# Smart Diet Diary: Real-Time Mobile Application for Food Recognition
## 3 Materials and Methods
### Volume Estimation  
The criterion for selecting a reference object was that it should be a common object possessed by most individuals or is available at the time of undergoing a diet, such as ID cards, thumbnails, or utensils. The problem with these objects was that they had either asymmetrical forms or varying sizes. We decided to use a NZD 2 coin because it is circular in shape and the horizontal and vertical pixels are constant. Additionally, it is easy to carry and people possess it most of the time. Other available coin options were also considered; however, NZD 2 coin was preferred due to its suitable size for the application. To determine the volume of an item from a 2D image, we applied segmentation methods such as GrabCut algorithm. It is an iterative algorithm that incorporates statistics and GraphCut to obtain exhaustive 2D segmentation. We used it to partition the image by extricating the foreground (food and coin) to estimate their sizes by counting pixels. This process only involves a coarse separation between the foreground and background. This is achieved by sketching a rectangle around the desired item. We carried out coarse partitioning during the classification stage, which returns a bounding box and label. GrabCut erases interior pixels that do not belong to an item. The steps are explained below in Algorithm 1. ```
1. Initialize the pixels in the foreground
2. Initialize the pixels in the background
3. Initialize the pixels of unknown part of the image
4. Use Orchard-Bouman clustering algorithm to model Foreground and Background as Gaussian Mixture Model. 5. Each pixel is assigned a Gaussian component most likely in the foreground GMMs and the rest is background GMMs. 6. GMMs newly learned from a collection of pixels that have been created in step 5 and 7. 7. Graph Cut is created, and it is used to search for a new classification of the pixels of the foreground and background. 8. Repeat steps 4-6 until classification is completed

Figure 5: Running faster R-CNN on TensorFlow. The foreground, background, and the unidentified portion of the picture are fed to the algorithm by sketching a rectangle encompassing the object of interest. The initial image segmentation is created with pixels inside the bounding box considered as the foreground and the pixels outside as the background. Both foreground and background are modeled as a Gaussian mixture model (GMM) by means of the Orchard-Bouman clustering algorithm [68]. GMMs are used to represent normally distributed subpopulations within an overall population without the knowledge of a relationship between subpopulation and data point. Each image is fed as an array of pixels, where each pixel represents the intensity, and GMM partitions the pixels into similar segments for further analysis. Each pixel in the foreground is given the most likely Gaussian factor and pixels in the background are treated in the same way. New GMMs are learned from the pixel sets created in the previous step. The Orchard-Bouman clustering algorithm is used, which provides an optimal solution for clusters with Gaussian distributions. GrabCut then finds a new classification of foreground and background pixels. This process is reiterated until the algorithm converges. The outcome of the segmentation was the food and reference item on a dark background. This technique suffers from the limitation that the height of the food cannot be calculated using this method and we can only provide an average serving size of items based on the shape of food items stored in the server database and use the corresponding volume equation to calculate the volume of the food. The code for volume estimation is provided in Appendix A. ### Calorie Estimation

Determining a food's calorie content through a single image is a complicated problem. It is difficult to obtain an accurate estimation without additional information. Existing studies use cameras with depth sensors or capture multiple pictures of food from different angles [14].