# A New Approach to Image-Based Estimation of Food Volume
## 2 Method
### Segmentation  
The objective of the segmentation step is to detect the exact location of the food items in the images. Food image segmentation can be very challenging. There are very different ways of cooking the same kind of food and of arranging food on a dish. These problems and many other factors cause food appearance to vary dramatically. Some studies impose constraints when taking pictures of food, in order to make segmentation easier. For example, [PERSON] [13] assumes that food lies on a dish that is brighter than the table cloth. Several other studies [14; 15; 16] impose that the dish, or the container where the food lies, has a pre-specified shape. Recently, attention has been paid to semi-automatic methods. In [17; 18], the user is requested to draw a bounding box around the food items, while in [19], the user must mark the initial seeds before starting to grow segments. In this work, a semi-automatic method is designed which can be run on smartphones in a user-friendly manner. The method is based on a customized interactive version of the graph-cut (GC) algorithm [20], which deals with image segmentation as a pixel-labeling problem. The graph-cut algorithm assigns a \"foreground\" (food) or \"background\" (not food) label to each pixel. We evaluated the general effectiveness of this method in a previous work [11]. In our customized version of GC, the user imposes some hard constraints by marking some food and non-food areas  
Figure 2: The camera positions of the six selected frames are shown along with the reconstructed model of a sample object. in one of the images. The application provides a user-friendly environment for marking images by simply touching the screen (see Figure 1). Algorithm 2 reports the pseudo-code of this process. The marked parts of the image are fed into a Gaussian Mixture Model (GMM) algorithm [21] that estimates the distribution of the Red-Green-Blue (RGB) values for the image foreground (food) and background (not food). The K-means algorithm is then used to divide the data from each RGB channel into ten clusters that are assumed to follow a normal distribution. Then, the GMM estimates the distribution by combining the normal models. Next, a value is assigned to every pixel, which denotes the likelihood of its belonging to the foreground or the background, based on the distribution estimated by the GMM. These likelihood values are used to seed the graph-cut algorithm, which produces the final segmentation. However, if the user is dissatisfied with the segmentation output, he/she can repeat the marking step to improve the result. ```
functionSegmentation(OriginalImage, MarkedImage) \(Pixels_{F}\leftarrow\textbf{ExtractFoodPixels}(OriginalImage,MarkedImage)\) \(Pixels_{B}\leftarrow\textbf{ExtractBackgroundPixels}(OriginalImage,MarkedImage)\) \(GMM_{F}\leftarrow\textbf{ComputeFoodGMM}(Pixels_{F})\) \(GMM_{B}\leftarrow\textbf{ComputeBackgroundGMM}(Pixels_{B})\) \(Like_{F}\leftarrow\textbf{CalcFoodLikelihood}(OriginalImage,GMM_{F})\) \(Like_{B}\leftarrow\textbf{CalcBackgroundLikelihood}(OriginalImage,GMM_{B})\) \(Segments\leftarrow\textbf{GraphCut}(Like_{F},Like_{B})\) returnSegments endfunction
```  
**Algorithm 2** Segmentation of a marked image using GMM and graph-cut  
Although the input to the segmentation method we propose consists of the six images which are automatically extracted from the short video, the user needs to mark only one of them. The segmentation for such an image obtained based on the marked pixels is then used to extend the process to the other images by recomputing the GMM on the whole segmented image and using such a more accurate estimate as the model used in segmenting the others. This is based on the primary assumption that the images acquired are characterized by a similar color distribution. Figure 3 shows the segmentation of four food images obtained by our method. Being based on six images, a small segmentation error in one image can be recovered by the results obtained on the other images.