# A New Approach to Image-Based Estimation of Food Volume
## 2 Method
### Image-Based Modeling  
In this step, a 3D model of the food item in the form of a point-cloud is generated based on a Structure from Motion (SfM) algorithm. We followed the same approach described in [27; 28], properly adapting it to this work. In fact, the base approaches have been designed to build a model based on many (up to a few thousands) images. Our algorithm is customized for using a lower number of images (six, in our case) and address processing time-related concerns. Algorithm 4 reports the pseudo-code of the method, while Figure 6 shows three examples of the reconstructed models. ```
functionModel(OriginalImages[], SegmentedImages[], CameraMatrix) \(Matches[]\leftarrow\textbf{Extract-SURF-BRISK-Matches}(OriginalImages[])\)\(\triangleright\) Prepare Matches \(Matches[]\leftarrow\textbf{PruneOutliers}(Matches[])\)\(BaseLine\leftarrow\textbf{FindTheBestPair}(Matches[])\)\(\triangleright\) Start Modeling \(PointCloud\leftarrow\textbf{Triangulate}(BaseLine,Matches[],CameraMatrix)\) while there is more un-processed image do \(NextImage\leftarrow\textbf{ChooseNextImage}(PointCloud,Matches[])\)\(NewMatrix\leftarrow\textbf{FindPoseAndCameraMatrix}(PointCloud,Matches[],CameraMatrix)\)\(PointCloud\leftarrow\textbf{PointCloud}+\textbf{Triangulate}(BaseLine,Matches[],NewMatrix)\) endwhile \(Corners\leftarrow\textbf{FindCheckerboardCorners}(OriginalImages[],CameraMatrix)\)\(\triangleright\) Find Checkerboard \(Scale\leftarrow\textbf{EstimateSizeScale}(Corners)\)\(SurfacePlane\leftarrow\textbf{EstimateSurfacePlane}(Corners)\)\(PointCloud\leftarrow\textbf{EliminateNoise}(PointCloud,SurfacePlane)\)\(\triangleright\) Refine PointCloud returnPointCloud, Scale endfunction
```  
**Algorithm 4** 3D modeling algorithm  
Figure 6: Three examples of models reconstructed by the proposed system. First, Speeded Up Robust Features (SURF) [29] of the images are extracted and their Binary Robust Invariant Scalable Keypoints (BRISK) [30] description is used to find matches between image pairs. A pair is considered to be relevant if it produces at least 30 matches, in which case the corresponding points are added to the point-cloud. There are 15 possible pairs in total to be taken into consideration for six images, which is generally a large number, which could affect the total processing time significantly. Therefore, only a limited number of image pairs, configurable by the user, are actually used instead of all possible pairs. Then, the Fundamental Matrix [31] is calculated for all image pairs using an 8-point Random Sample Consensus (RANSAC) [32]. The RANSAC outliers threshold is set to 0.6% of the maximum image dimension (i.e., the largest between height and width) and the level of confidence is set to 99%. The results of RANSAC are therefore also used to \"clean\" the point-cloud by removing outliers. In the next step, the best image pair is chosen as the starting point for the 3D reconstruction, using the results of a homographic transformation as a selection criterion. Homography is the mathematical term for mapping points on one surface to points on another. The transformation between images of each pair is calculated using the RANSAC with the outliers threshold set to 0.4% of the largest image dimension. This method, besides finding the transformation, classifies the matching features into inliers and outliers. The pair with the largest inlier rate is chosen to start the reconstruction. The quality of the selected image pair is further evaluated by checking the validity of its Essential Matrix [31]. If the selected pair fails the test, the next best pair is chosen as the baseline of the 3D reconstruction. Then, the 2D coordinates of the matching points belonging to the selected images are used to calculate the corresponding 3D coordinates.