# A New Approach to Image-Based Estimation of Food Volume
## 2 Method
### Image-Based Modeling
Assuming \(P_{i}\) to be the camera matrix of the \(i^{th}\) image and \(x_{i}\) and \(x_{j}\) the 2D coordinates of two matching points in the \(i^{th}\) and \(j^{th}\) image, the relationships \(x_{i}=P_{i}X\) and \(x_{j}=P_{j}X\) hold, where \(X\) denotes the 3D coordinate of the projected point. Moreover, for every point that is not too close to the camera or at infinity, one can say \(X=(x,y,z,1)t\), where \(t\) is the scaling factor. This creates an inhomogeneous linear system of the form \(AX=B\). The least-squares solution of this equation can be found by using the Singular Value Decomposition (SVD). Unfortunately, this method is not very accurate. The method introduced by [PERSON] and [PERSON] [33] iteratively re-calculates \(X\) by updating \(A\) and \(B\) based on the results of the previous step. This allows one to refine the estimate and to significantly increase accuracy. The other images are then successively taken into consideration in the triangulation process. Since the 3D coordinates can be estimated only up to a scaling factor \(t\), the 3D points belonging to different image pairs might have a different scale factor, and it may be therefore impossible to build a consistent 3D reconstruction. The other issue is the order in which the different views are taken into account. Taking into account a lower image-quality view before the others may affect the final model negatively. In order to tackle these issues, we first choose the best view among the remaining ones by calculating, for each view, the number of features that produce successful matches with the images that have been previously processed, and selecting the one for which it is largest. Then, the camera position for the selected view is estimated based on the previously reconstructed points. This is a classic Perspective-n-Point problem (PnP), which determines the position and orientation of a camera given its intrinsic parameters and a set of \(n\) correspondences between 3D points and their 2D projections. The \(solvePnPRansac\) function, provided by OpenCV, which finds the pose that minimizes the reprojection error, is used to solve it. The use of RANSAC makes the function robust to outliers. This approach also ensures that the 3D points added to the point-cloud based on the new images share the same scaling factor as the previous ones. Bundle Adjustment [34] is also applied to the triangulation output in order to refine the results. It is an optimization step that provides a maximum-likelihood estimation of the positions of both the 3D points and the cameras [34]. This result is obtained by finding the parameters of the camera view \(P_{k}\) and the 3D points \(X_{i}\), which minimize the reprojection error, i.e., the sum of squared distances between the observed image points \(x_{ki}\) and the reprojected image points \(P_{k}(X_{i})\)[31]. In the next step, all of the non-object points are removed from the point-cloud based on the output of the segmentation stage. The requirement by which a 3D point must appear in at least two images compensates for some segmentation errors that might have occurred in some images. Finally, outliers are removed from the point-cloud. Since the ground surface has been determined based on the checkerboard location, it is possible to apply restrictions to the maximum acceptable height of the food item (we set such limit to 20 cm). The point-cloud usually also includes isolated outliers that are relevantly far from the other points. To deal with this problem, we remove from the point-cloud the points whose distance from the closest neighbor is larger than a threshold. In practice, for each point, four types of distances are defined: the Euclidean distance and the distances along the three coordinate axes. Acceptable points should be no farther from their nearest neighbor than one standard deviation for each of the measures taken into consideration. Moreover, if an isolated set of less than five points is too far from its neighbors, it will be removed as well.