# Food Classification and Meal Intake Amount Estimation through Deep Learning
## 3 Food Classification and Meal Intake Amount Estimation through Deep Learning
### Meal Intake Amount Estimation  
The meal intake amounts are estimated as the differences in the food volumes between the pre- and post-meal images. Estimating a volume from an image is known to be a very challenging task. Nevertheless, we propose a method of the food volume estimation by assuming that the foods on the meal plate have a few simple 3D shape types. In the proposed method, the food is modeled in a 3D shape according to the type and the area as shown in Figure 7.  
Figure 6: Image correction to have same capturing environment: (**a**) pre- and post-meal images; (**b**) rectangles and vertices enclosing meal plate regions; (**c**) corrected post-meal image by homography transformation; (**d**) food regions of post-meal image before correction; (**e**) food regions after correction. Figure 7: Flow of food volume estimation. Food has a specific 3D shape depending on its food type. Foods in the rice and the soup categories are similar to a spherical cap shape as shown in Figure 8a. In Figure 8b, the foods which consist of bunches of an item are shaped like cones. Figure 8c shows that the shape of the food is close to cuboids if the food is one chunk. The proposed method determines the 3D shape types for the characteristic of food as a spherical cap, a cone, or a cuboid shape. Table 2 shows 3D shape types by the food types. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Category** & **Food Name** & **3D Shape Types** & **Category** & **Food Name** & **3D Shape Types** \\ \hline Rice & Rice & spherical cap & & Fruit salad & Cone \\ \hline \multirow{4}{*}{Soup} & Bean sprout soup & spherical cap & & Kimchi & cone \\  & Miso soup & spherical cap & & Pepper seasoned & cone \\  & Radish leaf soup & spherical cap & & Seastring seasoned & cone \\  & Seaweed soup & spherical cap & & Stir-fried fish cake & cone \\ \hline \multirow{4}{*}{Side-dish} & Tofu & cuboid & & Stir-fried mushroom & cone \\  & Jeon & cuboid & & Stir-fried octopus & cone \\  & Stewed fish & cuboid & & Stir-fried pork & cone \\ \cline{1-1}  & Grilled fish & cuboid & & Stir-fried squash & cone \\ \cline{1-1}  & Eggplant & cone & & Yellow pickled radish & cone \\ \hline \hline \end{tabular}
\end{table} 
Table 2: 3D shape types for food types in proposed method. Figure 8: Shapes of foods: (**a**) spherical cap shape type; (**b**) cuboid shape type; (**c**) cone shape type. The food volume is estimated through the base area and the 3D shape types. The base area is calculated through the actual meal plate area as follows:  
\[A_{food}=\frac{n(p)}{n(f)}\times A_{plate}, \tag{1}\]  
where \(n(. )\) is the number of pixels; \(p\) and \(f\) are the regions of the meal plate and the food, respectively; and \(A_{food}\) and \(A_{plate}\) are the base area and the meal plate area, respectively. Even though the base area and the shape type are determined, the 3D shape can be different. For example, the shape of a cone with a specific base depends on its height. It is very difficult to estimate the height through one image. However, the foods cannot be sloped beyond a certain angle, which is the angle of repose. Though the angles of repose are different depending on the material of the food, the proposed method supposes that the angle of repose is 30 degree which is the average angle [36]. In other words, the food is assumed to have the slope angle of 30 degree. The food volume is estimated through the 3D shape type of the food. For the food of the spherical cap type, the food height \(h\) is equal to \(R-a\) as shown in Figure 9a, where \(R\) is the radius of the whole sphere, and \(a\) is a distance between the spherical center and the base area.