# Food Classification and Meal Intake Amount Estimation through Deep Learning
## 2 Related Works for Food Detection and Meal Intake Amount Estimation  
Research on the food type classification and the meal intake amount estimation is classified into methods by sensors and based on images. The food type and meal intake amount can be calculated by the sensors that measure sound, pressure, inertia, and so on, caused by eating food. The meal intake amount is measured by attaching weight or pressure sensors to a tray where meal plates are placed [19; 20]. However, the estimation methods by the tray with the sensors have the disadvantage of extremely limited mobility. Some wearable devices can be utilized to measure the meal intake amount [21; 22; 23]. [PERSON] et al. [21] classifies the food type and measures the meal intake amount by template matching on the characteristics of sounds generated by chewing. [PERSON] et al. [22] recognizes the food types and the meal intake amount through a smartwatch with a three-axis accelerometer. However, these methods cannot estimate the meal intake amount while the wearable device is not worn. Recently, the technology for object detection in images has greatly been improved due to the development of artificial intelligence technology through deep neural networks. Various studies [2; 3; 4; 5] for the food type classification have been conducted through object detection networks such as YOLO [24], VGG19 [25], AlexNet [26], and Inception V3 [27]. [PERSON] et al. [28] pre-processes the meal image through morphological operators before detecting foods through an object detection network. The food detection methods through deep neural networks outperform the traditional methods [29], which are scale invariant feature transform (SIFT), histogram of oriented gradients (HOC), and local binary patterns (LBP). However, it is very difficult to measure the meal intake amount through only one image. For the image-based methods, the meal intake amount should be measured by images captured from two or more viewpoints, an RGB-D image, or prior modeling of the foods. The methods based on multiple images [6; 7] measure the food volume by reconstructing the 3D shape through correspondences among the pixels of images. [PERSON] et al. [6] finds some feature points in the food images captured with stereo vision in order to generate a point cloud that is a set of points on 3D space. The food volumes are measured through the point cloud for images. The food volume can also be estimated through the RGB-D image, which is added as a channel for the distance to subjects [8; 9; 10; 11]. [PERSON] et al. [8] detects the food regions and classifies the food types through convolution layers through RGB channels of the captured image. The food volumes are estimated by reconstructing 3D surfaces through a depth channel that has distance information. The 3D shape can be predicted by pre-modeling for the template of the food or bowl [30; 31; 32]. [PERSON] et al. [30] estimates the food volumes based on a known bowl shape. The food volume estimation through a single image requires a distinct reference object [12; 13; 14]. The meal intake amount can be estimated by the ratio of the number of pixels between the food and the reference object regions [12]. However, this estimation method has a large error for thin food. [PERSON] et al. [13] calculate a projection model from a food object to the image through the reference object region. After that, one 3D shape type among sphere, ellipsoid, and cuboid is manually assigned to the food region to estimate the food volume. [PERSON] et al. [14] crop a food region without background through Faster R-CNN, Grabcut algorithm, and median filter. The relative -food volume is estimated through the CNN network that learns the relationship between the food image without background and the food volume. The actual volume is calculated through the area ratio between the food and the size-known reference object. The estimation methods based on the reference object have an inconvenience where the reference object should be captured with food.