# Accurate Real-Life Chinese Dish Recognition

Authors: Shanzhen Lan; Chengjuan Wan; Yuxuan Pang; Mingxue Jin; Shaode Yu
Journal: International Transactions on Electrical Energy Systems
 (2022)

---

## Abstract

Deep learning is a new research direction in the field of machine learning, which was introduced into machine learning to bring it closer to its original goal. Accurate dish recognition becomes increasingly important in the multimedia community since it can help cuisine recommendation, calorie management, service improvement, and other food computing tasks. Many novel approaches have been developed on web recipes and menu pictures, while few are concerned real-life dish image analysis. In this study, a deep learning-based prototype system is deployed in a Chinese canteen, and 28 dish types, 16,904 images, and 45,061 instances have been collected. Specifically, in the prototype system, three practical issues are explored, including the backbone network selection, the training strategy determination, and the minimum number of samples for model upgrading. Experimental results suggest that fine-tuned Faster-RCNN can serve as the backbone network of the prototype system since it outperforms the other four fine-tuned networks on dish recognition (accuracy, 98.10%; recall, 97.20%; MAP (mean average precession), 98.30%) and satisfies real-time requirement (0.15 second per image). Meanwhile, the transferred backbone network achieves superior results (MAP, 96.48%) over the same architecture trained from image scratches (MAP, 87.84%). On model upgrading, a good (MAP, 91.34%) to better (MAP, 96.48%) outcome is obtained when the training size is increased from 50 to 200 samples per dish type, and 150 and more instances should be annotated if a new dish type is added to the system’s recognition list. Conclusively, the real-life deployment and evaluation of the prototype system indicate that deep learning is full of potential to enhance customer experience through accurate daily dish recognition.


---

## Body

## 1. Introduction

Accurate dish recognition becomes increasingly important in the multimedia community. It is useful in cuisine retrieval and recommendation, nutritional intake monitoring, food service improvement, and many other food computing applications [1]. On the other hand, it is yet very challenging. Except for the difference in the appearance of shape, texture, and color, and the difference in ingredient composition, cooking styles, and procedure attributes, dishes may be served in different environmental conditions, which causes large category variation in recognition context. All these issues pose difficulties inaccurate dish recognition.To address these challenges, many novel methods have been developed. In terms of the backbone architecture, dish recognition methods can be broadly divided into multistage approaches and end-to-end approaches. The former combines feature extraction and dish classification. Representative work includes the analysis of visual appearance and ingredient composition, the geolocalized modeling of restaurants, menus and dish images, and the use of other external information[2, 3]. Pouladzadeh developed a recognition method using support vector machine (SVM), and it involves the combinations of food shape, color, size, and texture characteristics [2]. Kawano and Yanai proposed a mobile approach to estimate nutrition, calorie, and users’ eating habits of food [4]. The approach integrates human interaction for food localization, Fisher vector representation of histogram of gradient and color patch as computational features, and linear SVM for food categorization. Herranz formulated the dish recognition problem as a probabilistic learning model that accounts for dish images, restaurant names, and geolocation information [5]. Furthermore, Xu incorporated the discriminative messages, including dish types, restaurants, and geolocalized settings, for the refinement of dish image recognition, and notably, geolocation and geolocalized models contribute to the improvement of food recognition performance. [6]. He presented a food classification method by using a multi kernel SVM, and dish images are acquired from several views [3]. Specifically, the method detects dish ingredients by combining deformable parts, and then, a texture assessment model is applied for food categories classification. In general, it takes massive time and labor for multistage approaches to select proper features and classifiers for enhancing dish recognition performance.Thanks to the development of deep learning architectures [7–12], the availability of massive data samples [13–17], and the upgrade of computational hardware, end-to-end approaches help boost the performance of dish recognition [18–27]. Wu exploits the semantic relationship among fine-grained food categories, and a multitask learning procedure is added to a convolutional neural network (CNN) [27]. Moreover, by using a smoothing procedure, the prediction of food classes is refined. Bolanos and Radeva adopted GoogLeNet [7] for simultaneous food localization and recognition [19]. The model uses global average pooling to produce bounding box proposals, and the bounding boxes are identified as food and nonfood objects. Wang designed a partially asymmetric multitask CNN for simultaneous recognition of restaurant locations and food categories from dish images [20]. The network contains a pathway to acquire the semantics of dish types and a pathway to find out the restaurant’s identity. To relieve the burden of data collection and the overfitting problem, Aguilar integrates multiple classifiers-based convolutional models, and the food recognition performance is improved [21]. Ege and Yanai proposed multitask learning [22]. They modify YOLO [8] with additional feature map output for dish detection and calorie estimation. Martinel develop a slice convolutional module layer accompanied by a deep residual neural network for dish recognition [23]. Through such network, good representation for dishes depends on nonspecific structure, leading to better performance on various datasets. Min presents a multiattention neural network [24]. The network aims to sequentially localize multiple informative food regions by using multiscale guidance from a coarse level of category to a fine level of ingredient analysis [13]. It generates an attentional image region from category-level supervision and then uses a long-short time memory network and spatial transformer to discover and identify potential image regions with ingredient-level scales. The method is evaluated on the dataset ISIA Food-200 which contains 200 food types, 319 ingredients, and about 200,000 images. Park prepared a Korean food dataset and design a deep learning network for dietary management [25]. The model outperforms the other evaluated deep networks on the dataset with accuracy of 91.3%, and time cost of 0.4 ms per image. Xiao presented a simple perception learning model [26]. They design a jumping convolution module to extract image features of food regions for reducing the CNN complexity, and in addition, a preprocessing step by image editing is used to get different visual cues of dish images. Gao proposed a neural network that applies handcrafted and deep learned features as well as local and global features for dish image recognition and dish health assessment. They introduce a local attention mechanism for localizing the key areas of the dishes, extract low-level features of ingredients and colors for learning deep features, and attention mechanisms using local- and global-scale feature analysis are combined to predict the dish/food taste as the outcome. Overall, massive dish samples are indispensable to train an end-to-end approach because hyper-parameters should be optimized.Meanwhile, according to the dish types presented on one plate, the methods can be generally grouped into single and mixed dish recognition approaches, and most methods concentrate on the classification of single dish types [2–6, 18–27]. In comparison to a single dish scenario, a mixed dish contains various dish types overlaid on one plate, and this kind of food is popular in China, Singapore, and other Southeast and Eastern Asian regions [28]. The major difficulty of mixed dish classification comes from the served presentation of different dishes and in particular, these dishes are overlaid with no clear border. To tackle these problems, Deng built a contextual relation network to encode the implicit contextual relationship among multiple dishes from region features as well as to transform the explicit contextual relationship from label-level co-occurrence analysis. Then, they use a domain adaption network to arrange local and global image features, decrease visual variances of dish instances, and eliminate domain gaps of dish features served in different restaurants [29]. Nagarajan designed a novel transfer learning framework for multilabel dish image recognition [30]. The framework aims to leverage the knowledge learned on simple single-label dish classification onto mixed dish recognition. Wang also presented a multilabel learning framework [31]. It integrates region-level prediction to detect food regions and multiscale analysis to tackle the difference in dish sizes, and two mixed dish datasets are built to assess the effectiveness and efficiency of the proposed framework.In this study, the real-life single-dish recognition problem is investigated. Many works have been devoted to dish recognition, most of them are tested on web images, while few concern the deployment and upgrading of dish recognition systems in real scenes. Specifically, a deep learning-based daily dish recognition system is presented and deployed in a Chinese canteen, and the contributions are summarized as follows:(1)
In a Chinese canteen, 28 dish types, 16,904 images, and 45,061 instances are prepared. Notably, the number of dish types, images, and instances is increasing offline for future dataset expansion and system updating.(2)
A prototype system is built and deployed in the canteen, and three practical issues are explored, including the backbone network selection, the training strategy determination, and the minimum size of samples for model upgrading.(3)
Experimental results show that fine-tuned Faster-RCNN serves as the backbone since it outperforms the other four transferred deep networks. Based on the backbone, the strategy of fine-tuning is found more effective than that of training from image scratches. In addition, 150 and more data samples per dish type can lead to satisfactory results on model upgrading.The remaining of this paper is organized as follows: Section2 briefly presents the related works of real-life dish recognition. Section 3 describes the prototype system, the Chinese dish dataset, and the experiment design. Section 5 illustrates the experimental results, including the selected backbone network and training strategy, and the minimum size of samples for model upgrading. The findings, limitations, and conclusion are presented in Section 6.


## 2. Related Works

A dozen of works have concerned the deployment of food recognition systems on smartphone or on cloud for real-world dish image analysis [32–49]. As to multistage approaches, Kawano and Yanai implemented a food recognition system on smartphone with the purpose of recording calories, nutrition, and eating habits [32]. This multistage system adopts linear SVM, bounding box adjustment, and food region estimation, and the classification accuracy of top 5 candidates reaches 79.2% on a 100-category food dataset. Notably, the system needs no servers and runs on smartphone in real-time. Pouladzadeh and Shirmohammadi design a multifood classification system to quantify the nutrition and calorie of dishes [33]. The system deployed in the cloud server involves the manual selection of bounding circles, computational region selection and representation using SVM classifier, deep learning-based region proposal, and food name prediction. On the FooDD dataset, its average recall rate, precision rate, and accuracy are 90.98%, 93.05%, and 94.11%, respectively. Liu presented deep food on the edge computing service infrastructure [34]. The system involves front-end component for food image preprocessing, back-end component for deep learning-based food recognition, and communication component for data upload and download between front- and back-end components [35]. Cheng proposed a practical dish recognition method that balances recognition accuracy and time cost [36]. Generally, a detector is built to localize the dish position, a reidentifier is used to recognize registered dish categories, and an additional module is designed to estimate the dish attributes. The system deployed in embedded environments runs about 5 frames per second and the tray accuracy is 82.1% on the UNIMIB2016 dataset. Ming designed a food photo recognition app that is tested on different food groups (such as Chinese, fruit, Western, and Japanese) and the app is also verified on laboratory photos and real user pictures. The top-5 recognition accuracy of real user photos (≤71%) is inferior to that of laboratory test photos (≥89%), and therefore, a big gap in recognition performance exists between the use of laboratory photos and real-life food images [38].As to end-to-end approaches, Mezgec designed a deep learning network, NutriNet, for the detection and recognition of food and drink in images. The model is tested on 520 food and drink items and 225,953 images, and its classification accuracy reaches 86.72%. Moreover, it is evaluated on images taken by using hand-held cameras, and its top-five accuracy is also promising. Fakhrou also used smartphone for image acquisition of dishes and fruits as the input of an end-to-end recognition system [40]. The system utilizes ensemble learning to fuse two deep neural architectures, and its accuracy achieves 95.55% on twenty-nine categories of customized dataset that benefits children with visual impairments. Merler developed a food recognition engine on smartphone for dietary logging [44]. They consider foods in restaurants and foods in wild, and fine-tuning GoogLeNet is found superior over k-nearest neighbor and ensemble SVM on the Food-500 dataset [41]. On foods from 6 restaurant chains dataset, its top-3 accuracy reaches 92.1%, while on foods across wild datasets, its accuracy should be further improved [43]. Aguilar proposed a deep network as a semantic food detector for smart restaurants [46]. It casts automatic food tray analysis into three parts of food localization, separation and recognition, and experiments on the dataset UNIMIB2016 (image size, 3264 × 2448) can achieve F-measure larger than 90%. Horiguchi build a sequential personalized classifier for food recognition on food-logging applications [45]. Notably, a simple classifier for each user is gradually personalized by using new dish samples from existing or related types, even if the dish type has a very small number of image samples. This effective framework integrating the mean classifier and deep learned features shows higher recognition accuracy over existing methods on realistic image analysis. Sundaravadivel designed a consumer electronics product by using an open internet-of-things platform for dish image analysis and storage [48]. It is embedded with a novel five-layer perceptron network and a Bayesian network for precise recognition. On 1000 meals and 8172 food items, the classification accuracy is 98.6%. Anzawa attempt to recognize multiple foods or dish items in one single picture [49]. A buffet restaurant is investigated, and a limited number of images per class become available because the menus change each meal. In detail, deep features learned from the last layer of average pooling of ResNet-50 are compared to the template image feature vectors by a nearest-neighbor search for item recognition.Table1 shows several dish image recognition systems from approach categories and hardware (CPU, central processing unit; GPU, graphic processing unit), tested datasets, classification accuracy, and time consumption. Both multistage approaches and end-to-end approaches achieve high classification accuracy in an acceptable time cost (<60s) on tested datasets, while the performance of recognition systems on real-life images drops considerably. For instance, the multistage system [37] achieves top-5 accuracy of 89.9% on laboratory test photos and 69.8% on real user photos, and the performance of the end-to-end system [39] drops dramatically from NutriNet dataset (accuracy, 94.47%) and UNIMIB2016 (accuracy, 86.39%) to real-world test dataset (top-5 accuracy, 55%).Table 1
A summary of dishes’ image recognition systems.
Method and hardwareDatasetAccuracyTime cost[32]Multistage approachUEC-FOOD100Top-5 accuracy, 79.2%0.85sClient: CPU (1.6 GHz)(1) 100 classes(2) 12,905 images(3) >100 images per class[33]Multistage approachFooDDRecall 90.98%(1) 30 classes(2) 7,000 imagesPrecision 93.05%Accuracy 94.11%[34]Multistage approachUEC-100Accuracy 95.2%≈60sServer: GPU (12 GB)UEC-256Accuracy 81.8%Client: CPU (2.5 GHz)Food-101Accuracy 94.0%[35]Multistage approachLogMealTop-5 accuracy 94%(1) 800 + classes(2) 1,348,507 images[36]Multistage approachDish132Top-5 accuracy 99.5%0.74s(1) 132 classes(2) 16,000 images[37]Multistage approachLaboratory test photosTop-5 accuracy 89.9%0.26sReal user photosTop-5 accuracy 69.8%[38]Multistage approachUEC-FOOD100Top-5 accuracy 93.7%0.25s(1) 100 classesClient: CPU (2.5 GHz)(2) 12,905 images(3)>100 images per class[39]End-to-end approachNutriNet datasetAccuracy 94.47%Server: GPU (24 GB)UNIMIB2016Accuracy 86.39%Clinent: GPU (12 GB)Real-world test datasetTop-5 accuracy 55%[40]End-to-end approachFood and fruitsAccuracy 95.55%0.482sServer: GPU (6 GB)(1) 29 classesClient: CPU (2.6 GHz)(2) 31,127 images(3)>1000 images per class[41]End-to-end approachFood201-MultiLabelAccuracy 50%<1s(1) 201 classes(2) 50,374 images[42]End-to-end approachOffice-31Accuracy 97.71%Server: GPU (12 GB)Meal-300Accuracy 72.45%[43]End-to-end approachUEH-VDRAccuracy 92.33%(1) 9 classesPrecision 94.13%(2) 7,848 imagesRecall 90.82%[44]End-to-end approachFOOD-500Accuracy 97.2%Server: CPU (1.6 GHz)(1) 500 classes(2) 148,408 images[45]End-to-end approachFoodieCal datasetAccuracy 89.93%(1) 23 classes(2) 23,000 imagesOursEnd-to-end approachChinaDishSetAccuracy 98.10%≈0.15sServer: GPU (8 GB)(1) 28 classesRecall 97.20%Client: CPU (2.4 GHz)(2) 16,904 imagesMAP 96.48%(3)>200 samples per type


## 3. Materials and Methods

This section presents the prototype system, the procedure of data collection, cleaning and labeling, and the workflow of dishes plate image acquisition, dishes localization and recognition, and failure cases-based model upgrading. After that, the experimental design is described in detail, including backbone network selection, training strategy comparison, and the effect of the training sample sizes on dishes recognition performance.


### 3.1. The Prototype System

A practical prototype system has been designed and deployed in a Chinese restaurant as shown in Figure1. It consists of a video recording system, a deep learning-based dishes recognition system, and the user interface for dish-sale analysis. Deep learning is mainly to learn the internal laws and representation levels of sample data. In the process of learning, people can get a lot of information, such as words, sounds, and images. The video recording system contains a miniature camera, and digital images are acquired when a dishes plate is placed under the camera. The imaging parameters, such as image resolution and frame rate, are provided. The dishes recognition system is embedded with a well-trained deep network, and its purpose is accurate dishes localization and recognition in real-time. Specifically, it is linked to food prices and helpful for automatic charging. In addition, the user interface facilitates dish-sale analysis and smart canteen management.Figure 1
Prototype system for a Chinese restaurant.Figure1 shows a prototype system deployed in a Chinese restaurant for real-time accurate dish recognition. It is implemented with a microcamera for dishes plate image acquisition and a pretrained deep network for candidate region localization and dishes classification. In addition, the user interface is provided for dish-sale analysis and canteen management.


### 3.2. Data Collection and Preparation

An annotated dataset of Chinese dishes is prepared. It contains 16,904 images and will be made available online for the further development, comparison, and reproducibility of algorithms for dishes (or Chinese dishes) recognition.


#### 3.2.1. Data Cleaning

Primarily, 30 hours of videos (i.e., 2 hours shooting per day × 15 days) and 1,080,000 images of daily dish-sale were recorded. It is worth noting that an image generally contains more than one dish instance.In the data cleaning stage, several mathematical methods have been employed to compute the similarity of dish images, including histogram matching, Cosine similarity, Hamming distance, Euclidean distance, and perceptual hash algorithm. It was found Cosine similarity was suitable for its low complexity and good performance, and the threshold was set to 95% to screen out background images. Then, Baidu AI EasyData1 is employed to help clean the rest data after the trial-and-error test on a few free platforms. Next, the whole dataset was divided into 15 small ones according to the shooting date. At last, EasyData helped find out the images with motion blur, focus blur, atomization, noise, jumble, and repeated or high-similarity content to ensure the visual quality of acquired dishes images.


#### 3.2.2. Image Annotation

About 30,000 images remained after automated data cleaning. A close look found out that many images were with no dishes or with dishes in plastic bags and thereby, manual cleaning became necessary. It is worth noting that the removal of these useless images was fulfilled in the image annotation stage.Image annotation was again performed on the public intelligent platform of Baidu AI EasyData. First, about 30% of images were manually labeled, and each dish category was annotated in more than 10 instances. Then, the images with annotated instances were uploaded to train the model embedded in the EasyData tool. Next, the tool was used to label the remaining 70% images. In the end, the dataset was fully annotated. Furthermore, the manual check was conducted offline to ensure the labeling quality of image annotation.


#### 3.2.3. The Dataset ChinaDishSet

After removing these categories with less than 200 instances, 28 categories remain. Figure2 shows representative instances of each category. Due to the difference in lighting conditions, shooting time, food ingredients, cooking procedure, and background, the perceived quality of dish images varies.Figure 2
Representative samples of 28 categories of Chinese dishes in the ChinaDishSet dataset.The dataset ChinaDishSet contains a total of 45,061 instances. Figure3 shows the distribution of instance numbers in each dish category. It is observed that rice is the most popular food in daily diet and reaches 6,496 samples, followed by scrambled eggs with cucumber. According to the number of the dish with the most and the dish with the least annotated instances, the maximum imbalance ratio reaches 26.62.Figure 3
The distribution of dishes instances in each category.The largest category is rice with 6,496 samples, the smallest category is a dumpling with 244 samples, and the average number is around 1,609 instances per category.The average number of annotated data instances is 1,609 per category. In the stage of data cleaning, a category with less than 200 instances has been set aside for further dataset upgrading. In a Chinese restaurant, the categories of dishes change due to the join or leave of the chefs and the season food supplies. On the other hand, setting aside the categories with few samples can avoid the data imbalance which may cause an adverse impact on model training.


### 3.3. The Workflow of the Prototype System

The workflow of the deployed system shown in Figure4 contains dish plate image acquisition, deep learning-based dish localization and recognition, and failure cases (or new dish types) based on model updating.Figure 4
The workflow of the dish recognition system.It contains dish plate image acquisition, deep learning-based dish localization and recognition, and failure cases (or new dish types) based on model updating. Note that the failure cases and images of new dish types will be manually annotated for backbone training.


#### 3.3.1. Dish Plate Image Acquisition

After the prototype system is activated each time, the first task is to acquire an imageIt0 as the background. After that, how to determine whether a dish’s plate is under the camera, and how to define whether the plate is stationary, both become crucial for follow-up deep learning-based dishes recognition.In this study, assuming at timetk, a dish plate is detected on the way to be placed under the camera, and an image Itk is acquired that Itk−It0>ϵ. Then, a time interval δt should be defined to guarantee that the plate is stationary, i.e., Itk+δt−Itk≈0. Offline experiment indicates that the threshold of absolute intensity difference ε=104 and the time interval δt=0.2s satisfies the requirement of the prototype system.


#### 3.3.2. Deep Learning-Based Dish Localization and Recognition

The networks that can simultaneously localize and recognize dishes are considered. In this study, five deep networks off-the-shelf are investigated. Neural networks are all about taking a set of inputs, performing progressively complex computations on them, and providing outputs to solve real-world problems like classification. The networks include YOLO, SSD, Faster-RCNN, RetinaNetand Cascade-RCNN. The network architectures are shown in Figure5. In general, YOLO casts object detection as a regression task by associating isolated bounding boxes to class probabilities, and it can process images in real time. SSD discretizes the output bounding boxes into a series of default rectangles with different scales and aspect ratios according to the location of feature maps, and in default boxes, the presence of each object category is scored and predicted [9]. Faster-RCNN adds a region proposal network that predicts object boundaries and objectless scores at potential positions at the same time [10]. RetinaNet develops the focal loss with the purpose of training on a sparse set of difficult examples, and the network addresses data imbalance by weighting the lossless to these easy examples [11]. Cascade-RCNN is composed of a series of sample detectors that are trained with intersection over union for improving hypotheses quality [12]. Notably, the implementation of the five deep networks is available online.Figure 5
The architectures of deep neural networks of YOLOV3, Faster-RCNN, SSD, RetinaNet, and Cascade-RCNN.Their performance with different training strategies on dishes recognition is compared for selecting the backbone network of the dish recognition system. Different recognition performance is mainly manifested in the recall rate, precision rate, and recognition speed in the recognition process.


#### 3.3.3. System Upgrading

The prototype system will be upgraded in two situations. The difference in recognition performance mainly lies in the comparison of recall, precision, and recognition speed in the recognition process. One is to handle wrongly classified dish images. Intuitively, failure cases are representative of some environmental change and unobservable dish patterns, and thereby important and valuable. In failure cases, the dishes in the plate images are annotated manually. When the number of plate images reaches 200, the system is ready for updating. The other is to add new dish types into the prototype system. To guarantee recognition performance, the minimum number of dish instances should be well considered.


### 3.4. Experiment Design

Table2 lists the parameters in model fine-tuning, including the batch size, number of epochs, base learning rate, weight decay, and its coefficient and optimizer. If the learning rate is too large, the loss function may directly exceed the global optimal point. If the learning rate is too small, the change rate of the loss function is very slow. The learning rate decay type is PiecewiseDecay, and the decay scheduler is set at the 25th, 30th, and 40th epoch. If not specified, the parameters of deep neural networks are set as default.Table 2
The default parameters used in model fine-tuning.
Cascade-RCNNRetinaNetFaster-RCNNSSDYOLOV3BatchSize222128Epochs1515151515BaseLearningRate1.0×10−51.0×10−51.0×10−56.25×10−51.56×10−5WeightDecayL2L2L2L2L2WeightDecayCoeff1.0×10−41.0×10−41.0×10−45.0×10−55.0×10−4OptimizerMomentumMomentumMomentumAdamMomentumMomentum0.90.90.90.9


#### 3.4.1. Candidate Deep Networks for Backbone Selection

On the performance comparison of deep networks on dish recognition, 10 times of experiments are conducted. At each time, the 16,904 images in the ChinaDishSet are divided into a training set with 80% images and a testing set with the rest images. To a deep network, the performance is quantified with the classification accuracy, recall, and mean average precision (MAP) on average.The five networks [8–12] have been well trained on the dataset Microsoft COCO [14]. In this part, the networks are fine-tuned on the training set of images and further inference on the testing set. The parameters shown in Table 2 are used.


#### 3.4.2. Deep Network Training Strategies

After the backbone network is determined, two different training strategies, fine-tuning the pretrained backbone and using the backbone network trained from image scratch, are conducted, and the classification performance is compared.At this time, the parameters for fine-tuning are set as shown in Table2, while for training the backbone network from image scratches, the parameters are set as the same except for the base learning rate which is set at 0.005. In addition, different numbers of data instances are considered in the training stage.


#### 3.4.3. The Training Data Size for System Upgrading

The effect of training data sizes on dish recognition performance is explored. It is highly related to the system upgrading, and the minimum number of data samples should be annotated when a new category is added to the dish recognition list of the system. To address this issue, in the experiment, the training data size of each category is increased from 50 to 200 with an equal interval of 50 samples, and the classification metrics are analyzed.


### 3.5. Software and Platform

The algorithms are implemented with PyTorch (version 1.10.1) andPython (version 3.7). The deep learning framework PaddlePaddle (version 1.8.4) is deployed on a workstation (Windows 10, 64 bit OS, 12-cored CPU (2.4 GHz), 40 GB RAM) with one GPU card (NVIDIA Tesla P4, 8.0 GB)).


## 3.1. The Prototype System

A practical prototype system has been designed and deployed in a Chinese restaurant as shown in Figure1. It consists of a video recording system, a deep learning-based dishes recognition system, and the user interface for dish-sale analysis. Deep learning is mainly to learn the internal laws and representation levels of sample data. In the process of learning, people can get a lot of information, such as words, sounds, and images. The video recording system contains a miniature camera, and digital images are acquired when a dishes plate is placed under the camera. The imaging parameters, such as image resolution and frame rate, are provided. The dishes recognition system is embedded with a well-trained deep network, and its purpose is accurate dishes localization and recognition in real-time. Specifically, it is linked to food prices and helpful for automatic charging. In addition, the user interface facilitates dish-sale analysis and smart canteen management.Figure 1
Prototype system for a Chinese restaurant.Figure1 shows a prototype system deployed in a Chinese restaurant for real-time accurate dish recognition. It is implemented with a microcamera for dishes plate image acquisition and a pretrained deep network for candidate region localization and dishes classification. In addition, the user interface is provided for dish-sale analysis and canteen management.


## 3.2. Data Collection and Preparation

An annotated dataset of Chinese dishes is prepared. It contains 16,904 images and will be made available online for the further development, comparison, and reproducibility of algorithms for dishes (or Chinese dishes) recognition.


### 3.2.1. Data Cleaning

Primarily, 30 hours of videos (i.e., 2 hours shooting per day × 15 days) and 1,080,000 images of daily dish-sale were recorded. It is worth noting that an image generally contains more than one dish instance.In the data cleaning stage, several mathematical methods have been employed to compute the similarity of dish images, including histogram matching, Cosine similarity, Hamming distance, Euclidean distance, and perceptual hash algorithm. It was found Cosine similarity was suitable for its low complexity and good performance, and the threshold was set to 95% to screen out background images. Then, Baidu AI EasyData1 is employed to help clean the rest data after the trial-and-error test on a few free platforms. Next, the whole dataset was divided into 15 small ones according to the shooting date. At last, EasyData helped find out the images with motion blur, focus blur, atomization, noise, jumble, and repeated or high-similarity content to ensure the visual quality of acquired dishes images.


### 3.2.2. Image Annotation

About 30,000 images remained after automated data cleaning. A close look found out that many images were with no dishes or with dishes in plastic bags and thereby, manual cleaning became necessary. It is worth noting that the removal of these useless images was fulfilled in the image annotation stage.Image annotation was again performed on the public intelligent platform of Baidu AI EasyData. First, about 30% of images were manually labeled, and each dish category was annotated in more than 10 instances. Then, the images with annotated instances were uploaded to train the model embedded in the EasyData tool. Next, the tool was used to label the remaining 70% images. In the end, the dataset was fully annotated. Furthermore, the manual check was conducted offline to ensure the labeling quality of image annotation.


### 3.2.3. The Dataset ChinaDishSet

After removing these categories with less than 200 instances, 28 categories remain. Figure2 shows representative instances of each category. Due to the difference in lighting conditions, shooting time, food ingredients, cooking procedure, and background, the perceived quality of dish images varies.Figure 2
Representative samples of 28 categories of Chinese dishes in the ChinaDishSet dataset.The dataset ChinaDishSet contains a total of 45,061 instances. Figure3 shows the distribution of instance numbers in each dish category. It is observed that rice is the most popular food in daily diet and reaches 6,496 samples, followed by scrambled eggs with cucumber. According to the number of the dish with the most and the dish with the least annotated instances, the maximum imbalance ratio reaches 26.62.Figure 3
The distribution of dishes instances in each category.The largest category is rice with 6,496 samples, the smallest category is a dumpling with 244 samples, and the average number is around 1,609 instances per category.The average number of annotated data instances is 1,609 per category. In the stage of data cleaning, a category with less than 200 instances has been set aside for further dataset upgrading. In a Chinese restaurant, the categories of dishes change due to the join or leave of the chefs and the season food supplies. On the other hand, setting aside the categories with few samples can avoid the data imbalance which may cause an adverse impact on model training.


## 3.2.1. Data Cleaning

Primarily, 30 hours of videos (i.e., 2 hours shooting per day × 15 days) and 1,080,000 images of daily dish-sale were recorded. It is worth noting that an image generally contains more than one dish instance.In the data cleaning stage, several mathematical methods have been employed to compute the similarity of dish images, including histogram matching, Cosine similarity, Hamming distance, Euclidean distance, and perceptual hash algorithm. It was found Cosine similarity was suitable for its low complexity and good performance, and the threshold was set to 95% to screen out background images. Then, Baidu AI EasyData1 is employed to help clean the rest data after the trial-and-error test on a few free platforms. Next, the whole dataset was divided into 15 small ones according to the shooting date. At last, EasyData helped find out the images with motion blur, focus blur, atomization, noise, jumble, and repeated or high-similarity content to ensure the visual quality of acquired dishes images.


## 3.2.2. Image Annotation

About 30,000 images remained after automated data cleaning. A close look found out that many images were with no dishes or with dishes in plastic bags and thereby, manual cleaning became necessary. It is worth noting that the removal of these useless images was fulfilled in the image annotation stage.Image annotation was again performed on the public intelligent platform of Baidu AI EasyData. First, about 30% of images were manually labeled, and each dish category was annotated in more than 10 instances. Then, the images with annotated instances were uploaded to train the model embedded in the EasyData tool. Next, the tool was used to label the remaining 70% images. In the end, the dataset was fully annotated. Furthermore, the manual check was conducted offline to ensure the labeling quality of image annotation.


## 3.2.3. The Dataset ChinaDishSet

After removing these categories with less than 200 instances, 28 categories remain. Figure2 shows representative instances of each category. Due to the difference in lighting conditions, shooting time, food ingredients, cooking procedure, and background, the perceived quality of dish images varies.Figure 2
Representative samples of 28 categories of Chinese dishes in the ChinaDishSet dataset.The dataset ChinaDishSet contains a total of 45,061 instances. Figure3 shows the distribution of instance numbers in each dish category. It is observed that rice is the most popular food in daily diet and reaches 6,496 samples, followed by scrambled eggs with cucumber. According to the number of the dish with the most and the dish with the least annotated instances, the maximum imbalance ratio reaches 26.62.Figure 3
The distribution of dishes instances in each category.The largest category is rice with 6,496 samples, the smallest category is a dumpling with 244 samples, and the average number is around 1,609 instances per category.The average number of annotated data instances is 1,609 per category. In the stage of data cleaning, a category with less than 200 instances has been set aside for further dataset upgrading. In a Chinese restaurant, the categories of dishes change due to the join or leave of the chefs and the season food supplies. On the other hand, setting aside the categories with few samples can avoid the data imbalance which may cause an adverse impact on model training.


## 3.3. The Workflow of the Prototype System

The workflow of the deployed system shown in Figure4 contains dish plate image acquisition, deep learning-based dish localization and recognition, and failure cases (or new dish types) based on model updating.Figure 4
The workflow of the dish recognition system.It contains dish plate image acquisition, deep learning-based dish localization and recognition, and failure cases (or new dish types) based on model updating. Note that the failure cases and images of new dish types will be manually annotated for backbone training.


### 3.3.1. Dish Plate Image Acquisition

After the prototype system is activated each time, the first task is to acquire an imageIt0 as the background. After that, how to determine whether a dish’s plate is under the camera, and how to define whether the plate is stationary, both become crucial for follow-up deep learning-based dishes recognition.In this study, assuming at timetk, a dish plate is detected on the way to be placed under the camera, and an image Itk is acquired that Itk−It0>ϵ. Then, a time interval δt should be defined to guarantee that the plate is stationary, i.e., Itk+δt−Itk≈0. Offline experiment indicates that the threshold of absolute intensity difference ε=104 and the time interval δt=0.2s satisfies the requirement of the prototype system.


### 3.3.2. Deep Learning-Based Dish Localization and Recognition

The networks that can simultaneously localize and recognize dishes are considered. In this study, five deep networks off-the-shelf are investigated. Neural networks are all about taking a set of inputs, performing progressively complex computations on them, and providing outputs to solve real-world problems like classification. The networks include YOLO, SSD, Faster-RCNN, RetinaNetand Cascade-RCNN. The network architectures are shown in Figure5. In general, YOLO casts object detection as a regression task by associating isolated bounding boxes to class probabilities, and it can process images in real time. SSD discretizes the output bounding boxes into a series of default rectangles with different scales and aspect ratios according to the location of feature maps, and in default boxes, the presence of each object category is scored and predicted [9]. Faster-RCNN adds a region proposal network that predicts object boundaries and objectless scores at potential positions at the same time [10]. RetinaNet develops the focal loss with the purpose of training on a sparse set of difficult examples, and the network addresses data imbalance by weighting the lossless to these easy examples [11]. Cascade-RCNN is composed of a series of sample detectors that are trained with intersection over union for improving hypotheses quality [12]. Notably, the implementation of the five deep networks is available online.Figure 5
The architectures of deep neural networks of YOLOV3, Faster-RCNN, SSD, RetinaNet, and Cascade-RCNN.Their performance with different training strategies on dishes recognition is compared for selecting the backbone network of the dish recognition system. Different recognition performance is mainly manifested in the recall rate, precision rate, and recognition speed in the recognition process.


### 3.3.3. System Upgrading

The prototype system will be upgraded in two situations. The difference in recognition performance mainly lies in the comparison of recall, precision, and recognition speed in the recognition process. One is to handle wrongly classified dish images. Intuitively, failure cases are representative of some environmental change and unobservable dish patterns, and thereby important and valuable. In failure cases, the dishes in the plate images are annotated manually. When the number of plate images reaches 200, the system is ready for updating. The other is to add new dish types into the prototype system. To guarantee recognition performance, the minimum number of dish instances should be well considered.


## 3.3.1. Dish Plate Image Acquisition

After the prototype system is activated each time, the first task is to acquire an imageIt0 as the background. After that, how to determine whether a dish’s plate is under the camera, and how to define whether the plate is stationary, both become crucial for follow-up deep learning-based dishes recognition.In this study, assuming at timetk, a dish plate is detected on the way to be placed under the camera, and an image Itk is acquired that Itk−It0>ϵ. Then, a time interval δt should be defined to guarantee that the plate is stationary, i.e., Itk+δt−Itk≈0. Offline experiment indicates that the threshold of absolute intensity difference ε=104 and the time interval δt=0.2s satisfies the requirement of the prototype system.


## 3.3.2. Deep Learning-Based Dish Localization and Recognition

The networks that can simultaneously localize and recognize dishes are considered. In this study, five deep networks off-the-shelf are investigated. Neural networks are all about taking a set of inputs, performing progressively complex computations on them, and providing outputs to solve real-world problems like classification. The networks include YOLO, SSD, Faster-RCNN, RetinaNetand Cascade-RCNN. The network architectures are shown in Figure5. In general, YOLO casts object detection as a regression task by associating isolated bounding boxes to class probabilities, and it can process images in real time. SSD discretizes the output bounding boxes into a series of default rectangles with different scales and aspect ratios according to the location of feature maps, and in default boxes, the presence of each object category is scored and predicted [9]. Faster-RCNN adds a region proposal network that predicts object boundaries and objectless scores at potential positions at the same time [10]. RetinaNet develops the focal loss with the purpose of training on a sparse set of difficult examples, and the network addresses data imbalance by weighting the lossless to these easy examples [11]. Cascade-RCNN is composed of a series of sample detectors that are trained with intersection over union for improving hypotheses quality [12]. Notably, the implementation of the five deep networks is available online.Figure 5
The architectures of deep neural networks of YOLOV3, Faster-RCNN, SSD, RetinaNet, and Cascade-RCNN.Their performance with different training strategies on dishes recognition is compared for selecting the backbone network of the dish recognition system. Different recognition performance is mainly manifested in the recall rate, precision rate, and recognition speed in the recognition process.


## 3.3.3. System Upgrading

The prototype system will be upgraded in two situations. The difference in recognition performance mainly lies in the comparison of recall, precision, and recognition speed in the recognition process. One is to handle wrongly classified dish images. Intuitively, failure cases are representative of some environmental change and unobservable dish patterns, and thereby important and valuable. In failure cases, the dishes in the plate images are annotated manually. When the number of plate images reaches 200, the system is ready for updating. The other is to add new dish types into the prototype system. To guarantee recognition performance, the minimum number of dish instances should be well considered.


## 3.4. Experiment Design

Table2 lists the parameters in model fine-tuning, including the batch size, number of epochs, base learning rate, weight decay, and its coefficient and optimizer. If the learning rate is too large, the loss function may directly exceed the global optimal point. If the learning rate is too small, the change rate of the loss function is very slow. The learning rate decay type is PiecewiseDecay, and the decay scheduler is set at the 25th, 30th, and 40th epoch. If not specified, the parameters of deep neural networks are set as default.Table 2
The default parameters used in model fine-tuning.
Cascade-RCNNRetinaNetFaster-RCNNSSDYOLOV3BatchSize222128Epochs1515151515BaseLearningRate1.0×10−51.0×10−51.0×10−56.25×10−51.56×10−5WeightDecayL2L2L2L2L2WeightDecayCoeff1.0×10−41.0×10−41.0×10−45.0×10−55.0×10−4OptimizerMomentumMomentumMomentumAdamMomentumMomentum0.90.90.90.9


### 3.4.1. Candidate Deep Networks for Backbone Selection

On the performance comparison of deep networks on dish recognition, 10 times of experiments are conducted. At each time, the 16,904 images in the ChinaDishSet are divided into a training set with 80% images and a testing set with the rest images. To a deep network, the performance is quantified with the classification accuracy, recall, and mean average precision (MAP) on average.The five networks [8–12] have been well trained on the dataset Microsoft COCO [14]. In this part, the networks are fine-tuned on the training set of images and further inference on the testing set. The parameters shown in Table 2 are used.


### 3.4.2. Deep Network Training Strategies

After the backbone network is determined, two different training strategies, fine-tuning the pretrained backbone and using the backbone network trained from image scratch, are conducted, and the classification performance is compared.At this time, the parameters for fine-tuning are set as shown in Table2, while for training the backbone network from image scratches, the parameters are set as the same except for the base learning rate which is set at 0.005. In addition, different numbers of data instances are considered in the training stage.


### 3.4.3. The Training Data Size for System Upgrading

The effect of training data sizes on dish recognition performance is explored. It is highly related to the system upgrading, and the minimum number of data samples should be annotated when a new category is added to the dish recognition list of the system. To address this issue, in the experiment, the training data size of each category is increased from 50 to 200 with an equal interval of 50 samples, and the classification metrics are analyzed.


## 3.4.1. Candidate Deep Networks for Backbone Selection

On the performance comparison of deep networks on dish recognition, 10 times of experiments are conducted. At each time, the 16,904 images in the ChinaDishSet are divided into a training set with 80% images and a testing set with the rest images. To a deep network, the performance is quantified with the classification accuracy, recall, and mean average precision (MAP) on average.The five networks [8–12] have been well trained on the dataset Microsoft COCO [14]. In this part, the networks are fine-tuned on the training set of images and further inference on the testing set. The parameters shown in Table 2 are used.


## 3.4.2. Deep Network Training Strategies

After the backbone network is determined, two different training strategies, fine-tuning the pretrained backbone and using the backbone network trained from image scratch, are conducted, and the classification performance is compared.At this time, the parameters for fine-tuning are set as shown in Table2, while for training the backbone network from image scratches, the parameters are set as the same except for the base learning rate which is set at 0.005. In addition, different numbers of data instances are considered in the training stage.


## 3.4.3. The Training Data Size for System Upgrading

The effect of training data sizes on dish recognition performance is explored. It is highly related to the system upgrading, and the minimum number of data samples should be annotated when a new category is added to the dish recognition list of the system. To address this issue, in the experiment, the training data size of each category is increased from 50 to 200 with an equal interval of 50 samples, and the classification metrics are analyzed.


## 3.5. Software and Platform

The algorithms are implemented with PyTorch (version 1.10.1) andPython (version 3.7). The deep learning framework PaddlePaddle (version 1.8.4) is deployed on a workstation (Windows 10, 64 bit OS, 12-cored CPU (2.4 GHz), 40 GB RAM) with one GPU card (NVIDIA Tesla P4, 8.0 GB)).


## 4. Results

This section shows the results, including the backbone network selection, the training strategy determination, and the effect of training data sizes on dish recognition performance. After that, one daily dish recognition result of the prototype system deployed in the canteen is presented.


### 4.1. The Backbone Network Selection

Figure6 shows the MAP values regarding different confidence thresholds on dish classification. It is observed that when the threshold value is set larger than 0.20, all five networks obtain good classification results (MAP ≥ 0.80). Notably, even if the confidence threshold is set 0.0, both Faster-RCNN and Cascade-RCNN achieve MAP > 0.85. Among the networks, when the confidence thresholds change, Faster-RCNN and Cascade-RCNN achieve consistently superior MAP values, followed by YOLOV3. In addition, RetinaNet obtains satisfactory results when the cutoff value of the confidence threshold is set to 0.40, while the MAP values of SSD keep no large than 0.90.Figure 6
The MAP values with different confidence thresholds. When the threshold value is set larger than 0.20, all the networks achieve MAP values larger than 0.80 on dishes classification.Figure7 shows the distribution of MAP values in each dish category. The networks achieve superior recognition in most of the categories. On the other hand, 3 networks obtain relatively low values (<0.90) in the zucchini category. Except for less than 500 annotation samples, an offline check of the zucchini shows that the image quality is relatively low, and the zucchini are presented in various forms with vague objectiveness. In addition, the SSD model achieves low MAP values (<0.90) on 8 dish categories.Figure 7
The distribution of MAP values regarding different dish categories on the ChinaDishSet.Table3 summarizes the classification results of the networks embedded in the prototype system. It shows that the three networks (Cascade-RCNN, RetinaNet, and Faster-RCNN) using R50_FPN as the backbone achieve promising results, followed by YOLOV3 and SSD. Specifically, the recall and MAP values of SSD are less than 90.0%. In addition, Faster-RCNN runs the fastest (146 ms), and the time cost of all the networks is around 200 ms, which satisfies real-time requirements in real-life scenes. To balance the classification performance of accuracy, recall, MAP, and time cost, Faster-RCNN is chosen as the backbone of the prototype system for Chinese dishes recognition.Table 3
The prediction performance of Chinese dishes.
NetworkAccuracy (%)Recall (%)MAP (%)Time cost (ms)YOLOV3MobileNetV197.5096.7097.60165SSDMobileNetV193.1084.3086.50185Cascade-RCNNR50_FPN97.0097.0097.80201RetinaNetR50_FPN97.1097.0098.50234Faster-RCNNR50_FPN98.1097.2098.30146The highest result of each metric is in boldface.


### 4.2. The Training Strategy Determination

Figure8 shows the performance of the backbone network Faster-RCNN trained with different strategies. When using the same number of training sizes per dish type, it is found that fine-tuning leads to much higher MAP values over that training from image scratches (Table 3). For instance, when 150 instances per dish type are used, the model based on pretrained parameters achieves MAP 96.15% ± 0.00%, which is nearly 12% higher than the model trained using randomly initialized parameters (MAP, 84.74% ± 0.00%). Notably, it is observed that the model using pretrained parameters achieves an impressive MAP value of around 80% at the end of the first epoch. In contrast, the model using random initialization obtains a low MAP value, even if the number is set to 200 sample instances per dish type.Figure 8
Performance comparison of different training strategies (fine-tuning the pretrained backbone network and training the backbone from image scratch) on dish recognition.


### 4.3. The Effect of Training Data Sizes on Dish Recognition Performance

Table4 summarizes the dish recognition performance when the training data sizes change. It indicates that increasing training samples of each dish type improves the dish recognition results. When training the model from image scratches, the model’s MAP value changes from 61.46% ± 0.02% (50 instances per dish type) to 87.84%±0.00% (200 instances per dish type); and when fine-tuning the pretrained model, the MAP value keeps high from 91.33% ± 0.00% (50 instances per dish type) to 96.48% ± 0.01% (200 instances per dish type). It is also found that when the number of training samples per type increases from 150 to 200, the MAP values of the fine-tuned model have a small improvement (≈0.33%). This finding may suggest that, when a new type of dish is added, 150 and more annotated samples may keep the performance after system upgrading.Table 4
The effect of training data sizes on dishes recognition performance.
50100150200Training61.46% ± 0.02%76.92% ± 0.01%84.74% ± 0.00%87.84% ± 0.00%Fine-tuning91.34% ± 0.00%94.74% ± 0.00%96.15% ± 0.00%96.48% ± 0.01%


### 4.4. Daily Dish Recognition in the Canteen

Figure9 shows several examples of prediction results when the prototype system is deployed in the smart Chinese canteen. It shows that the dishes are correctly localized and identified using Faster-RCNN, even if the dishes (such as the rice) are served in different shapes. Interestingly, the soup in a round bowl is not localized nor predicted. This finding reveals that the prototype system can perform well as expected since the category of soup has not been involved due to its limited annotation samples.Figure 9
Examples of dishes recognition using the deployed prototype system.Figure10 shows the confound matrix of dish recognition results of one daily sale. A total of 7,274 dish samples have been served, and 25 instances are misclassified. The classification accuracy, recall, and MAP values are 98.10%, 97.21%, and 98.34%, respectively. In general, in each dish type, no more than 4 samples are wrongly predicted. For example, 2 potato_meat samples are recognized as zucchini_meat, and 2 samples are misclassified as cucumber_meat. The timely check shows that the ingredient composition and cooking procedure of potato_meat causes appearance change and thereby, the wrong prediction into zucchini_meat or cucumber_meat. The failure cases are annotated and kept for further system updating.Figure 10
The confound matrix of recognition results of one daily sale.


## 4.1. The Backbone Network Selection

Figure6 shows the MAP values regarding different confidence thresholds on dish classification. It is observed that when the threshold value is set larger than 0.20, all five networks obtain good classification results (MAP ≥ 0.80). Notably, even if the confidence threshold is set 0.0, both Faster-RCNN and Cascade-RCNN achieve MAP > 0.85. Among the networks, when the confidence thresholds change, Faster-RCNN and Cascade-RCNN achieve consistently superior MAP values, followed by YOLOV3. In addition, RetinaNet obtains satisfactory results when the cutoff value of the confidence threshold is set to 0.40, while the MAP values of SSD keep no large than 0.90.Figure 6
The MAP values with different confidence thresholds. When the threshold value is set larger than 0.20, all the networks achieve MAP values larger than 0.80 on dishes classification.Figure7 shows the distribution of MAP values in each dish category. The networks achieve superior recognition in most of the categories. On the other hand, 3 networks obtain relatively low values (<0.90) in the zucchini category. Except for less than 500 annotation samples, an offline check of the zucchini shows that the image quality is relatively low, and the zucchini are presented in various forms with vague objectiveness. In addition, the SSD model achieves low MAP values (<0.90) on 8 dish categories.Figure 7
The distribution of MAP values regarding different dish categories on the ChinaDishSet.Table3 summarizes the classification results of the networks embedded in the prototype system. It shows that the three networks (Cascade-RCNN, RetinaNet, and Faster-RCNN) using R50_FPN as the backbone achieve promising results, followed by YOLOV3 and SSD. Specifically, the recall and MAP values of SSD are less than 90.0%. In addition, Faster-RCNN runs the fastest (146 ms), and the time cost of all the networks is around 200 ms, which satisfies real-time requirements in real-life scenes. To balance the classification performance of accuracy, recall, MAP, and time cost, Faster-RCNN is chosen as the backbone of the prototype system for Chinese dishes recognition.Table 3
The prediction performance of Chinese dishes.
NetworkAccuracy (%)Recall (%)MAP (%)Time cost (ms)YOLOV3MobileNetV197.5096.7097.60165SSDMobileNetV193.1084.3086.50185Cascade-RCNNR50_FPN97.0097.0097.80201RetinaNetR50_FPN97.1097.0098.50234Faster-RCNNR50_FPN98.1097.2098.30146The highest result of each metric is in boldface.


## 4.2. The Training Strategy Determination

Figure8 shows the performance of the backbone network Faster-RCNN trained with different strategies. When using the same number of training sizes per dish type, it is found that fine-tuning leads to much higher MAP values over that training from image scratches (Table 3). For instance, when 150 instances per dish type are used, the model based on pretrained parameters achieves MAP 96.15% ± 0.00%, which is nearly 12% higher than the model trained using randomly initialized parameters (MAP, 84.74% ± 0.00%). Notably, it is observed that the model using pretrained parameters achieves an impressive MAP value of around 80% at the end of the first epoch. In contrast, the model using random initialization obtains a low MAP value, even if the number is set to 200 sample instances per dish type.Figure 8
Performance comparison of different training strategies (fine-tuning the pretrained backbone network and training the backbone from image scratch) on dish recognition.


## 4.3. The Effect of Training Data Sizes on Dish Recognition Performance

Table4 summarizes the dish recognition performance when the training data sizes change. It indicates that increasing training samples of each dish type improves the dish recognition results. When training the model from image scratches, the model’s MAP value changes from 61.46% ± 0.02% (50 instances per dish type) to 87.84%±0.00% (200 instances per dish type); and when fine-tuning the pretrained model, the MAP value keeps high from 91.33% ± 0.00% (50 instances per dish type) to 96.48% ± 0.01% (200 instances per dish type). It is also found that when the number of training samples per type increases from 150 to 200, the MAP values of the fine-tuned model have a small improvement (≈0.33%). This finding may suggest that, when a new type of dish is added, 150 and more annotated samples may keep the performance after system upgrading.Table 4
The effect of training data sizes on dishes recognition performance.
50100150200Training61.46% ± 0.02%76.92% ± 0.01%84.74% ± 0.00%87.84% ± 0.00%Fine-tuning91.34% ± 0.00%94.74% ± 0.00%96.15% ± 0.00%96.48% ± 0.01%


## 4.4. Daily Dish Recognition in the Canteen

Figure9 shows several examples of prediction results when the prototype system is deployed in the smart Chinese canteen. It shows that the dishes are correctly localized and identified using Faster-RCNN, even if the dishes (such as the rice) are served in different shapes. Interestingly, the soup in a round bowl is not localized nor predicted. This finding reveals that the prototype system can perform well as expected since the category of soup has not been involved due to its limited annotation samples.Figure 9
Examples of dishes recognition using the deployed prototype system.Figure10 shows the confound matrix of dish recognition results of one daily sale. A total of 7,274 dish samples have been served, and 25 instances are misclassified. The classification accuracy, recall, and MAP values are 98.10%, 97.21%, and 98.34%, respectively. In general, in each dish type, no more than 4 samples are wrongly predicted. For example, 2 potato_meat samples are recognized as zucchini_meat, and 2 samples are misclassified as cucumber_meat. The timely check shows that the ingredient composition and cooking procedure of potato_meat causes appearance change and thereby, the wrong prediction into zucchini_meat or cucumber_meat. The failure cases are annotated and kept for further system updating.Figure 10
The confound matrix of recognition results of one daily sale.


## 5. Discussion

Accurate real-life dish recognition is indispensable in a smart canteen for food ordering, charging, and payment. Many multistage and end-to-end approaches have been developed, and most existing works aim for single dish recognition. Notably, a dozen of systems have been deployed, while the performance is not yet satisfactory on real user photos [37, 39]. In this study, a prototype system (Figure 1) is designed, and its performance is evaluated on daily dish images acquired in the Chinese canteen. Specifically, three technical issues are explored, including the backbone network selection, the training strategy determination, and the minimum sizes for system upgrading.A real-life dish dataset ChinaDishSet is prepared, and the dish images are acquired from the Chinese canteen using a video recording system (Figure1). After data cleaning and annotation, the dataset contains 28 types of dishes, 16,904 images, and 45,061 instances (Figures 2 and 3). At present, many large-scale high-quality dish image datasets [15–17] are available [15]. These datasets cover a broad range of western, Japanese, Chinese, Korean, and miscellaneous food with a high diversity of meat, cereals, vegetables, fish, fruits, and other categories, affording the ability to train high-capacity deep networks on multimodal data analysis [16]. However, most of these images are retrieved from web recipes and menu pictures. To bridge the gap between the laboratory experiment and the real-life situation, Ming conduct experiments on assessing the classification performance of laboratory images and real-user photos and found that the accuracy of real-user data is considerably (≈20%) lower than that of laboratory data in terms of top-one accuracy. Horiguchi design a simple classifier for each user, and by adapting incrementally users’ interest domain of food from a small number of dish samples, the classifier becomes personalized and effective based on learned deep features [47]. These studies [37, 47] indicate laboratory models are promising for real-world tasks. In this study, the task of real-life dish recognition is simplified by deploying the prototype system in a Chinese canteen that benefits plate image acquisition, dish annotation, model training, performance evaluation, and system upgrading. Most importantly, more dish types, images, and instances are already collected offline, and dataset expansion is helpful for system upgrading [17].Faster-RCNN serves the prototype system as the backbone network since it generally outperforms the other four deep networks (Figure6 and Table 2). The backbone network is highly related to performance, and previous studies have shown the superiority of end-to-end approaches in dish recognition (Table 1). Therefore, in this study, the five off-the-shelf deep networks [8–12] are considered, and importantly, these networks can simultaneously localize and recognize the objects which have been verified in large-scale image analysis [18]. According to Table 2, Faster-RCNN, RetinaNet, Cascade-RCNN, and YOLOV3 achieve competitive results, while Faster-RCNN is singled out for the least time cost on dish recognition. By integrating extracted features, proposed regions, bounding boxes, and prediction into seamless learning, Faster-RCNN has been accelerated in visual object analysis. Notably, in the task of dish recognition, Faster-RCNN has been used as a food identifier of dishes in a food image [50, 51], and in [52], its architecture has been modified by adding cross-connected layers to capture low- and high-level messages, and an attention mechanism is additionally used to make the model highlight the contexture of the dish regions. It should be mentioned that some deep networks are kept on update, such as YOLO, and their capacity is kept on improving.The strategies of fine-tuning a deep network and training the network from image scratches are investigated, and as expected, fine-tuning is found more effective than pure training (Figure8 and Table 3). Technically, there are many ways to fine-tune the parameters in a pretrain network by fixing part of or the whole model layers [53]. Fine-tuning, as one of the transfer learning applications, enables a pretrained deep model on a close target. Not only it can relieve the requirement of massive instances for training a deep architecture but also dish recognition performance can be considerably improved. According to Figure 8, it is obvious that fine-tuning leads to much higher MAP values than that pure training. For instance, when 50 instances per dish type are used, it is 91.34% (fine-tuning) versus 61.46% (pure training), and a big difference of ≈30% exists (Table 3). In addition, fine-tuning results in the model training at a high MAP value at the beginning. For instance, when 200 instances per dish type are used, fine-tuned Faster-RCNN obtains a MAP value larger than 80% after the first epoch (Figure 8). Interestingly, Nagarajan proposed a new transfer learning framework for leveraging the message learned on a simple food recognition onto a multilabel task, and the framework uses prior knowledge to tackle the data bias between single and multiple label tasks. In this study, the parameters of pretrained Faster-RCNN are fine-tuned using dish images and a very small learning rate (Table 2) that softly transfers knowledge learned from common objects in context for real-life dish recognition.Furthermore, the system upgrading is considered, and the minimum number of 150 samples is determined when a new dish type is added to the recognition list. As shown in Table3 and Figure 8, it is found that a larger number of each type leads to a higher MAP value in dish recognition. Using fine-tuned Faster-RCNN, the MAP value improves from 91.34% to 96.48% when the number of each dish type increases from 50 to 200 instances, while the improvement is not obvious when 150 samples of each type are changed to 200 samples. It might suggest that 150 and more samples are required when the recognition list is updated. Collecting and annotating dish samples is a rigorous yet time-consuming and arduous task. In practice, failure cases are valuable in system upgrading, since some unobservable and unexpected patterns might exist in these images. While to decrease the frequency of model updating, the number of failure cases is set larger than 200 instances in this study.The daily dish recognition results in the canteen have further verified the success of the prototype system. On the one hand, the dish types in the recognition list, such as the roast_chicken in different orientations, can be identified precisely (accuracy 98.10%, recall 97.10%, and MAP 98.34%) as shown in Figure10. On the other hand, the dish types not in the recognition list, such as the soup in a round bowl, are left alone with no localization or caption (Figure 9). In addition, it is found that 25 out of 7,274 dish instances are wrongly classified, while the timely check shows that failure cases come from the same ingredients or visual appearance (Figure 10). Therefore, the system performs well in daily work, and its capacity can be enhanced by system upgrading in the future.Several limitations exist in the current study. For boosting the recognition performance, multiple evidence can be exploited, including the analysis of visual appearance, ingredient compositions, inherent semantic relationships among fine-grained classes, food procedural attributes, and external knowledge [3, 5, 6, 18]. Meanwhile, novel techniques should be considered and developed, such as multiview and multiscale representation learning [54], local and global feature aggregation, pretraining using large-scale dish datasets [42], and multimodel fusion. Most urgently, the video recording system should be upgraded with a high-resolution microcamera for high-quality image acquisition.


## 6. Conclusions

Accurate dish recognition can help improve dining service, nutritional intake monitoring, food retrieval, and cuisine recommendation. However, a large gap exists between laboratory test photos and real user photos, hampering the deployment of many novel approaches. In this study, we simplify real-life dish recognition by deploying a prototype system in a Chinese canteen. After the dish image dataset is built, three practical issues related to dish recognition performance have been investigated. Experimental results suggest that fine-tuned Faster-RCNN can serve as the backbone of the system, fine-tuning is more effective than pure training, and when a new dish type is added to the recognition list of the system, at least 150 samples per type should be prepared. In the future, the prototype system will be deployed in other canteens, and we will explore the cues of visual appearance, ingredient composition and food procedure attributes, and the techniques of multiscale and multiview representation for boosting the recognition performance.



---

*Source: 1917704-2022-09-15.xml*