Zero-Annotation Plate Segmentation
Using a Food Category Classifier and a Food/Non-Food Classifier
Wataru Shimoda and Keiji Yanai
The University of Electro Communications, Tokyo, JapanAbstract

2. Method
In this study, we propose a weakly supervised segmentation method for food images. In order to achieve it, wealso train a food-plate segmentation model without pixelwise annotation. We make use of the inference results of theplate areas and apply it to improve an accuracy of weakly
supervised segmentation.
In this research, we synthesize segmentation masks forlearning a segmentation model that infers plate areas of foodimages. In order to generate the plate areas, we use visualization results of a food class classifier and a food/nonfood classifier. We assume that vO = CAM (x; θcl,O ) ∈RC×H×W is a visualization result of the C-class food classifier for input image x generated by Class Activation Mapping (CAM) [4]. In the similar manner, the visualization
results of the food/non-food classifier are represented byvF = CAM (x; θcl,F ) ∈ R2×H×W , where θcl,O and θcl,F
are parameters for the classifiers. vF will have large scoresin food regions of images. On the other hand, the visualization result vO of the food category classifier will havelarge responses in regions that are important for class identification. Both the visualization results are expected torespond to the area of the food regions of images. However, there is a difference between these visualizations. In
particular, while the visualization of the food/non-food re-To refine a weakly-supervised segmentation method in
the food image domain, we propose a novel method to infer
plate regions without any pixel-wise annotation. We synthesize segmentation masks for training a plate segmentation
model by difference between two types of visualization. In
this paper, we train two types of classifiers: a food category classifier and a food/non-food classifier. These twoclassifiers can highlight food regions by visualization forthe classifiers. Our finding is that there is a difference between visualizations of food regions of two types of classifiers and the difference corresponds to strong co-occurencewith foods, that is a plate of food. We demonstrate the proposed approach can capture a region of a plate of foods andwe also utilize the plate segmentation results for boosting
the weakly-supervised food segmentation accuracy.

1. Introduction
In this paper, we propose a novel method to synthesize
segmentation masks for food plate areas without pixel-wiseannotation and we utilize the plate segmentation results for
improvement of the weakly supervised food segmentation.
In Fig.1, we show illustration for the idea of the approach tosynthesize segmentation masks for food plate areas withoutpixel-wise annotation.
In order to deduce plate areas without pixel-wise annotation, in this study, we train not only food class classifiers
but also food/non-food classifiers. In the recognition offood/non-food, plate areas will respond because those havestrong co-occurrence with foods. On the other hand, in therecognition result of the food category, plates are included
in most of food images, so the contribution to the recognition of the food category is not large. That is, in the visualization of the food class classifier and the food/non-foodclassifier is different and the difference may have correlation with plate areas. In this study, the difference between
the visualization results of the dish area in these two classifiers is used to infer the dish area without the area levelannotation.

Figure 1. An illustration of the proposed approach for synthesizing
plate segmentation masks using visualization.

1

turn clear responses in regions of strong co-occurrence withthe food images, the visualization of the food category will
not respond in the regions of strong co-occurrence withthe food images because the objects which have strong cooccurrence with foods are included in most of food images
and the objects are useless for recognition of food categories. In this paper, we assume that the regions that havestrong co-occurrence with foods are plates of the food andwe synthesize plate segmentation masks by utilizing the difference between the visualization results of the food/nonfood classifier and the food category classifier.
We denote the steps of synthesizing of segmentation
masks for plate areas from the visualization results. First,
we obtain binary segmentation masks mF whose pixels
represent belonging to foods or non-food objects from vF .
Secondly, we obtain segmentation masks my for categorylabels y assigned to images from visualization results. IfmF and my are able to extract correctly, the difference inthe masks should be objects have strong co-occurrence withfood. We define the segmentation masks synthesized by theabove processing as mP . Here, we define a set of pixels
of mP of plate regions as SP . This set can be represented
by SP = SFf g − Syf g , where Syf g is a set of the foregroundof the discriminative food region and SFf g is a set of thefore ground of the whole food region. We train the platesegmentation model by synthesized background, plate areaand food area ternary masks mP . The learning loss of themodel as follows:∑ ∑1Lplate = − ∑log(hku (x; θP )),|Sk | k=(0,1,2) u∈Skk=(0,1,2)(1)where hku is conditional probability of observing any labelk at any location u. Sk is a set of pixels for a class k of themask mP , where S0 = SFbg , S1 = Syf g and S2 = SP .
We apply CRF to the probability map of the plate segmentation model and used the CRF applied results as thefinal plate segmentation results.

3. Experiments
In the experiments, we used the UEC-FOOD100dataset [2]. The UEC-FOOD100 dataset [2] consist of 100
class food categories and each category includes 100 images. It should be noted that each food item has an annotated bounding box, but there are no annotation for pixelwise segmentation masks. Therefore, we add new semantic
segmentation annotation to 10% of UEC-FOOD100 dataset.
We used this pixel-wise annotation for only evaluation.
Fig.2 shows examples of the food and plate area inferredby the plate segmentation model. We observed that the proposed method can infer various types of plate areas, insteadof simply relying on color features or estimating only circular objects. These results are excellent, considering that
we did not use any pixel-wise annotation for training of thesegmentation model.

Figure 2. Examples of the plate segmentation model.
Table 1. Comparison with existing methods.

MethodBaseline method [3]BB annotation + grabcut [1]
Proposed

mIoU
50.2
51.1
52.3

Pix acc
77.5
81.9
80.4

Table 1 shows the comparison between the proposed
method and the existing method. The first baseline method
is the method [3] used by the proposed method as a baseframework. BB annotation + grabcut is a method that utilizes bounding box annotation proposed in [1]. This approach with usage of bounding boxes is expected to have agreat advantage over the approach using only class labels,
and can be considered a powerful baseline. Astonishingly,
although the proposed method is a method using only classlabels, it achieves accuracy close to the method using thebounding box. In addition, regarding the mIoU, the proposed approach using the inference result of the plate areaby the proposed method achieved the higher accuracy thanthe methods using not only the baseline method but also thebounding box, which is an impressive result.

4. Conclusions
In this paper, we proposed a method to synthesize segmentation masks for food plate by visualization. Actually,
we used a food category classifier and a food/non-food classifier for visualization and extracted strong co-occurrence
regions with foods from the difference between the visualization results of the two types of the classifiers. In addition,
we demonstrated that we can boost weakly supervised foodsegmentation by inference results of a plate segmentation
model trained by the synthesized masks.

References[1] A. Khoreva, R. Benenson, J. Hosang, M. Hein, and B. Schiele. Simple
does it: Weakly supervised instance and semantic segmentation. InCVPR, 2017. 2[2] Y. Matsuda, H. Hoashi, and K. Yanai. Recognition of multiple-foodimages by detecting candidate regions. In ICME, pages 1554–1564,
2012. 2[3] W. Shimoda and K. Yanai. Self-supervised difference detection forweakly-supervised semantic segmentation. In ICCV, 2019. 2[4] B. Zhou, A. Khosla, A. Lapedriza, and A. Oliva, A. Torralba. Learning
deep features for discriminative localization. In CVPR, 2016. 1