ICME 2020WEAKLY-SUPERVISED PLATEAND FOOD REGIONSEGMENTATION
Wataru Shimoda and Keiji Yanai
The University of Electro Communications,Tokyo, Japanⓒ 2020 UEC Tokyo.

Food segmentation• Various applications– Food volume estimation, Food calorie estimation• Problem– Pixel-wise annotation cost

• Approach– Weakly-supervised learning• Train with image-level labelsⓒ 2020 UEC Tokyo.

Motivation• A specific problem on food segmentation– Should be plate regions in food region?
• Desirable segmentation is different in case by case

Include plate?
Which is better?
Foodsegmentation

We can choiceaccording to situations
by plate segmentationⓒ 2020 UEC Tokyo.

Problem statement• Food plate segmentation– Train plate segmentation model without pixelwise annotation– Use image-level labels for food categories– Separate food images into three categories• Background• Food regions• Plate regionsⓒ 2020 UEC Tokyo.

Key idea
𝑓𝑔

Food category classifier
Not respond plate regions
𝑓𝑔
𝑆𝐹 Food/Non-Food classifier𝑓𝑔
𝑓𝑔
Respond plate regions
𝑆𝑃 = 𝑆𝐹 − 𝑆𝐿 , 𝑦 ∈ 𝐿
𝑆𝑃 Difference
Plate regions

𝑆𝐿Discriminativefood region

Input image

Food categoryclassifier

Visualization

Plate regionFood region

Food/non-foodclassifier

Visualizationⓒ 2020 UEC Tokyo.

Approach• Visualization– Class activation map[1]– 𝑣𝐹 = 𝐶𝐴𝑀(𝑥; 𝜃𝐿 ) ∈ ℝ2×𝐻×𝑊 , 𝑣𝐿 = 𝐶𝐴𝑀 𝑥; 𝜃𝐿 ∈ ℝ𝐶×𝐻×𝑊• Mask from visualization– 𝑚𝐹,𝑐𝑎𝑚 : food/non-food mask, from 𝑣𝐹
𝑦– 𝑚𝐿,𝑐𝑎𝑚 : food category mask, from 𝑣𝐿 and label 𝑦𝑘𝑟• 𝑚𝐿,𝑐𝑎𝑚: unreliable regions, upper k class of recognition results• Plate mask

𝑓𝑔𝑓𝑔

𝑆𝑃 = 𝑆𝐹 − 𝑆𝐿 , 𝑦 ∈ 𝐿– 𝑚𝑃,𝑐𝑎𝑚 : difference of the masks 𝑆𝑃 :a set of pixels form 𝑚𝑃,𝑐𝑎𝑚– 𝑚𝑃,𝑜𝑢𝑡 : CRF applied mask
[1] Learning deep features for discriminative localization, Zhou et al., CVPR 2016ⓒ 2020 UEC Tokyo.

𝑓𝑔

𝑆𝐹 :a set of pixels form 𝑚𝐹,𝑐𝑎𝑚Combination withweakly-supervised food segmentation• Train plate segmentation model with weaklysupervised segmentation model in end-to-end manner• Weakly-supervised food segmentation– Image-label to segmentation– Output setting• not include food plate regions– Base method• [1] Self-Supervised Difference Detection, ICCV 2019[1] Self-supervised difference detection for weakly-supervised semantic segmentation, Shimoda et al., ICCV 2019ⓒ 2020 UEC Tokyo.

SSDD module• Integrates two candidate segmentation masks
using difference detection for stable refinement– [1] Self-Supervised Difference Detection, ICCV 2019
Proposal A

SSDD module

Integration

Input image
DDNet
Proposal B
DDNet[1] Self-supervised difference detection for weakly-supervised semantic segmentation, Shimoda et al., ICCV 2019ⓒ 2020 UEC Tokyo.

Architecture- We use a SSDD module for integration of CAM and food+plate segmentation mask
- We make consistency between the food segmentation model and the platesegmentation model in the food regions with two techniques①Constraining food regions by plate regions②Penalizing background prediction using plate segmentation
𝒎𝑳,𝒄𝒂𝒎𝒎𝑳,𝒕𝒄𝒉CNN

Food segmentationSeg
mask(Teacher)Seg
pred𝒎𝑷CAM
SSDD
module𝒎𝑳,𝒑𝒍𝒕Segmask(Food+plate)

Input image
IntegrationSeg
mask(Prediction)

𝒎𝑳,𝒐𝒖𝒕Plate segmentationⓒ 2020 UEC Tokyo.

Architecture①Constraining Food Regions by Plate Regions②Penalizing Background Prediction Using Plate Segmentation

Avoid mixing of the food regions and the plate region by integration basedconstraining
𝒎𝑳,𝒄𝒂𝒎𝒎𝑳,𝒕𝒄𝒉

CNN

Food segmentationSeg
mask(Teacher)Seg
pred𝒎𝑷CAM

SSDD
module𝒎𝑳,𝒑𝒍𝒕Seg
mask(Food+plate)𝑚𝐿,𝑝𝑙𝑡 =𝑚𝐿,𝑜𝑢𝑡 if (𝑚𝑃,𝑜𝑢𝑡 = food class)

Input image
IntegrationSeg
mask(Prediction)

𝒎𝑳,𝒐𝒖𝒕Plate segmentationⓒ 2020 UEC Tokyo.

𝑩𝑮 if (𝑚𝑃,𝑜𝑢𝑡 = 𝑩𝑮 or plate class)Architecture①Constraining Food Regions by Plate Regions②Penalizing Background Prediction Using Plate Segmentation
To limit the outputs of background, we constrain the outputs of the foodsegmentation model on the background class using a penalty loss𝒎𝑳,𝒄𝒂𝒎𝒎𝑳,𝒕𝒄𝒉

CNN

Food segmentationSeg
mask(Teacher)Seg
pred𝒎𝑷CAM

SSDD
module𝒎𝑳,𝒑𝒍𝒕
Seg
mask(Food+plate)

Input image
IntegrationSeg
mask(Prediction)

𝒎𝑳,𝒐𝒖𝒕Plate segmentationⓒ 2020 UEC Tokyo.

𝐿𝑝𝑒𝑛𝑎𝑙𝑡𝑦 =1− 𝑓𝑜𝑜𝑑 ෍ log(𝑢ℎ෨ 𝑏𝑔 )𝑆𝑃,𝑜𝑢𝑡 𝑢Experiments• Dataset– UEC FOOD100• 100 classes, 10000 images• 100 images per class• Image-level label and bounding box annotation• We annotated pixel-wise annotation to 1000 images forevaluation– Train• 9000 images with image-level labels• 8155 non-food images from Web and Twitter– Test
• 1000 images with pixel-level labelsⓒ 2020 UEC Tokyo.

Plate segmentation results
Input
image

Raw
plate seg

CRF
applied

Input
image

ⓒ 2020 UEC Tokyo.

Rawplate seg

CRF
appliedFailure cases
Input
image

Raw platesegmentation

CRF applied
segmentationⓒ 2020 UEC Tokyo.

Input
image

Raw platesegmentation

CRF applied
segmentation

Comparison with other weaklysupervised segmentation methods
Quantitative evaluation was performed using weakly supervised food segmentation.
Because we only have pixel-wise ground truth for the food category masks

CAM [1]
Base method [2]Simple does it [3]†mIoU
30.7
49.7
51.1

Pixel Acc
65.1
78.3
81.9

PFSeg(proposed)

55.4

82.6

[1] Learning deep features for discriminative localization, Zhou et al., CVPR 2016[2] Self-supervised difference detection for weakly-supervised semantic segmentation, Shimoda et al., ICCV 2019[3] Simple does it: Weakly supervised instance and semantic segmentation, Khoreva et al., CVPR 2017† use Bounding box annotation and GrabCutⓒ 2020 UEC Tokyo.

Ablation study

methodConstraining

PenalizingmIoU

Pixel Acc

(I)--

49.7

78.3(II)

✔

-

42.9

75.4

(III)

-

✔

52.6

81.0

(IV)

✔

✔

55.4

82.6

ⓒ 2020 UEC Tokyo.

InputCAM

SSDD

Simple
Does Itⓒ 2020 UEC Tokyo.

PFseg(Proposed)

GroundtruthSummary• Predict plate regions without pixel-wiseannotations– Boost weakly supervised segmentation accuracy
using plate segmentation• Future work
– Improve inference of plate segmentation on theboundaries in the plate regions and thebackground– Further applicationsⓒ 2020 UEC Tokyo.