Hungry Networks: 3D Mesh Reconstruction ofa Dish and a Plate from a Single Dish Image
for Estimating Food Volume
Shu Naritomi, Keiji Yanai
The University of Electro-Communications, Tokyo, Japan

Introduction- Dietary calorie management has been an important topic.
- There is a lot of research on calorie estimation in the multimedia community.
2D based

Depth based

Sensor based[Ege et al., IEICE2018][CalorieCaptorGlass, IEEE VR 2020][Im2Calories, ICCV 2015]

2Introduction- Reconstruct 3D dish (food + plate) volume and 3D plate volumefrom a single dish image- Achieve consistency between the plate part of the two reconstructedvolumes introducing plate consistency loss.

3

Appropriate 3D representation- Purpose: estimate the food volume.
- Desired features.

Self-intersection [Mesh R-CNN, ICCV2019]- The volume can be easily obtained.
- Matching plate part shape of dish and plate.

The shapes of the dishes
do not match.

4

Appropriate 3D representation- Purpose: estimate the food volume.
- Desired features.
- The volume can be easily obtained.
- Matching plate part shape of dish and plate.

The shapes of the dishes
do not match.

Self-intersection [Mesh R-CNN, ICCV2019]

No Self-intersection&
Watertight Mesh5

Appropriate 3D representation- Purpose: estimate the food volume.
- Desired features.

Self-intersection [Mesh R-CNN, ICCV2019]- The volume can be easily obtained.
- Matching plate part shape of dish and plate.

The shapes of the dishes
do not match.

No Self-intersection&
Watertight Mesh

Occupancy representationis reasonable
6

Hungry Networks : inference
Increase the resolution. (N times)
occupancy
probabilities marchingcubespoint∈ ℝ3
image

decoder1occupancy
probability ∈ ℝ

decoder2

occupancy
probability ∈ ℝ

encoderpoint∈ ℝ3
occupancy
probabilities

Increase the resolution. (N times)

marching
cubes
7Hungry Networks : inference
Increase the resolution. (N times)
Extract
occupancy
Image featureprobabilities marching
cubes

point∈ ℝ3
image

decoder1

occupancy
probability ∈ ℝ

decoder2

occupancy
probability ∈ ℝ

encoderpoint∈ ℝ3
occupancy
probabilities

Increase the resolution. (N times)

marching
cubes
8Hungry Networks : inference
Image feature Increase the resolution. (N times)
occupancy+
probabilities marchingcoordinate p ∈ ℝ3
cubes

point∈ ℝ3
image

decoder1

occupancy
probability ∈ ℝ

decoder2

occupancy
probability ∈ ℝ

encoderpoint∈ ℝ3
occupancy
probabilities

Increase the resolution. (N times)

marching
cubes
9Hungry Networks : inference
Image feature Increase the resolution. (N times)
occupancy+
probabilities marching
coordinate p ∈ ℝ3
cubes

point∈ ℝ3
image

decoder1

occupancy
probability ∈ ℝ

decoder2

occupancy
probability ∈ ℝ

encoderpoint∈ ℝ3
occupancy
probabilities

Increase the resolution. (N times)

Inferenceoccupancymarching
cubes
10Increase the resolution
only at the boundary
surface of the object.

Hungry Networks : inference

Image feature Increase the resolution. (N times)
occupancy+
probabilities marching
coordinate p ∈ ℝ3
cubespoint∈ ℝ3
image

decoder1

occupancy
probability ∈ ℝ

decoder2

occupancy
probability ∈ ℝ

encoderpoint∈ ℝ3
occupancy
probabilities

Increase the resolution. (N times)

Inferenceoccupancymarching
cubes
11Hungry Networks : inference
Increase the resolution. (N times)
occupancy
probabilities marching

cubes

point∈ ℝ3
image

decoder1
encoder

decoder2

occupancy
Finally,probability∈ ℝ apply the obtained occupancy

field to the Marching Cube to extract
the mesh.
occupancy
probability ∈ ℝpoint∈ ℝ3
occupancy
probabilities

Increase the resolution. (N times)

marchingcubes
12Hungry Networks : training- Learning the occupancy is actually a binary classification. (inside or outside)-Binary cross entropy loss𝑝 ∈ 𝑅3

: input point coordinate𝑥: image feature vector𝑜 𝑝 ∈𝑅: occupancy of point p

𝑓𝑑 𝑥, 𝑝 ∈ 𝑅 : decoder that outputs occupancy
13

Hungry Networks : training- Plate consistency loss (proposal method)-

Loss function for matching plate parts of the 3D shape of dish and plat

Dish occupancy
𝑓𝑑1 𝑥, 𝑝0

Plate occupancy
𝑓𝑑2 𝑥, 𝑝0

𝑓𝑑2 𝑥, 𝑝− 𝑓𝑑1 𝑥, 𝑝101011-1
10014

Hungry Networks : training- Plate consistency loss (proposal method)-

Loss function for matching plate parts of the 3D shape of dish and plat

Dish occupancy
𝑓𝑑1 𝑥, 𝑝0

Plate occupancy
𝑓𝑑2 𝑥, 𝑝0

𝑓𝑑2 𝑥, 𝑝− 𝑓𝑑1 𝑥, 𝑝101011-1
100

Without Plateconsistency loss
With Plateconsistency loss15

Hungry Networks : training- Plate consistency loss (proposal method)-There is a problem
if the difference is 1.

Loss function for matching plate parts of the 3D shape of dish and plat

Dish occupancy
𝑓𝑑1 𝑥, 𝑝0

Plate occupancy
𝑓𝑑2 𝑥, 𝑝0

𝑓𝑑2 𝑥, 𝑝− 𝑓𝑑1 𝑥, 𝑝

10
10
1
1-1
100

Without Plateconsistency loss
With Plateconsistency loss

16

Hungry Networks : training- Mini batch loss
𝑓𝑒 𝐼𝑖 Encoder that outputs image feature𝐼𝑖

i-th imageℬmini batch17

Training dataset- There is no dataset containing a 3D mesh of dish.
-Build a new dataset- 240 Dish 3D models、38 plate 3D models.
- Using a commercially available 3D scanner.

18

Experiment : Qualitative evaluation

19

Experiment : Quantitative evaluation- weighting plate consistency loss

20

Experiment : Quantitative evaluation- weighting plate consistency loss

plate consistency loss
contributes to reducing
volume error.

21

Conclusion-Hungry Networks- Reconstruct 3D dish (food + plate) volume and 3D plate volume from a single dish image-Introducing plate consistency loss
- Matching plate parts of the 3D shape of dish and plate- Contributes to the accuracy of volume estimation-

Creating a 3D food dataset- We showed that it can correspond to the real dish image.
22