3D Mesh Reconstruction of Foods
from a Single Image
Shu Naritomi, Keiji Yanai

The University of Electro-Communications, Tokyo, Japan

Our recent work
- Hungry Networks: 3D Mesh Reconstruction of a Dish and a Plate from aSingle Dish Image for Estimating Food Volume. [1]- ACM Multimedia Asia 2020.

- Pop’n Food: 3D Food Model Estimation System from a Single Image. [2]- IEEE MIPR 2021
[1] S. Naritomi, and K. Yanai. Hungry Networks: 3D Mesh Reconstruction of a Dish and a Plate from a Single Dish Image forEstimating Food Volume. In Proc. of ACM Multimedia Asia 2020.

[2] S. Naritomi, and K. Yanai. Pop’n Food: 3D Food Model Estimation System from a Single Image.
In Proc. of IEEE 4th International Conference on Multimedia Information Processing and Retrieval 2021

2

Introduction- Dietary calorie management has been an important topic.
- There is a lot of research on calorie estimation in the multimedia community.
2D based

Depth based

Sensor based[Ege et al., IEICE2018][CalorieCaptorGlass, IEEE VR 2020][Im2Calories, ICCV 2015]3

Introduction- Reconstruct 3D dish (food + plate) volume and 3D plate volumefrom a single dish image- Achieve consistency between the plate part of the two reconstructedvolumes introducing plate consistency loss.

4

Appropriate 3D representation- we want to estimate the food volume.
- Voxel : ✖ Not suitable for high resolution- Point cloud: ✖ The connection between points is unknown.
- Mesh: 〇 It is easy to achieve high resolution.
the volume can be calculated easily if the conditions are met.

- Conditions for obtaining volume from Mesh
- Watertight- no self-intersection
Self-intersection [Mesh R-CNN, ICCV2019]5

Appropriate 3D representation- Mesh Template : self-intersection occurs frequency.
- Occupancy, SDF: When a marching cube is used to[Occupancy Networks, CVPR2019]

extract the mesh, it is watertight and does not self-intersect.

- The problem of situations where the shapes of the plate do not match.
- The point 𝑝 ∈ 𝑅3 contained inside the plate is not contained inside the dish mesh.

occupancy isreasonable
6

Hungry Networks : inference
Increase the resolution. (N times)
occupancy
probabilities marchingcubespoint∈ ℝ3image

decoder1

occupancy
probability ∈ ℝdecoder2

occupancy
probability ∈ ℝ

encoderpoint∈ ℝ3
occupancy
probabilities

Increase the resolution. (N times)

marching
cubes
7Hungry Networks : inference
Increase the resolution. (N times)
Extract
occupancy
Image featureprobabilities marching
cubes

point∈ ℝ3
image

decoder1

occupancy
probability ∈ ℝ

decoder2

occupancy
probability ∈ ℝ

encoderpoint∈ ℝ3
occupancy
probabilities

Increase the resolution. (N times)

marching
cubes
8Hungry Networks : inference
Image feature Increase the resolution. (N times)
occupancy+
probabilities marching
coordinate p ∈ ℝ3
cubespoint∈ ℝ3
image

decoder1

occupancy
probability ∈ ℝ

decoder2

occupancy
probability ∈ ℝ

encoderpoint∈ ℝ3
occupancy
probabilities

Increase the resolution. (N times)

marching
cubes
9Hungry Networks : inference
Image feature Increase the resolution. (N times)
occupancy+
probabilities marching
coordinate p ∈ ℝ3
cubes

point∈ ℝ3
image

decoder1

occupancy
probability ∈ ℝ

decoder2

occupancy
probability ∈ ℝ

encoderpoint∈ ℝ3
occupancy
probabilities

Increase the resolution. (N times)

Inferenceoccupancymarching
cubes
10Increase the resolution
only at the boundarysurface of the object.

Hungry Networks : inference

Image feature Increase the resolution. (N times)
occupancy+
probabilities marching
coordinate p ∈ ℝ3
cubes

point∈ ℝ3
image

decoder1

occupancy
probability ∈ ℝ

decoder2

occupancy
probability ∈ ℝ

encoderpoint∈ ℝ3
occupancy
probabilities

Increase the resolution. (N times)

Inferenceoccupancymarching
cubes
11Hungry Networks : inference
Increase the resolution. (N times)
occupancy
probabilities marching

cubes

point∈ ℝ3
image

decoder1
encoder

decoder2

occupancy
Finally,probability∈ ℝ apply the obtained occupancy

field to the Marching Cube to extract
the mesh.
occupancy
probability ∈ ℝpoint∈ ℝ3
occupancy
probabilities

Increase the resolution. (N times)

marchingcubes
12Hungry Networks : training- Learning the occupancy is actually a binary classification.
-Binary cross entropy loss𝑝 ∈ 𝑅3

: input point coordinate𝑥: image feature vector𝑜 𝑝 ∈𝑅: occupancy of point p[Occupancy Networks, CVPR2019]

𝑓𝑑 𝑥, 𝑝 ∈ 𝑅 : decoder that outputs occupancy
13

Hungry Networks : training- Plate consistency loss (proposal method)-Loss function for matching plate parts of the 3D shape of dish and plat

Dish occupancy
𝑓𝑑1 𝑥, 𝑝0

Plate occupancy
𝑓𝑑2 𝑥, 𝑝0

𝑓𝑑2 𝑥, 𝑝− 𝑓𝑑1 𝑥, 𝑝101011-1
10014

Hungry Networks : training- Plate consistency loss (proposal method)-

Loss function for matching plate parts of the 3D shape of dish and plat

Dish occupancy
𝑓𝑑1 𝑥, 𝑝0

Plate occupancy
𝑓𝑑2 𝑥, 𝑝0

𝑓𝑑2 𝑥, 𝑝− 𝑓𝑑1 𝑥, 𝑝101011-1100

Without Plateconsistency loss
With Plateconsistency loss

15Hungry Networks : training- Plate consistency loss (proposal method)-There is a problem
if the difference is 1.

Loss function for matching plate parts of the 3D shape of dish and plat

Dish occupancy
𝑓𝑑1 𝑥, 𝑝0

Plate occupancy
𝑓𝑑2 𝑥, 𝑝0

𝑓𝑑2 𝑥, 𝑝− 𝑓𝑑1 𝑥, 𝑝

101011-1100

Without Plateconsistency loss
With Plateconsistency loss

16

Hungry Networks : training- Mini batch loss
𝑓𝑒 𝐼𝑖 Encoder that outputs image feature𝐼𝑖

i-th imageℬmini batch17

Training dataset- There is no dataset containing a 3D mesh of dish.
-Build a new dataset- 240 Dish 3D models、38 plate 3D models.
- Using a commercially available 3D scanner.

18

Training dataset- The mesh output by the scanner cannot be learned as it is.
- problem- (1) The center of the model does not coincide with the origin.
- (2) Not watertight.
- (3) The size is not unified.

- (4) Containing noise.
- (5) The coordinates of the plate parts of a dish mesh and a corresponding plate meshdo not match to each other.
19

Training dataset- The mesh output by the scanner cannot be learned as it is.
- problem- (1) The center of the model does not coincide with the origin.
- (2) Not watertight.
- (3) The size is not unified.

- (4) Containing noise.
- (5) The coordinates of the plate parts of a dish mesh and a corresponding plate meshdo not match to each other.
20

Training dataset : modify scanned mesh data.
- (2) Not watertight.
- The 3D model taken by the scanner lacks the surface that was in contact with thefloor.
- Apply Poisson Surface Reconstruction- (4) Contains noise- Eliminate using TSDF Fusion

21Training dataset : modify scanned mesh data.
- (5) The coordinates of the plate parts of a dish mesh and a corresponding
plate mesh do not match to each other.
- Align dish and plate meshes using ICP (Iterative closest point)22

Training dataset : image- Rendered using blender as well as 3D-R2N2 [13].
- Two patterns of images are available, with background or withoutbackground.

23

Experiment : Qualitative evaluation- ResNet18- 𝜆3=20- Without background24Experiment : Qualitative evaluation- ResNet18- 𝜆3=20- With background

25Experiment : Qualitative evaluation- ResNet18- 𝜆3=20- With background26

Experiment : Quantitative evaluation- weighting plate consistency loss

27

Experiment : Quantitative evaluation- weighting plate consistency loss

plate consistency loss
contributes to reducing
volume error.

28

Experiment : Quantitative evaluation- 2 patterns of learning image with / without background- Image with background + ResNet18 + 𝜆3=20 Is the most accurate.

29

Application

https://youtu.be/YyIu8bL65EE

30

Conclusion-Hungry Networks- Reconstruct 3D dish (food + plate) volume and 3D plate volume from a single dish image-

Introducing plate consistency loss
- Matching plate parts of the 3D shape of dish and plate- Contributes to the accuracy of volume estimation-

Creating a 3D meal dataset for training- We showed that it can correspond to the real dish image.
31

Method objective

33

Method objective

34

Method objective

35

Appropriate 3D representation

36

Proposed networks- Hungry Networks-

Reconstruct two meshes of dish and plate from a single dish image-Extend Occupancy Networks [17], an occupancy-based method- Introducing plate consistency loss-

Loss function for matching plate parts of the 3D shape of dish and plate40