Refinement of Hair Geometryby Strand Integration
Ryota Maeda (University of Hyogo),
Kenshi Takayama, Takafumi Taketomi (CyberAgent)Digital Human• Create 3D models of real people by capturing images

Multi-view camera

Video production2

Difficulty of Hair Modeling
Hand modeling by CG artists

Using multi-view images

photograph[Maya XGen Official Doc.]••

Technical expertise and artistic skill
Laborious and time-consuming task

COLMAP[Nam et al.]•Conventional MVS does not work

3

Related Works : Hair Reconstruction from Multi-view Images[Paris et al, SIGGRAPH2004][Luo et al, CVPR2012][Paris et al, SIGGRAPH2008][Hu et al, SIGGRAPH2014][Luo et al, CVPR2013][Luo et al, SIGGRAPH2013]4

LPMVS (Line-based PatchMatch Multi-View Stereo) [Nam et al., CVPR2019]5

LPMVS (Line-based PatchMatch Multi-View Stereo) [Nam et al., CVPR2019]

LPMVS[Nam+, CVPR2019]

Input: Multi-view images

Output: 3D lines (unfilterd)

6

Our Contribution: Strand Integration[Proposed method]
LPMVS[Nam+, CVPR2019]

StrandIntegration

Input: Multi-view images
7

General Pipeline forReconstruction of Hair Geometry

8

Reconstruction of Hair Geometry from Multi-view Images

Input

Multi-view images

Reconstruction Algorithm

OutputHair geometry

Downstream tasks(i.e. neural rendering)[Rosu et al., ECCV2022][Wang et al., CVPR2022]

9

Reconstruction of Hair Geometry from Multi-view Images

Input

Esitimate 3D lines

Merge into long strands

OutputOur target

Multi-view images

Hair geometry

10

Reconstruction of Hair Geometry from Multi-view Images
Esitimate 3D lines

Input

Estimate
depth and direction map
for every viewMerge all views
into a set of 3D lines

Multi-view images

11

Reconstruction of Hair Geometry from Multi-view Images
Esitimate 3D lines

Input

Estimate
depth and direction mapfor every view

[Proposed method]

Strand Integration

Merge all views
into a set of 3D lines

Multi-view images

12

Representation of Hair Geometry by 3D Lines

Depth map !

Direction map "••Position ! = #, %, &Direction ' = (! , (" , (#13Problem of the Existing Method (LPMVS [Nam et al.])Hair strands are smoothly connected…
Problem
Does NOT consider
the spatial coherence of 3D lines.

••Position ! = #, %, &Direction ' = (! , (" , (#14Our Method: Strand Integration

Hair strands are smoothly connected…
Problem
Does NOT consider
the spatial coherence of 3D lines.

••Position ! = #, %, &Direction ' = (! , (" , (#Our Method: Strand Integration
Refine the position of hair strands byusing the direction.

15

Strand Integration

16

Loss Function
Find the depth map ) which minimizesℒ " = $! ℒ! " + ℒ" (")
Direction loss

Depth loss

17

Direction Loss: ℒ!
●Harness the position and direction information for improving
geometrical coherence.

●Inspired by Normal Integration.

18

Normal Integration: Depth from NormalDifferentiation ∇Integration ∫Normal map !

Depth map "[Image from ICIP2019 Tutorial: Photometric 3D Reconstruction]

19

Strand Integration: Depth from Direction

Differentiation ∇Integration ∫Direction map #Depth map "20

Strand Integration: Depth from Direction=∇!

Orientationof the strandDirection map #Differentiation of depth map ∇! "21

Direction Loss: ℒ!

ℒ% " = $ %&'()' − ∇* (")+∇&22

Loss Function
Find the depth map ) which minimizesℒ " = $! ℒ! " + ℒ" (")
Direction loss

Depth loss

23

Depth Loss : ℒ"●

Use prior depth map as anchor points.

●

Incorporate the multi-view constraint into the refinement process.

ℒ, " = $ " − "&'()'+Contain inaccurate
depth values
24

3D Line Consistency Map

Inaccurate depth

Low consistency

distortthe refinement result

use as per-pixel weightto ignore
the inaccurate depth

Prior depth map

Consistency Map
with respect toneighboring views

25

Depth Loss (w/ Consistency Map) : ℒ"
●Use prior depth map as anchor points.

●

Incorporate the multi-view constraint into the refinement process.

ℒ, " = $ * " − "&'()'+26

Loss Function
Find the depth map ) which minimizesℒ " = $! ℒ! " + ℒ" (")
Direction loss

Depth loss

27

Loss Function with Normal Loss
Find the depth map ) which minimizesℒ " = $# ℒ# " + $! ℒ! " + ℒ" (")
Normal loss
(optional)

Direction loss

Depth loss

28

Normal Loss (optional): ℒ#●If we obtain the normal map , we can use it as optional loss.

●Normal should be perpendicular to the strand direction.

Photometricstereo
Light stage system

Photometric images

Normal map

29Normal Loss (optional): ℒ#●

If we obtain the normal map , we can use it as optional loss.

●

Normal should be perpendicular to the strand direction.

ℒ- " = $ + , -&'()'+∇&
30

Loss Function with Normal Loss
Find the depth map ) which minimizesℒ " = $# ℒ# " + $! ℒ! " + ℒ" (")
Normal loss(optional)

Direction loss

Depth loss

31

Overall Pipeline of Strand Integration

StrandIntegration

LPMVS[Nam et al.]Depth

Input images

Direction

Depth (refined)

Optional (if normal available)

Photometric
StereoPhotometric images

Normal32

Experiment33

Implementation●LPMVS [Nam et al.]○○●The official implementation is not publicly available.
CPU-based reimplementation in C++.

Strand Integration○○

Use PyTorch as a general gradient descent solver.
20 min for each view on Apple M1 Max. 1 1 MP(2730x4096) image.

34

Synthetic Data●

Rendered with pbrt-v4●60 views●2730x4096●4 hairstyles

TopBack

Straight

Curly

Wavy

Wavythin35

Result: Straight Hair$#
Ground Truth

LPMVS[Nam et al.]Strand Integration(ours)"
36

Result: Curly Hair$#
Ground Truth

LPMVS[Nam et al.]Strand Integration(ours)"
37

Result: Wavy Hair$#
Ground Truth

LPMVS[Nam et al.]Strand Integration(ours)"
38

Result: Wavythin Hair$#
Ground Truth

LPMVS[Nam et al.]Strand Integration(ours)"
39

Error Analysis on Depth Map

Abs. error

Ground Truth

Abs. error

LPMVS [Nam et al.]Strand Integration_

MAE : 30.05
RMSE: 52.57

MAE : 9.67
RMSE: 14.9240

Error Analysis on Depth Map

MAE / RMSE

Straight

Curly

Strand Integration
9.76 / 15.03

LPMVS
30.05 / 52.57

LPMVS
41.47 / 67.87

Wavy

LPMVS
34.21 / 60.30

Strand Integration
18.10 / 27.07

Wavythin

Strand Integration
12.20 / 20.57

LPMVS
34.96 / 58.54

Strand Integration
11.34 / 19.36

41

Ablation Study of Normal Loss

MAE / RMSE

Straight

Curlyw/ normal
9.67 / 14.92

w/o normal
9.76 / 15.03

w/o normal
18.10 / 27.07

Wavy

w/o normal
12.20 / 20.57

w/ normal
17.86 / 26.97Wavythin

w/ normal
12.25 / 20.53

w/o normal
11.34 / 19.36

w/ normal
11.23 / 19.13

42

Result: Merged Point Cloud

LPMVS [Nam+]

Strand Integration

43

Result: Merged Point Cloud
All point cloud

Numberof points

After filtered./0×10$./0×10$

12×10$

302×10$LPMVS [Nam+]

Strand Integration

LPMVS [Nam+] _Strand Integration
44

Real Data
●60 views●

5315x8001●

Long hair

45

Result: Real Data

LPMVS [Nam+]

Strand Integration46

Result: Real Data
All point cloud

Numberof points

After filtering

3425×10$3425×10$.36×10$744×10$LPMVS [Nam+]

Strand Integration

LPMVS [Nam+] _Strand Integration
47

Limitation●

Assume that hair is continuous and coherent everywhere.

●

Break when the hair is strongly curled or scattered.
Curly

MAE / RMSE

18.10 / 27.07

Straight

9.76 / 15.03
48

Official Code on GitHub
Official implementation

Strand Integration
Unofficial implementation

LPMVS[Nam el al., CVPR2019]

Synthetic data ofmulti-view images
Project page49

Conclusion●

Strand Integration: Refine the inaccurate
hair strand by integrating the gradientalong the hair strand.

Project page50