This CVPR paper is the Open Access version, provided by the Computer Vision Foundation.
Except for this watermark, it is identical to the accepted version;
the final published version of the proceedings is available on IEEE Xplore.

MVCPS-NeuS: Multi-view Constrained Photometric Stereo
for Neural Surface Reconstruction
Hiroaki Santo
Fumio Okura
Yasuyuki Matsushita
Graduate School of Information Science and Technology, Osaka University{santo.hiroaki, okura, yasumat}@ist.osaka-u.ac.jp

Abstract
Multi-view photometric stereo (MVPS) recovers a highfidelity 3D shape of a scene by benefiting from both multiview stereo and photometric stereo. While photometric stereo
boosts detailed shape reconstruction, it necessitates recording images under various light conditions for each viewpoint.
In particular, calibrating the light directions for each view
significantly increases the cost of acquiring images. To make
MVPS more accessible, we introduce a practical and easy-toimplement setup, multi-view constrained photometric stereo(MVCPS), where the light directions are unknown but constrained to move together with the camera. Unlike conventional multi-view uncalibrated photometric stereo, ourconstrained setting reduces the ambiguities of surface normal estimates from per-view linear ambiguities to a single
and global linear one, thereby simplifying the disambiguation process. The proposed method integrates the ambiguous
surface normal into neural surface reconstruction (NeuS) to
simultaneously resolve the global ambiguity and estimate
the detailed 3D shape. Experiments demonstrate that our
method estimates accurate shapes under sparse viewpoints
using only a few multi-view constrained light sources.

1. Introduction
Multi-view photometric stereo (MVPS) is a 3D reconstruction approach that combines photometric stereo (PS) andmulti-view stereo (MVS). In a conventional setting, a scene
is recorded from multiple viewpoints, and at each viewpoint, multiple images are captured under varying light directions [14, 19, 29, 40]. With the advancement of neuralsurface reconstruction [37, 41], recent works [14, 40] haveachieved high-fidelity 3D reconstruction by incorporating
the surface normal derived from PS via inverse rendering.
To eliminate the cost of calibrating the cameras and light
directions in MVPS, it is generally favored to work in an
uncalibrated setting, where the camera can freely move andlight directions are treated unknown. For determining thecamerastrobe⋯rotation stage
Figure 1. The minimal setting for multi-view constrained photometric stereo, equipping a camera, three strobes, and a manual rotation
stage with markers. The camera and light sources move together
with respect to the target object.

camera parameters including its postures, we can safely rely
on mature Structure-from-Motion (SfM) methods; however,uncalibrated light directions pose a problem in determining surface details. Namely, when the light directions are
unknown, the problem becomes uncalibrated photometricstereo (UPS), and it is known that surface normal can only
be estimated up to a linear ambiguity. As a result, in uncalibrated MVPS, it yields an ambiguous surface normal map
for each view, where each of them has a different liner ambiguity. Due to the per-view linear ambiguity in the estimatedsurface normal maps, it remains challenging to fully takeadvantage of PS’s capability in uncalibrated MVPS.
In this paper, we consider a setting of MVPS, where light
directions are unknown but constrained to move together
with a camera, which we call multi-view constrained photometric stereo (MVCPS). Such a setup can be easily achievedby employing a rig that secures a camera and light sources,
or by using a rotation stage to move a target object in frontof a camera and light sources, as shown in Fig. 1. In fact, thesetting is hardly new, and most existing single-view or multiview PS works employ such setups [19, 22, 25, 27, 31, 33–20475

In this section, we first review previous works of singleand multi-view PS. We then describe recent works of neuralsurface representations.

Multi-view Photometric Stereo (MVPS) Early works of
MVPS start with estimating a base shape from multi-view
observations, such as a mesh [29], SDF volume [22], andsparse point cloud [19], and then refine the shape using PS
to achieve detailed shape recovery. Kaya et al. [13] propose
an integration of recent deep learning-based MVS and PS.
Subsequently, their method has been extended [12, 14] toaccount for the uncertainties in the depth and normal estimations. While these methods achieve good 3D reconstruction,
they assume dense observations, i.e., the number of viewpoints and light sources is large enough to obtain accurate
estimations from both MVS and PS.
Along with the recent advances in neural surface reconstruction, Yang et al. [40] propose PS-NeRF, which incorporates normal maps estimated by PS into neural surface
reconstruction. They estimate per-view normal maps in advance and use them to optimize a neural surface. They use
the state-of-the-art learning-based UPS method [2] for normal map estimation for each view. However, when only alimited number of light sources are available, UPS fails toaccurately estimate the normal maps, resulting in a collapsed
shape estimation.
More recently, MVPSNet [44] proposes a feature extractor to leverage shading information observed under varying
lights, thereby facilitating improved stereo matching. They
also use the UPS method [2] to estimate the light directions.
The proposed method shares the spirit of PS-NeRF. However, to overcome the challenges posed by limited viewpoints
and light sources, we propose to simultaneously optimize
shapes and disambiguate surface normal in neural surface
reconstruction instead of involving per-view disambiguationof the surface normal.

Single-view Photometric Stereo Conventional PS [36,38] started with the Lambertian assumption [17] and known
light directions. The assumption of known light directions
has been later relaxed to deal with uncalibrated settings,
where light directions are treated unknown.
UPS simultaneously estimates scene shape and light directions and eliminates the necessity of light source calibration,
while there exists a linear ambiguity. Early works [1, 9]
employ the Lambertian assumption and factorize the input
observations into surface normals and light directions. To
resolve the ambiguity, the uses of known surface albedo [9],shadows [16], and specularity [4, 5] have been explored.
Recent works [2] use a learning-based approach to resolve the ambiguity by data prior. As discussed in [3], thelearning-based method also implicitly uses shadows andspecularity to disambiguate the estimation. More recently,
inverse rendering-based approaches [11, 18, 20] have been
proposed. While they achieve accuracy comparable to thatof calibrated settings in scenes with dense light sources, they
still face challenges when the light sources are sparse.

Neural Surface Reconstruction Neural implicit surfaces
have achieved remarkable advancements in novel view synthesis [26] and shape recovery [28, 37, 41, 42]. While thesemethods achieve high-quality shape reconstruction, they require a large number of images to accurately optimize theneural surface.Scenes with sparse viewpoints have been challenging, and several works tackle this problem. Long etal. [23] propose a pre-trained encoder that estimates a coarse
shape from sparse inputs. In the direction of using priorfrom a pre-trained model, Wu et al. [39] introduce the use ofmulti-view consistency, which has been used in conventional
MVS [7]. Yu et al. [43] propose the more explicit use of apre-trained prior, i.e., a monocular depth and normal estimation method. They estimate the per-view depth and normal
maps from a single RGB image by [6] for supervision.
Using prior knowledge from a large amount of data iseffective in cases where the training data covers test scenes.
In contrast, the proposed method uses additional observations under a few different light sources and demonstrates
superior performance in scenes with sparse viewpoints.

35, 45] because of the ease of implementation.
We show that, in MVCPS with unknown light directions,the ambiguity of surface normal can be reduced from perview ones to a single and global one. By jointly factorizing
the multi-view & multi-light observations, our method derives multi-view surface normal maps that have a unique andcommon linear ambiguity. Leveraging the surface normal
maps with reduced ambiguity, we develop a neural surface
reconstruction method that jointly resolves the ambiguity,
thereby achieving high-fidelity 3D shape recovery. The proposed method also introduces a confidence estimation forsurface normal based on the reduced ambiguity, enhancing
robustness against outliers like shadows.
Our experimental results demonstrate that the proposed
method yields more accurate estimations under minimal
light directions and sparse views, e.g., three light sources
and four viewpoints, compared to state-of-the-art multi-view
uncalibrated photometric stereo methods. To sum up, thispaper provides the following three contributions:• We show that our MVCPS reduces the per-view ambiguities into a single and global ambiguity, which allows forbetter disambiguation and shape recovery.
• We develop an outlier detection method based on the
global ambiguity for robust estimation.
• The proposed method achieves detailed shape reconstruction by integrating the ambiguous surface normal into
neural surface reconstruction.

2. Related Work20476

#light

3. Proposed Method

We begin with a naive extension of factorization-based UPS
for multi-view observations. We then show that, by assuming
the camera and lights move together, we can jointly factorize the multi-view observations to reduce the ambiguity ofsurface normal maps.
Singular Value Decomposition-based multi-view UPS
Hayakawa [9] propose a UPS method that decomposes single view observations under varying lights using singularvalue decomposition (SVD). It can be simply extended tothe multi-view context by solving UPS for each view. The
observation at i-th view, Mi ∈ Rp×l , can be decomposed
into left- and right-singular vectors Ui ∈ Rp×p , Vi ∈ Rl×land singular values Σi ∈ Rp×l as:
  \vM _i = \vU _i \vSigma _i \vV _i^\top . \nonumber⊤U⊤
i Ui = Ip , Vi Vi = Il , the diagonal elements of Σi are
singular values, and In is an n × n identity matrix. With theLambertian assumption, it becomes rank Mi = 3 for l ≥ 3
under different and non-coplanar light directions, the observations M can be written as a product of low-dimensional
matrices as  \vM _i = \underbrace {\vU '_i \vSigma '_i}_{\hat {\vN }_i} \underbrace {\vV _i^{'\top }}_{\hat {\vL }_i^\top }, \label {eq:single_svd_ups}(1)

where U′i ∈ Rp×3 and Vi′ ∈ Rl×3 are first three singular
vectors of Ui and Vi , respectively, and Σ′i is a 3 × 3 diagonal matrix containing singular values. The decomposedN̂i ∈ Rp×3 and L̂i ∈ Rl×3 denote the estimates of the surface normals and light directions for i-th view, respectively.
The solution is known to contain a linear ambiguity Xi ∈ R3×3 (where det(Xi ) ̸= 0) for each view, whichmeans any invertible 3 × 3 matrix Xi can be inserted as  \vM _i = \left (\hat {\vN }_i \vX _i\right ) \left (\vX _i^{-1} \hat {\vL }_i^\top \right ) = \hat {\vN }_i' \hat {\vL }_i' \label {eq:per_view_ambiguity}

(2)

#view

𝑵

⋯

ambiguity

light
𝑳

𝑿𝑿
𝑴

𝑵
⋯
⋯

3.1. Multi-view Constrained Photometric Stereo(MVCPS)𝑴⋯

Our method observes a target object from v viewpoints under l light directions and obtains observations {M1 , · · · , Mv } ⊂ Rp×l , where p represents the number of pixels. We assume that cameras’ intrinsics and extrinsics are known by SfM or markers placed in the scene. We
further assume that foreground masks {s1 , · · · , sv } ⊂ Rp
are available. The proposed method jointly decomposes theobservations {Mi } into per-view surface normal maps andshared light directions w.r.t. the camera. Subsequently, aneural surface is optimized with the supervision of color observations, foreground masks, and the decomposed surface
normals. In this section, we describe these details.

normalview-independent

Figure 2. Decomposition of multi-view observations by HO-GSVD

to obtain different combinations of surface normals and light
directions. As such, a straightforward application of UPS
to the multi-view setting suffers from per-view ambiguities {Xi }.
Our MVCPS via Higher-Order Generalized SVD Different from the general uncalibrated case described above,
our MVCPS setting assumes that the camera and lights move
together. In this setting, we can assume unknown but consistent light directions L̂i across all views with respect to thecamera as  \forall i, \hat {\vL }_i = \hat {\vL }.

(3)

With the knowledge of consistent light directions L̂,
we cast the problem of multi-view UPS to matrix factorization based on higher-order generalized SVD (HOGSVD) [15, 30]. With the HO-GSVD, we can decompose
an arbitrary number of observations {Mi } into {Ui }, {Σi },and common V ∈ Rl×l as  \vM _i = \vU _i \vSigma _i \vV ^\top .

(4)

Following Eq. (1), we obtain estimates of surface normal{N̂i } and a consistent light direction L̂:  \vM _i = \hat {\vN }_i \vX \vX ^{-1} \hat {\vL }^\top , \label {eq:multi_svd_ups}(5)

where N̂i = U′i Σ′i and L̂ = V′ ∈ Rl×3 is the first three
singular vectors of V. The matrix X ∈ R3×3 (det(X) ̸= 0)
is the global linear ambiguity. Compared to the per-view
linear ambiguities {Xi } that appeared in Eq. (2), with our
method, we can reduce the ambiguity to a global one X
shared by all the views (Fig. 2). While the decomposition ofobservations {Mi } into {N̂i } and L̂ can also be achieved
by applying SVD to the concatenated observation matri⊤⊤
ces M⊤, HO-GSVD offers a more accurate
1 , M2 , · · ·low-rank approximation (Eq. (5)) when more than threelight sources are available. As we will see in the next section, the proposed method resolves the ambiguity X throughneural surface reconstruction using the supervision by thedecomposed per-view surface normals {N̂i }.

20477

decomposed
normalvolume rendering

disambiguatednormal
𝑵𝑿

𝑵
𝑿

loss

SDF

Figure 3. Optimization of SDF by decomposed surface normal

3.2. Neural Surface Reconstruction for MVCPS
Once we have per-view surface normals {N̂i } with ambiguity X, we fuse them to a neural surface reconstruction
method by extending NeuS [37] to recover the detailed 3D
shape and resolve the ambiguity X simultaneously (Fig. 3).
Following the work of NeuS [37], we represent a scene
surface S by a zero-level set of a neural signed distancefunction (SDF) fθ : R3 → R as
  \vS = \{\vx \in \doubleR ^3| f_\theta (\vx )=0\}, \nonumber
where x is a 3D point, and θ is a learnable parameter. Our
MVCPS provides the surface normal maps {N̂i } with an
unknown global ambiguity represented as X. We will call{N̂i X} as disambiguated surface normal maps hereafter.
Normal consistency loss The gradient of the SDF,∇fθ (x) at a point x, represents the surface normal of thepoint. By applying volume rendering along a camera ray,p(t) = o + tv (t > 0), where o ∈ R3 and v ∈ S2 represent
a camera position and viewing direction, respectively, we
can compute the surface normal n∗ from the SDF as  \vn ^*(\vo , \vv ) = \int _0^{\infty } w(t) \grad f_\theta \left (\vp (t)\right ) dt. \nonumber
The weight w(t) for a point on the ray is computed from thevolume density σ(t) as  w(t) = T(t) \sigma (t), \nonumber
 Rtwhere T (t) = exp − o σ(u)du is an accumulated transmittance. We employ the normalized S-density [37] as thevolume density σ(t) computed from the SDF fθ . We also
follow the uniform and near-surface sampling strategy in[37] for computing the integration.
To optimize the ambiguity matrix X ∈ R3×3 throughbackpropagation, we introduce a normal consistency loss
between the SDF surface normal n∗ and the disambiguatedsurface normal {N̂i X}:
  \mathcal {L}_{\rm normal} = \sum _{(\vo ,\vv ) \in \chi } || \tau _{\rm c}\left (\vn ^*(\vo , \vv ) \right ) -\hat {\vn }(\vo , \vv ) ||_1, \label {eq:normal_loss}(6)

where τc denotes the transformation of a surface normal fromthe world coordinates to camera coordinates, and n̂(o, v) is

the surface normal derived from the corresponding pixel ofdisambiguated surface normals {N̂i X}.
Although the normal loss Lnormal does not resolve theambiguities by itself, the SDF surface normals n∗ (o, v) arealso constrained by photo and mask consistency losses in
the case of NeuS, which constrains the ambiguity matrix X.
Concurrently, the SDF is refined using the disambiguatedsurface normals, which contain detailed shape information
derived from MVCPS.
While photo-consistency loss is only applicable when thesame scene point is observed from multiple viewpoints, thenormal loss can constrain the surface orientation even when
the scene point is observed only from a single view, whichenables the recovery of shape from sparse observations.
Other loss functions We follow the original NeuS implementation for the remaining losses, considering color, mask,
and regularization. We here briefly recap them.
To optimize SDF fθ by color observations, we follow thevolume rendering of a radiance field cϕ : R3 × S2 → R3+
proposed by NeRF [26] and render the color C∗ ∈ R3+ as  \vC ^*(\vo , \vv ) = \int _0^{\infty } w(t) \vc _\phi \left (\vp (t), \vv \right ) dt. \nonumber
The color loss is computed between rendered and observed intensities as  \mathcal {L}_{\rm color} = \sum _{(\vo , \vv ) \in \chi } ||\vC ^*(\vo , \vv ) - \vC (\vo , \vv )||_1, \nonumber
where χ is the set of sampled rays and C(o, v) is theobserved intensity of the corresponding pixel in the input images. We employ the L1 loss for robust estimation.
For the observed intensities C, we use the median imageof input images captured under different light directions,as following [19], or images captured under natural illumination when available. We also employ the mask loss
and Eikonal loss [8] as described in [37]. The mask loss
Lmask = BCE (s∗ (o, v), s(o, v)) is defined as the binary
cross entropy between the rendered mask s∗ (o, v) and input
mask s(o, v). The Eikonal loss regularizes the norm of thegradients of the SDF fθ as
  \mathcal {L}_{\rm reg} = \sum _{\vx \in \psi } \left ( || \grad f_\theta \left ( \vx \right ) ||_2 - 1 \right )^2, \label {eq:eikonal_loss}(7)

where ψ is the set of sampled points on the rays.
Training The SDF fθ and radiance field cϕ are represented
by multilayer perceptrons (MLPs). We optimize the parameters of MLPs, θ and ϕ, and the ambiguity matrix X using
the following overall loss function:
  \mathcal {L} = \mathcal {L}_{\rm color} + \lambda _\text {mask} \mathcal {L}_{\rm mask} + \lambda _\text {normal} \mathcal {L}_{\rm normal} + \lambda _\text {reg} \mathcal {L}_{\rm reg}, \label {eq:loss}
(8)

20478

where λmask , λnormal , and λreg are weights for the losses. To
achieve better convergence in the simultaneous optimizationof the SDF parameters and ambiguity matrix X, we use acoarse-to-fine optimization strategy with positional encoding [21], which gradually enables high-frequency encoding
as the training progresses.

BLOBBYcameraobjectrotation stage

3.3. Confidence Estimation
The proposed factorization method in MVCPS assumes aLambertian surface without shadows. However, in the realworld, outliers like shadows and specularity cannot be neglected, as they lead to estimation errors. Particularly witha limited number of light sources, e.g., only three, accurate
estimation becomes difficult even using robust optimization
algorithms or recent learning-based methods. In the proposed
method, inspired by the fact that the ambiguity matrix X isshared across all views, we introduce a confidence estimation during training to migrate adverse effects arising frominaccurate estimations.
Let us represent the loss function L as a function of theambiguity matrix X and the view index i where rays are
sampled, denoted as L(X, i). We consider two different
views, u and v, and assess the changes in the loss at u-thview by applying the gradient of the loss at v-th view. Thechanges ∆L is written as
  \Delta \mathcal {L}_{u,v} = \mathcal {L}(\vX , u) - \mathcal {L}\left (\vX - \alpha \frac {\partial \mathcal {L}(\vX , v)}{\partial \vX }, u \right ), \label {eq:confidence_change_loss}(9)

where α is the step size of the updates. When assuming thatthe gradient at the v-th view, ∂L(X,v)∂X , improves the ambiguity matrix X, it is expected that the loss at u-th view should
improve, given that the ambiguity matrix X is shared acrossall views. Nonetheless, if the decomposed surface normal isincorrect at the u-th view, for example, due to shadows, theambiguity matrix X becomes irrelevant for such pixels. As
a result, the losses at these pixels may increase or decrease,regardless of the improvements in X.
Building on this premise, we propose a variance-based
confidence estimation method. We compute the changes ofthe loss ∆Lu,v for each pixel over different training steps
and construct the variance maps {q1 , · · · , qv } ⊂ Rp+ . Although it is not guaranteed that the gradient at the v-th viewis always accurate, from a statistical standpoint, the ambiguity matrix X is expected to improve as training progresses. Therefore, by statistically analyzing the changes inloss across different training steps, we expect the surface
normals of pixels exhibiting lower variance to have higher
confidence. From the variance maps {qi }, we compute theconfidence {q̄i } ∈ [0, 1]p by applying exponential mapping,  \bar {\vq _i} = \exp \left (-\kappa \frac {\vq _i}{\max (\vq _i)}\right ),

LEDs

BUNNY

(10)

Figure 4. Our synthetic dataset Figure 5. Setup of our real-world
experimentBEAR

BUDDHARABBIT

CUBE

FROG

Figure 6. Scenes for our real-world experiments

where κ is a hyperparameter to define the confidence forqi
a pixel with maximum variance, i.e., max(q= 1. In ouri)implementation, the parameter κ is set to 5. The normal loss
Lnormal is weighted by the confidence of each pixel.

4. Experiments
We evaluate the proposed method on our synthetic dataset,a public MVPS real-world dataset, DiLiGenT-MV, and ourreal-world dataset. In the following sections, we detail theexperimental settings and present the evaluation results.
Comparison methods We compare the proposed methodwith NeuS [37], PS-NeRF [40], and MonoSDF [43]. NeuS
and MonoSDF are MVS methods based on neural surface reconstruction, taking RGB images as inputs. As RGB images,
we use median images of images captured with varying light
directions. PS-NeRF is a state-of-the-art multi-view UPS
that optimizes the neural surface using RGB images and perview normal maps, which are estimated from single-view
images captured under multiple unknown lighting directions.
For a fair comparison, we adopt the same neural surface
representation as proposed by NeuS across all methods but
use the respective loss functions. More specifically, whentraining the comparison methods, we replace the normal lossof Eq. (6) by:
  \mathcal {L}_{\rm normal} = \sum _{(\vo ,\vv ) \in \chi } || \tau _{\rm c}\left (\vn ^*(\vo , \vv ) \right ) -\hat {\vn }'(\vo , \vv ) ||_1, \label {eq:normal_loss_comparison}(11)

where surface normal n̂′ is computed in the individual comparison methods. For PS-NeRF, the input surface normal n̂′is computed using a learning-based UPS [3], which is theextended version of [2] used in the original PS-NeRF. For
MonoSDF, we use a monocular normal and depth estimation

20479

method [6] to obtain per-view normal and depth maps from asingle image. We input the estimated normal maps as n̂′ andalso incorporate a scale- and shift-invariant loss, proposedin MonoSDF, between the estimated depth and the rendered
depth of the SDF. The weight of the depth loss is set to be
the same as that of the normal loss.
Implementation details We implement the proposed
method based on the official implementation of NeuS1 . We
train the model for 100K iterations with batch size 1024.
The training takes about 10 hours on a NVIDIA A6000 GPU.
We use Adam optimizer with default parameters. We use
different learning rates for updating parameters of MLPs,
θ and ϕ and the ambiguity matrix X, set to 5 × 10−4 and1 × 10−3 , respectively.
We set the weights of losses as λmask = 0.1 andλreg = 0.1. The weight of the normal loss λnormal is initially
set to 0.03 and linearly increased to 0.1 over the first 50K
iterations. The coarse-to-fine optimization via positional encoding is also adapted from 0 to 50K iterations. To computethe confidence map, we store the change of loss, Eq. (9),every 500 iterations with downscaled resolution. The update
step α in Eq. (9) is set to the learning rate for the ambiguity matrix. For the first 20K iterations, we do not use theconfidence weighting to ensure stable computation of thevariance. To avoid collapsed estimation, we fix the weight of
the normal loss λnormal for PS-NeRF and MonoSDF to 0.03.
Evaluation metrics For evaluation, we extract a meshfrom the optimized SDF using the Marching cubes algorithm [24]. Invisible surfaces from any cameras are excluded
using a rasterizer-based renderer [32], and we compute theChamfer distance between the point clouds extracted fromthe estimated and the ground truth meshes. We additionallyevaluate the angular error between the rendered normal mapsof the optimized SDF and the ground truth.

4.1. Synthetic Experiments
We render two scenes for our evaluation, BLOBBY [10] andBUNNY2 , using a rendering software, Blender3 . We use 24
viewpoints, rotating the camera at equal intervals aroundthe object, and for each viewpoint, render 3 images under
different directional lights, which move together with thecamera. As shown in Fig. 4, we use a textureless surface for
BLOBBY and a wood texture4 for BUNNY.
1 NeuS implementation, https://github.com/Totoro97/NeuS/,

last accessed on March 25, 2024.
2 Stanford Bunny, https://graphics.stanford.edu/data/

3Dscanrep/, last accessed on March 25, 2024.
3 Blender 3.3, https://www.blender.org/, last accessed on March

25, 2024.
4 Poly Haven, https://polyhaven.com/, last accessed on March 25,

2024.

As discussed in [3], specularity is useful for accurate
estimation of UPS; hence, we include specularities in the
input images for PS-NeRF and MonoSDF. Conversely, theproposed method and NeuS assume the Lambertian surface,
and thus, we render them as diffuse surfaces.
We use four views located at 90-degree intervals and threelight directions. Figure 7 and Table 1 present the estimatednormal maps and estimation errors, respectively. Here, we
visualize only one view per scene, and more complete results
can be found in our supplementary material. Due to sparse
viewpoints, NeuS fails to reconstruct correct surfaces. In
contrast, MonoSDF, which uses the same inputs as NeuS, canaccurately recover the rough shapes. However, MonoSDF
tends to produce over-smoothed surfaces, as observed in the
BUNNY scene. PS-NeRF provides a reasonable estimation,
but due to the lower accuracy of surface normal estimation
by UPS, the accuracy of its optimized shapes is not as highas that of the proposed method. In terms of both the Chamferdistances and mean angular errors, we observe the accurate
estimation achieved by the proposed method.

4.2. Real-world Experiments
We use two datasets for real-world evaluation, DiLiGenTMV [19] and our own dataset. We first describe the datasetsand then introduce the experimental results.
DiLiGenT-MV: DiLiGenT-MV captures five objects from
20 viewpoints and uses fixed 96 light directions. Since theproposed method assumes the Lambertian reflectance, we
here show the results for two scenes with relatively diffuse
materials, BEAR and BUDDHA shown in Fig. 6. The remaining scenes are shown in our supplementary material.
Our dataset: For recording real-world data, we use a polarimetric camera (FLIR BFS-U3-51S5PC-C) for obtaining
the diffuse-only and diffuse+specular observations for comparison methods. For just running our method, in practice,
we can simply attach an off-the-shelf polarization filter to an
ordinary camera, as shown in Fig. 1.
We capture three objects: RABBIT, CUBE, and FROG,
shown in Fig. 6. RABBIT and FROG are made of pottery,
while CUBE is made of plastic. We use our capturing setup,shown in Fig. 5, to capture images from 60 viewpoints with
3 light directions, rotating the target object. The ground truth
is obtained by a laser scanner and aligned with the estimatedmeshes by NeuS, using all of the 60 views.
Results: Figures 7 and 8 visualize the estimated normal
maps and meshes, respectively, and Table 1 represents the estimation errors. Similar to our synthetic experiments, NeuS
presents larger errors for most of the scenes, and PS-NeRF
faces challenges in recovering the normal map from limited observations. Although our dataset (RABBIT, CUBE,
and FROG) is captured using cross-polarization to reduce
specular reflections, BEAR and BUDDHA contain specu-

20480

Table 1. Comparison results with four views. PS-NeRF and ours use three light sources. “CD” denotes the Chamfer distance (↓) between themesh extracted from the estimated SDF and the ground truth. “MAE” denotes mean angular errors (↓) in degrees across all available views.
We show the MAE of rendered normal maps of the estimated SDF and estimated normal maps fed to the optimization. The estimated normal
maps by the proposed method are disambiguated by the estimated ambiguity matrix X. Bold font and underline are used to denote the bestand second-best results, respectively.

BLOBBY
BUNNY
BEAR
BUDDHARABBIT
CUBE
FROG

CD
24.3
16.3
10.2
26.6
2.17
95.2
40.1

NeuS
MAE (SDF)
14.0
21.9
10.4
30.1
13.1
34.6
33.7
NeuS

MonoSDF
MAE (SDF) MAE (Est.)
8.40
30.9
11.5
20.6
5.81
13.0
23.1
26.2
14.2
26.2
9.64
14.1
23.6
26.2

CD
25.3
5.38
6.32
13.4
3.81
9.50
13.5

MonoSDFPS-NeRFMAE (SDF) MAE (Est.)
16.0
7.05
8.53
17.1
7.02
18.1
29.8
22.5
13.4
29.8
14.7
32.6
19.3
25.6

CD
8.83
1.79
1.30
2.18
1.65
0.85
2.01

Ours
MAE (SDF)
2.28
5.38
5.11
14.5
9.67
8.34
14.8

PS-NeRFMAE (Est.)
7.01
12.4
12.9
19.3
15.2
12.6
18.2

Ours

Normal (SDF)

Normal (SDF)

Normal (Est.)

Normal (SDF)

Normal (Est.)

Normal (SDF)

Normal (Est.)

11.5

9.7

29.4

5.1

11.0

1.9

5.5

24.5

9.7

10.6

8.1

15.7

6.8

14.2

13.4

5.5

11.4

10.1

15.6

5.0

12.1

33.8

25.1

26.1

18.9

35.7

13.9

15.3

14.2

15.3

23.9

16.0

28.8

11.5

17.6

38.7

9.3

13.7

8.6

27.1

7.8

13.3

CUBE

RABBIT

BUDDHA

BEAR

BUNNY

BLOBBY

Ground truth

CD
27.3
7.61
1.7
12.2
2.20
0.90
20.2

FROG

high ◦
45

low
0

35.1

23.6

27.1

18.8

21.5

14.9

19.1

Figure 7. Evaluation of the normal maps. For each scene and method, we present the rendered normal map of the SDF, the estimated normal
map fed to the optimization, and corresponding error maps side-by-side. The numbers under the error maps represent mean angular errors indegrees. The estimated normal maps by the proposed method are disambiguated by the estimated ambiguity matrix X.

lar reflections. Nevertheless, the proposed method robustly
estimates accurate shapes.
In scenes such as BEAR and CUBE, the monocular normal estimation in MonoSDF performs well, leading to accurate estimations. However, for example, in the CUBE

scene, the detailed shape is lacking. In contrast, the proposed
method can recover detailed shapes in the CUBE scene.
BUDDHA and FROG present challenges for all methods;
however, the proposed method achieves globally accurate
shape recovery.

20481

Table 2. Comparison of the proposed method with and without HO-GSVD. “MAE (UPS)” and “MAE (SDF)” denote the mean angularerrors (↓) in degrees for the disambiguated normal maps and rendered normal maps of SDF, respectively. “Proj. mat.” represents the error
between the estimated ambiguity matrix and the ground truth measured by the Frobenius norm (↓). Bold font indicates better results.

Ours
w/o HO-GSVD

NeuS

4 views
Proj. mat.
0.00230.0050

MonoSDF

MAE (SDF)
1.78
3.00

PS-NeRFMAE (UPS)
0.085
0.59

Ours

FROG

CUBE

RABBIT

BUDDHABEAR

GT

MAE (UPS)
0.13
0.62

Figure 8. Estimated meshes for the real-world scenes.

Est.

Err. map Conf. map

1

w/o Conf.

Ours

GT

0
45°
0°

Figure 9. Results of confidence estimation. The left-hand side showsthe estimated meshes for the BUNNY scene, both ours and ours
without confidence estimation (“w/o Conf.”). On the right-handside, for each row, the first two columns present the normal maps ofthe ground truth and the one estimated, projected by the estimatedambiguity matrix. The last two columns show the error maps (“Err.
map”) of the estimated normal and confidence maps (“Conf. map”).

5. Discussion
This paper presents a practical and easy-to-implement 3D
reconstruction method, MVCPS. Our constrained setting allows us to decompose the observations into per-view surface

8 views
Proj. mat.
0.00100.0037MAE (SDF)
1.32
2.01

MAE (UPS)
0.054
0.421

20 views
Proj. mat.
0.00055
0.0026MAE (SDF)
1.29
1.77

normal maps and shared light directions w.r.t. the camera by
HO-GSVD, reducing the per-view ambiguities in UPS to asingle and global linear ambiguity. We demonstrate that, byintegrating the decomposed normal maps into neural surface
reconstruction, the proposed method can jointly estimate
accurate 3D shapes and the ambiguity matrix. We compare
the proposed method with the state-of-the-art methods andshow the proposed method’s effectiveness in the challenging
setting of sparse views and lights.
One of the limitations of our method is the Lambertianassumption, which is rarely met in the real world. Dealing
with non-Lambertian objects under sparse viewpoints andlight sources is one of our future venues.
Ablation study on HO-GSVD To assess the impact of HOGSVD compared to SVD-based UPS, we evaluate the accuracy of disambiguation in the proposed optimization stage.
We assume that the factorization perfectly works and use
the normal maps of the ground truth as input. We use the
BUNNY scene with 4, 8, and 20 views. Table 2 comparesthe disambiguated accuracy of the surface normal using ashared linear ambiguity across all views (HO-GSVD), withthose using per-view linear ambiguities (SVD). Since we canonly disambiguate the surface normals of the views used in
the optimization when using view-independent ambiguities,
we compare the mean angular errors for those specific viewsin this experiment. This result demonstrates the consistentadvantage of HO-GSVD for more accurate ambiguity resolution. We further investigate the effectiveness of HO-GSVD
in improving factorization accuracy, with details provided inour supplementary material.
Ablation study on confidence estimation Figure 9 shows
the results of the proposed method with and without confidence estimation. We also visualize the disambiguated normal maps, the corresponding error maps, and the estimatedconfidence maps. As seen in the error maps and the confidence maps, pixels with higher errors tend to have lower
confidence, as expected. In the BEAR scene, both shadowed and specular pixels exhibit lower confidence, which
contributes to robust estimation. Observing the estimatedmesh without confidence estimation reveals that the shape isheavily affected by the shadow on the ear.

Acknowledgement
This work was supported by JSPS KAKENHI Grant Numbers JP22K17910 and JP23H05491.

20482

References[1] P.N. Belhumeur, D.J. Kriegman, and A.L. Yuille. The basrelief ambiguity. In Computer Vision and Pattern Recognition(CVPR), 1997. 2[2] Guanying Chen, Kai Han, Boxin Shi, Yasuyuki Matsushita,
and Kwan-Yee K. Wong. SDPS-Net: Self-calibrating deep
photometric stereo networks. In Computer Vision and Pattern
Recognition (CVPR), 2019. 2, 5
[3] Guanying Chen, Michael Waechter, Boxin Shi, Kwan-Yee K.
Wong, and Yasuyuki Matsushita. What is learned in deep
uncalibrated photometric stereo? In European Conference on
Computer Vision (ECCV), 2020. 2, 5, 6
[4] Ondrej Drbohlav and Mike Chaniler. Can two specular pixels
calibrate photometric stereo? In International Conference on
Computer Vision (ICCV), 2005. 2[5] Ondřej Drbohlav and Radim Šára. Specularities reduce ambiguity of uncalibrated photometric stereo. In European Conference on Computer Vision (ECCV), 2002. 2[6] Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir
Zamir. Omnidata: A scalable pipeline for making multi-task
mid-level vision datasets from 3D scans. In International
Conference on Computer Vision (ICCV), 2021. 2, 6
[7] Yasutaka Furukawa and Jean Ponce. Accurate, dense, androbust multiview stereopsis. Transactions on Pattern Analysis
and Machine Intelligence (PAMI), 32(8):1362–1376, 2009. 2[8] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, andYaron Lipman. Implicit geometric regularization for learning
shapes. In International Conference on Machine Learning(ICML), 2020. 4[9] Hideki Hayakawa. Photometric stereo under a light sourcewith arbitrary motion. Journal of the Optical Society of America (JOSA), 11(11):3079–3089, 1994. 2, 3
[10] Micah K. Johnson and Edward H. Adelson. Shape estimation in natural illumination. In Computer Vision and Pattern
Recognition (CVPR), 2011. 6[11] Berk Kaya, Suryansh Kumar, Carlos Oliveira, Vittorio Ferrari,and Luc Van Gool. Uncalibrated neural inverse rendering forphotometric stereo of general surfaces. In Computer Vision
and Pattern Recognition (CVPR), 2021. 2[12] Berk Kaya, Suryansh Kumar, Carlos Oliveira, Vittorio Ferrari,and Luc Van Gool. Uncertainty-aware deep multi-view photometric stereo. In Computer Vision and Pattern Recognition(CVPR), 2022. 2[13] Berk Kaya, Suryansh Kumar, Francesco Sarno, Vittorio Ferrari, and Luc Van Gool. Neural radiance fields approach to
deep multi-view photometric stereo. In IEEE Winter Conference on Applications of Computer Vision (WACV), 2022.
2[14] Berk Kaya, Suryansh Kumar, Carlos Oliveira, Vittorio Ferrari,and Luc Van Gool. Multi-view photometric stereo revisited.
In IEEE Winter Conference on Applications of Computer
Vision (WACV), 2023. 1, 2
[15] Idris Kempf, Paul J. Goulart, and Stephen R. Duncan. A
higher-order generalized singular value decomposition for
rank-deficient matrices. SIAM Journal on Matrix Analysis
and Applications, 44(3):1047–1072, 2023. 3[16] David J. Kriegman and Peter N. Belhumeur. What shadows
reveal about object structure. Journal of the Optical Societyof America (JOSA), 18(8):1804–1813, 2001. 2[17] JH Lambert. Photometria. Augustae Vindelicorum, 1760. 2[18] Junxuan Li and Hongdong Li. Self-calibrating photometricstereo by neural inverse rendering. In European Conference
on Computer Vision (ECCV), 2022. 2[19] Min Li, Zhenglong Zhou, Zhe Wu, Boxin Shi, Changyu Diao,
and Ping Tan. Multi-view photometric stereo: A robust solution and benchmark dataset for spatially varying isotropic
materials. IEEE Transactions on Image Processing, 29:4159–
4173, 2020. 1, 2, 4, 6
[20] Zongrui Li, Qian Zheng, Boxin Shi, Gang Pan, and Xudong
Jiang. Dani-net: Uncalibrated photometric stereo by differentiable shadow handling, anisotropic reflectance modeling,
and neural inverse rendering. In Computer Vision and Pattern
Recognition (CVPR), 2023. 2[21] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon
Lucey. BARF: Bundle-adjusting neural radiance fields. In
International Conference on Computer Vision (ICCV), 2021.
5[22] F. Logothetis, R. Mecca, and R. Cipolla. A differential volumetric approach to multi-view photometric stereo. In International Conference on Computer Vision (ICCV), 2019. 1,
2[23] Xiaoxiao Long, Cheng Lin, Peng Wang, Taku Komura, andWenping Wang. SparseNeuS: Fast generalizable neural surface reconstruction from sparse views. In European Conference on Computer Vision (ECCV), 2022. 2[24] William E. Lorensen and Harvey E. Cline. Marching cubes:
A high resolution 3D surface construction algorithm. In
SIGGRAPH, New York, NY, USA, 1987. Association for
Computing Machinery. 6[25] Lilika Makabe, Heng Guo, Hiroaki Santo, Fumio Okura, andYasuyuki Matsushita. Near-light photometric stereo withsymmetric lights. In IEEE International Conference on Computational Photography (ICCP), 2023. 1[26] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik,
Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF:
Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV),
2020. 2, 4[27] Kazuma Minami, Hiroaki Santo, Fumio Okura, and Yasuyuki Matsushita. Symmetric-light photometric stereo. In
IEEE Winter Conference on Applications of Computer Vision(WACV), 2022. 1[28] Michael Oechsle, Songyou Peng, and Andreas Geiger.
UNISURF: Unifying neural implicit surfaces and radiancefields for multi-view reconstruction. In International Conference on Computer Vision (ICCV), 2021. 2[29] Jaesik Park, Sudipta N Sinha, Yasuyuki Matsushita, Yu-Wing
Tai, and In So Kweon. Multiview photometric stereo using
planar mesh parameterization. In International Conference
on Computer Vision (ICCV), 2013. 1, 2[30] Sri Ponnapalli, Michael Saunders, Charles Loan, and Orly
Alter. A higher-order generalized singular value decomposition for comparison of global mrna expression from multiple
organisms. PloS one, 6:e28072, 2011. 3

20483

[31] Yvain Quéau, Bastien Durix, Tao Wu, Daniel Cremers,
François Lauze, and Jean-Denis Durou. LED-based photometric stereo: Modeling, calibration and numerical solution.
Journal of Mathematical Imaging and Vision, 60(3):313–340,
2018. 1[32] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia
Gkioxari. Accelerating 3D deep learning with PyTorch3D.
arXiv:2007.08501, 2020. 6[33] Hiroaki Santo, Michael Waechter, and Yasuyuki Matsushita.
Deep near-light photometric stereo for spatially varying reflectances. In European Conference on Computer Vision(ECCV), 2020. 1[34] Hiroaki Santo, Masaki Samejima, Yusuke Sugano, Boxin Shi,and Yasuyuki Matsushita. Deep photometric stereo networks
for determining surface normal and reflectances. Transactions
on Pattern Analysis and Machine Intelligence (PAMI), 44(1):114–128, 2022.
[35] Boxin Shi, Zhipeng Mo, Zhe Wu, Dinglong Duan, Sai-Kit
Yeung, and Ping Tan. A benchmark dataset and evaluation for
non-Lambertian and uncalibrated photometric stereo. Transactions on Pattern Analysis and Machine Intelligence (PAMI),
41(2):271–284, 2019. 2[36] William M. Silver. Determining shape and reflectance using
multiple images. Master’s thesis, Massachusetts Institute ofTechnology, 1980. 2[37] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku
Komura, and Wenping Wang. NeuS: Learning neural implicit
surfaces by volume rendering for multi-view reconstruction.
In Neural Information Processing Systems (NeurIPS), 2021.
1, 2, 4, 5[38] Robert J. Woodham. Photometric method for determining surface orientation from multiple images. Optical engineering,19(1):139–144, 1980. 2[39] Haoyu Wu, Alexandros Graikos, and Dimitris Samaras. SVolSDF: Sparse multi-view stereo regularization of neural
implicit surfaces. In International Conference on Computer
Vision (ICCV), 2023. 2[40] Wenqi Yang, Guanying Chen, Chaofeng Chen, Zhenfang
Chen, and Kwan-Yee K. Wong. PS-NeRF: Neural inverse
rendering for multi-view photometric stereo. In European
Conference on Computer Vision (ECCV), 2022. 1, 2, 5[41] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, MatanAtzmon, Basri Ronen, and Yaron Lipman. Multiview neuralsurface reconstruction by disentangling geometry and appearance. In Neural Information Processing Systems (NeurIPS),
2020. 1, 2[42] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. In Neural Information Processing Systems (NeurIPS), 2021. 2[43] Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. MonoSDF: Exploring monoculargeometric cues for neural implicit surface reconstruction. In
Neural Information Processing Systems (NeurIPS), 2022. 2,5[44] Dongxu Zhao, Daniel Lichy, Pierre-Nicolas Perrin, JanMichael Frahm, and Soumyadip Sengupta. MVPSNet: Fast

generalizable multi-view photometric stereo. In International
Conference on Computer Vision (ICCV), 2023. 2[45] Zhenglong Zhou and Ping Tan. Ring-light photometric stereo.
In European Conference on Computer Vision (ECCV), 2010.
220484