GNeRP: Gaussian-guided Neural Reconstruction of Reflective Objects with Noisy Polarization Priors
Authors: LI Yang
RUIZHENG WU
Jiyong Li
Ying-Cong Chen
Venue: ICLR 2024 poster (2024)
Keywords: Neural 3D Reconstruction
Specular Reflection
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Learning surfaces from neural radiance field (NeRF) became a rising topic in Multi-View Stereo (MVS).  Recent Signed Distance Function (SDF)-based methods demonstrated their ability to reconstruct exact 3D shapes of Lambertian scenes. However, their results on reflective scenes are unsatisfactory due to the entanglement of specular radiance and complicated geometry. To address the challenges, we propose a Gaussian-based representation of normals in SDF fields. Supervised by polarization priors, this representation guides the learning of geometry behind the specular reflection and capture more details than existing methods.  Moreover, we propose a reweighting strategy in optimization process to alleviate the noise issue of polarization priors. To validate the effectiveness of our design, we capture polarimetric information and ground truth meshes in additional reflective scenes with various geometry. We also evaluated our framework on PANDORA dataset. Both qualitative and quantitative comparisons prove our method outperforms existing neural 3D reconstruction methods in reflective scenes by a large margin.

Published as a conference paper at ICLR 2024
GNERP: GAUSSIAN-GUIDED NEURAL RECONSTRUC- TION OF REFLECTIVE OBJECTS WITH NOISY POLAR- IZATION PRIORS
Yang LI
Ruizheng WU
Jiyong LI
Yingcong CHEN12
AI Thrust, HKUST(GZ). Nansha, Guangzhou, China
DDpaarmment of Computer Science & Engineering, HKUST, Clear Water Bay, HongKong SAR, China Department of Computer Science, Sun Yat-sen University, Panyu, Guangzhou, China *R & D Center, SmartMore, Qianhai, Shenzhen, China
f Corresponding Author
y118038ccnnect. usk rzwuêcse, cubk. .k, 11jy3730ma112 sysu, edu, on,
yingcongcheneust..ig
ABSTRACT
Learning surfaces from neural radiance field (NeRF) became a rising topic Multi-View Stereo (MVS). Recent Signed Distance Function (SDF)- based meth- ods demonstrated their ability to reconstruct accurate 3D shapes Lambertian scenes. However, their results reflective scenes unsatisfactory due the entanglement of specular radiance and complicated geometry. To address the challenges, we propose Gaussian-based representation of normals in SDF fields. Supervised by polarization priors, this representation guides the learning of geom- etry behind the specular reflection and captures more details than existing meth- ods. Moreover, we propose reweighting strategy in the optimization process to alleviate the noise issue of polarization priors. To validate the effectiveness of our design, we capture polarimetric information, and ground truth meshes in addi- tional reflective scenes with various geometry. also evaluated our framework the PANDORA dataset. Comparisons prove our method outperforms existing neural 3D reconstruction methods in reflective scenes by large margin.
INTRODUCTION
Reconstructing 3D shapes from 2D images (Furukawa et al.) 2013) is a fundamental problem in computer vision and graphics, with downstream applications such 3D printing (Chen & Yang 2014), autonomous driving (Chen et al., 2017), and Computer Aided Design (Furukawa et al. 2010) Although diffuse objects precisely reconstructed, reflective and textured-less scenes remain chal- lenging. Traditional Vulti-View Stereo (MVS) methods (Bregler et al.) (2000) rely on stereo match- ing across views, which is hindered in the presence of specular surfaces and texture absence. Recent methods utilizing implicit neural representation learning for 3D reconstruction have shown promis- ing accuracy (Mescheder al. 2019 Yariy al., (02T), yet they overlook the specular reflection between light rays and surfaces, failing to adequately handle glossy objects with figh-frequency specular reflection.
Existing methods (Zhang et al. 202 Liu et al. 2023 Dave et al. 2022 attempt to separate spec- ular reflection components from radiance to improve reconstruction process. These methods model the interaction of light rays and surfaces by Bidirectional Reflectance Distribution Functions (BRDFs) and estimate them by neural networks. However, the inverse problem posed by BRDFs formulation is highly ill-posed (Guo et al. 2014), and low-frequency bias (Xu et al., (2019) of neural BRDFs making the learned geometry over smoothed (Liu et al. (023). Therefore, geometry with specular reflection shown in Fig. I (a) intractable for them. Besides, few meth- ods employ polarization priors to facilitate the learning of specular reflection because they reveal information about surface normals. However, polarization information is always concentrated in

---

Published as a conference paper at ICLR 2024
2023 making the recon- specular- dominnt regions and noisy in diffuse regions (Kajiyama al. struction process in deffuseedominant regions distorted.
Faced with the bias of neural BRDF and noise issues of polarization priors, present novel per- spective for reconstructing the detailed geometry reflective objects. Our key idea is to extend the geometry representation from scalar SDFs Gaussian fields of normals supervised by polar- ization priors. Given a surface point, the normals within its neighborhood are approximated by a 3D Gaussian. And it's more informative representation of geometry. The mean shows the over- all fow-freqqeene orientation of the surface, while the covariance captures high-frequency details. Coincidentally, the representation can splatted into the image plane 2D Gaussians, as illus- trated in Fig. [T (b). The splatting skips the disentangled specular radiance. Learning of the 2D Gaussians can be directly supervised by the polarization information about surface normals. Hence, it circumvents the separation of complex geometry and specular reflection and manages to learn detailed geometry.
2D Gaussian
Polarization Príor
rays
specular radiance
surface
Rendered Gaussian
Gaussian
(a) Neural 3D Reconstruction
(b) GNeRP
Figure 1: Visualization of Gaussians of normals in Neural Reconstruction pipelines. 2D Gaussians can be rendered from 3D Gaussians of learned normals.
Furthermore, to tackle the noise issues of polarization priors, we introduce Degree of Polarization (DoP) based reweighting strategy. This strategy adaptively balances the supervision of radiance and polarization priors, enhancing the reconstructing accuracy in diffuse-dominant regions.
In summary, our contributions are as follows:
We propose novel polarization-basee Gaussian representation of detailed geometry to guide the learning of geometry behind specular reflection.
We propose a DoP reweighing strategy alleviate noise and imbalance distribution prob- lems of polarization priors.
We collect new challenging multi-view dataset consisting of both radiance and polarimet- ric images with more diverse and challenging scenes.
RELATED WORK
2.1 MULTI-VIEW 3D RECONSTRUCTION
Traditional Multi-view Stereo focuses on the extraction of cross-view features to generate 3D points. (Schönberger al. 2016 Galliani et 2015) try estimate the depth map of the observed scene with multi-view consistency and fuse the depth maps into dense point clouds. These methods suf- fer from accumulating errors due to complex pipelines, and features are hard to be extracted from reflective objects. (Mescheder et al. 2019) explicitly models the objects' occupancy in a voxel grid guarantee a complete object model created. However, the resolution of the voxel limits the ac- curacy of the reconstructed surface. Recently, the success of NeRF (Mildenhall et al. (2020), which uses a simple MLP to encode the color and density information for scene, inspired researchers to resort to implicit representation for multi-view 3D reconstruction. The representative works are Unisurf (Oechsle et al. 2021), NeuS (Wang et al., 2021), and VoISDF (Yariv et al., 2021), which exploit MLP model Signed Distance Function (SDF) for target scene. These methods opti- mize the implicit representation, i.e., SDF, by minimizing the MSE loss between the rendered pixel's radiance value and the corresponding pixel's radiance value in GT images. Such a paradigm works
2

---

Published as a conference paper at ICLR 2024
well with Lambertian surfaces. However, only view-conditioned radiance fields fail in reflective scenes.
2.2 BRDF FOR REFLECTIVE OBJECTS RECONSTRUCTION
In the regions with complex geometry, BRDFs always exhibit high-frequency variations due to the normals terms, while the low-frequency implicit bias of neural networks (Xu al. 2019) disables neural BRDFs from predicting these abrupt changes. It always results in over-smoothed geometry. For example, NeRO (Liu et al. 2023) adopts Micro-facet BRDF (Cook & Torrance 1981) parame- terized by material and normal distribution terms. Although its results of smooth mirror-like objects are excellent, the spatial continuity of neural BRDF is a barrier to the combination of complex ge- ometry and specular reflection. In the regions with complex geometry, sole multi-view images with disentangled radiance result in severe ill-posedness of the inverse problem, as is shown in Fig. (a). Moreover, explicit estimation of anisotropic normals distribution has been used in rendering delicate objects, such as anisotropy shading of hairs (Banks) 1994), to improve the perception of orientation and shapes (Ament Dachsbacher 2015). However, anisotropic normals distribution in neural SDFs for 3D reconstruction remains under-defined and non-trivial. Our method proposes 3D Gaus- sians, of which anisotropic 3D covariance is more informative than the scalar normals distribution term in NeRO. The latter only measures the concentration of normals surface point.
2.3 MULTI-VIEW 3D RECONSTRUCTION WITH POLARIZATION
Polarization prior reveals the azimuth angle of the surface normal, i.e., the angle between the nor- mal projection onto the image plane and the positive x-axis of the image. Shape-frome foo--frooppo frrompooaaizzaion has been investigated by other papers (Atkinson Hancock 2006:] Foster al. [2018] Fukao al.]
2021 Cui al. 2017: Kadambi et al. 2015 Zhao et al. (020) before the invention of neural 3D reconstruction. But most of them are focused on common scenes. For example, PMVIR (Zhao et al.) 2020 exploits the relation of the polarization angle and the azimuth angle of normals but with only Lambert shading, and thus it cannot treat reflective objects at all. Neural 3D Reconstruction with polarization priors has also been explored. Sparse Ellipsometry (Hwang et al., (222)) develops de- vice to capture polarimetric information and 3D shapes concurrently. However, their reconstruction always disturbed by the noise in diffuse-dominant regions. For example, PANDORA (Dave al., 2022) extends radiance in BRDF into polarimetric dimensions while the geometry of diffuse regions cannot be learned properly.
2.4 GAUSSIANS IN 3D SCENE REPRESENTATION
Gaussians are used to represent the attributes of 3D scenes. Mip-NeRF (Barron et al. 2021 encodes Gaussian regions of space rather than infinitesimal points for anti-aliasing. (Zwicker et al. 2001) proposes Gaussian splatting that taking volume data as 3D Gaussians and nearly projects the 3D Gaussian the 2D one (Kerbl al. 2023). (Kerbl al., 2023) implements the splatting pipeline on the NeRF for real-time rendering. In numerical geometry, (Berkmann & Caelli 1994) calculates the covariance matrix from the projections of the normal vectors to highlight the edges and local geometry of surfaces. Inspired by them, we demonstrate a further fact that taking surface normals as 3D Gaussians and going through similar splatting pipeline would exactly be transformed into Gaussians. Our 2D Gaussians are coincidentally available for polarization priors. Thus, supervised by polarization priors, the learned 3D Gaussians capture more details, which represent the average orientation of normals by means and the changes within the neighborhood by covariance matrices.
METHODS
3.1 PRELIMINARY OF POLARIZATION
Here, introduce the concept of polarization and its mathematical relation to surface normals projected to the captured images. The prior contributes to the disentanglement of specular radiance and geometry.
3

---

Published as a conference paper at ICLR 2024
of Reflection
for
Polarized Direction
Camera Center
Reflected Light
X-Y Projection
rined Direction
Incident Light
Surface Normal
Polarization Camera
Surface Normal
Figure 2: Illustration of polarization shift in specular reflection. The right figure is detailed de- scription of the geometry relation between AoP and the surface normal. vi is the azimuth angle, yp is the AoP, which is the angle from the positive x-axis to the polarized direction.
Polarimetry describes the vibration status of light waves. Since light is type of transverse wave that only oscillates in the plane perpendicular the light path (Colleti 2005). the full polarimetric cues of rays are always represented by planar ellipses (Smith & Ward 1974). The magnitude of vectors inside these ellipses alludes to the amplitude of the light wave vibration along the vectors, as shown in Fig. [ Common light sources, such as sunlight and LED spotlights, emit unpolarized our captured scenes, objects are mostly assume that the incident light is unpolarized. During reflection, the vibration each direction absorbed unequally, and unpolarized incident light turns into partially polarized reflected light captured by polarization cameras. The Angle of Polarization (AoP) and Degree of Polarization (DoP) are two cues of the polarization ellipse functionally related to projected surface normals at the points of reflection, which be formulated as:
light, i.e., the light vibrates equally in all directions. illuminated directly by light sources,
(i. arctan $2(1,))
1
(i((i.)) + (i
p(i.j)
{s. E
(1)
S1 (i,
so(i,j
where s: P are AoP and DoP, (i,j) is the pixel index, and s = [So, S1, $2, S3] is Stokes vector rectly calculated from polarization capture. Generally, in specularity-domine
regions, the relation
between projected normals and AoP fixed as bb and the equation /i + 2 = & mod m.
Moreover, DoP is significantly higher in these regions. Details of polarization analysis are shown in the Appendix.
3.2 GAUSSIAN GUIDED POLARIMETRIC NEURAL 3D RECONSTRUCTION PIPELINE
Polarimetric neural 3D reconstruction refers to reconstructing surfaces by neural implicit surface learning, given N calibrated multi-view images X' {C,).y with pixcl-aligned polarization pri- ors y = (sp;, p; 1.V First, we introduce general pipeline of learning surface by volume rendering, taking NeuS (Wang et al. 202I) 1) as an example. Sec. 3.2.2 introduces the 3D Gaussian of surface normals and its transforms 2D Gaussian in the image plane. represents the geometry of sur- face points more precisely and thus can separate detailed geometry from high-frequency specular radiance. 3.2.3 presents full optimization containing radiance loss and Gaussian loss, which measures the gap between these 2D) Gaussians and polarization priors. We propose a DoP reweigh- ing strategy alleviate the aforementioned noise and imbalanced distribution of polarization priors. It balances the influence of radiance and polarimetric cues adaptively. Finally, Sec. An overview of the entire framework is shown in Fig.
3.2.1 LEARNING SURFACE BY VOLUME RENDERING
NeRF Mildenhall et al. 2020) proposed novel render pipeline with combination of spatial neu- ral radiance fields and volume rendering (Kajyya Von Herzen 1984) to synthesize high-quality novel view images. Unlike traditional explicit meshes, the representation of 3D scenes in NeRF
4

---

Published as a conference paper at ICLR 2024
Goometry Feature
Surfice Net
Radiance Net
Tgb
U(x)
(d
Vd
Rendering
Radiance Image
N(n(x), E(x))
(n, (u) Ep(w))
Perspective Projection
mmean
Lov
Splatting
q (u)
Ep(
3D Gaussian Dist.
Caussian Dist.
Polarization Pršor
Figure Illustration of our method.
is decomposed into spatial-dependent density fields and radiance fields depending on both spatial position and viewing direction. Then, the color of an arbitrary pixel with a ray td passed through can be rendered by volume composition along the ray:
K
1-1
C(r)
Σ (r,, d), T; exp (- Σ a5oz), -- -
(2)
where K points fx,)+ the ray sampled. o; and c; approximated volume density and radiance predicted by neural networks with position x and viewing direction as inputs. 6, is the length of sampled interval 4:] a; and T; denote the transmittance and alpha value of points, and by them the final color is alpha composited (Max 1995). The neural network is optimized by the mean square error between the ground truth color C(r) in the image and the rendered color C(r).
Despite realistic novel view images, the geometry of scenes extracted from learned density fields is inaccurate with floating artifacts since the shape is not defined in the density field. NeuS (Wang al.] 2021) defines surfaces the zero-level set of Signed Distance Field (SDF), and density derived from SDF:
@e (x D, (d (x++1)) 0) $, (d:))
[d(x.), f(x:)] f(xx), a; max
c(x,, n. d, f(), (3)
where e and f are the geometry network and radiance network, d(x,) is the signed distance the surface and f; f(x:) is the geometry feature. is defined by SDF with Laplace distribution p, (a:) (1 + -=====- where the variance is a trainable parameter. The volume rendering pro- cess is analogous to NeRF, while the radiance network takes normals n, Vxd(x,) and geometry feature fú as additional inputs.
3.2.2 GAUSSIAN SPLATTING NORMALS
Neural bDF-based 3D reconstruction excels smooth Lambertian objects. With neural BRDF defining the specular reflection between rays and surfaces, smooth surfaces of reflective objects can also be properly learned. However, the low-frequency implicit bias of neural networks (Xu al., 2019) is barrier for both of them to recover delicate geometry behind specular reflection, such as abrupt normal changes in NeRO (L.uu et al., (023). Thus, we propose a 3D Gaussian estimation of distributions of normals splatted to the image plane, making it available for 2D polarization supervision.
additional representation of geometry details. We show how it can be
Instead of separate vectors assigned to each point, the normal within the neighborhood of an arbitrary position X; is assumed as Gaussian:
exp (-16x-m(xx)) "xxx) (x-xxxx)))) ,
G(x(x,) = N (n(xx), E(x))
(2T)
(4)
where n E 3 is the normal, and Σ E k3x3 is the covariance of the Gaussian. Given a ray with discretization {x.|x x + K k= additional M positions within the neighhorhood are super-sampled
5

---

Published as a conference paper at ICLR 2024
to estimate the covariance. In this paper, M is 6 containing X,-1, X and four positions around the ray. Hence, the unbiased estimation of Gaussian can be formulated as:
M Σ (n(x)) - m(xx)) (n(x²) - m(xx)) jm1
((x(xx,) = (n(xx), E(x,)) =N n(xi), M
where n(x() x dd( n(x) E R However, those 3D Gaussians are not accessible in captured 2D images, and volume rendering in Sec. 3.2.1 takes 3D scalar fields into account, making the projection of 3D Gaussians non-trivial. Aftematively, (Zwicker et al., 2001) presents splatting ap- proach treating colors in 3D space as Gaussian kernels and visualizing them on the image plane. We apply analogous transforms and further prove our normal-based 31D Gaussians exactly splatted to 2D Gaussians. Given viewpoint, the transform can be formulated as:
((x|x))p = -N(J JWE( (=) =N ([ np(xi) (1n) ]. Ep (x,) [
1)
(6)
where W Rx33 J R3x3 are viewing transform matrix and normal projection matrix (Chen itt al. 2022), respectively, Derivation of them is shown in the Appendix. It shows that only the first two rows the transformed mean vector and the upper 2 square block of the transformed covariance matrix remain non-zero, splatting 3D Gaussians to 2D Gaussians in the image plane. For simplification, 2D Gaussians are also denoted by (xx = (Nnn(x;), Ex:x)), where np E R Σp (x:)) R2x2, Moreover, the SVD of the covariance matrix Σp VÁVT cir- cumvents the ill-posedness of the covariance matrix and reveals its relation to anisotropic normal distribution. Intuitively, the geometry appears smooth from the imaging perspective, then the corresponding normals of the neighborhood will be projected to similar vectors, resulting in an in- significant deviation of the eigenvalues. Otherwise, the deviation would be significant. Eigenvectors also show the local shape of the position, as shown in Fig. 4 (e). Finally, all 2D Gaussians on the ray passing through the pixel
composited by volume rendering:
((u) - N Σ
( Txmmp((s)
K
Σ TToEEp(x:)) = (u(nn(u), Ep),), k=1
K
where T; and og are in Eq. (u) E RR, Ep) E R2x2, The mean of 3D Gaussians n(x:).
which is splatted to mp (u). represents the overall orientation of x4. And the covariance Σxx:) and splatted Ep)u in the image model the high-frequener details. this way, our representation captures more details than NeuS and other methods based on the neural BRDF parameterized by isotropic normals distribution. Another main strength of those 2D Gaussians is direct supervision by polarization priors, which is introduced in Sec. 3.2.3
3.2.3 OPTIMIZATION WITH REWEIGHTED POLARIZATION PRIORS
(a) Scene (b) AoP
(c) DoP
(d) R. AoP
Visualization of DoP Reweighing
(e) 2D Gaussians in Polarization
Figure 4: Visualization of Reweighted AoP Priors. Red boxes bound specular reflection dominant regions, and the blue boxes bound diffuse ones. (d) is the AoP map reweighted by DoP. Saturation in (c) indicates the degree of anisotropy, and color represents the direction of the singular vector of 2D Gaussians' covariance. A few 2D Gaussians are drawn ellipses for intuition.

---

Published as a conference paper at ICLR 2024
The 2D Gaussian in Sec. 3.2.2 can be directly extracted from AoP priors i in Eq. by:
(p(ulu,) -N(ūp Ep(u,)
M' Σ (v( ((b()) (v(+(()))) - v(bb())))) ==1
=N sv((b(u:)),
M
where 11() = )x the corresponding pixel index of (super-sampled) points on the ray. There- fore, M' = 4 since x,-1 and X++1 are the same ray x;- v(0) represents 2D unitary vector rotated by 0. V 40 + mod π is the azimuth angle normals, derived from the AoP in Eq. I And is scale factor. Similar Sec. 3.2.2] the estimated covariance matrix is decomposed into Σ VÁVT. We define the degree of anisotropy (DoA) of those 2D Gaussians as Ag 2D Gaussians saturated by DoA are visualized in Fig. (e). In this Fig, color is concentrated and coherent along the edges of the scene. It shows DoA is higher in the region with complicated geom- etry and surface changes most dramatically along singular vectors of covariance. Before optimiza- tion, the polarization prior AoP is reweighted by DoP to alleviate the aforementioned observational noise and imbalanced distribution problem in Sec. I The noise of AoP is mainly generated by diffuse reflection because it's always weakly polarized (Kajiyama et al. [023). The DoP in diffuse- dominant regions significantly lower than secular-dominant ones, as shown Fig. 4 (c) and (d). Thus, the reweighted AoP defined & p is proposed as alternative supervision with less noise. Meanwhile, radiance is disentangled with surroundings in specular reflection dominant areas. To adaptively balance radiance and polarization priors, our full loss function during reconstructing is defined as:
L L-a(1 - (oo(1-))ccoloo Pcooor + 8pp(Cmeau + Lcor) + +Yeik + 8Lmask. Ccolor C(u)
C(u) l1,, mmean =|1 (p(p((())) -p(( 112,
Leov
+ V V (u), Leik = K Σ C(IV 1)?,
Ao
Au
where Leolor and maask are the radiance rendering loss and the BCE loss of object masks in 2021 Splatted 2D Gaussians is supervised by (b()) and Ep(u) in Eq. 8 (p(np(u) (u)) (
NeuS (Wang et al.
mod π and vb is the azimuth angle of normals. The supervision of ra- diance and polarization priors are reweighted by the DoP p. Especially, only Anisotropy (ratio of singular values) and eigenvectors are supervised to avoid scaling and numerical issues. the local shape is like plane, normals will change smoothly in all directions, and the Anisotropy approaches 1. If there are some details like edges, normals tend to change abruptly and exhibit directionality, represented by eigenvectors. Coik is
et al. 2020 a, y and 6 are regularization term of the gradient of SDF widely used (Gropp
4 EXPERIMENTS
To evaluate the effectiveness of our method, we tested GNeRP on objects from multiple scenes and compared them with existing state-of-the-art neural 3D reconstruction methods.
PolRef Dataset The methods are evaluated on the PANDORA dataset (Daee al. 2022) and captured scenes by ourselves. The PANDORA includes reflective objects (Owl, vlackvase, and Gnome) with polarization priors. However, their ground truth shapes are unavailable for quantitative evaluation. Moreover, the diversity of materials, geometry, and illumination is not enough for over- all comparisons. Only the geometry of the Gnome scene complicated but less reflective. Only mirror-like ball in the Blackvase reflects surroundings other than highlights. Other common datasets, including Shiny Blender (Verbin et al., 2022), lack polarization priors for our method. To compre- hensively evaluate the performance of 3D reconstruction methods, a new challenging multi-view dataset named PolRef was collected, consisting of objects with reflective and less- textured surfaces captured with various illumination. Radiance images and aligned polarization priors captured in one shot using polarization cameras. To obtain precise and complete ground truth shapes, objects were produced using SLA 3D printers, with accuracy tolerance of mmmmm. Detailed descrip-
7

---

Published as a conference paper at ICLR 2024
tions shown in the Appendix. The dataset will be released to facilitate further research on 3D reconstruction in more challenging scenes the future.
Experimental Settings GNeRP is built upon NeuS (Wang et al. 2021). The geometry network and radiance network in Fig. is the same as that of NeuS. Since the covariance loss Ce in Eq. refines the details of the geometry, it will not be activated during the initial 50K steps. The model trained for 200k iterations and takes about hours on a server with 4 NVIDIA RTX 3090 Ti GPUs for the reconstruction. After optimization, the meshes extracted from learned SDF by Marching Cubes (Lorensen & Cline 1998) with resolution of 5123. The hyper-parameter settings are shown in the Appendix D.3
4.1 COMPARISON WITH STATE-OF-THE-ART METHODS
We conducted the comparison of reconstruction accuracy between our methods and several state-of of- the-art methods, including baseline methods for neural 3D reconstruction (Unisurf (Oechsle al., 2021). VolSDF Yariv et al.) (202T) and NeuS (Wang et al. 2021)). two view-consistency based methods (Neural Warp (Darmon et al. 2022) and Geo-NeuS (Fu al. 2022)), two new methods for reconstruction of reflective objects (NeRO (l.iu et al. 2023) and Ref-NeuS (Ge et al. 2023) and
polarization- based method (PANDORA (Dave et al. 2022)).
Scene
GNcRP(Ours)
PANDORA
NeRO
Ref-NeuS
Figure 5: Visual comparison of our method and state-of-the-art methods.
qualitative comparison between our method and state-of-the-au methods specially designed for reflective objects is shown in Fig. 7 which demonstrates that our method significantly improves the geometry details and accuracy normals. In the Ironman scene, NeRO reconstructed an over- smoothed geometry. Due to the spatial continuity of neural BRDF, it failed to reconstruct the high- frequency armor details with abrupt normal changes. The shape Ref-NeuS more accurate, but the sole scalar SDF is not able to predict the geometry details. The duck scene is more reflective with a combination of highlights and reflection of surroundings. Although Ref-NeuS detected the reflective regions, it was still misled by the environment radiance and reconstructing concave holes. The results PANDORA are over-smoothed in Ironman and disturbed by noise polarization priors in Duck. Additional comparisons of different scenes shown in the Appendix. We conduct
8

---

Published as a conference paper at ICLR 2024
Methods
Ironman
Duck
Cow
Snorlax
Mean
Unisurf*
Occhsle et al.
2021
3,97
10.83
14.43
14.33
10.89
VoISDF Yarıv al. 2021
2.72
5.16
5.95
3.20
4.26
NeuS (Wang et al. 2021
2.28
2.12
3.82
2.11
2.58
Geo-NeuS Luu et al. 2022
4.77
10.12
17.48
10.39
10.82
NeuralWarp
Darmon et al. 2022
12.44
19.78
5.41
20.57
14.55
NeRO*
et
2023
2.29
23.75
2.95
24.30
13.32
Ref-NeuS
Ge et al.
2023
1.88
1.93
3.66
1.99
2.34
PANDORA
(Dave et
2022)
4.61
5.28
7.96
5.73
5.90
GNeRP
1.34
1.63
1.39
1.05
1.35
Table I: Quantitative comparison with state-of-the- methods. The lower is better. indicates the method doesn't use object masks. i refers to the use of polarization priors. The best scores are bold, the second best scores are double underlined, and the third best scores underlined.
quantitative comparisons on the four scenes with ground truth meshes in our dataset. The evaluation metric is Chamfer Distance according to NeuS (Wang et al. (021) and Unisurf (Oechsle et al. 2021). Scores are reported in Table I which shows our method reconstructs more precise meshes in all four scenes. NeRO (Liu et al. 2023) needs environment information to calculate occlusion loss, and Unisurf also learns occupancy from backgrounds. Training them with masks failed directly, we report the scores without masks in Tab. [1] denoted by * Geo-NeuS needs sparse points from (SfM) Scchöbeerger al. (2016) calculate SDF loss and select the pairs based on SfM for the warping process. We did the sparse reconstruction in COLMAP and followed the pairs selection method in NeuralWarp. Full polarimetric acquisition (Stokes vector (So $1 s2], see in the Appendix) is required by PANDORA. We processed the raw polarization capture to follow its data conventions. Sparse reconstruction of reflective scenes was noisy and incomplete, resulting in the worst accuracy by Geo-NeuS and NeuralWarp. Ref-NeuS demonstrated comparable scores on all scenes, our method still outperformed.
4.2 ABLATION STUDY
Scene
NeuS
w/ Lyean
w/ Loov
w/ ReW. Liean
w/ ReW. Leov
Full
Snorlax
2.11
2.03 2.72
3.01 5.54
1.81
2.07 2.29
1.05
Cow
3.82
1.94
1.39
Table 2: Ablation Study. Lieans Leov are in Eq.
To validate the effectiveness of the proposed modules, test the following three settings shown Tab. [ W/L.nea refers to the naive supervision of and azimuth angle of normals SDE Due to the noise, the results worse. W/L refers the polarization supervision with only covariance. The reconstruction is focused on details and results in the worst scores. W/ ReW. Lmean indicates the reweighted losses P) + -h-maan- Similarly, w/ReW. Lrow represents (1 - p) Ccolor + pLcov. The reweighting does improve the efficiency of polarization priors. Finally, the full setting shows the best scores. Additional visualization is shown the Appendix.
5 CONCLUSION
We propose GNeRP to reconstruct the detailed geometry of reflective scenes. In GNeRP, propose a new Gaussian-basee representation of normals and introduce polarization priors to supervise it. We propose DoP reweighing strategy resolve noise issues in polarization priors. We collect new, challenging multi-view dataset with non-Lamen Lambbererta scenes to evaluate existing methods more comprehensively. Experimental results demonstrate the superiority of our method.

---

Published as a conference paper at ICLR 2024
REFERENCES
Marco Ament and Carsten Dachsbacher. Anisotropic ambient volume shading. IEEE Transactions on Visualization and Computer Graphics, 22(1):111-41024, 2015.
Gary A Atkinson and Edwin R Hancock. Recovery of surface orientation from diffuse polarization.
JEEE Transactions on Image Processing, 15(6):1653-164, 2006.
David C Banks. Illumination in diverse codimensions. In SIGGRAPH, pp. 327-334, 1994.
Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin Martin-Brualla, and
Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.
ICCV, 2021.
J. Berkmann and T. Caelli. Computation of surface geometry and segmentation using covariance techniques. IEEE Transactions Pattern Analysis and Machine Intelligence, pp. 1114-1116, Jan 1994.
Christoph Bregler, Aaron Hertzmann, and Henning Biermann. Recovering non-rigid 3d shape from image streams. In CVPR, pp. 690-696, 2000.
Guangcheng Chen, Li He, Yisheng Guan, and Hong Zhang. Perspective phase angle model for polarimetric 3d reconstruction. In ECCV, pp. 398-414, 2022.
Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In CVPR, pp. 1907-1915, 2017.
Yi Ping Chen and Ming Der Yang. Micro-scale manufacture of 3d printing. In Applied Mechanics and Materials, volume 670, pp. 936-941, 2014.
Edward Collett. Field guide to polarization. In SPIE, 2005.
Robert L. Cook and Kenneth Torrance. reflectance model for computer graphics. In SIG- GRAPH, 1981.
Zhaopeng Cui, Jinwei Gu, Boxin Shi, Ping Tan, and Jan Kautz. Polarimetric multi-view stereo. In CVPR, pp. 1558 1558-1567, 1567, 2017.
François Darmon, Bénédicte Bascle, Jean-Clément Devaux, Pascal Monasse, and Mathieu Aubry. Improving neural implicit surfaces geometry with patch warping. In CVPR, pp. 6260-6263 2022.
Akshat Dave, Yongyi Zhao, and Ashok Veeraraghavan. Pandora: Polarization-aide neural decom- position of radiance. arXiv preprint arXiv:2203. 13458, 2022.
James Foster, Shelby E Temple, Martin J How, Ilse M Daly, Camilla R Sharkey, David Wilby, and Nicholas W Roberts. Polarisation vision: overcoming challenges of working with property of light we barely see. The Science of Nature, 105(3): 2018.
Qiancheng Fu, Qingshan Xu, Yew-Soon Ong, and Wenbing Tao. Geo-neus: Geometry-consistent neural implicit surfaces learning for multi-view reconstruction. NIPS, 2022.
Yoshiki Fukao, Ryo Kawahara, Shohei Nobuhara, and Ko Nishino. Polarimetric normal stereo. In CVPR, pp. 682-690, 2021.
Yasutaka Furukawa, Brian Curless, Steven Seitz, and Richard Szeliski. Towards internet-scale multi-view stereo. In CVPR, pp. 1434-1441, 2010.
Yasutaka Furukawa, Carlos Hernández, al. Multi-view stereo: A tutorial. Foundations and Trends@ in Computer Graphics and Vision, 91-2):11144, 2015.
Silvano Galliani, Katrin Lasinger, and Konrad Schindler. Massively parallel multiview stercopsis by surface normal diffusion. In ICCV, pp. 873-881, 2015.
Wenhang Ge, Tao Hu, Haoyu Zhao, Shu Liu, and Ting-Cong Chen. Ref-neus: Ambiguity-reduce reduced neural implicit surface learning for multi-view reconstruction with reflection. arXiv preprint arXiv:2303. 10840, 2023.
10

---

Published as a conference paper at ICLR 2024
Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regu- larization for learning shapes. arXiv preprint arXiv:2002.10099, 2020.
Xiaojie Guo, Xiaochun Cao, and Yi Ma. Robust separation of reflection from multiple images. In
CVPR. pp. 2195-119222 2202,, 2014.
Inseung Hwang, Daniel S. Jeon, Adolfo Muñoz, Diego Gutierrez, Xin Tong, and Min H. Kim. Sparse ellipsometry: Portable acquisition polarimetric svbrdf and shape with unstructured flash photography. TOG, 41(4), 2022.
Achuta Kadambi, Vage Taamazyan, Boxin Shi, and Ramesh Raskar. Polarized 3d: High-quality depth sensing with polarization cues. In ICCV, pp. 3370-3378, 2015.
James T Kajiya and Brian P Von Herzen. Ray tracing volume densities. ACM SIGGRAPH computer graphics, 1853):165-11 1984.
Soma Kajiyama, Taihe Piao, Ryo Kawahara, and Takahiro Okabe. Separating partially-polarized diffuse and specular reflection components under unpolarized light sources. In WACV, pp. 2549- 2558, 2023.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023.
Virginia Klema and Alan Laub. The singular value decomposition: Its computation and some appli- cations. IEEE Transactions on automatic control, 25(2):164-176, 1980.
Yuan Liu, Peng Wang, Cheng Lin, Xiaoxiao Long, Jiepeng Wang, Lingjie Liu, Taku Komura, and Wenping Wang. Nero: Neural geometry and brdf reconstruction of reflective objects from multi- view images. In SIGGRAPH, 2023.
William Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. In Seminal graphics: pioneering efforts that shaped the field, pp. 347-353. 1998.
Nelson Max. Optical models for direct volume rendering. IEEE Transactions Visualization and Computer Graphics, 1())999-108, 1995.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR, pp. 4460-4470, 2019.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. Mitsuba 2: A retargetable forward and inverse renderer. TOG, 38(6): 2019.
Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In ICCV, pp. 5589-5599, 2021.
Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Mare Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In ECCV, pp. 501-518, 2016.
Johannes Lutz Schönberger, Enliang Zheng, Mare Pollefeys, and Jan-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. In ECCV, 2016.
Bruce D Smith and Stan H Ward. On the computation of polarization ellipse parameters. Geophysics, 3966)8677866, 1974.
Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srini- vasan. Ref-nerf: Structured view-dependent appearance for neural radiance fields. CVPR, pp. 5481-5490, 2022.
Peng Wang. Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2 106.10668 10689, 2021.
11

---

Published as a conference paper at ICLR 2024
Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. arXiv preprint arxiv:1901.000 2019.
Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. NIPS, 34:4805 4815, 2021.
Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. PhySG: Inverse rendering with spherical gaussians for physics-basee material editing and relighting. CVPR, 2021.
Jinyu Zhao, Yusuke Monno, and Masatoshi Okutomi. Polarimetric multi-view inverse rendering. In ECCV, pp. 85-102, 2020.
Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. Ewa volume splatting. In VIS, pp. 29-538, 2001.
A METHOD DETAILS
A.1 POLARIZATION ANALYSIS
The primal polarization information captured in one shot is a four-directional polarization image obtained by using four directional linear polarizers at angles 09, 45% 90% and 135". After the de- mosaicing process, the four directional images can be denoted as Io. I45. Ion. and I 1 respectively. For computational efficiency, the Stokes representation is widely used, such as PANDORA Dave et al.. (2022), which is defined as follows:
Sg
Io I9o
S
S1
Iu
Igo
IIR
(10)
$2
L45
1a5
is noted that the fourth component Stokes vector is zero because it represents circular polariza- tion, which is not captured by linear polarizers. The vector also covers radiance information, which can be derived as:
Lad
2 S0.
(11)
However, the Stokes vector is always noisy and redundant for 3D reconstruction. This is because, in addition to the orientation of surfaces, the Stokes vector is affected by environmental variables and the nature of objects. These variables include, but are not limited to, illumination, roughness, and material properties. PANDORA Dave et al. (2022) supervises the full vector for inverse rendering, but it's unnecessary for learning geometry only. Therefore, we refine the Stokes vector into the AoP and DoP cues, as they are most related to the geometry within the Stokes vector. Calculating AoP and DoP from Stokes vector can formulated as follows:
1
(2
(i, 2
arctan
$1 (i,
(p, ER
(12)
(i = (si + s2(1.))2
Su(i,))
where ji., is the AoP at the pixel (i.j), lying in the interval of [0, x]. And p(t,j) the DoP ranging from ( to 1.
A.2 PERSPECTIVE TRANSFORM OF GAUSSIANS
Similar to (wicker et al. 2001). transforms of 3D Gaussians to 2D Gaussians are performed based on the lemma:
Lemma Given 3D affine transform u +x() Mx + 3D Gaussian g(x) NN E) is transformed into:
g' (u) = (z)) MEM")
(13)
It be proven by replacing with Φ- (n)
12

---

Published as a conference paper at ICLR 2024
During the imaging, points in the world space are projected to 2D pixels through the view transform matrix and perspective projection matrix. The view transform converts the world coordinates to the camera ones. It can be formulated as (xx) = Rx + t, where R is a rotation matrix and t is translation vector. For bound normals, the translation will not change their orientation, and thus the corresponding matrix W R. However, the perspective project matrix of normals is quite different from points. For 3D points, the transform is non-linear projecting points to the image plane. On the contrary, the projection of normals proven be linear, which can be formulated as (Chen al. 2022):
U2
0
Us Vy
x (vXn) Ug + U272, Ry + Uy 0)"
0
Jn,
(14)
0
where is the normal vector, is the vector points to the location of the normal vector, i.e. the direction of the ray, and [0, 0, 1]". Since both of the view transform and projection in Eq. [4] are affine, the structure of Gaussian holds. Moreover, the last row of J is all-zero, making only the
upper 2 x 2 square block of JMJT and the first two rows of Jx are non-zero. It explains the last term Eq.
A.3 GAUSSIANS ESTIMATION AND DECOMPOSITION
Given a normal vector of 3D point n(x;) with normal vectors of super-sampled points with the neighborhood (n(x)) M.M : 3D Gaussian M(n(x;). Exx)) can be estimated by Eq Then, the 3D Gaussian is transformed into the camera coordinates N (mnp(x)), (x:))) by Eq. 6 Finally, all
3D Gaussians along the same ray are composited into 2D Gaussian (nn (u). Ep(u)) in the pixel
by volume rendering in Eq. 7 where x, o(u) + f,d(u). Then Singular Value Decomposition
(SVD) (Klema & Laub 1980 is performed get A (u):
(u) V VÁVT(u) (u)
(15)
For 2D polarization priors, 2D Gaussians are estimated by Eq. For the sample x/. the correspond- ing 2D pixel can be located by the ray x + t since d' and the pixel u² is bijective. From the AoP pu)), we can derive the orientation of the projected normal vector sb. It's represented by:
(vb()))) = (u'),, sini")(sinnn(uu) (in()) ((') = (<((rr) +
(16)
mad
where s is the scale factor equal to the magnitude of the projected normal vector. Through Eq. X 2D Gaussians of polarization images can be estimated. Finally the same SVD is performed to get A.
B POLREF DATASET
Since ground truth geometry is inaccessible in the most of existing polarimetric multi-view datasets, we sampled a new dataset named PolRef Dataset to evaluate our method comprehensively. The dataset is split into real captured scenes and synthetic scenes.
Dataset Collection The dataset consists of 8 scenes (Ironman, Snorlax, Duck, Cow, Cat, Vase, Bunny, and Dragon). For real scenes, the capture pipeline illustrated in Fig. (a). Radiance im- ages and aligned polarimetric priors were captured in one shot using polarization cameras (LUCID PHX050S-6 Q and HIKIVISION MV-CH050- 10UP). We captured multiple views around the object. The objects were put on a calibration disk designed for 360 capture to get poses. The data process- ing formulation to extract polarization priors listed in Sec. A. To obtain precise and complete ground truth shapes, 4 objects (Ironman, Snorlax, Duck, and Cow) were produced using SLA 3D printers, with an accuracy tolerance of -0.100mmmm given STL files as ground truth shapes. To enhance the diversity of the dataset, two non-3D printed objects were also included in the data collection pro- cess. Moreover, to increase the diversity and evaluate our method more comprehensively, synthetic data (Bunny and Dragon) is generated using the Mitsuba renderer (Nmier- David et al. 2019). Mit- suba is able to render polarization priors (Stokes vectors) from meshes with pre-defined attributes of scenes, illustrated in Fig. 6 (b). In addition polarization priors, ground truth normal maps
13

---

Published as a conference paper at ICLR 2024
accessible through rendering, making the evaluation of normals accuracy available. The dataset will be released to facilitate further research on 3D reconstruction in more challenging scenes in the future.
990
Polarization Camera
I
50
I 135'
Real Scene
So
Mitsuba Renderer
10
+
S
Training Data
So
5o
$1
72
S2
S2
mask
p (DoP) Φ (AoP)
Stokes
Stokes Normals
(a) Real Dataset Capture
(b) Syn. Dataset
Figure 6: Pipeline of PolRef Dataset.
Evaluation Protocol For real and synthetic datasets, the accuracy of reconstructed meshes is mea- sured by Chamfer Distance (CD), which be formulated as:
1 Σ lz Nearest Pi)l, CD = 2n Σ b Nearest (2,, P2)1 + 2m 5=1
(17)
where is the vertex, n1, are the number of vertices in meshes, and Nearest P) argmin || 1 x'll is nearest neighboring function. Mean Angular Error (MAE) of normals introduced to the evaluation of synthetic data since ground truth normals are available, which can be formulated as:
MAE (ú, n) = NM arecos
(18)
where nf is the ground truth normal vector pixel i the view j. and N, M are the number of views and pixels, respectively. Compared to CD. MAE is more sensitive details of shapes, where normals change abruptly.
C ADDITIONAL COMPARISONS WITH EXISTING METHODS
C.1 VISUAL COMPARISON ON PANDORA DATASET
Visual comparison is shown in Fig. In the Owl scene, all baseline methods fail to reconstruct feathers, of which surfaces are mistakenly concave. Because baseline methods recognize dark radi- ance as the cue of deeper surface points. In the Black Vase and Cat scenes, Unisurf and NeuS suffer from severe shape-radiance ambiguity as they cannot disentangle specular reflection with surface color and wrongly reconstruct the shapes of reflected scenes. VoISDF distinguishes reflection and color more clearly than them but still worse than our method glossy areas, as shown in the Cat scene. In the Vase scene, even though we have tried as many combinations of hyper-parameters as we can, Unisurf still fails to reconstruct rough geometry. VoISDF isn 't able to find correct sur- face points from dark radiance, the hole of the vase wrongly recognized The reflection area is also distorted in the shape reconstructed by NeuS. Collectively, our method outperforms baseline methods by large margin in all four scenes.
convex surface.
14

---

Published as a conference paper at ICLR 2024
Scene
Unisurf
VoISDF
NeuS
Ours
Figure 7: Visual comparison of our method and baseline methods. The Owl and Black Vase scenes are PANDORA Dave et al. (2022) dataset. The Cat scene sampled with indoor illumination and the Vase with single light source.
C.2 COMPARISON ON SYNTHETIC POLREF DATASET
We also evaluated our method on the generated dataset. In contrast to real datasets, ground truth normal maps are accessible in generated scenes, we introduced the Mean Angular Error (MAE) of normals as an additional evaluation metric. Compared to Chamfer Distance (CD), MAE can reflect the accuracy of reconstructed details better. Results demonstrate that our method still outperforms existing methods for reconstructing reflective objects. Tab. 3 shows our method outperforms both
GNeRP (Ours)
Ref-NeuS
NeRO
PANDORA
Scene
CD &
MAE
CD 4
MAE 4
CD 4
MAE 1
CD 4
MAE!
Bunny Dragon
0.72 0.59
0.78
1.09
1.03 1.23
1.41 2.15
2.55 3.47
3.77
18.15
1.03
0.82
5.48
10.98
Table 3: Comparison on PolReF Synthetic Dataset.
in mesh distance (CD) and accuracy of normals (MAE) quantitatively. Visualization of normals is shown in Fig. 8 Similar to other experiments, the normals reconstructed by existing methods always be over-smoothed, resulting in lack of details. For instance, the scale details of the dragon scene are omitted in other methods, particularly in the NeRO method. PANDORA fails in both scenes because estimating Stokes vectors with these complex geometries is challenging.
C.3 VISUAL COMPARISON ON DIFFUSE DATASET
To validate the generalization ability of our method, evaluation ducted the Camera scene captured in PMVIR (Zhao et al., 2020). Visual comparison with baseline method NeuS is shown in Fig. 9 Quantitative comparisons are unavailable since the ground truth mesh isn't collected in the dataset.
objects con-
15

---

Published as a conference paper at ICLR 2024
Radiance
GT Normals
GNeRP(Ours)
Ref-NeuS
NeRO
PANDORA
Figure 8: Visual comparison of normals PolRef Synthetic Dataset.
Reference
GNeRP(Ours)
NeuS
Figure 9: Visual comparison of a diffuse object with the baseline method.
As is shown in the figure, our method reconstructs more details of small structures such as buttons, knobs, and slots of the camera. proves our method can handle diffuse objects concurrently.
C.4 VISUAL COMPARISON NORMALS WITH REF-NERF
ReR-NeRF Verbin et al. (022) is state-of-the-ar method for reflective object rendering. However, its mesh is inaccessible, and the normals are noisy due to the Integrated Position Encoding (IPE), we did not involve it in the overall comparison. Instead, we show example of the Cat scene to show the incomparable normals Fig 10
D ADDITIONAL ABLATION ANALYSIS
D.1 VISUALIZATION OF EFFECTIVENESS OF DoP REWEIGHTING
DoP efficiently alleviates the noise in polarization prior. One the most significant scenes is Gnome
in the PANDORA dataset, is shown in Fig. 11
16

---

Published as a conference paper at ICLR 2024
(a) Ours
(b) Ref-NeRF
Figure 10: Normals comparison between Ref-NeRF and ours.
ReW
Noisy Polarization
win DaP Reweighinng
Neu8
Figure 11: Ablation study of DoP reweighting. Top-Down: w/ DoP ReW., w/o DoP ReW., NeuS
D.2 VISUALIZATION OF EFFECTIVENESS OF COVARIANCE
Amean is our key design regarding to Gaussians of normals. In our experiments, significantly improves the reconstruction of abrupt normal changes at the mouth of the Duck. It's shown in Fig.
12
(a) w/ ReW. Cmean (b) w/ ReW. Cmeany Leov
Figure 12: Ablation study on covariance loss.
D.3 ABLATION STUDY OF
In our experiments, hyperparameters include weights loss functions (a, B, 7. and 8) and the number of super-sampled points M. 7 and 5 are fixed 0.1 to follow previous methods. & and 3
17

---

Published as a conference paper at ICLR 2024
fixed at 0.1. However, is set to either 0.1 or depending the overall ratio of reflection regions. Increased reflection regions indicate a decrease in the reliability of radiance cues, and therefore, the value of should be decreased.
D.3.1 WEIGIIT OF RADIANCE LOSS
We test different weights of the radiance loss function on the Bunny scene. Results are shown in Fig.
Fig.
(d) demonstrates that if a is too low, the method cannot extract efficient radiance
(a)AoP
(b)o=1.0
(c)a=0.1 CD=0.72, MAE=0.78
(d)a=0.01 CD=2.49, MAE=3.07
CD=0.94, MAE=0.66
Figure 13: Ablation study of the weight a. CD refers to Chamfer Distance, and MAE refers to Mean Angular Error.
cues, and then the regions with less polarization prior are incorrectly interpreted holes.
D.3.2 NUMBER OF SUPER-SAMPLING SAMPLING WITHIN NEIGHBORHOOD
In the paper, M 6 additional points within the neighborhood of x; are sampled for simplification. To validate the robustness of this choice, we sampled double points around the ray. Results are shown in Tab. 4 shows that increasing M will not enhance the accuracy significantly.
Scene M
M 10
Duck
1.63
1.62
Table 4: Ablation study on M.
E LIMITATION
We observe that major limitation of our method is that the reconstruction relies on polarimetric imaging. As illustrated in Fig. 6 radiance images and polarization priors are generated through polarimetric imaging. Moreover, due hardware limitations, the imaging quality of polarimetric cameras is generally slightly lower compared to regular RGB cameras, and it is more prone to noise shaded scenes. Furthermore, existing acceleration techniques for NeRF, such as Instant-NGP and Voxel, have not been incorporated into this method. As result, the training and inference speed of the model is lower compared to existing NGP-based methods.
18

---

Published as a conference paper at ICLR 2024
E.1
FAILURE CASE
Figure 14: Visualization of distorted mesh resulted by shading.
We present a failure case in Fig. 14 The region bounded by the boxes is shaded by self-ocelusion, making the extraction of polarimetric cues noisy shown in the middle sub-figure. Moreover, the radiance is too dim to learn geometry properly. Consequently, a wrong dent exists the region of the reconstructed mesh.
19


Reviews:
[{"review_id": "r1PAZxFWs3", "reviewer": "Reviewer_ANa9", "summary": "The article introduces a 3D reconstruction technique based on Nerf, which focuses on capturing 3D geometries of glossy objects. Building on recent progress in utilizing signed distance functions (SDF), as seen in approaches like NeuS or Ref-Neus, this method also incorporates it. However, unlike the scalar SDF in these approaches, this proposed method additionally integrates polarization cues to constrain the surface normal. These added constraints are formulated using a Gaussian splatting approach. As a result, the proposed method demonstrates notably improved accuracy compared to state-of-the-art (SOTA) approaches.", "questions": "- For the parameter selection of alpha and beta, the authors did not specify the use of identical parameters across all experiments. The exact values for these parameters were not explicitly mentioned in the paper.\n\n- In Section 2.4, the authors stated that \"the learned 3D Gaussians imply the anisotropic normals distribution of 3D\npoints and capture more details of surface geometry.\" Could the authors elaborate on the description or provide some references?\nDoes it mean the learned 3D Gaussian is anisotropic? And the anisotropicity results in more details?\n\n- The ablation study could include an examination of the impact of alpha and beta. Additionally, the authors did not provide a clear rationale for why they included only two objects in Table 2 while incorporating four objects in Table 1. The ablation study of the other two objects is not supporting?\n\n- In the last paragraph of Section 3.2.3, the authors stated \"Intuitively, the first term measures the complexity of the geometry, while the second term reveals the specific geometric shape.\" Could the authors elaborate on this description?\n\n\nAdditional comments\n+ Typographical errors:\n\t- alpha in the first paragraph of page 5\n\t- common scenes in the first paragraph of section 2.3\n\t- Fig. 4(g) in the first paragraph of page 7. In fact, there is no Fig.4 (g).\n\t- In Fig. 4 caption: \"blue boxes bound diffuse ones\".\n\t\n\n+ \"exact 3D shapes\" in the abstract should be replaced by another term such as \"accurate\".", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The primary advantage of the proposed method lies in its utilization of polarization cues to improve the estimated geometry. This strategy is commonly employed to enhance accuracy in conventional 3D reconstruction techniques.\n- The additional constraint is devised using a Gaussian splatting technique. Specifically, the original 3D Gaussian on the 3D surface normal is transformed into a 2D Gaussian in the image plane and is constrained by polarization cue.\n- The polarization reweighting strategy can guide the proposed method to employ more polarization cues on the area with strong polarization information and employ more radiance on low polarization information.\n- The proposed method excels in reconstructing 3D geometry with high precision and capturing intricate details.", "weaknesses": "- The proposed method only works for glossy objects that exhibit clear polarization information.\n- The proposed method comes with many hyperparameters which are hard for users to set in practice. Parameters like alpha and beta typically vary between 0.1 and 1, contingent on the intricacy of the geometry. Regrettably, the authors did not carry out any experiments to assess the optimal selection of these parameters, potentially hindering the method's practical implementation and performance.\n- Only the results of a few objects are shown in the experiments.\n- There are typographical errors and improper terms, which are described in the questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cjyGuOy6He", "reviewer": "Reviewer_Kf6J", "summary": "Neural SDF-based 3D reconstruction excels at smooth Lambertian objects. The paper proposes a 3D Gaussian-based representation of normals in SDF fields, and splat it to 2D Gaussians in the image plane. It shows that the 2D Gaussian can be directly extracted from Angle of Polarization (AoP). This paper proposes to use the proposed 2D Gassians represenation as the additional constraints for 3D normal recovery. Moreover, it proposes to use the Degree of Polarization (DoP) which indicates the complexity of the surface to reweight  AoPalleviate the noise issue of polarization priors.  Their experimental results on PANDORA dataset domstrate the effiectiveness of the proposed method.", "questions": "Please explain in detial about how to get $\\hat{\\Sigma}, \\hat{\\Lambda}$ and $\\tilde{\\Sigma} \\tilde{\\Lambda}$ in EQ7~9. Pleae explain the experimental setting to get the polarized data. More details are needed for the new multi-view polarized dataset, and how it collected.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper proposed the 3D Gaussian representation of surface normal for the volume rendering proposed in NeRF(2020) in EQ4., and furture entend it to 2D Gaussian by using the splatting approach (Zwicker et al). The research demonstrates that the 2D Gaussian representation can be efficiently computed using the Angle of Projection (AoP). Additionally, it is revealed that the Degree of Polarization (DoP) is strongly correlated with the surface complexity. Consequently, the authors propose a novel regularization technique for NeRF, involving the reweighting of constraints on the 2D Gaussian representation of surface normals. The experimental results clearly indicate that the proposed method excels in handling high-frequency BRDF surfaces.", "weaknesses": "The paper presents experimental results only on the PANDORA dataset. It is essential to include discussions regarding failure cases and the limitations of the proposed method. The authors highlight \"the creation of a new and challenging multi-view dataset\" as a significant contribution. Nevertheless, there is a notable lack of information and discussion about this new dataset in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gSgRpcWB4K", "reviewer": "Reviewer_1dU4", "summary": "This paper proposes a neural volume-based method that utilizes Gaussian-based normal representation and polarization information to improve multi-view 3D reconstruction of highly specular objects. Its volume representation and rendering framework are from NeuS. The major novelty is to model the normal direction of sampled 3D points along the ray with 3D Gaussian distributions, which then can be projected to 2D image plane using Gaussian splatting. A new loss is proposed by comparing the projected normal distribution with the normal distribution computed from polarization information. Experiments on datasets containing highly specular objects shows that the proposed method can achieve much more accurate geometry reconstruction.", "questions": "My major questions are the first two points listed in the weakness section.\n1. Can we explain in more details how to compute 2D normal distribution from a polarized image? \n2. Can we further verify if 3D Gaussian distribution is necessary? If we only compute the normal following standard volume ray tracing and then use reweighted $L_{\\text{mean}}$ loss, will the results be similar or worse?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. High-quality geometry reconstruction for highly specular objects.\nMulti-view reconstruction of highly specular surfaces is known to be a difficult problem. This paper shows compelling reconstruction quality on several challenging real-world objects. \n\n2. Convincing improvements compared to prior method on highly specular objects.\nBoth quantitative and qualitative comparisons show that the proposed method achieve higher geometry reconstruction accuracy compared to prior state-of-the-arts, including a method that also utilizes polarization information and a method designed for specular object. \n\n3. A polarization-based geometry reconstruction method that does not require complicated setups. \nThe proposed method does not make strong assumptions on the material properties and capturing environment, which makes it practical for many applications.", "weaknesses": "1. I have some questions about Sec. 3.2.3, which I would like to understand better. They are not necessarily the weaknesses of the paper.\n\n(a) How to decide the scale factor $s$? Or do we assume the mean loss should be scale invariant and we only care its orientation?\n\n(b) Do we compute $\\mu_i^{j}$ by projecting every $x_{i}^{j}$ along the ray to 2D image, i.e. $M'$ equal to $6M$?\n\n(c) I can see that 2D convariance matrices computed from polarization and splatting can both indicate if geometry is changing rapidly in the neighborhood. But it is a bit difficult for me to understand why they should be equal. A side by side visualization of both 2D Gaussian distributions will be useful. In addition, will noisy AoP of diffuse surface cause the 2D covariance matrix from polarization to have very large eigen values?\n\n(d) Typos: there is an extra $)$ in $L_{\\text{mean}}$ and $L_{\\text{conv}}$. \n\n2. Further ablation studies can be useful to verify the effectiveness of Gaussian normal representation. \nSince Gaussian normal representation is the major novelty, it might be good to further verify its effectiveness with more ablation studies. One simple baseline is to render normal through standard volume ray tracing and then use reweighted $L_{\\mean}$ loss for training. That may give us a clear answer if we need to sample extra points to compute a Gaussian distribution. \n\n3. Results on less specular objects may be interesting.\nI notice that all real objects shown in the paper are quite glossy. Do we have results on less glossy objects? As shown in the paper, the AoP of diffuse surfaces can be noisy but hopefully the DoP weight can fix this issue. \n\n4. Missing reference.\nOne related paper is cite is: \"Sparse Ellipsometry: Portable Acquisition of Polarimetric SVBRDF and Shape with Unstructured Flash Photography, Hwang et al.\"\n\nVery minor comments:\nFigure 2: the polarized direction may use a different color from that of the x-axis. Zenith angle $\\theta$ is mentioned by never discussed in the paper. It may be more useful to label out angle $\\varphi$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "32cjvDyHJb", "reviewer": "Reviewer_BtpD", "summary": "This paper proposes a method called GNERP for reconstructing the detailed geometry of reflective objects from multi-view images. The key idea is to extend the geometry representation from scalar Signed Distance Fields (SDFs) to Gaussian fields of normals supervised by polarization priors. The paper introduces a pipeline for learning the surface by volume rendering, and presents a DoP reweighing strategy to alleviate noise and imbalance distribution problems of polarization priors. Additionally, a new multi-view dataset called PolRef, consisting of objects with reflective and less-textured surfaces, is collected to evaluate the performance of3D reconstruction methods. Experimental results show that GNERP improves the geometry details and accuracy of geometry and normals of reflective surfaces compared to existing state-of-the-art methods.", "questions": "Please see weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper integrates recent advances in Gaussian Splatting to model the polarization priors for reflective surface recovery, which is innovative and aligns with the current research trends.\n\n- The proposed reweighted polarization priors appear to serve effectively as supervision.", "weaknesses": "- The explanation of the proposed method is unclear. What is the input? Is a polarization camera required for reconstruction? If not, is the polarization prior regressed during optimization?\n\n- The proposed approach neglects the effect of self-occlusions, which can be severe for complicated objects. The paper should at least include a qualitative discussion about the effects of self-occlusions.\n\n- The property of polarization is closely related to materials. The proposed approach relies on the intermediate output of a coordinate network and hence seems to handle only objects with a uniform material, as shown in Fig. 5.\n\n- The experiments are conducted only on relatively simple objects. It would be beneficial to show results on more complicated objects to better understand its performance.\n\n- It would be great if the authors could provide an illustration of how the polarization priors intuitively improve the performance of NeuS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)