SAMURAI: Shape And Material from Unconstrained Real-world Arbitrary Image collections
Authors: Mark Boss
Andreas Engelhardt
Abhishek Kar
Yuanzhen Li
Deqing Sun
Jonathan T. Barron
Hendrik Lensch
Varun Jampani
Venue: NeurIPS 2022 Accept (2022)
Keywords: inverse rendering
reflectance decomposition
illumination estimation
pose estimation
neural fields

Abstract: Inverse rendering of an object under entirely unknown capture conditions is a fundamental challenge in computer vision and graphics. Neural approaches such as NeRF have achieved photorealistic results on novel view synthesis, but they require known camera poses. Solving this problem with unknown camera poses is highly challenging as it requires joint optimization over shape, radiance, and pose. This problem is exacerbated when the input images are captured in the wild with varying backgrounds and illuminations. Standard pose estimation techniques fail in such image collections in the wild due to very few estimated correspondences across images. Furthermore, NeRF cannot relight a scene under any illumination, as it operates on radiance (the product of reflectance and illumination). We propose a joint optimization framework to estimate the shape,  BRDF, and per-image camera pose and illumination. Our method works on in-the-wild online image collections of an object and produces relightable 3D assets for several use-cases such as AR/VR. To our knowledge, our method is the first to tackle this severely unconstrained task with minimal user interaction.

SAMURAI: Shape And Material from Unconstrained Real-world Arbitrary Image collections
Mark Bosst University of Tübingen
Andreas Engelhardt' University of Tübingen
Abhishek Kar Google
Yuanzhen Li Google
Deqing Sun Google
Jonathan T. Barron
Hendrik P. A. Lensch University Tübingen
Varun Jampani Google
Google
Abstract
Inverse rendering an object under entirely unknown capture conditions a fun- damental challenge in computer vision and graphics. Neural approaches such as NeRF have achieved photorealistic results on novel view synthesis, but they re- quire known camera poses. Solving this problem with unknown camera poses is highly challenging as requires joint optimization over shape, radiance, and pose. This problem is exacerbated when the input images captured in the wild with varying backgrounds and illuminations. Standard pose estimation tech- niques fail in such image collections in the wild due very few estimated cor- respondences across images. Furthermore, NeRF cannot relight a scene under any illumination, it operates radiance (the product of reflectance and il- lumination). We propose a joint optimization framework to estimate the shape, BRDF, and per-image camera pose and illumination. Our method works in- the-wild online image collections of an object and produces relightable 3D assets for several use-cases such AR/VR. To knowledge, to tackle this severely unconstrained task with minimal user interaction. Project page: https::/markboo wtt.t//sssseeeppbiiaaiiioooooooi
method is the first
samurai/
1 Introduction
Capturing high-quality 3D shapes and materials of real-world objects is essential for many graphics applications in AR, VR, games, movies, etc. Using active multi-view object capture setups can pro- vide high-quality 3D assets 6.43) but cannot scale large-scale set objects that are present in the world. By contrast, image collections provided by image search results product review images exist for nearly every object. this work, we propose a -agegory-agnostic technique to estimate the 3D shape and material properties of objects from such Internet image collections. Estimating 3D shapes and materials from Internet object image collections poses several challenges as the images highly unconstrained with varying backgrounds, illuminations, and camera intrinsics. Fig. (left) shows a sample image collection of an object which forms the input to our technique.
Concretely, we estimate the 3D shape and BRDF material properties 16] while also estimating per- image illumination, camera poses, and intrinsics. Several contemporary works on shape and material estimation [6] 11 52 assume constant camera intrinsics, near- seerfet segmentation masks as well as almost-correct camera poses given by COLMAP 49 50]. However, is tedious to annotate object masks in the input images manually. We also observe that COLMAP often fails due to insufficient correspondences real-world image collections with highly varying illuminations and backgrounds even when constrain the correspondences to lie within object masks. Instead,
Work done during Student Researcher position Google.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Camera Poses
Shape + BRDF
AR
Game Insertion
Neural
Field
Per-Image Illumination
Material Editing
In-the-wild Images
Optimization Targets
Applications
Figure I: Sample SAMURAI outputs and applications. Sample input collection and the out- puts on challenging real-wor wolld unconstrained image collection. We extract meshes with material properties from the learned volumes enabling several applications in AR/VR, material editing etc.
rough quadrant- quadrant-based pose initialization, e.g. (Front, Above, Right), (Front, Below, Left) etc., as in NeRS [61], which usually takes only a few minutes annotation time per image collection. We base our technique the recent Neural-PIL [12] method that proposes to learn illumination pri- ors along with novel pre-integration illumination network for estimating a neural volume with 31) shape, BRDF, and per-image illumination. Neural-PIL. 12 assumes perfect camera poses and the same camera intrinsics across images. Given that traditional camera pose estimations like COLMAP may fail in-the-will wiid images, propose SAMURAI to jointly optimize camera poses well as intrinsics with a carefully designed optimization protocol (Fig. [T. Furthermore, Neural-PIL requires perfect object masks, whereas leverage automatically estimated object masks and deal with noisy masks using a posterior scaling loss. Some key distinguishing features of SAMURAI include:
Flexible camera parametrization for varying distances. Standard techniques such as NeRF 40 assume fixed near/far clipping planes with equidistant cameras to the object. In contrast, we define the neural volume in global coordinates and propose learn clipping planes per image.
Camera multiplex optimization. Optimizing a single camera per image is prone getting stuck in local minima. We propose using multiplex camera where optimize several camera poses per image and then phase out the incorrect poses throughout the optimization. Although camera multiplexes are previously used in mesh optimization [21], optimizing camera multiplex with neural volumes is challenging due to inefficient ray-based neural volume rendering.
Posterior scaling of inpul images. As different input images have different noise characteristics (e.g., noisy masks), some images would be more useful for the optimization. We propose to posterior scaling input images that weighs the influence of different images on the optimization.
Mesh extraction. We extract explicit meshes with BRDF texture from the learned neural volume making the resulting 3D models readily usable in existing graphics engines.
We observe that existing related datasets such as NeRD-dataset [II] do not capture the variations present in image collections in the wild. For instance, NeRD-dataset images have non-varying background making it casier for COLMAP to work. In addition, the illumination variations are more drastic in internet images captured by different people/cameras and different times. To evaluate the practical in-the- whhd setting, we collected image collections with 8 objects in which each image is captured under unique background and illumination conditions. In addition, also vary the cameras used for capturing the images. Experiments on our new image collections and existing datasets demonstrate better view synthesis and relighting results with SAMURAI compared to existing works. In addition, explicit mesh extraction allows for seamless use of learned 3D assets in graphics applications such as object insertion in AR or games and material editing etc. Fig. (right) shows some sample application results with 3D assets estimated using SAMURAL.
2
Related works
Neural fields encode spatial information in the ML.P network weights, and we can retrieve the
information by simply querying the coordinates [5 39 45 53 With these MI.Ps, we can store

---

alpha or density values and then explicitly render the volume using ray marching [36]. Recent works such as NeRF 40 leverage this neural volume rendering to achieve photo-realistic view synthesis results with view-dependent appearance variations. Rapid research in neural fields followed, which
alternated the surface representations 44
55
provided general improvements to the method [4]
[44
reduced the long training times [1 35
48
56) and inference times [1 22 26
35
41
60)
enabled extraction of 3D geometry and materials
II
42
611.
added generalization capabilities 13
(56 65] or enabled relighting of scenes 55 22
63 64]. However, most methods rely on COLMAP poses, which can fail in complex settings, such as varying illumination and locations.
27
37
52
Joint camera and shape estimation is highly ambiguous task. An accurate shape reconstruc- tion is only possible with accurate poses and vice versa. Often techniques rely on correspondences across images to estimate camera poses 44 [50]. Recently, several methods combined camera cal- ibration with a joint neural volume training. Jeong ef al. [24] (SCNeRFF rely on correspondences, and BARF R4 proposes coarse to fine optimization using a varying number of Fourier frequencies and requires rough camera poses and NeRF-- [57 requires training the neural volume twice while
keeping the previous camera parameter optimization. GNeRF (38 proposes to use discriminator on randomly sampled views to learn a pose estimation network on synthesized views jointly. Over time the pose estimation network can estimate the real camera poses, which can then be used for the full neural volume training. A concurrent work LASSI [58] optimizes articulated shapes and camera poses for animal image collections via part discovery, but LASSIE do not estimate illumination or material properties. NeRS [6]]] deforms a sphere to a specific shape using coordinate- based MLPs and converts the deformation field to mesh; while also optimizing camera poses and single illumi- nation. Compared to previous work, our method does not rely on correspondences (us. SCNeRF), which might be hard to obtain in varying illuminations, can have extremely coarse poses due the camera multiplexing (vs. BARF). works in multiple Illumination (vs. all prior art), does not require training twice (us. NeRF--) or GAN-style training (vs. GNeRF).
BRDF and illumination estimation is a challenging ambiguous research problem. One needs con- trolled laboratory capture setups for high-quality BRDF capture 3 9 28/30 Casual estimation
enables on-site material acquisition with simple cameras and a co-located camera flash. These tech- niques often constrain the problem to planar surfaces with either a single shot [[]] 17
23 32 47),
few-shot [ or multi-shot 22. 9 [0) captures. This casual capture setup can also be extended to joint BRDF and shape reconstruction (5/7 TO
entire scenes B3 51. Most of these methods require a known active illumination like co-loceted flash. Recovering a BRDF under unknown passive illumination is significantly more challenging and ambiguous as it requires disentangling the BRDF from the illumination. Often the specular parameter is constrained to be non-spatially varying or omitted [31 59 633. Recently, neural field-based decomposition achieved decomposition of scenes under varying illumination [2 or fixed illumination [63, but require known, near-perfec camera poses. This can fail on challenging datasets, and our method enables the decomposition of these in-the tn-the-wild datasets.
25
43
47
3 Method
Problem setup. The input is a collection of q object images C E x3; (1....,q] captured with different backgrounds, cameras and illuminations; and can also have varying resolutions. We denote the value of a specific pixel as C In addition, we roughly annotate camera pose quadrants with 3 simple binary questions: Left vs. Right, Above vs. Below, and Front Back. We automat- ically estimate foreground segmentation masks M, E {0, 1]sx1 using U2-Net (-66., which can be imperfect. Given these, we jointly optimize 3D neural volume with shape and BRDF material information along with per-image illumination, camera poses, and intrinsics. This practical capture setup allows the conversion of most 2D image collections into a 3D representation with little manual work. The rough pose quadrant annotation takes about few (3-5) minutes for a typical 80 image collection. At each point x E R in the 3D neural volume V. we estimate the BRDF parameters for the Cook-Torrance model [16] b E R (basecolor b. E R metallic b,, E RR,, roughness b, E R), unit-length surface normal n E R and volume density a E R. We also estimate the latent per-image illumination vectors 2 E IR 12 E (1....,4) .4) used in Neural-PII. [2]. We also estimate per-image camera poses and intrinsies, which we represent using a 'look-at' parameterization that we explain later. Next, we provide brief overview of prerequisites: NeRF 40) and Neural-PII. [12
3

---

Optimizable
Prameters
Illumination zj
Annealed Fourier
Reflectance Field
Intrinsics
Pi-- RleBBadleerr
MLP
BRDFs b,
Color &
Extrinsies ry. ug. tz
Densities 01
Direction
Positions ate
Figure 2: Overview, We jointly optimize the intrinsic (f,) and extrinsic camera (r Ug tÿ) param- eters alongside the shape (o) and BRDF (b) in Reflectance Field and per-image illumination (z,) ). The shape encoded in the density a and also used to integrate all BRDFs along a ray in direction d. The composed BRDF is then rendered using Neural-PIL. 12 in deferred rendering-style.
models neural volume for novel view synthesis with two Bulti-Layeer-Pereptronn (MLP). The MLPs take 3D location E IR3 and view direction d R3 as input and outputs a view-dependent output color c E IR3 and volume density a E RR. These output colors for target view computed by casting camera ray r(t) o id into the volume, with ray origin o E IRS and view direction d. The final color is then approximated via numerical quadrature the integral:
Brief overview NeRF 44). NeRF 40
ē(r) = J (()((()) di with T(t) = exp(- Sr (1) dí). using the near and far bounds of the ray t, and y respectively [40] The first MLP is trained to learn a coarse representation by sampling the volume in fixed sampling pattern along each ray. A finer sampling pattern is created using the coarse density distribution in the coarse MLP, placing more samples on high-density areas. The second MLP is then trained using both fine and coarse sampling.
Brief overview of Neural-PIL [12]. Similar to NeRF [40], Neural-PIL [12 also uses two MLPs to learn a neural volume. The first MLP is similar to NeRF with an additional GLO (generative latent optimization) embedding that models the changes in appearances (due to different illumi- nations) across images. In the second MLP, breaking with NeRF, Neural-PIL. predicts not only a view-dependent output color but also BRDF parameters each 3D location. The second MLP takes 3D location as input and outputs volume density and BRDF parameters (diffuse, specular, rough- ness, and normals). Unlike NeRF, this second MLP does not take the view direction input but can model view-dependent and relighting effects in the rendering due to explicit BRDF decomposi- tion. A key distinguishing factor of Neural-PII. is the of latent illumination embeddings and specialized illumination pre-integration (PIL. network for fast rendering, which we refer to as 'PII. rendering'. More concretely, Neural-PIL optimizes per-image illumination embedding z to model image-specific illumination. The rendered output color is equivalent to NeRF's output c, but due the explicit BRDF decomposition and illumination modeling, enables relighting.
3.1 SAMURAI Joint optimization of shape, BRDF, cameras, and illuminations
The main limitations of Neural-PIL include the assumption of near-perfect camera poses and the availability of perfect object segmentation masks. We observe that COLMAP either produces in- correct poses or completely fails due to an insufficient number of correspondences across images when the backgrounds and illuminations are highly varying across the image collection. In addition, camera intrinsics could vary across image collections. and the automatically estimated object masks could also be noisy. We propose a technique (we refer as 'SAMURAI') for joint optimization of 3D shape, BRDF, per-image camera parameters, and illuminations for given in in-the-wild image col- lection. This is highly under-constrained and challenging optimization problem when only image collections and rough camera pose quadrants are given as input. We address this highly challenging problem with carefully designed camera parameterization and optimization schemes.
Architecture overview. high-level overview of SAMURAI architecture shown Fig. which mostly follows the architecture of Neural-PIL [22 and NeRD [II]. However, we do not use coarse network for efficiency reasons and only use the fine or decomposition network with an MLP with layers of 128 features. Overall we sample 128 points along the ray in fixed steps. The latent embed- ding from the main network is interpreted with a linear layer outputting feature for the density and an MLP with a hidden layer for the view and appearance-conditione radiance. We also leverage

---

Ray
Look-at
Far Bound
Arnealing
Ag ConrseZFine
Aa Advanced
BRDF
Origin
Lox Sealer
Near
0.6
Focal Length
Bound
Volume
Camera Multiplex
Bound e
Figure 3: Ray parametrization and Camera Multiplex. (Left) Our ray bounds are defined by a world space sphere. The distance from the origin to the near and far points of the sphere define the sampling range. By defining a glob- ally consistent sampling range cameras be placed in arbitrary distances. (Right) When optimizing multiple camera hypotheses, only the best camera should optimize the shape and appearance, here visualized in deep green. Cameras which are not aligned that well are visualized in off-white and cannot influence the reflectance field. When the non-aligned camera poses improve during the training. they may become relevant for the network optimization.
Steps
-105
Figure 4: Optimization scheme. Dur- ing optimization, we use a gradual change in the loss weights using three À variables. Additionally, we perform a Fourier frequency annealing in the first phase the training and delay the train- ing of the focal length for later stages in the training. The BRDF estimation is mainly regulated by the A parameter.
BRDF decoder similar to NeRD [ which first compresses the feature output of the main network to features and compresses the embedding further to the BRDF (base color, metallic, roughness). We encourage sparsity in the embedding space using: Lee Sparsity = -1/N Σ lesl. an Lo-Loss on the BRDF embedding and smoothness loss Le Smooth = 1/2 Σx [f e.) f(0: e, + e)), where N denotes the number of random rays. (0) the BRDF decoder with the weights and E is normal distributed Gaussian noise with standard deviation of 0.01. Similar to NeRD, we also predict regular direction-dependent radiance ē in the early stages of the training. This is mainly used for training stabilization in the early stages. As this direct color prediction is only used in the early stages, we omitted it for simplicity in Fig. [2] Inspired by Tancik et al. [53], we add a Gaussian distributed noise the Fourier embedding. However, as also leverage BARF's annealing, we add these random frequencies as offsets to the logarithmically spaced frequencies. Without these random offsets, artifacts from the axis-aligned frequencies, like stripes, can occur. Further details are available in the supplementary material.
34 Fourier
Rough camera pose quadrant initialization. We observe that camera pose optimization is a highly non-convex problem and tends to get stuck quickly in local minima. To combat this, we propose to annotate camera pose quadrants with simple binary questions: Left vs. Right, Above vs. Below, and Front vs. Back. This only takes about 4-5 minutes for our typical 80 image collection. Note that our pose quadrant initialization is much noisier than adding some noise around GT camera poses as in some related works such as NeRF (57). This rough pose initialization is in line with recent works such as NeRS [61 that also use rough manual pose initialization.
Flexible object-centric camera parameterization for varying camera distances. We define the trainable per-image camera parameters using 'look-at' parameterization with a 3D look-at vector T E j {I..... 9). a scalar up rotation uj E R[-T,, π| and a 3D camera position L; E R as well as focal length ʃ E IR. Furthermore, these are stored as offsets to the initial parame- ters, enabling easier regularization. We additionally store the offset vertical focal lengths Sy a
compressed manner similar to NeRF-.: f, V5/h [57], where h is the image height in pixels. The cameras are initialized based on the given pose quadrants and an initial field of view of 53.13 degrees. We optimize a perspective pinhole camera with fixed principal point but per-image focal lengths. The cameras are not always equidistant to the object for the in-the-wild image collections. To account for variable camera-object distances, we do not set fixed near and far bounds for each ray which is a standard practice neural volumetric optimizations such as NeRF 40). Instead, we define sampling range based on the camera distance to origin, e.g. the near bound is |o and the
5

---

far bound is |o] + where is defined as our sampling radius with a diameter of 1. We illustrate this sphere with near and far bounds in Fig. E] This explicit computation of near and far bounds for each ray enables placing the cameras at arbitrary distances from the object. This is not possible with the existing neural volume optimization techniques that use fixed near and far bounds for each camera ray. The cameras are then placed based on the quadrants and placed at distance to make the entire neural volume v visible. This look-at parametcrization is more flexible for optimizing object-centric neural volumes than more commonly used 3D rotation matrices.
Camera multiplexes. We observe that camera pose optimization gets stuck in local minima even with rough quadrant pose initialization. To combat this, inspired by mesh optimization works (e.g. (2I)), we propose to optimize a camera multiplex with 4 randomly jättered poses around the quadrant center direction for each image. Optimizing multiple cameras per image would reduce the number of rays we can cast in a single optimization step due to memory and computational limita- tions. This makes camera multiplex optimization noisy and challenging learning neural volumes. We propose techniques to make camera multiplex learning more robust by dynamically re- re-weighing the loss functions associated with different cameras in a multiplex during the optimization, This process is visualized in Fig. Specifically, we compute the mask reconstruction loss LMasj e RR associated with each camera and image j. We then re- re-weigh each camera loss in a multiplex with Sj = (ooffmar(-A,,cc (mssk)) where Sj- E IR and X is a scalar that is gradually increased during the optimization. That is, we re-weigh the loss with [ = =; SJ,, Leewwokk This dynamic re- weighing reduces the influence of bad camera poses while learning the shape and materials. Since we can only render random set of rays within each batch, we update the camera multiplex weights Sj with a memory bank and momentum across the batches. See the supplements for more details.
scaling of input images. Some images are noisier than others (e.g., due to camera shake) or noisy object masks My. To be robust against such noisy data, propose we-weigh images in the given collection. We keep a circular buffer of around 1000 elements with the recent mask losses and rendered image losses with multiplex scaling applied. We use this buffer to calculate the mean 14 and standard deviation o; of these losses. Given the recent loss statistics we also create a loss pu (Exua Cinss,)
Posterior
scalar using: R₃ = max max(tanh
+ 11, 1). In a similar way to the camera posterior scaling, we employ it per-image basis using: LNeworrk R2 LNewwork..
3.2 Losses and Optimization
Image reconstruction loss is a Chabonnier loss: (g.p) = Vlg- p)² + 0.001 between the input color from C' for pixel s and the corresponding predicted color of the networks غ. We additionally calculate the loss with the rendered color č.
Mask losses consist of two terms. One is the binary cross- cross-entropy loss LBCE between the volume- rendered mask and the estimated foreground object mask. The second one is the background loss Baukkgouud from NeRD [1. which forces all samples for rays cast towards the background to be O. We combine these losses as the mask loss:
LMask
LBCE + Baackroomd
Regularization losses We compute the gradient of the density to estimate the surface normals. We use the normal direction loss Ludir from (54) to constrain the normals to face the camera until the ray reaches the surface. This helps in obtaining sharper surfaces without cloud-like artifacts.
BRDF losses. The task of joint estimation of BRDF and illumination is quite challenging where the illumination can fall into a local minimum. For example, the object can be tinted in bluish color, and the illumination is an orange color to express a more neutral color tone. As our image collections have multiple illuminations, we can force the base color b. to replicate the pixel color from the images. This way, a mean color over the dataset is learned and prevents falling into the local minima. We leverage the Mean Squared Error (MSE) for this: Lisi LMSE (C* (C*, b.). Additionally, we find that a smoothness loss LSmooth for the normal, roughness and metallic parameters similar to the one used in UNISURF 44 further regularizes the solution.
Overall network and camera losses. The final loss to optimize the decomposition network is then defined as LNetwork Xtohimage (Ce,ū) + (1 - Li)gee (C*, é) + LMas + XaLini + Andir + Asmooth Symoneh + >Dec Smonth &Dec Smooth + >Dec Sparsity LDec Sparsity- Here, > and X, are the optimization scheduling variables. Furthermore, the camera posterior scaling is applied to these losses. For the camera optimization, we leverage the same losses as described in LNewwock However, as the camera

---

should constantly be optimized, we do not apply posterior scaling to the losses when optimizing cameras. This allows cameras to leave the local minima. Here, we only calculate the mean loss, and therefore badly initialized camera poses can still recover over the training duration. Additionally, we define an LI loss on the look-at vector v, to constrain the camera pose to look at the object. The loss is defined as Llookat- We also use volume padding loss, which prevents cameras from going too far into our volume bound t: LBounds = max((u - [t,1)2,0).
Optimization scheduling. Fig. 4 shows the optimization schedule of different loss weights. We use three fading À variables to transition the optimization schedule smoothly. The Xe is mainly used to increase image resolution and reduce the number of active multiplex cameras. The direct color è optimization is faded to the BRDF optiiization using As and some losses are scaled by X as defined earlier. Furthermore, we perform the BARF 34 frequency annealing in the early stages of the training and delay the focal length optimization to the later stages of the training. We use two different optimizers. The networks are optimized by an Adam optimizer with a learning rate of le-4 and exponentially decayed by order of magnitude every 300k steps. The camera optimization is performed with a learning rate of 3e-3 and exponentially decayed by order of magnitude every 70k steps. Further details of the optimization schedule are available in the supplementary material.
Poses Nor Knowa (3in
Poses Available (5) Sat
Metled
Pose Inii
PSNRT
SSIM
Transhtion]
Rotation 11
BARF-A
SAMURAL
Mathed
PSNR
23.66
16.9
SSIM
0.99
PSNR|
22.94
19.7
SSIM"
0.75
8:39
Tranelati)
23.34
Roseien '1
2.95
0.96
McRS C1
BARF [3 CINERE
5S
Runden
Qualrant
Quadram
14.96
12:44
20.3
0.47
0.61
0:68
81.22
34.64
32.27
0.55
2,59
0.77
NeRD (T]
26 94
0.95
SAMURAI
Quadrant
21.08
1.76
33.95
0.71
27.73
MtRD NeL
11
GT
23.86
0.88
Table 1: Novel View Synthesis on varying il- lumination datasets. We split our datasets into those where we have poses, and highly challeng- ing ones where the poses were not recoverable with classical methods. SAMURAI achieves con- siderably better performance compared to BARF- A. For reference, we also show the metrics from NeRD and Neural-PIL which require GT poses and do not work on images with unknown poses.
P
GT
23.95
0.50
Table 2: Novel View Synthesis on single illu- mination datasets. For two scenes under single illumination the poses are easily recoverable. Furthermore, we can now compare with GNeRF which does not require pose initialization. As seen our method achieves a good performance. For reference the view synthesis metrics with NeRD and Neural- PPLL that use GT known poses are also shown.
Method
PSNRT
SSIM
w/o Camera Multiplex
23.01
0.87
w/o Posterior Scaling
23.51
0.88
w/o Reeeoling/Resouution
22.73
0.83
w/o Random Fourier Offsets
24.01
0.91
w/o Regularization
21.77
0.86
Full
24.31
0.92
Figure 5: SAMURAI-Datasel Novel Views. Here, we show novel views and illuminations from the SAMURAI datasets alongside our recon- struction. The illumination conditions are accu-
Table 3: Ablation study, view synthesis and re- lighting results on two scenes (Garbage Truck and NeRD car) show that ablating any of the proposed aspects of SAMURAI can result in worse results demonstrating their importance.
rately reproduced.
4 Experiments
Datasets For evaluations, we created new image collections of 8 objects (each with 80 images) captured under unique illuminations and locations and few different cameras. We refer this dataset as the SAMURAI dataset. The images reflect the practical and challenging input scenario we are targeting in this work. Common methods such as COLMAP fail to estimate correspondences and camera poses for this dataset. Therefore, we cannot run methods that require poses on this dataset. Additionally, we evaluate on CC-licensed image collections from online sources of the statue of liberty and chair. We also use the 3 synthetic and 2 real-world datasets of NeRD [1] under varying illumination, where poses are available. Lastly, to showcase the performance with other methods, use the 2 real-world datascts from NeRD, which are taken under fixed illumination. In total,
7

---

Robot
Robot
Engine
Fire Engine
Ou
BARFA
Figure 6: Comparison with BARF-A. When comparing novel view synthesis and relighting results of SAMURAI (top) with BARF-A (bottom), SAMURAI produces more accurate camera poses and captures the object better. BARF-A is sometimes unable to recover the shape and poses.
Input
Basecolor
Metallic
Roughness
Normal
Re-render
Ours
Neurat-PLL
Figure 7: Comparison with Neural-PII, decomposition. Notice our accurate pose alignment, plausible geometry, reduced floaters and accurate BRDF decomposition, compared to Neural-PIL., even without relying on near perfect poses.
evaluate SAMURAI on 17 scenes. Please refer the supplementary material for an overview of the SAMURAI datasets along with other datasets we experimented with.
Baselines. Currently, there exists no prior art that can tackle varying illumination input images while jointly estimating camera poses. So, compare with modified BARF (34) technique, which can store per-image appearances a latent vector. We call this baseline BARF-A. Additionally, on scenes with fixed illumination, we can compare with GNeRF (38), the regular BARF, and modified version of NeRS 66 (details in the supplement). On the datasets where poses are easily recovered given, we can also compare with NeRD [1] and Neural-PIL [12], which require known, near- perfect camera poses. We provide BARF, BARF-A, and NeRS with the same pose initialization used in SAMURAI.
Evaluation. We perform novel view synthesis using learned volumes and use standard PSNR and SSIM metrics w.r.t. ground-truth views. We held out every 16th image for testing. We optimize the cameras and illuminations on the test images for evaluation purposes but do not allow the test images to affect the main decomposition network or camera training. Additionally, we perform the Procrustes analysis on the recovered camera poses to evaluate the pose optimization.
Results on varying illumination datasets. We divide the varying illumination datasets into the ones without GT poses, and those with accurate camera poses (either via GT or COLMAP). Since these datasets have varying illumination, we need to perform both view synthesis and relighting to obtain target views. Table [ shows the metrics computed w.r.t. test views. Results show that we considerably outperform BARF-A in both PSNR and SSIM metrics while at the same time solving more challenging BRDF decomposition task. Visual results in Fig. flaso clearly demonstrate better view synthesis and relighting results compared to BARF-A. BARF-A fails to align the camera poses, whereas SAMURAI achieves more accurate camera poses and drastically improved reconstruction quality. Only slightly perturbed poses are leveraged as starting positions in the original BARF method. Our coarse pose initialization is too noisy for the method to work accurately. SAMURAI overcomes this issue with the camera multiplex and other optimization strategies.
Fig. 7 shows the visual comparison of BRDF decompositions on MotherChild dataset from NeRD II along with the corresponding results from Neural-PIL. [12. In general, our method can decompose the scene even with unknown camera poses. SAMURAI also produces fewer float- ing artifacts and creates more coherent surface. The roughness parameter is also more plausible in our result, as the object is rough, whereas Neural-PII. estimated a near mirror-like surface. Further results from the SAMURAI dataset are shown in Fig.5] We show novel views and relighting results w.r.t the target test views. Visual results clearly show that SAMURAI can recover the pose and provide a consistent illumination w.r. the ground-truth target views.
8

---

SXMURAI
BARE
Head
SOMERAI
BARE
ONCF
Nett
ONAR
NOBS
Cape
SAMURAI
BAR
Cape 2
SAMURAL
BARK
ON-FF
Noks
GNRR
SORS
Figure 8: Comparison with Baselines. When comparing SAMURAI with the baselines (GNeRF,
BARF, and NeRS) ours outperforms all methods in reconstruction quality and pose estimation.
Results on fixed illumination datasets. For image collections captured under fixed illumination, we can compare with more techniques. We compare with GNeRF, the default BARF and NeRS. We additionally can compare with Neural-PIL [2 and NeRD [T] on the near-perfect camera poses re- covered from COLMAP. Results in Table show that SAMURAI outperforms the baselines BARF, GNeRF, and NeRS and is also close Neural-PIL. and NeRD that uses GT camera poses. GNeRF does not require rough pose initialization. Overall our method also achieves good pose recovery, where NeRS only slightly outperforms our method in the translational error due some outliers in our case. These outliers do not degrade our reconstructions due to our image posterior loss.
Fig. shows sample view synthesis results of SAMURAI, BARF, GNeRF, and NeRS sample single illumination datasets. Visuals indicate better results with SAMURAI compared to GNeRF and BARF NeRS seems capture more apparent detail, but the general decomposition quality is significantly better in our method, where the cape gold material is represented more accurately. NeRS also introduces misaligned face texture in the Head scene, where two faces are visible.
Direction
Direction 2
Direction
BARF
SAMURAI
Figure 9: BARF directional conditioning entanglement. BARF's radiance output is conditioned based on the view direction. When we manipulate the view direction while keeping the camera static, we can see that the embedding is still entangled with the pose. BARF improves the pho- tometric training error of unaligned poses by shifting the texture. SAMURAI instead estimates view-independent BRDFs, where this entanglement is not possible. Here, we provide magnifica- tions on edges where this entanglement is noticeable. Due to different camera coordinate systems and optimizations, we provide similar views from SAMURAI.

---

Entanglement of View direction and Radiance Conditioning in BARE Based on the view direc- tion, BARF conditions the output radiance similar to NeRF [40]. In Fig. we show the effect of fixed camera with varying directional conditioning of the radiance. The texture starts to shift on the surface. BARF reduced the photometric training error of unaligned poses by slightly shifting the texture for these views. This can result in shifting textures in novel view synthesis. With the shifting textures, the shape representation worsens, and the pose alignment quality is further limited.
Our BRDF decomposition creates a view-independent texture representation, and our illumination global and does not influence the static texture information. With this representation shifting the textures is impossible, and the texture can only remain static. Especially in pose alignment, this is highly beneficial as each camera pose has to align to a globally static model. This also explains our improvement in quality in novel view synthesis in Table I and Table 2 In the supplementary, we also show the effects of appearance conditioning in BARF-A.
Ablation Study. We perform an ablation study where we ablate different aspects of the SAMU- RAI model to analyze their importance. Table 3 shows the novel view synthesis average metrics on the Garbage Truck collection from the SAMURAT dataset and the synthetic car dataset from NeRD I.. Metrics show that regularization and the coarse-to-fine optimization scheme (Fourier annealing and resolution) are the most significant contributing factors the final reconstruction quality. The multiplex cameras and the posterior scaling also improve the reconstruction quality, stabilizing the training and preventing cameras to get stuck in local minima. Visual comparisons of the specific ablations are available in the supplements.
Applications. One of the contributions of this work is the extraction of explicit mesh with material properties from the learned neural reflectance volume. The process is described in the supplementary material. The resulting mesh can be realistically placed in an Augmented Reality (AR) scene or in 3D game. In addition, one could edit the BRDF materials the recovered mesh. See Fig. [ for sample results of these applications, where our recovered 3D assets blend well in a given 3D scene.
Limitations. SAMURAI achieves large strides in the decomposition of in-the- the-wild image collec- tions compared to prior art. However, we still rely rough pose initialization. GNeRF proposes a reconstruction technique without any pose initialization but fails on the challenging in-the-wild datasets. Furthermore, SAMURAI produces slightly blurry textures. This is especially noticeable texture. Reconstruction
in the cape scene in Fig. 8 Here, the cape has a repeating, of this high-frequency texture requires near-perfect camera poses. Since this dataset is in sin- gle location and illumination, COLMAP-based pose estimation outperforms SAMURAI based pose alignment. However, SAMURAI enables the reconstruction of highly challenging datasets of online image collections where COLMAP completely fails. Our BRDF and illumination decomposition is also not capable of modelling shadowing and inter-reflections. As we mainly tackle object decom- position, the shadows and reter-reflections are not crucial. Removing the need for pose initialization along with modeling shadows and inter-reflections form an important future work.
5 Conclusion
SAMURAI is a carefully designed optimization framework for joint camera, shape, BRDF, and illu- mination estimation. It can work on in-the-wild image collections captured in varying backgrounds and illuminations and with different cameras. Results on existing and our new challenging dataset demonstrate good view synthesis and relighting results, where several existing techniques fail. In addition, our mesh extraction allows the resulting 3D assets to be readily used in several graphics applications such as AR/VR, gaming, material editing etc.
Acknowledgments and Disclosure of Funding
This work has been partially funded by the Deutsche Forschungsermiincchatt (DFG, German Re- search Foundation) under Germany's Excellence Strategy EXC number 2064/1 - Project number 390727645 and SFB 1233, TP 02- Project number 276693517. It was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FK7: 011S1 11S38039A
10

---

References
[1] Miik Aittala, Timo Aila, and Jaakko Lehtinen. Reflectance modeling by neural texture syn- thesis. In ACM Transactions on Graphics (ToG), 2018.
[2] Rachel Albert, Dorian Yao Chan, Dan B. Goldman, and James F. O'Brian. Approximate svBRDF estimation from mobile phone video. In Eurographies Symposium on Rendering, 2018.
[3] Louis-Philippe Asselin, Denis Laurendeau, and Saan-François Lalonde. Deep SVBRDF esti- mation real materials. In International Conference on 3D Vision (3DV), 2020.
[4] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti anti-aliasing neural radiance fields. IEEE Interaational Conference on Computer Vision (ICCV), 2021.
[5] Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yan- nick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. Neural reflectance fields for appearance acquisition. ArXiv e-prints, 2020.
[6] Sai Bi, Zexiang Xu, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Krieg- man, and Ravi Ramamoorthi. Deep reflectance volumes: Relightable reconstructions from multi-view photometric images. In European Conference on Computer Vision (ECCV), 2020.
[7] Sai Bi, Zexiang Xu, Kalyan Sunkavalli, David Kriegman, and Ravi Ramamoorthi. Deep 3d capture: Geometry and reflectance from sparse multi-view images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[8] Mark Boss and Hendrik P.A. Lensch. Single image brdf parameter estimation with condi- tional adversarial network. In ArXiv e-prints, 2019.
[9] Mark Boss, Fabian Groh, Sebastian Herholz, and Hendrik P. A. Lensch. Deep Dual Loss BRDF Parameter Estimation. In Workshop on Material Appearance Modeling, 2018.
[10] Mark Boss, Varun Jampani, Kihwan Kim, Hendrik P.A. Lensch, and Jan Kautz. Two-shot spatially-varying BRDF and shape estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[11] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, and Hendrik P.A. Lensch. NeRD: Neural reflectance decomposition from image collections. IEEE Interna- tional Conference on Computer Vision (ICCV), 2021.
[12] Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan T. Barron, and Hendrik P.A. Lensch. Neural-pil: Neural pre-integrated lighting for reflectance decomposition. In Advances Neural Information Processing Systems (NeurIPS), 2021.
[13] Eric Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-GAN: Periodic implicit generative adversarial networks for 3D-aware image synthesis. In IEEE Con- ference Computer Vision and Pattern Recognition (CVPR), 2021.
[14] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. TensoRF: Tensorial radiance fields. In ArXiv e-prints, 2022.
[15] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[16] Robert L. Cook and Kenneth E. Torrance. A reflectance model for computer graphics. ACM Transactions on Graphics (ToG), 1982.
[17] Valentin Deschaintre, Miika Aitalla, Fredo Durand, George Drettakis, and Adrien Bousseau. Single-image SVBRDF capture with a rendering-awant deep network. In ACM Transactions on Graphics (ToG), 2018.

---

[18] Valentin Deschaintre, Miik Aitalla, Fredo Durand, George Drettakis, and Adrien Bousseau. Flexible SVBRDF capture with a mulli-image deep network. In Eurographics Symposium on Rendering, 2019.
[19] Valentin Deschaintre, George Drettakis, and Adrien Bousseau. Guided fine-tuning for large- scale material transfer. In Eurographics Symposiun on Rendering, 2020.
[20] Duan Gao, Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. Deep inverse rendering for high- resolution SVBRDF estimation from an arbitrary number of images. In ACM Transactions on Graphics (SIGGRAPH), 2019.
[21] Shubham Goel, Angjoo Kanazawa, and Jitendra Malik. Shape and viewpoint without key- points. In European Conference on Computer Vision (ECCV), 2020.
[22] Peter Hedman, Pratul P. Srinivasan, Ben Mildenhall, Jonathan T. Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. In IEEE International Conference on Computer Vision (ICCV), 2021.
[23] Philipp Henzler, Valentin Deschaintre, Niloy Mitra, and Tobias Ritschel. Generative mod- elling of BRDF textures from flash images. ACM Transactions Graphics (SIGGRAPH
ASIA), 2021. 2021.
[24] Yoonwoo Jeong, Scokjun Ahn, Christopher Choy, Animashree Anandkumar, Minsu Cho, and Jaesik Park. Self-calibrating neural radiance fields. In IEEE International Conference Computer Vision (ICCV), 2021.
[25] Berk Kaya, Suryansh Kumar, Carlos Oliveira, Vittorio Ferrari, and Luc Van Gool. Uncalibrated neural inverse rendering for photometric stereo of general surfaces. In IEEE International Conference on Computer Vision (ICCV), 2021.
[26] Petr Kellnhofer, Lars Jebe, Andrew Jones, Ryan Spicer, Kari Pulli, and Gordon Wetzstein. Neural lumigraph rendering. In IEEE Conference Computer Vision and Puttern Recognition (CVPR), 2021.
[27] Zhengfei Kuang. Kyle Olszewski, Menglei Chai, Zeng Huang, Panos Achlioptas, and Sergey Tulyakov. NeROIC: Neural object capture and rendering from online image collections. ArXiv e-prints, 2022.
[28] Jason Lawrence, Szymon Rusinkiewicz, and Ravi Ramamoorthi. Efficient BRDF importance sampling using factored representation. ACM Transactions on Graphics (ToG), 2004.
[29] Hendrik P. A. Lensch, Jan Kautz, Michael Gosele, and Hans-Peter Seidel. Image-based recon- struction of spatially varying materials. In Eurographics Conference on Rendering. 2001.
[30] Hendrik P.A. Lensch, Jochen Lang, M. Sa Asla, and Hans-Peter Seidel. Planned sampling of spatially varying BRDFs. In Computer Graphics Forum, 2003.
[31] Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. Modeling surface appearance from a single photograph using seff-augmented convolutional neural networks. In ACM Transactions on Graphics (ToG), 2017.
[32] Zhengqin Li, Kalyan Sunkavalli, and Manmohan Chandraker. Materials for masses: SVBRDF acquisition with a single mobile phone image. In European Conference on Computer Vision (ECCV), 2018.
[33] Zhengqin Li, Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and SVBRDF from a single image. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2020.
[34] Chen-Hsuan Lin, Wei-Chiu Ma. Antonio Torralba, and Simon Lucey. Barf: Bundle-adjusting neural radiance fields. In IEEE International Conference on Computer Vision (ICCV), 2021.
[35] Lingje Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
12

---

[36] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. ACM Transactions on Graphics (ToG), 2019.
[37] Ricardo Martin-Brualla Noha Radwan, Mchdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the Wild: Neural Radiance Fields for Uncon- strained Photo Collections. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[38] Quan Meng, Anpei Chen, Haimin Luo, Minye Wu, Hao Su, Lan Xu, Xuming He, and Jingyi Yu. GNeRF: GAN-based Neural Radiance Field without Posed Camera. In IEÉE International Conference on Computer Vision (ICCV), 2021.
[39] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In IEEE Con- ference Computer Vision and Pattern Recognition (CVPR), 2019.
[40] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV), 2020.
[41] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Transactions on Graphics (ToG), 2022.
[42] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Mueller, and Sanja Fidler. Extracting Triangular 3D Models, Materials, and Lighting From Images. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[43] Giljoo Nam, Diego Gutierrez, and Min H. Kim. Practical SVBRDF acquisition of 3d objects with unstructured flash photography. In ACM Transactions on Graphics (SIGGRAPH ASIA), 2018.
[44] Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit sur- faces and radiance fields for multi-view reconstruction. In IEEE International Conference on Computer Vision (ICCV), 2021.
[45] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In IEEE Conference on Computer Vision and Pattern Recognition JCVPR),June 2019.
[46] Xuebin Qin, Zichen Zhang, Chenyang Huang, Masood Dehghan, Osmar Zaiane, and Martin Jagersand. U2-net: Going deeper with nested u-structure for salient object detection. In Pattern Recognition, volume 106, 2020.
[47] Shen Sang and Manmohan Chandraker. Single-shot neural relighting and SVBRDF estimation.
In European Conference Computer Vision (ECCV), 2020.
[48] Sara Fridovich-Keil and Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[49] Johannes Lutz Schönberger and Jan-Michael Frahm. Structure- fronee-foom-aee motion revisited. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[50] Johannes Lutz Schönberger, Enliang Zheng, Mare Pollefeys, and Jan-Michael Frahm. Pixel- wise view selection for unstructured multi-view stereo. In European Conference on Computer Vision (ECCV), 2016.
[51] Soumyadip Sengupta, Jinwei Gu, Kihwan Kim, Guilin Liu, David W. Jacobs, and Jan Kautz. Neural inverse rendering of an indoor scene from a single image. In IEEE International Con- ference on Computer Vision (ICCV), 2019.
[52] Pratul P. Srinivasan, Boyang Deng, Xiuming Zhang, Matteww Tancik, Ben Mildenhall, and Jonathan T. Barron. NeRV: Neural reflectance and visibility fields for relighting and view synthesis. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
13

---

[53] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenball, Sara Fridovich- Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems (NeurIPS), 2020.
[54] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T. Barron, and Pratul P. Srinivasan. Ref-neRF: Structured view-dependent appearance for neural radiance fields. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[55] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang- NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[56] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[57] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. NeRF-- Neural radiance fields without known camera parameters. ArXiv e-prints, 2021.
[58] Chun-Han Yao. Wei-Chih Hung, Yuanzhen Li, Michael Rubinstein, Ming- Hsuan Yang, and Varun Jampani. Lassie: Learning articulated shapes from sparse image ensemble via 3d part discovery. In Advances Neural Information Processing Systems (NeurIPS), 2022.
[59] Wenjie Ye, Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. Single image surface appearance modeling with self self-augmented cnns and inexact supervision. Computer Graphics Forum, 2018.
[60] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. In IEEE International Conference on Com- puter Vision (ICCV), 2021.
[61] Jason Zhang, Gengshan Yang, Shubham Tulsiani, and Deva Ramanan. NeRS: Neural flectance surfaces for sparse-view 3d reconstruction in the wild. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2021.
[62] Jianzhao Zhang, Guojun Chen, Yue Dong, Jian Shi, Bob Zhang, and Enhua Wu. Deep inverse rendering for practical object appearance scan with uncalibrated illumination. In Advances in Computer Graphics, 2020.
[63] Kai Zhang, Fujun Luan, Qianqian Wang. Kavita Bala, and Noah Snavely. PhySG: Inverse rendering with spherical Gaussians for physics-based material editing and relighting. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2021.
[64] Xiuming Zhang, Pratul Srinivasan, Boyang Deng, Paul Debevec, William Freeman, and Jonathan T. Barron. NeRFactor: Neural factorization of shape and reflectance under un- known illumination. In ACM Transactions on Graphics (SIGGRAPH ASIA), 2021.
[65] Yuxuan Zhang, Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, Antonio Torralba, and Sanja Fidler. Image GANs meet differentiable rendering for inverse graphies and interpretable 3d neural rendering. In International Conference on Learning Representations (ICLR), 2021.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A] No directly negative societal impacts are easily apparent from our method.
14

---

(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material as URL)? [No] We will release
it.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'te re using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "z2k7n3cn2ra", "reviewer": "Reviewer_eZCF", "summary": "This paper uses a neural field-based approach to estimate the shape, BDRF parameters, and per-image camera pose and illumination from the in-the-wild image collections. It is the first method that is able to estimate all parameters simultaneously. Conventional neural field-based approaches require an almost-correct camera pose to estimate the shape and BDRF parameters. The proposed method replaces the need for accurate camera pose with a rough user-initialized camera pose quadrant. This paper introduces a novel in-the-wild image collection dataset in which COLMAP is hard to estimate the camera pose. Experimental results show that the proposed method achieves much better accuracy in the challenging dataset (pose not known).", "questions": "- In L157, the authors claim that they do not use the coarse-to-fine network. What if the proposed method use coarse network? Wouldn’t the accuracy be better? In addition, what is the coarse-to-fine optimization in the ablation study?\n- Why couldn’t the proposed method achieve GT camera pose? What if the proposed method utilizes deep camera pose estimation (such as Wide-Baseline Relative Camera Pose Estimation with Directional Learning) to improve the accuracy? Would it be possible?\n", "limitations": "Yes, the authors has adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n- The proposed method is able to solve a highly under constrained problem with rough camera pose initialization. It estimates the BRDF parameters, unit-length surface normal, volume density, latent-per-image illumination vectors, per image camera pose and intrinsics.\n- Flexible camera parameterization for varying distances. As the proposed method targets for in-the-wild image collection, the near and far bound of conventional methods cannot be applied. Thus, the proposed method places the camera in a distance where the is visible given the field of view.\n- Camera multiplex optimization. It is challenging to optimize the camera pose due to local minima. The proposed methods optimize multiple camera poses with their corresponding weights based on the camera loss to find the best possible camera pose. \n- Posterior scaling of input images. Similar to camera multiple optimization, the proposed method also optimizes the image collection used for training. It gives different weights based on the noise level for each image.\n- The proposed method can be applied for various applications, such as AR & VR, which makes it easy for those applications to insert the real-world objects without huge efforts. \n\n\n# Weaknesses\n\n- The authors claim that standard pose estimation techniques fail on the challenging images, but there is no justification whether the proposed dataset is a challenging dataset. Proof of pose of estimation failure should be included.\n- While the conventional methods automatically estimate the camera pose with COLMAP, the proposed method requires user interaction for each dataset to roughly annotate the position. However, there is no justification why the proposed rough pose estimation is preferred. In addition, there is no ablation study of the camera pose initialization using 3 simple binary questions.\n- The intention of the proposed method is to do proof-of-concept of large-scale set objects. However, the qualitative evaluation of large-scale set objects is unavailable in the main manuscript. It is questionable whether the proposed method can be scaled to large-scale set objects. In the supplementary material, there are two online image collections, but the results are not satisfying. In addition, comparison with state-of-the-art methods is unavailable.\n    - As in Modeling the World from Internet Photo Collections, the camera pose of large objects images might be estimated using correspondences.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "tQ8LCZ-VhFN", "reviewer": "Reviewer_hfWC", "summary": "This paper proposed a method that works on in-the-wild image collections of an object to estimate the shape, BRDF, per-image camera pose, camera intrinsics and illumination in a jointly optimized framework.\nThis problem is very challenging and under constraint as the camara parameters, illuminations and even the background of the images may vary a lot across the online image collections. To my knowledge, this is the first method that aims at solving the shape, BRDF and camera poses with this challenging setup. \n\nThe paper proposed several components that based on the recent Neural-PIL-Rendering technique to construct the entire pipeline, including: an object-centric camera parametrizetion to learn the clipping planes per image; camera multiplex optimization to avoid local minima during the optimization; posterior scaling of input images to suppress the influence of corrupts images; a two-stage mesh extraction for refined mesh reconstruction.", "questions": "See weaknesses part.", "limitations": "The most of the limitations are discussed in the paper. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n- The paper is well written and easy to follow. The figures, tables and videos help the understanding of the paper and identify its contributions.\n\n- The proposed camera multiplexes, although been used in mesh optimization works [22] before, is the first time being used in the context of neural volume rendering. This technique will dynamic re-weighing the each camera loss to reduce the influence of bad camera poses during the optimization of the shape and materials.\n\n- The proposed posterior scaling of input images also makes sense in the context of suppressing the influence of corrupted images. This technique can also be used in a broader application: when dealing with online image collections and images with poor quality can be dynamically re-weigh.\n\n- The evaluation of the method over real and synthetic datasets demonstrates the effectiveness of this method in recovering shape and material under unknown camera poses, changing illuminations and varying backgrounds. (From Table 1 and Table 3, Table A3)\n\n- The Ablation study in table 2 also validates the effectiveness of each component of the methods. \n\nWeaknesses\n- In table 1, the SAMURAI's performance is lower when the camera poses are available. Why?\n\n- I have a concern about the quality of the reconstructed meshes. In Figure A4, the reconstructed mesh of \"Statue of Liberty\" and \"Chair\" is inaccurate, and the texture is also blurry. \n\n- The novel view synthesis quality is very low compared to the prior \"fix-illumination\" or \"known camera pose\" methods. As can be seen in Table 1, Table 3 where the reconstructed images have much lower PSNR compared to prior works. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5h2e2G_girj", "reviewer": "Reviewer_m5j4", "summary": "This paper proposes a NeRF-based single-object inverse rendering pipeline where the camera intrinsics is unknown and extrinsics are coarsely initialized. The neural implicit based reflectance (BRDF) is also optimized separately for relighting/material editing. Compared with previous methods that uses camera intrinsics and extrinsics estimated from a third-party pipeline (eg. COLMAP), the proposed method is more robust in cases where COLMAP fails to estimate accurate poses due to the camera optimization module. A small dataset consisting of images captured with different background/illumination for 8 objects is collected and used in evaluation of the proposed method. The experiment results show that the proposed method outperforms the compared state-of-the-art methods that also use unposed input images. \n", "questions": "Please see the questions in the weakness section.", "limitations": "The method does not specifically handles shadows, as the author pointed out in the paper.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n+The joint optimization of the BRDF, occupancy represented as neural implicit functions and the camera parameters (poses and intrinsics) given unposed images captured under different illuminations/backgrounds is a challenging task. The recipes used in this submission can be of good reference for the following works that pushes the NeRF-based inverse rendering towards in-the-wild scenarios.\n\n+Compared with the similar method (BARF), the performance boost of the proposed pipeline is obvious.\n\n+A small dataset is proposed for challenging cases where the COLMAP fails (L276-L277). This dataset can be useful for testing algorithms using unposed images with varying illumination as inputs.\n\n\nWeakness\n\n-My main concern is about the limitation in the cases where the proposed method performs better than the compared method: \nOther methods, such as NeRF-W, also uses internet photos with varying background/illumination/camera intrinsics as inputs; the key difference between the work and the previous works is that this work is robust to cases where COLMAP fails to estimate good camera poses or does not work at all, versus the assumption that COLMAP works for in-the-wild dataset in other methods. Although this work has proposed a small dataset on which COLMAP is not working at all and shows that the proposed method works on this dataset, the following question is not answered: if COLMAP does estimate some noisy camera poses, can this method with optimized pose method still output perform the counterparts (either the other parts of this method + COLMAP pose, or NeRD + COLMAP pose) ? From Tab.3 of this paper, it shows that if COLMAP generates good camera pose, then the performance of the proposed method is not as good as the compared methods. This seems to reduce the application scope of this proposed method: it only works better if COLMAP does not work, which is not always the case given that COLMAP has decent robustness.\n\n-Self occlusion/shadowing is not considered. This poses a challenge for objects with concave shapes (grooves) that generate cast shadows, which may be reconstructed as albedo variations.\n\n-Based on L181-L199, the 'valid' sampling ranges for the rays are determined by the intersections between the predefined object-centric sphere and the camera rays, which are intern determined by the camera pose and intrinsics. As a result, the sampling location along the rays depends on the camera intrin/extrin. If that is the case, how the dependence is dealt with during the optimization process is not well explained. \n\n-The weighting scheme in camera multiplexes depends on the reliability of each candidate pose. How about the cases where none of the positions from one multiplex is reliable? Are the corresponding images ignored during reconstruction?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept