Structure from Duplicates: Neural Inverse Graphics from a Pile of Objects
Authors: Tianhang Cheng
Wei-Chiu Ma
Kaiyu Guan
Antonio Torralba
Shenlong Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: 3d reconstruction
inverse rendering
pose estimation
single view reconstruction
nerf
duplicates
TL;DR: Inverse rendering from a pile of objects from single image with NeRF

Abstract: Abstract Our world is full of identical objects (\emph{e.g.}, cans of coke, cars of same model). These duplicates, when seen together, provide additional and strong cues for us to effectively reason about 3D. Inspired by this observation, we introduce Structure from Duplicates (SfD), a novel inverse graphics framework that reconstructs geometry, material, and illumination from a single image containing multiple identical objects. SfD begins by identifying multiple instances of an object within an image, and then jointly estimates the 6DoF pose for all instances. An inverse graphics pipeline is subsequently employed to jointly reason about the shape, material of the object, and the environment light, while adhering to the shared geometry and material constraint across instances.
Our primary contributions involve utilizing object duplicates as a robust prior for single-image inverse graphics and proposing an in-plane rotation-robust Structure from Motion (SfM) formulation for joint 6-DoF object pose estimation. By leveraging multi-view cues from a single image, SfD generates more realistic and detailed 3D reconstructions, significantly outperforming existing single image reconstruction models and multi-view reconstruction approaches with a similar or greater number of observations.

Structure from Duplicates: Neural Inverse Graphics from a Pile of Objects
Tianhang Cheng'
Kaiyu Guan Antonio Torralba² Shenlong Wang' 'University of Illinois Jrbana-Champatn Massachusetts Institute of Technology (tcheng12, kaiyug, shenlong) eillinois edu {weichium, torralba] (arralba]ene @mi edu
Wei-Chiu Ma²
Input RGB
DoF pose
Albedo
Surface normal
Roughness/Metalle
Object insertion
Material editing I
Material editing
Relighting
Relighting 2
Figure I: Structure from duplicates (SfD) novel inverse graphics framework that reconstructs geometry, material, and illumination from a single image containing multiple identical objects.
Abstract
Our world is full of identical objects (e.g., cans of coke, cars of same model). These duplicates, when seen together, provide additional and strong cues for us to effectively reason about 3D. Inspired this observation,
introduce Structure from Duplicates (SfD), a novel inverse graphics framework that reconstructs geom- etry, material, and illumination from single image containing multiple identical objects. SfD begins by identifying multiple instances an object within an image, and then jointly estimates the 6DoF pose for all instances. An inverse graphics pipeline subsequently employed to jointly reason about the shape, material the object, and the environment light, while adhering to the shared geometry and material constraint across instances. Our primary contributions involve utilizing robust prior for single-image inverse graphics and proposing object duplicates robust Structure from Motion (SfM) formulation for joint an in-plane
6-DoF object pose estimation. By leveraging multi-view cues from single image, SfD generates more realistic and detailed 3D reconstructions, significantly outper- forming existing single image reconstruction models and multi-view reconstruction approaches with similar or greater number of observations. Code available at https://gituub.com/ htw.r//ww...oo//aaann.naea Tranhang-Cheng/SS
Introduction
Given single/set of image(s), the goal of inverse rendering is recover the underlying geometry, material, and lighting the scene. The task is of paramount interest to many applications in computer vision, graphics, and robotics and has drawn extensive attention across the communities over the past few years[74; 47: 21; 45].
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Figure 2: Repetitions in the visual world. Our physical world is full of identical objects (e.g., cans of coke, cars of the same model, chairs in classroom). These duplicates, when seen together, provide additional and strong cues for us to effectively reason about 3D.
Since the problem is ill-posed, prevailing inverse rendering approaches often leverage multi-view observations to constrain the solution space. While these methods have achieved state-of-the-art performance, in practice, is sometimes difficult, or even impossible, to obtain those densely captured images. To overcome the reliance multi view information, researchers have sought to incorporate various structural priors, either data-driven or handcrafted, into the models[5; 36]. By utilizing the regularizations, these approaches are able to approximate the intrinsic properties (e.g.. material) and extrinsic factors (e.g., illumination) even from one single image. Unfortunately, the estimations may be biased due to the priors imposed. This makes one ponder: is it possible that take the best of both worlds? Can we extract multi-view information from a single image under certain circumstances?
Fortunately the answer is yes. Our world is full of repetitive objects and structures. Repetitive patterns single images can help us extract and utilize multi-view information. For instance, when we enter an auditorium, we often see many identical chairs facing slightly different directions. Similarly, when we go to a supermarket, we may observe multiple nearly-identical apples piled on the fruit stand. Although we may not see the exact same object from multiple viewpoints in just one glance, we do see many of the "identical twins" from various angles, which is equivalent to multi-view observations and develop a computational model that can effectively infer the underlying 3D representations from a single image by harnessing the repetitive structures of the world.
more (see Sec. 3 for more details). Therefore, the goal of this paper is to
With these motivations in mind, we present Structure from Duplicates (SfD), a novel inverse rendering model that is capable of recovering high-quality geometry, material, and lighting of the objects from single image. SfD builds upon insights from structure from motion (SfM) as well as recent advances on neural fields. At its core lies two key modules: (i) in-plane rotation robust pose estimation module, and (ii) a geometric reconstruction module. Given an image a scene with duplicate objects, exploit the pose estimation module to estimate the relative 6 DoF poses of the objects. Then, based on the estimated poses, we align the objects and create multiple "virtual cameras." This allows us effectively map the problem from single-via view multi-object setup a multi-view single-object setting (see Fig. 3). Finally, once we obtain multi-view observations, we can leverage the geometric module to recover the underlying intrinsic and extrinsic properties of the scene. Importantly, SfD can be easily extended to multi-image setup. It can also be seen as superset of existing NeRF models, where the model will reduce to NeRF when there is only one single object in the scene.
We validate the efficacy of our model on a new dataset called Dup, which contains synthetic and real-world samples of duplicated objects since current multi-view datasets lack duplication samples. This allows us to benchmark inverse rendering performance under single-view or multi-view settings. Following previous work [60; 65; 47; 69; 74], evaluate rendering, relighting and texture quality with MSE, PSNR, SSIM, LPIPS [70], geometry with Chamfer Distance (CD), and environment light with MSE.
Our contributions are as follows: 1) We proposed a novel setting called "single-view duplicate objects" (S-M), which expands the scope of the inverse graphics family with a multi-view single- instance (M-S) framework. 2) Our method produces more realistic material texture than the existing multi-view inverse rendering model when using the same number of training views. 3) Even only relying on a single-view input, our approach can still recover comparable or superior materials and geometry compared to baselines that utilize multi-view images for supervision.

---

Single Input Image
Mulli-view patches
Structure from motion
Geometry
Ground Iruth
PBRMaterial
Illumination
compare
Segmentation
From single view to mult-view
DoF pose
Rendering
Method overview: (Left) SfT begins by identifying multiple instances of object within an image, and then jointly estimates the 6DoF pose for all instances. (Right) An inverse graphics pipeline is subsequently employed to reason about the shape, material of the object, and the environment light, while adhering to the shared geometry and material constraint across instances.
Figure 3:
2 Related Work
Inverse rendering: The task of inverse rendering can be dated back to more than half a century
ago [31; 23; 24; 2]. The goal is to factorize the appearance of an object scene in the observed image(s) into underlying geometry, material properties, and lighting conditions [53; 43; 66]. Since the problem is severely under-constrained, previous work mainly focused on controlled settings or simplifications of the problem [19; 22; 1J. For instance, they either assume the reflectance of an object is spatially invariant [69], presume the lighting conditions are known [57], assume the materials are lambertian [76; 41J, or presume the proxy geometry is available [52; 32; 10; 30J. More recently, with the help of machine learning, in particular deep learning. researchers have gradually moved towards more challenging in in-the-wild settings [73; 47]. By pre-training on large amount of synthetic yet realistic data [36] or baking inductive biases into the modeling pipeline [7; 72]. these approaches can better tackle unconstrained real-world scenarios (e.g., unknown lighting conditions) and recover the underly physical properties more effectively [47: 68). For example, through properly modeling the indirect illumination and the visibility of direct illumination, Zhang et al. [74] is able to recover interreflection- and shadow-free SVBRDF materials (e.g., albedo, roughness). Through disentangling complex geometry and materials from lighting effects, Wang et al. [62] can faithfully relight and manipulate a large outdoor urban scene. Our work builds upon recent advances in neural inverse rendering. Yet instead of grounding the underlying physical properties through multi-view observations as in prior work, we focus on the single image setup and capitalize on the duplicate objects in the scene for regularization. The repetitive structure not only allows us to ground the geometry, but also provide additional cues on higher- lighting effects (e.g., cast shadows). As will show in the experimental section, we recover the geometry, materials, and lighting much more effectively even when comparing to multi-view observations.
Recovering the spatial layout of the cameras and the geometry of the scene from single or collection of images is a longstanding challenge in computer vision. It is also the cornerstone for various downstream applications in computer graphics and robotics such as inverse rendering [43; 60; 62]. 3D editing [39; 63), navigation [42; 67], and robot manipulation [27; 38]. Prevailing 3D reconstruction systems, such as structure from motion (SfM), primarily rely on multi-view geometry to estimate the 3D structure of a scene [40; 20; 54]. While achieving significant successes, they rely on densely captured images, limiting their flexibility and practical use cases. Single image 3D reconstruction, on the other hand, aims to recover metric 3D information from monocular image [14; 6; 50; 49]. Since the problem is inherently ill-posed and lacks the ability to leverage multi-view geometry for regularization, these methods have to resort to (learned) structural priors to resolve the ambiguities. While they offer greater flexibility, their estimations may inherit biases from the training data. In this paper, we demonstrate that, under certain conditions, it is possible to incorporate multi-view geometry into a single image reconstruction system. Specifically, leverage repetitive objects within the scene anchor the underlying 3D structure. By treating
3D Reconstruction:
3

---

each of these duplicates as an observation from different viewpoints, we can achieve highly accurate metric 3D reconstruction from a single image.
Repetitions: Repetitive structures and patterns are ubiquitous in natural images. They play important roles in addressing numerous computer vision problems. For instance, a single natural image often contains substantial redundant patches [77]. The recurrence of small image patches allows one to learn a powerful prior which can later be utilized for various tasks such as super- resolution [18; 26]. image deblurring [44], image denoising [15], and texture synthesis [12]. Moving beyond patches, repetitive primitives or objects within the scene also provide informative cues about their intrinsic properties [33; 64]. By sharing or regularizing their underlying representation, one can more effectively constrain and reconstruct their 3D geometry [25; 16; 91, as well as enable various powerful image/shape manipulation operations [34; 59]. In this work, we further push the boundary and attempt to recover not just the geometry, but also the materials (e.g., albedo, roughness), visibilities, and lighting conditions of the objects. Perhaps closest to our work is [75]. Developed independently and concurrently. Zhang et al. build a generative model that aims to capture object intrinsics from single image with multiple similar/same instances. However, there exist several key differences: 1) we explicitly recover metric-accurate camera poses using muuti-geometty, whereas Zhang et al. learn this indirectly through GAN-loss; 2) we parameterize and reason realistic PBR material and environmental light; 3) we handle arbitrary poses, instead of needing to incorporate a prior pose distribution. Some previous work also tried to recover 6D object pose from crowded scene or densely packed objects[46; 11] from RGB-D input, but our model only require RGB input.
3 Structure from Duplicates
In this paper, seek to devise method that can precisely reconstruct the geometry, material properties, and lighting conditions of an object from a single image containing duplicates of it. We build our model based the observation that repetitive objects in the scene often have different poses and interact with the environment (e.g., illumination) differently. This allows one to extract rich multi-vien information even from one single view and enables one recover the underlying physical properties of the objects effectively.
We start by introducing a method for extracting the "multi-view" information from duplicate objects. Then we diseuss how to exploit recent advances in neural inverse rendering to disentangle both the object intrinsics and environment extrinsics from the appearance. Finally, describe our learning procedure and design choices.
3.1 Collaborative 6-DoF pose estimation
As we have alluded to above, single image with multiple duplicate objects contains rich multi- view information. It can help us ground the underlying geometry and materials of the objects, and understand the lighting condition of the scene.
Our key insight is that the image can be seen as a collection of multi-view images stitching together. essentially transform the single image into set of multi-view images of the object from various viewpoints. One can then leverage structure from motion (SfM) [55] to estimate the relative poses among the multi-view images, thereby aggregating the information needed for inverse rendering. Notably, the estimated camera poses can be inverted to recover the 6 DoF poses of the duplicate objects. As we will elaborate in Sec. 3.2, this empowers us more effectively model the extrinsic lighting effect (which mainly depends on world coordinate) as well as to properly position objects in perspective and reconstruct the exact same scene.
By cropping each object,
To be more formal, let I HR#xxx3 x3 be an image with N duplicate objects. Let ( xxxx be the corresponding object image patches. We first leverage SiM [55; 51] to estimate the camera poses of the multi-view cropped images' {E. E (E))
(SM (I Iahi ..., Johj
E
Next, since there is only one real camera in practice, {E:) N to obtain the 6 DoF poses of the duplicate objects. Without loss of generality and for
simply align the N virtual cameras
"In practice, the cropping operation will change the intrinsic matrix of the original camera during implemen- tation. For simplicity, we assume the intrinsics are properly handled here.

---

simplicity, we align all the cameras to a reference coordinate _.e The 6 DoF poses of the duplicate objects thus become c:- Gre ((:"m) where is matrix multiplication for pose composition.
In practice, we first employ a state-of-the-un panoptic segmentation model [8] to segment all objects in the scene. Then we fit predefined bounding box to each object and crop it. Lastly, we run COLMAP [55] to estimate the 6 DoF virtual camera poses, which in turn provides us with the DoF object poses. Fig. 3(left) depicts our collaborative 6- 6-DoF pose estimation process.
Caveats of random object poses: Unfortumately, naively feeding these object patches into SIM would often Icads to failure, as little correspondence can be found. This is due to the fact that state-of-the-art correspondence estimators [51] are trained on Internet vision data, where objects are primarily upright. The poses of the duplicate objects in our case, however, vary significantly. Moreover, the objects are often viewed from accidental viewpoints [17]. Existing estimators thus struggle to match effectively across such extreme views.
Rotation-aware data augmentation: Fortunately. the scene contains numerous duplicate objects. While estimating correspondences reliably across arbitrary instances may not always be possible. there are certain objects whose viewpoints become significantly similar after in-plane rotation. Hence, we have developed an in-plane rotation- aware data augmentation for correspondence estimation.
Specifically, when estimating correspondences between a pair of images, we don't It match the images directly. Instead, we gradually rotate one image and then perform the match. The number of correspondences at each rotation is recorded and then smoothed using a running average. We take the argmax to determine the optimal rotation angle. Finally, we rotate the correspondences from the best match back to the original pixel coordinates. As we will demonstrate in Sec. 4, this straightforward data augmentation strategy significantly improves the accuracy of 6 DoF pose estimation. In practice, we rotate the image by 4 per step. All the rotated images are batched together, enabling us to match the image pairs in a single forward pass.
3.2 Joint shape, material, and illumination estimation
J Suppose now we have the 6 DoF poses of the objects [EThi), The next step is to aggregate the information across duplicate objects to recover the intrinsics properties of the objects (e.g., geometry, materials) and the extrinsic factors of the world (e.g., illumination). We aim to reproduce these attributes as faithfully as possible, so that the resulting estimations can be utilized for downstream tasks such relighting and material editing. Since the task is under-constrained and joint estimation often leads to suboptimal results, we follow prior art [74; 62] and adopt a stage-wise procedure.
We start by reconstructing the geometry of the objects. In line with NeuS [60], we represent the object surfaces as the zero level set of a signed distance function (SDF).
Geometry reconstruction:
We parameterize the SDF with a multi-layer perceptron (MLP) S : x - that maps a 3D point under object coordinate øb E RR3 to signed distance value R. Different from NeuS [60], model the geometry of objects in local object space. This allows us to guarantee shape consistency across instances by design.
We can also obtain the surface normal by taking the gradient of the SDF: n(x = V V S. To learn the geometry from multi-view images, we additionally adopt an auxiliary appearance MLP C' ::: {x, x n, n phj d, dr } + c that takes as imput a 3D point x, xhi surface normal n, nh and view direction d, ah under both coordinate systems and outputs the color E R The input from world coordinate system helps the MLP to handle the appearance inconsistencies across instances caused by lighting or occlusion. We tied the weights of the early layers of C' those of S so that the gradient from color can be propagated to geometry. We determine which object coordinate to use based on the object the ray hits. This information can be derived either from the instance segmentation or another allocation MLP A: {x, d} + (distilled from SDF MLP), where E N is the instance index. After we obtain the geometry MLP S, we discard the auxiliary appearance MLP C. As we will discuss in the next paragraph, we model the object appearance using physics-based rendering (PBR) materials that it can handle complex real-world lighting scenarios.
Material and illumination model:
Now we have recovered the geometry of the objects, the next step is to estimate the environment light of the scene as well as the material of the object. We assume
5

---

GT
M-S
S-M.
GT
Zhang et al. [74]
Ours
Diffuse
Diffuse
Relighting
Relighting
Figure 4: Multi-vies inverse rendering.
Figure 5: Multi-view single object (M-S) single-view multi-objects (S-M).
all lights come from infinitely faraway sphere and only consider direct illumination. Therefore, the illumination emitted to a 3D point certain direction determined solely by the incident light direction w, and is independent of the point's position. Similar to [69; 74], we approximate the environment light with 128 Spherical Gaussians (SGs):
L, (wi) = = kn Mge A ( (u
(2)
where À E R++ is the lobe sharpness, is the lobe amplitude, and ø is the lobe axis. This allows us to effectively represent the illumination and compute the rendering equation (Eq. 3) in closed-form.
As for object material, we adopt the simplified Disney BRDF formulation [3; 29] and parameterized as MLP M x + {a, m). Here, E _ denotes albedo, E [0, 1] corresponds to roughness, and m [0, 1] signifies metallic. Additionally, inspired by Inv InvRender [74], we incorporate a visibility [0, to approximate visibility for each world coordinate faster reference. The difference that we use the sine activation function[56] rather than the ReLU activation for finer detail. Since the visibility field is in world space,
MLP V: (x, w.)
model both inter-object self-casted shadows and inter-object occlusions for multiple instances in our setup. We query only surface points, which be derived from the geometry MIP S using volume rendering. The material MLP M also operates in object coordinates like S, which ensure material consistency across all instances. Moreover, the variations in lighting conditions between instances help better disentangle the effects of lighting from the materials. We set the dielectrics Fresnel term to Fo = 0.02 and the general Fresnel term F (I Fo ma be compatible with both metals and dielectrics.
Combining all of these components, we can generate high-quality images by integrating the visible incident lights from hemisphere and modeling the effects of BRDF [28]:
Lo (ww: x) = S/
wo; (w; n)dw,
Here, w; is the incident light direction, while w. is the viewing direction. The BRDF function fr can be derived from our PBR materials. We determine the visibility of an incident light either through sphere tracing or following Zhang al. [74] to upproximate it with the visibility MLP V. We refer the readers to the supplementary materials for more details.
3.3 Optimization
Optimizing shape, illumination, and material jointly from scratch is challenging. Taking inspiration from previous successful approaches [74; 69], we implement a multi-stage optimization pipeline. We progressively optimize the geometry first, then visibility, and finally, material and illumination.
Geometry optimization: We optimize the geometry model by minimizing the difference between rendered cues and observed cues.
min Ecotor + X₂ Ereg + A + X Egormaal: S.C
where >1 = 0.1, A2 = >g 0.5. And each term is defined as follows:

---

Multi-view
Albedo
Roughness
Relighting
Env Light
Geureery
Single-view
Albedo
Roughness
Relighting
Env Light
Geometry
PSNR
MSF 1
PSNR
MSF 1
CT L
PSNR f
MSF. 1
PSNR 1
MSF 1
CT)
PhySG
16.233
0.087
21.323
0.054
0.024
PhySG
14.977
0.255
18.504
0.082
0.033
Nv-DiflRec
16.123
0.116
17.418
0.168
0.268
Nv-DiflRec*
14.021
0.165
17.214
0.067
0.050
InvRender
16.984
0.084
22.224
0.067
0.024
InvRender
14.724
0.247
17.998
0.082
0.033
Ours
21.961
0.026
25.486
0.029
0.011
Ours
17.629
0.062
21.374
0.052
0.034
Table 1: (Left) Multi-view inverse rendering on synthetic data. Both model and the baseline
are trained on multi-view images. Our model is significantly better than baseline terms of geometry and PBR texture. (Right) Single-view inverse rendering on synthetic data. While our model is trained on single-view image, the baselines * are trained on 10 multi-view images of the same scene.
The color consistency term Feooor a 1.2 color consistency loss between the rendered color с and the observed color è for all pixel rays: Ecotor = Σn ||c, Crl12-
The normal consistency term Emurmal measures the rendered normal á and a predicted normal n: Enoraal = Er || n. Dr + [Í, fhll- Our monocular predicted normal ń is obtained using
pretrained Omnidata model [13].
The mask consistency term Emask measures the discrepancy between the rendered mask m, and the observed mask ti,, terms of binary cross-entropy (BCE): Lmask - Σ, BCE (m, mī,). Finally, inspired by NeuS [60], incorporate Eikonal regularization to ensure the neural field
is a valid signed distance field: Teikona = Σx S 1)
Ambient occlusion and self-cast shadows pose challenges the accuracy of inverse rendering, as it's difficult to separate them from albedo when optimizing photometric loss. However, with an estimated geometry, we can already obtain a strong visibility cue. Consequently, we utilize ambient occlusion mapping to prebake the visibility map onto the object surfaces obtained from the previous stage. We then minimize the visibility consistency term ensure the rendered visibility e from MLP V aligns with the derived visibility t, miny Σ, BCE(U,, 0z).
Visibility optimization:
Material and illumination optimization: In the final stage, given the obtained surface geometry and the visibility network, we jointly optimize the environmental light and the PBR material network. The overall objective is follows:
min Ecolor En Esmonth M.w.d
A Emetals
where >4 0.01, As 0.1, Ag 0.01. The four terms defined follows:
The color consistency term Exctor minimizes the discrepancy between the rendered color and observed color, akin to the geometry optimization stage. However, use PBR-shaded color in place of the color queried from the auxiliary radiance field.
The sparse regularization Exuaree constrains the latent code of the material network to closely align with a constant target vector p': E KL (pllp'). We set p' = 0.05.
The smooth regularization Esmooth force the BRDF decoder to yield similar value for close la- [| D(z + dz)ll where dz is randomly sampled from normal distribution (0; 0.011.
tent code z: Esmooth
Lastly, inspired by the fact that most common objects are either metallic or not, we introduce metallic regularization Limeaal to encourage the predicted metallic value to be close to either 0 or 1: Tmmetal Σx (1 -m
4
Experiment
In this section, we evaluate the effectiveness of our model on synthetic and real-world datasets, analyze its characteristics, and showcase its applications.
4.1
Experiment setups
Data: Since existing multi-view datasets do not contain duplicate objects, we collect Dup, a novel inverse rendering dataset featuring various duplicate objects. Dup consists of 13 synthetic and 6
7

---

Albedo
Roughness
Env Light
Mesh
Rotation
Translation
Albedo
Roughness
Metallic
Relighting
Num
PSNR
MSE 4
MSE 1
CD 1
degree
length +
Model
PSNR
MSE
MSE
PSNR
6
18.48
0.13
0.116
0.025
2.515
0.070
full model
17.27
0.30
0.02
19.95
8
18.26
0.09
0.172
0.017
0.600
0.003
w/o clcan seg
16.68
0.23
0.13
19.12
10
18.99
0.09
0.204
0.029
0.186
0.003
w/o metal loss
17.09
0.19
0.08
19.02
19.22
0.09
0,063
0.021
0.299
0.005
w/o latent smooth
17.00
0.23
0.02
19.71
30
19.69
0.07
0.119
0.020
0,448
0.006
normal
16.51
0.42
0.97
18.81
50
18.56
0.08
0.153
0.038
0.464
0.008
w/o cik loss
16.79
0.51
0.97
19.31
60
16.45
0.14
0.150
0.091
51.512
0,644
w/o mask
15.14
0.29
0.02
18.11
Table 2: (Left) Performance vs. number of duplicates for color box" dataset. We highlight the
best
second and third values. (Right) Ablation study for the contribution of each loss term.
Rendering
Albedo
Roughness
Relighting
Env Light
Geometry
PSNR
SSIM 1
LPIPS +
PhySG* 20.624
0.641
0,263
PSNR
MSE + 0.096
PSNR f
MSE J 0.045
CD 1
Nv-DiffRec
18.818
0.569
0.282
M-S
20,229
21.328
0.010
InvRender*
20.665
0.639
0.262
S-M
23.448
0.050
24.254
0.052
0.007
Ours
20.326
0.660
0.192
Table 3: (Left) Single-view inverse rendering on real-world data. indicates that the baselines
are trained on multi-view observations. (Right) Multi-view single-object (M-S) vs. single-view multi-object (S-M).
real-world scenes, each comprising 5-10 duplicate objects such as cans, bottles, fire hydrants. etc. For synthetic data, we acquire 3D assets from PolyHaven and utilize Blender Cycles for physics-based rendering. We generate 10-300 images per scene. As for the real-world data, we place the objects in different environments and capture 10-15 images using mobile phone. The data allows for comprehensive evaluation of the benefits of including duplicate objects in the scene for inverse rendering. We refer the readers to the supp. materials for more details.
Metrics: Following prior art [74; 69], we employ Peak Signal-to-Nonn Ratio (PSNR), Structural Similarity (SSIM), and LPIPS [71] to assess the quality of rendered and relit images. For materials, we utilize PSNR to evaluate albedo, and mean-square mean-squared error (MSE) to quantify roughness and environmental lighting. And following[60; 65; 47], we leverage the Chamfer Distance to measure the distance between our estimated geometry and the ground truth.
Baselines: We compare against three state-of-the-ee the--r--ttt-t multi-view inverse rendering approaches: Physg [69], InvRender [74], and NVdiffrec [47]. Physg and InvRender employ implicit representa- tions to describe geometric and material properties, while NVdiffrec utilizes differentiable mesh representation along with UV textures. Additionally, we enhance Physg by equipping it with spatially-varying roughness.
We the Adam optimizer with initial learning rate le-4. All experiments are conducted on a single Nvidia A40 GPU. The first stage takes 20 hours, and the 2nd and 3rd stage takes about hours. Please refer to the supp. material for more details.
Implementation details:
4.2 Experimental results
Single-view inverse rendering: We first evaluate our approach on the single-image multi-object setup. Since the baselines are not designed for this particular setup, randomly select another views, resulting in a total of 10 multi-view images, to train them. As shown Tab. (rright), we able leverage duplicate objects to constrain the underlying geometry, achieving comparable performance to the multi-view baselines. The variations in lighting conditions across instances also aid in better disentangling the effects of lighting from the materials.
Multi-view inverse rendering: Our method can be naturally extended to the multi-view setup, allowing us to validate its effectiveness traditional inverse rendering scenarios. We utilize synthetic data to verify its performance. For each scene, train our model and the baselines using 100 different views and evaluate the quality of the reconstruction results. Similar to previous work, we
hbttss//poyyhwven.oommmdddd

---

N)diftiod477
(6ys
Ins Rawdereer([11]
Ours
Figure 6: The surface normal and rendering result on real-world cola image. Our model has the smoothest surface normal compared with other baselines.
assume the ground truth poses are provided. As shown in Tab. I(left), outperform the baselines all aspects. We conjecture that this improvement stems from our approach explicitly incorporating a material- and geometry-sharing mechanism during the modeling process. As result, we have access to significantly larger number of "effective views" during training compared to the baselines. We show some qualitative results in Fig. 4(left).
Real-world single-view inverse rendering: Up to this point, we have showcased the effectiveness of our approach in various setups using synthetic data. Next, evaluate our model real-world data. Due to the challenge of obtaining highly accurate ground truth for materials and geometry, we focus our comparison solely on the rendering results. As indicated in Tab. 3(Left), our method achieves comparable performance to the multi-view baselines, even when trained using only single view. We visualize some results Fig. 1, Fig. and Fig. 7.
Ablation study for each loss term: As shown in Tab. 2(right), since the metallieness of natural materials are usually binary, incorporating the metallic loss can properly regularize the underlying metallic component and prevent overfitting: Eikinol loss and mask loss help us better constrain the surface and the boundary of the objects, making them more accurate and smooth. Removing either term will significantly affect the reconstructed geometry and hence affect the relighting performance; The pre- trainrd surface normal provides a strong geomeotry prior, allowing us to reduce the ambiguity of sparse view inverse rendering. Removing it degrades the performance on all aspects.
4.3 Analysis
Importance of the number of duplicate objects: Since model utilize duplicate objects as a prior for 3D reasoning, one natural question to ask is: how many duplicate objects do we need? To investigate this, render 9 synthetic scenes with 4~60 duplicates with the same object. We train our model under each setup and report the performance in Tab. 2. As expected, increasing the number of duplicates from 30 improves the accuracy of both material and geometry, since it provides more constraints on the shared object intrinsics. However, the performance decrease after 30 instances due the accumulated 6 DoF pose error brought by heavier occlusion, the limited capacity of the visibility field, and limited image resolution.
The inffuence of different geometry representation: To demonstrate that our method does not rely on a specific neural network architecture, we replace the vanilla MLP using Fourier po- sition encoding to triplane representation from PET-NeuS[61] and hash position encoding from neuralangelo respectively. The result (please refer to supplementary material) shows that the triplane (28.0MB) can recover better geometry and texture than our original model (15.8MB) because the explicit planes provide higher resolution features and can better capture local detail[4]. However, the model of hash positional encoding (1.37GB) and produces high frequency noisy geometry, indicating that it overfits the training view.

---

Diffows
Roughness
RGR
Deffise
Ronghness
Invienneer
Oun
IncReroker
Figure 7: Qualitative comparison on real-world data. InvRender [74] takes as input 10 images, while we only consider one single view. Yet our approach is able to recover the underlying geometry and materials more effectively.
Multi-view single object (M-S) vs. single-view multi-objects (S-M): Is observing an object from multiple views equivalent to observing multiple objects from a single view? Which scenario provides more informative data? To address this question, we first construct a scene containing 10 duplicate objects. Then, we place the same object into the same scene and capture 10 multi-view images. We train our model under both setups. Remarkably, the single-view setting outperforms the multi-view setting in all aspects (see Tab. 3(right)). We conjecture this discrepancy arises from the fact that different instances of the object experience environmental lighting from various angles. Consequently, we are better able disentangle the lighting effects from the material properties in the single-view setup. Fig. (right shows some qualitative results.
Applications: Our approach supports various scene edits. Once we recover the material and geometry of the objects, as well as the illumination of the scene, we can faithfully relight existing objects, edit their materials, and seamlessly insert new objects into the environment as if they were originally present during the image capturing process (see Fig. 1).
Limitations: One major limitation of our approach is that we require the instances in each image to be nearly identical. Our method struggles when there are substantial deformations between different objects, as we adopt geometry/materiil-shhare loosen the sharing constraints and model instance-win wise variations. We explore this possibility in the supplementary material. Furthermore, our model requires accurate instance segmentation masks. Additionally, our approach currently requires decent 6 DoF poses as input and keeps the poses fixed. We found that jointly optimizing the 6 DoF pose and SDF field with BARF [37] will cause the pose error to increase. This because recovering camera pose from sparse views and changing light difficult non-convex problem and prone local minima. We believe the more recent works, like CamP[48] and SPARF[58] could potentially further refine our estimations. Moreover, our model cannot effectively model/constrain the geometry of unseen regions similar to existing neural fields methods.
strategy. One potential way to address this is to
5 Conclusion
We introduce novel inverse rendering approach for single images with duplicate objects. We exploit the repetitive structure to estimate the 6 DoF poses of the objects, and incorporate a geometry and material- matrrial-sharing mechanism to enhance the performance of inverse rendering. Experiments show that our method outperforms baselines, achieving highly detailed and precise reconstructions.
10

---

References
[1] Jonathan T Barron and Jitendra Malik. Shape, illumination, and reflectance from shading. TPAMI, 2014.
[2] Harry Barrow, Tenenbaum, A Hanson, and E Riseman. Recovering intrinsic scene charactcristies. Comput. vis. syst, 1978.
[3] Brent Burley and Walt Disney Animation Studios. Physically-base based shading at disney, Siggraph, 2012. Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremhlay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Fffficient geometry-aware 3D generative adversarial networks. In arXiv, 2021.
[5] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. arXiv preprint arXiv:2303. 13873, 2023,
[6] Weifeng Chen, Zhao Fu, Dawei Yang, and Jia Deng. Single-image depth perception in the wild. NeurlPS, 2016.
[7] Wenzheng Chen, Joey Litalien, Jun Gao, Zian Wang, Clement Fuji Tsang, Sameh Khamis, Or Litany, and Sanja Fidler. Dib-r++: learning to predict lighting and material with hybrid differentiable renderer.
NeurIPS, 2021.
[8] Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexander G Schwing. Mask2former for video instance segmentation. arXiv, 2021.
[9] Paul E Debevec, Camillo Taylor, and Jitendra Malik. Modeling and rendering architecture from photographs: A hybrid geometry-and image-hased approach. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 4655474. 2023.
[10] Yue Dong. Guojun Chen, Pieter Peers, Jiawan Zhang, and Xin Tong. Sppearance-frommmonne
Recovering spatially varying surface reflectance under unknown lighting. ACM Transactions on Graphics (TOG), 33(6):1-12, 2014.
[11] Andreas Doumanoglou, Rigas Kouskouridas, Sotiris Malassiotis, and Tae-Kyun Kim. Recovering 6d object pose and predicting next-best-view in the crowd. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3583 3592, 2016.
[12] Alexei A Efros and Thomas K Leung. Texture synthesis by non-parametric sampling. In ICCV, 1999.
[13] Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir. Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In Proceedings of the IEEE/CVF International
Conference on Computer Vision. pages 10786-10796, 2021.
[14] David Eigen. Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. NeurlPS, 2014.
[15] Michael Elad and Michal Aharon. Image denoising via sparse and redundant representations over learned dictionaries. TIP, 2006.
[16] Qi Fang, Qing Shuai, Junting Dong, Hujun Bao, and Xiaowei Zhou. Reconstructing 3d human pose by watching humans in the mirror. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 128 12844-1822, 2021.
[17] William T Freeman. The generic viewpoint assumption in a framework for visual perception. Nature, 1994.
[18] Daniel Glasner. Shai Bagon, and Michal Irani. Super-resolution from single image. In ICCV, 2009.
[19]
Roger Grosse, Micah K Johnson, Edward H Adelson, and William T Freeman. Ground truth datasct and baseline evaluations for intrinsic image algorithms. In ICCV, 2009.
[20] Richard Hartley and Andrew Zisserman. Multiple view geometry computer vision. Cambridge university press, 2003.
[21] Jon Hasselgren, Nikolai Hofmmann, and Jacob Munkberg. Shape. Light, and Material Decomposition from
Images using Monte Carlo Rendering and Denoising, AxXv:22060033888 2022.
[22] Daniel Hauagge, Scott Wehrwein, Kavita Bala, and Noah Snavely. Photometric ambient occlusion. In
CVPR, 2013.
[23] Berthold KP Horn. Determining lightness from an image. Computer graphics and image processing, 1974.
[24] Berthold KP Horn. Obtaining shape from shading information. The psychology of computer vision, 1975.
[25] Bo Hu. Christopher Brown, and Randal Nelson. Multiple-view 3-d reconstruction using a mirror. 2005.
[26] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed
self-exemplars. In CVPR, 2015.
[27] Jeffrey Ichnowski, Yahav Avigal, Justin Kerr. and Ken Goldberg. Dex-nerf: Using neural radiance field
to grasp transparent objects. arXiv, 2021.
[28] James T Kajiya. The rendering equation. In Proceedings of the 13th annual conference on Computer graphics and interactive techniques, 1986.
[29] Brian Karis and Epic Games. Real shading in unreal engine 4, Proc. Physically Based Shading Theory Practice, 2013.
[30] Pierre-Yves Laffont, Adrien Bousseau, and George Drettakis. Rich intrinsic image decompasition of
outdoor scenes from multiple views. IEEE transactions on visualization and computer graphics. 2012.
[31] Edwin H Land and John J McCann. Lightness and retincx thcory. Josa, 1971.
[32]
Hendrik PA Lensch, Jan Kautz, Michael Goesele, Wolfgang Heidrich, and Hans-Peter Seidel. Image-based reconstruction of spatial appearance and geometric detail. TOG, 2003.
[33] Yikai Li, Jiayuan Mao, Xiuming Zhang, Bill Freeman, Josh Tenenbaum. Noah Snavely, and Jiajun Wu.
Multi- program induction with 3d box priors. NewrIPS, 2020.
11

---

[34]
Yikai Li, Jiayuan Mao, Xiuming Zhang, William T Freeman. Joshua B Tenenbaum, and Jiajun Wu. Perspective plane program induction from single image. In CVPR, 2020.
[35] Zhaoshuo Li. Thomas Müller, Alex Evans, Russell H Taylor, Mathias Unberath, Ming- Yu Liu, and Chen-
Hsuan Lin. Neuralangelo: High-fidelity neural surface reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition ( (CVPR), 2023.
[36] Daniel Lichy. Jiaye Wu, Soumyadip Sengupta, and David Jacobs.
Shape and material capture at home. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages
6123-6133, 2021.
[37] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. Barf: Bundle-adjusting neural
radiance fields. In ICCV, 2021.
[38] Yen-Chen Lin. Pete Florence, Andy Zeng, Jonathan T Barron, Yilun Du Wei-Chiu Ma, Anthony Simeonov, Alberto Rodriguez Garcia, and Phillip Isola. Mira: Mental imagery for robotic affordances. In CoRL, 2023. [39] Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun- Yan Zhu, and Bryan Russell. Editing conditional radiance fields. In ICCV. 2021.
[40] H Christopher Longuet-Higgins, A computer algorithm for reconstructing a scene from two projections. Nature, 1981.
[41] Wei-Chiu Ma, Hang Chu, Bolei Zhou, Raquel Urtasun, and Antonio Torralba. Single image intrinsic decomposition without a single intrinsic image, In ECCV, 2018
[42] Wei-Chiu Ma, Ignacio Tartavull, loan Andrei Bärsan, Shenlong Wang, Min Bai, Gellert Mattyus Namdar Homayounfar, Shrinidhi Kowshika Lakshmikanth, Andrei Pokrovsky, and Raquel Urtasun. Exploiting sparse semantic hd maps for self-driving vehicle localization. In IROS, 2019.
[43]
Stephen Robert Marschner. Inverse rendering for computer graphics. Cornell University, 1998.
[44]
Tomer Michaeli and Michal Trani. Blind deblurring using internal patch recurrence. In ECCV, 2014.
[45]
Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron. Ravi Ramamoorthi, and Ren Ng.
Nerf: Representing scenes as neural radiance fields for view synthesis. ECCV, 2020.
[46]
Chaitanya Mitash, Bowen Wen, Kostas Bekris, and Abdeslam Boularias, Scene-level pose estimation for multiple instances of densely packed objects. In Conference on Robot Learning, pages 1133-1145. PMLR, 2020.
[47] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting triangular 3d models, materials, and lighting from images. In CVPR, 2022. [48] Philipp: Mildenhall Ben; Barron Jonathan T.; Martin-Bruall Brualla Ricardo Park, Keunhong; Henzler. Camp:
Camera preconditioning for neural radiance fields. ACM Trans. Graph., 2023.
[49] René Ranftl, Alexcy Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In ICCV,
2021.
[50] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. TPAMI. 2020. Paul-Edouard Sarlin, Danicl DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning
[51]
feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer
vision and pattern recognition, pages 4938-49477 18884497,222 2020.
[52] Imari Sato, Yoichi Sato, and Katsushi Ikeuchi. Illumination from shadows. IEEE Transactions on Pottern
Analysis and Machine Intelligence, 2003,
[53] Yoichi Sato, Mark D Wheeler, and Katsushi Ikeuchi. Object shape and reflectance modeling from observation. In SIGGRAPH, 1997.
[54] Johannes L Schonberger and Jan-Michael Frahm. Structure-frome frotmmmmooo revisited. In CVPR, 2016.
[55] Johannes Lutz Schönberger and Machhel Frahm. Structure-frot from- motion revisited. In Conference on Computer Vision and Puttern Recognition (CVPR). 2016.
[56] Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Proc. NewrlPS, 2020.
[57] Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron. Nerv: Ncural reflectance and visibility fields for relighting and view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitiinn, pages 7495-7504, 2021.
[58] Prune Truong. Marie-Juli Rakotosaona, Fabian Manhardt, and Federico Tombari. Sparf: Neural radiance fields from sparse and noisy poses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4190-4200, 2023.
[59] Jingkang Wang. Sivabalan Manivasagam, Yun Chen, Ze Yang. Ioan Andrei Bårsan, Anqi Joyce Yang. Wei- Chiu Ma, and Raquel Urtasun. Cadsim: Robust and scalable in-the-wild 3d reconstruction for controllable sensor simulation. In 6th Annual Conference on Robot Learning.
[60] Peng Wang, Lingji Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2 106. 10689, 2021.
[61] Yiqun Wang. Ivan Skorokhodov, and Peter Wonka. Pet-neus: Positional encoding triplanes for neural surfaces. 2023.
[62] Zian Wang, Tianchang Shen. Jun Gao, Shengyu Huang, Jacob Munkberg, Jon Hasselgren, Zan Gojeic, Wenzheng Chen, and Sanja Fidler. Neural fields meet explicit geometric representation for inverse rendering of urban scenes. arXiv. 2023.
[63] Guandao Yang, Serge Belongie, Bharath Hariharan, and Vladlen Koltun. Geometry processing with neural fields. NewrIPS, 2021.
12

---

[64] Ze Yang, Yun Chen. Jingkang Wang. Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel
Urtasun. Unisim: A neural closed-loop sensor simulator. CVPR. 2023.
[65]
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. NeurIPS, 2020. Yizhou Yu, Paul Debevec, Jitendra Malik, and Tim Hawkins. Inverse global illumination: Recovering reflectance models of real scenes from photographs. Proceedings of the 26th annual conference on Computer graphics and interactive techniques, 1999.
[66]
[67]
Albert Zhai and Shenlong Wang. Peanut: Predicting and navigating to unseen targets. arXiv, 2022.
[68]
Kai Zhang, Fujun Luan, Zhengqi Li, and Noah Snavely. Iron: Inverse rendering by optimizing neural sdfs
and materials from photometric images. In CVPR, 2022.
[69]
Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5453-5462, 2021.
[70] Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer
vision and pattern recognition, pages 586 595,, 2018.
[71] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as perceptual metric. IEEE, 2018.
[72] Xiuming Zhang, Pratul Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan Barron. Nerfactor: Neural factorization shape and reflectance under an unknown illumination. ACM Transactions on Graphics (TOG), 40(6);1-18, 2021.
[73] Yuxuan Zhang, Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, Antonio Toralba, and Sanja Fidler. Image gans meet differentiable rendering for inverse graphics and interpretable 3d neural rendering. arXiv, 2020.
[74] Yuanqing Zhang. Jiaming Sun, Xingyi He, Huan Fu, Rongfei Jia, and Xiaowei 7hou. Modeling indirect illumination for inverse rendering. In Proceedings of the IEEE/CVF Conference Computer Vision and
Puttern Recognition. pages 18643-18652, 2022.
[75]
Yunzhi Zhang, Shangzhe Wu, Noah Snavely, and Jiajun Wu. Seeing a rose in five thousand ways, arXiv, 2022.
[76]
Tinghui Zhou, Philipp Krahenbuhl, and Alexei A Efros. Learning data-driven reflectance priors for intrinsic
image decomposition. In ICCV, 2015.
[77] Maria Zontak and Michal Trani, Internal statistics a single natural image. CVPR, IEFF, 2011.
13


Reviews:
[{"review_id": "NlNEgTF2Y7", "reviewer": "Reviewer_vUuZ", "summary": "This paper presents \"Structure from Duplicates\" (SfD), a novel inverse graphics framework introduced to reconstruct the 3D structure, material, and illumination of multiple identical objects from a single image. The key steps include identifying these duplicate objects in an image and estimating their 6DoF pose. Then, the model applies an inverse graphics pipeline to deduce information about the objects' shape, material, and the lighting of the scene, while taking into account that these objects share the same geometry and material properties.", "questions": "- How would the model perform when applied to real-world scenarios with more noise, variability, and less regularity in object structures?\n\n- Could you discuss any strategies for mitigating the potential overfitting that might occur if you increase the capacity of the model to allow for instance-wise variations? \n\n- Could you elaborate on the potential of integrating your approach with BARF.  What kind of improvements or challenges do you expect?", "limitations": "Limitations are discussed above in the weaknesses section. Overall I am on the fence due to the limitation of the proposed approach, which is highly dependent on the similarity of the multi-objects in a single image. ", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "In my opinion, the key strengths of the proposed inverse rendering method are:\n\n- The method outperforms baseline techniques in single image multi-object setup and multi-view inverse rendering scenarios, as shown in the experiments.\n\n- The approach can leverage duplicate objects to constrain underlying geometry and better disentangle the effects of lighting from materials.\n\n- It can be extended to multi-view setups.\n\n- The model makes effective use of duplicate objects, with accuracy improving as the number of duplicates increases.\n\n- It performs well with real-world data, achieving comparable performance to multi-view baselines when trained using only a single view.\n\n- The model supports various scene edits - once the material and geometry of the objects are recovered, along with the scene's illumination, it can faithfully relight existing objects, edit their materials, and insert new objects into the environment.", "weaknesses": "The potential weaknesses seem to be:\n\n- Dependence on Similarity. A significant limitation is that the instances in each image need to be nearly identical; the method struggles when there are substantial deformations between different objects.\n\n- Pose Estimation Errors. The model currently requires accurate 6 DoF poses as input and keeps the poses fixed, which may limit its applicability in certain scenarios.\n\n- Potential Segmentation Errors. SfD begins by identifying multiple instances of an object within an image. This is prone to errors when the segmentation is inaccurate which is more of a problem on glossy and specular objects. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dependence on object similarity**: Over the years, the community has been actively investigating how to harness multi-view information from videos or sparse, extreme-view images, and push forward the frontier of 3D reconstruction and inverse graphics. Our work can be seen as an attempt in such a stride. To our knowledge, this is the first effort to conduct structure from motion from a single image. We agree with the reviewer that the setup may not be as common as others in practice. We, however, note that the problem by itself is an interesting scientific attempt. Furthermore, based on our preliminary experiments, our approach also has the potential to deal with slight variations. Specifically, we test our approach on the `crane` image that [3] provided, where each instance is slightly different. By augmenting our geomtry backbone with a instance-specific deformation field, we are able to reconstruct resonable poses and recover sensible shape and material and high-quality reconstruction (see Figure 1 in rebuttal pdf). We will explore further and include our investigations in the final version. \n\n**Robustness to pose estimation error**: As we have stated in the limitation section, one potential solution is to combine our approach with BARF and optimize the 6 DoF poses jointly with the underlying representations. To validate the conjecture, we test our joint optimization on a real-world image containing 70+ instances (see 2nd row of Figure 3 in pdf). Since each instance only takes a fraction of the pixels, the poses derived from SfM are very noisy. However, with the help of BARF, we are able to significantly reduce the pose error and recover the underlying object intrinsics and environmental extrinsics.\n\n**Robustness to segmentation error**: To understand the effectiveness of the segmentation mask, we compare the performance of our approach using both the ground truth (GT) mask and the estimated mask on a randomly selected scene. As shown in the table below and in the pdf, the results are comparable, indicating that our model is robust to the mask to a certain degree.\n\n \n|                        | Rendering |        |         | Albedo  |        |         | Roughness | Metallic | Relighting |        |           |\n|------------------------|-----------|--------|---------|---------|--------|---------|-----------|----------|------------|--------|-----------|\n|                        | PSNR ↑    | SSIM ↑ | LPIPS ↓ | PSNR ↑  | SSIM ↑ | LPIPS ↓ | MSE ↓     | MSE ↓    | PSNR ↑     | SSIM ↑ | LIPIPS ↓  |\n| full model             | 21.348    | 0.621  | 0.227   | 17.267  | 0.546  | 0.265   | 0.304     | 0.021    | 19.948     | 0.582  | 0.232     |\n| w/o clean segmentation | 20.700    | 0.602  | 0.228   | 16.681  | 0.509  | 0.266   | 0.232     | 0.133    | 19.124     | 0.565  | 0.227     |\n\nTable 1: Robustness test for noisy segmentation.\n"}, {"review_id": "t2TDBE7sPs", "reviewer": "Reviewer_sRgJ", "summary": "The paper presents a method for reconstructing the geometry, material, and illumination of an object using as input an image containing multiple copies of the object. The method leverages the appearance of multiple instances of an object in a single image to essentially create a multi-view supervision signal. The method first segments the different instances of the object in the image, then performs 6DoF pose estimation followed by a structure from motion pipeline to create an \"artificial\" multi-view setup. The object geometry and appearance is modeled with Neural Fields following NeuS. The whole pipeline is supervised using a photometric loss between the renderings and the original image.", "questions": "I would like to see what are some failure cases of the method. I couldn't find any in the main paper or supplementary material.", "limitations": "The paper has a decent discussion on the limitations of the proposed method. I could not identify any immediate potential societal impact of this work.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. I find this paper particularly interesting. I really like the idea of formulating a duality between multiple copies of an object in a single image and multiple views of a single object. It is a very nice example of thinking outside the box.\n2. I enjoyed reading the paper. It is well-written and easy to follow.\n3. The individual components used are properly justified and the engineering behind the method is also really good. It's a lot of individual components (segmentation, pose estimation, SfM, geometry and light representation) stitched together and it requires a lot of effort to make them work.\n4. The qualitative and quantitative results (Tables 2 & 3) are very good.\n\nI would like to see this paper get accepted.\n\n\n", "weaknesses": "1. I can't find any particularly important weakness. One issue is that the problem the paper attempts to solve is not something someone will encounter very often in real-world scenarios, but it's definitely very interesting from a scientific perspective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition! The authors were extremely thrilled when first coming up with the idea of formulating a duality between multiple copies of an object in a single image and multiple views of a single object, and how to leverage it for inverse graphics. We are extremely excited that the reviewer enjoyed reading our paper and shared similar similar fascination as we do. \n\n**Failure cases**: \n- Since our approach is category-agnostic and does not rely on pre-trained priors, our model cannot effectively model/constrain the geometry of unseen regions (similar to existing neural fields methods, *e.g.*, NeRF and NeuS). For instance, the bottom of the fire extinguisher should be hollow in practice. But since the region is not visible in the image, our model will predict a convex shape instead. One potential solution is to pretrain the networks on a large corpus of data/objects to equip the model with priors over the world. This however will  introduce additional assumptions (implicitly) regarding the objects we will be facing. We leave the trade-off for future study. \n- Furthermore, while our approach significantly improves the performance over the baselines, it sometimes still struggles with reconstructing intricate, fine-grained details. We conjecture the limitation stems from the representation power of NeuS[1], which forms basis of our geometry backbone. We, however, note that our pipeline is backbone-agnostic. As the field progresses, our framework can readily incorporate more powerful neural representations, such as Neuralangelo[2]. \n"}, {"review_id": "sfB0TTQKBW", "reviewer": "Reviewer_qVSc", "summary": "This paper tackles the inverse graphics task of predicting the geometry, material and illumination from a single image containing multiple identical objects. The key insight is to leverage the multiple instances depicted in the image to frame this single-view multi-object reconstruction problem into a better constrained multi-view single-object reconstruction problem. Specifically, the proposed approach (dubbed SfD) first identify and crop the instances in the image, then estimate their 6DoF poses using SfM to create a set of calibrated multi-view images. Finally, a geometric reconstruction module based on prior MVS works is optimized using the resulting multi-views. The framework is validated through experiments conducted on a custom dataset dubbed Dup and introduced in this work.", "questions": "- Table 1 is overflowing and appears way too early in the presentation (harms the overall readability)\n- L142: for completeness, I would suggest formulating the changes in the intrinsic matrix when cropping is performed (in the supplementary material)\n- Eq (4) and (5): there is no scaling factors to weight the different loss terms?\n- in general, the paper looks crowed because spaces have been modified; I think the presentation would gain in readability by leveraging the extra page to make it breath a bit more (especially true for the tables)", "limitations": "There is a small limitation section, which I think should be more detailed and illustrated to help the reader better understand the proposed method (see W2)", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. Leveraging duplicated instances to frame a single-view multi-object reconstruction problem into a multi-view single-object one is an interesting idea. Besides, this idea is rather novel; it has only been studied by the concurrent CVPR'23 work of [66]\n\nS2. Experiment section looks strong; the compared methods are relevant and the reported performances are better than prior works\n\nS3. The presentation is very clear and well written. It is easy to walk through the method formulation, which does not contain any major technical flaw. Figures are neat, well designed, and help the understanding.", "weaknesses": "**W1. New problem with low applicability**\n\nAlthough conceptually interesting, this work is tackling a new problem which by essence is very limited in terms of applicability to real-case scenarios, thus questioning its usefulness. Indeed, I see very few cases where the approach developed here could be successful, and quite a lot of failure cases: for example, I think the proposed approach would fail on most of the examples advertised in Figure 2 (see W2 for details).\n\n**W2. Lack of analysis**\n\nThe current presentation lacks some crucial analyses to better understand the proposed approach. In particular, I would expect experiments outlining:\n\n- the contribution of each key component (quantitative ablation study): each of the 6 loss terms, rotation aware data augmentation, etc.\n- its robustness to real data: in particular not exact duplicates like leaves of a tree\n- its robustness to noisy input: an image with duplicates + some other objects (e.g. chairs and tables in Fig. 2) or an image with only one-side instance views (e.g. cars in Fig. 2)\n- its robustness to high number of duplicates (e.g. 100/1000 objects like the pile of screws in Fig. 2)\n- the failure cases in the general setting (e.g., from the video in the supmat, it seems the geometry of the coke can is actually not so great, why?)\n\n**W3. Highly engineered method**\n\nIn addition to the low applicability of the formulated problem, the proposed approach is heavily engineered (6 loss terms, two-level approach including an optimization level with multiple stages, etc). Such amount of engineering typically makes it hard to apply the method to other cases or even to modify and build on top of it to improve the method.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Importance of the objectives**: To better understand the contribution of the loss terms, we start from the full model and subtract each loss respectively. As shown in the table below, removing either component will degrade the performance.\n\n- *Metallic loss*: Since the metallicness of natural materials are usually binary, incorporating the metallic loss can properly regularize the underlying metallic component and prevent overfitting. \n- *Eikinol loss and mask loss*: The two objectives help us better constrain the surface and the boundary of the objects, making them more accurate and smooth. Removing either term will significantly affect the reconstructed geometry and hence affect the relighting performance.\n- *Pretrained normal*: The pre-trained surface normal provides a strong geomeotry prior, allowing us to reduce the ambiguity of sparse view inverse rendering. Removing it degrades the performance on all aspects.\n\n|                        | Rendering |        |         | Albedo  |        |         | Roughness | Metallic | Relighting |        |           |\n|------------------------|-----------|--------|---------|---------|--------|---------|-----------|----------|------------|--------|-----------|\n|                        | PSNR ↑    | SSIM ↑ | LPIPS ↓ | PSNR ↑  | SSIM ↑ | LPIPS ↓ | MSE ↓     | MSE ↓    | PSNR ↑     | SSIM ↑ | LIPIPS ↓  |\n| full model             | 21.348    | 0.621  | 0.227   | 17.267  | 0.546  | 0.265   | 0.304     | 0.021    | 19.948     | 0.582  | 0.232     |\n| w/o binary metal loss  | 21.511    | 0.624  | 0.230   | 17.088  | 0.514  | 0.312   | 0.195     | 0.081    | 19.025     | 0.561  | 0.250     |\n| w/o latent smooth      | 21.313    | 0.620  | 0.245   | 17.001  | 0.543  | 0.284   | 0.230     | 0.021    | 19.710     | 0.575  | 0.245     |\n| w/o pretrained normal  | 20.467    | 0.584  | 0.241   | 16.512  | 0.516  | 0.261   | 0.420     | 0.971    | 18.806     | 0.547  | 0.257     |\n| w/o eik loss           | 20.844    | 0.576  | 0.367   | 16.790  | 0.483  | 0.420   | 0.508     | 0.971    | 19.310     | 0.555  | 0.298     |\n| w/o mask loss          | 19.609    | 0.503  | 0.472   | 15.138  | 0.386  | 0.556   | 0.286     | 0.021    | 18.114     | 0.480  | 0.378     |\n\nTable 2: Ablation study for the contribution of each loss term.  \n\n**Applicability and complexity**: \nOver the years, the community has been actively investigating how to harness multi-view information from videos or sparse, extreme-view images, and push forward the frontier of 3D reconstruction and inverse graphics. Our work can be seen as an attempt in such a stride. To our knowledge, this is the first effort to conduct structure from motion from a single image. We agree with the reviewers that the setup may not be as common as others in practice. We, however, note that the problem by itself is an interesting scientific attempt. Furthermore, based on our preliminary experiments, our approach also has the potential to deal with slight variations. Specifically, we test our approach on the `crane` image that [3] provided, where each instance is slightly different. By augmenting our geomtry backbone with a instance-specific deformation field, we are able to reconstruct resonable poses and recover sensible shape and material (see the pdf). Finally, while our full pipeline consists of multiple components, *why they are used* and *how they are used* are all carefully designed (*e.g.*, enforcing geometry/texture sharing through re-parameterizing the query space). The pipeline is also backbone-agnostic, allowing us to replace individual backbones with latest models. For instance, we can replace the geometry backbone from NeuS to [2] and potentially improve the geometry. We will investigate further and include them in the final version.\n\n\n**Robustness**:\n- *Robustness to non-identical objects*: We applied our method to the crane image provided by [3], where each instance varies slightly. By augmenting our geometry backbone with an instance-specific deformation field, we successfully reconstruct reasonable poses and recover coherent shape and material properties (see the PDF).\n- *Robustness to a high number of objects*: We evaluated our method on an image taken in the real world featuring 70 duplicated objects. The results indicate precise reconstruction and high-quality rendering (see the PDF).\n- *Duplicates + background objects*: Our instance segmentation is used to mask out background objects, enabling the method to concentrate on the objects of interest.\n\n\n**Missing scaling factors in Eq. 4 and Eq. 5**: Thanks for the catch. We will fix them.\n\n**Adjust the formulation of the intrinsic matrix explicitly**: In the main paper, we deliberately omit the changes for the sake of simplicity and readability (as noted in the footnote). Nonetheless, we totally agree with the reviewer that it will be helpful to explicitly formulate the changes in the supplementary material. We will revise it.\n\n**Expanding the limitation section**: Thanks for the suggestion! We will incorporate more comprehensive discussions and analyses (*e.g.*, the ones raised by the reviewers and those addressed during the rebuttal process) in the final version.\n\n\n**Paper formatting**: We will adjust the tables and spacing to make the paper less crowded in the final version.\n\n"}, {"review_id": "C1aEpbCJ0D", "reviewer": "Reviewer_4hmW", "summary": "This paper presents a pipeline for recovering object shape and surface materials from one single of multiple identical duplicate instances. The pipeline first extracts instance masks, registers camera poses using COLMAP (with in-plane rotation augmentation) and recovers shape, materials and environment lighting through inverse rendering.", "questions": "It might be hard to change my opinion on this paper, as most of the technical aspects are clear to me. Rather, this is a slight trivial task, especially given [66] has demonstrated results on a more generic case. Yet, I'm still open to objection.", "limitations": "The paper attributes the challenge of the modeling instance variation to the shape/material model, but the fundamental challenge is rather the pose estimation with COLMAP to begin with. COLMAP generally fails terribly with deformable objects - also the reason that recent deformable NeRFs typically mask out deformable objects running COLMAP to register cameras.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "### S1 - A sound inverse rendering pipeline\n- The proposed pipeline is carefully designed, starting from instance segmentation from Mask2former, to a camera pose estimation step using COLMAP with in-plane rotation augmentation, and finally to inverse rendering of shape, BRDF material and SG env map similar to [65].\n- Given that the instances are mostly identical, this pipeline in general works well.\n\n### S2 - Good writing\n- The paper is well written, with clear descriptions of the problem setup and most technical details.\n", "weaknesses": "### W1 - Quite a trivial task assuming *identical* instances\n- Fig. 1: \"the world is full of identical objects\" - it might be difficult to argue that different instances that appear in one scene are *identical*. This is in contrast to the claim in [66], which says \"no two roses in the natural world are identical\".\n- This paper focuses more on manufactured and synthetic objects, which appear more similar thanks to the consistent manufacturing process, but in reality, different objects often vary in both shape and appearance, due to wearing or natural variations, for instance, multiple apples in a basket.\n- This is the fundamental assumption for COLMAP to work on these instances. It can easily fail even if there is a slight variation in geometry and appearance, which raises significant concern on the method's robustness to instance variations.\n- Once the multiple instances are registered, the rest of the pipeline seems to be a standard inverse rendering task, and is highly inspired from existing work, which I do not find sufficiently interesting.\n\n### W2 - A special case of the rose paper [66]\n- Line 111: \"independently and concurrently, Zhang et al. [66]\" - NeurIPS has a two-month rule for claiming concurrent work, including arXiv papers. [66] appeared on arXiv in Dec 2022, which is almost half a year from the NeurIPS submission deadline. It might be difficult to claim concurrent work.\n- Essential, duplicates are an extreme form of the definition of similarity in [66]. The instance variation is ultimately what makes [66] interesting, as it breaks the assumption for a standard multi-view stereo approach, hence the generative modeling.\n- If the proposed method can handle instance variations well, such as different rose instances, without assuming a prior pose distribution, it will make a much more compelling argument.\n\n### Other comments\n- I suppose the material network is inherited from [65], as the \"latent code\" $\\mathbf{\\rho}$ and its regularizer are the same as [65]. But this is not mentioned in the paper.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The task is NOT trivial**: **We respectfully disagree with the statement that \"the task is trivial.\"** How to make inverse graphics/3D reconstruction more robust and work under more extreme scenarios is a challenging and longstanding problem in computer vision. In this work, we take a step forward by exploring the potential of performing structure from motion and recovering object intrinsics and environmental extrinsics *from a single image without pre-trained priors*. Specifically, we focus on the scenarios where there are multiple (near-)identical objects within the scene. By carefully formulating a duality between multiple copies of an object in a single image and multiple views of a single object, we are able to resolve the ambiguities in 3D and effectively recover the properties of interest. **While the idea may seem straightforward AFTER learned, how to design the low-level details and how to make it work in practice requires careful insights and thoughts (as acknowledged by other reviewers).** We urge the reviewer to acknowledge this.\n\nTo our knowledge, we are the first attempt to conduct structure from motion from a single image. We agree with the reviewers that the setup may not be as common as others in practice. However, it's important to emphasize that the problem, in its essence, is profoundly ill-posed and cannot be resolved without relying on certain priors. While some methods, such as [3], may ease the \"identical\" constraint, they also introduce other strong assumptions (e.g., knowledge of the camera distribution and vast amounts of objects for the generative approach (which we discuss in greater detail below). In contrast, our method chooses a distinct approach, addressing the issue from a purely multi-view geometric perspective. Moreover, our preliminary findings, detailed below, suggest that our methodology can also accommodate minor variations.\n\n**[3] is NOT a superset of our approach**: \nWe agree the setup of the two papers are indeed similar. However, instead of one paper being the superset of the other, the two approaches are **orthogonal**. \n- **Assumptions**: First and foremost, while [3] is able to model the variations among the instances, they impose other strong assumptions such as knowing the camera distribution in advance. The strong camera assumption allows them to sidestep the pose estimation step (*i.e.*, SfM) and focus on modeling the variation. In contrast, we assume no knowledge about the poses and attempt to solve for the full inverse rendering pipeline from the beginning. We thus resort to the (near-)identical instances to recover the exact 6 DoF poses. \n- **Approaches**: Secondly, [3] tackle the task through ***generative modeling***. Since they need to train a generative model *per scene*, their approach is very data-hungry. In contrast, our approach mainly exploits ***multi-view geometry*** to recover the underlying intrinsic and extrinsic properties. By explicitly baking the constraints into the modeling procedure, our approach becomes much more data-efficient. To validate our conjecture, we train [3] on three randomly selected scenes from our dataset, each of which has 10 identical instances. As shown in the pdf, the generative model failed to recover either of them. For comparison, we also test our approach on the `crane` image that [3] provided (the only publicly available data), where each instance is slightly different. By augmenting our geomtry backbone with a instance-specific deformation field, we are able to reconstruct resonable poses and recover sensible shape and material. \n- **Extrinsics**: Thirdly, [3] assume a simple phong shading model and assume a dominant directional light, whereas we parameterize our materials with PBR materials and the lighting with enironmental map, allowing us to model complex real world scenarios more effectively. Finally, it is unclear how to extend [3] to multi-view setup. In contrast, our method is naturally compatible with multi-view observations.\n\n\n**Concurrent to [3]**: We are aware of the NeurIPS policy and we do acknowledge that [3] is on arXiv more than 2 months ago. The two works were developed independently, and as evident from our paper, our design choices were not influenced by [3]. The authors did see the paper and were very excited that other researchers share similar interests and have been working on simlar topics. In our submission, rather than overlooking [3], we carefully discussed the differences between the two studies in the \"related work\" section. We tried our best to put [3] in the light it deserves, despite by the time of submission the paper was not accepted yet and the code was not released. Moreover, now that the code for [3] is available, we are eager to test our method using their dataset and vice versa, aiming for a comprehensive analysis and discussion in the final paper (preliminary results for both can be found in the PDF). Finally, it's crucial to highlight that, while the contexts of our studies are similar, our approaches to addressing the problem are significantly different, as elaborated above.\n\n\n### Reference\n[1] Wang, Y., Skorokhodov, I., & Wonka, P. (2023). PET-NeuS: Positional Encoding Tri-Planes for Neural Surfaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12598-12607).\n[2] Li, Z., Müller, T., Evans, A., Taylor, R. H., Unberath, M., Liu, M. Y., & Lin, C. H. (2023). Neuralangelo: High-Fidelity Neural Surface Reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8456-8465).\n[3] Zhang, Y., Wu, S., Snavely, N., & Wu, J. (2023). Seeing a Rose in Five Thousand Ways. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 962-971)."}, {"review_id": "QgujXjUOar", "reviewer": "Reviewer_vbZQ", "summary": "The proposed method in Structures from Duplicates leverages multiple appearance of an identical objects in a single image to reconstruct the geometry and material properties of this object. Each instance is assigned a virtual camera, such that the shared object representation is aligned in the same space. Therefore multi-view reconstruction can be applied reconstructing each instance in a separate view.\nFirst each object is identified using a segmentation method and cropped from the image. Initial camera poses are estimated using Structure from Motion (SfM) method COLMAP on the cropped images of each segmented object. To avoid the issues that come with extreme differences in the view points for SfM methods, images of instances are rotated and optimal pairs identified.\nFrom estimated view point and image pairs the surface geometry following NeuS, the object's decomposed BRDF and environmental lighting following InvRender are optimized following known inverse rendering approaches.\nResults on synthetic data show comparable quality with multi-view reconstruction approaches, while using duplicates of the same object in a single image and present better performance in the multi-view case, leveraging virtual views from duplicates. Additional results on real data are presented that validate the result on synthetic data. \n", "questions": "Partially addressed in the weakness sections:\n- Were ablation studies performed on all objects (e.g. 2-10 instances per object category), please clarify?", "limitations": "I think the authors adequately addressed all major limitations, such as a required instance segmentation and the reliance on COLMAP poses. Additionally they addressed potential solutions. I would suggest to add a short paragraph on the scope of the paper with respect to the experiments, that I suggested in the weakness section. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors present an interesting solution to a common setting in the real world. The given introduction and related works motivates this setting of multiple instances well, which is an interesting setting by itself, that has not been explored by prior work in a similar way. The method explanation is thorough on all major details. Building on well-known and established decomposed neural rendering architectures and ideas (NeuS, simplified BRDF) is acceptable because this representation is not the main focus of the paper and the method does not lack to my understanding. \n\nPresented results justify most claims made in the beginning and ablation studies validate design choices. Especially showing that explicitly modelling lighting, geometry and material properties has an impact on the reconstruction quality over multi-view reconstruction on a single object is an important insight of the presented work.\n\nOne could argue, that this paper lacks novelty and just applies known concepts, such as instances from repeating object classes in the same images and building a prior from that. In my opinion neglecting a pre-trained prior is a strong advantage of the paper.", "weaknesses": "A major weakness of the presented paper is the variety in the results. The synthetic and real dataset are both on the smaller side and pretty homogeneous, as elaborated in the following:\n\n- While the motivation is convincing and Fig.2 shows a good variety of examples for the presented setting, the presented results in the main paper lack some variety and complexity that can be seen in the presented real world examples like the screws and chairs. The main paper presents only simple, closed object geometries, such as bottles, cans, apples, or the toy plane in the synthetic data with a lack of thin structures and complex textures. On the other side the alarm clock and coffee maker in the supplementary of this work show that the reconstruction can lead to overly smooth geometry on complex structures. This is expected on the limited number of views, but not properly addressed in the main paper.\n\n- Not purely a weakness, but a suggestion to strengthen the claims I would ask the authors to add results on more complex real geometries and scenes. To my best understanding this can be a complex task to capture and reconstruct, but it would have a positive impact on the limited scope provided by cans, bottles and packaging.\n\nAblation studies on the number of objects from 2 to 10 objects on synthetic data show results, in which a higher number of objects significantly improves the results on almost all metrics of the reconstructed object. I would argue that even a higher number could improve results further and with respect to the presented motivation such experiments are not unlikely in the real world. Therefore an upper bound for improvements could further strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Clarification on abaltion studies**: The analysis on the number of instances were performed on a randomly selected scene. To validate our analysis holds for other scenes, we conduct the same experiments on all objects. Due to resource constraints, we only managed to fnish training on two more *randomly selected* objects. The rest is still running in the background. As shown in the table below (Table 3 and Table 4), the performance of our approach scales with respect to the number of instances. We will include the numbers for all objects in the final version.\n\n\n**Overly-smoothed geometry**: We conjecture the limitation stems from the representation power of NeuS [1] which forms basis of our geometry backbone. We, howver, note that our pipeline is backbone-agnostic. We can always replace existing components with the latest and great models. We originally chose NeuS because of its simplicity. We are currently exploring more advanced representations (*e.g.*, [2]) and will include our investigations in the final version.\n\n\n**Expanding the limitation section**: Thanks for the great suggestion. We will include an additional paragraph discussing the scope of the paper in the final version.\n\n \n| Cleaner | Rendering |        |         | Albedo  |        |         | Roughness | Relighting |        |          | Env Light | Geometry |             |          |         |\n|---------|-----------|--------|---------|---------|--------|---------|-----------|------------|--------|----------|-----------|----------|-------------|----------|---------|\n| Num     | PSNR ↑    | SSIM ↑ | LPIPS ↓ | PSNR ↑  | SSIM ↑ | LPIPS ↓ | MSE ↓     | PSNR ↑     | SSIM ↑ | LIPIPS ↓ | MSE ↓     | CD ↓     | Precision ↑ | Recall ↑ | F1 ↑    |\n| 2       | 14.390    | 0.457  | 0.439   | 12.358  | 0.412  | 0.504   | 0.077     | 14.090     | 0.423  | 0.457    | 0.045     | 0.050    | 0.808       | 0.576    | 0.673   |\n| 4       | 15.957    | 0.538  | 0.337   | 13.244  | 0.487  | 0.401   | 0.081     | 15.318     | 0.510  | 0.357    | 0.052     | 0.025    | 0.780       | 0.828    | 0.803   |\n| 6       | 17.260    | 0.630  | 0.290   | 14.052  | 0.551  | 0.354   | 0.100     | 16.542     | 0.608  | 0.297    | 0.048     | 0.029    | 0.744       | 0.762    | 0.753   |\n| 8       | 19.381    | 0.689  | 0.242   | 15.370  | 0.607  | 0.304   | 0.159     | 18.083     | 0.661  | 0.256    | 0.064     | 0.018    | 0.872       | 0.967    | 0.917   |\n\nTabel 3: Experiment result for different number of duplicated objects (Cleaner).\n\n \n| Fire | Rendering |        |         | Albedo  |        |         | Roughness | Relighting |        |          | Env Light | Geometry |             |          |         |\n|------|-----------|--------|---------|---------|--------|---------|-----------|------------|--------|----------|-----------|----------|-------------|----------|---------|\n| Num  | PSNR ↑    | SSIM ↑ | LPIPS ↓ | PSNR ↑  | SSIM ↑ | LPIPS ↓ | MSE ↓     | PSNR ↑     | SSIM ↑ | LIPIPS ↓ | MSE ↓     | CD ↓     | Precision ↑ | Recall ↑ | F1 ↑    |\n| 2    | 16.902    | 0.430  | 0.345   | 13.788  | 0.380  | 0.410   | 0.306     | 16.082     | 0.402  | 0.340    | 0.092     | 0.038    | 0.692       | 0.600    | 0.643   |\n| 4    | 18.157    | 0.531  | 0.271   | 14.799  | 0.461  | 0.325   | 0.311     | 17.224     | 0.500  | 0.272    | 0.080     | 0.029    | 0.713       | 0.781    | 0.745   |\n| 6    | 19.086    | 0.560  | 0.252   | 15.674  | 0.502  | 0.287   | 0.233     | 18.088     | 0.531  | 0.242    | 0.057     | 0.025    | 0.774       | 0.825    | 0.799   |\n| 8    | 21.029    | 0.622  | 0.248   | 16.632  | 0.531  | 0.301   | 0.305     | 19.602     | 0.582  | 0.238    | 0.072     | 0.028    | 0.847       | 0.747    | 0.794   |\n\nTabel 4: Experiment result for different number of duplicated objects (Fire extinguisher).\n\n"}]

Author rebuttal:
We thank the reviewers for their insightful comments and valuable suggestions. We are very excited that the reviewers appreciated the novelty of our approach [`Reviewer sRgJ`, `Reviewer qVSc`], found the idea particularly interesting (*e.g.*, " a nice example of thinking outside the box") [`Reviewer sRgJ`, `Reviewer qVSc`,  `Reviewer vbZQ`],  acknowledged our extensive evaluation and impressive results [`Reviewer vUuZ`, `Reviewer sRgJ`, `Reviewer qVSc`], and enjoyed our presentation [`Reviewer sRgJ`, `Reviewer qVSc`, `Reviewer 4hmW`, `Reviewer vbZQ`]

---
**Novelty and contributions** 

How to make inverse graphics/3D reconstruction more robust and work under more extreme scenarios is a challenging and longstanding problem in computer vision. In this work, we take a step forward by exploring the potential of performing structure from motion and recovering object intrinsics and environmental extrinsics *from a single image without pre-trained priors*. Specifically, we focus on the scenarios where there are multiple (near-)identical objects within the scene. By carefully formulating a duality between multiple copies of an object in a single image and multiple views of a single object, we are able to resolve the ambiguities in 3D and effectively recover the properties of interest. 

Over the years, the community has been actively investigating how to harness multi-view information from videos or sparse, extreme-view images, and push forward the frontier of 3D reconstruction and inverse graphics. Our work can be seen as an attempt in such a stride. To our knowledge, this is the first effort to conduct structure from motion from a single image. We agree with the reviewers that the setup may not be as common as others in practice. We, however, note that the problem by itself is an interesting scientific attempt (as acknowledged by several reviewers). Furthermore, based on our preliminary experiments, our approach also has the potential to deal with slight variations. Specifically, we test our approach on the `crane` image that [3] provided, where each instance is slightly different. By augmenting our geomtry backbone with a instance-specific deformation field, we are able to reconstruct resonable poses and recover sensible shape and material. We hope it can shed light on future research along similar directions, such as handling articulate objects or objects with large deformation. 

Finally, while our full pipeline is inspired by multiple exisiting components, *why they are used* and *how they are used* are all carefully designed (*e.g.*, enforcing geometry/texture sharing through re-parameterizing the query space). It is neither a simple extension nor a naive composition. It is also far from trivial. Exploiting existing algorithms to realize a novel idea does not mean there is no technical contribution. We hope the reviewers, in particular `Reviewer 4hmW`, can acknowledge this.


---
**Additional experiments and visualizations**

Based on reviewers' request, we have included new experimental results in the attached pdf and outlined them below. For instance, we provide a more comprehensive analysis of the robustness of our approach (w.r.t camera noise, segmentation error, and the amount of instances). We also present an additional comparison with prior art [3] and incorporate more detailed ablation studies. We thank the reviewers for the great suggestions and we strongly encourage the reviewers to take a look at the additional analyses.

---
We now address the concerns of each reviewer individually.


Meta-review: This paper initially received divergent reviews and garnered extensive discussion during the rebuttal period.  On the positive side, reviewers appreciated the clear presentation, the technical soundness of the proposed method, and the strong experimental validation of the claims made in the paper.  On the negative side, reviewers were concerned with the narrowness of the task definition (reconstruction of scenes with many identical instances of the same object) and the separation from recent work (Zhang et al. CVPR 2022 [65] and Zhang et al. CVPR 2023 [66]).

During the discussion period, the authors responded with a number of clarifications and additional results.  Most of the reviewers engaged in the discussion and updated their final opinions with concrete rationales.  After the dust has settled, two reviewers remained negative due to the above two concerns, and three reviewers indicated they remain positive and would like to see the work accepted.

The AC finds that: 1) the paper and the additional clarifications during the discussion period elaborate concretely differentiations relative to the relevant recent work; and 2) the remaining concern regarding narrowness of the task definition and potential limited applicability of the contribution are not sufficient to merit rejection of the paper.  Given that the work is technically sound, communicated clearly, and within scope of the venue, it has the potential for positive impact and that is something that will be organically determined by the community in the longer term rather than by reviewer opinion in this review cycle.  The AC therefore recommends acceptance and strongly encourages the authors to incorporate the clarifications and additional details provided during the discussion period into their manuscript.
Decision: Accept (poster)