Title: Structure from Duplicates: Neural Inverse Graphics from a Pile of Objects
Venue: NeurIPS 2023
Accepted: yes
Keywords: 3d reconstruction

inverse rendering

pose estimation

single view reconstruction

nerf

duplicates
Abstract: Abstract Our world is full of identical objects (\emph{e.g.}, cans of coke, cars of same model). These duplicates, when seen together, provide additional and strong cues for us to effectively reason about 3D. Inspired by this observation, we introduce Structure from Duplicates (SfD), a novel inverse graphics framework that reconstructs geometry, material, and illumination from a single image containing multiple identical objects. SfD begins by identifying multiple instances of an object within an image, and then jointly estimates the 6DoF pose for all instances. An inverse graphics pipeline is subsequently employed to jointly reason about the shape, material of the object, and the environment light, while adhering to the shared geometry and material constraint across instances.
Our primary contributions involve utilizing object duplicates as a robust prior for single-image inverse graphics and proposing an in-plane rotation-robust Structure from Motion (SfM) formulation for joint 6-DoF object pose estimation. By leveraging multi-view cues from a single image, SfD generates more realistic and detailed 3D reconstructions, significantly outperforming existing single image reconstruction models and multi-view reconstruction approaches with a similar or greater number of observations.

Structure from Duplicates:

Abstract

Our world is full of identical objects ( e.g. , cans of coke, cars of same model). These duplicates, when seen together, provide additional and strong cues for us to effectively reason about 3D. Inspired by this observation, we introduce Structure from Duplicates (SfD), a novel inverse graphics framework that reconstructs geometry, material, and illumination from a single image containing multiple identical objects. SfD begins by identifying multiple instances of an object within an image, and then jointly estimates the 6DoF pose for all instances. An inverse graphics pipeline is subsequently employed to jointly reason about the shape, material of the object, and the environment light, while adhering to the shared geometry and material constraint across instances. Our primary contributions involve utilizing object duplicates as a robust prior for single-image inverse graphics and proposing an in-plane rotation-robust Structure from Motion (SfM) formulation for joint 6-DoF object pose estimation. By leveraging multi-view cues from a single image, SfD generates more realistic and detailed 3D reconstructions, significantly outperforming existing single image reconstruction models and multi-view reconstruction approaches with a similar or greater number of observations. Code available at https://github.com/Tianhang-Cheng/SfD

1 Introduction

Given a single/set of image(s), the goal of inverse rendering is to recover the underlying geometry, material, and lighting of the scene. The task is of paramount interest to many applications in computer vision, graphics, and robotics and has drawn extensive attention across the communities over the past few years[74; 47; 21; 45].

lMilling

fiuedu

Since the problem is ill-posed, prevailing inverse rendering approaches often leverage multi-view observations to constrain the solution space. While these methods have achieved state-of-the-art performance, in practice, it is sometimes difficult, or even impossible, to obtain those densely captured images. To overcome the reliance on multi-view information, researchers have sought to incorporate various structural priors, either data-driven or handcrafted, into the models[5; 36]. By utilizing the regularizations, these approaches are able to approximate the intrinsic properties ( e.g. , material) and extrinsic factors ( e.g. , illumination) even from one single image. Unfortunately, the estimations may be biased due to the priors imposed. This makes one ponder: is it possible that we take the best of both worlds? Can we extract multi-view information from a single image under certain circumstances?

Fortunately the answer is yes. Our world is full of repetitive objects and structures. Repetitive patterns in single images can help us extract and utilize multi-view information. For instance, when we enter an auditorium, we often see many identical chairs facing slightly different directions. Similarly, when we go to a supermarket, we may observe multiple nearly-identical apples piled on the fruit stand. Although we may not see the exact same object from multiple viewpoints in just one glance, we do see many of the 'identical twins' from various angles, which is equivalent to multi-view observations and even more (see Sec. 3 for more details). Therefore, the goal of this paper is to develop a computational model that can effectively infer the underlying 3D representations from a single image by harnessing the repetitive structures of the world.

With these motivations in mind, we present Structure from Duplicates (SfD), a novel inverse rendering model that is capable of recovering high-quality geometry, material, and lighting of the objects from a single image. SfD builds upon insights from structure from motion (SfM) as well as recent advances on neural fields. At its core lies two key modules: (i) a in-plane rotation robust pose estimation module , and (ii) a geometric reconstruction module . Given an image of a scene with duplicate objects, we exploit the pose estimation module to estimate the relative 6 DoF poses of the objects. Then, based on the estimated poses, we align the objects and create multiple 'virtual cameras.' This allows us to effectively map the problem from a single-view multi-object setup to a multi-view single-object setting (see Fig. 3). Finally, once we obtain multi-view observations, we can leverage the geometric module to recover the underlying intrinsic and extrinsic properties of the scene. Importantly, SfD can be easily extended to multi-image setup. It can also be seen as a superset of existing NeRF models, where the model will reduce to NeRF when there is only one single object in the scene.

We validate the efficacy of our model on a new dataset called Dup , which contains synthetic and real-world samples of duplicated objects since current multi-view datasets lack duplication samples. This allows us to benchmark inverse rendering performance under single-view or multi-view settings. Following previous work [60; 65; 47; 69; 74], we evaluate rendering, relighting and texture quality with MSE, PSNR, SSIM, LPIPS [70], geometry with Chamfer Distance (CD), and environment light with MSE.

Our contributions are as follows: 1) We proposed a novel setting called "single-view duplicate objects" (S-M), which expands the scope of the inverse graphics family with a multi-view singleinstance (M-S) framework. 2) Our method produces more realistic material texture than the existing multi-view inverse rendering model when using the same number of training views. 3) Even only relying on a single-view input, our approach can still recover comparable or superior materials and geometry compared to baselines that utilize multi-view images for supervision.

Multi-view

Structurefrommotion

Geometry

patches

PBRMaterial

Illumination

compare

Segmentation

Fromsingleviewto

multi-view

aC

Rendering

2 Related Work

Inverse rendering: The task of inverse rendering can be dated back to more than half a century ago [31; 23; 24; 2]. The goal is to factorize the appearance of an object or a scene in the observed image(s) into underlying geometry, material properties, and lighting conditions [53; 43; 66]. Since the problem is severely under-constrained, previous work mainly focused on controlled settings or simplifications of the problem [19; 22; 1]. For instance, they either assume the reflectance of an object is spatially invariant [69], presume the lighting conditions are known [57], assume the materials are lambertian [76; 41], or presume the proxy geometry is available [52; 32; 10; 30]. More recently, with the help of machine learning, in particular deep learning, researchers have gradually moved towards more challenging in-the-wild settings [73; 47]. By pre-training on a large amount of synthetic yet realistic data [36] or baking inductive biases into the modeling pipeline [7; 72], these approaches can better tackle unconstrained real-world scenarios ( e.g. , unknown lighting conditions) and recover the underly physical properties more effectively [47; 68]. For example, through properly modeling the indirect illumination and the visibility of direct illumination, Zhang et al. [74] is able to recover interreflection- and shadow-free SVBRDF materials ( e.g. , albedo, roughness). Through disentangling complex geometry and materials from lighting effects, Wang et al. [62] can faithfully relight and manipulate a large outdoor urban scene. Our work builds upon recent advances in neural inverse rendering. Yet instead of grounding the underlying physical properties through multi-view observations as in prior work, we focus on the single image setup and capitalize on the duplicate objects in the scene for regularization. The repetitive structure not only allows us to ground the geometry, but also provide additional cues on higher-order lighting effects ( e.g. , cast shadows). As we will show in the experimental section, we can recover the geometry, materials, and lighting much more effectively even when comparing to multi-view observations.

3D Reconstruction: Recovering the spatial layout of the cameras and the geometry of the scene from a single or a collection of images is a longstanding challenge in computer vision. It is also the cornerstone for various downstream applications in computer graphics and robotics such as inverse rendering [43; 60; 62], 3D editing [39; 63], navigation [42; 67], and robot manipulation [27; 38]. Prevailing 3D reconstruction systems, such as structure from motion (SfM), primarily rely on multi-view geometry to estimate the 3D structure of a scene [40; 20; 54]. While achieving significant successes, they rely on densely captured images, limiting their flexibility and practical use cases. Single image 3D reconstruction, on the other hand, aims to recover metric 3D information from a monocular image [14; 6; 50; 49]. Since the problem is inherently ill-posed and lacks the ability to leverage multi-view geometry for regularization, these methods have to resort to (learned) structural priors to resolve the ambiguities. While they offer greater flexibility, their estimations may inherit biases from the training data. In this paper, we demonstrate that, under certain conditions, it is possible to incorporate multi-view geometry into a single image reconstruction system. Specifically, we leverage repetitive objects within the scene to anchor the underlying 3D structure. By treating

each of these duplicates as an observation from different viewpoints, we can achieve highly accurate metric 3D reconstruction from a single image.

Repetitions: Repetitive structures and patterns are ubiquitous in natural images. They play important roles in addressing numerous computer vision problems. For instance, a single natural image often contains substantial redundant patches [77]. The recurrence of small image patches allows one to learn a powerful prior which can later be utilized for various tasks such as superresolution [18; 26], image deblurring [44], image denoising [15], and texture synthesis [12]. Moving beyond patches, repetitive primitives or objects within the scene also provide informative cues about their intrinsic properties [33; 64]. By sharing or regularizing their underlying representation, one can more effectively constrain and reconstruct their 3D geometry [25; 16; 9], as well as enable various powerful image/shape manipulation operations [34; 59]. In this work, we further push the boundary and attempt to recover not just the geometry, but also the materials ( e.g. , albedo, roughness), visibilities, and lighting conditions of the objects. Perhaps closest to our work is [75]. Developed independently and concurrently, Zhang et al. build a generative model that aims to capture object intrinsics from a single image with multiple similar/same instances. However, there exist several key differences: 1) we explicitly recover metric-accurate camera poses using multi-geometry, whereas Zhang et al. learn this indirectly through a GAN-loss; 2) we parameterize and reason realistic PBR material and environmental light; 3) we handle arbitrary poses, instead of needing to incorporate a prior pose distribution. Some previous work also tried to recover 6D object pose from crowded scene or densely packed objects[46; 11] from RGB-D input, but our model only require RGB input.

3 Structure from Duplicates

In this paper, we seek to devise a method that can precisely reconstruct the geometry, material properties, and lighting conditions of an object from a single image containing duplicates of it . We build our model based on the observation that repetitive objects in the scene often have different poses and interact with the environment ( e.g. , illumination) differently. This allows one to extract rich multi-view information even from one single view and enables one to recover the underlying physical properties of the objects effectively.

We start by introducing a method for extracting the 'multi-view' information from duplicate objects. Then we discuss how to exploit recent advances in neural inverse rendering to disentangle both the object intrinsics and environment extrinsics from the appearance. Finally, we describe our learning procedure and design choices.

3.1 Collaborative 6-DoF pose estimation

As we have alluded to above, a single image with multiple duplicate objects contains rich multiview information. It can help us ground the underlying geometry and materials of the objects, and understand the lighting condition of the scene.

Our key insight is that the image can be seen as a collection of multi-view images stitching together . By cropping out each object, we can essentially transform the single image into a set of multi-view images of the object from various viewpoints. One can then leverage structure from motion (SfM) [55] to estimate the relative poses among the multi-view images, thereby aggregating the information needed for inverse rendering. Notably, the estimated camera poses can be inverted to recover the 6 DoF poses of the duplicate objects. As we will elaborate in Sec. 3.2, this empowers us to more effectively model the extrinsic lighting effect (which mainly depends on world coordinate) as well as to properly position objects in perspective and reconstruct the exact same scene.

To be more formal, let I ∈ R H × W × 3 be an image with N duplicate objects. Let {I obj i } N i =1 ∈ R w × h × 3 be the corresponding object image patches. We first leverage SfM [55; 51] to estimate the camera poses of the multi-view cropped images * { ξ i ∈ SE (3) } N i =1 :

Next, since there is only one real camera in practice, we can simply align the N virtual cameras { ξ i } N i =1 to obtain the 6 DoF poses of the duplicate objects. Without loss of generality and for

simplicity, we align all the cameras to a reference coordinate ξ ref . The 6 DoF poses of the duplicate objects thus become ξ obj i = ξ ref ◦ ( ξ cam i ) -1 , where ◦ is matrix multiplication for pose composition.

In practice, we first employ a state-of-the-art panoptic segmentation model [8] to segment all objects in the scene. Then we fit a predefined bounding box to each object and crop it. Lastly, we run COLMAP [55] to estimate the 6 DoF virtual camera poses, which in turn provides us with the 6 DoF object poses. Fig. 3(left) depicts our collaborative 6-DoF pose estimation process.

Caveats of random object poses: Unfortunately, naively feeding these object patches into SfM would often leads to failure, as little correspondence can be found. This is due to the fact that state-of-the-art correspondence estimators [51] are trained on Internet vision data, where objects are primarily upright. The poses of the duplicate objects in our case, however, vary significantly. Moreover, the objects are often viewed from accidental viewpoints [17]. Existing estimators thus struggle to match effectively across such extreme views.

Rotation-aware data augmentation: Fortunately, the scene contains numerous duplicate objects. While estimating correspondences reliably across arbitrary instances may not always be possible, there are certain objects whose viewpoints become significantly similar after in-plane rotation. Hence, we have developed an in-plane rotation-aware data augmentation for correspondence estimation.

Specifically, when estimating correspondences between a pair of images, we don't match the images directly. Instead, we gradually rotate one image and then perform the match. The number of correspondences at each rotation is recorded and then smoothed using a running average. We take the argmax to determine the optimal rotation angle. Finally, we rotate the correspondences from the best match back to the original pixel coordinates. As we will demonstrate in Sec. 4, this straightforward data augmentation strategy significantly improves the accuracy of 6 DoF pose estimation. In practice, we rotate the image by 4 ◦ per step. All the rotated images are batched together, enabling us to match the image pairs in a single forward pass.

3.2 Joint shape, material, and illumination estimation

Suppose now we have the 6 DoF poses of the objects { ξ obj i } N i =1 . The next step is to aggregate the information across duplicate objects to recover the intrinsics properties of the objects ( e.g. , geometry, materials) and the extrinsic factors of the world ( e.g. , illumination). We aim to reproduce these attributes as faithfully as possible, so that the resulting estimations can be utilized for downstream tasks such as relighting and material editing. Since the task is under-constrained and joint estimation often leads to suboptimal results, we follow prior art [74; 62] and adopt a stage-wise procedure.

Geometry reconstruction: We start by reconstructing the geometry of the objects. In line with NeuS [60], we represent the object surfaces as the zero level set of a signed distance function (SDF).

We parameterize the SDF with a multi-layer perceptron (MLP) S : x obj ↦→ s that maps a 3D point under object coordinate x obj ∈ R 3 to a signed distance value s ∈ R . Different from NeuS [60], we model the geometry of objects in local object space. This allows us to guarantee shape consistency across instances by design.

We can also obtain the surface normal by taking the gradient of the SDF: n ( x obj ) = ∇ x obj S . To learn the geometry from multi-view images, we additionally adopt an auxiliary appearance MLP C : { x , x obj , n , n obj , d , d obj } ↦→ c that takes as input a 3D point x , x obj , surface normal n , n obj , and view direction d , d obj under both coordinate systems and outputs the color c ∈ R 3 . The input from world coordinate system helps the MLP to handle the appearance inconsistencies across instances caused by lighting or occlusion. We tied the weights of the early layers of C to those of S so that the gradient from color can be propagated to geometry. We determine which object coordinate to use based on the object the ray hits. This information can be derived either from the instance segmentation or another allocation MLP A : { x , d } ↦→ q (distilled from SDF MLP), where q ∈ N is the instance index. After we obtain the geometry MLP S , we discard the auxiliary appearance MLP C . As we will discuss in the next paragraph, we model the object appearance using physics-based rendering (PBR) materials so that it can handle complex real-world lighting scenarios.

Material and illumination model: Now we have recovered the geometry of the objects, the next step is to estimate the environment light of the scene as well as the material of the object. We assume

GT

Zhang

[74]

Ours

Diffuse

Relighting

GT

M-S

S-M.

Diffuse

Relighting

all lights come from an infinitely faraway sphere and only consider direct illumination. Therefore, the illumination emitted to a 3D point in a certain direction is determined solely by the incident light direction w i and is independent of the point's position. Similar to [69; 74], we approximate the environment light with M = 128 Spherical Gaussians (SGs):

where λ ∈ R + is the lobe sharpness, µ is the lobe amplitude, and ϕ is the lobe axis. This allows us to effectively represent the illumination and compute the rendering equation (Eq. 3) in closed-form.

As for object material, we adopt the simplified Disney BRDF formulation [3; 29] and parameterized it as a MLP M : x obj ↦→{ a , r, m } . Here, a ∈ R 3 denotes albedo, r ∈ [0 , 1] corresponds to roughness, and m ∈ [0 , 1] signifies metallic. Additionally, inspired by InvRender [74], we incorporate a visibility MLP V : ( x , w i ) ↦→ v ∈ [0 , 1] to approximate visibility for each world coordinate faster reference. The difference is that we use the sine activation function[56] rather than the ReLU activation for finer detail. Since the visibility field is in world space, we can model both inter-object self-casted shadows and inter-object occlusions for multiple instances in our setup. We query only surface points, which can be derived from the geometry MLP S using volume rendering. The material MLP M also operates in object coordinates like S , which ensure material consistency across all instances. Moreover, the variations in lighting conditions between instances help us better disentangle the effects of lighting from the materials. We set the dielectrics Fresnel term to F 0 = 0 . 02 and the general Fresnel term to F = (1 -m ) F 0 + m a to be compatible with both metals and dielectrics.

Combining all of these components, we can generate high-quality images by integrating the visible incident lights from hemisphere and modeling the effects of BRDF [28]:

Here, w i is the incident light direction, while w o is the viewing direction. The BRDF function f r can be derived from our PBR materials. We determine the visibility of an incident light either through sphere tracing or following Zhang et al. [74] to approximate it with the visibility MLP V . We refer the readers to the supplementary materials for more details.

3.3 Optimization

Optimizing shape, illumination, and material jointly from scratch is challenging. Taking inspiration from previous successful approaches [74; 69], we implement a multi-stage optimization pipeline. We progressively optimize the geometry first, then visibility, and finally, material and illumination.

Geometry optimization: We optimize the geometry model by minimizing the difference between rendered cues and observed cues.

where λ 1 = 0 . 1 , λ 2 = λ 3 = 0 . 5 . And each term is defined as follows:

Table 1: (Left) Multi-view inverse rendering on synthetic data . Both our model and the baseline are trained on multi-view images. Our model is significantly better than baseline in terms of geometry and PBR texture. (Right) Single-view inverse rendering on synthetic data. While our model is trained on a single-view image , the baselines ∗ are trained on 10 multi-view images of the same scene.

The color consistency term E color is a L2 color consistency loss between the rendered color c and the observed color ˆ c for all pixel rays: E color = ∑ r ∥ c r -c r ∥ 2 .

The normal consistency term E normal measures the rendered normal ˆ n and a predicted normal ˆ n : E normal = ∑ r ∥ 1 -ˆ n T r n r ∥ 1 + ∥ ˆ n r -n r ∥ 1 . Our monocular predicted normal ˆ n is obtained using a pretrained Omnidata model [13].

The mask consistency term E mask measures the discrepancy between the rendered mask m r and the observed mask ˆ m r , in terms of binary cross-entropy (BCE): L mask = ∑ r BCE ( m r , ˆ m r ) .

Finally, inspired by NeuS [60], we incorporate an Eikonal regularization to ensure the neural field is a valid signed distance field: L eikonal = ∑ x ( ∥∇ x S ∥ 2 -1) 2 ,

Visibility optimization: Ambient occlusion and self-cast shadows pose challenges to the accuracy of inverse rendering, as it's difficult to separate them from albedo when optimizing photometric loss. However, with an estimated geometry, we can already obtain a strong visibility cue. Consequently, we utilize ambient occlusion mapping to prebake the visibility map onto the object surfaces obtained from the previous stage. We then minimize the visibility consistency term to ensure the rendered visibility v r from MLP V aligns with the derived visibility ˆ v r : min V ∑ r BCE ( v r , ˆ v r ) .

Material and illumination optimization: In the final stage, given the obtained surface geometry and the visibility network, we jointly optimize the environmental light and the PBR material network. The overall objective is as follows:

where λ 4 = 0 . 01 , λ 5 = 0 . 1 , λ 6 = 0 . 01 . The four terms are defined as follows:

The color consistency term E color minimizes the discrepancy between the rendered color and observed color, akin to the geometry optimization stage. However, we use PBR-shaded color in place of the color queried from the auxiliary radiance field.

The sparse regularization E sparse constrains the latent code ρ of the material network to closely align with a constant target vector ρ ′ : E latent = KL ( ρ || ρ ′ ) . We set ρ ′ = 0 . 05 .

The smooth regularization E smooth force the BRDF decoder to yield similar value for close latent code z : E smooth = ∥ D ( z ) -D ( z + dz ) ∥ 1 , where dz is randomly sampled from normal distribution N (0; 0 . 01) .

Lastly, inspired by the fact that most common objects are either metallic or not, we introduce a metallic regularization L metal to encourage the predicted metallic value to be close to either 0 or 1: L metal = ∑ r m r (1 -m r ) .

4 Experiment

In this section, we evaluate the effectiveness of our model on synthetic and real-world datasets, analyze its characteristics, and showcase its applications.

4.1 Experiment setups

Data: Since existing multi-view datasets do not contain duplicate objects, we collect Dup , a novel inverse rendering dataset featuring various duplicate objects. Dup consists of 13 synthetic and 6

real-world scenes, each comprising 5-10 duplicate objects such as cans, bottles, fire hydrants, etc. For synthetic data, we acquire 3D assets from PolyHaven † and utilize Blender Cycles for physics-based rendering. We generate 10-300 images per scene. As for the real-world data, we place the objects in different environments and capture 10-15 images using a mobile phone. The data allows for a comprehensive evaluation of the benefits of including duplicate objects in the scene for inverse rendering. We refer the readers to the supp. materials for more details.

Metrics: Following prior art [74; 69], we employ Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and LPIPS [71] to assess the quality of rendered and relit images. For materials, we utilize PSNR to evaluate albedo, and mean-squared error (MSE) to quantify roughness and environmental lighting. And following[60; 65; 47], we leverage the Chamfer Distance to measure the distance between our estimated geometry and the ground truth.

Baselines: We compare against three state-of-the-art multi-view inverse rendering approaches: Physg [69], InvRender [74], and NVdiffrec [47]. Physg and InvRender employ implicit representations to describe geometric and material properties, while NVdiffrec utilizes a differentiable mesh representation along with UV textures. Additionally, we enhance Physg by equipping it with a spatially-varying roughness.

Implementation details: We use the Adam optimizer with an initial learning rate 1e-4. All experiments are conducted on a single Nvidia A40 GPU. The first stage takes 20 hours, and the 2nd and 3rd stage takes about 2 hours. Please refer to the supp. material for more details.

4.2 Experimental results

Single-view inverse rendering: We first evaluate our approach on the single-image multi-object setup. Since the baselines are not designed for this particular setup, we randomly select another 9 views, resulting in a total of 10 multi-view images, to train them. As shown in Tab. 1(right), we are able to leverage duplicate objects to constrain the underlying geometry, achieving comparable performance to the multi-view baselines. The variations in lighting conditions across instances also aid us in better disentangling the effects of lighting from the materials.

Multi-view inverse rendering: Our method can be naturally extended to the multi-view setup, allowing us to validate its effectiveness in traditional inverse rendering scenarios. We utilize synthetic data to verify its performance. For each scene, we train our model and the baselines using 100 different views and evaluate the quality of the reconstruction results. Similar to previous work, we

Physg[69]

InvRender[74]

Ours

assume the ground truth poses are provided. As shown in Tab. 1(left), we outperform the baselines on all aspects. We conjecture that this improvement stems from our approach explicitly incorporating a material- and geometry-sharing mechanism during the modeling process. As a result, we have access to a significantly larger number of "effective views" during training compared to the baselines. We show some qualitative results in Fig. 4(left).

Real-world single-view inverse rendering: Up to this point, we have showcased the effectiveness of our approach in various setups using synthetic data. Next, we evaluate our model on real-world data. Due to the challenge of obtaining highly accurate ground truth for materials and geometry, we focus our comparison solely on the rendering results. As indicated in Tab. 3(Left), our method achieves comparable performance to the multi-view baselines, even when trained using only a single view. We visualize some results in Fig. 1, Fig. 6 and Fig. 7.

Ablation study for each loss term: As shown in Tab. 2(right), since the metallicness of natural materials are usually binary, incorporating the metallic loss can properly regularize the underlying metallic component and prevent overfitting; Eikinol loss and mask loss help us better constrain the surface and the boundary of the objects, making them more accurate and smooth. Removing either term will significantly affect the reconstructed geometry and hence affect the relighting performance; The pre-trained surface normal provides a strong geomeotry prior, allowing us to reduce the ambiguity of sparse view inverse rendering. Removing it degrades the performance on all aspects.

4.3 Analysis

Importance of the number of duplicate objects: Since our model utilize duplicate objects as a prior for 3D reasoning, one natural question to ask is: how many duplicate objects do we need? To investigate this, we render 9 synthetic scenes with 4 ∼ 60 duplicates with the same object. We train our model under each setup and report the performance in Tab. 2. As expected, increasing the number of duplicates from 6 to 30 improves the accuracy of both material and geometry, since it provides more constraints on the shared object intrinsics. However, the performance decrease after 30 instances due to the accumulated 6 DoF pose error brought by heavier occlusion, the limited capacity of the visibility field, and limited image resolution.

The influence of different geometry representation: To demonstrate that our method does not rely on a specific neural network architecture, we replace the vanilla MLP using Fourier position encoding to triplane representation from PET-NeuS[61] and hash position encoding from neuralangelo[35] respectively. The result (please refer to supplementary material) shows that the triplane (28.0MB) can recover better geometry and texture than our original model (15.8MB) because the explicit planes provide higher resolution features and can better capture local detail[4]. However, the model of hash positional encoding (1.37GB) and produces high frequency noisy in geometry, indicating that it overfits the training view.

RGB

Diffuse

Normal

Roughness

RGB

Diffuse

Normal

Roughness

InvRender

Ours

InvRender

Ours

EST

Multi-view single object (M-S) vs. single-view multi-objects (S-M): Is observing an object from multiple views equivalent to observing multiple objects from a single view? Which scenario provides more informative data? To address this question, we first construct a scene containing 10 duplicate objects. Then, we place the same object into the same scene and capture 10 multi-view images. We train our model under both setups. Remarkably, the single-view setting outperforms the multi-view setting in all aspects (see Tab. 3(right)). We conjecture this discrepancy arises from the fact that different instances of the object experience environmental lighting from various angles. Consequently, we are better able to disentangle the lighting effects from the material properties in the single-view setup. Fig. 5(right) shows some qualitative results.

Applications: Our approach supports various scene edits. Once we recover the material and geometry of the objects, as well as the illumination of the scene, we can faithfully relight existing objects, edit their materials, and seamlessly insert new objects into the environment as if they were originally present during the image capturing process (see Fig. 1).

Limitations: One major limitation of our approach is that we require the instances in each image to be nearly identical. Our method struggles when there are substantial deformations between different objects, as we adopt a geometry/material-sharing strategy. One potential way to address this is to loosen the sharing constraints and model instance-wise variations. We explore this possibility in the supplementary material. Furthermore, our model requires accurate instance segmentation masks. Additionally, our approach currently requires decent 6 DoF poses as input and keeps the poses fixed. We found that jointly optimizing the 6 DoF pose and SDF field with BARF [37] will cause the pose error to increase. This is because recovering camera pose from sparse views and changing light is a difficult non-convex problem and prone to local minima. We believe the more recent works, like CamP[48] and SPARF[58] could potentially further refine our estimations. Moreover, our model cannot effectively model/constrain the geometry of unseen regions similar to existing neural fields methods.

5 Conclusion

We introduce a novel inverse rendering approach for single images with duplicate objects. We exploit the repetitive structure to estimate the 6 DoF poses of the objects, and incorporate a geometry and material-sharing mechanism to enhance the performance of inverse rendering. Experiments show that our method outperforms baselines, achieving highly detailed and precise reconstructions.

References

Jonathan T Barron and Jitendra Malik. Shape, illumination, and reflectance from shading. TPAMI , 2014.

Harry Barrow, J Tenenbaum, A Hanson, and E Riseman. Recovering intrinsic scene characteristics. Comput. vis. syst , 1978.

Brent Burley and Walt Disney Animation Studios. Physically-based shading at disney. In Siggraph , 2012.

Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-aware 3D generative adversarial networks. In arXiv , 2021.

Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. arXiv preprint arXiv:2303.13873 , 2023.

Weifeng Chen, Zhao Fu, Dawei Yang, and Jia Deng. Single-image depth perception in the wild. NeurIPS , 2016.

Wenzheng Chen, Joey Litalien, Jun Gao, Zian Wang, Clement Fuji Tsang, Sameh Khamis, Or Litany, and Sanja Fidler. Dib-r++: learning to predict lighting and material with a hybrid differentiable renderer. NeurIPS , 2021.

Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexander G Schwing. Mask2former for video instance segmentation. arXiv , 2021.

Paul E Debevec, Camillo J Taylor, and Jitendra Malik. Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2 , pages 465-474. 2023.

Yue Dong, Guojun Chen, Pieter Peers, Jiawan Zhang, and Xin Tong. Appearance-from-motion: Recovering spatially varying surface reflectance under unknown lighting. ACM Transactions on Graphics (TOG) , 33(6):1-12, 2014.

Andreas Doumanoglou, Rigas Kouskouridas, Sotiris Malassiotis, and Tae-Kyun Kim. Recovering 6d object pose and predicting next-best-view in the crowd. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3583-3592, 2016.

Alexei A Efros and Thomas K Leung. Texture synthesis by non-parametric sampling. In ICCV , 1999.

Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir. Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10786-10796, 2021.

David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. NeurIPS , 2014.

Michael Elad and Michal Aharon. Image denoising via sparse and redundant representations over learned dictionaries. TIP , 2006.

Qi Fang, Qing Shuai, Junting Dong, Hujun Bao, and Xiaowei Zhou. Reconstructing 3d human pose by watching humans in the mirror. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12814-12823, 2021.

William T Freeman. The generic viewpoint assumption in a framework for visual perception. Nature , 1994.

Daniel Glasner, Shai Bagon, and Michal Irani. Super-resolution from a single image. In ICCV , 2009.

Roger Grosse, Micah K Johnson, Edward H Adelson, and William T Freeman. Ground truth dataset and baseline evaluations for intrinsic image algorithms. In ICCV , 2009.

Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision . Cambridge university press, 2003.

Jon Hasselgren, Nikolai Hofmann, and Jacob Munkberg. Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising. arXiv:2206.03380 , 2022.

Daniel Hauagge, Scott Wehrwein, Kavita Bala, and Noah Snavely. Photometric ambient occlusion. In CVPR , 2013.

Berthold KP Horn. Determining lightness from an image. Computer graphics and image processing , 1974.

Berthold KP Horn. Obtaining shape from shading information. The psychology of computer vision , 1975.

Bo Hu, Christopher Brown, and Randal Nelson. Multiple-view 3-d reconstruction using a mirror. 2005.

Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In CVPR , 2015.

Jeffrey Ichnowski, Yahav Avigal, Justin Kerr, and Ken Goldberg. Dex-nerf: Using a neural radiance field to grasp transparent objects. arXiv , 2021.

James T Kajiya. The rendering equation. In Proceedings of the 13th annual conference on Computer graphics and interactive techniques , 1986.

Brian Karis and Epic Games. Real shading in unreal engine 4. Proc. Physically Based Shading Theory Practice , 2013.

Pierre-Yves Laffont, Adrien Bousseau, and George Drettakis. Rich intrinsic image decomposition of outdoor scenes from multiple views. IEEE transactions on visualization and computer graphics , 2012.

Edwin H Land and John J McCann. Lightness and retinex theory. Josa , 1971.

Hendrik PA Lensch, Jan Kautz, Michael Goesele, Wolfgang Heidrich, and Hans-Peter Seidel. Image-based reconstruction of spatial appearance and geometric detail. TOG , 2003.

Yikai Li, Jiayuan Mao, Xiuming Zhang, Bill Freeman, Josh Tenenbaum, Noah Snavely, and Jiajun Wu. Multi-plane program induction with 3d box priors. NeurIPS , 2020.

Yikai Li, Jiayuan Mao, Xiuming Zhang, William T Freeman, Joshua B Tenenbaum, and Jiajun Wu. Perspective plane program induction from a single image. In CVPR , 2020.

Zhaoshuo Li, Thomas Müller, Alex Evans, Russell H Taylor, Mathias Unberath, Ming-Yu Liu, and ChenHsuan Lin. Neuralangelo: High-fidelity neural surface reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Daniel Lichy, Jiaye Wu, Soumyadip Sengupta, and David W Jacobs. Shape and material capture at home. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6123-6133, 2021.

Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. Barf: Bundle-adjusting neural radiance fields. In ICCV , 2021.

Yen-Chen Lin, Pete Florence, Andy Zeng, Jonathan T Barron, Yilun Du, Wei-Chiu Ma, Anthony Simeonov, Alberto Rodriguez Garcia, and Phillip Isola. Mira: Mental imagery for robotic affordances. In CoRL , 2023.

Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun-Yan Zhu, and Bryan Russell. Editing conditional radiance fields. In ICCV , 2021.

H Christopher Longuet-Higgins. A computer algorithm for reconstructing a scene from two projections. Nature , 1981.

Wei-Chiu Ma, Hang Chu, Bolei Zhou, Raquel Urtasun, and Antonio Torralba. Single image intrinsic decomposition without a single intrinsic image. In ECCV , 2018.

Wei-Chiu Ma, Ignacio Tartavull, Ioan Andrei Bârsan, Shenlong Wang, Min Bai, Gellert Mattyus, Namdar Homayounfar, Shrinidhi Kowshika Lakshmikanth, Andrei Pokrovsky, and Raquel Urtasun. Exploiting sparse semantic hd maps for self-driving vehicle localization. In IROS , 2019.

Stephen Robert Marschner. Inverse rendering for computer graphics . Cornell University, 1998.

Tomer Michaeli and Michal Irani. Blind deblurring using internal patch recurrence. In ECCV , 2014.

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. ECCV , 2020.

Chaitanya Mitash, Bowen Wen, Kostas Bekris, and Abdeslam Boularias. Scene-level pose estimation for multiple instances of densely packed objects. In Conference on Robot Learning , pages 1133-1145. PMLR, 2020.

Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting triangular 3d models, materials, and lighting from images. In CVPR , 2022.

Philipp; Mildenhall Ben; Barron Jonathan T.; Martin-Brualla Ricardo Park, Keunhong; Henzler. Camp: Camera preconditioning for neural radiance fields. ACM Trans. Graph. , 2023.

René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In ICCV , 2021.

René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. TPAMI , 2020.

Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4938-4947, 2020.

Imari Sato, Yoichi Sato, and Katsushi Ikeuchi. Illumination from shadows. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2003.

Yoichi Sato, Mark D Wheeler, and Katsushi Ikeuchi. Object shape and reflectance modeling from observation. In SIGGRAPH , 1997.

Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In CVPR , 2016.

Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2016.

Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Proc. NeurIPS , 2020.

Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7495-7504, 2021.

Prune Truong, Marie-Julie Rakotosaona, Fabian Manhardt, and Federico Tombari. Sparf: Neural radiance fields from sparse and noisy poses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4190-4200, 2023.

Jingkang Wang, Sivabalan Manivasagam, Yun Chen, Ze Yang, Ioan Andrei Bârsan, Anqi Joyce Yang, WeiChiu Ma, and Raquel Urtasun. Cadsim: Robust and scalable in-the-wild 3d reconstruction for controllable sensor simulation. In 6th Annual Conference on Robot Learning .

Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689 , 2021.

Yiqun Wang, Ivan Skorokhodov, and Peter Wonka. Pet-neus: Positional encoding triplanes for neural surfaces. 2023.

Zian Wang, Tianchang Shen, Jun Gao, Shengyu Huang, Jacob Munkberg, Jon Hasselgren, Zan Gojcic, Wenzheng Chen, and Sanja Fidler. Neural fields meet explicit geometric representation for inverse rendering of urban scenes. arXiv , 2023.

Guandao Yang, Serge Belongie, Bharath Hariharan, and Vladlen Koltun. Geometry processing with neural fields. NeurIPS , 2021.

Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun. Unisim: A neural closed-loop sensor simulator. CVPR , 2023.

Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. NeurIPS , 2020.

Yizhou Yu, Paul Debevec, Jitendra Malik, and Tim Hawkins. Inverse global illumination: Recovering reflectance models of real scenes from photographs. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques , 1999.

Albert J Zhai and Shenlong Wang. Peanut: Predicting and navigating to unseen targets. arXiv , 2022.

Kai Zhang, Fujun Luan, Zhengqi Li, and Noah Snavely. Iron: Inverse rendering by optimizing neural sdfs and materials from photometric images. In CVPR , 2022.

Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5453-5462, 2021.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. IEEE , 2018.

Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan T Barron. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. ACM Transactions on Graphics (TOG) , 40(6):1-18, 2021.

Yuxuan Zhang, Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, Antonio Torralba, and Sanja Fidler. Image gans meet differentiable rendering for inverse graphics and interpretable 3d neural rendering. arXiv , 2020.

Yuanqing Zhang, Jiaming Sun, Xingyi He, Huan Fu, Rongfei Jia, and Xiaowei Zhou. Modeling indirect illumination for inverse rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18643-18652, 2022.

Yunzhi Zhang, Shangzhe Wu, Noah Snavely, and Jiajun Wu. Seeing a rose in five thousand ways. arXiv , 2022.

Tinghui Zhou, Philipp Krahenbuhl, and Alexei A Efros. Learning data-driven reflectance priors for intrinsic image decomposition. In ICCV , 2015.

Maria Zontak and Michal Irani. Internal statistics of a single natural image. In CVPR . IEEE, 2011.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe proposed method in Structures from Duplicates leverages multiple appearance of an identical objects in a single image to reconstruct the geometry and material properties of this object. Each instance is assigned a virtual camera, such that the shared object representation is aligned in the same space. Therefore multi-view reconstruction can be applied reconstructing each instance in a separate view.\nFirst each object is identified using a segmentation method and cropped from the image. Initial camera poses are estimated using Structure from Motion (SfM) method COLMAP on the cropped images of each segmented object. To avoid the issues that come with extreme differences in the view points for SfM methods, images of instances are rotated and optimal pairs identified.\nFrom estimated view point and image pairs the surface geometry following NeuS, the object's decomposed BRDF and environmental lighting following InvRender are optimized following known inverse rendering approaches.\nResults on synthetic data show comparable quality with multi-view reconstruction approaches, while using duplicates of the same object in a single image and present better performance in the multi-view case, leveraging virtual views from duplicates. Additional results on real data are presented that validate the result on synthetic data.\n\nSTRENGTHS:\nThe authors present an interesting solution to a common setting in the real world. The given introduction and related works motivates this setting of multiple instances well, which is an interesting setting by itself, that has not been explored by prior work in a similar way. The method explanation is thorough on all major details. Building on well-known and established decomposed neural rendering architectures and ideas (NeuS, simplified BRDF) is acceptable because this representation is not the main focus of the paper and the method does not lack to my understanding. \n\nPresented results justify most claims made in the beginning and ablation studies validate design choices. Especially showing that explicitly modelling lighting, geometry and material properties has an impact on the reconstruction quality over multi-view reconstruction on a single object is an important insight of the presented work.\n\nOne could argue, that this paper lacks novelty and just applies known concepts, such as instances from repeating object classes in the same images and building a prior from that. In my opinion neglecting a pre-trained prior is a strong advantage of the paper.\n\nWEAKNESSES:\nA major weakness of the presented paper is the variety in the results. The synthetic and real dataset are both on the smaller side and pretty homogeneous, as elaborated in the following:\n\n- While the motivation is convincing and Fig.2 shows a good variety of examples for the presented setting, the presented results in the main paper lack some variety and complexity that can be seen in the presented real world examples like the screws and chairs. The main paper presents only simple, closed object geometries, such as bottles, cans, apples, or the toy plane in the synthetic data with a lack of thin structures and complex textures. On the other side the alarm clock and coffee maker in the supplementary of this work show that the reconstruction can lead to overly smooth geometry on complex structures. This is expected on the limited number of views, but not properly addressed in the main paper.\n\n- Not purely a weakness, but a suggestion to strengthen the claims I would ask the authors to add results on more complex real geometries and scenes. To my best understanding this can be a complex task to capture and reconstruct, but it would have a positive impact on the limited scope provided by cans, bottles and packaging.\n\nAblation studies on the number of objects from 2 to 10 objects on synthetic data show results, in which a higher number of objects significantly improves the results on almost all metrics of the reconstructed object. I would argue that even a higher number could improve results further and with respect to the presented motivation such experiments are not unlikely in the real world. Therefore an upper bound for improvements could further strengthen the paper.\n\nQUESTIONS:\nPartially addressed in the weakness sections:\n- Were ablation studies performed on all objects (e.g. 2-10 instances per object category), please clarify?\n\nLIMITATIONS:\nI think the authors adequately addressed all major limitations, such as a required instance segmentation and the reliance on COLMAP poses. Additionally they addressed potential solutions. I would suggest to add a short paragraph on the scope of the paper with respect to the experiments, that I suggested in the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a pipeline for recovering object shape and surface materials from one single of multiple identical duplicate instances. The pipeline first extracts instance masks, registers camera poses using COLMAP (with in-plane rotation augmentation) and recovers shape, materials and environment lighting through inverse rendering.\n\nSTRENGTHS:\n### S1 - A sound inverse rendering pipeline\n- The proposed pipeline is carefully designed, starting from instance segmentation from Mask2former, to a camera pose estimation step using COLMAP with in-plane rotation augmentation, and finally to inverse rendering of shape, BRDF material and SG env map similar to [65].\n- Given that the instances are mostly identical, this pipeline in general works well.\n\n### S2 - Good writing\n- The paper is well written, with clear descriptions of the problem setup and most technical details.\n\nWEAKNESSES:\n### W1 - Quite a trivial task assuming *identical* instances\n- Fig. 1: \"the world is full of identical objects\" - it might be difficult to argue that different instances that appear in one scene are *identical*. This is in contrast to the claim in [66], which says \"no two roses in the natural world are identical\".\n- This paper focuses more on manufactured and synthetic objects, which appear more similar thanks to the consistent manufacturing process, but in reality, different objects often vary in both shape and appearance, due to wearing or natural variations, for instance, multiple apples in a basket.\n- This is the fundamental assumption for COLMAP to work on these instances. It can easily fail even if there is a slight variation in geometry and appearance, which raises significant concern on the method's robustness to instance variations.\n- Once the multiple instances are registered, the rest of the pipeline seems to be a standard inverse rendering task, and is highly inspired from existing work, which I do not find sufficiently interesting.\n\n### W2 - A special case of the rose paper [66]\n- Line 111: \"independently and concurrently, Zhang et al. [66]\" - NeurIPS has a two-month rule for claiming concurrent work, including arXiv papers. [66] appeared on arXiv in Dec 2022, which is almost half a year from the NeurIPS submission deadline. It might be difficult to claim concurrent work.\n- Essential, duplicates are an extreme form of the definition of similarity in [66]. The instance variation is ultimately what makes [66] interesting, as it breaks the assumption for a standard multi-view stereo approach, hence the generative modeling.\n- If the proposed method can handle instance variations well, such as different rose instances, without assuming a prior pose distribution, it will make a much more compelling argument.\n\n### Other comments\n- I suppose the material network is inherited from [65], as the \"latent code\" $\\mathbf{\\rho}$ and its regularizer are the same as [65]. But this is not mentioned in the paper.\n\nQUESTIONS:\nIt might be hard to change my opinion on this paper, as most of the technical aspects are clear to me. Rather, this is a slight trivial task, especially given [66] has demonstrated results on a more generic case. Yet, I'm still open to objection.\n\nLIMITATIONS:\nThe paper attributes the challenge of the modeling instance variation to the shape/material model, but the fundamental challenge is rather the pose estimation with COLMAP to begin with. COLMAP generally fails terribly with deformable objects - also the reason that recent deformable NeRFs typically mask out deformable objects running COLMAP to register cameras.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the inverse graphics task of predicting the geometry, material and illumination from a single image containing multiple identical objects. The key insight is to leverage the multiple instances depicted in the image to frame this single-view multi-object reconstruction problem into a better constrained multi-view single-object reconstruction problem. Specifically, the proposed approach (dubbed SfD) first identify and crop the instances in the image, then estimate their 6DoF poses using SfM to create a set of calibrated multi-view images. Finally, a geometric reconstruction module based on prior MVS works is optimized using the resulting multi-views. The framework is validated through experiments conducted on a custom dataset dubbed Dup and introduced in this work.\n\nSTRENGTHS:\nS1. Leveraging duplicated instances to frame a single-view multi-object reconstruction problem into a multi-view single-object one is an interesting idea. Besides, this idea is rather novel; it has only been studied by the concurrent CVPR'23 work of [66]\n\nS2. Experiment section looks strong; the compared methods are relevant and the reported performances are better than prior works\n\nS3. The presentation is very clear and well written. It is easy to walk through the method formulation, which does not contain any major technical flaw. Figures are neat, well designed, and help the understanding.\n\nWEAKNESSES:\n**W1. New problem with low applicability**\n\nAlthough conceptually interesting, this work is tackling a new problem which by essence is very limited in terms of applicability to real-case scenarios, thus questioning its usefulness. Indeed, I see very few cases where the approach developed here could be successful, and quite a lot of failure cases: for example, I think the proposed approach would fail on most of the examples advertised in Figure 2 (see W2 for details).\n\n**W2. Lack of analysis**\n\nThe current presentation lacks some crucial analyses to better understand the proposed approach. In particular, I would expect experiments outlining:\n\n- the contribution of each key component (quantitative ablation study): each of the 6 loss terms, rotation aware data augmentation, etc.\n- its robustness to real data: in particular not exact duplicates like leaves of a tree\n- its robustness to noisy input: an image with duplicates + some other objects (e.g. chairs and tables in Fig. 2) or an image with only one-side instance views (e.g. cars in Fig. 2)\n- its robustness to high number of duplicates (e.g. 100/1000 objects like the pile of screws in Fig. 2)\n- the failure cases in the general setting (e.g., from the video in the supmat, it seems the geometry of the coke can is actually not so great, why?)\n\n**W3. Highly engineered method**\n\nIn addition to the low applicability of the formulated problem, the proposed approach is heavily engineered (6 loss terms, two-level approach including an optimization level with multiple stages, etc). Such amount of engineering typically makes it hard to apply the method to other cases or even to modify and build on top of it to improve the method.\n\nQUESTIONS:\n- Table 1 is overflowing and appears way too early in the presentation (harms the overall readability)\n- L142: for completeness, I would suggest formulating the changes in the intrinsic matrix when cropping is performed (in the supplementary material)\n- Eq (4) and (5): there is no scaling factors to weight the different loss terms?\n- in general, the paper looks crowed because spaces have been modified; I think the presentation would gain in readability by leveraging the extra page to make it breath a bit more (especially true for the tables)\n\nLIMITATIONS:\nThere is a small limitation section, which I think should be more detailed and illustrated to help the reader better understand the proposed method (see W2)", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a method for reconstructing the geometry, material, and illumination of an object using as input an image containing multiple copies of the object. The method leverages the appearance of multiple instances of an object in a single image to essentially create a multi-view supervision signal. The method first segments the different instances of the object in the image, then performs 6DoF pose estimation followed by a structure from motion pipeline to create an \"artificial\" multi-view setup. The object geometry and appearance is modeled with Neural Fields following NeuS. The whole pipeline is supervised using a photometric loss between the renderings and the original image.\n\nSTRENGTHS:\n1. I find this paper particularly interesting. I really like the idea of formulating a duality between multiple copies of an object in a single image and multiple views of a single object. It is a very nice example of thinking outside the box.\n2. I enjoyed reading the paper. It is well-written and easy to follow.\n3. The individual components used are properly justified and the engineering behind the method is also really good. It's a lot of individual components (segmentation, pose estimation, SfM, geometry and light representation) stitched together and it requires a lot of effort to make them work.\n4. The qualitative and quantitative results (Tables 2 & 3) are very good.\n\nI would like to see this paper get accepted.\n\nWEAKNESSES:\n1. I can't find any particularly important weakness. One issue is that the problem the paper attempts to solve is not something someone will encounter very often in real-world scenarios, but it's definitely very interesting from a scientific perspective.\n\nQUESTIONS:\nI would like to see what are some failure cases of the method. I couldn't find any in the main paper or supplementary material.\n\nLIMITATIONS:\nThe paper has a decent discussion on the limitations of the proposed method. I could not identify any immediate potential societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents \"Structure from Duplicates\" (SfD), a novel inverse graphics framework introduced to reconstruct the 3D structure, material, and illumination of multiple identical objects from a single image. The key steps include identifying these duplicate objects in an image and estimating their 6DoF pose. Then, the model applies an inverse graphics pipeline to deduce information about the objects' shape, material, and the lighting of the scene, while taking into account that these objects share the same geometry and material properties.\n\nSTRENGTHS:\nIn my opinion, the key strengths of the proposed inverse rendering method are:\n\n- The method outperforms baseline techniques in single image multi-object setup and multi-view inverse rendering scenarios, as shown in the experiments.\n\n- The approach can leverage duplicate objects to constrain underlying geometry and better disentangle the effects of lighting from materials.\n\n- It can be extended to multi-view setups.\n\n- The model makes effective use of duplicate objects, with accuracy improving as the number of duplicates increases.\n\n- It performs well with real-world data, achieving comparable performance to multi-view baselines when trained using only a single view.\n\n- The model supports various scene edits - once the material and geometry of the objects are recovered, along with the scene's illumination, it can faithfully relight existing objects, edit their materials, and insert new objects into the environment.\n\nWEAKNESSES:\nThe potential weaknesses seem to be:\n\n- Dependence on Similarity. A significant limitation is that the instances in each image need to be nearly identical; the method struggles when there are substantial deformations between different objects.\n\n- Pose Estimation Errors. The model currently requires accurate 6 DoF poses as input and keeps the poses fixed, which may limit its applicability in certain scenarios.\n\n- Potential Segmentation Errors. SfD begins by identifying multiple instances of an object within an image. This is prone to errors when the segmentation is inaccurate which is more of a problem on glossy and specular objects.\n\nQUESTIONS:\n- How would the model perform when applied to real-world scenarios with more noise, variability, and less regularity in object structures?\n\n- Could you discuss any strategies for mitigating the potential overfitting that might occur if you increase the capacity of the model to allow for instance-wise variations? \n\n- Could you elaborate on the potential of integrating your approach with BARF.  What kind of improvements or challenges do you expect?\n\nLIMITATIONS:\nLimitations are discussed above in the weaknesses section. Overall I am on the fence due to the limitation of the proposed approach, which is highly dependent on the similarity of the multi-objects in a single image.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}