SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Structure fromDuplicates: Neural Inverse Graphics from a Pile of Objects
Tianhang Cheng<sup>1</sup> Wei-Chiu Ma<sup>2</sup> Kaiyu Guan<sup>1</sup> Antonio Torralba<sup>2</sup> Shenlong Wang<sup>1</sup>  
<sup>1</sup>University of Illinois Urbana-Champaign <sup>2</sup>Massachusetts Institute of Technology {tcheng12, kaiyug, shenlong}@illinois.edu {weichium, torralba}@mit.edu
6 DoF pose
Roughness/Metallic
Object insertion
Material editing I
Material editing 2
Figure 1: Structure from duplicates (SfD) is a novel inverse graphics framework that reconstructs geometry, material, and illumination from a single image containing multiple identical objects.
Relighting 1
Relighting 2
Our world is full of identical objects (e.g., cans of coke, cars of same model). These duplicates, when seen together, provide additional and strong cues for us to effectively reason about 3D. Inspired by this observation, we introduce Structure from Duplicates (SfD), a novel inverse graphics framework that reconstructs geometry, material, and illumination from a single image containing multiple identical objects. SfD begins by identifying multiple instances of an object within an image, and then jointly estimates the 6DoF pose for all instances. An inverse graphics pipeline is subsequently employed to jointly reason about the shape, material of the object, and the environment light, while adhering to the shared geometry and material constraint across instances. Our primary contributions involve utilizing object duplicates as a robust prior for single-image inverse graphics and proposing an in-plane rotation-robust Structure from Motion (SfM) formulation for joint 6-DoF object pose estimation. By leveraging multi-view cues from a single image, SfD generates more realistic and detailed 3D reconstructions, significantly outperforming existing single image reconstruction models and multi-view reconstruction approaches with a similar or greater number of observations. Code available at https://github.com/Tianhang-Cheng/SfD
Given a single/set of image(s), the goal of inverse rendering is to recover the underlying geometry, material, and lighting of the scene. The task is of paramount interest to many applications in computer vision, graphics, and robotics and has drawn extensive attention across the communities over the past few years[74; 47; 21; 45].
Figure 2: Repetitions in the visual world. Our physical world is full of identical objects (e.g., cans of coke, cars of the same model, chairs in a classroom). These duplicates, when seen together, provide additional and strong cues for us to effectively reason about 3D.
Since the problem is ill-posed, prevailing inverse rendering approaches often leverage multi-view observations to constrain the solution space. While these methods have achieved state-of-the-art performance, in practice, it is sometimes difficult, or even impossible, to obtain those densely captured images. To overcome the reliance on multi-view information, researchers have sought to incorporate various structural priors, either data-driven or handcrafted, into the models[5; 36]. By utilizing the regularizations, these approaches are able to approximate the intrinsic properties (e.g., material) and extrinsic factors (e.g., illumination) even from one single image. Unfortunately, the estimations may be biased due to the priors imposed. This makes one ponder: is it possible that we take the best of both worlds? Can we extract multi-view information from a single image under certain circumstances?
Fortunately the answer is yes. Our world is full of repetitive objects and structures. Repetitive patterns in single images can help us extract and utilize multi-view information. For instance, when we enter an auditorium, we often see many identical chairs facing slightly different directions. Similarly, when we go to a supermarket, we may observe multiple nearly-identical apples piled on the fruit stand. Although we may not see the exact same object from multiple viewpoints in just one glance, we do see many of the "identical twins" from various angles, which is equivalent to multi-view observations and even more (see Sec. 3 for more details). Therefore, the goal of this paper is to develop a computational model that can effectively infer the underlying 3D representations from a single image by harnessing the repetitive structures of the world.
With these motivations in mind, we present Structure from Duplicates (SfD), a novel inverse rendering model that is capable of recovering high-quality geometry, material, and lighting of the objects from a single image. SfD builds upon insights from structure from motion (SfM) as well as recent advances on neural fields. At its core lies two key modules: (i) a in-plane rotation robust pose estimation module, and (ii) a geometric reconstruction module. Given an image of a scene with duplicate objects, we exploit the pose estimation module to estimate the relative 6 DoF poses of the objects. Then, based on the estimated poses, we align the objects and create multiple "virtual cameras." This allows us to effectively map the problem from a single-view multi-object setup to a multi-view single-object setting (see Fig. 3). Finally, once we obtain multi-view observations, we can leverage the geometric module to recover the underlying intrinsic and extrinsic properties of the scene. Importantly, SfD can be easily extended to multi-image setup. It can also be seen as a superset of existing NeRF models, where the model will reduce to NeRF when there is only one single object in the scene.
We validate the efficacy of our model on a new dataset called Dup, which contains synthetic and real-world samples of duplicated objects since current multi-view datasets lack duplication samples. This allows us to benchmark inverse rendering performance under single-view or multi-view settings. Following previous work [60; 65; 47; 69; 74], we evaluate rendering, relighting and texture quality with MSE, PSNR, SSIM, LPIPS [70], geometry with Chamfer Distance (CD), and environment light with MSE.
Our contributions are as follows: 1) We proposed a novel setting called "single-view duplicate objects" (S-M), which expands the scope of the inverse graphics family with a multi-view single-instance (M-S) framework. 2) Our method produces more realistic material texture than the existing multi-view inverse rendering model when using the same number of training views. 3) Even only relying on a single-view input, our approach can still recover comparable or superior materials and geometry compared to baselines that utilize multi-view images for supervision.
Figure 3: Method overview: (Left) SfD begins by identifying multiple instances of an object within an image, and then jointly estimates the 6DoF pose for all instances. (Right) An inverse graphics pipeline is subsequently employed to reason about the shape, material of the object, and the environment light, while adhering to the shared geometry and material constraint across instances.
Inverse rendering: The task of inverse rendering can be dated back to more than half a century ago [31; 23; 24; 2]. The goal is to factorize the appearance of an object or a scene in the observed image(s) into underlying geometry, material properties, and lighting conditions [53; 43; 66]. Since the problem is severely under-constrained, previous work mainly focused on controlled settings or simplifications of the problem [19; 22; 1]. For instance, they either assume the reflectance of an object is spatially invariant [69], presume the lighting conditions are known [57], assume the materials are lambertian [76; 41], or presume the proxy geometry is available [52; 32; 10; 30]. More recently, with the help of machine learning, in particular deep learning, researchers have gradually moved towards more challenging in-the-wild settings [73; 47]. By pre-training on a large amount of synthetic yet realistic data [36] or baking inductive biases into the modeling pipeline [7; 72], these approaches can better tackle unconstrained real-world scenarios (e.g., unknown lighting conditions) and recover the underlying physical properties more effectively [47; 68]. For example, through properly modeling the indirect illumination and the visibility of direct illumination, Zhang et al. [74] is able to recover interreflection- and shadow-free SVBRDF materials (e.g., albedo, roughness). Through disentangling complex geometry and materials from lighting effects, Wang et al. [62] can faithfully relight and manipulate a large outdoor urban scene. Our work builds upon recent advances in neural inverse rendering. Yet instead of grounding the underlying physical properties through multi-view observations as in prior work, we focus on the single image setup and capitalize on the duplicate objects in the scene for regularization. The repetitive structure not only allows us to ground the geometry, but also provide additional cues on higher-order lighting effects (e.g., cast shadows). As we will show in the experimental section, we can recover the geometry, materials, and lighting much more effectively even when comparing to multi-view observations.
3D Reconstruction: Recovering the spatial layout of the cameras and the geometry of the scene from a single or a collection of images is a longstanding challenge in computer vision. It is also the cornerstone for various downstream applications in computer graphics and robotics such as inverse rendering [43; 60; 62], 3D editing [39; 63], navigation [42; 67], and robot manipulation [27; 38]. Prevailing 3D reconstruction systems, such as structure from motion (SfM), primarily rely on multi-view geometry to estimate the 3D structure of a scene [40; 20; 54]. While achieving significant successes, they rely on densely captured images, limiting their flexibility and practical use cases. Single image 3D reconstruction, on the other hand, aims to recover metric 3D information from a monocular image [14; 6; 50; 49]. Since the problem is inherently ill-posed and lacks the ability to leverage multi-view geometry for regularization, these methods have to resort to (learned) structural priors to resolve the ambiguities. While they offer greater flexibility, their estimations may inherit biases from the training data. In this paper, we demonstrate that, under certain conditions, it is possible to incorporate multi-view geometry into a single image reconstruction system. Specifically, we leverage repetitive objects within the scene to anchor the underlying 3D structure. By treating
each of these duplicates as an observation from different viewpoints, we can achieve highly accurate metric 3D reconstruction from a single image.
Repetitions: Repetitive structures and patterns are ubiquitous in natural images. They play important roles in addressing numerous computer vision problems. For instance, a single natural image often contains substantial redundant patches [77]. The recurrence of small image patches allows one to learn a powerful prior which can later be utilized for various tasks such as super-resolution [18; 26], image deblurring [44], image denoising [15], and texture synthesis [12]. Moving beyond patches, repetitive primitives or objects within the scene also provide informative cues about their intrinsic properties [33; 64]. By sharing or regularizing their underlying representation, one can more effectively constrain and reconstruct their 3D geometry [25; 16; 9], as well as enable various powerful image/shape manipulation operations [34; 59]. In this work, we further push the boundary and attempt to recover not just the geometry, but also the materials (e.g., albedo, roughness), visibilities, and lighting conditions of the objects. Perhaps closest to our work is [75]. Developed independently and concurrently, Zhang et al. build a generative model that aims to capture object intrinsics from a single image with multiple similar/same instances. However, there exist several key differences: 1) we explicitly recover metric-accurate camera poses using multi-geometry, whereas Zhang et al. learn this indirectly through a GAN-loss; 2) we parameterize and reason realistic PBR material and environmental light; 3) we handle arbitrary poses, instead of needing to incorporate a prior pose distribution. Some previous work also tried to recover 6D object pose from crowded scene or densely packed objects[46; 11] from RGB-D input, but our model only require RGB input.
3 Structure fromDuplicates
In this paper, we seek to devise a method that can precisely reconstruct the geometry, material properties, and lighting conditions of an object from a single image containing duplicates of it. We build our model based on the observation that repetitive objects in the scene often have different poses and interact with the environment (e.g., illumination) differently. This allows one to extract rich multi-view information even from one single view and enables one to recover the underlying physical properties of the objects effectively.
We start by introducing a method for extracting the "multi-view" information from duplicate objects. Then we discuss how to exploit recent advances in neural inverse rendering to disentangle both the object intrinsics and environment extrinsics from the appearance. Finally, we describe our learning procedure and design choices.
3.1 Collaborative 6-DoF pose estimation
As we have alluded to above, a single image with multiple duplicate objects contains rich multiview information. It can help us ground the underlying geometry and materials of the objects, and understand the lighting condition of the scene.
Our key insight is that the image can be seen as a collection of multi-view images stitching together. By cropping out each object, we can essentially transform the single image into a set of multi-view images of the object from various viewpoints. One can then leverage structure from motion (SfM) [55] to estimate the relative poses among the multi-view images, thereby aggregating the information needed for inverse rendering. Notably, the estimated camera poses can be inverted to recover the 6 DoF poses of the duplicate objects. As we will elaborate in Sec. 3.2, this empowers us to more effectively model the extrinsic lighting effect (which mainly depends on world coordinate) as well as to properly position objects in perspective and reconstruct the exact same scene.
To be more formal, let  $\mathcal{I} \in \mathbb{R}^{H \times W \times 3}$  be an image with  $N$  duplicate objects. Let  $\{\mathcal{I}_i^{\mathrm{obj}}\}_{i=1}^N \in \mathbb{R}^{w \times h \times 3}$  be the corresponding object image patches. We first leverage SfM [55; 51] to estimate the camera poses of the multi-view cropped images\*  $\{\pmb{\xi}_i \in \mathbb{S}\mathbb{E}(3)\}_{i=1}^N$ :
$$
\boldsymbol {\xi} _ {1} ^ {\mathrm {c a m}}, \boldsymbol {\xi} _ {2} ^ {\mathrm {c a m}}, \dots , \boldsymbol {\xi} _ {N} ^ {\mathrm {c a m}} = f ^ {\mathrm {S f M}} \left(\mathcal {I} _ {1} ^ {\mathrm {o b j}}, \mathcal {I} _ {2} ^ {\mathrm {o b j}}, \dots , \mathcal {I} _ {K} ^ {\mathrm {o b j}}\right). \tag {1}
$$
Next, since there is only one real camera in practice, we can simply align the  $N$  virtual cameras  $\{\pmb{\xi}_i\}_{i=1}^N$  to obtain the 6 DoF poses of the duplicate objects. Without loss of generality and for
*In practice, the cropping operation will change the intrinsic matrix of the original camera during implementation. For simplicity, we assume the intrinsics are properly handled here.
simplicity, we align all the cameras to a reference coordinate  $\pmb{\xi}^{\mathrm{ref}}$ . The 6 DoF poses of the duplicate objects thus become  $\pmb{\xi}_i^{\mathrm{obj}} = \pmb{\xi}^{\mathrm{ref}} \circ (\pmb{\xi}_i^{\mathrm{cam}})^{-1}$ , where  $\circ$  is matrix multiplication for pose composition.
In practice, we first employ a state-of-the-art panoptic segmentation model [8] to segment all objects in the scene. Then we fit a predefined bounding box to each object and crop it. Lastly, we run COLMAP [55] to estimate the 6 DoF virtual camera poses, which in turn provides us with the 6 DoF object poses. Fig. 3(left) depicts our collaborative 6-DoF pose estimation process.
Caveats of random object poses: Unfortunately, naively feeding these object patches into SfM would often leads to failure, as little correspondence can be found. This is due to the fact that state-of-the-art correspondence estimators [51] are trained on Internet vision data, where objects are primarily upright. The poses of the duplicate objects in our case, however, vary significantly. Moreover, the objects are often viewed from accidental viewpoints [17]. Existing estimators thus struggle to match effectively across such extreme views.
Rotation-aware data augmentation: Fortunately, the scene contains numerous duplicate objects. While estimating correspondences reliably across arbitrary instances may not always be possible, there are certain objects whose viewpoints become significantly similar after in-plane rotation. Hence, we have developed an in-plane rotation-aware data augmentation for correspondence estimation.
Specifically, when estimating correspondences between a pair of images, we don't match the images directly. Instead, we gradually rotate one image and then perform the match. The number of correspondences at each rotation is recorded and then smoothed using a running average. We take the  $\arg\max$  to determine the optimal rotation angle. Finally, we rotate the correspondences from the best match back to the original pixel coordinates. As we will demonstrate in Sec. 4, this straightforward data augmentation strategy significantly improves the accuracy of 6 DoF pose estimation. In practice, we rotate the image by  $4^{\circ}$  per step. All the rotated images are batched together, enabling us to match the image pairs in a single forward pass.
3.2 Joint shape, material, and illumination estimation
Suppose now we have the 6 DoF poses of the objects  $\{\pmb{\xi}_i^{\mathrm{obj}}\}_{i=1}^N$ . The next step is to aggregate the information across duplicate objects to recover the intrinsics properties of the objects (e.g., geometry, materials) and the extrinsic factors of the world (e.g., illumination). We aim to reproduce these attributes as faithfully as possible, so that the resulting estimations can be utilized for downstream tasks such as relighting and material editing. Since the task is under-constrained and joint estimation often leads to suboptimal results, we follow prior art [74; 62] and adopt a stage-wise procedure.
Geometry reconstruction: We start by reconstructing the geometry of the objects. In line with NeuS [60], we represent the object surfaces as the zero level set of a signed distance function (SDF).
We parameterize the SDF with a multi-layer perceptron (MLP)  $S:\mathbf{x}^{\mathrm{obj}}\mapsto s$  that maps a 3D point under object coordinate  $\mathbf{x}^{\mathrm{obj}}\in \mathbb{R}^3$  to a signed distance value  $s\in \mathbb{R}$ . Different from NeuS [60], we model the geometry of objects in local object space. This allows us to guarantee shape consistency across instances by design.
We can also obtain the surface normal by taking the gradient of the SDF:  $\mathbf{n}(\mathbf{x}^{\mathrm{obj}}) = \nabla_{\mathbf{x}^{\mathrm{obj}}}S$ . To learn the geometry from multi-view images, we additionally adopt an auxiliary appearance MLP  $C:\{\mathbf{x},\mathbf{x}^{\mathrm{obj}},\mathbf{n},\mathbf{n}^{\mathrm{obj}},\mathbf{d},\mathbf{d}^{\mathrm{obj}}\} \mapsto \mathbf{c}$  that takes as input a 3D point  $\mathbf{x}$ ,  $\mathbf{x}^{\mathrm{obj}}$ , surface normal  $\mathbf{n}$ ,  $\mathbf{n}^{\mathrm{obj}}$ , and view direction  $\mathbf{d}$ ,  $\mathbf{d}^{\mathrm{obj}}$  under both coordinate systems and outputs the color  $\mathbf{c} \in \mathbb{R}^3$ . The input from world coordinate system helps the MLP to handle the appearance inconsistencies across instances caused by lighting or occlusion. We tied the weights of the early layers of  $C$  to those of  $S$  so that the gradient from color can be propagated to geometry. We determine which object coordinate to use based on the object the ray hits. This information can be derived either from the instance segmentation or another allocation MLP  $A:\{\mathbf{x},\mathbf{d}\} \mapsto q$  (distilled from SDF MLP), where  $q \in \mathbb{N}$  is the instance index. After we obtain the geometry MLP  $S$ , we discard the auxiliary appearance MLP  $C$ . As we will discuss in the next paragraph, we model the object appearance using physics-based rendering (PBR) materials so that it can handle complex real-world lighting scenarios.
Material and illumination model: Now we have recovered the geometry of the objects, the next step is to estimate the environment light of the scene as well as the material of the object. We assume
Zhang et al.[74]
Dffse
Rei
Figure 4: Multi-view inverse rendering.
Figure 5: Multi-view single object (M-S) vs single-view multi-objects (S-M).
M-S
Dffse
all lights come from an infinitely faraway sphere and only consider direct illumination. Therefore, the illumination emitted to a 3D point in a certain direction is determined solely by the incident light direction  $\boldsymbol{w}_i$  and is independent of the point's position. Similar to [69; 74], we approximate the environment light with  $M = 128$  Spherical Gaussians (SGs):
$$
L _ {i} \left(\omega_ {i}\right) = \sum_ {k = 1} ^ {M} \boldsymbol {\mu} _ {k} e ^ {\lambda_ {k} \left(\boldsymbol {w} _ {i} \cdot \boldsymbol {\phi} _ {k} - 1\right)}, \tag {2}
$$
where  $\lambda \in \mathbb{R}^{+}$  is the lobe sharpness,  $\mu$  is the lobe amplitude, and  $\phi$  is the lobe axis. This allows us to effectively represent the illumination and compute the rendering equation (Eq. 3) in closed-form.
As for object material, we adopt the simplified Disney BRDF formulation [3; 29] and parameterized it as a MLP  $M: \mathbf{x}^{\mathrm{obj}} \mapsto \{\mathbf{a}, r, m\}$ . Here,  $\mathbf{a} \in \mathbb{R}^3$  denotes albedo,  $r \in [0,1]$  corresponds to roughness, and  $m \in [0,1]$  signifies metallic. Additionally, inspired by InvRender [74], we incorporate a visibility MLP  $V: (\mathbf{x}, \boldsymbol{w}_i) \mapsto v \in [0,1]$  to approximate visibility for each world coordinate faster reference. The difference is that we use the sine activation function[56] rather than the ReLU activation for finer detail. Since the visibility field is in world space, we can model both inter-object self-casted shadows and inter-object occlusions for multiple instances in our setup. We query only surface points, which can be derived from the geometry MLP  $S$  using volume rendering. The material MLP  $M$  also operates in object coordinates like  $S$ , which ensure material consistency across all instances. Moreover, the variations in lighting conditions between instances help us better disentangle the effects of lighting from the materials. We set the dielectrics Fresnel term to  $F_0 = 0.02$  and the general Fresnel term to  $\mathbf{F} = (1 - m)F_0 + ma$  to be compatible with both metals and dielectrics.
Combining all of these components, we can generate high-quality images by integrating the visible incident lights from hemisphere and modeling the effects of BRDF [28]:
$$
L _ {o} \left(\boldsymbol {w} _ {o}; \mathbf {x}\right) = \int_ {\Omega} L _ {i} \left(\boldsymbol {w} _ {i}\right) f _ {r} \left(\boldsymbol {w} _ {i}, \boldsymbol {w} _ {o}; \mathbf {x}\right) \left(\boldsymbol {w} _ {i} \cdot \mathbf {n}\right) d \boldsymbol {w} _ {i}. \tag {3}
$$
Here,  $\boldsymbol{w}_i$  is the incident light direction, while  $\boldsymbol{w}_o$  is the viewing direction. The BRDF function  $f_r$  can be derived from our PBR materials. We determine the visibility of an incident light either through sphere tracing or following Zhang et al. [74] to approximate it with the visibility MLP  $V$ . We refer the readers to the supplementary materials for more details.
Optimizing shape, illumination, and material jointly from scratch is challenging. Taking inspiration from previous successful approaches [74; 69], we implement a multi-stage optimization pipeline. We progressively optimize the geometry first, then visibility, and finally, material and illumination.