SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Senthil Purushwalkam*
Salesforce AI Research spurushwalkam@salesforce.com
Nikhil Naik  
Salesforce AI Research  
nnaik@salesforce.com
We present a novel method for reconstructing 3D objects from a single RGB image. Our method leverages the latest image generation models to infer the hidden 3D structure while remaining faithful to the input image. While existing methods[1, 2] obtain impressive results in generating 3D models from text prompts, they do not provide an easy approach for conditioning on input RGB data. Naive extensions of these methods often lead to improper alignment in appearance between the input image and the 3D reconstructions. We address these challenges by introducing Image Constrained Radiance Fields (ConRad), a novel variant of neural radiance fields. ConRad is an efficient 3D representation that explicitly captures the appearance of an input image in one viewpoint. We propose a training algorithm that leverages the single RGB image in conjunction with pretrained Diffusion Models to optimize the parameters of a ConRad representation. Extensive experiments show that ConRad representations can simplify preservation of image details while producing a realistic 3D reconstruction. Compared to existing state-of-the-art baselines, we show that our 3D reconstructions remain more faithful to the input and produce more consistent 3D models while demonstrating significantly improved quantitative performance on a ShapeNet object benchmark.
Humans possess the ability to accurately infer the full 3D structure of an object even after observing just one viewpoint. Since the RGB and depth observed for one viewpoint does not provide sufficient information, we have to rely on our past experiences to make intelligent inferences about a realistic reconstruction of the full 3D structure. This ability helps us navigate and interact with the 3D world around us seamlessly. Capturing a similar capability of generating the full 3D structure from a single image has been a long-standing problem in Computer Vision. Such systems could facilitate advances in robotic manipulation and navigation systems, and has applications in video games, augmented reality and e-commerce. Despite the importance of this capability, success on solving this problem has been limited.
This lack of success can partly be attributed to the lack of useful large scale data. While we have made progress on collecting large scale image datasets, the scale of 3D object or multi-view image datasets has been severely limited. The availability of billions of RGB images and captions [3] has facilitated significant advances in semantic understanding of images [4-6]. Over the last few years in particular, approaches for generative modeling of the image manifold have demonstrated capabilities including controllable generation of highly realistic images [7, 8], inpainting of occluded images [9] and realistic image editing [10-12]. These success stories demonstrate the ability of the 2D generative models to capture prior knowledge about the visual world. In light of this evidence,
*Project webpage: https://www.senthilpurushwalkam.com/publication/conrad/
Figure 1: ConRad representation: We propose a novel radiance field, ConRad, which allows conditioning on a single image. In contrast to regular radiance fields, our approach can accurately model the input image in one reference viewpoint of the radiance field without any training. Utilizing ConRad representations simplifies the optimization of a radiance field for generating a 3D model from a single image.
we ask the question—can this 2D prior knowledge be leveraged to infer the hidden 3D structure of objects?
DreamFusion [1] generates 3D models based on an input text prompt, using the distribution captured by a pretrained diffusion model to update the parameters of a neural radiance field (NERF) [13]. While DreamFusion demonstrates impressive capabilities, generating arbitrary instances of objects from text prompts severely limits its applications. In recent concurrent work, RealFusion [14] and NeuralLift360 [15] extend DreamFusion to accommodate image input. These methods propose to optimize additional objectives to reconstruct the given input image in one reference viewpoint of the NERF while still relying on a diffusion model to generate a realistic reconstruction in novel viewpoints.
In this work, we propose a novel method for generating a 3D object from a single input image. Instead of designing additional objectives to reconstruct the input image, we propose to rethink the underlying 3D representation. We introduce Image Constrained Radiance Fields (ConRad), a novel variant of NERFs that can explicitly capture an input image without any training. Given an input image and a chosen arbitrary reference camera pose, ConRad utilizes multiple constraints to incorporate the appearance of the object and the estimated foreground mask into the color and density fields respectively. These constraints ensure that the rendering of the radiance field from the chosen viewpoint exactly matches the input image while allowing the remaining viewpoints to be optimized.
The proposed ConRad representation significantly simplifies the process of generating a consistent 3D model for the input image using any pretrained diffusion model. Since the representation accounts for the reference view, the training simply has to focus on distilling the diffusion model prior for the other viewpoints. Therefore, we show that we can leverage a training algorithm that has minimal deviations from DreamFusion. We propose a few key improvements to the training algorithm which leads to more robust training and higher quality 3D reconstructions.
We demonstrate results using images gathered from multiple datasets including CO3D [16], ShapeNet [17], and images generated from pretrained generative models. ConRad produces 3D reconstructions that are consistently faithful to the input image while producing high quality full 3D reconstructions. In contrast to RealFusion and NeuralLift360, ConRad has fewer hyperparameters and does not rely on balancing the trade off between image reconstruction vs satisfying the diffusion prior, leading to a more robust training pipeline. We conduct a quantitative comparison on objects from the ShapeNet dataset and observe that our approach demonstrates significant gains in reconstruction quality compared to state-of-the-art methods.
3D Representations Over the last decade, research in Computer Vision and Graphics has led to development of efficient parametric representations of 3D scenes [13, 18, 19]. Traditional voxel
grid representations [20-22] maintain a dense grid of 3D scene parameters but consume a lot of memory. Neural Radiance Fields (NERFs) [13] offer a continuous parametric representation of the scene modeled as a neural network while requiring significantly lower memory while being computationally inefficient. Several approaches [23, 18] have been proposed to take advantage of both forms of representation. Instant-NGP [18] is one such approach that models a 3D scene using a multi-resolution hash-encoding followed by a shallow neural network. Our proposed ConRad representation builds on Instant-NGP but can be used with any radiance field.
Learning to Estimate 3D Structure To recover the full 3D structure from a single image, many works propose to learn category-specific 3D representations [24] then later fit to an input image. These approaches primarily relied on coarse representations like point clouds [24] and compositions of primitive shapes [25]. Furthermore, these methods focused on the geometry and could not infer the hidden appearance of the objects. NERFs demonstrated the capability to accurately infer novel viewpoints of a scene using densely sampled images. This has inspired numerous approaches to train similar radiance fields using sparse samples of a scene or object using semantic and geometric priors. [26-30] train models on large scale multi-view data to estimate radiance fields given a single or few images. The limited diversity of available 3D data restricts the generalizability of these approaches.
Generating 3D from Pretrained 2D models Large scale 2D image data is much more readily available compared to 3D or multi-view data. This has led to significantly larger advances in modeling semantics in images [4-6] and generative modeling of images [7, 8]. In light of this, several works have attempted to leverage the knowledge from these 2D models for the task of inferring the radiance fields from one or few images. Our proposed method belongs to this category of approaches since we rely on the distribution modeled by a pretrained image diffusion model.
DreamFields [31] generates 3D models from text prompts by optimizing the CLIP [4] distance between the NERF renderings and the textual CLIP embedding. DreamFusion [1] proposes an approach called Score Distillation Sampling (SDS) for distilling knowledge from a pretrained text-conditioned Diffusion Model [8] into a NERF representation (see Section 3 for more details). Magic3D [2] proposes a coarse-to-fine strategy that also leverages SDS to first train a NERF and then finetune a mesh representation. Similar to DreamFields, DietNERF [32] proposed to train a NERF on a few input images while enforcing consistency of CLIP [4] features in other unknown views. However, this approach fails to generate 3D objects using a single image input. Pretrained image generative models have also been used to infer radiance fields in existing work [33] but only focuses on estimating geometry of the regions visible in the input image.
RealFusion [14] and NeuralLift360 [15] are two recent methods that are very relevant to our proposed approach. Inspired by DreamFusion, these approaches leverage a pretrained Stable Diffusion model to infer the appearance and geometry of unknown viewpoints and optimize several additional objectives to enforce consistency to the input image in one reference viewpoint. In concurrent work, Neri[34] also follows a similar approach of using a reconstruction loss and depth loss for the known viewpoint, and uses diffusion model priors for unknown viewpoints. Unless properly tuned, these approaches lead to poor alignment between the input image and reconstructed 3D model (more details in Section 5). In contrast, ConRad focuses on modifying the underlying 3D representation to easily incorporate the input image and generates a more consistent 3D reconstruction.
We first provide a concise summary of the prerequisite concepts from generative modeling of images and 3D objects that we build upon for ConRad.
Diffusion Models A diffusion model is a recently developed generative model that synthesizes images by iteratively denoising a sample from a Gaussian distribution. For training a diffusion model, noise is added to a real image  $I$  iteratively for  $T$  timesteps to synthesize training data  $\{I_0,I_1,\dots,I_T\}$ . Specifically, the noised image for timestep  $t$  can be computed as  $I_{t} = \sqrt{\alpha_{t}} I + \sqrt{1 - \alpha_{t}\epsilon}$  where  $\epsilon \in \mathcal{N}(\mathbf{0},\mathbf{I})$  and  $\alpha_{t}$  is predefined noising schedule. This data is used to a train a denoising model  $\hat{\epsilon}_{\phi}(I_t,y,t)$  which estimates the noise  $\epsilon$  using the noisy image  $I_{t}$ , the timestep  $t$  and an optional embedding  $y$  of the caption associated with the image. A pretrained diffusion model can be used to synthesize images by following the inverse process. First, a noise sample  $I_{T}$  is drawn from  $\mathcal{N}(0,\mathbf{I})$ . Then  $\hat{\epsilon}_{\phi}$  is iteratively used to estimate the sequences of images  $\{I_{T - 1},I_{T - 2},\ldots ,I_0\}$  where  $I_0$  is the finally synthesized image.
Neural Radiance Fields A Neural Radiance Field (NERF) [13] is a parametric representation of a 3D scene as a continuous volume of emitted color  $c$  and density  $\sigma$ . Formally, it can be written as a mapping  $\mathcal{F}_{\theta}:(x)\to (c(x),\sigma (x))$  from the 3D coordinates of a point  $x$ , to its associated color and density. These radiance fields allow rendering of 2D images by accumulating color over points sampled on camera rays. For a camera ray  $\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}$ , the accumulated color is expressed as:
$$
C (\mathbf {r}) = \int_ {t} T (t) \sigma (r (t)) c (r (t)) d t \tag {1}
$$
where  $T(t) = \exp(-\int_0^t \sigma(r(s)) ds)$  is the accumulated transmittance of the volume along the ray. Learning an accurate representation  $\mathcal{F}_{\theta}$  of a 3D scene generally requires supervision in the form of several ground truth samples of  $C(\mathbf{r})$  i.e. several images of a scene taken from different viewpoints. Representing  $\mathcal{F}_{\theta}$  as a neural network has been shown to demonstrate several desirable properties like the ability to synthesize novel views and realistic high-resolution renderings of complex scenes.
Image Diffusion to Radiance Fields DreamFusion proposed an approach to leverage a pretrained image diffusion model and a text prompt to optimize a 3D radiance field. The key idea is to optimize parameters  $\theta$  such that the rendering of the radiance field from any viewpoint looks like a sample from the diffusion model. This is accomplished by randomly sampling camera poses  $p$  during training, rendering images from the radiance field for these viewpoints  $\bar{I}_{\theta}^{p}$  and using a Score Distillation Sampling objective to optimize the radiance field. The gradient of the Score Distillation Sampling (SDS) objective  $\mathcal{L}_{\mathrm{SDS}}$  is defined as:
$$
\nabla_ {\theta} \mathcal {L} _ {\mathrm {S D S}} (\phi , I _ {\theta} ^ {p}, y) = \mathbb {E} _ {t, e} \left[ w (t) \left(\hat {\epsilon} _ {\phi} \left(\sqrt {\alpha_ {t}} I _ {\theta} ^ {p} + \sqrt {1 - \alpha_ {t}} \epsilon , y, t\right) - \epsilon\right) \nabla_ {\theta} I _ {\theta} ^ {p} \right] \tag {2}
$$
where  $w(t)$  is a timestep dependent weighting function. We refer the readers to [1] for the derivation of this objective. In practice, each update is computed using a randomly sampled timestep  $t$  and noise sample  $\epsilon$ . Intuitively, this is equivalent to first perturbing the rendered image using  $\epsilon, t$  and then updating the radiance field using the difference between the diffusion model estimated noise and  $\epsilon$ .
The goal of this work is to optimize a 3D radiance field  $\mathcal{F}_{\theta}$  to capture the visible appearance and geometry of an object depicted in an input image, while inferring a realistic reconstruction of the unknown/hidden parts. Let the input image be represented by  $\hat{I}$  and  $\hat{p}$  be a reference camera pose (which can be arbitrarily chosen) associated with the image. Let  $I_{\theta}^{p}$  be the image obtained as a differentiable rendering of  $\mathcal{F}_{\theta}$  viewed from camera pose  $p$ . The optimal desired representation  $F_{\hat{\theta}}$  should satisfy two criteria: (i)  $I_{\hat{\theta}}^{\hat{p}} = \hat{I}$ , and (ii) For all viewpoints  $p$ ,  $I_{\hat{\theta}}^{p}$  should be semantically and geometrically consistent.
A simple approach for satisfying requirement (i) could be to optimizing an  $L_{2}$  distance objective  $I_{\theta}^{\hat{p}}$  and  $\hat{I}$ . Furthermore, Score Distillation Sampling (see Section 3) can be used to satisfy requirement (ii) by optimizing randomly rendered viewpoints using diffusion model priors. This approach forms the basis of concurrent works RealFusion[14] and NeuralLift360[15]. We observe in experiments (Sec 5) that this approach often leads to misaligned final appearance of novel viewpoints.
4.2 ConRad: Image Constrained Radiance Fields
We propose a novel variant of neural radiance fields called Image Constrained Radiance Fields (ConRad) that allows us to effectively satisfy the two objectives. Intuitively, we wish to formulate a radiance field that accurately depicts the input image in one viewpoint while allowing us to optimize and infer the other viewpoints. First, we note that the input image depicts a "visible" part of an object's surface. Under some assumptions on the object properties<sup>2</sup>, we can make inferences about the 3D space around this object. Any point that lies on/in front of the visible surface is known to us and does not need to be re-inferred in our radiance field. Specifically, points between the reference camera and the surface should have zero density, and points on the surface should have high density
2The proposed constraints rely on the assumption that the object is opaque.
and color equal to the corresponding pixel in the input image. Unfortunately, since we do not have access to the depth map corresponding to the reference view, it is not possible to infer which 3D points lie in front or behind the surface. As a workaround, we leverage the radiance field to obtain an estimate of the depth. Using this estimated depth, we propose to incorporate these observations into the color and density of the radiance field to leverage the input image as an explicit constraint.
Concretely, let the ray corresponding to pixel  $(i,j)$  in the reference view be  $\mathbf{r}_{\hat{p}}^{(i,j)}(t) = \mathbf{o}_{\hat{p}} + t\mathbf{d}_{\hat{p}}^{(i,j)}$ . We define the visibility depth of this pixel  $V_{\hat{p}}[i,j]$  as the value of  $t$  such that
$$
1 - \frac {\int_ {0} ^ {t} T (t) \sigma \left(r _ {p} ^ {(i , j)} (t)\right)}{\int_ {s = 0} ^ {\infty} T (s) \sigma \left(r _ {p} ^ {(i , j)} (s)\right)} = \eta \tag {3}
$$
where  $\eta$  is a small value set to 0.1 in our experiments. Intuitively, the visibility depth for each pixel is a point on the ray beyond which the contribution of color is minimal (less than  $10\%$ ).
Let  $Q_{p}:\mathbb{R}^{3}\to [-1,1]\times [-1,1]$  be the camera projection matrix corresponding to projection from world coordinates to the normalized reference image coordinates. We can reformulate the color  $c(x)$  of the radiance field  $\mathcal{F}_{\theta}$  as follows:
$$
v _ {x} = \mathbb {1} \left(\left\| x - \mathbf {0} _ {\hat {p}} \right\| <   V _ {\hat {p}} \left[ Q _ {\hat {p}} (x) \right]\right) \tag {4}
$$
$$
c ^ {\prime} (x) = v _ {x} * I ^ {\hat {p}} \left(Q _ {\hat {p}} (x)\right) + (1 - v _ {x}) * c (x) \tag {5}
$$
where we use bilinear interpolation to compute pixel values  $I^{\hat{p}}(Q_{\hat{p}}(x))$ . This constraint enforces the appearance of the reference viewpoint to explicitly match the image. Since we only estimate the depth of each pixel based on a potentially incorrect density field, we enforce that all points in between the camera and the estimated surface along the ray should have color equal to the corresponding pixel. As shown in Figure 1, this still constrains the reference view to match the input image but allows the density to be optimized through our training process.
Additionally, the foreground mask of the input image informs the density of the radiance field. We estimate the binary foreground mask  $\hat{M}$  using a pretrained foreground estimation method[35]. We know that any point on the reference rays corresponding to the background pixels should have zero density. We reformulate the density  $\sigma(x)$  as:
$$
m _ {x} = \hat {M} \left[ Q _ {\hat {p}} (x) \right] \tag {6}
$$
$$
\sigma^ {\prime} (x) = m _ {x} * \sigma (x) \tag {7}
$$
In summary, ConRad is the radiance field defined by the constrained color  $c'(x)$  and density  $\sigma'(x)$ .
4.3 Optimizing Image Constrained Radiance Fields
In this section, we present our approach for training a ConRad representation using a single input image  $\hat{I}$ . First, we preprocess the input image to extract several supervisory signals. Since we wish to leverage a text-conditioned diffusion model, we need to generate text embeddings for a caption of the input image. We use the caption "a photo of a <token>" where the embedding of the special token is inferred using Textual Inversion[36]. In order to learn an embedding that accurately captures the input image, we pass in multiple synthetically generated augmentations of the input image by randomly cropping, resizing and blurring it. Due to space constraints, we presented more details in the supplementary material. We also estimate a depth map for the reference view  $\hat{D}$  using MiDaS[37].
Diffusion Prior One of the key advantages of ConRad is the simplicity of the training pipeline. Since ConRad already incorporates the appearance of the reference view, the training algorithm has minimal deviations from DreamFusion. We simply compute SDS updates on random viewpoints using the pretrained Diffusion Model conditioned on the previously obtained text embeddings and update the radiance field to infer the unknown density and color regions.
Depth Loss Since the depth of the reference viewpoint is unknown, we find that providing additional supervision through the estimated reference depth map  $\hat{D}$  sometimes improves the results of our training. The preprocessed depth estimate  $\hat{D}$  only provides relative depth values[38]. Therefore, the ground truth depth should be close to  $\hat{D}$  up to a scalar scaling and translation factor. We accommodate
Figure 2: ConRad Image-to-3D Reconstruction: Here we visualize the 3D structures generated by our proposed approach using images taken from different sources. We observe that the proposed ConRad representation leads to high quality realistic reconstructions of the objects while completely preserving input image details. Please see Appendix for comparison to RealFusion and NeuralLift360 on these input images.
this in the depth loss  $\mathcal{L}_{\mathrm{dep}}$  by formulating it as the Pearson Correlation coefficient between  $D$  and  $\hat{D}$ . NeuralLift360[15] adopts a similar idea and uses a ranking loss for providing depth supervision.
Warm Start The visibility and mask values in Eq 4 & 6 are non-differentiable and also cause a sharp boundary in the radiance field. For some input objects, we observe that this disrupts training. This can easily be resolved by performing a "warm start". We multiply the visibility and mask scores in Eq 4 & 6 with a scalar  $\alpha$  that is linearly annealed from 0 to 1 over the first  $50\%$  of the updates and then kept constant at 1.
The simplicity of Image Constrained Radiance Fields representations facilitates a straightforward implementation of the training process. We use Instant-NGP[18] as the base representation comprising of a shared multi-resolution hash encoding and two separate small MLPs for the unconstrained color  $c(x)$  and density  $\sigma(x)$  respectively. For all experiments in this paper, we choose the reference view camera to lie at a distance of 3.2 from origin, azimuth angle 0 and elevation 0 except where specified. We precompute and store rays for the camera corresponding to this reference view. Before each update, we estimate the Visibility Depth (Eq 4) by evaluating on points along these rays and choosing the closest solution. For each update, a random camera pose  $p$  is obtained by uniformly sampling elevation angle in  $[-15^\circ, 45^\circ]$ , azimuth angle in  $[0^\circ, 360^\circ]$  and distance to origin in [3, 3.5]. We then render the image  $I_p$  for this viewpoint using the constrained color  $c'(x)$  and density  $\sigma'(x)$ . We also compute the estimated reference view depth  $D_{\hat{p}}$  using the precomputed rays and radiance field density  $\sigma'(x)$ . We compute the SDS sampling update (Eq 2) using  $I_p$  and the gradient of the Depth Loss  $D_{\hat{p}}$  to update all the parameters. Additionally, the regularizations proposed in [1] are adopted to enforce smoothness of surface normals and encourage outward facing surface normals in the radiance field. We keep all the hyperparameters unchanged for all experiments except when explicitly indicated (Sec 5.4). Please refer to the supplementary material for details of regularizations and their associated weights, hyperparameters of Instant-NGP, optimizer hyperparameters and pseudo-code of the training algorithm.