SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note that we do not need to render the image of the reference view during training. This eliminates the need for tuning viewpoint sampling strategies and additional reference loss hyperparameters, leading to a simpler and more robust training pipeline. Computation of visibility depth also does not significantly increase GPU memory consumption since we do not compute its gradients.
5.2 Visualizing 3D Reconstructions
We first visualize the final 3D reconstructions of ConRad representations trained on single images taken from different sources. In Figure 2, we present the RGB and surface normal reconstructions of the reference view and three novel viewpoints. Observe that the final representation always accurately reconstructs the input viewpoint due to the constraints incorporated in ConRad. Furthermore, the novel viewpoints generally depict a realistic reconstruction of the input object.
We also observe that the reconstructions are faithful to the specific instance of the object, presenting a smooth transition from reference viewpoint to unobserved viewpoints. This can be attributed to two aspects of the model: (i) The textual inversion embedding passed to the diffusion model captures the details of the object and (ii) Since ConRad depicts the input image accurately from beginning of training, it provides a strong prior for the appearance for the diffusion model, especially around the reference viewpoint. In Figure 2, "Novel View 1" presents a viewpoint close to the reference view.
Our proposed approach can also be used to generate 3D model from text prompts. To accomplish this, we can leverage pretrained Stable Diffusion to first generate a 2D image from the text prompt and then use ConRad to learn a 3D reconstruction of this object. Figure 2 Rows 3 & 4 demonstrate this capability on two prompts.
5.3 Comparison to existing work
RealFusion[14] relies on alternating between rendering the reference viewpoint and a random viewpoint. For the reference viewpoint, additional mean squared error objectives are used to distill
Figure 3: Qualitative comparisons: We perform qualitative comparisons of our approach to state-of-the-art image to 3D generation methods. For fair comparison and to demonstrate robustness, we evaluate on images taken from the respective papers. We observe that ConRad is able to generate higher quality 3D models while more accurately preserving input image details.
the reference RGB and foreground mask values into the 3D representation. For other viewpoints, Score Distillation Sampling is used to update the representation. Similar to our approach, textual inversion is used to compute the conditioning text embedding for Stable Diffusion.
NeuralLift360[15] proposes a similar approach to RealFusion. In addition to reference RGB and foreground mask, NeuralLift360 also utilizes an estimated depth map and the CLIP features of the input image as supervision. A ranking loss is used to account for the scale and translation ambiguity in the estimated depth map. This is similar in spirit to our proposed depth objective in Section 4.3. NeuralLift360 also encourages the CLIP features of renderings from all viewpoints to be consistent with the input viewpoint.
Qualitative Comparisons In Figure 3, we present qualitative comparisons to ConRad. While both RealFusion and NeuralLift360 released official implementations, we observed that they require tuning of hyperparameters or fail to generate meaningful reconstructions for several objects. For fair comparison, we evaluate our method on images taken from the respective papers. We observe
ConRad can better capture the details of the input image consistently. In contrast, RealFusion and NeuralLift360 often generate a similar instance of the same category with a different appearance.
Evaluation Metrics The task of image to 3D generation is difficult to quantitatively evaluate due to the inherent ambiguity in the expected output i.e. there are several possible realistic reconstructions of the same object. NeuralLift360[15] evaluates the ability to capture semantic content by measuring the CLIP feature[4] distance between all viewpoints of the generated object and the single input reference image. We build upon this idea and propose metrics to evaluate the ability to generate different viewpoints of an object.
Table 1: Quantitative Comparisons: We perform quantitative comparisons using 3D object data of 20 objects depicting instances of 10 categories taken from the ShapeNet dataset. We evaluate the CLIP semantic similarity of rendered object viewpoints to ground truth viewpoint samples. We demonstrate that generating 3D models with ConRad leads to significant improvements over state-of-the-art across all metrics.
Method	All Views	Near Reference
dref	dall	doracle	dref	dall	doracle
PointE[39]	0.496	0.482	0.453	0.496	0.499	0.399
RealFusion[14]	0.495	0.486	0.460	0.483	0.482	0.464
NeuralLift-360[15]	0.528	0.516	0.498	0.543	0.544	0.534
ConRad (ours)	0.332	0.316	0.273	0.286	0.257	0.230
First, we render 20 objects from 10 categories of the ShapeNet[17] dataset viewed from 68 different camera poses to create a ground truth (GT) set. We choose a front-facing view from each object as input to an Image-to-3D approach. We then render the generated object from the same 68 camera poses. Due to ambiguity of depth, corresponding camera poses between GT and rendered images could depict very different object poses. Using these two sets of images, we compute three metrics -  $d_{ref}$  is the mean CLIP feature distance between the reference image and all the rendered viewpoints (same as [15]).  $d_{all}$  is the mean CLIP feature distance between all pairs of GT and rendered images.  $d_{oracle}$  is the solution to a linear sum assignment problem[40] where the cost of assigning a GT view to a rendered image is the CLIP feature distance between them. This evaluates the ability of the representation to generate images as diverse as the ground truth while preserving semantic content.
We evaluate these three metrics for the two sets of 68 images ("All Views"). We also evaluate on a subset of camera poses that lie within a  $15^{\circ}$  elevation change and  $45^{\circ}$  azimuth change ("Near Reference") giving us 15 images each for ground truth and rendered images. This measures the semantic consistency in viewpoints where parts of the input image are visible.
Figure 4: Ablation Study: We present a qualitative investigation of the importance of various components proposed in our approach on two prototypical examples. The components are arranged left to right in decreasing order of importance for final reconstruction quality.
5.4 Analysis of ConRad Components
Quantitative Evaluation In Table 1, we compare ConRad to RealFusion, NeuralLift360, and PointE. PointE[39] is a point cloud diffusion model was trained on several million 3D models. It can directly generate point clouds using CLIP features of the input image. Since there is no corresponding reference view in the output, we synthesize 150 random views for PointE instead of the 68 chosen views. For all methods, we use the implementations and hyperparameters provided by the authors. We observe that ConRad outperforms these methods across all the metrics by a significant margin. We present a per-object breakdown of these metrics in the supplementary material.
We now investigate the various components of ConRad to understand their significance. In Figure 4, we present a visualization of the effect of removing each component on two typical examples. In most experiments, we observe that performing a warm start is not necessary but leads to crisper final 3D structure. We also observe that removing the Depth Loss significantly reduces computational cost, but this leads to incorrect 3D structure for some objects (see row 2 of Figure 4). Textual Inversion is crucial to maintain consistent appearance of the object from all viewpoints. Moreover, removing the color constraint (Eq 4) generally still leads to a realistic 3D model but depicts an arbitrary object. Finally, removing both color and density constraints (still using the depth loss) leads to very unrealistic objects.
5.5 Failure Cases and Limitations
While ConRad significantly improves the simplicity and robustness of learning the 3D structure, it occasionally demonstrates issues that are common to Image-to-3D methods. In Figure 5, we present two typical examples of failure cases. Row 1 demonstrates the Janus effect, where the final 3D model has two faces. We also observe that performing score distillation sampling using Stable Diffusion leads to saturated colors in the 3D model. Leveraging textual inversion embeddings mitigates this issue to some extent, but still occasionally leads to incorrect appearance in novel viewpoints, as demonstrated in Row 2. We also observe that for a few objects, parts of the generated 3D object is semi-transparent (see videos in supplementary material).
Novel Front View
Novel Back View
Figure 5: Failure Cases: On some objects, our proposed approach suffers from the Janus effect and is affected by the bias of Stable Diffusion to produce saturated colors.
In this work, we present a novel radiance field variant, ConRad, that significantly simplifies the training pipeline for generating 3D models from a single image. By explicitly incorporating the input image into the color and density fields, we show that we can eliminate tedious tuning of additional hyperparameters. We demonstrate that ConRad representations lead to more accurate depiction of the input image, produce higher quality and more realistic 3D models compared to existing work.
[39] A. Nichol, H. Jun, P. Dhariwal, P. Mishkin, and M. Chen, "Point-e: A system for generating 3d point clouds from complex prompts," arXiv preprint arXiv:2212.08751, 2022.
[40] H. W. Kuhn, “The hungarian method for the assignment problem,” Naval research logistics quarterly, vol. 2, no. 1-2, pp. 83–97, 1955.
A Additional Implementation Details
Here we present more details about the implementation of ConRad representation to facilitate reproducibility of results.
Base Representation As indicated in the main text, we use the Instant-NGP[18] representation followed by two MLPs to model the density  $\sigma(x)$  and color  $c(x)$ . For the multi-resolution hash encoding of Instant-NGP representation, we use 16 levels with a 2-dimensional encoding at each level. The Instant-NGP encoding is passed to two 3-layer MLPs with a hidden dimension of 64. The color MLP output is passed through a sigmoid activation to obtain the RGB values additionally. The density MLP output is passed to the exponential  $e^x$  activation to obtain the density value.
Regularization Losses In addition to SDS and the Depth Loss, we use three regularizations proposed in [1] to produce coherent objects. We encourage the radiance field to have either very low or very high density at any point along the rays by using the following entropy regularizer:
$$
\mathcal {L} _ {e n t} = \sum_ {x} - \alpha (x) * \log \alpha (x) - (1 - \alpha (x)) \log (1 - \alpha (x)) \tag {8}
$$
where  $\alpha (x)$  is the rendering weight at point  $x$
We encourage the surface normals in each rendered view to point towards the camera using the orientation regularizer:
$$
\mathcal {L} _ {\text {o r i e n t}} = \sum_ {x} \alpha (x) * \max  (<   \mathbf {n} (x), \mathbf {d} >, 0) ^ {2} \tag {9}
$$
where  $\mathbf{n}$  is the normal at the point  $x$  computed using the finite differences method with the density  $\sigma(x)$  and  $\mathbf{d}$  is the viewing direction.
Finally, we encourage the smoothness of normals computed at each point along the rays using the smoothness regularizer:
$$
\mathcal {L} _ {\text {s m o o t h}} = \sum_ {x} | \mathbf {n} (x) - \mathbf {n} (x + \delta) | \tag {10}
$$
where  $\delta$  is a random perturbation for each point with a maximum perturbation of 0.01 along each axis.
Algorithm 1: Optimizing a ConRad representation to reconstruct 3D from a single image.
Data: Image  $\hat{I}$  , Estimated Depth  $\hat{D}$  , Foreground Mask  $\hat{M}$  , Reference Pose  $\hat{p}$    
1 Initialize ConRad  $\mathcal{F}_{\theta}$    
2 Compute reference rays  $r_{\hat{p}}$  .   
3 Compute text embeddings  $t$  using textual inversion on image  $\hat{I}$    
4 for  $i = 0$  to 5000 do   
5  $\hat{V}\gets$  Visibility Depth using  $r_{\hat{p}}$  and  $\sigma_{\theta}(x)$  (Equation 3);   
6  $D_{\hat{p}}\gets$  Radiance Field Depth along rays  $r_{\hat{p}}$  using  $\sigma^{\prime}(x)$  ;   
7  $\mathcal{L}_{dep}\gets$  Pearson Correlation between  $D_{\hat{p}}$  and  $\hat{D}$  .   
8  $p\gets$  sample random camera pose;   
9  $r_p\gets$  compute rays for viewpoint  $p$  .   
10  $I_{p}\gets$  Render reference viewpoint image using  $r_{\hat{p}},c^{\prime}(x),\sigma^{\prime}(x)$  .   
11  $\nabla_{\theta}\mathcal{L}_{\mathrm{SDS}}(\phi ,I_{\theta}^{p},t)\gets$  Using Stable Diffusion, text embedding t and  $I_{p}$  (Eq. 2);   
12  $\mathcal{L}_{\text{orient}}\gets$  Compute orientation regularization along  $r_p$  .   
13  $\mathcal{L}_{\text{smooth}}\gets$  Compute smoothness regularization along  $r_p$  .   
14  $\mathcal{L}_{ent}\gets$  Compute entropy regularization along  $r_p$  .   
15  $\theta \gets \theta -\eta (\nabla_{\theta}\mathcal{L}_{\mathrm{SDS}}(\phi ,I_{\theta}^{p},t) + \nabla_{\theta}(10*\mathcal{L}_{\text{depth}} + 0.01*\mathcal{L}_{\text{ent}} + 0.01*\mathcal{L}_{\text{orient}} + 10*\mathcal{L}_{\text{smooth}}));$
Loss Weighting The above losses are all jointly optimized using the following weighted sum:
$$
\mathcal {L} = \mathcal {L} _ {\mathrm {S D S}} + 1 0 * \mathcal {L} _ {\text {d e p t h}} + 0. 0 1 * \mathcal {L} _ {\text {e n t}} + 0. 0 1 * \mathcal {L} _ {\text {o r i e n t}} + 1 0 * \mathcal {L} _ {\text {s m o o t h}} \tag {11}
$$
In experiments, we observed that the orientation and entropy regularizer had minimal effect on the final output and could be turned off on most input images without any loss in quality. However, we retain these regularizers in all experiments to ensure similarity to DreamFusion.
Textual Inversion Since we aim to use only an image as input, we need to synthesize a text prompt to pass as conditioning to the text-conditioned diffusion model. Additionally, we need the text-prompt to be sufficiently descriptive to capture all the details of the specific object instance. Most pretrained captioning algorithms provide a coarse caption that doesn't capture such details. Therefore, in recent research, Textual Inversion[36] proposed an approach that uses input images and a diffusion model to infer the text embedding of a special token ("<token>"). This token along with its learned embedding can be passed to the diffusion model to synthesize novel images of the same object. This approach generally requires 3 to 5 images to accurately capture an object. Since we have access to only one image, we rely on synthetically augmenting the image to run Textual Inversion. Specifically, we generate images by randomly flipping horizontally with probability 0.5, extract a random crop covering  $50\%$  to  $100\%$  of the image, Gaussian blurring the image with a kernel size 5 and standard deviation randomly sampled in [0.1, 2] and jittering the hue, saturation, contrast and brightness by a random value in [0, 0.1]. Furthermore, we perform classification of the input image using a pretrained CLIP model[4]. The text embedding of the obtained class label is used to initialize the embedding of the special token "<token)".
Optimization We use the Adan optimizer with 0.005 learning rate and weight decay of 0.00002. We use a batch size of 1 for each update i.e. one random viewpoint. We keep the learning rate fixed and train the representation for 5000 updates. The optimization process takes approximately 20 minutes on a single A100 GPU.
B Additional Quantitative Evaluation
In Sec 5.3 of the main text, we present quantitative comparisons of our work to RealFusion[14], NeuralLift-360[15] and Point-E[39] on objects from the ShapeNet[17] dataset. Here we present more details on this evaluation to facilitate easy reproduction.
Evaluation Data We choose 20 objects from 10 categories. While we randomly sampled objects, we manually removed some non-prototypical objects like a triangular bed. In Figure 6, we present the chosen input image for each object, the ShapeNet object ID and category.
Table 2: Quantitative Comparisons: We present quantitative comparisons using 3D object data of 20 objects the ShapeNet dataset. We present an object-wise breakdown of the performance for each method. The performance is evaluated on all (68) views of the object. Please refer to the main text for more details on the evaluation metrics.
Category	Object ID	\(d_{ref}\)	\(d_{all}\)	\(d_{oracle}\)
RealFusion	NeuralLift-360	Point-E	ConRad	RealFusion	NeuralLift-360	Point-E	ConRad	RealFusion	NeuralLift-360	Point-E	ConRad
airplane	2c9797...	0.418	0.753	0.412	0.251	0.462	0.833	0.441	0.263	0.432	0.833	0.420	0.236
airplane	7d89d6...	0.539	0.543	0.426	0.246	0.658	0.676	0.513	0.279	0.639	0.668	0.487	0.238
bench	5d9880...	0.642	0.737	0.507	0.374	0.583	0.710	0.457	0.355	0.557	0.692	0.427	0.297
bench	62cc45...	0.575	0.585	0.395	0.272	0.599	0.598	0.408	0.309	0.578	0.592	0.366	0.249
bus	2ba84d...	0.533	0.541	0.564	0.335	0.492	0.548	0.501	0.329	0.461	0.537	0.469	0.259
bus	642b3d...	0.485	0.531	0.571	0.356	0.500	0.474	0.574	0.340	0.462	0.416	0.540	0.275
bus	6cc0a...	0.654	0.588	0.582	0.340	0.610	0.534	0.523	0.295	0.587	0.502	0.490	0.252
car	1abeca...	0.490	0.458	0.578	0.469	0.487	0.455	0.551	0.401	0.460	0.411	0.517	0.350
car	35155f...	0.514	0.396	0.527	0.368	0.504	0.340	0.527	0.311	0.481	0.300	0.501	0.265
car	9807c1...	0.543	0.449	0.582	0.371	0.503	0.377	0.561	0.281	0.482	0.347	0.535	0.238
chair	8ce2e4...	0.562	0.837	0.508	0.370	0.470	0.773	0.432	0.304	0.446	0.773	0.395	0.269
chair	ea9181...	0.548	0.517	0.485	0.259	0.522	0.495	0.471	0.248	0.500	0.490	0.452	0.208
flowerpot	8e7642...	0.338	0.505	0.497	0.319	0.299	0.483	0.481	0.307	0.283	0.469	0.467	0.290
flowerpot	f25999...	0.399	0.252	0.467	0.198	0.446	0.288	0.491	0.227	0.425	0.264	0.471	0.186
guitar	4c082a...	0.440	0.402	0.558	0.335	0.363	0.340	0.474	0.304	0.336	0.313	0.435	0.274
motorbike	61b17f...	0.407	0.586	0.482	0.327	0.391	0.597	0.508	0.297	0.353	0.593	0.483	0.252
motorbike	6e3761...	0.409	0.395	0.521	0.270	0.410	0.348	0.512	0.267	0.379	0.314	0.491	0.230
sofa	6f8494...	0.551	0.556	0.458	0.328	0.517	0.516	0.424	0.337	0.493	0.512	0.386	0.281
table	5d9f9e...	0.481	0.471	0.387	0.294	0.503	0.478	0.395	0.297	0.477	0.474	0.365	0.252
table	bdd12e...	0.380	0.458	0.406	0.560	0.399	0.457	0.394	0.568	0.376	0.451	0.370	0.562
Detailed Quantitative Results In Table 2, we present an object-wise breakup of the results presented in the main text. We observe that for most objects, ConRad based training leads to significantly improved results on all metrics.
Figure 6: ShapeNet Evaluation Data: We present the 20 input images used to evaluate performance ConRad and existing works in this domain.
C Additional Qualitative Comparisons
In Figure 2, we present visualizations of 3D models produced by ConRad. Here we present visualizations for the 3D models produced by RealFusion[14] and NeuralLift360[15] using the same input images in Figure 7. We observe that both methods generally produce low quality generations when arbitrary images are used as inputs. We found that NeuralLift360 was successful more frequently compared to RealFusion but produced inferior 3D models compared to ConRad.
NeuralLift-360 [15]
Figure 7: Additional Qualitative Baseline Results: We present additional qualitative baseline results using reference images presented in the main paper. For both NeuralLift-360 and RealFusion, we use the official implementations released by the authors.
RealFusion [14]