ConRad: Image Constrained Radiance Fields for 3D Generation from a Single Image
Authors: Senthil Purushwalkam
Nikhil Naik
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D
generation
diffusion
viewpoint

Abstract: We present a novel method for reconstructing 3D objects from a single RGB image. Our method leverages the latest image generation models to infer the hidden 3D structure while remaining faithful to the input image. While existing methods obtain impressive results in generating 3D models from text prompts, they do not provide an easy approach for conditioning on input RGB data. Naive extensions of these methods often lead to improper alignment in appearance between the input image and the 3D reconstructions. We address these challenges by introducing Image Constrained Radiance Fields (ConRad), a novel variant of neural radiance fields. ConRad is an efficient 3D representation that explicitly captures the appearance of an input image in one viewpoint. We propose a training algorithm that leverages the single RGB image in conjunction with pretrained Diffusion Models to optimize the parameters of a ConRad representation. Extensive experiments show that ConRad representations can simplify preservation of image details while producing a realistic 3D reconstruction. Compared to existing state-of-the-art baselines, we show that our 3D reconstructions remain more faithful to the input and produce more consistent 3D models while demonstrating significantly improved quantitative performance on a ShapeNet object benchmark.

ConRad: Image Constrained Radiance Fields for 3D Generation from a Single Image
Senthil Purushwalkam
Nikhil Naík Salesforce AI Research
Salesforce AT Research
spurushwalkame sales forcee..e.ceeooooo.
maake@alesforce.com
Abstract
We present novel method for reconstructing 3D objects from a single RGB image. Our method leverages the latest image generation models to infer the hidden 3D structure while remaining faithful the input image. While existing methods[1, obtain impressive results in generating 3D models from text prompts, they do not provide an easy approach for conditioning on input RGB data. Naïve extensions of these methods often lead to improper alignment in appearance between the input image and the 3D reconstructions. We address these challenges by introducing Image Constrained Radiance Fields (ConRad), novel variant of neural radiance fields. ConRad is an efficient 3D representation that explicitly captures the appear- ance of an input image in one viewpoint. We propose training algorithm that leverages the single RGB image conjunction with pretrained Diffusion Models optimize the parameters of ConRad representation. Extensive experiments show that ConRad representations can simplify preservation of image details while producing realistic 3D reconstruction. Compared existing state-of-the-art base- lines, we show that our 3D reconstructions remain more faithful to the input and produce more consistent 3D models while demonstrating significantly improved quantitative performance on a ShapeNet object benchmark.
Introduction
Humans posses the ability to accurately infer the full 3D structure of an object even after observing just viewpoint. Since the RGB and depth observed for one viewpoint does not provide sufficient information, we have to rely on our past experiences to make intelligent inferences about a realistic reconstruction of the full 3D structure. This ability helps navigate and interact with the 3D world around us seamlessly. Capturing a similar capability of generating the full 3D structure from a single image has been long-standing problem in Computer Vision. Such systems could facilitate advances in robotic manipulation and navigation systems, and has applications video games, augmented reality and e-commerce. Despite the importance of this capability, success on solving this problem has been limited.
This lack of success can partly be attributed to the lack of useful large scale data. While have made progress on collecting large scale image datasets, the scale of 3D object or multi-view image datasets has been severely limited. The availability of billions of RGB images and captions [3] has facilitated significant advances semantic understanding of images [4-6]. Over the last few years in particular, approaches for generative modeling of the image manifold have demonstrated capabilities including controllable generation of highly realistic images [7, 8], inpainting of occluded images [9] and realistic image editing [10-12]. These success stories demonstrate the ability of the 2D generative models to capture prior knowledge about the visual world. In light of this evidence,
Project webpage: https: /www http://pp..ppppoo.......ooo///aa
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Regular Radiance Field Initialization
Input Image
ConRad
+
Figure 1: ConRad representation: We propose novel radiance field,ConRad, which allows conditioning on a single image. In contrast to regular radiance fields, our approach can accurately model the input image in one reference viewpoint of the radiance field without any training. Utilizing ConRad representations simplifies the optimization of a radiance field for generating a 3D model from single image.
ask the question-cen can this 2D prior knowledge be leveraged to infer the hidden 3D structure of objects?
DreamFusion [1] generates 3D models based on an input text prompt, using the distribution captured
by pretrained diffusion model update the parameters of neural radiance field (NERF) [13]. While DreamFusion demonstrates impressive capabilities, generating arbitrary instances of objects from text prompts severely limits its applications. recent concurrent work, RealFusion [14] and BeuralL.ift360 [15] extend DreamFusion to accommodate image input. These methods propose to optimize additional objectives to reconstruct the given input image in one reference viewpoint of the NERF while still relying on diffusion model to generate realistic reconstruction in novel viewpoints.
In this work, we propose a novel method for generating 3D object from single input image. Instead of designing additional objectives to reconstruct the input image, we propose rethink the underlying 3D representation. We introduce Image Constrained Radiance Fields (ConRad), novel variant of NERFs that can explicitly capture an input image without any training. Given an input image and a chosen arbitrary reference camera pose, ConRad utilizes multiple constraints to incorporate the appearance fields respectively. These constraints ensure that the rendering of the radiance field from the chosen viewpoint exactly matches the input image while allowing the remaining viewpoints be optimized.
object and the estimated foreground mask into the color and density
The proposed ConRad representation significantly simplifies the process of generating a consistent 3D model for the input image using any pretrained diffusion model. Since the representation accounts for the reference view, the training simply has to focus on distilling the diffusion model prior for the other viewpoints. Therefore, we show that we can leverage a training algorithm that has minimal deviations from DreamFusion. We propose a few key improvements to the training algorithm which leads to more robust training and higher quality 3D reconstructions.
We demonstrate results using images gathered from multiple datasets including CO3D [16]. ShapeNet [17], and images generated from pretrained generative models. ConRad produces 3DD reconstructions that are consistently faithful to the input image while producing high quality full 3D) reconstructions. In contrast RealFusion and NeuralLift360, ConRad has fewer hyperparameters and does not rely on balancing the trade off between image reconstruction vs satisfying the diffusion prior, leading more robust training pipeline. We conduct quantitative comparison objects from the ShapeNet dataset and observe that our approach demonstrates significant gains in reconstruction quality compared to of tottte-ff-hhaaat art methods.
Related Work
3D Representations Over the decade, research in Computer Vision and Graphies has led to development of efficient parametric representations of 3D scenes [13, 18, 19]. Traditional voxel

---

grid representations [20-22] maintain a dense grid of 3D scene parameters but consume a lot of memory. Neural Radiance Fields (NERFs) [13] offer contimuous parametric representation of the scene modeled as a neural network while requiring significantly lower memory while being computationally inefficient. Several approaches [23, 18] have been proposed to take advantage of both forms of representation. Instant-NGP [18] is one such approach that models 3D scene using a multi- resolution hash-encoding followed by a shallow neural network. Our proposed ConRad representation builds on Instant-NGP but can be used with any radiance field.
Learning Estimate 3D Structure To recover the full 3D structure from single image, many works propose to learn category 3D representations [24] then later fit to an input image. These approaches primarily relied on coarse representations like point clouds [24] and compositions of primitive shapes [25]. Furthermore, these methods focused on the geometry and could not infer the hidden appearance of the objects. NERFs demonstrated the capability to accurately infer novel viewpoints of a scene using densely sampled images. This has inspired numerous approaches to train similar radiance fields using sparse samples of a scene or object using semantic and geometric priors. [26-30] train models on large scale multi-view data to estimate radiance fields given single or few images. The limited diversity of available 3D data restricts the generalizability of these approaches.
Generating 3D from Pretrained 2D models Large scale 2D image data is much more readily available compared to 3D or muet data. This has led to significantly larger advances in modeling semantics in images [4-6] and generative modeling of images [7, 8]. In light of this, several works have attempted leverage the knowledge from these 2D models for the task of inferring the radiance fields from one or few images. Our proposed method belongs to this category of approaches since we rely on the distribution modeled by a pretrained image diffusion model.
DreamFields [31] generates 3D models from text prompts by optimizing the CLIP [4] distance between the NERF renderings and the textual CLIP embedding. DreamFusion [1] proposes an approach called Score Distillation Sampling (SDS) for distilling knowledge from a pretrained text-conditioned Diffusion Model [8] into a NERF representation (see Section for more details). Magic3D [2] proposes coarse-to-finn fine strategy that also leverages SDS to first train a NERF and then finetune mesh representation. Similar DreamFields, DietNERF [32] proposed to train a NERF on few input images while enforcing consistency of CLIP [4] features in other unknown views. However, this approach fails to generate 3D objects using a single image input. Pretrained image generative models have also been used to infer radiance fields in existing work [33] but only focuses on estimating geometry of the regions visible in the input image.
RealFusion [14] and NeuralLift360 [15] are two recent methods that are very relevant to our proposed approach. Inspired by DreamFusion, these approaches leverage a pretrained Stable Diffusion model to infer the appearance and geometry of unknown viewpoints and optimize several additional objectives to enforce consistency to the input image in one reference viewpoint. In concurrent work, Nerdi[34] also follows a similar approach of using reconsruction loss and depth loss for the known viewpoint, and uses diffusion model priors for unknown viewpoints. Unless properly tuned, these approaches lead to poor alignment between the input image and reconstructed 3D model (more details in Section 5). In contrast, ConRad focuses on modifying the underlying 3D representation to easily incorporate the input image and generates a more consistent 3D reconstruction.
3 Preliminaries
We first provide a concise summary of the prerequisite concepts from generative modeling of images and 3D objects that we build upon for ConRad.
recently developed generative model that synthesizes images by iteratively denoising a sample from a Gaussian distribution. For training diffusion model, noise is added a real image iteratively for T timesteps to synthesize training data {40, 41,.., } Specifically, the noised image for timestep t can be computed as I, Vaii I + /1 are where e E (0. I) and a is predefined noising schedule. This data is used to a train a denoising model Eo( y. t) which estimates the noise using the noisy image It, the timestep t and an optional embedding y of the caption associated with the image. A pretrained diffusion model can be used to synthesize images by following the inverse process. First, a noise sample IT is drawn from (N(0, I). Then é is iteratively used to estimate the sequences of images (TT-I, -1, TT- ... Jo) where To the finally synthesized image.
Diffusion Models A diffusion model
3

---

Neural Radiance Fields A Neural Radiance Field (NERF) [13] is parametric representation of a 3D scene as a continuous volume of emitted color and density σ. Formally, it can be written as mapping Fg (r) - (c(x)), or (x)) from the 3D coordinates of a point to it's associated color and density. These radiance fields allow rendering of 2D images by accumulating color over points sampled on camera rays. For camera ray r(t) = o ++ id, the accumulated color is expressed as:
C(r) - ((t)(((())))(((tt))tt
where T(I) exp(- (r()))ss) is the accumulated transmittance of the volume along the ray. Learning an accurate representation Fo of a 3D scene generally requires supervision in the form of several ground truth samples of C(r) i.e. several images of a scene taken from different viewpoints. Representing Fg as a neural network has been shown to demonstrate several desirable properties like the ability synthesize novel views and realistic high-resolution renderings of complex scenes.
Image Diffusion to Radiance Fields DreamFusion proposed an approach to leverage a pretrained image diffusion model and a text prompt to optimize a 3D radiance field. The key idea is to optimize parameters 8 such that the rendering of the radiance field from any viewpoint looks like a sample from the diffusion model. This is accomplished by randomly sampling camera poses p during training, rendering images from the radiance field for these viewpoints 1/ and using a Score Distillation Sampling objective optimize the radiance field. The gradient of the Score Distillation Sampling (SDS) objective Lass is defined as:
V VoLsus (0, 1/. y) = Bie / (otelelVa) + +VI a,,, - e) -))
where wt) is a timestep dependent weighting function. We refer the readers to [1] for the derivation of this objective. In practice, each update is computed using randomly sampled timestep t and noise sample e. Intuitively, this is equivalent to first perturbing the rendered image using c, f and then updating the radiance field using the difference between the diffusion model estimated noise and
4 Method
4.1
Problem Setup
The goal of this work is to optimize 3D radiance field Fg to capture the visible appearance and geometry of an object depicted in an imput image, while inferring realistic reconstruction of the unknown/hidden parts. Let the input image be represented by î and e be reference camera pose (which can be arbitrarily chosen) associated with the image. Let Th be the image obtained as a differentiable rendering of Fa viewed from camera pose p. The optimal desired representation F should satisfy two criteria: (f) I I. and (ii) For all viewpoints p, I should be semantically and geometrically consistent.
simple approach for satisfying requirement (i) could be to optimizing an L2 distance objective I and f. Furthermore, Score Distillation Sampling (see Section 3) can be used to satisfy requirement (ii) by optimizing randomly rendered viewpoints using diffusion model priors. This approach forms the basis of concurrent works RealFusion[ [4] and NeuralLitt360155] We observe in experiments (Sec 5) that this approach often leads to misaligned final appearance of novel viewpoints.
4.2 ConRad: Image Constrained Radiance Fields
We propose a novel variant of neural radiance fields called Image Constrained Radiance Fields (Con- Rad) that allows us to effectively satisfy the two objectives. Intuitively, we wish to formulate a radiance field that accurately depicts the input image in one viewpoint while allowing us to optimize and infer the other viewpoints. First, we note that the input image depicts a "visible" part of an object's is surface. Under some assumptions on the object properties", we can make inferences about the 3D space around this object. Any point that lies on/in front of the visible surface is known to us and does not need to be re-inferred in our radiance field. Specifically, points between the reference camera and the surface should have zero density, and points on the surface should have high density
"The proposed constraints rely on the assumption that the object opaque.

---

and color equal to the corresponding pixel in the input image. Unfortunately, since we do not have access to the depth map corresponding to the reference view, it is not possible to infer which 3D points lie in front or behind the surface. As workaround, we leverage the radiance field to obtain an estimate of the depth. Using this estimated depth, we propose to incorporate these observations into the color and density of the radiance field to leverage the input image as an explicit constraint.
Concretely, let the ray corresponding to pixel () in the reference view be (1.1) (t) 0p + td (1.1)
r We define the visibility depth of this pixel Vili. as the value of such that
Jo (())(r) (t))
J (1)) (s) =
(3)
where 17 is a small value set to 0.1 in our experiments. Intuitively, the visibility depth for each pixel is a point on the ray beyond which the contribution of color is minimal (less than 10%).
Let Qp : RR [-1,1] x 1- 1] be the camera projection matrix corresponding to projection from world coordinates to the normalized reference image coordinates. We can reformulate the color c(z) of the radiance field Fg as follows:
Us 1(1 opll [=])s()] d(x)= U (²)) + + (1-1v) Uz) * c(x)
(4)
(5)
where we use bilinear interpolation to compute pixel values (Q₂)))))) This constraint enforces the appearance of the reference viewpoint to explicitly match the image. Since we only estimate the depth of each pixel based on a potentially incorrect density field, we enforce that all points in between the camera and the estimated surface along the ray should have color equal the corresponding pixel. As shown in Figure 1, this still constrains the reference view to match the input image but allows the density to be optimized through our training process.
Additionally, the foreground mask of the input image informs the density of the radiance field. We estimate the binary foreground mask M using a pretrained foreground estimation m35hod[35]. We know that any point on the reference rays corresponding to the background pixels should have zero density. We reformulate the density a (I) as:
mz = [x)]] d' (x) = m2 * (x)
(6)
(7)
In summary, ConRad is the radiance field defined by the constrained color c'(x) and density o' (zx).
4.3 Optimizing Image Constrained Radiance Fields
In this section, we present our approach for training a ConRad representation using single input image Í. First, we preprocess the input image to extract several supervisory signals. Since we wish to leverage a text-conditioned diffusion model, we need to generate text embeddings for caption of the input image. We use the caption "a photo of <token>" " where the embedding of the special token is inferred using Textual Inversion[36]. In order to learn embedding that accurately captures the input image, we pass in multiple synthetically generated augmentations of the input image by randomly cropping, resizing and blurring it. Due space constraints, we presented more details in the supplementary material. We also estimate a depth map for the reference view D using MiDaS[37].
Diffusion Prior
One of the key advantages of ConRad is the simplicity of the training pipeline. Since ConRad already incorporates the appearance of the reference view, the training algorithm has minimal deviations from DreamFusion. We simply compute SDS updates random viewpoints using the pretrained Diffusion Model conditioned on the previously obtained text embeddings and update the radiance field to infer the unknown density and color regions.
Depth Loss Since the depth of the reference viewpoint is unknown, we find that providing additional supervision through the estimated reference depth map H sometimes improves the results of our training. The preprocessed depth estimate D only provides relative depth values[38]. Therefore, the ground truth depth should be close to D up to scalar scaling and translation factor. We accommodate
5

---

Input Image
Input View
Novel View
Novel View
Novel View
CO3D Dataset
CO3D Dataset
Stable Drilusion "A photo frommen"
Stabll Dillusion "A photo of stuffed dinosaur"
ShapeNet Dataset
Figure ConRad Image-to-3D Reconstruction: Here we visualize the 3D structures generated by our proposed approach using images taken from different sources. We observe that the proposed ConRad representation leads to high quality realistic reconstructions of the objects while completely preserving input image details. Please see Appendix for comparison to RealFusion and NeuralLift360 on these input images.

---

this in the depth loss deep by formulating it as the Pearson Correlation coefficient between D and D. 56 adopts a similar idea and uses a ranking loss for providing depth supervision.
Warm Start The visibility and mask values in Eq 4 & 6 are non-differentiable and also cause a sharp boundary in the radiance field. For some input objects, we observe that this disrupts training. This can easily be resolved by performing a "warm start". We multiply the visibility and mask scores in Eq 4 & with a scalar a that linearly annealed from over the first 50% of the updates and then kept constant at I.
Experiments and Results
5.1 Implementation Details
The simplicity of Image Constrained Radiance Fields representations facilitates a straightforward implementation of the training process. We use Instant-NGP[ as the base representation comprising of shared multi-resolution hash encoding and two separate small MLPs for the unconstrained color c(z) and density (z)) respectively. For all experiments in this paper, we choose the reference view camera to lie at distance of 3.2 from origin, azimuth angle O and elevation 0 except where specified. We precompute and store rays for the camera corresponding to this reference view. Before each update, we estimate the Visibility Depth (Fiq 4) by evaluating on points along these rays and choosing the closest solution. For each update, a random camera pose p is obtained by uniformly sampling elevation angle in [-15,, 45%), azimuth angle in [0 360%] and distance to origin in [3, 3.5]. We then render the image I, for this viewpoint using the constrained color e' (x) and density a d'(x). We also compute the estimated reference view depth D3 using the precomputed rays and radiance field density o' (x). We compute the SDS sampling update (Eq 2) using Ip and the gradient of the Depth Loss D3 update all the parameters. Additionally, the regularizations proposed in [1] are adopted to surface normals and encourage outward facing surface normals in the radiance field. We keep all the hyperparamters unchanged for all experiments except when explicitly indicated (Sec 5.4). Please refer to the supplementary material for details of regularizations and their associated weights, hyperparameters of Instant-NGP, optimizer hyperparameters and pseudo-code of the training algorithm.
enforce smoothness
Note that we do not need to render the image of the reference view during training. This eliminates the need for tuning viewpoint sampling strategies and additional reference loss hyperparameters, leading to simpler and more robust training pipeline. Computation of visibility depth also does not significantly increase GPU memory consumption since we do not compute its gradients.
5.2 Visualizing 3D Reconstructions
We first visualize the final 3D reconstructions of ConRad representations trained on single images taken from different sources. In Figure 2, we present the RGB and surface normal reconstructions of the reference view and three novel viewpoints. Observe that the final representation always accurately reconstructs the input viewpoint due the constraints incorporated in ConRad. Furthermore, the novel viewpoints generally depict a realistic reconstruction of the input object.
We also observe that the reconstructions are faithful to the specific instance of the object, presenting smooth transition from reference viewpoint to unobserved viewpoints. This can be attributed to two aspects of the model: (i) The textual inversion embedding passed to the diffusion model captures the details of the object and (ii) Since ConRad depicts the input image accurately from beginning of training. it provides strong prior for the appearance for the diffusion model, especially around the reference viewpoint. In Figure 2, "Novel View 1" presents a viewpoint close to the reference view.
Our proposed approach can also be used to generate 3D model from text prompts. To accomplish this, we can leverage pretrained Stable Diffusion to first generate 2D image from the text prompt and then use ConRad to learn a 3D reconstruction of this object. Figure 2 Rows 3 & 4 demonstrate this capability on two prompts.
5.3 Comparison to existing work
RealFusion 14] relies on alternating between rendering the reference viewpoint and random viewpoint. For the reference viewpoint, additional mean squared error objectives are used to distill
7

---

RealFusion Melas-Kyriazi et. al
ConRad
Jalan Feom paper
NeuralLift-366 et.
ConRad
peper)
Figure Qualitative comparisons: We perform qualitative comparisons of our approach to state- of-the-art image to 3D generation methods, For fair comparison and to demonstrate robustness, we evaluate on images taken from the respective papers. We observe that ConRad able to generate higher quality 3D models while more accurately preserving input image details.
the reference RGB and foreground mask values into the 3D representation. For other viewpoints, Score Distillation Sampling is used to update the representation. Similar to our approach, textual inversion is used to compute the conditioning text embedding for Stable Diffusion.
NeuralLin336(15) proposes a similar approach to RealFusion. In addition to reference RGB and foreground mask, NeuralLift360 also utilizes estimated depth map and the CLIP features of the input image as supervision. ranking loss is used to account for the scale and translation ambiguity in the estimated depth map. This is similar in spirit to proposed depth objective in Section 4.3. Neurall 3f0 also encourages the CLIP features of renderings from all viewpoints to be consistent with the input viewpoint.
Qualitative Comparisons In Figure 3, we present qualitative comparisons to ConRad. While both RealFusion and Neurall released official implementations, we observed that they require tuning of hyperparameters fail to generate meaningful reconstructions for several objects. For fair comparison, evaluate our method images taken from the respective papers. We observe

---

ConRad can better capture the details of the input image consistently. In contrast, RealFusion and NeuralLift360 often generate similar instance of the same category with a different appearance.
Evaluation Metrics The task of image to 3D generation is difficult to quantitatively evaluate due to the inherent ambiguity in the expected output L.e. there are several possible realistic reconstructions of the same object. NeuralLitt360(15) evaluates the ability to capture semantic content by measuring the CLIP [eature[4] distance between all viewpoints of the generated object and the single input reference image. We build upon this idea and propose metrics to evaluate the ability to generate different viewpoints of an object.
Table 1: Quantitative Comparisons: We perform quantitative comparisons using 3D object data of 20 objects depicting instances of 10 categories taken from the ShapeNet dataset. We evaluate the CLIP semantic similarity of rendered object viewpoints to ground truth viewpoint samples. We demonstrate that generating 3D models with ConRad leads to significant improvements over state-of-the-arr across all metrics.
All Views daul
Near Reference
Method
dres
doracle
dref
daul
doracle
PointE[39]
0.496
0.482
0.453
0.496
0.499
0.399
RealFusion[ [4]
0.495
0.486
0.460
0.483
0.482
0.464
NeuralLift-360(15)
0.528
0.516
0.498
0,543
0.544
0.534
ConRad (ours)
0.332
0.316
0.273
0.286
0.257
0.230
First, we render 20 objects from 10 categories of the 117 dataset viewed from 68 different camera poses to create ground truth (GT) set. We choose a front-facing view from each object as input to an Image-to-3D approach. We then render the generated object from the same 68 camera poses, Due to ambiguity of depth, corresponding camera poses between GT and rendered images could depict very different object poses. Using these two sets of images, we compute three metrics dref is the mean CLIP feature distance between the reference image and all the rendered viewpoints (same as [15]), dau is the mean CLIP feature distance between all pairs of GT and rendered images. doracte is the solution to a linear sum assignment problem[ 40] where the cost of assigning a GT view to rendered image is the CLIP feature distance between them. This evaluates the ability of the representation to generate images as diverse as the ground truth while preserving semantic content.
We evaluate these three metrics for the two sets of 68 images ("All Views"). We also evaluate on subset of camera poses that lie within 15" elevation change and 45 azimuth change ("Near Reference") giving us 15 images each for ground truth and rendered images. This measures the semantic consistency in viewpoints where parts of the input image are visible.
Input Image
ConRad
ConRad
ConRad RGB Constr.
ConRad Textual Inv.
ConRad Depth Loss
ConRad Warm
RGB Constr,
Figure 4: Ablation Study: We present a qualitative investigation of the importance of various components proposed in our approach on two prototypical examples. The components are arranged left to right in decreasing order of importance for final reconstruction quality.

---

5.4 Analysis of ConRad Components
Quantitative Evaluation In Table we compare ConRad to RealFusion, NeuralLift360, and PointE. PointE[39] is a point cloud diffusion model was trained on several million 3D models. It can directly generate point clouds using CLIP features of the input image. Since there no corresponding reference view in the output, we synthesize 150 random views for PointE instead of the 68 chosen views. For all methods, we use the implementations and hyperparameters provided by the authors. We observe that ConRad outperforms these methods across all the metrics by a significant margin. We present a per-object breakdown of these metries in the supplementary material.
We now investigate the various components of ConRad to understand their significance. In Figure 4, we present a visualization of the effect of removing each component on two typical examples. In most experiments, we observe that performing a warm start is not necessary but leads to crisper final 3D structure. We also observe that removing the Depth Loss significantly reduces computational cost, but this leads to incorrect 3D structure for some objects (see row 2 of Figure 4). Textual Inversion crucial maintain consistent appearance the object from all viewpoints. Moreover, removing the color constraint (Eq 4) generally still leads to a realistic 3D model but depicts an arbitrary object. Finally, removing both color and density constraints (still using the depth loss) leads to very unrealistic objects.
5.5 Failure Cases and Limitations
While ConRad significantly improves the sim- plicity and robustness of learning the 3D struc- ture, it occasionally demonstrates issues that common to Image-to-3D methods. In Figure 5, we present two typical examples of failure cases. Row 1 demonstrates the Janus effect, where the final 3D model has two faces. We also observe that performing score distillation sampling us- ing Stable Diffusion leads to saturated colors in the 3D model. Leveraging textual inversion em- beddings mitigates this issue some extent, but still occasionally leads to incorrect appearance in novel viewpoints, as demonstrated Row We also observe that for few objects, parts of the generated 3D object mri-trranppreet (see videos in supplementary material).
Input Image
Novel Front View
Novel Back View
#EEGED
Figure 5: Failure Cases: On some objects, our proposed approach suffers from the Janus effect and is affected by the bias of Stable Diffusion to produce saturated colors.
Conclusion
In this work, we present a novel radiance field variant, ConRad, that significantly simplifies the training pipeline for generating 3D models from single image. By explicitly incorporating the input image into the color and density fields, we show that we can eliminate tedious tuning of additional hyperparameters. We demonstrate that ConRad representations lead to more accurate depiction of the input image, produce higher quality and more realistic 3D models compared to existing work.
10

---

References
[1] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall, "Dreamfusion: Text-to-3d using diffusion," arXiv preprint arXiv:2209. 14988, 2022.
[2] C.-H. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng, X. Huang, K. Kreis, S. Fidler, M.-Y. Liu, and T-- Y. Lin, "Magic3d: High-resolution text-to-3d content creation," arXiv preprint arXiv:2211.10440 2022.
[3] C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and
A. Komatsuzaki, "Laioo-400m: Open dataset of clip-filtered 400 million image-text pairs," arXiv preprint arXiv:2111.02114 2021.
[4] A. Radford, W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin.
J. Clark, et al., "Learning transferable visual models from natural language supervision," in International conference machine learning, pp. 8748-878 8763, PMLR. 2021.
[5] J. Li, D. Li, C. Xiong, and S. Hoi. "Blip: Bootstrapping language-imas pre- pre-training for unified vision- language understanding and generation." in International Conference Machine Learning, pp. 12888- 12900, PMLR, 2022.
[6] Li. D. S. Savarese, and Hoi, "Blip-2:
Bootstrapping language-image pre-training with frozen
image encoders and large language models," arXiv preprint wwxxi:2301..2597 2023.
[7] Ho, A. Jain, and P. Abbeel, "Denoising diffusion probabilistic models," " Advances in Neural Information P Processing Systems, vol. 33, pp. 6840-6851, 2020.
[8] R. Rombach, A. Blattmann, D. Lorenz. P. Esser. and B. Ommer. "High-resolution image synthesis with latent diffusion models," in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
[9] A. Lugmayr, M. Danelljan. A. Romero, F. Yu, R. Timofte, and Van Gool, "Repaint: Inpainting using denoising diffusion probabilistic models," " in Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1461-11471, 2022.
[10] G. Couairon, J. Verbeek, H. Schwenk, and M. Cord, "Diffedit: Diffusion-basse -baood semantic image editing with mask guidance," arXiv preprint ar0:::2210.1 2022.
[11] T. Brooks, A. Holynski, and A. A. Efros, "Instructpixppix: Learning to follow image editing instructions," arXiv preprint arXiv:2211 1,09000, 2022.
[12] B. Wallace, A. Gokul, and N. Naik, "Edict: Exact diffusion inversion via coupled transformations," arXiv preprint arxiv:2211.12440 2022.
[13] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. "Nerf: Representing scenes as neural radiance fields for view synthesis," Communications of the ACM, vol. 65, no. 1, pp. 99-106, 2021.
[14] L. PeyrazK, C. Rupprecht. I. Laina, and A. Vedaldi, "Realfusion: 360 ('ddeg) reconstruction of any object from single image." arXiv preprint arXiv:2302. 10663, 2023.
[15] D. Xu, Y. Jiang, P. Wang, Z. Fan, Y. Wang, and Z. Wang. "Neurallift-36: Lifting in-the-wild 2d photo to 3d object with 360 { (eee views," arXiv preprint 2.. 2022.
[16] Reizenstein, R. Shapovalov, Henzler. L. Sbordone, P. Labatut, and D. Novotny, "Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction," in Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10901-10911, 2021.
[17] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song. H. Su, al., "Shapenet: An information-rich 3d model repository," arXiv preprint 1r12v:151..03012. 2015.
[18] T. Müller, A. Evans, C. Schied, and Keller, "Instant neural graphics primitives with a multiresolution hash encoding," ACM Transactions on Graphics (ToG), vol. 41, no. 4, pp. 1-15, 2022.
[19] J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. A multiscale representation for auti-aliasing neural radiance fields." in Proceedings of the IEEE/CVF International Conference Computer Vision, 5855-5866 2021.
and P. P. Srinivasan, "Mip-nerf:

---

[20] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao, "3d shapenets: A deep representation for volumetric shapes." in Proceedings of the IEEE conference on computer vision and puttern recognition. pp. 1912-1920, 2015.
[21] G. Riegler, A. Osman Ulusoy, and A. Geiger, "Oetnet: Learning deep 3d representations at high resolutions." in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3577-3586, 2017.
[22] C. Jiang, A. Sud, A. Makadia, J. Huang, M. Nießner, T. Funkhouser, et al., "Local implicit grid repre- sentations for 3d scenes," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6001-6010, 2020.
[23] A. Yu, R. Li, M. Tancik, H. Li, R. Ng, and A. Kanazawa, "Plenoctrees for real-time rendering of neural radiance fields." in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5752- 5761, 2021.
[24] S. Tulsiani, A. Kar, J. Carreira, and J. Malik, "Learning category-specific deformable 3d models for object reconstruction," IEEE transactions on pattern analysis and machine intelligence, vol. 39, no. 4, pp. 719 2016.
[25] S. Tulsiani, H. Su, L. Guibas, A. A. Efros, and J. Malik, "Learning shape abstractions by assembling volumetric primitives." in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2635 2005-2643,222 2017.
[26] A. Yu. V. Ye, M. Tancik, and A. Kanazawa, "pixelnerf: Neural radiance fields from one or few images." in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 4578-4587, 2021.
[27] J. Kulhááek, E. Derner, T. Sattler, and R. Babuška, "Viewformer: Nerf-frree neural rendering from few images using transformers," in Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV, pp. 198 198-216, Springer, 2022.
[28] C. B. Choy, D. Xu, J. Gwak, K. Chen, and S. Savarese. "3d-r2n2: A unitied approach for single and multi- view 3d object reconstruction." in Computer Aision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 1-14, 2016, Proceedings, Part VIII 14, pp. 628-644, Springer, 2016.
[29] H. Xie, H. Yao, X. Sun, S. Zhou, and S. Zhang. "Pix2vox: Context-aware 3d reconstruction from single and multi-view images," in Proceedings of the IEEE/CVF international conference on computer vision, pp. 2690 2698, 2019.
[30] F. Yagubbayli, Y. Wang, A. Tonioni, and F. Tombari, "Legoformer: Transformers for block-by- by-blockk multi-view 3d reconstruction," arXiv preprint arXiv:2106. 2021.
[31] A. Jain, Mildenhall. J. T. Barron, P. Abbeel, and Poole. "Zero-shot text-guided object generation with dream fields," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 867-876, 2022.
[32] A. Jain, M. Tancik, and P. Abbeel, "Putting nerf on diet: Semantically consistent few-shot view synthesis." in Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 5885-5894, 2021.
[33] X. Pan, B. Dai, Z. Liu, C. C. Loy, and P. Luo, "Do 2d gans know 3d shape? unsupervised 3d shape reconstruction from 2d image gans," arXiv preprint arXiv:2011.008. 2020.
[34] C. Deng, C. Jiang, C. R. Qi, X. Yan, Y. Zhou, L. Guibas, D. Anguelov, et al., "Nerdi: Single-view nerf synthesis with language- guided diffusion general image priors," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20637 1337-20677,2 2023.
[35] M. S. Lee, W. Shin, and S. W. Han, "Tracer: Extreme attention guided salient object tracing network." arXiv preprint arXiv: 2021.
[36] R. Gal, Y. Alaluf, Y. Atzmon, O. Patashnik, A. H. Bermano, G. Chechik, and D. Cohen-Or, "An im- age is worth one word: Personalizing text-to-image generation using textual inversion." arXiv preprint 20Xiv:2008.01618, 2022.
[37] R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun, "Towards robust monocular depth estimation: Mixing datasets for shot cross-dataset transfer," IEEE Transactions on Puttern Analysis and Machine Intelligence (TPAMI), 2020.
[38] R. Ranftl, K. Lasinger, D. Hafnnr, K. Schindler, and V. Koltun, "Towards robust monocular depth estimation: Mixing datasets for zero-shot cross dataset transfer." IEEE transactions on pattern analysis and machine intelligence, vol. 44, no. 3. pp. 1623- 1237,1637,000 2020.
12

---

[39] A. Nichol, H. Jun, P. Dhariwal, P. Mishkin, and M. Chen, "Point-e: A system for generating 3d point clouds from complex prompts," arXiv preprint urX1::2112.08751, 2022.
[40] H. W. Kuhn, "The hungarian method for the assignment problem," Naval research logistics quarterly. vol. 2, no. 1-2, pp. 83-97, 1955.
13

---

A
Additional Implementation Details
Here we present more details about the implementation of ConRad representation to facilitate reproducibility of results.
Base Representation As indicated in the main text, we use the Instant-NGP( representation followed by two MLPs to model the density (x)) and color c(z). For the multi-resolution hash encoding of Instant-NGP representation, we use levels with a 2-dimensional encoding at each level. The Instant-NGP encoding is passed to two 3-layer MLPs with a hidden dimension of 64. The color MLP output is passed through sigmoid activation to obtain the RGB values additionally. The density MLP output is passed to the exponential e activation to obtain the density value.
Regularization Losses In addition to SDS and the Depth Loss, we use three regularizations proposed in [1] to produce coherent objects. We encourage the radiance field to have either very low or very high density at any point along the rays by using the following entropy regularizer:
Cent Σ a(r) * log (x)) - (1 a(x)) log(1- - a(x))
(8)
where a<(x) is the rendering weight at point x.
We encourage the surface normals in each rendered view to point towards the camera using the orientation regularizer:
Corient Σa(r) max(< n(x). d 2,0)
a())
(9)
where n is the normal at the point æ computed using the finite differences method with the density (x) and d is the viewing direction.
Finally, encourage the smoothness of normals computed at each point along the rays using the smoothness regularizer:
Losmooth Σ [nx) n(x 5)]
(10)
where 8 is random perturbation for each point with maximum perturbation of 0.01 along each axis.
Algorithm 1: Optimizing a ConRad representation to reconstruct 3D) from a single image.
Data: Image i, Estimated Depth i, Foreground Mask M, Reference Pose ý
Initialize ConRad Fe:
2 Compute reference rays rp:
Compute text embeddings using textual inversion image 1; for i = to 5000 do
V + Visibility Depth using F and aa (2) (Equation 3);
D3 + Radiance Field Depth along rays r3 using o' d((x):
Ldep + Pearson Correlation between Dg and D;
p - sample random camera pose;
Tp compute rays for viewpoint p;
10
I + Render reference viewpoint image using Tp, (z))
11
18, VoLsns 12, < Using Stable Diffusion, text embedding L and Ip (Eq, 2);
Lorient Compute orientation regularization along rpi Lsmooth -- Compute smoothness regularization along rp: 13
14
Lent & Compute entropy regularization along Tp:
0 0 - nVVoaas Ig. Ldo0phhh + Lent + 0.01 Corient 10 Lsmooth))
Loss Weighting
The above losses are all jointly optimized using the following weighted sum:
L = Laps + 10 * deepth + 0.01 Lent + 0.01 Corient + 10 * Lemooth
(11)
14

---

In experiments, we observed that the orientation and entropy regularizer had minimal effect on the final oulput and could be turned off on most input images without any loss in quality. However, we retain these regularizers in all experiments to ensure similarity to DreamFusion
Textual Inversion Since we aim to use only an image as input, need to synthesize text prompt to pass as conditioning to the cox-ccondtiooee diffusion model. Additionally, we need the text-prompt be sufficiently descriptive capture all the details of the specific object instance. Most pretrained captioning algorithms provide a coarse caption that doesn't I capture such details. Therefore, recent research, Textual Inversion[36] proposed an approach that uses input images and diffusion model to infer the text embedding of special token ("<token>"). This token along with its learned embedding can be passed to the diffusion model to synthesize novel images of the same object. This approach generally requires 3 to 5 images to accurately capture an object. Since we have access to only one image, we rely on synthetically augmenting the image to run Textual Inversion. Specifically, we generate images by randomly flipping horizontally with probability 0.5, extract random crop covering 50% to 100% of the image, Gaussian blurring the image with kernel size and standard deviation randomly sampled in [0.1, 2] and jittering the hue, saturation, contrast and brightness by random value in [0, 0.1]. Furthermore, we perform classification of the input image using a pretrained CLIP model[4]. The text embedding of the obtained class label is used to initialize the embedding of the special token "<token>".
Optimization We use the Adan optimizer with 0.005 learning rate and weight decay of 0.00002. We use a batch size of for each update i.e. one random viewpoint. We keep the learning rate fixed and train the representation for 5000 updates. The optimization process takes approximately 20 minutes on a single A100 GPU.
B Additional Quantitative Evaluation
In Sec 5.3 of the main text, we present quantitative comparisons of our work RealFusion| 14], NeuralL.itt330(115 and Point-E[39] on objects from the ShapeNet[17] [7]] dataset. Here we present more details on this evaluation facilitate easy reproduction.
Evaluation Data We choose 20 objects from 10 categories. While we randomly sampled objects, we manually removed some non-prototypical objects like triangular bed. In Figure 6, we present the chosen input image for each object, the ShapeNet object ID and category.
Table 2: Quantitative Comparisons: We present quantitative comparisons using 3D object data of
20 objects the ShapeNet dataset. We present an object-wiss wise breakdown of the performance for each method. The performance is evaluated on all (68) views of the object. Please refer to the main text for more details on the evaluation metries.
Congory
ID
berade
RealFinion
Namil 30-
Poire-F
CooBtal
RestFasion
Nerrall 10-160
Point-P:
Cunitad
RraiFusson
Naunil (0--50
Poire-F
Conloal
arplaae
2e919)...
0.418
0,751
0.412
0.151
0.062
0.833
(1:41
0.263
0.433
0,813
0.420
0.236
airplane
705946...
0.246
0.513
0.639
0.668
0.238
5a0650..
0.642
0.374
0.457
0.557
0.602
0.427
0.297
beach
62co45..
0.505
0.373
0.408
0.575
0.902
0.366
0.349
Ihulid....
0.541
0.564
0.338
0.548
0.501
0.461
0.537
0.469
0.259
bus
64213d..
0.485
0.531
0.571
0.356
8500
0.474
0.574
0.380
0.462
0.416
0,500
0.375
tus
destlal...
0.644
0.588
0.582
0,340
0610
0.534
0.533
0.296
0.587
0,902
0,490
0.232
car
labeca..
0.400
0.455
0,575
0.469
0.887
0.155
0.551
0.801
0.460
0.411
0,517
0,350
car
351.55...
0.314
0.396
0.527
0.368
0.504
0.340
0.527
0.311
0.48]
0,300
0,501
0.365
9901e1..
0.541
0.440
0.582
0.371
0.503
0.377
0.561
(L261
0.433
0.347
0.535
6.13
char
Scelel...
0.562
0.370
0.445
0.773
0.395
0,369
char
0.480
0.208
nowerpor
867642...
0.338
0.505
0.319
0.451
0.283
0.469
0.467
0.290
flwrrput
12599...
0.390
0:252
0.19%
0.368
0.401
0.255
0.364
0,471
0.186
puiar
4c082a..
0.480
0,401
0,558
0.335
0.363
0.340
01454
0.381
0.336
0.313
0,435
0.174
nokebbike
61b17E...
0.407
0.586
0,482
0.327
0.391
0.597
0.508
0.297
0.353
0,593
0,483
0.252
mokerbike
6x3761...
0:409
0.965
0.521
0,170
0.400
01 348
0152
0.267
0.370
0.314
0.491
6.130
sofa
668194...
0.553
0.556
0,458
0.338
0.17
0.516
0.424
0.337
0.493
0,512
0.385
0
utic
Sa592..
0.481
0.471
0.387
0.294
0.503
0.478
0.395
0.297
0.47
0.474
0.365
0.252
table
0.189
0.458
0,406
0.560
0.394
0.37
0.451
0,570
0.562
Detailed Quantitative Results In Table 2, we present an object-wise breakup of the results presented in the main text. We observe that for most objects, ConRad based training leads to significantly improved results on all metrics.
15

---

Figure 6: ShapeNet Evaluation Data: We present the 20 input images used to evaluate performance ConRad and existing works in this domain.
16

---

C Additional Qualitative Comparisons
In Figure 2, we present visualizations of 3D models produced by ConRad. Here we present visualiza- tions for the 3D models produced by RealFusion[ 14] and |euralLift360[[5] using the same input images in Figure 7. We observe that both methods generally produce low quality generations when arbitrary images used inputs. We found that NeuralLift360 was successful more frequently compared to RealFusion but produced inferior 3D models compared to ConRad.
NeuralLift-300 [15]
RealFusion [14]
Figure 7: Additional Qualitative Baseline Results: We present additional qualitative baseline
results using reference images presented in the main paper. For both BeuralL.ftt-361 and RealFusion, we use the official implementations released by the authors.
17


Reviews:
[{"review_id": "BXzOt2IlQH", "reviewer": "Reviewer_yDN8", "summary": "This paper presents ConRad, a new method for reconstructing 3D objects from a single RGB image. At the core of ConRad is a neural radiance field that by design satisfies the reference view constraint. That is, the representation will always render the input RGB image at the reference view. This hard constraint gets rid of the need for training in the reference view, and enables ConRad to be trained in a similar way as DreamFusion. Experiments show that ConRad can produce 3D reconstructions more faithful to the input and produce more consistent 3D models than RealFusion and NeuralLift-360.", "questions": "Authors may respond to the weaknesses mentioned above.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The idea of introducing reference-view reconstruction as a hard constraint is interesting and novel. It by design projects the image color to the 3D radiance field and gets rid of the need for training in the reference view.  This design also serves as a useful prior during training and better preserves the texture and properties of the input image. I believe this design would be interesting to the community.\n\n- The qualitative comparison shows that ConRad performs much better than RealFusion and NeuralLift-360 in reference view reconstruction and multiview consistency. The quantitative results of ConRad is also significantly better.\n\n- The writing is clear and easy to follow.", "weaknesses": "-  Most parts of the pipeline are based on previous works, such as the texture inversion, multiview SDS loss, and depth guidance, which makes the technical contribution not strong.\n\n- At line 174, it is mentioned that $\\eta$ is set to 0.1 and \"the visibility depth for each pixel is a point on the ray beyond which the contribution of color is minimal (less than 10%).\" I find this design and explanation not convincing and maybe wrong.  \nIn my understanding, when $\\eta = 0.1$, it means the ray from the camera to the visibility depth has only contributed 10% color, and the ray after the visibility depth will contribute the remaining 90% color, which is the opposite of the explanation in the paper. Therefore, I think it makes more sense to set $\\eta$ to 0.9. Authors should clarify this and provide an ablation study on the choice of $\\eta$.\n\n- In Fig. 3, the authors should show the qualitative results of all baselines (RealFusion and NeuralList-360) for each example. There is enough horizontal space to do this. Besides, there are not enough qualitative comparisons in the current submission. Authors should provide more qualitative comparisons in the supplementary materials.\n\n- For quantitative study, why not report the scores under the same setting as RealFusion for a fair comparison?\n\n- From the video, it can be seen that the 3D object is semi-transparent in many cases.\n\n- At line 237, it is mentioned that \"Computation of visibility depth also does not significantly increase GPU memory consumption since we do not compute its gradients.\" However, since the method uses a depth loss, I believe the gradient need to be back-propagated through depth anyway. Then this argument does not make sense.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer yDN8 for the appreciation of our work, overall positive review and constructive feedback. We clarify the questions raised by the reviewer here.\n\n> 1. Line 174, ... I think it makes more sense to set $\\eta$ to 0.9.\n\nThis is an error in Equation (3). The equation should be \n$$ 1 - \\frac{\\int^t_{0} T(t) \\sigma( r^{(i,j)}_p(t) )}{\\int T(s) \\sigma( r^{(i,j)}_p(s) ) } = \\eta $$\nThank you for pointing this out. We will fix this in the paper. We observed in experiments that the effect of $\\eta$ varies with the volume and complexity of the object. However, setting it to 0.1 generally works for all objects. We will present additional visualizations in the final version with varying $\\eta$. \n\n> 2. In Fig. 3, the authors should show the qualitative results of all baselines\n\nWe have added more qualitative visualizations of the baselines in the attached rebuttal PDF. We observe that ConRad can consistently produce higher quality reconstructions (compare to the visualizations in the main paper). For fair comparisons to the baseline works, in the main paper, we chose to compare on the images chosen by the respective authors in Figure 3. We will add these additional visualizations to the supplementary material. \n\n> 3. why not report the scores under the same setting as RealFusion for a fair comparison?\n\nWe faced several issues with the metric proposed in RealFusion:\n- First, the authors did not release an implementation of this metric. This required us to attempt to reproduce the results presented in RealFusion.\n- The metric presented in RealFusion requires conversion of the radiance field to a mesh using the marching cubes algorithm, followed by Iterated Closest Point (ICP) algorithm to match the estimated mesh to the ground truth mesh. Both these algorithms are known to be sensitive to the hyperparameters and would require manual tuning per object. This makes the metric difficult to reproduce. \n\nIn contrast, we took inspiration from the metric proposed in NeuralLift-360. The \"All View $d_ref$\" metric presented in our work is the same as the metric used in NeuralLift-360. We also report additional metrics ($d_{all}$, $d_{oracle}) that build on top of this idea while maintaining simplicity. We will also release code for this metric along with the final version of the paper. \n\n> 4. From the video, it can be seen that the 3D object is semi-transparent in many cases.\n\nThis is a common issue faced by image to 3D approaches that rely on NERF representations. Similar results can be observed in RealFusion and NeuralLift-360. We will include this in our discussion on limitations. One potential solution is to use mesh-based representations. However, this exploration would be beyond the scope of our work and would require additional research.  \n\n> 5. At line 237, it is mentioned that \"Computation of visibility depth also does not significantly increase GPU memory consumption since we do not compute its gradients.\" However, since the method uses a depth loss, I believe the gradient need to be back-propagated through depth anyway. \n\nThis is referring to the computation of the \"*visibility depth*\" defined in Equation (3). This is not the same as the depth estimate used in the computation of the depth loss. It is true that the depth loss requires back propagation of gradients. This statement is alluding to the fact that the computation of *visibility depth* does not add additional memory consumption on top of that. \n"}, {"review_id": "w95nilUSnA", "reviewer": "Reviewer_GS6t", "summary": "This paper introduces Image Constrained Radiance Fields (ConRad), a novel 3D representation that constrains initial radiance fields to a reference view image without requiring training. ConRad is adept at accurately modeling the input image in one reference view and is effectively integrated with Dreamfusion-style training to convert single images to 3D shapes. The results illustrate that ConRad’s 3D reconstructions are of high quality and closely resemble the original input.", "questions": "see the strengths and weaknesses.", "limitations": "NA", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper introduces an innovative image-conditioned radiance field, ConRad, which is capable of accurately modeling a reference view without the need for training, and significantly enhances the optimization of single image-conditioned NeRFs. Despite its simplicity, ConRad produces remarkable results in converting single images to 3D.\n2. The authors adeptly tackle several technical challenges by employing depth loss and a warm start strategy. ConRad’s effectiveness is convincingly demonstrated through both qualitative and quantitative results.\n3. The ablation studies presented in Figure 4 are methodologically sound and illustrate the efficacy of each module proposed. Additionally, the authors provide insightful analyses.\n4. The paper is well-structured, with clear and easily comprehensible presentation.", "weaknesses": "1. It would be beneficial for the authors to include information on the training time required for ConRad to convert a single image to 3D, and compare this with the training times of original Dreamfusion or Dreambooth3D. It is pertinent to understand if the ConRad representation contributes to accelerating the convergence speed of NeRF optimization.\n2. The paper should explore the impact of varying hyperparameters within ConRad. Additionally, as Instant-NGP is utilized to represent the 3D scene, it would be valuable for the authors to investigate the influence of different NeRF representations.\n3. ConRad still exhibits issues such as color saturation, akin to those found in Dreamfusion-style optimization. It is recommended that the authors consider incorporating recent advancements in Dreamfusion, such as ProlificDreamer, to further enhance ConRad’s performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer GS6t for their appreciation of our work, positive review and valuable suggestions to improve our paper. \n\n> 1. It would be beneficial for the authors to include information on the training time required for ConRad to convert a single image to 3D, and compare this with the training times of original Dreamfusion or Dreambooth3D.\n\nWe hoped that ConRad would lead to faster convergence since the reference viewpoint is instantaneously captured. However, in practice, we did not observe convincing improvements in speed across all objects compared to the baseline (RealFusion). We keep the number of updates equal to RealFusion which works well for all objects. Both methods take approximately 20 minutes on a single A100 GPU. NeuralLift-360 uses twice the number of updates and takes 1 hour on the same hardware setup. We will add these details to the paper. \n\n> 2. explore the impact of varying hyperparameters within ConRad ... it would be valuable for the authors to investigate the influence of different NeRF representations.\n\nThank you for the suggestion. We will report these in the final version. Switching the representation from Instant-NGP to other NeRF representations would require additional experimentation which precludes us from reporting it here due to time and resource limitations. \n\n> 3. ConRad still exhibits issues such as color saturation, akin to those found in Dreamfusion-style optimization. It is recommended that the authors consider incorporating recent advancements in Dreamfusion, such as ProlificDreamer, to further enhance ConRad’s performance.\n\nAs an active area of research, there have been several recent advancements in this domain. Since ConRad improves the underlying representation, we believe that these advancements can be applied to ConRad. We thank the reviewer for the suggestion. We plan to investigate this and report any improvements in the final version. \n\n"}, {"review_id": "DFel3Ux8Nc", "reviewer": "Reviewer_ZVcc", "summary": "This work proposes an approach for 3D generation from a single input image. Given an input image, a monocular depth estimate and an estimated instance mask, an image constrained radiance field is optimized following the score distillation approach from DreamFusion. They key idea is to enforce a hard constraint on the radiance field that guarantees that the input image can be reconstructed exactly whereas existing methods only encourage this with a reconstruction loss.\n", "questions": "Please also see Weaknesses.\n\nL.262 mentions that NeuralLift360 uses mask supervision. However, this was not clear to me from the original NeuralLift360 paper. Could you please explain again how NeuralLift360 uses foreground masks and provide a pointer where this is stated in the original paper?\n\nYour approach seems to achieve a higher image fidelity than the baselines. Is this only a result of more stable training or which component, e.g. compared to NeuralLift360, enables the higher image fidelity?\n\nMinor:\nL.277 “improved metrics”, it is not shown that the metrics improve over existing metrics, so I suggest removing “improved”\n", "limitations": "In general, the limitations and the broader impact were adequately discussed. From the supplementary video it looks like sometimes objects are not fully opaque (bird). This should be added to the limitations section. \n", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is easy to follow and well-written. Both, qualitative and quantitative analysis indicate that the proposed method outperforms the existing baselines. \nIn general, inferring 3D representations from single input images is an interesting and challenging task with high relevance to the community. The idea of integrating the input image as a hard constraint in optimization is intuitive and might be interesting to the community.\n", "weaknesses": "My main concern is that the central claim of the paper, that integrating the image as a hard constraint over using a reconstruction loss makes the training more robust and that “The proposed ConRad representation significantly simplifies the process of a generating a consistent 3D model for the input image” (L.55), requires more experimental support. While the supplementary video shows a few failure cases for the baselines these results could be hand-picked. Importantly, it is also not clear if the stable training indeed results from using the proposed image conditioned radiance fields. For this, I would have liked to see an ablation study where the training pipeline is exactly the same except that once an image conditioned radiance field is used (hard constraint) and once a reconstruction loss is used (soft constraint), also ablating different strengths/weights of the reconstruction loss. \n\nUsing pearson correlation as a depth loss for monocular depth was already proposed in [1] which is not cited. Further, [1] is missing from the baseline comparison and while there is no code available, a qualitative comparison could be performed similarly to the other baselines, i.e. using results from the original paper. \n\n[1] Deng et al, “NeRDi: Single-View NeRF Synthesis with Language-Guided Diffusion as General Image Priors”, CVPR2023\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer ZVcc for the constructive feedback and suggestions. We address the reviewer's comments here with additional experiments and discussion. \n\n> 1. My main concern is that the central claim of the paper, that integrating the image as a hard constraint over using a reconstruction loss makes the training more robust and that “The proposed ConRad representation significantly simplifies the process of a generating a consistent 3D model for the input image” (L.55), requires more experimental support.\n\nThank you for raising the concern that this claim does not seem sufficiently supported. Here we will provide additional discussion and results, and re-iterate some of our intuitions for this claim. \n\n- Our proposed approach ConRad removes any need for additional reconstruction loss objectives. This in turn eliminates any associated hyperparameters. Furthermore, existing works (like RealFusion and NeuralLift-360) perform an \"alternating optimization\" strategy to optimize two separate objectives which is generally an unstable optimization algorithm (except in special cases). Therefore, we believe that removal of these components intuitively leads to a *simplification* of the process.\n\n- Nevertheless, we agree that additional experimental evidence could help support this claim further. Based on the reviewer's suggestion, In Table 1 of the rebuttal PDF, we compare ConRad to models constructed by learning the representation using the reconstruction losses (for RGB and foreground mask). We investigate different strengths of reconstruction losses $\\lambda$ and experiment with 20 ShapeNet objects. We observe that on most objects (18 out of 20), ConRad produces better reconstructions based on the \"All View $d_{oracle}$\" metric. We also observe the same result across all six metrics but omit this due to space constraints. \n\n\n> 2. Using pearson correlation as a depth loss for monocular depth was already proposed in [1] which is not cited. Further, [1] is missing from the baseline comparison and while there is no code available, a qualitative comparison could be performed similarly to the other baselines, i.e. using results from the original paper. \n[1] Deng et al, “NeRDi: Single-View NeRF Synthesis with Language-Guided Diffusion as General Image Priors”, CVPR2023\n\nThank you for pointing us to this reference. We will include this citation. To the best of our knowledge, this was unpublished work at the time of submission. However, we will add comparison to this work in the final version. \n\n> 3. L.262 mentions that NeuralLift360 uses mask supervision. However, this was not clear to me from the original NeuralLift360 paper. Could you please explain again how NeuralLift360 uses foreground masks and provide a pointer where this is stated in the original paper?\n\nThis detail is not mentioned in the NeuralLift-360 paper. Please refer here for the implementation detail:\n- https://github.com/VITA-Group/NeuralLift-360#data-preparation\n- https://github.com/VITA-Group/NeuralLift-360/blob/main/nerf/utils_neurallift.py#L377-L383\n- https://github.com/VITA-Group/NeuralLift-360/blob/main/nerf/utils_neurallift.py#L620-L628\n\n> 4. Your approach seems to achieve a higher image fidelity than the baselines. Is this only a result of more stable training or which component, e.g. compared to NeuralLift360, enables the higher image fidelity?\n\nWe believe that the higher fidelity can be attributed to the stability achieved by capturing the reference view accurately using ConRad. There are no other major differences compared to NeuralLift-360 and RealFusion (except the addition of depth loss). \n\n> 5. Minor: L.277 “improved metrics”, it is not shown that the metrics improve over existing metrics, so I suggest removing “improved”.  \n\nWe will update this in the paper. \n"}, {"review_id": "WNPhGssFpQ", "reviewer": "Reviewer_1JJW", "summary": "This paper proposes a novel parametrization for NeRF designed to facilitate the task of single image (+ foreground mask) to 3D model generation. The authors modify the volumetric rendering equation of the NeRF volume to include explicit constraints given by the single available view. In particular, by construction, points intersected by rays corresponding to the background of the conditioning image will have their density set to zero and points intersected by rays corresponding to the foreground will have an RGB color equal to the one in the conditioning image (instead of the one encoded in the radiance field). Given this modified NeRF rendering procedure the authors use textual inversion + an SDS loss similar to Dreamfusion to optimize a full 3D model. The proposed method can be used both to generate plausible 3D models of real objects from a single view as well as to generate 3D models from pure text using a text2image model to get the conditioning view. The method is compared to concurrent works and achieves better quality and more crisp 3D models. ", "questions": "## Questions\n\n1. How are you handling the background when training with the SDS? The NeRF being optimized does not have any background, are you adding a random one before feeding the image to Stable Diffusion to compute the SDS?\n\n2. Can you clarify any possible misunderstandings with respect to what I wrote in weakness “b”?\n\n3. The procedure used to initialize the text token described between line 38 and 40 of the appendix it’s not very clear. How do you perform classification using CLIP in this context?\n\n## Typos\n\n* line 54: “the process of a generating” → “the process of generating”\n", "limitations": "No flag for negative societal impact.\n\nLimitations have been discussed in Sec. 5.5.\n", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "+ The proposed solution is both simple and elegant while achieving convincing results. The modification to the rendering equation guarantees, by construction, that the single input view is going to be respected. This allows the capacity of the NeRF volume to focus on modeling the missing parts and recovering the geometry of the first view (when not using the monocular depth loss). \n\n+ Great presentation, the paper is well written and quite self contained. Most details regarding the implementation of the method are provided as part of the appendix. \n\n+ I appreciated the effort of the authors in Sec. 5.3 Tab. 1 to try to propose a novel metric for evaluating the quality of the inferred 3D representations in terms of their 3D consistency. \n", "weaknesses": "## Major\n\nI have not found major weaknesses in the work. \n \n## Minor\n\na. **Scale Ambiguity**: The scale of the 3D reconstructed model is inherently ambiguous due to the unknown depth associated with the first view. This means that depending on how the density of the first view converges the same 3D object could be represented as a big object far away or a small object close to the original camera. Of course this, in turns, implies that moving the camera once the model is fitted will have drastically different effects. This is also recognized by the authors between line 287 and 290. This is an inherent ambiguity of any methods based on a single view but I wonder if a simple regularization to push the density of the NeRF towards the central area of the volume could have helped to standardize the scale of the fitted 3D models.  \n\nb. **Possibly entangled text inversion**: to keep faithful details of the object the authors propose to perform textual inversion of a Stable Diffusion model to find a text token corresponding to the appearance of the object they are trying to reconstruct. To do so they apply various augmentation to the single view of the object to generate a training set for textual inversion and directly optimize an input token. Since all the images used for textual inversion are generated from a single seed image the textual inversion token might pick up on specific details of the single view available rather than on the specific object that the authors are trying to reconstruct. An example of unwanted entangle representation could be to pick up on  the background of the single view. Randomly cutting and pasting the foreground object into random positions over random backgrounds might be a viable solution to reduce this risk. \n\nc. **Some ad hoc components per experiments**: From the manuscript it seems that the authors used the depth loss and the warm start only for some experiments but not for all of them. This makes evaluating the experimental evaluation a bit more confusing. If the two additional components do not have any negative impact on the final performance also for models which would not need them (as it seems the case from the ablations in Fig. 4) I would suggest just presenting all results with both components switched on for all experiments. If instead in some cases those have a negative effect I would mention it explicitly in the manuscript.  \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 1JJW for the appreciation of our work, positive review and the constructive discussion. We discuss some of the questions raised by the reviewer here.\n\n### Minor Weaknesses\n> 1. **Scale Ambiguity**: ... an inherent ambiguity of any methods based on a single view but I wonder if a simple regularization to push the density of the NeRF towards the central area of the volume could have helped to standardize the scale of the fitted 3D models.\n\nWe adopt a slightly different idea to achieve this goal. The density of the radiance field is initialized with a gaussian at the center with a chosen standard deviation. This biases the model towards a specific scale. The same idea is used in DreamFusion[1], RealFusion[14] and NeualLift-360 [15]. \n\n> 2. **Possibly entangled text inversion**: ...  An example of unwanted entangle representation could be to pick up on the background of the single view. Randomly cutting and pasting the foreground object into random positions over random backgrounds might be a viable solution to reduce this risk.\n\nThis is definitely a possible issue when performing textual inversion on a single image. Using segmented foreground with random backgrounds might help alleviate this issue to some extent. Our approach of carefully initializing the learned special token also addresses this issue to some extent.\n\nSpecifically, as briefly discussed in the supplementary material, we first classify the reference image using CLIP. This can be done by finding the noun $n$ in the CLIP vocabulary that minimizes the distance between the CLIP text embedding of \"A photo of <$n$>\" and the CLIP image embedding. The embedding of the special textual inversion token is then be initialized with the text embedding of $n$. This ensures that the token focuses on the object in the image. For example, for an image of a red cat, the textual inversion token would be initialized with \"cat\". Textual inversion would then ideally update the embedding to capture \"red cat\". However, it is still possible but less likely that the embedding would drastically drift to capture background elements.\n\n> 3.  **Some ad hoc components per experiments**:  From the manuscript it seems that the authors used the depth loss and the warm start only for some experiments but not for all of them. ... I would suggest just presenting all results with both components switched on for all experiments.\n\nWe do keep both components turned on except for the ablation experiments. This was possibly not communicated properly in the text. We will rephrase the text to make this clearer. \n\n--- \n\n### Questions\n> How are you handling the background when training with the SDS?\n\nFor every update, we randomly sample a RGB color and create a uniform background with this pixel value. Thank you for pointing this out. We will add this detail to the implementation details. \n \n"}, {"review_id": "XpXuNqTSTX", "reviewer": "Reviewer_mCGc", "summary": "This paper proposes image constraint neural radiance field, a representation that takes a reference image into account. Such a representation helps the task of 3D reconstruction from single image with the guidance from pretrained diffusion models. Experiments demonstrates the proposed method can help boost the quality of the reconstructed objects.", "questions": "Please refer to weaknesses.", "limitations": "The authors have mentioned some of the limitations in Sec 5.5.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method is simple and effective.\n2. The experiments show the effectiveness of the proposed image constraint neural radiance field.", "weaknesses": "1. For viewpoints other than the reference view, the shapes and images are blurry. The proposed method can only enforce the appearance under the reference view to be realistic, giving limited improvement to views that do not have overlap with the reference view.\n2. In line 49, '…can explicitly capture an input image without any training' which is ambiguous. The underlying depth of the input image still requires depth loss to constrain the learning of the radiance field.\n3. The proposed methods claims to fully utilize the information from the reference view compared to the previous methods(L55-57). The quality is better than the baselines, but how about the efficiency? Does it take shorter time to reach a good reconstruction result?\n4. There are some typos in the paper, e.g., L54 'a' should be removed; Eq.(4) 'o_p' should be bold to align with L172", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer mCGc for the positive review and constructive feedback. We clarify the concerns raised by the reviewer here.\n\n> 1. For viewpoints other than the reference view, the shapes and images are blurry.\n\nCompared to the reference view, the renderings do appear less crisp in novel viewpoints. However, this is a limitation of most image to 3D approaches. In fact, we demonstrate in results that compared to existing works, our approach produces higher quality reconstructions even in novel viewpoints (Figure 3 and Table 1). \n\n> 2. In line 49, '…can explicitly capture an input image without any training' which is ambiguous. The underlying depth of the input image still requires depth loss to constrain the learning of the radiance field.\n\nWe will rephrase this in the paper to make it clearer. This statement is referring to the rendering of the radiance field from the reference viewpoint. The proposed method can indeed capture an input image *in the reference viewpoint* without any training. The constraints applied on the radiance field are instantaneous and do not require any optimization. For example, in Figure 1, we show a visualization of the constraints applied using the image of a chair. As we show on the right, the rendering from the reference viewpoint is accurately capture without any training. \n\n> 3. The quality is better than the baselines, but how about the efficiency? Does it take shorter time to reach a good reconstruction result?\n\nIntuitively, we hoped to see improvements in efficiency since ConRad captures the reference view accurately. However, in practice we did not observe convincing speed improvements in the number of updates required. Therefore, we do not make this claim in the paper. We keep the number of updates equal to RealFusion which works well for all objects. Both methods take approximately 20 minutes on a single A100 GPU. NeuralLift-360 uses twice the number of updates and takes 1 hour on the same hardware setup. We will add these details to the paper.\n\n> 4. There are some typos in the paper, e.g., L54 'a' should be removed; Eq.(4) 'o_p' should be bold to align with L172\n\nThank you for pointing this out. We will make these changes. \n"}]

Author rebuttal:
We thank all the reviewers for their time and feedback. We address individual comments to each reviewer separately. 
Please find supporting material attached here as a PDF.  

Meta-review: The paper presents ConRad, a single-image 3D generation. All the reviewers are positive about its novelty (especially the ability to preserve the reference view without training) and the quality of the results. The authors rebuttal further clarified some of the initial concerns. The AC reads the discussions and agrees with the reviewers' assessment.
Decision: Accept (poster)