SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We seek to learn a model that, given a single image of a 3D scene enables us to sample from the distribution over 3D scenes that are consistent with the observation. We expect that 3D regions visible in the image are reconstructed faithfully, while unobserved parts are generated plausibly. Every time we sample, we expect a different plausible 3D generation. Signals S are 3D scenes, and observations are 2D images  $\mathbf{O}$  and their camera parameters  $\phi$ . At training time, we assume that we have access to at least two image observations and their camera parameters per scene, such that we can assemble tuples of  $(\mathbf{O}^{\mathrm{ctx}}, \phi^{\mathrm{ctx}}, \mathbf{O}^{\mathrm{trgt}}, \phi^{\mathrm{trgt}})$ , with 2D images  $\mathbf{O}^{\mathrm{ctx}}, \mathbf{O}^{\mathrm{trgt}}$ , and camera parameters  $\phi^{\mathrm{ctx}}, \phi^{\mathrm{trgt}}$ .
Scope. We note that our treatment of inverse graphics exceeds a mere application of the presented framework. In particular, we not only integrate the differentiable rendering forward function, but further propose a novel 3D-structured denoise function. Here, we enable state-of-the-art conditional generation of complex, real-world 3D scenes.
Related Work. Few-shot reconstruction of 3D scene representations via differentiable rendering was pioneered by deterministic methods [9, 31, 32, 32-41] that blur regions of the 3D scene unobserved in the context observations. Probabilistic methods have been proposed that can sample from the distribution of novel views trained only on images [4, 5, 42-45]. While results are impressive, these methods do not allow sampling from the distribution of 3D scenes, but only from the distribution of novel views. Generations are not multi-view consistent. Obtaining a 3D scene requires costly post-processing via score distillation [6]. Several approaches [2, 3] use a two-stage design: they first reconstruct a dataset of 3D scenes, and then train a 3D diffusion model. However, pre-computing large 3D datasets is expensive. Further, to obtain high-quality results, dense observations are required per scene. RenderDiffusion [7] and HoloDiffusion [20] integrate differentiable forward rendering with an unconditional diffusion model, enabling unconditional sampling of simple, single-object scenes. Similar to us, RenderDiffusion performs denoising in the image space, while HoloDiffusion uses a 3D denoising architecture. Other methods use priors learned by text-conditioned image diffusion models to optimize 3D scenes [46-48]. Here, the generative model does not have explicit knowledge about the 3D information of scenes. These methods often suffer from geometric artifacts.
Structure of S and forward model render. We can afford only an abridged discussion here - please see the supplement for a more detailed description. We use NeRF [49] as the parameterization of 3D scenes, such that S is a function that maps a 3D coordinate  $\mathbf{p}$  to a color  $\mathbf{c}$  and density  $\sigma$  as  $\mathbf{S}(\mathbf{p}) = (\sigma, \mathbf{c})$ . We require a generalizable NeRF that is predicted in a feed-forward pass by an encoder that takes a set of  $M$  context images and corresponding camera poses  $\{(\mathbf{O}_i, \phi_i)\}_i^M$  as input.
Figure 4: Qualitative Comparison for Inverse Graphics application. We benchmark with SparseFusion [5] and the deterministic pixelNeRF [9]. SparseFusion samples 2D novel views conditioned on a deterministic rendering (Diffusion Out.), and generates multi-view consistent 3D scenes only after Score Distillation. Our method consistently generates higher-quality scenes, while directly sampling 3D scenes.
We base our model on pixelNeRF [9]. pixelNeRF first extracts image features  $\{\mathbf{F}_i\}_i$  from each context observation via an encoder enc as  $\mathbf{F}_i = \mathrm{enc}(\mathbf{O}_i)$ . Given a 3D point  $\mathbf{p}$ , it obtains its pixel coordinates in each context view via  $\mathbf{p}_i^{\mathrm{pix}} = \pi (\mathbf{p},\phi_i)$  via the projection operator  $\pi$ , and recovers a corresponding feature as  $\mathbf{f}_i = \mathbf{F}_i(\mathbf{p}_i^{\mathrm{pix}})$  by sampling the feature map at pixel coordinate  $\mathbf{p}_i^{\mathrm{pix}}$ . It then parameterizes  $\mathbf{S}$  via an MLP as:
$$
\mathbf {S} (\mathbf {p}) = (\sigma (\mathbf {p}), \mathbf {c} (\mathbf {p})) = \operatorname {M L P} \left(\left\{\left(\mathbf {f} _ {i} \oplus \mathbf {p} _ {i} \right\} _ {i} ^ {M}\right), \right. \tag {10}
$$
where  $\oplus$  is concatenation and  $\mathbf{p}_i$  is the 3D point  $\mathbf{p}$  transformed into the camera coordinates of observation  $i$ . The number of context images  $M$  is flexible, and we may condition  $\mathbf{S}$  on a single or several observations. It will be convenient to refer to a pixelNeRF that is reconstructed from context and target observations  $(\mathbf{O}^{\mathrm{ctxt}},\phi^{\mathrm{ctxt}})$  and  $(\mathbf{O}^{\mathrm{trgt}},\phi^{\mathrm{trgt}})$  as
$$
\mathbf {S} (\cdot \mid \operatorname {e n c} (\mathbf {O} ^ {\mathrm {c t x t}}), \operatorname {e n c} (\mathbf {O} ^ {\mathrm {t r g t}})), \tag {11}
$$
where we make the pixelNeRF encoder enc explicit and drop the poses  $\phi^{\mathrm{trgt}}$  and  $\phi^{\mathrm{ctxt}}$ . We leverage differentiable volume rendering [49] as forward model, such that
$$
\mathbf {O} = \operatorname {r e n d e r} (\mathbf {S}, \phi), \tag {12}
$$
where  $\mathbf{S}$  is rendered from a camera with parameters  $\phi$ .
Implementation of denoise. Fig. 2 gives an overview of the denoising procedure. Following our framework, we obtain the denoised target observation  $\hat{\mathbf{O}}_{t - 1}^{\mathrm{trgt}}$  as:
$$
\hat {\mathbf {O}} _ {t - 1} ^ {\operatorname {t r g t}} = \operatorname {r e n d e r} \left(\mathbf {S} _ {t - 1}, \phi^ {\operatorname {t r g t}}\right), \quad \text {w h e r e} \tag {13}
$$
$$
\mathbf {S} _ {t - 1} = \mathbf {S} (\cdot | \operatorname {e n c} _ {t = 0} \left(\mathbf {O} ^ {\mathrm {c t x t}}\right), \operatorname {e n c} _ {t} \left(\mathbf {O} _ {t} ^ {\mathrm {t r g t}}\right)), \tag {14}
$$
where the image encoder  $\mathsf{enc}_t$  is now conditioned on the timestep  $t$ . In other words, we will generate a target view  $\hat{\mathbf{O}}_{t-1}^{\mathrm{trgt}}$  by rendering the pixelNeRF conditioned on the context and noisy target observations. However, feeding the noisy  $\mathbf{O}_t^{\mathrm{trgt}}$  directly to pixelNeRF is insufficient. This is because the pixel-aligned features  $\mathsf{enc}_t(\mathbf{O})$  are obtained from each view separately - thus, the features generated by  $\mathsf{enc}_t(\mathbf{O}_t^{\mathrm{trgt}})$  will be uninformative. To successfully generate a 3D scene, we have to augment the  $\mathbf{O}_t^{\mathrm{trgt}}$  with information from the context view. We propose to generate conditioning information for  $\mathbf{O}_t^{\mathrm{trgt}}$  by rendering a deterministic estimate  $\mathbf{O}_{\mathrm{det}}^{\mathrm{trgt}} = \text{render}(\mathbf{S}(\cdot|\mathsf{enc}_{t=0}(\mathbf{O}^{\mathrm{ctx}})), \phi^{\mathrm{trgt}})$ . I.e., we condition pixelNeRF only on the context view, and render an estimate of the target view via volume rendering. However, in the extreme case of a completely uncertain target view, this results in a completely blurry image. We thus propose to additionally render high-dimensional features. Recall that any 3D point  $\mathbf{p}$ , we have  $(\sigma(\mathbf{p}), \mathbf{c}(\mathbf{p})) = \mathsf{MLP}_t(\mathbf{p})$ . We modify  $\mathsf{MLP}_t$  to also output a high-dimensional feature and
Single-Image Motion Prediction
Input
Motion Field Samples
Determ.
Figure 5: Qualitative Results for Single-Image Motion Prediction (left) and GAN Inversion (right).
Probabilistic GAN Inversion
Patch
Determ
render a deterministic feature map to augment  $\mathbf{O}_t^{\mathrm{trgt}}$  (only RGB shown in figure). We generate the final 3D scene as  $\mathbf{S}_{t - 1} = \mathbf{S}(\cdot |\mathsf{enc}_{t = 0}(\mathbf{O}^{\mathrm{ctxt}}),\mathsf{enc}_t(\mathbf{O}_{\mathrm{det}}^{\mathrm{trgt}}\oplus \mathbf{O}_t^{\mathrm{trgt}}))$ . The final denoised target view is then obtained according to the rendering Eq. 13 above.
Loss and Training. Our loss consists of simple least-squares terms on re-rendered views, identical to the general loss terms presented in Eqs. 8 and 9, in addition to regularizers that penalize degenerate 3D scenes. We discuss these regularizers, as well as training details, in the supplement.
4.1.1 Results
Datasets We evaluate on two challenging real-world datasets. We use Co3D hydrants [50] to evaluate our method on object-centric scenes. For scene-level 3D synthesis, we use the challenging RealEstate10k dataset [51], consisting of indoor and outdoor videos of scenes.
Baselines We compare our approach with state-of-the-art approaches in deterministic and probabilistic 3D scene completion. We use pixelNeRF as the representative method for deterministic methods that takes a single image as input and deterministically reconstructs a 3D scene. Our method is the first to probabilistically reconstruct 3D scenes in an end-to-end manner. Regardless, we compare with the concurrent SparseFusion [52] that learns an image-space generative model over novel views of a 3D scene. Score distillation of this generative model is required every time we want to obtain a multi-view consistent 3D scene, which is costly.
Qualitative Results. In Fig. 3, we show multiple samples of 3D scenes sampled from a monocular image. For the indoor scenes of RealEstate10k, there are large regions of uncertainty. We can sample from the distribution of valid 3D scenes, resulting in significantly different 3D scenes with plausible geometry and colors. The objects are faithfully reconstructed for the object-centric Co3D scenes, and the uncertainty in the scene is captured. We can sample larger 3D scenes and render longer trajectories by autoregressive sampling, i.e., we treat intermediate diffused images as additional context observations to sample another target observation. The Co3D results in Fig. 3 were generated autoregressively for a complete 360 degrees trajectory. In Fig. 4, we compare our results with pixelNeRF [9] and SparseFusion [5]. pixelNeRF is a deterministic method and thus leads to very blurry results in uncertain regions. SparseFusion reconstructs scenes by score-distillation over a 2D generative model. This optimization is very expensive, and does not lead to natural-looking results.
Quantitative Results. For the object-centric Co3D dataset, we evaluate the accuracy of novel views using PSNR and LPIPS [53] metrics. Note that PSNR/LPIPS are not meaningful metrics for large scenes since the predictions have a large amount of uncertainty, i.e., a wide range of novel view images can be consistent with any input image. Thus, we report FID [54] and KID [55] scores to evaluate the realism of reconstructions in these cases. Our approach outperforms all baselines for LPIPS, FID, and KID metrics, as our model achieves more realistic results. We achieve slightly lower PSNR compared to pixelNeRF [9]. Note that PSNR favors mean estimates, and that we only evaluate our model using a single randomly sampled scene for an input image due to computational constraints.
4.2 Single-Image Motion Prediction
Here, we seek to train a model that, given a single static image, allows us to sample from all possible motions of pixels in the image. Given, for instance, an image of a person performing a task, such as kicking a soccer ball, it is possible to predict potential future states. This is a stochastic problem, as there are multiple possible motions consistent with an image. We train on a dataset of natural videos [56]. We only observe RGB frames and never directly observe the underlying motion, i.e., the pixel correspondences in time are unavailable. We use tuples of two frames from videos within a small temporal window, and use them as our context and target observations for training.
3D Scene Completion
Co3D	RealEstate10k
PSNR↑	LPIPS↓	FID↓	KID↓	FID↓	KID↓
pixelNeRF	17.93	0.54	180.20	0.14	195.40	0.14
SparseFusion	12.06	0.63	252.13	0.16	99.44	0.04
Ours	17.47	0.42	84.63	0.05	42.84	0.01
GAN Inversion
FFHQ
FID↓	KID↓
Determ.	25.7	0.019
Ours	7.45	0.002
Table 1: Quantitative evaluation. (left) We benchmark our 3D generative model with state-of-the-art baselines pixelNeRF [9] and SparseFusion [5]. (right) We benchmark with a deterministic baseline on GAN inversion, which we drastically outperform.
Related Work. Several papers tackle this problem, where motion in the form of optical flow [57-59], 2D trajectories [60, 61], and human motion [62, 63] are recovered from a static image; however, all these methods assume supervision over the underlying motion. Learning to reason about motion requires the neural network to learn about the properties and behavior of the different objects in the world. Thus, this serves as a useful proxy task for representation learning, and can be used as a backbone for many downstream applications [60, 64].
Structure of S and forward model warp. Our signal S stores the appearance and motion information in a 2D grid. At any pixel  $\mathbf{u}$ , the signal is defined as  $\mathbf{S}(\mathbf{u}) = (\mathbf{S}_c(\mathbf{u}),\mathbf{S}_m(\mathbf{u}))$ , where  $\mathbf{S}_c(\mathbf{u})\in \mathbb{R}^3$  is the color value, and  $\mathbf{S}_m(\mathbf{u})\in \mathbb{R}^2$  is a 2D motion vector. The forward model is a warping operator, such that  $\operatorname {warp}(\mathbf{S},\phi)(\mathbf{u} + \phi \mathbf{S}_m(\mathbf{u})) = \mathbf{S}_c(\mathbf{u})$  and  $\phi$  is a scalar that changes the magnitude of motion. We implement this function using a differentiable point splatting operation [65].
Implementation of denoise. The inset figure illustrates our design. We use a 2D network that takes  $\mathbf{O}^{\mathrm{ctx}}$ ,  $\mathbf{O}_t^{\mathrm{trgt}}$ , and  $t$  as input, and generates the motion map  $\mathbf{S}_m$  as the output. The signal is then reconstructed as  $\mathbf{S} = (\mathbf{O}^{\mathrm{ctx}}, \mathbf{S}_m)$ . Context and target frames correspond to parameters  $\phi^{\mathrm{ctx}} = 0$  and  $\phi^{\mathrm{trgt}} = 1$ , and can be reconstructed from the signal using warp.
Loss and Evaluation. Similar to inverse graphics, we use reconstruction and regularization losses. The reconstruction losses are identical to Eqs. 8 and 9, and the regularization loss is a smoothness term that encourages a natural motion of the scene, see supplement for details. We show results in Fig. 5 (left), where we can estimate a diverse set of possible motion flows from monocular images. By smoothly interpolating  $\phi$ , we can generate short video sequences, even though our model only saw low-framederate video frames during training. We also train a deterministic baseline, which only generates a single motion field. Due to the amount of uncertainty in this problem, the deterministic estimate collapses to a near-zero motion field regardless of the input image, and thus, fails to learn any meaningful features from images.
4.3 GAN Inversion
Projecting images onto the latent space of generative adversarial networks is a well-studied problem [8, 66], and enables interesting applications, as manipulating latents along known directions allows a user to effectively edit images [67-69]. Here, we solve the problem of projecting partial images: given a small visible patch in an image, our goal is to model the distribution of possible StyleGAN2 [8] latents that agree with the input patch. There are a diverse set of latents that can correspond to the input observation, and we train our method without observing supervised (image, latent) pairs. Instead, we train on pairs of  $(\mathbf{O}^{\mathrm{ctx}},\mathbf{O}^{\mathrm{trgt}})$  observations, where  $\mathbf{O}^{\mathrm{ctx}}$  are the small patches in images, and  $\mathbf{O}^{\mathrm{trgt}}$  are the full images.
Related Work. While most GAN inversion methods focus on inverting a complete image into the generator's latent space [70-78], some also reconstruct GAN latents from small patches via supervised training. Inversion is not trivial, and papers often rely on regularization [77] or integrate the inversion with editing tasks [79] for higher quality. We also integrate the inpainting task with the inversion, and seek to model the uncertainty of the GAN inversion task given only a partial observation (patch) of the target image.
Structure of S and forward model synthesize. Our signal  $\mathbf{S} \in \mathbb{R}^{512}$  is a 512 dimensional latent code representing the "w" space of StyleGAN2 [8] trained on the FFHQ [80] dataset. The forward model synthesize  $(\mathbf{S}, \phi) = \mathrm{GAN}(\mathbf{S})[\phi]$  first reconstructs the image corresponding to  $\mathbf{S}$  using a
forward pass of the GAN. It then extracts a patch using the forward model's parameters  $\phi$  that encode the patch coordinates.
Implementation of denoise, Loss, and Evaluation. Please see the inset figure for an illustration of the method. The denoising network receives  $\mathbf{O}^{\mathrm{ctx}}$ ,  $\mathbf{O}_t^{\mathrm{trgt}}$ , and timestep t as input, and generates an estimate of the StyleGAN latent w. The loss function is identical to Eq. 8 and compares the reconstructed sample with ground truth.
We show results in Fig. 5 (right). We obtain diverse samples that are all consistent with the input patch. We also compare with a deterministic baseline that minimizes the same loss but only produces a single estimate. While this deterministic estimate also agrees with the input image, it does not model the diversity of outputs. We consequently achieve significantly better FID [54] and KID [55] scores than the deterministic baseline, reported in Tab. 1 (right).
Limitations. While our method makes significant advances in generative modeling, it still has several limitations. Sampling 3D scenes at test time can be very slow, due to the expensive nature of the denoising process and the cost of volume rendering. We need multi-view observations of training scenes for the inverse graphics application. Our models are not trained on very large-scale datasets, and can thus not generalize to out-of-distribution data.
Conclusion We have introduced a new method that tightly integrates differentiable forward models and conditional diffusion models. Our model learns to sample from the distribution of signals trained only using their observations. We demonstrate the efficacy of our approach on three challenging computer vision problems. In inverse graphics, our method, in combination with a 3D-structured conditioning method, enables us to directly sample from the distribution of real-world 3D scenes consistent with a single image observation. We can then render multi-view consistent novel views while obtaining diverse samples of 3D geometry and appearance in unobserved regions of the scene. We further tackle single-image conditional motion synthesis, where we learn to sample from the distribution of 2D motion conditioned on a single image, as well as GAN inversion, where we learn to sample images that exist in the latent space of a GAN that are consistent with a given patch. With this work, we make contributions that broaden the applicability of state-of-the-art generative modeling to a large range of scientifically relevant applications, and hope to inspire future research in this direction.
Acknowledgements. This work was supported by the National Science Foundation under Grant No. 2211259, by the Singapore DSTA under DST00OECI20300823 (New Representations for Vision), by the NSF award 1955864 (Occlusion and Directional Resolution in Computational Imaging), by the ONR MURI grant N00014-22-1-2740, and by the Amazon Science Hub. We are grateful for helpful conversations with members of the Scene Representation Group David Charatan, Cameron Smith, and Boyuan Chen. We thank Zhizhuo Zhou for thoughtful discussions about the SparseFusion baseline. This article solely reflects the opinions and conclusions of its authors and no other entity.
Author contributions. Ayush and Vincent conceived the idea of diffusion with forward models, designed experiments, generated most figures, and wrote most of the paper. Ayush contributed the key insight to integrate differentiable rendering with diffusion models by denoising in image space while generating 3D scenes. Ayush and Vincent generalized this to general forward models, and conceived the single-image motion application. Vincent contributed the 3D-structured conditioning and generated the overview and methods figures. Ayush wrote all initial code and ran all initial experiments. Ayush and Tianwei implemented the inverse graphics application and generated most of the 3D results of our model, while George helped with the baseline 3D results. Ayush executed all single-image motion experiments. George conceived, implemented, and executed all GAN inversion experiments. Semon helped formalizing the method and wrote the proposition and its proof. Frédo and Bill were involved in regular meetings and gave valuable feedback on results and experiments. Josh provided intriguing cognitive science perspectives and feedback on results and experiments, and provided a significant part of the compute. Vincent's Scene Representation Group provided a significant part of the compute, and the project profited from code infrastructure developed by and conversations with other members of the Scene Representation Group.
[67] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. Proc. NeurIPS, 2020. 9
[68] Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhofer, and Christian Theobalt. Stylerig: Rigging stylegan for 3d control over portrait images. In Proc. CVPR, 2020. 9
[69] Yujun Shen, Ceyuan Yang, Xiaou Tang, and Bolei Zhou. Interfacegan: Interpreting the disentangled face representation learned by gans. IEEE transactions on pattern analysis and machine intelligence, 44(4):2004-2018, 2020. 9
[70] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proc. ICCV, 2019. 9
[71] Rameen Abdul, Yipeng Qin, and Peter Wonka. Image2stylegan++: How to edit the embedded images? In Proc. CVPR, 2020. 9
[72] David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, and Antonio Torralba. Semantic photo manipulation with a generative image prior. arXiv preprint arXiv:2005.07727, 2020.9
[73] Yuval Alaluf, Or Patashnik, and Daniel Cohen-Or. Restyle: A residual-based stylegan encoder via iterative refinement. In Proc. ICCV, 2021. 9
[74] Shanyan Guan, Ying Tai, Bingbing Ni, Feida Zhu, Feiyue Huang, and Xiaokang Yang. Collaborative learning for faster stylegan embedding. arXiv preprint arXiv:2007.01758, 2020. 9
[75] Stanislav Pidhorskyi, Donald A Adjeroh, and Gianfranco Doretto. Adversarial latent autoencoders. In Proc. CVPR, 2020. 9
[76] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: a stylegan encoder for image-to-image translation. In Proc. CVPR, 2021. 9
[77] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG), 40(4):1-14, 2021. 9
[78] Tengfei Wang, Yong Zhang, Yanbo Fan, Jue Wang, and Qifeng Chen. High-fidelity gan inversion for image attribute editing. In Proc. CVPR, 2022. 9
[79] Ayush Tewari, Mohamed Elgharib, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, and Christian Theobalt. Pie: Portrait image embedding for semantic control. ACM Transactions on Graphics (TOG), 39(6):1-14, 2020. 9
[80] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proc. CVPR, 2019. 9