Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Models

Authors: ziyi  wu
yulia  rubanova
rishabh  kabra
drew a hudson
igor  gilitschenski
yusuf  aytar
sjoerd  van steenkiste
kelsey r allen
thomas  kipf
google  deepmind
google  research
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 02:09:31

Abstract: We address the problem of multi-object 3D pose control in image diffusion models. Instead of conditioning on a sequence of text tokens, we propose to use a set of per-object representations, *Neural Assets*, to control the 3D pose of individual objects in a scene. Neural Assets are obtained by pooling visual representations of objects from a reference image, such as a frame in a video, and are trained to reconstruct the respective objects in a different image, e.g., a later frame in the video. Importantly, we encode object visuals from the reference image while conditioning on object poses from the target frame, which enables learning disentangled appearance and position features. Combining visual and 3D pose representations in a sequence-of-tokens format allows us to keep the text-to-image interface of existing models, with Neural Assets in place of text tokens. By fine-tuning a pre-trained text-to-image diffusion model with this information, our approach enables fine-grained 3D pose and placement control of individual objects in a scene. We further demonstrate that Neural Assets can be transferred and recomposed across different scenes. Our model achieves state-of-the-art multi-object editing results on both synthetic 3D scene datasets, as well as two real-world video datasets (Objectron, Waymo Open).

Main paper: 
 introduction: 
 From animation movies to video games, the field of computer graphics has long relied on a traditional workflow for creating and manipulating visual content. This approach involves the creation of 3D assets, which are then placed in a scene and animated to achieve the desired visual effects. With the recent advance of deep generative models [26, 50, 77, 82] , a new paradigm is emerging. Diffusion models have achieved promising results in content creation [22, 44, 70, 79] by training on web-scale text-image data [85] . Users can now expect realistic image generation, depicting almost everything describable in text. However, text alone is often insufficient for precise control over the output image.To address this challenge, an emerging body of work has investigated alternative ways to control the image generation process. One line of work studies different forms of conditioning inputs, such as depth maps, surface normals, and semantic layouts [59, 103, 116] . Another direction is personalized image generation [30, 58, 81] , which aims to synthesize a new image while preserving particular aspects of a reference image (e.g., placing an object of interest on a desired background). However, these approaches are still fundamentally limited in their 3D understanding of objects. As a result, they cannot achieve intuitive object control in the 3D space, e.g., rotation. While some recent works introduce 3D geometry to the generation process [8, 61, 67] , they cannot handle multi-object real-world scenes as it is hard to obtain scalable training data (paired images and 3D annotations). We address these limitations by taking inspiration from cognitive science to propose a scalable solution to 3D-aware multi-object control. When humans move through the world, their motor systems keep track of their movements through an efference copy and proprioceptive feedback [4, 96] . This allows the human perceptual system to track objects accurately across time even when the object's relative pose to the observer changes [28] . We use this observation to propose the use of videos of multiple objects as a scalable source of training data for 3D multi-object control. Specifically, for any two frames sampled from a video, the naturally occurring changes in the 3D pose (e.g., 3D bounding boxes) of objects can be treated as training labels for multi-object editing.With this source of training data, we propose Neural Assets -per object latent representations with consistent 3D appearance but variable 3D pose. Neural Assets are trained by extracting their visual appearances from one frame in a video and reconstructing their appearances in a different frame in the video conditioned on the corresponding 3D bounding boxes. This supports learning consistent 3D appearance disentangled from 3D pose. We can then tokenize any number of Neural Assets and feed this sequence to a fine-tuned conditional image generator for precise, multi-object, 3D control.Our main contributions are threefold: (i) A Neural Asset formulation that represent objects with disentangled appearance and pose features. By training on paired video frames, it enables fine-grained 3D control of individual objects. (ii) Our framework is applicable to both synthetic and real-world scenes, achieving state-of-the-art results on 3D-aware object editing. (iii) We extend Neural Assets to further support compositional scene generation, such as swapping the background of two scenes and transferring objects across scenes. We show the versatile control ability of our model in Fig. 1 . 
related work: 
 2D spatial control in diffusion models (DMs). With the rapid growth of diffusion-based visual generation [22, 44, 70, 79, 82, 94] , there have been many works aiming to inject spatial control to pre-trained DMs via 2D bounding boxes or segmentation masks. One line of research achieves this by manipulating text prompts [11, 32, 51] , intermediate attention maps [12, 16, 17, 27, 41, 52, 110] or noisy latents [25, 64, 69, 90] in the diffusion process, without the need to change model weights.Closer to ours are methods that fine-tune pre-trained DMs to support additional spatial conditioning inputs [5, 29, 33, 45, 112, 114] . GLIGEN [59] introduces new attention layers to condition on bounding boxes. InstanceDiffusion [103] further supports object masks, points, and scribbles with a unified feature fusion block. To incorporate dense control signals such as depth maps and surface normals, ControlNet [116] adds zero-initialized convolution layers around the original network blocks. Recently, Boximator [99] demonstrates that such 2D control can be extended to video models with a similar technique. Several existing works [3, 107] leverage natural motion observed in video and similar to our work propose to train on paired video frames to achieve pixel-level control. In our work, we build upon pre-trained DMs and leverage 3D bounding boxes as spatial conditioning, which enables 3D-aware control such as object rotation and occlusion handling.3D-aware image generation. Earlier works leverage differentiable rendering to learn 3D Generative Adversarial Networks (GANs) [34] from monocular images, with explicit 3D representations such as radiance fields [14, 15, 37, 71, 86, 111] and meshes [18, 19, 31, 73, 74] . Inspired by the great success of DMs in image generation, several works try to lift the 2D knowledge to 3D [49, 60, 62, 66, 75, 89, 98, 105] . The pioneering work 3DiM [105] and follow-up work Zero-1-to-3 [61] directly train diffusion models on multi-view renderings of 3D assets. However, this line of research only considers single objects without background, which cannot handle in-the-wild data with complex backgrounds. Closest to ours are methods that process multi-object real-world scenes [2, 72, 84, 115] . OBJect-3DIT [67] studies language-guided 3D-aware object editing by training on paired synthetic data, limiting its performance on real-world images [115] . LooseControl [8] converts 3D bounding boxes to depth maps to guide the object pose. Yet, it cannot be directly applied to edit existing images. In contrast, our Neural Asset representation captures both object appearance and 3D pose. It can be easily trained on real-world videos to achieve multi-object 3D edits. From a methodology perspective, there have been prior works learning disentangled appearance and pose representations for 3D-aware multi-object image editing [71, 100, 111] . However, they are all based on the GAN framework [34] and do not learn generalizable object representations via an encoder. In contrast, we build upon a large-scale pre-trained image diffusion model [79] and powerful feature extractors [13] , enabling editing of complex real-world scenes.Personalized image generation. Since the seminal works DreamBooth [81] and Textual Inversion [30] which perform personalized generation via test-time fine-tuning, huge efforts have been made to achieve this in a zero-shot manner [47, 58, 88, 101, 106, 109] . Most of them are only able to synthesize one subject, and cannot control the spatial location of the generated instance. A notable exception is Subject-Diffusion [65] , which leverages frozen CLIP embeddings for object appearance and 2D bounding boxes for object position. Still, it cannot explicitly control the 3D pose of objects.Object-centric representation learning. Our Neural Asset representation is also related to recent object-centric slot representations [48, 63, 91, 92, 108] that decompose scenes into a set of object entities. Object slots provide a useful interface for editing such as object attributes [93] , motions [87] , 3D poses [46] , and global camera poses [83] . Nevertheless, these models show significantly degraded results on real-world data. Neural Assets also consist of disentangled appearance and pose features of objects. Different from existing slot-based models, we fine-tune self-supervised visual encoders and connect them with large-scale pre-trained DMs, which scales up to complex real-world data.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations of this work in Sec. 5. 
method: neural assets: 
 Inspired by 3D assets in computer graphics software, we propose Neural Assets as learnable objectcentric representations. A Neural Asset comprises an appearance and an object pose representation, which is trained to reconstruct the object via conditioning a diffusion model (Sec. 3.2). Trained on paired images, our method learns disentangled representations, enabling 3D-aware object editing and compositional generation at inference time (Sec. 3.3). Our framework is summarized in Fig. 2 .3D object models, or 3D assets, are basic components of any 3D scene in computer graphics software, such as Blender [20] . A typical workflow includes selecting N 3D assets {â 1 , ..., âN } from an asset library and placing them into a scene. Formally, one can define a 3D asset as a tuple âi (A i , P i ), where A i is a set of descriptors defining the asset's appearance (e.g., canonical 3D shape and surface textures) and P i describes its pose (e.g., rigid transformation and scaling from its canonical pose).Inspired by 3D assets in computer graphics, our goal is to enable such capabilities (i.e., 3D control and compositional generation) in recent generative models. To achieve this, we define a Neural Asset as a tuple a i (A i , P i ), where A i ∈ R (K×D) is a flattened sequence of K D-dimensional vectors describing the appearance of an asset, and P i ∈ R D is a D -dimensional embedding of the asset's pose in a scene. In other words, a Neural Asset is fully described by learnable embedding vectors, factorized into appearance and pose. This factorization enables independent control over appearance and pose of an asset, similar to how 3D object models can be controlled in traditional computer graphics software. Importantly, besides the 3D pose of assets, our approach does not require any explicit mapping of objects into 3D, such as depth maps or the NeRF representation [68] .In the following, we describe how both the appearance A i and the pose P i of a Neural Asset a i are obtained from visual observations (such as an image or a frame in a video). Importantly, the appearance and pose representations are not necessarily encoded from the same observation, i.e., they can be encoded from two separate frames sampled from a video. We find this strategy critical to learn disentangled and controllable representations, which we will discuss in detail in Sec. 3.3.Appearance encoding. At a high level, we wish to obtain a set of N Neural Asset appearance tokens A i from a visual observation x src , where x src can be an image or a frame in a video. While one could approach this problem in a fully-unsupervised fashion, using a method such as Slot Attention [63] to decompose an image into a set of object representations, we choose to use readily-available annotations to allow fine-grained specification of objects of interest. In particular, we assume that a 2D bounding box b i is provided for each Neural Asset a i , specifying which object should be extracted from x src . Therefore, we obtain the appearance representation A i as follows:EQUATIONwhere H i is the output feature map of a visual encoder Enc. RoIAlign [38] extracts a fixed size feature map using the provided bounding box b i which is flattened to form the appearance token A i . This factorization allows us to extract N object appearances from an image with just one encoder forward pass. In contrast, previous methods [65, 109] crop each object out to extract features separately, and thus requires N encoder passes. This becomes unaffordable if we jointly fine-tune the visual encoder, which is key to learning generalizable features as we will show in the ablation study.Pose encoding. The pose token P i of a Neural Asset a i is the primary interface for controlling the presence and 3D pose of an object in the rendered scene. In this work, we assume that the object pose is provided in terms of a 3D bounding box, which fully specifies its location, orientation, and size in the scene. Formally, we take four corners spanning the 3D bounding box 1 and project them to the image plane to get {c j i = (h j i , w j i , d j i )} 4 j=1 , with the projected 2D coordinate (h j i , w j i ), and the 3D depth d i,j . We obtain the pose representation P i for a Neural Asset as follows:EQUATIONwhere we first concatenate the four corners c j i to form C i ∈ R 12 , and then project it to P i ∈ R D via an MLP. We tried the Fourier coordinate encoding in prior works [59, 103] but did not find it helpful.There are alternative ways to represent 3D bounding boxes (e.g., concatenation of center, size, and rotation commonly used in 3D object detection [57] ), which we compare in Appendix B. 4 . In this work, we assume the availability of training data with 3D annotations -obtaining high-quality 3D object boxes for videos at scale is still an open research problem, but may soon be within reach given recent progress in monocular 3D detection [102] , depth estimation [7, 113] , and pose tracking [9] .Serialization of multiple Neural Assets. We encode a set of N Neural Assets into a sequence of tokens that can be appended to or used in place of text embeddings for conditioning a generative model. In particular, we first concatenate the appearance token A i and the pose token P i channel-wise, and then linearly project it to obtain a Neural Asset representation a i as follows:EQUATIONChannel-wise concatenation uniquely binds one pose token with one appearance representation in the presence of multiple Neural Assets. An alternative solution is to learn such association with positional encoding. Yet, it breaks the permutation-invariance of the generator against the order of input objects and leads to poor results in our preliminary experiments. Finally, we simply concatenate multiple Neural Assets along the token axis to arrive at our token sequence, which can be used as a drop-in replacement for a sequence of text tokens in a text-to-image generation model.Background modeling. Similar to prior works [71, 111] , we found it helpful to encode the scene background separately, which enables independent control thereof (e.g., swapping out the scene, or controlling global properties such as lighting). We choose the following heuristic strategy to encode the background: to avoid leakage of foreground object information, we mask all pixels within asset bounding boxes b i . We then pass this masked image through the image encoder Enc (shared weights with the foreground asset encoder) and apply a global RoIAlign, i.e., using the entire image as region of interest, to obtain a background appearance token A bg ∈ R (K×D) . Similar to a Neural Asset, we also attach a pose token P bg to A bg . This can either be a timestep embedding of the video frame (relative to the source frame) or a relative camera pose embedding, if available. In the serialized representations, the background token is treated the same as Neural Assets, i.e., we concatenate A bg and P bg channel-wise and linearly project it. Finally, the foreground assets a i and the background token are concatenated along the token dimension and used to condition the generator.To generate images from Neural Assets, we make minimal assumptions about the architecture or training setup of the generative image model to ensure compatibility with future large-scale pretrained image generators. In particular, we assume that the generative image model accepts a sequence of tokens as conditioning signal: for most base models this would be a sequence of tokens derived from text prompts, which we can easily replace with a sequence of Neural Asset tokens.As a representative for this class of models, we adopt Stable Diffusion v2.1 [79] for the generative decoder. See Appendix C for details on this model. Starting from the pre-trained text-to-image checkpoint, we fine-tune the entire model end-to-end to accept Neural Assets tokens instead of text tokens as conditioning signal. The training and inference setup is explained in the following section.Learning from frame pairs. As outlined in the introduction, we require a scalable data source of object-level "edits" in 3D space to effectively learn multi-object 3D control capabilities. Video data offers a natural solution to this problem: as the camera and the content of the scene moves or changes over time, objects are observed from various view points and thus in various poses and lighting conditions [78] . We exploit this signal by randomly sampling pairs of frames from video clips, where we take one frame as the "source" image x src and the other frame as the "target" image x tgt .As described earlier, we obtain the appearance token A i of Neural Assets from the source frame x src by extracting object features using 2D box annotations. Next, we obtain the pose token P i for each extracted asset from the target frame x tgt , for which we need to identify the correspondences between objects in both frames. In practice, such correspondences can be obtained, for example, by applying an object tracking model on the underlying video. Finally, with the associated appearance and pose representations, we condition the image generator on them and train it to reconstruct the target frame x tgt , i.e., using the denoising loss of Stable Diffusion v2.1 in our case. Such a paired frame training strategy forces the model to learn an appearance token that is invariant to object pose and leverage the pose token to synthesize the new object, avoiding the trivial solution of simple pixel-copying. Figure 3 : Single-object editing results on OBJect unseen object subset. We evaluate on the Translation, Rotation, and Removal tasks. We follow 3DIT [67] to compute metrics inside the edited object's bounding box. Our results are averaged over 3 random seeds. Test-time controllability. The learned disentangled representations naturally enable multi-object scene-level editing as we will show in Sec. 4.3. Since we encode 3D bounding boxes to pose tokens P i , we can move, rotate, and rescale objects by changing the box coordinates. We can also compose Neural Assets a i across scenes to generate new scenes. In addition, our background modeling design supports swapping the environment map of the scene. Importantly, as we will see in the experiments, our image generator learns to naturally blend the objects into their new environment at new positions, with realistic lighting effects such as rendering and adapting shadows correctly.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss potential positive and negative society impacts in Appendix D. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiments: 
 In this section, we conduct extensive experiments to answer the following questions: (i) Can Neural Assets enable accurate 3D object editing? (ii) What practical applications does our method support on real-world scenes? (iii) What is the impact of each design choice in our framework?Datasets. We select four datasets with object or camera motion, which span different levels of complexity. OBJect [67] is introduced in 3DIT [67] , which is one of our baselines. It contains 400k synthetic scenes rendered by Blender [20] with a static camera. Up to four Objaverse [21] assets are placed on a textured ground and only one object is randomly moved on the ground. For a fair comparison with 3DIT, we use 2D bounding boxes plus rotation angles as object poses, and follow them to base our model on Stable Diffusion v1.5 [79] . MOVi-E [36] consists of Blender simulated videos with up to 23 objects. It is more challenging than OBJect as it has linear camera motion and there can be multiple objects moving simultaneously. Objectron [1] is a big step up in complexity as it captures real-world objects with complex backgrounds. 15k object-centric videos covering objects from nine categories are recorded with 360 • camera movement. Waymo Open [97] is a real-world self-driving dataset captured by car mounted cameras. We follow prior work [111] to use only the front view and filter out cars that are too small. See Appendix A.1 for more details on datasets.Baselines. We compare to methods that can perform 3D-aware editing on existing images and have released their code. 3DIT [67] fine-tunes Zero-1-to-3 [61] on the OBJect dataset to support translation and rotation of objects. However, it cannot render big viewpoint changes as it does not encode camera poses. Following [67] , we create another baseline (dubbed Chained) by using SAM [55] to segment the object of interest, removing it using Stable Diffusion inpainting model [79] , running Zero-1-to-3 to rotate and scale the object, and stitching it to the target position. Since none of these baselines can control multiple objects simultaneously, we apply them to edit all objects sequentially.Evaluation settings. We report common metrics to measure the quality of the edited image -PSNR, SSIM [104] , LPIPS [117] , and FID [42] . Following prior works [49, 67] , we also compute objectlevel metrics on cropped out image patches of edited objects. To evaluate the fidelity of edited objects, we take the DINO [13] feature similarity metric proposed in [81] . On video datasets, we randomly sample source and target images in each testing video and fix them across runs for consistent results.Implementation Details. For all experiments, we resize images to 256 × 256. DINO self-supervised pre-trained ViT-B/8 [13] is adopted as the visual encoder Enc, and jointly fine-tuned with the generator. All our models are trained using the Adam optimizer [53] with a batch size of 1536 on 256 TPUv4 chips. For inference, we generate images by running the DDIM [95] sampler for 50 steps.For more training and inference details, please refer to Appendix A.4.Single-object editing. We first compare the ability to control the 3D pose of a single object on the OBJect dataset. Fig. 3 presents the results on the unseen object subset. We do not show FID here as it mainly measures the visual quality of generated examples, which does not reflect the editing accuracy. For results on the seen object subset and FID, please refer to Appendix B.1, where we observe similar trends. Compared to baselines, our model does not condition on text (e.g., the category name of the object to edit) as in 3DIT and is not trained on curated multi-view images of 3D assets as in Zero-1-to-3. Still, we achieve state-of-the-art performance on all three tasks. This is because our Neural Assets representation learns disentangled appearance and pose features, which is able to preserve object identity while changing its placement smoothly. Also, the fine-tuned DINO encoder generalizes better to unseen objects compared to the frozen CLIP visual encoder used by baselines.Multi-object editing. Fig. 4 shows the results on MOVi-E, Objectron, and Waymo Open, where multiple objects are manipulated in each sample. Similar to the single-object case, we compute metrics inside the object bounding boxes, and leave the image-level results to Appendix B.1. Our model outperforms baselines by a sizeable margin across datasets. Fig. 5 presents the qualitative results. When there are multiple objects of the same class in the scene (e.g., boxes in the MOVi-E example and cars on Waymo Open), 3DIT is unable to edit the correct instance. In addition, it generalizes poorly to real-world scenes. Thanks to the object cropping step, Chained baseline can identify the correct object of interest. However, the edited object is simply pasted to the target location, leading to unrealistic appearance due to missing lighting effects such as shadows. In contrast, our model is able to control all objects precisely, preserve their fidelity, and blend them into the background naturally. Since we encode the camera pose, we can also model global viewpoint change as shown in the third row. See Appendix B.1 for additional qualitative results.In this section, we show versatile control of scene objects on Waymo Open. For results on Objectron, please refer to Appendix B.3. As shown in Fig. 6 , we can translate and rotate cars in driving scenes.The model understands the 3D world as objects zoom in and out when moving, and show consistent The generator is able to harmonize objects with the new environment. For example, the car lights are turned on and rendered with specular highlight when using a background image from a night scene.Replace BackgroundWe study the effect of each component in the model. All ablations are run on Objectron since it is a real-world dataset with complex background, and has higher object diversity than Waymo Open.Visual encoder. Previous image-conditioned diffusion models [49, 61, 62] usually use the frozen image encoder of CLIP [76] to extract visual features. Instead, as shown in Fig. 9a , we found that both MAE [39] and DINO [13] pre-trained ViTs give better results. This is because CLIP's image encoder only captures high-level semantics of images, which suffices in single-object tasks, but fails in our multi-object setting. In contrast, MAE and DINO pre-training enable the model to extract more fine-grained features. Besides, DINO outperforms MAE as its features contain richer 3D information, which aligns with recent research [6] . Finally, jointly fine-tuning the image encoder learns more generalizable appearance tokens in Neural Assets, leading to the best performance.Background modeling. We compare our full model with two variants: (i) not conditioning on any background tokens (dubbed No-BG), and (ii) conditioning on background appearance tokens but not using relative camera pose as pose tokens (dubbed No-Pose). As shown in Fig. 9b , our background modeling strategy performs the best in image-level metrics as backgrounds usually occupy a large part of real-world images. Interestingly, our method also achieves significantly better object-level metrics. This is because given background appearance and pose, the model does not need to infer them from object tokens, leading to more disentangled Neural Assets representations. Training strategy. As described in Sec. 3.3, we train on videos and extract appearance and pose tokens from different frames. We compare such design with training on a single frame in Fig. 9c . Our paired frame training strategy clearly outperforms single frame training. Since the appearance token is extracted by a ViT with positional encoding, it already contains object position information, which acts as a shortcut for image reconstruction. Therefore, the model ignores the input object pose token, resulting in poor controllability. One way to alleviate this is removing the positional encoding in the image encoder (dubbed NO-PE), which still underperforms paired frame training. This is because to reconstruct objects with visual features extracted from a different frame, the model is forced to infer their underlying 3D structure instead of simply copying pixels. In addition, the generator needs to render realistic lighting effects such as shadows under the new scene configuration. 
conclusion: 
 In this paper, we present Neural Assets, vector-based representations of objects and scene elements with disentangled appearance and pose features. By connecting with pre-trained image generators, we enable controllable 3D scene generation. Our method is capable of controlling multiple objects in the 3D space as well as transferring assets across scenes, both on synthetic and real-world datasets. We view our work as an important step towards general-purpose neural-based simulators.Limitations and future works. One main failure case of our model is symmetry ambiguity. As can be seen from the rotation results in Fig. 10 (a) , the handle of the cup gets flipped when it rotates by 180 degree. Another failure case that only happens on Objectron is the entanglement of global camera motion and local object movement (Fig. 10 (b) ). This is because Objectron videos only contain camera motion while objects always stay static. Both issues will likely be resolved if we train our model on larger-scale datasets with more diverse object and camera motion.An ideal Neural Asset should enable control over all potential configurations of an object such as deformation (e.g., a walking cat), rigid articulation (e.g., opening of a scissor), and structural decomposition (e.g., tomatoes being cut). In this work, we first tackle the foremost important aspect, i.e., controlling 3D rigid object pose and background composition which applies to almost all the objects. Hence our current method does not allow for controlling structural changes. However, it can be adapted when suitable datasets are developed that capture other changes in objects.Another limitation is that our approach is currently limited to existing datasets that have 3D bounding box annotations. Yet, with recent advances in vision foundation models [9, 55, 113] , we may soon have scalable 3D annotation pipelines similar to their 2D counterparts. One notable example is OmniNOCS [56] , which works on both Waymo and Objectron (datasets we used in this work), and diverse, in-the-wild Internet images for a wide range of object classes. It can be used to create larger open domain datasets to learn Neural Assets. We see this as an interesting future direction. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: We discuss safeguards on our image generator in Appendix D.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We provide the license information in Appendix A.1 and Appendix A.2.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: In this section, we provide full details on the datasets, baselines, evaluation settings, and the training and inference implementation of our model.OBJect [67] consists of Blender [20] rendered scenes where multiple (up to four) objects are placed on a flat textured ground. The objects come from a 59k subset of Objaverse dataset [21] . A total of 18 background maps are used to provide environmental lighting. Four types of object-level editing are provided -translation, rotation, removal, and insertion, each with 100k simulated data. Notably, only one object is edited in each data, and the translation and rotation is always on the ground (i.e., perpendicular to the gravity vector). For a fair comparison with the 3DIT baseline [67] , we use the 2D rotated bounding box to represent object pose, which is composed of two corners of the 2D bounding box and the rotation angle over the gravity axis. This dataset is under the Open Data Commons Attribution License (ODC-By) 2 .MOVi-E [36] contains 10k videos simulated using Kubric [36] . Each scene contains 11 to 23 real-world objects from the Google Scanned Objects (GSO) repository [23] . At the start of each video, several objects are thrown to the ground to collide with other objects. Similar to OBJect, environmental lighting is provided by a randomly sampled environment map image. The camera follows a small linear motion. The full data generation pipeline is under the Apache 2.0 license3 .Objectron [1] contains 15k object-centric video clips of common daily objects covering nine categories. Each video comes with object pose tracking throughout the video, and we process it to obtain 3D bounding boxes. Since this dataset does not provide 2D bounding box labels, we project the eight corners of 3D boxes to the image, and take the tight bounding box of projected points as 2D boxes. Objectron is licensed under the Computational Use of Data Agreement 1.0 (C-UDA-1.0) 4 .Waymo Open [97] . The Waymo Open Dataset consists of 1k videos of self-driving scenes recorded by car mounted cameras. Following prior works [24, 111] , we take the front view camera and bounding box annotations of cars. Notably, the 3D bounding boxes only have a heading angle (rotation along the yaw-axis) annotation, and thus we treat the other two rotation angles as 0. Besides, the provided 2D boxes and 3D boxes are not aligned, preventing us from doing paired frame training. We instead project 3D boxes to get associated 2D boxes similar to on Objectron. Waymo Open is licensed under the Waymo Dataset License Agreement for Non-Commercial Use (August 2019) 5 .Data Pre-processing. For all datasets, we resize the images to 256 × 256 regardless of the original aspect ratio. On Objectron, we discard all videos from the bike class as it contains many blurry frames and inaccurate 3D bounding box annotations. On Waymo, we remove all cars whose 2D bounding box is smaller than 1% of the image area. We do not apply data augmentation except on Waymo Open, where we apply random horizontal flip and random resize crop following [24] .3DIT [67] fine-tunes Zero-1-to-3 [61] to support scene-level 3D object edits. We generate the editing instruction from the target object pose, such as the translation coordinate and the rotation angle. However, this method does not support large viewpoint changes as it does not encode camera poses. We take their official code and pre-trained weights of the Multitask variant. 3DIT is under the CreativeML Open RAIL-M license 6 .Chained. This baseline is inspired by [67, 72] , where we chain multiple models together to achieve 3D-aware object editing. An editing step usually contains three steps: (i) crop out the object of interest and inpaint its region with backgrounds, (ii) synthesize the object under the new pose, and (iii) place the object to the new location. For (i), we apply SAM [55] to segment the object using 2D bounding box prompt, and inpaint the original object region with Stable Diffusion v2 inpainting model [79] . For (ii), we run Zero-1-to-3 [61] to re-pose the object according to the target 3D bounding box. For (iii), following [61] , we first get the alpha mask of the re-posed object using an online tool 7 , and insert it to the new position via alpha blending. It is worth noting that Zero-1-to-3 does not support camera rotation over the roll axis. For all models, we take their official code and pre-trained weights. SAM is under the Apache 2.0 license 8 . Stable Diffusion v2 inpainting model is under the CreativeML Open RAIL++-M License 9 . Zero-1-to-3 is under the MIT license 10 . The online alpha mask extraction tool is under the Apache 2.0 license11 .We report PSNR, SSIM [104] , LPIPS [117] , and FID [42] to measure the accuracy of the edited image. We compute metrics both on the entire image, and within the 2D bounding box of edited objects. For box-level metrics, we follow [67] to crop out each object and directly run the metric without resizing. We also evaluate the identity preservation of objects using the DINO [13] feature similarity proposed in [81] , which runs a DINO self-supervised pre-trained ViT on cropped object patches to extract features and compute the cosine similarity between predicted and ground-truth image.Model architecture. We take Stable Diffusion (SD) v2.1 [79] as our image generator except for experiments on the OBJect dataset, where we use SD v1.5 for a fair comparison with baselines. Similar to prior works [49, 89] , we also observe clearly better performance using SD v2.1 compared to v1.5. However, we note that our Neural Assets framework generalizes to any image generator that conditions on a sequence of tokens. We implement the visual encoder Enc with a DINO selfsupervised pre-trained ViT-B/8 [13] , which outputs a feature map of shape 28 × 28 given a 256 × 256 image. For each object, we apply RoIAlign [38] to extract a 2 × 2 small feature map and flatten it, i.e., the appearance token A i has a sequence length of K = 4. Since the conditioning token dimension of pre-trained SD v2.1 is 1024, we use a two-layer MLP to transform the 3D bounding boxes input to D = 1024, and linearly project the concatenated appearance and pose token back to 1024. For background modeling, we mask all pixels within object boxes by setting them to a fixed value of 0.5, and extract features with the same DINO encoder. Instead, the pose token is obtained by applying a different two-layer MLP on the relative camera pose between the source and the target image.Training. We implement the entire Neural Assets framework in JAX [10] using the Flax [40] neural network library. We train all model components jointly using the Adam optimizer [53] with a batch size of 1536 on 256 TPUv5 chips (16GB memory each). We use a peak learning rate of 5 × 10 -5 for the image generator and the visual encoder, and a larger learning rate of 1 × 10 -3 for remaining layers (MLPs and linear projection layers). Both learning rates are linearly warmed up in the first 1,000 steps and stay constant. A gradient clipping of 1.0 is applied to stabilize training. We found that the model overfits more severely on real-world data with complex backgrounds compared to synthetic datasets. Therefore, we train the model for 200k steps on OBJect and MOVi-E which takes 24 hours, and 50k steps on Objectron and Waymo Open which takes 6 hours. In order to apply classifier-free guidance (CFG) [43] , we randomly drop the appearance and pose token (i.e., setting them as zeros) with a probability of 10%. CFG improves the performance and also alleviates overfitting in training.Inference. We run the DDIM sampler [95] for 50 steps to generate images. We found the model works well with CFG scale between 1.5 and 4, and thus choose to use 2.0 in all the experiments. Table 1 : Single-object editing results on OBJect. We evaluate on the Translation, Rotation, and Removal tasks. We follow 3DIT [67] to evaluate on both seen and unseen object subsets, and compute metrics inside the edited object's bounding box. Our results are averaged over 3 random seeds.Unseen Objects Model PSNR ↑ SSIM ↑ LPIPS ↓ FID ↓ PSNR ↑ SSIM ↑ LPIPS ↓ FIDWe present full quantitative results on OBJect in Tab. 1, and on MOVi-E, Objectron, and Waymo Open in Tab. 2. Compared to the main paper, we report additional FID metrics and results on the unseen object subset for OBJect, while for the other three datasets, we report additional FID and DINO feature similarity metrics, plus results computed over the entire image (Image-Level). Overall, we observe similar trends as in the main paper, where our Neural Assets model significantly outperforms baselines across all datasets. In Fig. 11 , we show additional qualitative comparisons.We present all quantitative results of our ablation studies on Objectron (Sec. 4.4) in Tab. 3, Tab. 4, and Tab. 5. We observe similar trends on all metrics at both image-and object-level. In Fig. 12 and Fig. 13 , we show controllable scene generation results on Objectron. Objectron videos only have global camera movement, while the objects are static. Still, our Neural Assets model learns disentangled foreground and background representations. As can be seen from the results, we can rotate the foreground objects while keeping the background fixed, or swap background between scenes. Importantly, our model inpaints the masked background regions not occupied by the novel object, and renders realistic shadows around the object, which is far beyond simple pixel copying.In Fig. 14 , we visualize the object pose representation we use. Given a 3D bounding box of an object, we project its four corners to the image space, and concatenate their 2D coordinates and depth values to obtain a 12-D pose vector. The 2D projected points resemble a local coordinate frame for the object, specifying its position, rotation, and scale. On the other hand, the depth is useful for determining the occlusion of objects.There are alternative ways to represent the object pose, e.g., the coordinate of the 3D box center C with its size and rotation which is commonly used in 3D object detection [57] . These representations achieve similar results on MOVi-E and Objectron. However, their learned rotation controllability is significantly worse than our representation on Waymo. This is because most of the cars on Waymo are not rotated (turn left / right), leading to very few training data on object rotation. If we directly input the rotation angle to the model, it tends to ignore it. In contrast, due to prospective projection, the projected local coordinate frame of unrotated cars still look "rotated" when they are not strictly in front of the ego vehicle. This provides much more training signal to learn the rotation of objects.Diffusion model [44, 94] is a class of generative models that learns to generate samples by iteratively denoising from a standard Gaussian distribution. It consists of a denoiser θ , usually implemented as a U-Net [80] , which predicts the noise added to the data x. Instead of denoising raw pixels, Stable Diffusion introduces a VAE [54] tokenizer to map images to low-dimensional latent code z and applies the denoiser on it. In addition, the denoiser is conditioned on text and thus supports text-to-image generation. In this work, we simply replace the text embeddings with Neural Assets a i and fine-tune the model to support appearance and pose control of 3D objects. Controllable visual generation is an important task in computer vision. Neural Assets equip generative models with an intuitive interface to control their behaviors, which enables more interpretable AI algorithms and may potentially benefit other fields such as computer graphics and robotics. We believe this work will benefit the whole research community and the society.Potential negative societal impacts. Since we fine-tune large-scale pre-trained generative models in our pipeline, we inherit limitations of these base models, such as dataset selection bias. Such bias might be problematic when human subjects are involved, though our current approach is only capable of rigid object control and does not consider humans as an "asset" yet. Further study on how such bias affects model performance is required for mitigating negative societal impacts that could arise from this work [35] .This work was carried out at Google. Igor Gilitschenski contributed to the project in an advisory capacity.NeurIPS Paper Checklist Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Only three corners are needed to fully define a 3D bounding box, but we found a 4-corner representation beneficial to work with. Previous research[118] also shows that over-parametrization can benefit model learning.https://huggingface.co/datasets/allenai/object-edit/blob/main/README.mdhttps://github.com/google-research/kubric/blob/main/LICENSEhttps://github.com/google-research-datasets/Objectron#licensehttps://waymo.com/open/termshttps://github.com/allenai/object-edit/blob/main/LICENSEhttps://github.com/OPHoperHPO/image-background-remove-toolhttps://github.com/facebookresearch/segment-anything#licensehttps://huggingface.co/stabilityai/stable-diffusion-2-inpaintinghttps://github.com/cvlab-columbia/zero123/blob/main/LICENSEhttps://github.com/OPHoperHPO/image-background-remove-tool/blob/master/LICENSE 


Decision: This paper proposes a method for controlled generation of compositional scenes using diffusion models with explicit pose control of the individual objects in the scene. To achieve this the authors propose a new Neural Assets representation, which disentangles the identity and the pose of the object. These representations are further appended into the text prompt provided to the diffusion model for conditioning. Additionally the authors propose a novel methodology to train their system using videos containing the objects in different poses in different frames, which further helps to disentangles the identity and expression representations. On several benchmark datasets the authors show significant improvement in performance over the state of the art and show for the first time 3D control for composes scenes instead of just single objects.

All reviewers appreciated the novelty and elegance of the proposed method and the high quality of the results achieved. They further appreciated the well-written nature of the paper. All reviewers further championed the acceptance of this work. This work represents a significant step forward in synthesizing 3D controlled scenes composes of individual objects. The AC concurs and recommends acceptance.

Reviews: review: paper_summary: paper_summary: This paper addresses the task of multi-object pose and scale control in image generation and editing using diffusion models. The authors introduce an object-centric representation with a disentangled pose and appearance called Neural Asset. The neural asset for each object in an image is estimated using pose and appearance encoders. The authors exploit the naturally occurring pose and appearance variations in training video datasets to train such encoders. To generate an image based on neural assets, the authors fine-tune an image diffusion model conditioned on the sequence of neural assets as its condition. The authors evaluate and compare their method with existing baselines, showcasing the ability of their method for pose and scale control, object removal, and background change.
strength_weakness: strength_weakness: - Based on the provided results, the background has small changes when an object in the image is edited even in the areas far from the edited object
- The proposed method is mainly limited to the domain of the training dataset as opposed to the recent training-free diffusion-based image editing methods
questions: questions: See the weaknesses
limitations: limitations: sufficiently discussed
paper_summary: paper_summary: The paper introduces an object-centric representation for multi-object 3D pose control in image diffusion models. Instead of text token sequences, it utilizes Neural Assets, which are per-object representations learned by pooling visual features from reference images and reconstructing objects in target frames. This disentangles appearance and pose features by fine-tuning a pre-trained text-to-image diffusion model while conditioning on target frame poses, enabling fine-grained 3D pose and placement control in scenes. The results are demonstrated on both synthetic and real-world datasets in multi-object editing tasks.
strength_weakness: strength_weakness: I value the performance the proposed model presented in manipulating objects while keeping the rest unchanged. I spent quite some time trying to comprehend how the current design leads to its performance, and still have the following questions, which might be helpful if they are discussed in the paper:  
[-] The paper utilizes projection and depth of the absolute object poses in the pose embedding. This is kind of confusing as the it requires the model to encode how the object should look like in the canonical space. If this is the case, the model should be able to perform per-object reconstruction with the object-centric representation through images under different poses and even maybe object pose estimation. A common practice is to use the relative pose change, as the authors discussed in the background modeling.  
[-] The authors leverage SD as base model, but one significant difference is that the proposed model put the learning bottleneck to the conditional module, compared to Zero-1-to-3 or ControlNet. All the useful features are embedded in the conditions now, including how the images should look like. In original SD, the conditional module is relatively light-weight. I'm wondering if there are limitations for the current design to capture the complex conditions and recover the input images.  
[-] One related concern is that from the results, it seems the reconstruction results are also decent. I'm wondering what role is the randomness of the noise play in the current model. Will the model output different images if the seed changes?  
[-] How should the model comprehend the objects in the target images that do not exist in the src images, especially during training?

Some other concerns:  
[-] The model may lack a total understanding of the scene, especially the interactions between objects, e.g.,  when one object is supported by another or when the 3D bounding boxes of two objects collide with each other. How will the model react in such scenarios?
[-] The proposed model faces limitations in fully manipulating 3D object and their relationships as naturally as humans do. The requirement of 2D/3D bounding box pose challenges to the detection models and the manipulation as projection points is not intuitive as humans who commonly use language.
questions: questions: See above. Overall, I think the paper offers a new perspective for object-centric learning, especially under the image generation settings. I still have some concerns and will consider raising my rating if the rebuttal resolves them.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper considers controlling the 3d poses of different objects in an image generated by a diffusion model. By conditioning the diffusion model on a sequence of per-object appearance and pose tokens instead of text tokens it is possible to finely control the object poses, and furthermore to e.g. transfer objects between scenes. The diffusion model is trained using videos with 3d bounding box annotations and correspondences between different frames, and it is shown experimentally to generate realistic images and to outperform related methods.
strength_weakness: strength_weakness: - There seems to be some issues with the modelling of the background for the Objectron dataset, though this is not an issue with Waymo Open. Failure cases are e.g. that rotations and scaling of objects also rotate and scale the background around the object, even outside the object bounding boxes. Did the authors try e.g. any alternative ways of modelling the background to fix this?
- This is not a major weakness, but there is no measure of 3d consistency, which is common in previous work, e.g.  zero-1-to-3 and follow up works often measure 3d reconstruction errors on e.g. Google Scanned Objects (GSO). It would be straight-forward to evaluate the 3d consistency of the proposed method on this dataset, and compare to e.g. zero-1-to-3. It should be mentioned though that this would not reflect the full strength of the proposed method since it can handle multiple objects and diverse backgrounds which is not captured when evaluating on that dataset.
questions: questions: Please see weaknesses

Can the authors clarify if code and models will be released? It is mentioned that the authors “will consider” releasing code upon acceptance (line 844). Will any trained models be released? Will code to replicate the training be released?
limitations: limitations: This is adequately addressed
paper_summary: paper_summary: This paper presents a novel method for 3D-aware editing of multi-object images. Given the original image and 2D bounding boxes to specify the objects to be edited, and the 3D pose as the target of the editing, the proposed method encodes the object appearance feature and their pose features, which are then taken as conditions to generate the edited images. The proposed approach disentangles the appearances and poses of the objects, and enables flexible and better control of the 3D pose of each object in the image.
strength_weakness: strength_weakness: Basically, this paper provides an excellent 3D-aware image editing model for multi-object images benefiting from the pre-trained image generation model and the large training dataset. The idea of disentangling the appearance and pose features of objects is the key for multi-object image editing. However, since there have been existing works on this topic, such as BlobGAN-3D, it is essential to add an in-depth discussion to validate the superiority of the proposed network architecture.

[1] BlobGAN-3D: A Spatially-Disentangled 3D-Aware Generative Model for Indoor Scenes
questions: questions: Line 244 claims that the proposed method is not pre-trained on multi-view rendering of 3D assets. It is actually trained on the different frames of videos, which is more appropriate for the multi-object image setting. This claim is true but I would not take it as an advantage of the proposed method.
limitations: limitations: This paper includes the discussions of limitations and broader impacts. However, it's better to present some failure cases of the proposed approach to better understand its performance.

Comments: title: Final Comment
Authors' Response
Post Rebuttal
comment: I thank the authors for their response. I keep my score.
Dear Reveiwer h1dc,

The authors have provided a response to the questions that you raised in your review. It would be great if you could read and acknowledge their response before 11:59 pm AOE August 13, 2024.

Best,
AC
I thank the authors for their answers. I keep my score as 7 (accept).
Thank you for the response. It addresses my questions well. I would maintain my rating as "accept".
I thank the authors for their responses.  They address my questions about the model design and qualitative results, and thus I have raised my score accordingly.