Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Models
Authors: Ziyi Wu
Yulia Rubanova
Rishabh Kabra
Drew A. Hudson
Igor Gilitschenski
Yusuf Aytar
Sjoerd van Steenkiste
Kelsey R Allen
Thomas Kipf
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Controllable generation
3D-aware editing
diffusion model
Primary area: diffusion_based_models
TL;DR: We propose a Neural Assets representation that enables 3D-aware multi-object control in real-world scenes

Abstract: We address the problem of multi-object 3D pose control in image diffusion models. Instead of conditioning on a sequence of text tokens, we propose to use a set of per-object representations, *Neural Assets*, to control the 3D pose of individual objects in a scene. Neural Assets are obtained by pooling visual representations of objects from a reference image, such as a frame in a video, and are trained to reconstruct the respective objects in a different image, e.g., a later frame in the video. Importantly, we encode object visuals from the reference image while conditioning on object poses from the target frame, which enables learning disentangled appearance and position features. Combining visual and 3D pose representations in a sequence-of-tokens format allows us to keep the text-to-image interface of existing models, with Neural Assets in place of text tokens. By fine-tuning a pre-trained text-to-image diffusion model with this information, our approach enables fine-grained 3D pose and placement control of individual objects in a scene. We further demonstrate that Neural Assets can be transferred and recomposed across different scenes. Our model achieves state-of-the-art multi-object editing results on both synthetic 3D scene datasets, as well as two real-world video datasets (Objectron, Waymo Open).

Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Models
Ziyi Wu Yulia Rubanova', Rishabh Kabra Drew A. Hudson', Igor Gilitschenski*4. Yusuf Aytar', Sjoerd van Steenkiste", Kelsey R. Allen', Thomas Kipf
Google DeepMind 2Google Research University of Toronto VVector Institute UCL
Abstract
We address the problem of multi-object 3D pose control in image diffusion models. Instead of conditioning on sequence of text tokens, we propose use a of per- object representations, Neural Assets, to control the 3D pose of individual objects in a scene. Neural Assets are obtained by pooling visual representations of objects from reference image, such as frame in a video, and are trained to reconstruct the respective objects different image, e.g., later frame in the video. Importantly, we encode object visuals from the reference image while conditioning on object poses from the target frame. This enables learning disentangled appearance and pose features. Combining visual and 3D pose representations in sequence-of- of- tokens format allows us to keep the text-to-image architecture of existing models, with Neural Assets in place of text tokens. By fine-tuning pre-trained text-to- image diffusion model with this information, our approach enables fine-grand mme-grained 3D pose and placement control of individual objects in scene. We further demonstrate that Neural Assets can be transferred and recomposed across different scenes. Our model achieves tate-offthhe-rr multi-object editing results on both synthetic 3D scene datasets, as well as two real-world video datasets (Objectron, Waymo Open). Additional details and video results available our project page.
Introduction
From animation movies to video games, the field of computer graphics has long relied on a traditional workflow for creating and manipulating visual content. This approach involves the creation of 3D assets, which are then placed in scene and animated to achieve the desired visual effects. With the recent advance of deep generative models [26, 50, 77, 82], a new paradigm is emerging. Diffusion models have achieved promising results in content creation [22, 44, 70, 79] by training on web-scale text-image data [85]. Users can expect realistic image generation, depicting almost everything describable in text. However, text alone is often insufficient for precise control over the output image.
To address this challenge, emerging body of work has investigated alternative ways to control the image generation process. One line of work studies different forms of conditioning inputs, such as depth maps, surface normals, and semantic layouts [59, 103, 116]. Another direction is personalized image generation [30, 58, 81], which aims to synthesize a new image while preserving particular aspects of a reference image (e.g.. placing object of interest However, these approaches are still fundamentally limited in their 3D understanding of objects. As recent
desired background).
result, they cannot achieve intuitive object control in the 3D space, e.g., rotation. While works introduce 3D geometry to the generation process [8, 61, 67]. they cannot handle multi-object real-world scenes as is hard obtain scalable training data (paired images and 3D annotations).
Work done while interning Google.
Contact: tkipf@google com. Project page: neural-assets seneets.githut .ithhu.......................... io
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Original Image
Translate Obj.
Rotate Obj.
Rescale Obj.
Replace Obj.
Replace Bg.
Figure 1: 3D-aware editing with our Neural Asset representations. Given souree image and object 3D bounding boxes, we can translate, rotate, and rescale the object. In addition, we support compositional generation by transferring objects or backgrounds across images.
We address these limitations by taking inspiration from cognitive science to propose scalable solution to 3D-aware multi-object control. When humans move through the world, their motor systems keep track of their movements through an efference copy and proprioceptive feedhack [4, 96]. This allows the human perceptual system to track objects accurately across time even when the object's relative pose the observer changes [28]. We use this observation propose the of videos of multiple objects as a scalable source of training data for 3D multi-object control. Specifically, for any two frames sampled from a video, the naturally occurring changes in the 3D pose (e.g., 3D bounding boxes) of objects can be treated as training labels for multi-object editing.
With this source of training data, propose Neural Assets per object latent representations with consistent 3D appearance but variable 3D pose. Neural Assets are trained by extracting their visual appearances from frame in video and reconstructing their appearances in different frame in the video conditioned on the corresponding 3D bounding boxes. This supports learning consistent 3D appearance disentangled from 3D pose. We then tokenize any number of Neural Assets and feed this sequence to a fine-tuned conditional image generator for precise, multi-object, 3D control.
Our main contributions are threefold: (i) A Neural Asset formulation that represent objects with disentangled appearance and pose features. By training on paired video frames, enables fine-grained 3D control of individual objects. (ii) Our framework is applicable to both synthetic and real- world scenes, achieving state-of state-of-the-ar the-art results on 3D-aware object editing. (iii) We extend Neural Assets to further support compositional scene generation, such as swapping the background of two scenes and transferring objects across scenes. We show the versatile control ability our model in Fig. 1.
2 Related Work
2D spatial control in diffusion models (DMs), With the rapid growth of diffusion- visual generation [22, 44, 70, 79, 82, 94], there have been many works aiming to inject spatial control to pre-trained DMs via 2D bounding boxes or segmentation masks. One line of research achieves this by manipulating text prompts [1 1, 32, 51], intermediate attention maps [12, 16, 17, 27, 52, 110] or noisy latents [25, 64, 69, 90] in the diffusion process, without the need to change model weights.
Closer to ours are methods that fine-tune pre-trained DMs to support additional spatial conditioning inputs [5, 29, 33, 45, 112, 114]. GLIGEN [59] introduces new attention layers to condition on bounding boxes. InstanceDiffusion [103] further supports object masks, points, and scribbles with unified feature fusion block. To incorporate dense control signals such as depth maps and surface normals, ControlNet [116] adds pero-initialized convolution layers around the original network blocks. Recently, Boximator [99] demonstrates that such 2D control can be extended to video models with similar technique. Several existing works [3, 107] leverage natural motion observed in video and similar to our work propose to train on paired video frames to achieve pixel-level control. In our work, we build upon pre-trained DMs and leverage 3D bounding boxes as spatial conditioning, which enables 3D-aware control such object rotation and occlusion handling.

---

3D-aware image generation. Earlier works leverage differentiable rendering to learn 3D Generative Adversarial Networks (GANs) [34] from monocular images, with explicit 3D representations such as radiance fields [14, 15, 37, 71, 86, 111] and meshes [18, 19, 31, 73, 74). Inspired by the great success of DMs in image generation, several works try to lift the 2D knowledge to 3D [49, 60, 62, 66, 75, 89, 98, 105]. The pioneering work 3DiM [105] and follow-up work 7ero-1-to-- [61] directly train diffusion models on multi-view renderings of 3D assets. However, this line of research only considers single objects without background, which cannot handle in-the- wild data with complex backgrounds. Closest to ours are methods that process multi-object real-world scenes [2, 72, 84, 115). OBJect- 3DIT [67] studies language- genuaagg-guieedd 3D- object editing by training paired synthetic data, limiting its performance on real-world images [115]. LooseControl [8] converts 3D bounding boxes to depth maps to guide the object pose. Yet, it cannot be directly applied to edit existing images. In contrast, our Neural Asset representation captures both object appearance and 3D pose. It can be easily trained on real-world videos to achieve multi-object 3D edits.
From a methodology perspective, there have been prior works learning disentangled appearance and pose representations for 3D-aware multi-object image editing [71, 100, 111]. However, they all based on the GAN framework [34] and do not learn generalizable object representations via an encoder. In contrast, we build upon a large-scale pre-trained image diffusion model [79] and powerful feature extractors [13], enabling editing of complex real real-world scenes.
Personalized image generation. Since the seminal works DreamBooth [81] and Textual Inver- sion [30] which perform personalized generation via test-time fine-tuning, huge efforts have been made to achieve this in zero-shot manner [47, 58, 88, 101, 106, 109]. Most of them are only able to synthesize one subject, and cannot control the spatial location of the generated instance. A notable exception is Subject-Diffusion [65], which leverages frozen CLIP embeddings for object appearance and 2D bounding boxes for object position. Still, it cannot explicitly control the 3D pose of objects.
Object-centric representation learning. Our Neural Asset representation is also related to recent object-centric slot representations [48, 63, 91, 92, 108] that decompose scenes into a set of object entities. Object slots provide a useful interface for editing such as object attributes [93], motions [87], 3D poses [46], and global camera poses [83]. Nevertheless, these models show significantly degraded results on real-world data. Neural Assets also consist of disentangled appearance and pose features of objects. Different from existing slot-based models, we fine-tune self-supervised visual encoders and connect them with large-scale pre-trained DMs, which scales up to complex
data.
3 Method: Neural Assets
Inspired by 3D assets in computer graphics software, we propose Neural Assets as learnable object- centric representations. A Neural Asset comprises an appearance and an object pose representation. which is trained to reconstruct the object via conditioning a diffusion model (Sec. 3.2). Trained on paired images, our method learns disentangled representations, enabling 3D-aware object editing and compositional generation at inference time (Sec. 3.3). Our framework is summarized in Fig. 2.
3.1 Background: 3D Assets in Computer Graphics
3D object models, or 3D assets, are basic components of any 3D scene in computer graphics software, such Blender [20]. A typical workflow includes selecting N 3D assets {āi,..., ..., ân) from an asset library and placing them into scene. Formally, one can define 3D asset tuple à; - (Ai,, Pt). where A; is a set of descriptors defining the asset's appearance (e.g., canonical 3D shape and surface textures) and P, describes its pose (c.g., rigid transformation and scaling from its canonical pose).
3.2 Neural Assets
Inspired by 3D assets in computer graphics, our goal is to enable such capabilities (i.e., 3D control and compositional generation) in recent generative models. To achieve this, we define a Neural Asser as a tuple a; & (A,, P.). where A; E (x is flattened sequence of K D-ddiennsioall vectors describing the appearance of an asset, and P; E R a D'-dimensional embedding of the asset's pose in scene. In other words, Neural Asset is fully described by learnable embedding vectors, factorized into appearance and pose. This factorization enables independent control over appearance and pose of asset, similar to how 3D object models can be controlled in traditional computer
3

---

(a) Extracting Neural Assets
(b) Controllable generation
(c) Test-time compositionality
Neural Asset
Assets
Rol
Appearance
Align
Target
Image diflusion
New Asset
Image diffusion
Image
Rotate
pose
BG
BG
Reconstructed image
New composed image
src image
1gt image
Figure 2: Neural Assets framework. (a) We train our model on pairs of video frames, which contain objects under different poses. We encode appearance tokens from source image with RoIAlign, and pose tokens from the objects' 3D bounding boxes in target image. They are combined to form our Neural Asset representations. (b) An image diffusion model conditioned on Neural Assets and separate background token to reconstruct the target image the training signal. (c) During inference, we can manipulate the Neural Assets to control the objects in the generated image: rotate the object's pose (blue) replace object by a different one from another image (pink).
graphics software. Importantly, besides the 3D pose of assets, approach does not require any explicit mapping of objects into 3D, such as depth maps or the NeRF representation [68].
3.2.1 Asset Encoding
In the following, we describe how both the appearance A, and the pose P. of Neural Asset a are obtained from visual observations (such as an image or a frame in a video). Importantly, the appearance and pose representations are not necessarily encoded from the same observation, i.e., they can be encoded from two separate frames sampled from a video. We find this strategy critical to learn disentangled and controllable representations, which will discuss in detail in Sec. 3.3.
Appearance encoding. At high level, we wish to obtain set of N Neural Asset appearance tokens A; from visual observation Isre, where Usre can be an image or frame in video. While one could approach this problem in fully-nnsuerrvised fashion, using method such as Slot Attention [63] to decompose an image into set of object representations, we choose to use readily-availaae annotations to allow fine-grained specification of objects of interest. In particular, we assume that a 2D bounding box 6; is provided for each Neural Asset a, specifying which object should be extracted from Trrc Therefore, we obtain the appearance representation A, as follows:
A; = Flatten (RoleeRRRRlllggg (H;,
H; = Enc((are).
(1)
where H; is the output feature map of visual encoder Enc. RolAlign [38] extracts a fixed size feature map using the provided bounding box b, which flattened to form the appearance token A, This factorization allows us to extract N object appearances from an image with just one encoder forward pass. In contrast, previous methods [65, 109] crop each object out to extract features separately, and thus requires N encoder passes. This becomes unaffordable if we jointly fine-tune the visual encoder, which is key to learning generalizable features as we will show the ablation study.
Pose encoding. The pose token P; of Neural Asset is the primary interface for controlling the presence and 3D pose of an object in the rendered scene. In this work, we assume that the object pose is provided in terms of a 3D bounding box, which fully specifies its location, orientation, and size the scene. Formally, we take four corners spanning the 3D bounding box! and project them to the image plane to get {c/ = (Ng, wi = with the projected 2D coordinate (h, w²), and the 3D depth di We obtain the pose representation P; for Neural Asset follows:
(2) where we first concatenate the four corners &: to form C₁ E 122 and then project it to P. E RD via an MLP. We tried the Fourier coordinate encoding in prior works [59, 103] but did not find it helpful.
P. = MLP(C). C: = =Concat|c2 2,, c3)),
There are alternative ways to represent 3D bounding boxes (e.g., concatenation center, size, and rotation commonly used in 3D object detection [57]), which we compare in Appendix B.4. In this
Only three corners are needed to fully detine 3D bounding box, but we found 4 cocorrr representation beneficial to work with. Previous research [1 [118] also shows that over over- parametrizzaiio can benefit model learning.

---

work, we assume the availability of training data with 3D annotations obtaining high-quality 3D object boxes for videos at scale is still an open research problem, but may soon be within reach given recent progress in monocular 3D detection [102], depth estimation [7, 113], and pose tracking [9].
Serialization of multiple Neural Assets. We encode set of N Neural Assets into sequence of tokens that can be appended to or used in place of text embeddings for conditioning a generative model. In particular, we first concatenate the appearance token A, and the pose token P. channel-wise, and then linearly project it to obtain a Neural Asset representation as follows:
a; Linear(@;)) (ā:), a; Concat[A, P.] EERRR 1KxD+D'
(3)
Channel-wist concatenation uniquely binds one pose token with one appearance representation in the presence of multiple Neural Assets. An alternative solution is to learn such association with positional encoding. Yct, it breaks the permutation inviriiacce of the generator against the order of input objects and leads to poor results in our preliminary experiments. Finally, we simply concatenate token sequence, which can be used as
multiple Neural Assets along the token axis to arrive drop-in replacement for a sequence of text tokens in a text-to-image generation model.
Background modeling. Similar to prior works [71, 111], we found it helpful to encode the scene background separately, which enables independent control thereof (e.g., swapping out the scene, or controlling global properties such as lighting). We choose the following heuristic strategy to encode the background: to avoid leakage of foreground object information, we mask all pixels within asset bounding boxes b,. We then pass this masked image through the image encoder Enc (shared weights with the foreground asset encoder) and apply a global RoLAlign, i.c., using the entire image as region of interest, to obtain a background appearance token Abg E (K)))) Similar to a Neural Asset, we also attach a pose token Pog to Abg. This can either be a timestep embedding of the video frame (relative to the source frame) or a relative camera pose embedding, if available. representations, the background token is treated the same as Neural Assets, i.e., we concatenate Ang and Pog channel-wise and linearly project it. Finally, the foreground assets a, and the background token are concatenated along the token dimension and used to condition the generator.
the serialized
3.2.2 Generative Decoder
To generate images from Neural Assets, we make minimal assumptions about the architecture or training setup of the generative image model to ensure compatibility with future aarge-scale pre- trained image generators. In particular, we assume that the generative image model accepts sequence of tokens conditioning signal: for most base models this would be sequence of tokens derived from text prompts, which we can easily replace with a sequence of Neural Asset tokens.
As representative for this class of models, we adopt Stable Diffusion v2.1 [79] for the generative decoder. See Appendix C for details on this model. Starting from the pre-trained text-to-image checkpoint, we fine-tune the entire model end-to-end to accept Neural Assets tokens instead of text tokens as conditioning signal. The training and inference setup is explained in the following section.
3.3 Learning and Inference
Learning from frame pairs. As outlined in the introduction, we require a scalable data source of object-level "edits" in 3D space to effectively learn multi-object 3D control capabilities. Video data offers natural solution to this problem: as the camera and the content of the scene moves or changes over time, objects are observed from various view points and thus in various poses and lighting conditions [78]. We exploit this signal by randomly sampling pairs of frames from video clips, where we take one frame as the "source" image Earc and the other frame as the "target" image Tigt-
As described carlier, we obtain the appearance token A, of Neural Assets from the source frame Tare by extracting object features using 2D box annotations. Next, we obtain the pose token P. for each extracted asset from the targer frame Figt, for which we need to identify the correspondences between objects in both frames. In practice, such correspondences can be obtained, for example, by applying an object tracking model on the underlying video. Finally, with the associated appearance and pose representations, we condition the image generator on them and train to reconstruct the target frame Tigt. i.e., using the denoising loss of Stable Diffusion v2.1 1 in our case. Such paired frame training strategy forces the model to learn an appearance token that is invariant to object pose and leverage the pose token to synthesize the new object, avoiding the trivial solution of simple pixel-copying.
5

---

30
0.5
Chained
3DIT
24-
0.5
0.4
Qurs
18
0.3
0.3
0.2-
Transiation
Rotation
Removal
Transiation
Rotation
Removal
Translation
Rotation
Removal
(a) PSNR (higher better)
(b) SSIM (higher is hetter)
(c) LPIPS (lower is better)
Figure 3: Single-object editing results OB,eet unseen object subset. We evaluate the Translation, Rotation, and Removal tasks. We follow 3DIT [67] to compute metrics inside the edited object's bounding box. Our results averaged over 3 random seeds.
20
0.5
0.6
Chained
1 Wayma
3DIT
17
0.4
Ours
14-
0.4
0.3
MOVI-E
Objectron
Waymo
MOVi-E
Objectron
Objectron
Wayma
(a) PSNR (higher better)
(b) SSIM (higher is better)
(c) LPIPS (lower is hetter)
Figure 4: Multi-object editing results on MOVi-E, Objectron, and Waymo Open (denoted as Waymo in the figures). We compute metrics inside the edited objects' bounding boxes.
Test-time controllability. The learned disentangled representations naturally enable multi-object scene-level editing we will show in Sec. 4.3. Since we encode 3D bounding boxes to pose tokens P. we can move, rotate, and rescale objects by changing the box coordinates. We can also compose Neural Assets across scenes to generate new scenes. In addition, our background modeling design supports swapping the environment map of the scene. Importantly, as we will see the experiments, our image generator learns to naturally blend the objects into their new environment at new positions, with realistic lighting effects such as rendering and adapting shadows correctly.
4 Experiments
In this section, we conduct extensive experiments to answer the following questions: (i) Can Neural Assets enable accurate 3D object editing? (ii) What practical applications does our method support on real-world scenes? (iii) What is the impact of each design choice in our framework?
4.1 Experimental Setup
Datasets. We select four datasets with object or camera motion, which span different levels of complexity. OBJect [67] is introduced 3DIT [67], which is one of our baselines. It contains 400k synthetic scenes rendered by Blender [20] with static camera. Up to four Objaverse [21] assets are placed on textured ground and only one object is randomly moved on the ground. For fair comparison with 3DIT, we use 2D bounding boxes plus rotation angles as object poses, and follow them to base our model on Stable Diffusion v1.5 [79]. MOVI-E [36] consists Blender simulated videos with up to 23 objects. It is more challenging than OBJect as it has linear camera motion and there can be multiple objects moving simultancously. Objectron [1] a big step up in complexity as it captures real-world objects with complex backgrounds. 15k object-centric videos covering objects from nine categories are recorded with 360 camera movement. Waymo Open [97] a real-world self-driving dataset captured by mounted cameras. We follow prior work [111] use only the front view and filter out cars that are too small. See Appendix A. for more details on datasets.
We compare to methods that can perform 3D-aware editing existing images and have released their code. 3D/T [67] fine-tunes Zero-I-to- [61] on the OBJect dataset to support translation and rotation of objects. However, it cannot render big viewpoint changes as it does not encode camera poses. Following [67], we create another baseline (dubbed Chained) by using SAM [55] to segment the object of interest, removing it using Stable Diffusion inpainting model [79], running Zero-I-to-3
Baselines.

---

Source Image
Chained
3DIT
Ours
Target Image
MOVi-E
Objectron
Waymo Open
Figure 5: Qualitative comparison on MOVi-E, Objectron, and Waymo Open. All models gen- erate new image given source image and the 3D bounding box of target objects. Our method performs the best in object identity preservation, editing accuracy, and background modeling.
rotate and scale the object, and stitching the target position. Since none of these baselines can control multiple objects simultancously, we apply them to edit all objects sequentially.
Evaluation settings. We report common metrics to measure the quality of the edited image PSNR. SSIM [104], LPIPS [117], and FID [42]. Following prior works [49, 67]. we also compute object- level metrics cropped out image patches of edited objects. To evaluate the fidelity of edited objects, we take the DINO [13] feature similarity metric proposed in [81]. On video datasets, we randomly sample source and target images in each testing video and fix them across runs for consistent results.
Implementation Details. For all experiments, we resize images to 256 256. DINO self-supervised pre-trained ViT-B/8 [13] is adopted as the visual encoder Enc, and jointly fine-tuned with the generator. All our models are trained using the Adam optimizer [53] with a batch size of 1536 on 256 TPUv4 chips. For inference, generate images by running the DDIM [95] sampler for 50 steps. For more training and inference details, please refer to Appendix A.4.
4.2 Main Results
Single-object editing. We first compare the ability control the 3D pose of single object on the OBJect dataset. Fig. presents the results on the unseen object subset. We not show FID here mainly measures the visual quality of generated examples, which does not reflect the editing accuracy.
7

---

Original Image
Translate
Rotate
Figure 6: Object translation and rotation by manipulating 3D bounding boxes on Waymo Open. See our project page for videos and additional object rescaling results.
Original Image
Reconstruction
Remove
Segment
Replace
Recompose
Figure 7: Compositional generation results Waymo Open. By composing Neural Assets, can remove and segment objects, as well as transfer and recompose objects between scenes.
For results on the seen object subset and FID, please refer to Appendix B. where observe similar trends. Compared to baselines, our model does not condition on text (c.g., the category name of the object to edit) as in 3DIT and is not trained on curated multi-view images of 3D assets in Zero-I-to-3, Still, we achieve state-of-the-art performance on all three tasks. This is because our Neural Assets representation learns disentangled appearance and pose features, which is able to preserve object identity while changing its placement smoothly. Also, the tune- DINO encoder generalizes better to unseen objects compared to the frozen CLIP visual encoder used by baselines.
Multi-object editing. Fig. 4 shows the results on MOVi-E, Objectron, and Waymo Open, where multiple objects are manipulated in each sample. Similar to the single-object case, we compute metrics inside the object bounding boxes, and leave the image-level results to Appendix B. Our model outperforms baselines by sizeable margin across datasets. Fig. 5 presents the qualitative results. When there are multiple objects of the same class in the scene (c.g., boxes in the MOVi-E example and cars Waymo Open), 3DIT is unable to edit the correct instance. In addition, it generalizes poorly to real-world scenes. Thanks to the object cropping step, Chained baseline can identify the correct object of interest. However, the edited object is simply pasted to the target location, leading to unrealistic appearance due to missing lighting effects such as shadows. In contrast, our model is able to control all objects precisely, preserve their fidelity, and blend them into the background naturally. Since we encode the camera pose, we can also model global viewpoint change shown in the third row. See Appendix B.1 for additional qualitative results.
4.3 Controllable Scene Generation
In this section, we show versatile control of scene objects on Waymo Open. For results on Objectron, please refer to Appendix B.3. As shown in Fig. 6, we can translate and rotate cars in driving scenes. The model understands the 3D world as objects zoom in and out when moving, and show consistent
8

---

Replace Background
Figure 8: Transfer backgrounds between scenes by replacing the background token on Waymo Open. The objects can adapt to new environments, e.g., the car lights are turned on at night.
PSNR
LPIPS
PSNR
LPIPS
PSNR
LPIPS L
18
16
16
0.28
15
14
0.27
14
0.24
0.22
0.22
CLIP
MAE
DINO
FT-DINO
CLIP
MAF
DINO FT-DINO
No 8G
No pase
Ours
No NG-8G
Ne-Pose
Ours
Single (Ne-PE)
Single
Paired
Single (No PE)
Single
Paired
(a) Visual encoder (FT: fine-tuned).
(b) Background modeling.
Training strategy.
Figure 9: Comparison of (a) visual encoders, (b) background modeling, and (c) training strate- gies on Objectron. Bold entry denotes our full model. See text for each variant. We report PSNR and LPIPS computed within object bounding boxes, and leave other metrics to Appendix B.2.
novel views when rotating. Fig. presents our ability of compositional generation, where objects removed, segmented out, and transferred across scenes. Notice how the model handles occlusion and inpaints the scene properly. Finally, Fig. demonstrates background swapping between scenes. The generator is able to harmonize objects with the new environment. For example, the car lights are turned on and rendered with specular highlight when using a background image from a night scene.
4.4 Ablation Study
We study the effect of each component in the model. All ablations real-world dataset with complex background, and has higher object diversity than Waymo Open.
Objectron since it is
Visual encoder. Previous congee-onditiioned diffusion models [49, 61, 62] usually use the frozen image encoder of CLIP [76] extract visual features. Instead, as shown Fig. 9a, we found that both MAE [39] and DINO [13] pre-trained ViTs give better results. This is because CLIP's image encoder only captures high-level semantics of images, which suffices in single-object tasks, but fails in our multi-object setting. In contrast, MAF and DINO pre-training enable the model to extract more fine- fine-graine -e--ggrained features. Besides, DINO outperforms MAE as its features contain richer 3D information, which aligns with recent research [6]. Finally, jointly fine-tuning the image encoder learns more generalizable appearance tokens in Neural Assets, leading to the best performance.
Background modeling, We compare our full model with two variants: (i) not conditioning any background tokens (dubbed No-BG), and (ii) conditioning on background appearance tokens but not using relative camera pose as pose tokens (dubbed No-Pose). As shown in Fig. 9b, our background modeling strategy performs best image-level metrics as backgrounds usually occupy large part of real-world images. Interestingly, our method also achieves significantly better object-level metrics. This is because given background appearance and pose, the model does not need to infer them from object tokens, leading to more disentangled Neural Assets representations.

---

(a) Symmetry ambiguity when rotating objects
(b) Object-camera motion entanglement
Figure 10: Failure case analysis. Our model mainly has two failure cases: (a) symmetry ambiguity, where the handle of the cup gets flipped when it rotates by 180 degrees; (b) camera-ohject motion entanglement, where the background also moves when we translate the foreground object. Both issues will likely be resolved if we train our Neural Assets model on more diverse data.
Training strategy. As described Sec. 3.3, we train on videos and extract appearance and pose tokens from different frames. We compare such design with training single frame in Fig. 9c. Our paired frame training strategy clearly outperforms single frame training. Since the appearance token is extracted by ViT with positional encoding, it already contains object position information, which acts as a shortcut for image reconstruction. Therefore, the model ignores the input object pose token, resulting in poor controllability. One way alleviate this is removing the positional encoding in the image encoder (dubbed NO-PE), which still underperforms paired frame training. This is because to reconstruct objects with visual features extracted from different frame, the model is forced infer their underlying 3D structure instead of simply copying pixels. In addition, the generator needs to render realistic lighting effects such shadows under the new scene configuration.
Conclusion
In this paper, we present Neural Assets, vector-based representations of objects and scene elements with disentangled appearance and pose features. By connecting with pre-trained image generators, we enable controllable 3D scene generation. Our method is capable of controlling multiple objects in the 3D space well as transferring assets across scenes, both on synthetic and real-world datasets. We view our work as an important step towards general-purpose neural-base based simulators.
Limitations and future works. One main failure case of our model is symmetry ambiguity. As can be seen from the rotation results in Fig. 10 (a), the handle of the cup gets flipped when it rotates by 180 degree. Another failure that only happens Objectron is the entanglement of global camera motion and local object movement (Fig. 10 (b)). This is because Objectron videos only contain camera motion while objects always stay static. Both issues will likely be resolved if train model on larger-scale datasets with more diverse object and camera motion.
An ideal Neural Asset should enable control over all potential configurations of object such as deformation (e.g.. walking cat), rigid articulation (e.g., opening scissor), and structural decomposition (e.g., tomatoes being cut). In this work, we first tackle the foremost important aspect, ie.. controlling 3D rigid object pose and background composition which applies to almost all the objects. Hence current method does not allow for controlling structural changes. However, it can be adapted when suitable datasets are developed that capture other changes in objects.
Another limitation is that our approach is currently limited to existing datasets that have 3D bounding box annotations. Yet, with recent advances in vision foundation models [9, 55, 113], we may soon have scalable 3D annotation pipelines similar to their 2D counterparts. One notable example is OmniNOCS [56], which works on both Waymo and Objectron (datasets we used in this work), and diverse, in-the-wild Internet inages for a wide range of object classes. It can be used to create larger open domain datasets to learn Neural Assets. We see this as an interesting future direction.
Acknowledgements
We would like to thank Etienne Pot, Klaus Greff, Shlomi Fruchter, and Amir Hertz for their advise regarding infrastructure. We would further like to thank Mehdi S. M. Sajjadi, João Carreira, Sean Kirmani, Yi Yang, Daniel Zoran, David Fleet, Kevin Murphy, and Mike Mozer for helpful discussions.
10

---

References
[1] Adel Ahmadyan, Liangkai Zhang, Artsiom Ablavatski, Jianing Wei, and Matthias Grundmann. Objectron: A large scale dataset of object-centric videos in the wild with pose annotations. In CVPR, 2021. 6, 18
[2] Hadi Alzayer, Zhihao Xia, Xuaner Zhang, Eli Shechtman, Jia-Bin Huang, and Michael Gharbi. Magic Fixup: Streamlining photo editing by watching dynamic videos. arXiv preprint arXiv:2403. /3044 2024. 3
[3] Hadi Alzayer, Zhihao Xia, Xuaner Zhang. Eli Shechtman, Jia-Bin Huang, and Michael Gharbi. Magic fixup: Streamlining photo editing by watching dynamic videos. arXiv preprint 13444 2024.
[4] Michael A Arbib. The handbook of brain theory and neural networks. MIT Press, 2003. 2
[5] Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin. SpaText: Spatio-textual representation for controllable image generation. In CVPR, 2023.
[6] Mohamed EI Banani, Amit Raj, Kevis-Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3d awareness of visual foundation models. In CVPR, 2024.
Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller ZoeDepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.12288 2023. 5
[8] Shariq Farooq Bhat, Niloy Mitra, and Peter Wonka. LooseControl: Lifting controlnet for generalized depth conditioning. In CVPR, 2024. 1,3
[9] Jan Kautz Stan Birchfield Bowen Wen, Wei Yang. FoundationPose: Unified 6d pose estimation and tracking of novel objects. In CVPR, 2024. 5, 10
[10] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milnn and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http://gi git.bb.ttuuutt.ggggg www.hhuww.oomeoeeeeeeeeeeeeeeeee coon/google/jaxx 19
[11] Tim Brooks, Aleksander Holynski, and Alexei A Efros. InstructPix2Pix: Learning to follow image editing instructions. In CVPR, 2023. 2
[12] Mingdeng Cao, Xintao Wang, Zhongang Qi. Ying Shan, Xiaohu Qie, and Yingiang Zheng. MasaCtrl: Tuning-fre -free mutual self-attention control for consistent image synthesis and editing. In ICCV, 2023. 2
[13] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021. 3, 7, 9, 19
[14] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-GAN: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In CVPR. 2021. 3
[15] Eric R Chan, Connor 7. Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry- aware 3d generative adversarial networks. In CVPR, 2022.
[16] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and and-Exeite Attention-based semantic guidance for text-to-image diffusion models. TOG, 2023. 2
[17] Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross- attention guidance. In WACV, 2024. 2

---

[18] Wenzheng Chen, Huan Ling, Jun Gao, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. Learning predict 3d objects with an interpolation- differentiable renderer. NeurlPS, 2019.
[19] Wenzheng Chen, Joey Litalien, Jun Gao, Zian Wang, Clement Fuji Tsang, Sameh Khamis, Or Litany, and Sanja Fidler. DIB-R++: learning to predict lighting and material with hybrid differentiable renderer. NeurIPS, 2021.
[20] Blender Online Community. Blender a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. URL http: //www./wwwww/w blender.
org- 3, 6, 18
[21] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In CVPR, 2023. 6, 18
[22] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurIPS, 2021. 1,2
[23] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google Scanned Objects: A high- quality dataset of 3d scanned household items. In ICRA, 2022. 18
[24] Gamaleldin Fathy Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael Curtis Mozer, and Thomas Kipf. SAVi++: Towards end-to-end object-centric learning from real-world videos. NeurIPS, 2022. 18
[25] Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self-guidance for controllable image generation. NeurlPS, 2023. 2
[26] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
[27] Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training- -free structured diffusion guidance for compositional text-to-imant synthesis. In ICLR, 2023. 2
[28] Leon Festinger and Lance Kirkpatrick Canon. Information about spatial location based on knowledge about efference. Psychological Review, 1965.
[29] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-A-Scenn ASecnee Scene-based text-to-image generation with human priors. ECCV, 2022.
[30] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth word: Personalizing text-to-image generation using textual inversion. In ICLR, 2023. 3,
[31] Jun Gao, Tianchang Shen, Zian Wang. Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler. GET3D: A generative model high quality 3d textured shapes learned from images. NewrIPS, 2022. 3
[32] Songwei Ge, Taesung Park, Jun-Yan Zhu, and Jia-Bin Huang. Expressive text-to-image generation with rich text. In ICCV, 2023. 2
[33] Vidit Goel, Elia Peruzzo, Yifan Jiang, Dejia Xu, Nicu Sebe, Trevor Darrell, Zhangyang Wang. and Humphrey Shi. PAIR-Diffusion: Object-level image editing with stuuture-umd-apppearacce paired diffusion models. In CVPR, 2024.7 2
[34] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. NeurIPS, 2014.
[35] Google. Safety & fairness considerations for generative models, 2023. URL https:// sevelopers.googlee www/wochineeeeeeeeeeeococceceee htw.://hwe....eeeeeeoeoooorrr//e wewaaaffttytt--aaaae ai. 24
12

---

[36] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, et al. Kubric: A scalable dataset generator. In CVPR, 2022. 6, 18
[37] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. StyleNeRF: A style-based 3d aware generator for high-resolution image synthesis. In ICLR, 2022. 3
[38] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, 2017. 19
[39] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In CVPR, 2022. 9
[40] Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc Zee. Flax: neural network library and ecosystem for JAX, 2020. URL
http:// http:/ggitiin github...... www.hhu...omeeeeeee www/ con//oogle/flar 19
[41] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. Prompt-to-Promp image editing with cross-attention control. In ICLR, 2023. 2
[42] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to a local nash equilibrium. NeurIPS, 2017. 7, 19
[43] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS Workshop, 2021. 19
[44] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020. 1,2,22
[45] Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate Anyone: Consistent and controllable image-to-video synthesis for character animation. arXiv preprint arXiv:2311.77117, 2023.
[46] Allan Jabri, Sjoerd van Steenkiste, Emiel Hoogeboom, Mehdi SM Sajjadi, and Thomas Kipf. DORSal: Diffusion for object-centric representations of scenes et In ICLR, 2024.
[47] Xuhui Jia, Yang Zhao, Kelvin CK Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, and Yu-Chuan Su. Taming encoder for zero fine-tuning image customization with text-to text-to-image diffusion models. arXiv preprint arXiv:2304.02642, 2023. 3
[48] Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn. Object-Centric slot diffusion. NeurIPS, 2023. 3
[49] Yash Kant, Ziyi Wu, Michael Vasilkovsky, Guocheng Qian, Jian Ren, Riza Alp Guler, Bernard Ghanem, Sergey Tulyakov, Igor Gilitschenski, and Aliaksandr Siarohin. SPAD: Spatially aware multiview diffusers. In CVPR, 2024. 3, 7, 9, 19
[50] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, 2020.
[51] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Trani. Imagic: Text-based real image editing with diffusion models. In CVPR, 2023. 2
[52] Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung- Woo Ha, and Jun-Yan Zhu. Dense text-to-image generation with attention modulation. In ICCV. 2023. 2
[53] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 7, 19
[54] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014. 22
13

---

[55] Alexander Kirillov, Eric Mintun. Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment Anything, In ICCV, 2023. 6, 10, 18
[56] Akshay Krishnan, Abhijit Kundu, Kevis-Kokits, Maninis, James Hays, and Matthew Brown. OmniÑOCS: A unified nocs dataset and model for 3d lifting of 2d objects. In ECCV, 2024. 10
[57] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. PointPillars: Fast encoders for object detection from point clouds. In CVPR, 2019. 22
[58] Dongxu Li, Junnan Li, and Steven Hoi. BLIP-Diffusinn: Pre-trained subject representation for controllable text-to-image generation and editing. NeurIPS, 2023. 1,3
[59] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang. Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. GLIGEN: Open-set grounded text-to-image generation. In CVPR, 2023.
2,
[60] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung-Yi Yi Lin. Magic3D: High-resolution text-to-3d content creation. CVPR, 2023.
[61] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov. Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object. In ICCV, 2023. 3, 6, 9, 18, 19
[62] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. SyneDremmer: Generating ICLR, 2024. 3, 9
images from single-view image. In
[63] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. NeurIPS, 2020. 3,4
[64] Grace Luo, Trevor Darrell, Oliver Wang, Dan Goldman, and Aleksander Holynski. Readout Guidance: Learning control from diffusion features. In CVPR, 2024.
[65] Jian Ma, Junhao Liang. Chen Chen, and Haonan Lu. Subject-Diffusion Open domain personalized text-to-image generation without test-time fine-tuning. In ACM SIGGRAPH Conference Proceedings, 2024. 3, 4
[66] Luke Melas-Kyrrazzi, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. RealFusion: 360deg reconstruction of any object from single image. In CVPR, 2023.
[67] Oscar Michel, Anand Bhattad, Eli VanderBilt, Ranjay Krishna, Aniruddha Kembhavi, and Tanmay Gupta. OBJECT 3DIT: Language-guider 3d-aware image editing. NeurIPS, 2023. 1,
3, 7, 18, 19, 20
[68] B Mildenhall, PP Srinivasan, M Tancik, JT Barron, Ramamoorthi, and Ng. NeRF: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020. 4
[69] Chong Mou, Xintao Wang, Jiechong Song. Ying Shan, and Jian Zhang. DragonDiffusion: Enabling drag-style manipulation on diffusion models. In ICLR, 2024.7 2
[70] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic
models. ICML, 2021. 1,2
[71] Michael Niemeyer and Andreas Geiger. GIRAFFE: Representing scenes as compositional
generative neural feature fields. In CVPR, 2021. 3,5
[72] Karran Pandey, Paul Guerrero, Matheus Gadelha, Yannick Hold-Geoffiee Karan Singh, and Niloy Mitra. Diffusion Handles: Enabling 3d edits for diffusion models by lifting activations 3d. In CVPR, 2024. 3, 18
[73] Dario Pavllo, Graham Spinks, Thomas Hofmann, Marie- Francine Moens, and Aurelien Lucchi. Convolutional generation of textured 3d meshes. NeurIPS, 2020. 3
14

---

[74] Dario Pavllo, Jonas Kohler, Thomas Hofmann. and Aurelien Lucchi. Learning generative models of textured 3d meshes from real-world images. In ICCV, 2021. 3
[75] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. DreamFusion: Text-to-3d using 2d diffusion. In ICLR, 2023. 3
[76] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021. 9
[77] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06/255, 2022.
[78] Helge Rhodin, Mathieu Salzmann, and Pascal Fua. Unsupervised geometry-aware representa: tion for 3d human pose estimation. In ECCV, 2018. 5
[79] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. CVPR, 2022. 2, 3, 5, 6, 19
[80] Olaf Romneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015. 22
[81] Natamiel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aber- man. DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation.
In CVPR, 2023. 1, 3, 19
[82] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022. 1,2
[83] Mehdi SM Sajaadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetic, Mario Lucic, Leonidas J Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. NeurIPS, 2022.
[84] Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Herrmann, Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, et al. ZeroNVS: Zero-shot 360-degree view synthesis from a single real image. In CVPR, 2024. 3
[85] Christoph Schuhmann, Romain Beaumont, Richard Veneu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. LAION-5B: An open large-scale dataset for training next generation image-text models. NeurIPS Datasets and Benchmarks Track, 2022.
[86] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. GRAF: Generative radiance fields for 3d-aware image synthesis. NeurIPS, 2020.
[87] Maximilian Seitzer, Sjoerd van Steenkiste, Thomas Kipf, Klaus Greff, and Mehdi SM Sajjadi. DyST: Towards dynamic neural scene representations on real-world videos. In ICLR, 2024. 3
[88] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. InstantBooth: Personalized text-to-image generation without test-time finetuning. arXiv preprint arXiv:2304.03411, 2023.
[89] Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. MVDream: Multi-view diffusion for 3d generation. In ICLR, 2024. 3, 19
[90] Yujun Shi, Chuhui Xue, Jiachun Pan, Wenqing Zhang, Vincent YF Tan, and Song Bai. DragDiffusion: Harnessing diffusion models for interactive paset-based image editing. In
CVPR, 2024. 2
[91] Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns to compose. In ICLR, 2021.
[92] Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. NeurIPS, 2022.
15

---

[93] Gautam Singh, Yeongbin Kim, and Sungjin Ahn. Neural systematic binder. In ICLR, 2023. 3
[94] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In ICML, 2015. 2, 22
[95] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2020. 7, 19
[96] Roger Wolcott Sperry. Neural basis of the spontaneous optokinetic response produced by visual inversion. Journal of Comparative and Physiological Psychology, 1950.
[97] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, 2020. 6, 18
[98] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich, Score Jacobian Chaining: Lifting pretrained 2d diffusion models for 3d generation. In CVPR, 2023.
3
[99] Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guoqiang Wei, Liping Yuan, and Hang Li. Boximator: Generating rich and controllable motions for video synthesis. arXiv preprint arXiv:2402.01556, 2024.
[100] Qian Wang, Yiqun Wang, Michael Birsak, and Peter Wonka. BlobGAN-3D: A spatially- disentangled 3d-aware generative model for indoor scenes. arXiv preprini vrXiv:2303.14700 2023. 3
[101] Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen. InstantlD: Zero-shot identity-preserving generation in seconds. arXiv preprint arXiv:2401.005119 2024.
[102] Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. FCOS3D: Fully convolutional one-stage monocular 3d object detection. In ICCV Workshop, 2021.
[103] Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. In- stanceDiffusion: Instance-level control for image generation. In CVPR, 2024. 1, 2,
[104] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro P Simoncelli. Image Quality Assess- ment: from error visibility to structural similarity. TIP, 2004. 7, 19
[105] Daniel Watson, William Chan, Ricardo Martin Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. Novel view synthesis with diffusion models. In ICLR, 2023. 3
[106] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. ELITE: Encoding visual concepts into textual embeddings for customized text-to-image generation. In ICCV, 2023. 3
[107] Olivia Wiles, Koepke, and Andrew Zisserman. X2Face: A network for controlling face generation using images, audio, and pose codes. In ECCV, 2018. 2
[108] Ziyi Wu, Jingyu Hu, Wuyue Lu, Igor Gilitschenski, and Animesh Garg. SlotDiffusion: Object-centric generative modeling with diffusion models. NeurIPS, 2023. 3
[109] Guangxuan Xiao, Tianwei Yin, William T Freeman, Frédo Durand, and Song Han. FastCom- poser: Tuming-frce ffree multi-subject image generation with localized attention. arXiv preprint 0rXiv:2305.10433 2023. 3,4
[110] Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. BoxDiff: Text-to-image synthesis with training-free box-constrained diffusion. In ICCV, 2023. 2
[111] Yinghao Xu, Menglei Chai, Zifan Shi, Sida Peng, Ivan Skorokhodov, Aliaksandr Siarohin, Ceyuan Yang, Yujun Shen, Hsin- Hsin-Ying Lee, Bolei Zhou, et al. DiscoScene: Spatially disentan- gled generative radiance fields for controllable 3d-aware scene synthesis. CVPR, 2023. 5, 6, 18
16

---

[112] Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, HtaWeei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. MagicAnimate: Temporally consistent human image animation using diffusion model. In CVPR, 2024. 2
[113] Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth Anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024. 5, 10
[114] Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, al. ReCo: Region-controlled text-to-image generation. In CVPR, 2023. 2
[115] Jiraphon Yenphraphai, Xichen Pan, Sainan Liu, Daniele Panozzo, and Saining Xie. Image Sculpting: Precise object editing with 3d geometry control. In CVPR, 2024.
[116] Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In ICCV, 2023. 1,2
[117] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as a perceptual metric. In CVPR, 2018. 7. 19
[118] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity rotation representations neural networks. In CVPR, 2019.
17

---

A Detailed Experimental Setup
In this section, we provide full details on the datasets, baselines, evaluation settings, and the training and inference implementation of our model.
A.1 Datasets
OBJeet [67] consists of Blender [20] rendered scenes where multiple (up four) objects are placed on a flat textured ground. The objects come from a 59k subset of Objaverse dataset [21]. A total of 18 background maps are used to provide environmental lighting. Four types of object-level editing are provided- - translation, rotation, removal, and insertion, each with 100k simulated data. Notably, only one object is edited in each data, and the translation and rotation is always on the ground (i.e., perpendicular to the gravity vector). For a fair comparison with the 3DIT baseline [67], we use the 2D rotated bounding box to represent object pose, which is composed of two corners of the 2D bounding box and the rotation angle over the gravity axis. This dataset is under the Open Data Commons Attribution License (ODC-By)².
MOVi-E [36] contains 10k videos simulated using Kubric [36]. Each scene contains 11 to 23 real-world objects from the Google Scanned Objects (GSO) repository [23]. At the start of each video, several objects are thrown to the ground to collide with other objects. Similar to OBJect, environmental lighting provided by a randomly sampled environment map image. The camera follows small linear motion. The full data generation pipeline is under the Apache 2.0 license".
Objectron [1] contains 15k object-centric video clips of common daily objects covering nine cate- gories. Each video comes with object pose tracking throughout the video, and we process it to obtain 3D bounding boxes. Since this dataset does not provide 2D bounding box labels, we project the eight corners of 3D boxes to the image, and take the tight bounding box of projected points as 2D boxes. Objectron is licensed under the Computational Use of Data Agreement 1.0 (C-UDA-1.0)).
Waymo Open [97]. The Waymo Open Dataset consists of 1K videos of self-driving scenes recorded by car mounted cameras. Following prior works [24, 111]. take the front view camera and bounding box annotations of cars. Notably, the 3D bounding boxes only have a heading angle (rotation along the yaw-axis) annotation, and thus we treat the other two rotation angles as O. Besides, the provided 2D boxes and 3D boxes are not aligned. preventing us from doing paired frame training. We instead project 3D boxes to get associated 2D boxes similar to Objectron. Waymo Open is licensed under the Waymo Dataset License Agreement for Non-Commercit Use (August 2019)².
Data Pre-processing. For all datasets, we resize the images to 256 x 256 regardless of the original aspect ratio. On Objectron, we discard all videos from the bike class as contains many blurry frames and inaccurate 3D bounding box annotations. On Waymo, we remove all cars whose 2D bounding box is smaller than 1 %% of the image area. We do not apply data augmentation except on Waymo Open, where we apply random horizontal flip and random resize crop following [24].
A.2 Baselines
3DIT [67] fine-tunes Zero-I-to-3 [61] to support scene-level 3D object edits. We generate the editing instruction from the target object pose, such as the translation coordinate and the rotation angle. However, this method does not support large viewpoint changes as it does not encode camera poses. We take their official code and pre-trained weights of the Multitask variant. 3DIT is under the CreativeML Open RAIL-M l
This baseline is inspired by [67, [2), where we chain multiple models together to achieve 3D-aware object editing. An editing step usually contains three steps: (i) crop out the object of interest and inpaint its region with backgrounds, (ii) synthesize the object under the new pose, and (iii) place the object to the new location. For (i), we apply SAM [55] segment the object using 2D bounding box prompt, and inpaint the original object region with Stable Diffusion v2 inpainting
Chained.
'nttpa://github. hoa.:oogggeeeeaarre/aaaatcccaaaa
'nttpa://waymo. www//pen/term....mmm htappp///aaamo.ccc///ee//aeemma hhtp:///ooooooooooccccacaaaaaaaa
hhttps: httpp://github... htt.ta/.woo/allllaaal/ecaaaaaaaa
18

---

model [79]. For (ii), we run Zero-I-to- [61] to re-pose the object according to the target 3D bounding box. For (iii). following [61], we first get the alpha mask of the re-posed object using an online tool', and insert it to the new position via alpha blending. It is worth noting that Zero- 1-t0-3 does not support camera rotation over the roll axis. For all models, we take their official code and pre-trained weights. SAM is under the Apache 2.0 license Stable Diffusion v2 inpainting model is under the CreativeML Open RAIL++MM License'. Zero-I-to is under the MIT license 10, The online alpha mask extraction tool is under the Apache 2.0 license"!
A.3 Evaluation Settings
We report PSNR, SSIM [104], LPIPS [117], and FID [42] to measure the accuracy of the edited image. We compute metrics both on the entire image, and within the 2D bounding box of edited objects. For box-level metrics, we follow [67] to crop out each object and directly run the metric without resizing. We also evaluate the identity preservation of objects using the DINO [13] feature similarity proposed in [81], which runs a DINO self-supervised pre-trained ViT on cropped object patches to extract features and compute the cosine similarity between predicted and ground-truth image.
A.4 Our Implementation Details
Model architecture. We take Stable Diffusion (SD) v2.1 [79] as our image generator except for experiments on the OBJect dataset, where we use SD v1.5 for fair comparison with baselines. Similar to prior works [49, 891, we also observe clearly better performance using SD v2.1 compared to v1.3. However, we note that our Neural Assets framework generalizes to any image generator that conditions supervised pre-trained ViT-B/8 [13], which outputs a feature map of shape 28 x 28 given a 256 x 256 image. For each object, we apply RolAlign [38] to extract a 2 2 small feature map and flatten it, i.e., the appearance token A, has a sequence length of K = 4. Since the conditioning token dimension of pre-trained SD v2.1 is 1024, to D' = 1024, and linearly project the concatenated appearance and pose token back to 1024. For background modeling, we mask all pixels within object boxes by setting them and extract features with the same DINO encoder. Instead, the pose token is obtained by applying a different two-layer MLP the relative camera pose between the source and the target image.
sequence of tokens. We implement the visual encoder Enc with DINO self-
use lwy-rayer MLP to transform the 3D bounding boxes input
fixed value of 0.5,
Training. We implement the entire Neural Assets framework in JAX [10] using the Flax [40] neural network library. We train all model components jointly using the Adam optimizer [53] with a batch size of 1536 on 256 TPUv5 chips (16GB memory each). We use a peak learning rate of 5 x 10-5 for the image generator and the visual encoder, and larger learning rate of x 10-3 3 for remaining
layers (MLPs and linear projection layers). Both learning rates are linearly warmed up in the first 1,000 steps and stay constant. A gradient clipping of 1.0 is applied to stabilize training. We found that the model overfits more severely on real-world data with complex backgrounds compared to synthetic datasets. Therefore, and 50k steps on Objectron and Waymo Open which takes 6 hours. In order to apply classifier free guidance (CFG) [43], we randomly drop the appearance and pose token (i.e., setting them zeros) with a probability of 10%. CFG improves the performance and also alleviates overfitting in training.
train the model for 200k steps OBJect and MOVi-E which takes 24 hours,
Inference. We run the DDIM sampler [95] for 50 steps to generate images. We found the model works well with CFG scale between 1.5 and 4, and thus choose to use 2.0 in all the experiments.
19

---

Table 1: Single-object editing results OBJect. We evaluate the Translation, Rotation, and Removal tasks. We follow 3DIT [67] to evaluate on both seen and unseen object subsets, and compute metrics inside the edited object's bounding box. Our results are averaged over 3 random seeds.
Seen Objects
Unseen Objects
Model
PSNR
SSIM
LPIPS
FID +
PSNR t
SSIM 1
LPIPS &
FID +
Task: Translation
Chained
13.70
0.309
0.485
0.94
14.13
0.326
0.467
0.97
3DI
15.21
0.300
0.472
0.24
15.20
0.292
0.477
0.25
Ours
20.58
0.039
0.273
0.15
20.13-0811
0.429
0.274
0.16
Task: Rotation
Chained 3DIT
13.18
0.269
0,540
1.00
12.85
0.270
0,538
1.69
16.86
0.382
0.429
0.25
16.28
0.366
0,447
0.24
Ours
18.52
0.391 100022
0.304
2.14
18.39 20.57
0.377
0.365 :00000
0.15
Task: Removal
Chained 12.49
0.383
0,465
0.80
12.12
0.379
0,459
1.05
3DIT
24.98
0.585
0.249
0.24
24.66
0.568
0,260
0.24
Ours
28.86
0.616:0005
0.167 1:0000
0.14
28.44 0091
0.613 10016
0.169 000.000
0.15 #0.005
Table 2: Multi-object editing results on MOVi-E, Objectron, and Waymo Open. We compute metrics on the entire image and inside the object bounding boxes. Ours are averaged over 3 seeds.
Image-Level
Object-Level
Model
PSNR
SSIM
LPIPS +
FID +
PSNR t
SSIM 1
LPIPS 4 DINO 1
Dataset: MOVi-E
Chained 3DIT
14.46
0.409
0.481
4,46
15.06
0.303
0.436
0.554
14.33
0.385
0.671
5.89
12.41
0.214
0.663
0.336
Ours
22.03
0.594
0.277
2.20
20.05
0.547
0.289 2000s
0.738
Dataset:
Objectron
Chained 3DIT
11.23
0.262
0.586
2.03
11.24
0.171
0.415
0.555
11.69
0.281
0.559
1.99
11.30
0.150
0.444
0.547
Ours
14.83
0.348.100
0.446.100
0.55
16.41 :0.33 0.477.1000 :00009
0.233 0:0008
0.790
Dataset:
Waymo Open
Chained
18.28
0.501
0.454
2.88
16,20
0.310
0.383
0.596
3DIT
17.32
0.421
0.474
3.32
14.41
0.174
0.537
0.449
Ours
18.71
0.494
0.404+-000
1.64
17.67 00.30
0.343+tom
0.348+:0.r
0.653
B Additional Experimental Results
B.1 Full Benchmark Results
We present full quantitative results on OBJect in Tab. 1, and on MOVi-E, Objectron, and Waymo Open in Tab. 2. Compared to the main paper, we report additional FID metrics and results on the unseen object subset for OBJect, while for the other three datasets, we report additional FID and DINO feature similarity metrics, plus results computed over the entire image (Image-Level) Overall, we observe similar trends as in the main paper, where our Neural Assets model significantly outperforms baselines across all datasets. In Fig. 11, we show additional qualitative comparisons.
B.2 Full Ablation Results
We present all quantitative results of our ablation studies on Objectron (Sec. 4.4) in Tab. 3, Tab. 4, and Tab. 5, We observe similar trends all metrics at both image- and object-level.
20

---

Source Image
Chained
3DIT
Ours
Target Image
MOVi-E
Objectron
Waymo Open
Figure More qualitative results on MOVi-E, Objectron, and Waymo Open.
B.3
Controllable Scene Generation
Fig. and Fig. 13, we show controllable scene generation results on Objectron. Objectron videos only have global camera movement, while the objects are static. Still, our Neural Assets model learns disentangled foreground and background representations. can be seen from the results, we can rotate the foreground objects while keeping the background fixed, or swap background between
21

---

Table 3: Ablation of image encoders Objectron. FT-DINO stands for fine-tuning DINO VIT.
Image-Level
Object-Level
Encoder
PSNR
SSIM 1
LPIPS 4
FID J
PSNR 1
SSIM
LPIPS
DINO
CLIP
12.95
0.309
0.532
0.66
14.13
0.339
0.333
0.709
MAE
13.64
0.317
0.501
0.59
14.93
0.369
0.296
0.735
DINO
13.75
0.325
0.498
0.57
15.03
0.388
0.296
0.747
FT-FINO
14.83
0.348
0.446
0.55
16.41
0.477
0.233
0.790
Table 4: Ablation of background modeling on Objectron. No-BG means doing background
modeling at all, while No-Pose stands for not using the relative camera pose between two frames.
Image-Level
Object-Levvl
Background PSNR f
SSIM f
LPIPS
FID
PSNR
SSIM t
LPIPS 4 DINO
No-BG
12.98
0.308
0.513
1.18
14.49
0.394
0.297
0.712
No-Pose
13.71
0.326
0.496
0.72
15.39
0.423
0.273
0.751
Ours
14.83
0.348
0.446
0.55
16.41
0.477
0.233
0.790
Table Ablation of training data on Objectron. Single and Paired refer to training on one image or source-target pairs. No-PE means removing the positional encoding in the ViT image encoder.
Image-Level
Object-Level
Training Data
PSNR 1
SSIM f
LPIPS
FID 4
PSNR 1
SSIM f
LPIPS 4 DINO
Single Single (No-PE)
12.63
0.298
0.544
1.07
13.41
0.259
0.381
0.651
13.74
0.323
0.503
1.01
14.51
0.315
0.337
0.683
Paired
14.83
0.348
0.446
0.55
16.41
0.477
0.233
0.790
scenes.
Importantly, our model inpaints the masked background regions not occupied by the novel object, and renders realistic shadows around the object, which is far beyond simple pixel copying.
B.4 Ablation on 3D Pose Representations
In Fig. 14, we visualize the object pose representation we use. Given a 3D bounding box of an object, project its four corners to the image space, and concatenate their 2D coordinates and depth values to obtain a 12-D pose vector. The 2D projected points resemble a local coordinate frame for the object, specifying its position, rotation, and scale. On the other hand, the depth is useful for determining the occlusion of objects.
There are alternative ways to represent the object pose, e.g., the coordinate of the 3D box center C with its size and rotation which is commonly used in 3D object detection [57]. These representations achieve similar results on MOVi-E and Objectron. However, their learned rotation controllability is significantly worse than our representation on Waymo. This is because most the cars on Waymo not rotated (turn left / right), leading to very few training data object rotation. If we directly input the rotation angle to the model, it tends to ignore it. contrast, due to prospective projection, the projected local coordinate frame of unrotated cars still look "rotated" when they not strictly in front of the ego vehicle. This provides much more training signal to learn the rotation of objects.
C Background on Stable Diffusion
Diffusion model [44, 94] is class of generative models that learns to generate samples by iteratively denoising from a standard Gaussian distribution. It consists of denoiser ca, usually implemented as a U-Net [80], which predicts the noise e added to the data Instead of denoising raw pixels, Stable Diffusion introduces a VAE [54] tokenizer to map images to low-dimensional latent code and applies the denoiser on it. In addition, the denoiser is conditioned on text and thus supports text-to-image generation. In this work, we simply replace the text embeddings with Neural Assets o; and fine-tune the model to support appearance and pose control of 3D objects.
22

---

Original Image
Translate
Rotate
Figure 12: Object translation and rotation results on Objectron. Although there is only camera movement this dataset (i.e., objects never move), the model still learns to disentangle the object pose and the camera pose. shown the object rotation results, the background stays fixed. See our project page for videos and additional object rescaling results.
Replace Background
Figure 13: Transfer backgrounds between scenes by replacing the background token on Objectron.
Note how the global camera viewpoint is adjusted fit the foreground object. In addition, the generator is able to synthetic lighting effects such as shadows on the surfaces.
23

---

P3
C
2
P1
Po
(a) Object pose as 3D bounding box
Sample pose representation
(c) Sample pose representation
Figure 14: Illustration of our object pose representation (a) and two examples (b). We project four corners Po, PI, P2, P3 of a 3D bounding box to the 2D image plane and concatenate them to obtain the pose token. The projected four corners form local coordinate system of the object.
D Broader Impacts
Controllable visual generation is important task in computer vision. Neural Assets equip generative models with an intuitive interface to control their behaviors, which enables more interpretable algorithms and may potentially benefit other fields such computer graphics and robotics. We believe this work will benefit the whole research community and the society.
Potential negative societal impacts. Since we fine-tune large-scale pre-trained generative models in our pipeline, we inherit limitations of these base models, such as dataset selection bias. Such bias might be problematic when human subjects are involved, though our current approach is only capable of rigid object control and does not consider humans as an "asset" yet. Further study on how such bias affects model performance is required for mitigating negative societal impacts that could arise from this work [35].
E Funding Disclosure
This work carried out at Google. Igor Gilitschenski contributed the project in advisory capacity.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We propose Neural Assets, new form of representations that supports multi- object controllable 3D scene generation. We have verified it with extensive experiments in Sec. 4 and Appendix B.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We discuss the limitations of this work Sec. 5.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
25

---

Answer:
[NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Ye]]
Justification: We provide all implementation details about dataset processing, model archi- tecture, and training and inference in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
26

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide the details about our implementations in Appendix A. and Appendix A.4. See our project page, neural-assed gi thtb..... io, for further details.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/
guw/guiiees//ee Cee////ooeubiiicc ssi aasionPoiccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all implementation details about dataset processing, model archi- tecture, evaluation metries, and training and inference in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
All main results averaged over 3 random seeds and report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
27

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide all required information in Appendix A.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : /ttrrr/eerrpp.eeeee//ee cc http/pulli//ttiisiuuuiiiii
Answer: [Yes]
Justification:
Our research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We discuss potential positive and negative society impacts Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
28

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We discuss safeguards on our image generator in Appendix D.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We provide the license information in Appendix A.l and Appendix A.2.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
29

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new asset is introduced in the submission.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "3qdUdLiYC1", "reviewer": "Reviewer_h1dc", "summary": "This paper addresses the task of multi-object pose and scale control in image generation and editing using diffusion models. The authors introduce an object-centric representation with a disentangled pose and appearance called Neural Asset. The neural asset for each object in an image is estimated using pose and appearance encoders. The authors exploit the naturally occurring pose and appearance variations in training video datasets to train such encoders. To generate an image based on neural assets, the authors fine-tune an image diffusion model conditioned on the sequence of neural assets as its condition. The authors evaluate and compare their method with existing baselines, showcasing the ability of their method for pose and scale control, object removal, and background change.", "questions": "See the weaknesses", "limitations": "sufficiently discussed", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written and easy to follow\n- The proposed method is reasonable and of sufficient novelty\n- The provided results show the efficacy of the proposed approach for modeling multiple objects in images and controlling their pose and scale\n- The experiments are sufficient and contain ablation study on different design choices", "weaknesses": "- Based on the provided results, the background has small changes when an object in the image is edited even in the areas far from the edited object\n- The proposed method is mainly limited to the domain of the training dataset as opposed to the recent training-free diffusion-based image editing methods", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed feedback and encouraging comments.\n\n**1. Small background changes in areas far from the edited objects.**\n\nA: We encode object appearance tokens by applying RoIAlign on the image feature map. Even if we use the paired frame training strategy for feature disentanglement, there is still background information leak to the foreground object representations. As a result, editing an object might lead to small changes in an irrelevant background region.\n\nThis is not specific to our work: small background variation is a common issue in 3D-aware image editing methods (e.g., see results on the website of Diffusion Handles [69] and LooseControl [6]), which might be resolved with more powerful base models and larger-scale training data. We thus leave it for future work.\n\n\n**2. Limited application domain of Neural Assets compared to training-free methods.**\n\nA: Please see our general response about the discussion of using datasets with 3D bounding box annotations. With recent general-purpose 3D object pose estimators [1], we can build a large-scale training dataset with more diverse scenes to learn Neural Assets of general objects.\n\n[1] Krishnan, Akshay, et al. \"OmniNOCS: A unified NOCS dataset and model for 3D lifting of 2D objects.\" ECCV. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mZ9gGDCDkS", "reviewer": "Reviewer_bNz4", "summary": "The paper introduces an object-centric representation for multi-object 3D pose control in image diffusion models. Instead of text token sequences, it utilizes Neural Assets, which are per-object representations learned by pooling visual features from reference images and reconstructing objects in target frames. This disentangles appearance and pose features by fine-tuning a pre-trained text-to-image diffusion model while conditioning on target frame poses, enabling fine-grained 3D pose and placement control in scenes. The results are demonstrated on both synthetic and real-world datasets in multi-object editing tasks.", "questions": "See above. Overall, I think the paper offers a new perspective for object-centric learning, especially under the image generation settings. I still have some concerns and will consider raising my rating if the rebuttal resolves them.", "limitations": "Yes.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Object-centric representation: the proposed approach introduces a novel object representation to tackle conditional image generation, which disentangles 3D appearance and 3D pose. This representation provides a new solution to tackle similar tasks in controllable 2D image / 3D scene editing tasks.   \n2. Experimental Results: The experiments demonstrate the effectiveness of the proposed Neural Assets in achieving fine-grained control over individual object appearances and poses within a scene. Its application is shown in both synthetic and real-world scenes, supporting tasks like background swapping and object transfers across different scenes.  \n3. The paper is overall well written, providing good motivations and comparison with related work. Some concerns in the method section could be further clarified (see below).", "weaknesses": "I value the performance the proposed model presented in manipulating objects while keeping the rest unchanged. I spent quite some time trying to comprehend how the current design leads to its performance, and still have the following questions, which might be helpful if they are discussed in the paper:  \n[-] The paper utilizes projection and depth of the absolute object poses in the pose embedding. This is kind of confusing as the it requires the model to encode how the object should look like in the canonical space. If this is the case, the model should be able to perform per-object reconstruction with the object-centric representation through images under different poses and even maybe object pose estimation. A common practice is to use the relative pose change, as the authors discussed in the background modeling.  \n[-] The authors leverage SD as base model, but one significant difference is that the proposed model put the learning bottleneck to the conditional module, compared to Zero-1-to-3 or ControlNet. All the useful features are embedded in the conditions now, including how the images should look like. In original SD, the conditional module is relatively light-weight. I'm wondering if there are limitations for the current design to capture the complex conditions and recover the input images.  \n[-] One related concern is that from the results, it seems the reconstruction results are also decent. I'm wondering what role is the randomness of the noise play in the current model. Will the model output different images if the seed changes?  \n[-] How should the model comprehend the objects in the target images that do not exist in the src images, especially during training?\n\nSome other concerns:  \n[-] The model may lack a total understanding of the scene, especially the interactions between objects, e.g.,  when one object is supported by another or when the 3D bounding boxes of two objects collide with each other. How will the model react in such scenarios?\n[-] The proposed model faces limitations in fully manipulating 3D object and their relationships as naturally as humans do. The requirement of 2D/3D bounding box pose challenges to the detection models and the manipulation as projection points is not intuitive as humans who commonly use language.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments. We are glad to see the positive assessment of our paper, and will include the below discussions in the final version of the paper.\n\n\n**1. The use of absolute object pose for pose tokens.**\n\nA: This is a great question. As discussed in Sec. 3.1, a 3D asset in traditional graphics pipelines is often represented by its canonical 3D shape and its pose. Therefore, it is actually our goal to encode the object in its canonical space in the appearance token of a Neural Asset. As shown in the experiments, our model is indeed able to synthesize an object under different poses.\n\nIn our preliminary experiments, we tried using relative pose changes to encode object pose tokens. However, it shows worse results compared to absolute poses. One potential cause is the use of the DINO encoder. Due to its pre-training strategy (self-distillation), DINO is encouraged to extract visual features that are invariant to image transformations (e.g., resize, translate). This objective aligns with our goal of extracting object appearance in its canonical space regardless of the observed pose.\n\n\n**2. Using only the conditional module (cross-attention) for Neural Assets conditioning.**\n\nA: The main reason we use cross-attention is that it allows us to condition the generator on individual Neural Assets which are vector-based representations, facilitating scene decomposition. It is unclear how we can apply dense conditioning such as concatenation (Zero-1-to-3) and addition (ControlNet) while still keeping the object-centric property of our method.\n\nWhile we do not use pixel-aligned conditioning, full fine-tuning of the visual encoder and diffusion model can greatly improve the generation result (see ablations in Fig. 9 (a)). We agree that there are still artifacts in local visual details. We tried increasing the RoIAlign size of each object ($K$ for the appearance tokens) but it did not help much. Using a stronger base model (e.g. Stable Diffusion XL) or image encoder (e.g. DINO v2) might solve this problem, which we leave for future works.\n\n\n**3. Effect of random seeds in the generation results.**\n\nA: As we condition the generator on object and background representations, we should expect small variations in the generated images. Yet, since novel-view synthesis is an ill-defined problem, the model needs to hallucinate new content in regions unobserved in the source image. Fig. 1 of the uploaded PDF shows the generation results from our model under three random seeds. The global scene layout and object geometry are identical among different seeds. In addition, our model synthesizes diverse but plausible variations in the local details of objects and backgrounds.\n\n\n**4. New objects in the target image that do not exist in the source image.**\n\nA: New objects will only have a valid pose token, while the appearance token is set to zero. The model is encouraged to hallucinate it. In fact, in order to apply classifier-free guidance (CFG), we intentionally set the appearance tokens to zero with a probability of 10% during training. Thus, the new objects will be treated similarly by the model without harming the performance.\n\n\n**5. The ability to model object interactions in the scene.**\n\nA: We leverage cross-attention to inject Neural Assets to the latent features. It is true that object tokens do not interact with each other directly in cross-attention. However, the following self-attention should learn the object interactions and generate an image with coherent content. Our hypothesis is that object interaction does not need to be architecturally hardcoded into the conditioning branch.\n\nAs shown in Fig. 6 and Fig. 7 of the main paper, our model handles object occlusions correctly. As shown in Fig. 8 of the main paper, our model adapts objects to new backgrounds, e.g., the car lights are turned on at night. These results prove that our model understands object-object and object-background interactions.\n\nRegarding object collision, the Waymo results on our website (see link in the paper or the supplementary material) show a few cases (Rotation on row 1 & 4). The colliding objects blend into each other, while the other parts of the image look normal. We argue that as a conditional generator, our model is tasked to follow the input object poses. If the 3D bounding boxes are physically wrong such as collision, the model will just generate implausible images instead of correcting it.\n\n\n**6. Limitation of using 3D bounding boxes for control & Extension to language control.**\n\nA: Please see our general response about the discussion of using datasets with 3D bounding box annotations. With recent general-purpose 3D object pose estimators [1], we can build a large-scale training dataset with more diverse scenes to learn Neural Assets of general objects.\n\n[1] Krishnan, Akshay, et al. \"OmniNOCS: A unified NOCS dataset and model for 3D lifting of 2D objects.\" ECCV. 2024.\n\nWe agree that language-based control is more intuitive. But it is harder to achieve precise spatial control of objects compared to using bounding boxes. We thus leave this as an interesting future direction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DvTudhvuhW", "reviewer": "Reviewer_W8vg", "summary": "This paper considers controlling the 3d poses of different objects in an image generated by a diffusion model. By conditioning the diffusion model on a sequence of per-object appearance and pose tokens instead of text tokens it is possible to finely control the object poses, and furthermore to e.g. transfer objects between scenes. The diffusion model is trained using videos with 3d bounding box annotations and correspondences between different frames, and it is shown experimentally to generate realistic images and to outperform related methods.", "questions": "Please see weaknesses\n\nCan the authors clarify if code and models will be released? It is mentioned that the authors “will consider” releasing code upon acceptance (line 844). Will any trained models be released? Will code to replicate the training be released?", "limitations": "This is adequately addressed", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The approach to replace text tokens with a sequence of per-object appearance and pose tokens is very clean and elegant.\n- The results are impressive and the method achieves state-of-the-art results for the problem. The closest related work is 3DIT and the authors compare to and outperform that method on the tested datasets.\n- While previous work mainly relied on synthetic data (e.g. 3DIT which introduces the OBJect dataset/task from objaverse assets) the authors propose to use 3d bounding boxes from real-world monocular videos as supervision, which has consistent appearance but different poses, which fits well for the task and can be used to generate training data.\n- Qualitatively, the generated images generally look very good, especially for Waymo Open. E.g. in fig. 8 there is realistic lighting dependent on the background, so the generated objects adapt well to the rest of the scene. Also, for the scaling examples one can see that the shadows around the object mostly scale accordingly.", "weaknesses": "- There seems to be some issues with the modelling of the background for the Objectron dataset, though this is not an issue with Waymo Open. Failure cases are e.g. that rotations and scaling of objects also rotate and scale the background around the object, even outside the object bounding boxes. Did the authors try e.g. any alternative ways of modelling the background to fix this?\n- This is not a major weakness, but there is no measure of 3d consistency, which is common in previous work, e.g.  zero-1-to-3 and follow up works often measure 3d reconstruction errors on e.g. Google Scanned Objects (GSO). It would be straight-forward to evaluate the 3d consistency of the proposed method on this dataset, and compare to e.g. zero-1-to-3. It should be mentioned though that this would not reflect the full strength of the proposed method since it can handle multiple objects and diverse backgrounds which is not captured when evaluating on that dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and valuable comments.\n\n**1. Background modeling issue on Objectron. Any attempts to fix it?**\n\nA: Objectron videos only have camera movement, while objects remain static throughout the video. Due to this data issue, the global camera motion and the local object motion are entangled, leading to background issues in the translation and rescaling results.\n\nWe have tried different foreground object pose representations (see Appendix B.4) and different camera parametrizations (relative vs absolute camera pose) which serve as background pose, but none of them can fix this issue.\n\nThis issue will likely disappear if we train a model on diverse data where both camera and object movement are observed, as shown in the Waymo results. Due to the lack of labeled datasets we have not pursued larger-scale training in this submission, but it should be possible in the near future thanks to recent general-purpose 3D object pose estimators. See our general response for detailed discussions on this.\n\n\n**2. Evaluation of 3D consistency on GSO.**\n\nA: Thanks for the suggestion. As the reviewer correctly pointed out, our work mainly focuses on multi-object generation, with results on real-world scenes with complex backgrounds. We compare with and outperform Zero-1-to-3 (the “Chained” baseline) in terms of novel-view synthesis. We believe the evaluations presented in the paper are sufficient to show the effectiveness of Neural Assets.\n\nOn the other hand, we do not optimize for 3D consistency – this would require joint denoising of multiple views like in recent SOTA works such as CAT3D [1]. In contrast, we generate object edits/views completely independently, resulting in unavoidable inconsistencies. It is an interesting future direction to explore scene-level 3D reconstruction once we have more diverse training data.\n\n[1] Gao, Ruiqi, et al. \"Cat3d: Create anything in 3d with multi-view diffusion models.\" arXiv. 2024.\n\n\n**3. Code release.**\n\nA: We are planning to release the inference code for Neural Assets which would allow users to attach their own training loop and dataset pipeline. It will come with an example data loading pipeline on MOVi-E and a training step implementation based on Diffusers Stable DIffusion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4DJUWJ1Pfu", "reviewer": "Reviewer_jimW", "summary": "This paper presents a novel method for 3D-aware editing of multi-object images. Given the original image and 2D bounding boxes to specify the objects to be edited, and the 3D pose as the target of the editing, the proposed method encodes the object appearance feature and their pose features, which are then taken as conditions to generate the edited images. The proposed approach disentangles the appearances and poses of the objects, and enables flexible and better control of the 3D pose of each object in the image.", "questions": "Line 244 claims that the proposed method is not pre-trained on multi-view rendering of 3D assets. It is actually trained on the different frames of videos, which is more appropriate for the multi-object image setting. This claim is true but I would not take it as an advantage of the proposed method.", "limitations": "This paper includes the discussions of limitations and broader impacts. However, it's better to present some failure cases of the proposed approach to better understand its performance.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The strengths of this paper include:\n(1)It proposes a novel approach to disentangle the appearance feature and pose feature of each object in the multi-object image.\n(2)The proposed approach outperforms the STOA 3D-aware image editing methods, especially in the multi-object setting.\n(3)It enables object editing and controllable scene generation applications.\n\nOverall, this paper presents a practical solution for the 3D-aware multi-object image editing and achieves the best performance. The network design is quite straightforward, but it indeed makes the best use of the pre-trained models and the large training set.", "weaknesses": "Basically, this paper provides an excellent 3D-aware image editing model for multi-object images benefiting from the pre-trained image generation model and the large training dataset. The idea of disentangling the appearance and pose features of objects is the key for multi-object image editing. However, since there have been existing works on this topic, such as BlobGAN-3D, it is essential to add an in-depth discussion to validate the superiority of the proposed network architecture.\n\n[1] BlobGAN-3D: A Spatially-Disentangled 3D-Aware Generative Model for Indoor Scenes", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and valuable comments.\n\n**1. Discussion about prior work BlobGAN-3D [1].**\n\nA: Thanks for bringing up this related work. We will include this paper and add the following discussions in the camera-ready version of the paper:\n\n> There have been prior works learning disentangled appearance and pose representations for multi-object image editing [1, 68, 105]. However, they are based on the Generative Adversarial Networks (GANs) framework, which are pre-trained on smaller datasets and have less expressive latent space compared to recent diffusion models. In contrast, we build upon large-scale pre-trained image diffusion models, enabling editing of complex real-world scenes.\n\n[1] Wang, Qian, et al. \"BlobGAN-3D: A spatially-disentangled 3D-aware generative model for indoor scenes.\" arXiv. 2023.\n\n\n**2. The claim of not pre-trained on multi-view images as an advantage.**\n\nA: Thanks for pointing this out. We will remove this statement in the final version of the paper.\n\n\n**3. Failure case of the proposed method.**\n\nA: Thanks for this great suggestion. We have already included failure case analysis in the website (see link in the paper or the supplementary material). We will add the following discussions in the final version of the paper:\n\n> Failure case analysis: (1) One main failure case of our model is symmetry ambiguity. As can be seen from the rotation results (Fig. 2 (a) in the uploaded PDF), the handle of the cup gets flipped when it rotates by 180 degree. (2) Another failure case that only happens on Objectron is the entanglement of global camera motion and local object movement (Fig. 2 (b) in the uploaded PDF). This is because Objectron videos only contain camera motion while objects always stay static. Both issues will likely be resolved if we train our model on larger-scale datasets with more diverse object and camera motion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for their helpful feedback and insightful comments.

We are glad that the reviewers find our paper “*well written*” (h1dc, bNz4), our Neural Assets framework “*elegant*” (W8vg) and “*novel*” (h1dc, bNz4, jimW). Also, our experiments are considered “*sufficient*” (h1dc), “*impressive*” (W8vg), and “*providing good motivations and comparison with related work*” (bNz4).

From the view of generative models, all reviewers acknowledge that our method supports multi-object pose control, enabling several editing tasks. Reviewer W8vg points out that “*the generated images generally look very good*” such as “*realistic lighting*”. Finally, reviewer jimW agrees that our model “*makes the best use of the pre-trained models and the large training set*”.

We have uploaded a PDF file that includes figures to address the reviewers’ feedback. Below we include a response to a general question raised by the reviewers. For other questions, please see our response to individual questions below each review. We will incorporate all our responses and additional results in the final version of the manuscript.

- **Limitation of using 3D bounding boxes for pose encoding** (reviewers h1dc, bNz4, and W8vg)

At the time of paper submission, there was no general-purpose 3D object pose estimator available. Therefore, our experiments are limited to datasets with 3D bounding box annotations. However, a recent work OmniNOCS [1] fills this gap by introducing a 3D pose estimator that works on both Waymo and Objectron (datasets we used in this work), and *diverse, in-the-wild Internet images* for a wide range of object classes. With recent advances in vision foundation models, we expect to see scalable 3D annotation pipelines similar to their 2D counterparts soon, which could be used to learn Neural Assets of more general objects in more diverse environments.

[1] Krishnan, Akshay, et al. "OmniNOCS: A unified NOCS dataset and model for 3D lifting of 2D objects." ECCV. 2024.

Meta-review: This paper proposes a method for controlled generation of compositional scenes using diffusion models with explicit pose control of the individual objects in the scene. To achieve this the authors propose a new Neural Assets representation, which disentangles the identity and the pose of the object. These representations are further appended into the text prompt provided to the diffusion model for conditioning. Additionally the authors propose a novel methodology to train their system using videos containing the objects in different poses in different frames, which further helps to disentangles the identity and expression representations. On several benchmark datasets the authors show significant improvement in performance over the state of the art and show for the first time 3D control for composes scenes instead of just single objects.

All reviewers appreciated the novelty and elegance of the proposed method and the high quality of the results achieved. They further appreciated the well-written nature of the paper. All reviewers further championed the acceptance of this work. This work represents a significant step forward in synthesizing 3D controlled scenes composes of individual objects. The AC concurs and recommends acceptance.
Decision: Accept (spotlight)