OBJECT 3DIT: Language-guided 3D-aware Image Editing
Authors: Oscar Michel
Anand Bhattad
Eli VanderBilt
Ranjay Krishna
Aniruddha Kembhavi
Tanmay Gupta
Venue: NeurIPS 2023 poster (2023)
Keywords: computer vision
image editing
generative modeling
diffusion models
3D
TL;DR: We present 3DIT, a model to edit individual objects within an image of a rich scene. This is trained on a new dataset, OBJect presented in this paper.

Abstract: Existing image editing tools, while powerful, typically disregard the underlying 3D geometry from which the image is projected. As a result, edits made using these tools may become detached from the geometry and lighting conditions that are at the foundation of the image formation process; such edits break the portrayal of a coherent 3D world. 3D-aware generative models are a promising solution, but currently only succeed on small datasets or at the level of a single object. In this work, we formulate the new task of language-guided 3D-aware editing, where objects in an image should be edited according to a language instruction while remaining consistent with the underlying 3D scene. To promote progress towards this goal, we release OBJect: a benchmark dataset of 400K editing examples created from procedurally generated 3D scenes. Each example consists of an input image, editing instruction in language, and the edited image. We also introduce 3DIT: single and multi-task models for four editing tasks. Our models show impressive abilities to understand the 3D composition of entire scenes, factoring in surrounding objects, surfaces, lighting conditions, shadows, and physically-plausible object configurations. Surprisingly, training on only synthetic scenes from \dataset, editing capabilities of 3DIT generalize to real-world images.

OBJECT 3DIT: Language-guided 3D-aware Image Editing
Oscar Michel
Anand Bhattad?
Eli VanderBilt' Tanmay Gupta'
Ranjay Krishna
Aniruddha Kembhavi'
Allen Institute for Artificial Intelligence. UUiierritty of Illinois Sebana-Champaige
University of Washington
Performance OBJect Dataset
Generalization CLEVR
Generalization to Real World
3DT (Ours)
Input
3DIT (Ours)
3DIT (Ours)
move beolcese
mebie thae Fee www.eeeeeee homten
Transi ation
mooe
rotuto tha cubbe
Rotation
the
rocate
nenrcvv baseballccg
Removal Insertion
NiITovs the sphaese
the
Figure 1: We present 3DIT, a model to edit individual objects in the context of a rich scene with language conditioning. 3DIT is able effectively edit objects while considering their scale and viewpoint, is able to add, remove and edit shadows to be consistent with the scene lighting and is able to account for object occlusions. Training generalizes to images in the CLEVR dataset as well as the real world.
benchmark OBJECT, 3DIT remarkably
Abstract
Existing image editing tools, while powerful, typically disregard the underlying 3D geometry from which the image is projected. As a result, edits made using these tools may become detached from the geometry and lighting conditions that the foundation of the image formation process. In this work, we formulate the new task of language-guides 3D-aware editing, where objects in image should be edited according to a language instruction in context of the underlying 3D scene. To promote progress towards this goal, release OBJECT: a dataset consisting of 400K editing examples created from procedurally generated 3D scenes. Each example consists of input image, editing instruction language, and the edited image. We also introduce 3DIT: single and multi-task models for four editing tasks. Our models show impressive abilities to understand the 3D composition
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

of entire scenes, factoring in surrounding objects, surfaces, lighting conditions, shadows, and peesically-plussbble object configurations. Surprisingly, training on only synthetic scenes from OBJECT, editing capabilities of 3DIT generalize to real-world images. More information can be found on the project page at https://prior allenai. httg/poreectssbbactteeeee
1 Introduction
In today's 's visually-oriented society, the art of image editing has become an indispensable necessity. With the proliferation of camera phones and influences from social media platforms, amateur photog- raphers want to transform ordinary snapshots into visual masterpieces. Unfortunately, the process of image editing is still in its infancy. Professional tools such as Photoshop allow pixel-Ievel edits that can adjust lighting, insert objects, remove clutter, and introduce new shadows; however, these tools, with their steep learning curves are often daunting for novices. With the hopes of pulling image editors out from the minutiae of painstaking pixel-level edits, generative models have been heralded as a promise for object-level edits [57 58 31 522.
Unfortunately, object-centric geometry of the original photograph-is out of reach for generative models [29 25 73 70
or rotating an object while preserving the 3D
Although recent strides can take a segmented object and rotate and translate it, they typically operate on objects in isolation and often disregard any scene and lighting context [47 [44 70]. Others require
multiple viewpoints to reconstruct an object in 3D (6 25 73]. There is need for models that can
edit objects from a single image while preserving the structure of 3D objects and re-render shadows for the edited scene with the original lighting conditions.
To enable 3D-aware editing of objects in an image, we introduce OBJECT, Objaverse Editing in ContexT,, a large-scale benchmark to train and evaluate language-conditioned models that edit objects in images. We develop OBJECT by combining Objaverse [14], recent 3D asset library, and Blender [12]. 3D rendering engine. OBJECT contains 400k editing examples derived from procedurally generated 3D scenes. Scenes consist of up to four objects, chosen from 59k unique objects, placed on a flat textured surface with an environment lighting map, a three-point lighting system that moves with the camera, and a directional light. As shown in Figure [] we support four types of object edits: (a) translation across the surface; (b) rotating around the axis orthogonal to the surface; (c) inserting new objects; and (d) removing existing ones. Our 3D rendering engine ensures that all edits are physically plausible and the generated images capture realistic changes in 3D geometry, illumination, and shading resulting from the underlying edit. For instance, rotation and translation require maintaining contact with the surface; inserting new objects requires identifying stable supported poses for new objects; and removing objects often requires rendering occluded objects. Each image contains language instruction describing one of the four edits and resulting ground truth edited image. Edited images are evaluated using quantitative metrics that capture realism and faithfulness to the ground truth.
We also introduce 3DIT (3D-aware Diffusion Image-editing with Text), a model which supports each of the four manipulation tasks with language conditioning. 3DIT is initialized with the Zero-1-to-3 47 diffusion model (which was trained to perform novel view synthesis) and finetuned on the OBJECT dataset for object-centric image editing. The resultant model has effectively been obtained using a three-stage learning curriculum, starting with massive stable diffusion's 's web-scale pre-training on image-text pairs, followed by Zero-1-to-3's s pre-training stage to enhance the model's s understanding of 3D objects, and finally with fine-tuning on OBJECT to enable object-centric edits.
On OBJECT'S test images, 3DIT outperforms baselines across all four tasks on metrics that capture the faithfulness of the scene edit. Given the known limitations of automatic quantitative metrics, we also provide human evaluation study. where 3DIT's 's outputs are preferred to the baselines over 70% of the time. Edits produced by 3DIT tend to preserve the original scene's structure and not just the edited object. 3DIT preserves the scale and viewpoint of objects, it removes and adds appropriate shadows wherever necessary, and even infills previously occluded portions of the image when the occluder is translated or removed. A multi-task variant of 3DIT performs well despite having to support all four transformations using single set of parameters. Finally, 3DIT generalizes surprisingly well to new image domains such as CLEVR, popular synthetic dataset for visual

---

reasoning, as well as real-world images (see Figure D. This highlights 3DIT's remarkable capability given that OBJECT is a synthetic, procedurally generated dataset.
2 Related work
Image editing with generative models: The goals of editing objects and semantic regions in images with language have been active for over decade [45]. Back then, productizable edits were limited
to simple changes like cropping, colorization and resizing to complex procedures such as object
Traditionally, these tasks were performed manually using tools like Adobe Photoshop. However, the origin of Generative Adversarial Networks (GANs) [23] revolutionized the field, propelling significant strides toward
removal, addition, and rearrangement [59] 30 18
46
16
85 22
53
automation. StyleGAN [36] 37 35] notably facilitated intricate modifications to the synthesized images, paving the way for sophisticated GAN-basee editing techniques with greater control and flexibility [66 84 II
60 TO 5 I 67 Since then, advancements in generative image architectures have been marked by the emergence of diffusion models [17] When coupled with the availability of large-scale image-text datasets [64]. these models have facilitated the generation of high- fiddeiidlitt, diverse scenes [5] [58 62
63 34). Concurrent with these developments, a new wave of image editing methodologies utilizing these large-scale diffusion models have been introduced [83 50 29
43 19 72). Despite these advancements, models lack the 3D awareness necessary for maintaining geometric and lighting consistency. Our dataset, OBJECT, aims bridge this gap by enhancing existing methods and serving to evaluate future methodologies.
3D image editing: A host of recent research, including StyleNeRF 25 ViewGen 3. EG3D [6].
SJC 733, DreamFusion [54) Z 47). and Make-It-3D 70), has explored lifting 2D images to 3D. By contrast,
-oodel-DDIT--. comprehensively considers the entire scene, not just the object of interest, encompassing geometry, lighting, and other salient attributes of the background.
Scene rearrangement: Current research in scene rearrangement tasks primarily involve solving rearrangement from robotic manipulation and embodied agents (39] 55
48 to provide more intuitive and human-like commands for scene manipulation and navigation. Specific attempts have
56
also been made to apply these techniques to room rearrangements 774 77, 78] using datasets like AI2-THOR [41], Habitat [69]. Gibson (00.3D-FRONT (20) or fine-tuning tuning diffusion models with convex decomposition [22.. For instance, LegoNet [78] focuses on room rearrangements without the need specify the goal state, learning arrangements that satisfy human criteria from professionally arranged datasets provided by 3D-FRÖNT 20). Distinct from these works, our research introduces a unique perspective. We focus object-level rearrangements with primary emphasis 3D-aware image editing using language instructions. 3DIT is trained with OBJECT to edit scenes with a high degree of realism and 3D coherence.
3D asset datasets: diverse set of 3D asset dataset such as ShapeNet 7 and the recent Obja- verse [14] have played pivotal role in 3D computer vision. ShapeNet provides richly-annotann large-scale dataset of 3D shapes that has found numerous applications in object recognition, scene understanding, and 3D reconstruction. Objaverse has offered a large collection of 3D objects that are semantically segmented and paired with natural language descriptions. Objaverse has been instrumental in the construction of OBJECT and also advancing several other related research areas, including generating textured meshes 88 [2]] (7] zero-shot single image 3D generation 47 and
enriching simulators 42 15 for Embodied AL.
Synthetic datasets for vision models: Diagnostic datasets such as CLEVR 33 and CLEV- FRER (81) provide rigorous test bed for the visual reasoning abilities of models. They con- tain synthetically generated images of 3D scenes with simple primitives and associated questions that require understanding of the scene's 's objects, attributes, and relations to answer correctly. Kubric (4 is an image and video dataset generation engine that can model physical interactions between objects. In a similar vein, OBJECT offers procedurally generated scenes of commonly occurring natural objects derived from VbjaVerse [4] with configurable 3D objects and associated language instructions.
Benchmarks for image editing: There is currently a scarcity of benchmarks to evaluate generative models (2), especially for 3D scene editing. Existing ones, including light probes (76). repopulating street scenes [77), GeoSim [9] and CADSim (75) are not publicly available. Our presented OBJECT benchmark will be made publicly available.
3

---

3 OBJECT: A benchmark for Object Editing in Context
Our goal is to design and evaluate image editing models capable of editing objects in scenes. To enable training and evaluation of such models, we develop OBJECT. OBJECT contains scenes with multiple objects placed on a flat textured surface and illuminated with realistic lighting. These edits are described to the model using combination of language and numerical values (e-g. pixel coordinates and object rotation angle). All edits result in structural changes to the scene which in turn affect illumination changes such as inter-object reflections and shadows. The model does not have access to the underlying 3D scene (including object segmentations, locations, 3D structure, and lighting direction); must infer these from the input pixels.
3.1 Object editing tasks
OBJECT supports four fundamental object editing tasks: Each of the following manipulations targets a single object within a scene that may contain multiple objects. We now describe each task and the capabilities required from image editing model to succeed at the task. For specifying locations an image, we use coordinate system where (0,0) represents the bottom-left corner and (1.1) the top-right corner. Objects are specified in each task using their crowdsourced descriptions.
Given the x-y coordinates original location in the scene to the target location while preserving its angular pose and surface contact. Since the camera is fixed relative to the scene, a change in object location requires to model to synthesize newly visible portions of the object. The model is required to change the object's s scale in the image due to perspective projection i.e. the objects should appear smaller when moved further away from the camera and vice-versa. The new location may also result in drastically different illumination the object.
Translation:
target location, specified object is moved from its
Rotation: A specified object is rotated counter-clockwise around the vertical axis passing through the object's center of mass and perpendicular the ground by a given angle. To succeed, the model must localize the object, extrapolate the object's 's shape from single viewpoint, and re- re-imagine the scene with the rotated object. Rotating objects leads intricate changes to the shadow projected on the ground plane which are challenging to accurately produce.
Given a language description, an ob- ject matching the description is added to the scene at designated x-y location. The model must perform object generation at the desired location with stable pose and surface contact. Besides modeling the object shape, the model also needs understand the interaction of the geometry with scene lighting to generate real- istic shadow for the object.
Insertion:
Removal: A specified object removed from the scene. The model must not only be able to locate and segment the object, but also in-paint the object region using scene context. This often requires inpainting an object that was previously partially or fully occluded.
Figure 2: Scene generation in OBJECT depicting camera constraints, directional lighting (environ- ment and three-point lighting not shown), and the resulting object shadows.
3.2 Benchmark curation
Paired image-& tttett data is plentiful on the internet and large corpora are commonly used to train text- to-image models. However, there is a lack of image editing data consisting of initial and edited image pairs, with a description of the edit. Gathering such dataset at scale from the real world requires significant manipulation and annotation effort. Our key insight is that while object manipulation data is difficult to acquire, it is much easier to synthesize large volumes of this data leveraging the latest advances photorealistic rendering and large 3D asset libraries. Therefore, OBJECT contains procedurally generated 3D scenes rendered with objects from these asset libraries.

---

Object source. OBJECT scenes are constructed using one to four 3D objects from the Objaverse [4t The entire Objaverse dataset contains more than 800k assets. Since objaverse contains objects with errors (some objects are not fully rendered or contain no texture), we filter the objects down to a set of 59k via a combination of Sketchfab metadata-based filtering and crowdsoureing. The resulting objects all have various textures, are easily recognizable, are of high quality and resolution, are free of copyrighted material, are in isolation (as opposed to a single asset with multiple objects), and are free floating (so that they may be placed on any surface in the generated scenes).
Each of these assets is annotated with one of 1613 unique semantic categories using crowdsourcing. Workers were shown rotating 3D rendering of a particular object and asked to apply a category label; they were provided with a handy autocomplete list of roughly 1400 categories sourced from LVIS categooies If, however, workers were unable to find an appropriate category, they had the option of generating a new category. After this they were asked to write a sentence that describes the object, pointing out any interesting or noteworthy details that would distinguish it from other objects in the same category. Finally, category names were cleaned up to remove spelling errors; we removed unusual or rare categories.
We randomly choose 1513 categories to be seen during training while holding out the remaining 100 as unseen categories for validation and testing. This category split helps quantify the generalization gap in editing previously seen vs novel objects. We use a library of 17 texture maps obtained from to simulate wooden, cobblestone, and brick flooring for the scenes.
[2
Scene construction. We limit all scenes minimum of one and maximum of four objects. To identify a natural resting pose for these objects, we perform a physical simulation in Blender where drop each object onto an XY ground plane and record its resting pose. Then to identify object placements, we sample a bounding box of the same x-y aspect ratio as the object and uniformly scale the object lie in this bounding box. We ensure that objects, when rotated, do not intersect each other: bounding boxes who's circumscribed circles intersect are rejected. To avoid tiny objects being placed in the same scene as very large objects, we enforce the ratio between the smallest and longest largest side of each bounding box to be greater than 0.8. We randomly place the camera in the upper hemisphere surrounding the plane and point it towards the origin which lies on the ground plane. We further constrain the camera elevation angle from the ground between 40 to 80 to ensure that the viewing angle is neither too close to the ground nor completely vertical which are both relatively unnatural. In each scene, there is a designated object that is manipulated. If this object is not visible from the camera, we move the camera away from the origin until the object visible both before and after the manipulation.
Scene lighting. We use several light sources to realistically illuminate the scene. First, we add a random environment lighting map, which are special images that capture the light in a real-world scene from all directions, giving the impression that our constructed scenes are imbedded in various indoor and outdoor locations in the real world. We download 18 of these environment maps with CCO licences from lattps://polyaawe com/ Next, we add a three-point lighting system that automatically adapts to the camera view. This involves placing the key light for primary illumination, the fill light to soften key light shadows, and the back light to distinguish the subject from the background. These lights serve to effectively shade the objects in the front of the camera so that their 3D form is apparent. Finally, the scene is illuminated with directional lighting with the direction randomly sampled within a conical neighborhood around the negative-z direction to simulate an overhead light source. This consists of parallel rays emitted by a single light source infinitely far away and therefore can be specified by intensity and direction without specifying source position.
We generate 100k training examples for each task, and 1024 scenes for validation and testing. The 3D scenes are automatically generated using Blender and its Cycles ray tracer for rendering each scene. We also render segmentation masks that denote object instances, plane and background pixels for all scenes.
3DIT: a scene-aware editing model
Task setup. Consider a 3D scene, S, filled with multiple objects. Let #1 E Hx an image of this scene produced by rendering function f. Let represent the text description of the edit, and e represent the task-specifee numerial values (i.e. angle for the rotation task and x.y coordinates for removal, insertion, and translation) to describe the desired edit to the scene S. In
represent
5

---

this paper, we consider object-centie manipulations including rotating, translating, inserting, and removing objects. Manipulating the objects in S can yield a new image 22 = (M,, where M applied the transformation 1, V in 3D.
Our goal is to produce æ2 without access the 3DD scene S and instead, directly editing the source image æ1. Importantly, we have no explicit information about the scene (including scene geometry and layout), no explicit information about the lighting (such as its location and intensity), and no access to the camera parameters. All this information must be implicitly inferred from the single source image F1. Concretely, we wish to produce the target image 22 = fo L, v), where f is a learned function with parameters 0.
Background. Diffusion models [61 have recently shown spectacular results in generating images conditioned on text descriptions. These models consist of an encoder & that maps an image into latent code = = Exx), a decoder, D that can map a latent code back to image space, and a U-Net eø with learned parameters used for denoising. Some diffusion models are trained on large training corpora such as LAION-51 5B [65] and are able to produce high-qualit, high-resolution images that faithfully represent imput text descriptions. The recently proposed Sero-I-to-4 model [47] finetunes Stable Diffusion|41 on the task of generating an image of single object from a novel viewpoint, conditioned on an imput view and a relative camera transformation.
3DIT. Our model, 3DIT, builds upon 1-to-3 We design fo(-) using the same base architecture
but make changes to its conditioning module ca(-). Our changes enable the conditioning module to accept edit instructions in the form of language and location information to precisely define the desired edit. In the cross-attention conditional module, Zero- uses a CLIP image encoding to dineniimnnio vector encoding camera
represent the initial image, followed by concatenating pose information. This -12-dimensional vector gets passed through a muuli--ayeered perceptron (MLP) to map it back down to a size of 768 dimensions. Similarly, we encode the source image 21 using the same CLIP image encoder. We encode u and concatenated the vector with the image representation and feed into the MLP. Next, we append the MLP outputs with edit text tokens 1, which are extracted using CLIP's text encoder.
We finetune our model from the 16, 500-step checkpoint of Zero-I-to-3, During training, the network takes a noised latent encoding of z₁, timestep / and conditioning information (( where ž is the latent representation of the target image at time step t. and produces a denoising score estimate eo( c0(2,, t, c(x1, 1, e)) where c(-) E 768xx x outputs sequence of conditional embedding vectors. We finetune the network with the standard diffusion loss [3 62]:
min (ende
5 Experiments
We now present experiments to evaluate our 3DIT model. First, we evaluate single task variants of 3DIT, i.e. one model for each of the four tasks - object rotation, translation, insertion and removal. For each of these tasks, we evaluate the performance of the model on novel scenes with objects seen training time, and with objects unseen at training time. We also provide evaluations for a multi-task model - trained to perform all four tasks.
5.1 Baselines
For each of the four tasks, we create strong baselines inspired by recent approaches like VisProg 28 and Socratic models (82) that chain multiple foundation models together create performant systems for various tasks including image editing.
We first use SAM [40] in the generation mode get candidate masks for the entire scene and select the mask that contains the point and occupies no more than a third of the area of the entire image. If no such mask is found, we attempt to get mask by directly using the point as input to SAM to get a mask. Then, we use Stable Diffusion (SD) to inpaint the masked region using the prompt "a rendering of uncluttered textured floor with objects".
Removal:
Insertion: This baseline uses SD and the target location to re-imagine the scene with an object of the provided category. The final image is generated by using the prompt "a 3D rendering of category on

---

Rotation
Removal
cemoon the brown $000
colai the hut
22"
roture the while
camora the gecn sports oal
couch by
rotane the pipw
CoT Taroowo the
Bloc cooc
139
Input
Baseline
3DIT (Ours)
Ground Truth
Input
Baseline
3DIT (Cuns)
Ground Truth
Insertion
Translation
Poart wood Satee
(.72..269)
maze the
98 0050 2000
breet un
insert wood radio at (0.76,0.55
Foo tbe Poreea oo 72 1000 3e0
Inser un antayy Lank at (0,27, 10,27,00.54 0.522
encoum Thee sanetto T0 (0 1..76.981 6777
Input
Baseling
3DIT (Curs)
Ground Truth
Input
Baselina
3DIT (Ouns)
Ground Truth
Figure 3: Generated examples from 3DIT as well as baselines for each of the four tasks in the OBJECT benchmark.
a textured floor" conditioned on the initial image and a fixed-size (200 x 200) square mask around the target location.
Translation requires localizing the object given the category name, removing it from the initial location, and inserting it at the target location. We use OWL-ViT 49 to localize the object given the category name. The detected bounding box is fed into SAM to generate the object segmentation mask which is then used for inpainting similar the Removal baseline. Finally, the segmented object is composited at the target location.
Translation:
Here we use Zero- 1-to-3 [47 baseline which requires the object to be tightly centered the image with white background. So, we first localize object using OWL-ViT, crop the localized region, and segment it using SAM to create the appropriate input for Zero-1-to- for performing the rotation. The rotated object is composited back onto the image and the remaining unfilled regions are inpainted using SD.
Rotation:
5.2 Quantitative evaluation
We follow Zero-I-to-3 and use four metrics to automatically evaluate the quality and accuracy of the edited image PSNR, SSIM, LPIPS, and FID. The first 3 directly compare the prediction to the ground truth image, while FID measures the similarity between the predicted and ground truth sets of images. Instead of computing the metrics for the whole image, we focus on the region where edits are targeted. To do this, we simply use ground truth segmentation mask to crop the targeted rectangular region of interest prior to computing the metrics. Since our model, well as our baselines, can generate multiple solutions for each input, our evaluation considers the best-of-four prediction as per the SSIM metric to compute the final scores for all metrics. This considers the
7

---

Table I: Quantitative evaluation using generated samples. For each method, four samples per test image were generated. The best image according to the PSNR metric is selected represent each sample, and these values are averaged across samples. To ensure that the metrics focus on the transformed object and not the background which mostly remains unchanged, metrics are computed using the region around the transformed object's s mask.
Seen Objects
Unseen Objects SSIM
Model
PSNR 1
SSIM f
LPIP +
FID
PSNR t
LPIP +
FID
Task: Translation
Baseline
13.699
0.309
0.485
0.942
14.126
0.326
0.467
0.968
3DIT(1-task)
14.546
0.273
0.494
0.254
14,400
0.262
0.498
0.261
3DIT(Multitask)
15.21
0.300
0.472
0.244
15.200
0.292
0.477
0.253
Task: Rotation
Baseline
13.179
0.269
0.540
0.997
12.848
0.270
0.538
1.693
3DIT (1ttsk)
16.828
0.386
0.428
0.291
16,293
0.372
0.445
0.280
3DIT(Multitask)
16.859
0.382
0.429
0.248
16,279
0.366
0.447
0.236
Task: Insertion
Baseline
12,297
0.269
0.594
0.969
12.542
0.275
0.584
1.325
3DIT(1-task)
13.469
0.267
0,549
0.254
12.974
0.261
0,566
0.233
3DIT(Multitask)
13.630
0,263
0.551
0.222
13.088
0.261
0.568
0.214
Task: Removal
Baseline
12.494
0.383
0,465
0.801
12.123
0.379
0.459
1.047
EDIT(I-task)
24.937
0.588
0.254
0.241
24.474
0.561
0.260
0,258
3DIT(Multitask)
24.980
0.585
0.249
0.236
24.661
0.568
0.260
0.240
typical use case for editing applications where a user has the flexibility to pick from range of generated solutions. We report metrics separately for seen and unseen object categories.
Table presents quantitative evaluations for 3DIT comparison to the baselines. 3DTTouperforms the baselines for all four tasks at the metrics PSNR, SSIM and LPIP. Notably, the multi task model does well in comparison to the single task variant, in spite of having to learn tasks using the same number of learnable parameters. The FID scores for the baseline models tend to be higher. This is because the baselines tend to cut/paste objects in the image (for e.g. in the translation task), which retains image fidelity, even if the scale of the object is incorrect. 3DIT on the other hand does not explicitly cut/paste segments and instead must render them using the diffusion process, and is thus prone to poorer fidelity. On the contrary, challenging changes to the underlying 3D scene when editing images, as shown Figure 4 Its worth noting that the automatic evaluation metrics have limitations and often do not capture editing nuances encompassing geometry, lighting, and fidelity to the instruction. This motivates the need for human evaluation studies.
model is able to properly account for a variety of
5.3 Human evaluation studies
We conduct human preference evaluations between 3DIT and the relevant baseline by showing two images and asking annotators to select the one that best matches the ground truth image. We measure (1) Geometric consistency This requires humans to consider the geometric correctness of the transformed object, including the scale, positioning of the object on the ground plane and its relationship to other objects. also requires humans consider the correctness of other objects in the scene which may get occluded or unoccluded as result of the transformation. the source caption. (2) Lighting consistency This requires humans to consider the lighting correctness of the transformed object, including the direction and scale of the shadow as a result of the directional lighting. also requires humans to consider the correctness of the shadows of other objects in scene which may get occluded unoccluded as result of the transformation. Both evaluations also allow a third option (Tie) to selected. Each pairwise evaluation carried out for 30 test samples for OBJECT evaluation and 20 samples for real world evaluation.

---

Perspective change
Novel view synthesis
Occlusion
Lighting change
Figure 4: The figure shows the ability of 3DIT to handle various challenges of 3D-aware image editing such as: (a) (Top left) perspective size changes; (b) (Top right) synthesizing novel view points; (c) (Bottom left) generating occluded regions; (d) (Bottom right) accounting for scene lighting while rendering objects and their shadows.
presenss human evaluation study of the 3DIT model (in single task setting) in comparison to the corresponding baseline for all four tasks for images from OBJECT and the real world. 3DIT is heavily favored by humans, consistently obtaining preference scores of 70 % and more across all four tasks for geometric as well lighting consistency. The tied scores refer to instances where both models did exceedingly poorly and where both models did close perfect job.
For the translation task, 3DIT is able to scale the object appropriately, well rendering the shadow correctly. The baseline, in particular, does a poor job of the shadow and gets the scale wrong, leading to physically implausible image. For the rotation task, 3DIT performs a rotation consistent with the ground plane and also renders superior shadow. For the removal task, 3DIT tends to inpaint occluded objects well, and correctly adjusts their shadows. It also does well at removing the entire extent of the correct object in contrast to the baseline.
5.4 Real-world transfer qualitative
While we train our models on simulated data, we the model's ability to transfer real-world images qualitatively. Figure shows our model's output for different prompts for the same input image for all four tasks. We find these preliminary results encouraging as the outputs not only respect the task description but also look reasonably photo-realistic with appropriate shadows despite never seeing real- world editing examples during training.
6 Limitations and Broader Impact
Our work explores the use of synthetic data for training physically plausible and scene-aware image editing models. Given that even training on scenes with limited realism and complexity results in models that transfer well to the real world, there is tremendous potential to significantly improve performance by using more advanced photo-realistic simulators. We give an analysis of the typical failure cases of our model in the appendix. Finetuning on a small set of hand-crafted real-worls editing examples may also improve transfer to real-world images and enable compelling editing applications. Our work leads the way towards easy-to-use and increasingly powerful image editing

---

Table 2: Outcome of the human evaluation. The table illustrates the evaluators' preferences for 3DIT assessed on geometric accuracy and 3D lighting consistency. Baseline methods rarely gained preference due to their limited capacity to maintain geometric quality and lighting consistency.
OBJECT evaluation
Task
Geometric consistency
Lighting consistency
Baseline
3DrT(Ours)
Tie
Baseline
(Ours)urs)
Tie
Translation
20.0 %
73.3 %
6.6 %
3.3 %
80.0 %
16.6 %
Rotation
3.3 %
80.0 %
16.6 %
6.6 %
73.3 %
20.0 %
Insertion Removal
13.3 %
70.0 %
16.6 %
10.0 %
73.3 %
16,6 %
3.3 %
86.6
10.0 %
0.0 %
86.6 %
13.3 %
Real world evaluation
Task
Geometric consistency
Lighting consistency
Baseline
ODrr()
Tie
Baseline
3DIT(Ours)
Tie
Translation
25.0 %
60.0 %
15.0 %
15.0 %
70.0 %
15.0 %
Rotation
10.0 %
80.0
10.0 %
10.0 %
80.0 %
10.0 %
Insertion Removal
35.0 %
55.0 %
10.0 %
35.0 %
55.0 %
10.0 %
10.0 %
75.0
15.0 %
5.0 %
80.0 %
15.0 %
Translation
Rotation
oaagg S2., 0.8
StoWn 12.. 0.31
187
270*
uppt
Net 2ad 0.5)
nod cm (2.55,059)
ned can (8, 2...22
ippt
182
Insertion
Removal
lppt
ALO ax)
32 2335..33
and book 35.0.00
iiput
1o0dlpatte
solscrs
mouul
Irout
0.65)
aog
bookanek www.nee//..2.....mo
Inpot
book
whide Book
Figure 5: 3DIT is able to generalize to the real world while only being trained synthetic dataset. Here we show varying prompts for each of the four editing tasks.
capabilities for the broader society in the near future. Like any generative model, our work could also potentially be misused for propagating misinformation.
Conclusion
This work presents 3DIT, a model capable of editing individual objects within images, given language instruction. 3DIT is trained on new dataset, OBJECT, consisting of 400k 3D scenes procedurally generated using Objaverse objects. 3DIT performs well across on OBJECT and shows promising generalization CLEVR well as the real world.
10

---

References
[1] R. Abdal, P. Zhu, N. J. Mitra, and P. Wonka. Styleflow: Attribute exploration of stylegan-generated images using conditional continuous normalizing flows. ACM Transactions on Graphics (ToG), 40(3):1-21, 2021.
[2] O. G. Art. OpeoGameArt.orgg 16-May-2023).
httpss//opengamerrt.oom 2023. [Online; accessed
[3] A. Bhattad, A. Dundar, G. Liu, A. Tao, and Catanzaro. View generalization for single image textured 3d models. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, 2021.
[4] A. Bhattad and D. Forsyth. Cut-and-pastr object insertion by enabling deep image prior for reshading. In 2022 International Conference on 3D Vision (3DV). IEEE, 2022.
[5] A. Bhattad and D. A. Forsyth. Stylitgan: Prompting stylegan to produce illumination conditions, 2023.
[6] E. R. Chan, C. Z. Lin, M. A. Chan, K. Nagano, B. Pan, S. De Mello, O. Gallo, L. Guibas, J. Tremblay, S. Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16123-16133, 2022.
[7] A. X. Chang. Funkhouser, L. Guibas, P. Hanrahan. Q. Huang, Li, Savarese, M. Savva, S. Song, H. Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: 1512.03012, 2015.
[8] D. Z. Chen, Y. Siddiqui, H.-Y. Lee, S. Tulyakov, and M. Nießner. Text2tex: Text-driven texture synthesis via diffusion models. arXiv preprint arXiv:2303.11300 2023.
[9] Y. Chen, F. Rong, S. Duggal, S. Wang, X. Yan, S. Manivasagam, S. Xue, E. Yumer, and R. Urtasun. Geosim: Realistic video simulation via geometry-aware composition for self-driving. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021.
[10] M. J. Chong and D. Forsyth. Jojogan: 1Xiv:2112.11641 2021.
One shot face stylization.
arXiv preprint
[11] M. J. Chong, H.-Y. Lee, and D. Forsyth. Stylegan of all trades: Image manipulation with only pretrained stylegan. arXiv preprint arXiv:2111.01619, 2021.
[12] B. O. Community. Blender a 3d modelling and rendering package, 2018.
[13] M. Deitke, R. Liu, M. Wallingford, H. Ngo, O. Michel, A. Kusupati, A. Fan, C. Laforte, V. Voleti, S. Y. Gadre, et al. Objaverse-xl: universe of 10m+ 3d objects. arXiv preprint
arXiv:2307.05663. 2023.
[14] M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi. Objaverse: A universe of annotated 3d objects. arXiv preprint arXiv:2212.08051, 2022.
[15] M. Deitke, E. VanderBilt, A. Herrasti, L. Weihs, J. Salvador, K. Ehsani, W. Han, E. Kolve, A. Farhadi, A. Kembhavi, et al. Procthor: Large-scale embodied ai using procedural generation. Conference on Neural Information Processing Systems, 2022.
[16] A. Deshpande, J. Lu, M.-C. Yeh, M. Jin Chong, and D. Forsyth. Learning diverse image col- orization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6837-6845, 2017.
[17] Dhariwal and Nichol. Diffusion models beat gans on image synthesis. Advances Neural Information Processing Systems, 34:8780-8799 8794,, 2021.

---

[18] A. A. Efros and W. T. Freeman. Image quilting for texture synthesis and transfer. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 341-346, 2001.
[19] D. Epstein, A. Jabri, B. Poole, A. A. Efros, and A. Holynski. Diffusion self-guidance for controllable image generation. arXiv preprint arXiv:2306.00000 2023.
[20] H. Fu, B. Cai, L. Gao, L.-X. Zhang. J. Wang, C. Li. Q. Zeng, C. Sun, R. Jia, B. Zhao, et al. 3d-front: 3d furnished rooms with layouts and semantics. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 10933-10942, 2021.
[21] Gao, T. Shen, Z. Wang, Chen, K. Yin, D. Li, O. Litany, Z. Gojeic, and S. Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. In Advances In Neural Information Prucessing Systems, 2022.
[22] L. A. Gatys, A. S. Ecker, and M. Bethge. Image style transfer using convolutional neural networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414-2423, 2016.
[23] J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. arXiv preprint arXiv: 1406.266,, 2014.
[24] K. Greff, F. Belletti, L. Beyer, C. Doersch, Y. Du, D. Duckworth, D. Fleet, D. Gnanapragasam, Golemo, C. Herrmann, et al. Kubric: A scalable dataset generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3749-3761, 2022.
[25] Gu, L. Liu, P. Wang, and C. Theobalt. Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis. arXiv preprint arXiv:21 10.08985, 2021.
[26] A. Gupta, Dollár, and R. Girshick. Lvis: A dataset for large vocabulary instance segmenta- tion. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5351-5359, 2019.
[27] A. Gupta, W. Xiong, Y. Nie,, I. Jones, and B. Oguz. 3dgen: Triplane latent diffusion for textured mesh generation. arXiv preprint arXiv:2303.05371, 2023.
[28] T. Gupta and A. Kembhavi. Visual programming: Compositional visual reasoning without training. ArXiv, abs/2211.115553 2022.
[29] A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-Or. Prompt-to- prompt image editing with cross attention control. arXiv preprint arXiv:2208.016 2022.
[30] A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H. Salesin. Image analogies. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 327-340 2001.
[31] J. Ho, A. Jain, and Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6857 2020.
[32] Y. Hu, B. Liu, Kasai, Y. Wang, M. Ostendorf, R. Krishna, and N. A. Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. arXiv preprint ar3iv:2303. 1897, 2023.
[33] J. Johnson, B. Hariharan, I.. Van Der Maaten, I.. Fei-Fei, C. Lawrence Zitnick, and R. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901- 2910, 2017.
[34] M. Kang, J.-Y. Zhu, R. Zhang, J. Park, E. Shechtman, S. Paris, and T. Park. Scaling up gans for text-to-image synthesis. arXiv preprint arXiv:23030055/1 2023.
[35] T. Karras, M. Aittala, Laine, F. Härkönen, J. Hellsten, Lehtinen, and Aila. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 34, 2021.
12

---

[36] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[37] Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, Analyzing and improving the image quality of stylegan. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, 2020.
[38] B. Kawar, S. Zada, O. Lang, O. Tov, H. Chang, T. Dekel, I. Mosseri, and M. Irani. Imagic: Text-based real image editing with diffusion models. In Conference on Computer Vision and Pattern Recognition 2023, 2023.
[39] J. E. King, M. Cognetti, and S. Srinivasa. Rearrangement planning using object-centric and robot-centric action spaces. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 3940-3947. IEEE, 2016
[40] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitchead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. B. Girshick. Segment anything. ArXiv, abs/2304.02643 2023.
[41] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, M. Deitke, K. Ehsani, D. Gordon, Y. Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arX1v:1712.05474, 2017.
[42] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, M. Deitke, K. Ehsani, D. Gordon, Y. Zhu, A. Kembhavi, A. K. Gupta, and A. Farhadi. Ai2-thor: An interactive 3d environment for visual ai. arXiv e-prints, pages arXiv-1712, 2017.
[43] N. Kumari, B. Zhang, R. Zhang, E. Shechtman, and J.-Y. Zhu. Multi-concept customization of text-to-image diffusion. In CVPR, 2023.
[44] LambdaLabs.
https//lambbaalas..cmm// stable- www.fausssoooo-iie----ooiiiiiiia
https://hugginggace... http:/aac....o/mmmmaaaaaalaaaaaa 2023. [Online; accessed 17-May-2023).
[45] G. P. Laput, M. Dontcheva, G. Wilensky, W. Chang, A. Agarwala, J. Linder, and E. Adar. Pixeltone: A multimodal interface for image editing. In Proceedings of the SIGCHI Conference on Human Factors Computing Systems, pages 2185-2194, 2013.
[46] Z. Liao, H. Hoppe, D. Forsyth, and Y. Yu. A subdivision-based representation for vector image editing. IEEE transactions on visualization and computer graphics, 2012.
[47] R. Liu, R. Wu, B. Van Hoorick, P. Tokmakov, S. Zakharov, and C. Vondrick. Zero-I-to-3; Zero-shot one image to 3d object. arXiv preprint arXiv:2303.11322 2023.
[48] W. Liu, C. Paxton, T. Hermans, and D. Fox. Structformer: Learning spatial structure for language-guided semantic rearrangement of novel objects. In 2022 International Conference on Robotics and Automation (ICRA), pages
IEEE, 2022.
[49] M. Minderer, A. A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn, Dosovitskiy, A. Mahendran, A. Arnab, M. Dehghani. Z. Shen, X. Wang, X. Zhai, T. Kipf, and N. Houlsby, Simple open opca-vocyy object detection with vision transformers. ArXiv, has/2205500230. 2022.
[50] C. Mou, X. Wang, L. Xie, Zhang, Z. Qi, Y. Shan, and X. Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.0m 08453, 2023.
[51] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112. 107741,, 2021.
[52] A. Q. Nichol and Dhariwal. Improved denoising diffusion probabilistic models. In Interna- tional Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
13

---

[53] Park, M.-Y. Liu, T.-C. Wang, and J.-Y. Zhu. Semantic image synthesis with peatially-adaptiin normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[54] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:22099 14988, 2022.
[55] A. H. Qureshi. A. Mousavian, C. Paxton, M. C. Yip, and D. Fox. Nerp: Neural rearrangement planning for unknown objects. arXiv preprint arXiv:2106.01355 2021.
[56] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell,
P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8766 PMLR, 2021.
[57] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:204.061125, 2022.
[58] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. I. PMLR, 2021.
[59] E. Reinhard, M. Adhikhmin, B. Gooch, and P. Shirley, Color transfer between images. IEEE Computer graphics and applications, 21(5):34 41, 2001.
[60] E. Richardson, Y. Alaluf, O. Patashnik, Y. Nitzan, Y. Azar, S. Shapiro, and D. Cohen-Or.
Encoding in style: a stylegan encoder for image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2287-2296, 2021.
[61] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models, 2021.
[62] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[63] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gon- tijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 14:36479-36494 2022.
[64] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08465 2022.
[65] C. Schuhmann, R. Beaumont, Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarezyk, and J. Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models, 2022.
[66] Y. Shen, C. Yang, X. Tang, and B. Zhou. Interfacegan: Interpreting the disentangled face representation learned by gans. IEEE transactions on pattern analysis and machine intelligence, 2020.
[67] A. Shoshan, N. Bhonker, I. Kviatkovsky, and G. Medioni. Gan-control: Explicitly controllable gans. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14083-14093, 2021.
[68] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint ar arXiv:2010.02502, 2020.
[69] A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. Turner, N. Maestre, M. Mukadam, D. Chaplot, O. Maksymets, A. Gokaslan, V. Vondrus. Dharur, F. Meier, W. Galuba, A. Chang,
Z. Kira, V. Koltun, J. Malik, M. Savva, and D. Batra. Habitat 2.0: Training home assistants
to rearrange their habitat. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
14

---

[70] J. Tang, T. Wang, B. Zhang. T. Zhang, R. Yi, L. Ma, and D. Chen. Make-it-3d: High-fidelity 3d creation from single image with diffusion prior. arXiv preprint arXiv:2303.14184 2023.
[71] D. Ulyanov, A. Vedaldi, and Lempitsky. Deep image prior. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2018.
[72] V. Vavilala, S. Jain, R. Vasanth, A. Bhattad, and D. A. Forsyth. Blocks2world: Controlling realistic scenes with editable primitives. CoRR, abs/2307 abss/237.00387,
[73] H. Wang, X. Du, J. Li, R. A. Yeh, and G. Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. arXiv preprint arXiv:2212.00774, 2022.
[74] H. Wang, W. Liang, and L.-F. Yu. Scene mover: Automatic move planning for scene arrangement by deep reinforcement learning, ACM Transactions on Graphics, 39(6), 2020.
[75] J. Wang, S. Manivasagam, Y. Chen, Z. Yang, I. A. Bårsan, A. J. Yang, W.-C. Ma, and R. Urtasun. Cadsim: Robust and scalable in-the-wild 3d reconstruction for controllable sensor simulation. In 6th Annual Conference on Robot Learning. 2022.
[76] Y. Wang. B. L. Curless, and S. M. Seitz. People Computer Vision, pages 438 454. Springer, 2020.
probes. In European Conference
[77] Y. Wang, A. Liu, R. Tucker, J. Wu, B. L. Curless, S. M. Seitz, and N. Snavely, Repopulating street scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[78] Q. A. Wei, S. Ding, J. Park, R. Sajnani, A. Poulenard, S. Sridhar, and L.. Guibas, Lego-net: Learning regular rearrangements of objects in rooms. arXiv preprint arXiv:2301.09629 2023.
[79] L. Weihs, M. Deitke, A. Kembhavi, and R. Mottaghi. Visual room rearrangement. In Proceed- ings of the IEEE/CVF conference computer vision and pattern recognition, pages 9922-5931, 2021.
[80] F. Xia, A. R. Zamir, Z.-Y. He, A. Sax, J. Malik, and S. Savarese. Gibson env: real-world perception for embodied agents. In Computer Vision and Pullern Recognition (CVPR), 2018 IEEE Conference on. IEEE, 2018.
[81] K. Yi, C. Gan, Y. Li, P. Kohli, J. Wu, A. Torralba, and J. B. Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442, 2019.
[82] A. Zeng, A. Wong. S. Welker, K. Choromanski, F. Tombari, A. Purohit, M. S. Ryoo, V. Sindhwani, J. Lee, V. Vanhoucke, and P. R. Florence. Socratic models: Composing zero-shot multimodal reasoning with language. ArXiv, abs/2204.00598. 2022.
[83] L. Zhang and M. Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
[84] Zhu, Y. Shen, D. Zhao, and B. Zhou. In-domain gan inversion for real image editing. In Proceedings of European Conference on Computer Vision (ECCV), 2020.
[85] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-imase translation using cycle-consistent adversarial networks. In Proceedings of the IEÉE international conference on computer vision, 2017.
15

---

8 Appendix
8.1 Training and Inference Details
closely follow the training procedure established by [47], with a few modifications. Our approach uses an effective batch size of 1024, which is smaller than the batch size of 1536 used by fero-1-to-3. This adjustment was necessary because of the additional memory requirements caused by the reintroduction of the CLIP text encoder. This batch size achieved by using local batch size of 64 across 40GB NVIDIA RTX A6000 GPUs, along with two gradient accumulation steps. Similar to Zero 13,, train images with resolution of 256 256, resulting in latent spatial dimension of 32 x 32. Following their protocol, we utilize the Adam W optimizer, with a learning rate of le-4 for all parameters of the model except for those of the concatenation MLP, which uses learning rate of le 3. Our training process runs for a total 20,000 steps. We then select the best checkpoint based on metrics computed from an unseen object validation As was the case in StableDiffusion freeze the CLIP text encoder during training. For inference, we generate images with the DDIM [68] sampler using 200 steps. We do not classifier-free guidance, i.e. the cfg term is 1.0.
8.2 Robustness to severity of transformation
We analyze the robustness of our method by measuring the performance of the single task rotation model the complexity of the scene and severity of transformation changes. In Figure [6] we show the average of our Mask PSNR metric as the number of objects in the scene varies from to 4, where slight drop in performance occurs the number of objects increases. In Figure show average Mask PSNR for rotations in a given angle range on a pie chart, where it can be seen that the model does better with smaller angle deviations.
Robustness Angle
60-120
16
120-180
14-
14
Maxt PONn
180-240
300-360
11
240-300
Figure 6: Average Mask PSNR of the single- task rotation model the number of objects in the scene varies.
Figure 7: Average Mask PSNR of the single- task rotation model for rotation angles falling within slice.
8.3 Model failure analysis
Here analyze the most common failure model of 3DIT.
Incorrect geometry: The model incorrectly changes the object's geometry.
Unintended global modifications: The model changes parts of the scene that are not intended to be edited.
Incorrect texture: The model changes details on the surface the object, failing to preserve its identity.
Incorrect localization: The model performs the requested task, but on an object that is different from the intended one.
16

---

Incored Mcalication
Other
10
Unintended global modifications
Incorrect geometry
Incanedt textur
21.3%
Incorrect texture
Incorrect localization instruction: block)
Figure 9: Frequency of most common failure cases. examine 50 random outputs of the model and bucket the model errors by failure type
Figure 8: Qualitative examples of the four must common failure cases produced by the model.
8.4 Initialization ablation
To test the effect of the model's initialization, initializations: image-conditioned Stable Diffusion 44] Zero123 [47] (our standard initialization). and the recently released Zero 13-XLL [I3,, which is Zero123 model trained
ablation training model three different
much larger version of the Objaverse dataset with 10M assets. We see that although Zero123-XL. was a more performant base model for the task of rotating single object, it does not cause improvement overall on the tasks we consider here.
Table 3: Quantitative results for training our single-task model across three different ablations: image-conditoined Stable Diffusion, Zerol23, 123, and the recently released Zero123-XI
Seen Objects
Unseen Objects SSIM
Model
PSNR
SSIM
LPIP
FID
PSNR
LPIP
FID 1
Task: Translation
SD
14.373
0.264
0.510
0.101
14.351
0.253
0.510
0.102
Zero123
15.210
0.300
0.472
0.244
15.200
0.292
0.477
0.253
Zero123-XL
15.121
0.294
0.477
0.252
15.052
0.286
0.478
0.239
Task: Rotation
SD
15.074
0.368
0.430
0.089
14.558
0.359
0.433
0.095
Zerol23
16.859
0,382
0.429
0.248
16.279
0.366
0.447
0.236
Zero123-XL
15.433
0.381
0.420
0,241
15.008
0.378
0.429
0.243
Task: Insertion
SD
13.220
0.253
0.570
0.108
13.131
0.255
0.572
0.100
Zero123
13.630
0.263
0.551
0,222
13.088
0.261
0.568
0.214
Zero XL
13.481
0.264
0.557
0.274
13.094
0.259
0.566
0.258
Task: Removal
SD
23.882
0.576
0.263
0.117
23.352
0.542
0,270
0.115
Zerol23
24.980
0.585
0.249
0.236
24.661
0.568
0.260
0.240
Zerol23 XL
24.775
0.585
0.255
0.247
24.830
0.568
0.253
0.215
8.5
Additional qualitative results
In Figure 10
show qualitative results from multitask model each of the four editing tasks.
17

---

Table 4: Summary of key statistics of the OBJECT dataset.
Total objects
62950
Total categories
1613
Object per category median
6.0
Object per category mean
39.03
Object per category std
138.00
8.6 Dataset Analysis
details about the composition and statistical makeup of our dataset. In Table 4 we show a statistical overview the dataset, including total number of objects and categories, well as the mean, median, and standard deviation of objects per category. We also visualize the distribution of objects across categories, as can be show in the histogram Figure [1 Finally, visualize the frequency of category names in the wordcloud in Figure 12
In this seciton, we provide
18

---

Task:
Task: Remove
rotate bottle by 224
remove the
remove the lego brick
rotate the lion by 189
rotate the blimp by 101
remove unicycle
rotate the tomb by
remove the robot
rotate radio by 220*
remove the blue toy figurine
rotate the tank by 338
remove the wooden blocks
Task: Insert
Task: Translate
insert a sofa at (0.32,0.76)
move the lettuce to (0.54,0.52)
insert a gold trophy (0.42,0.71)
move the rock to (0.48,0.61)
move the sofa to (0.84,0.55)
insert the plastic bottle at (0.37.0.72)
the wooden block archway at (0.49.0.70)
move the oil lamp to (0.54,0.52)
insert the first aid kit (0.77,0.45)
clay
jug to (0.75,0.70)
insert black boot (0.62,0.69)
the box to (0.76,0.46)
Prompt
Input Image
Ground Truth
Ours
Prompt
Image
Ground Truth
Ours
Figure 10: Generated examples by the 3DIT multitask model.
19

---

cup
airplane
Compu
Piepessn
peeent
ri
book
bowl
d
mhosse
Tecile
shield
lottle
Tantern
covestote
card
desk
d
Aid
trading
tree
hammer
bi
house
shōe
seleviaton
skull
Lampp
bui
lding
sword
table
dugger
plant Sot
gurinn inne
toy
m
bag
ingkife
vehicle
wewwe pister
st statue
S
sculpture
shelver vees arrc
Lagoie
character
statue cu
culpture
car
automobile
nachine gun mark
coin
chairiis
Model
bed
bar
pistol
mug shar
beqdinmnn
potter
vase
berrser
axe e
Frat
box
ball
computer
Object cabegory frequency (loq scall!
helmet projectile weapon
cabinet trunk
b
Object categories from seen and unseen splits sorted by frequency, in log scale.
Figure 11:
Figure 12: A wordeloud visualizing the fre- quency of various object category names.
20


Reviews:
[{"review_id": "54zh0IkTfY", "reviewer": "Reviewer_1T99", "summary": "The 3DIT model is a language-guided 3D-aware image editing tool that allows for effective object editing while considering scale, viewpoint, lighting, and object occlusions. The model builds upon previous work in scene rearrangement and image generation, and incorporates a diffusion process to render object transformations. The authors conducted human preference evaluations to measure geometric and lighting consistency, and found that 3DIT outperformed relevant baselines in both categories.\n\nOne of the key strengths of 3DIT is its ability to add, remove, or edit shadows to maintain consistency with scene lighting. This is achieved through a shadow generation module that takes into account the position and orientation of the light source, as well as the geometry of the objects in the scene. Additionally, 3DIT accounts for object occlusions by using a novel occlusion-aware rendering module that predicts the visibility of each object in the scene.\n\nThe authors also introduced a new benchmark dataset called OBJECT, which consists of 3D scenes with multiple objects and associated natural language descriptions. They trained 3DIT on this dataset and found that it generalized well to images in the CLEVR dataset as well as the real world. This demonstrates the robustness and versatility of the model, and suggests that it could be applied to a wide range of real-world scenarios. Overall, by enabling users to edit objects in a natural and intuitive way, 3DIT opens up new possibilities for creative expression and visual communication.\n", "questions": "N/A", "limitations": "See above", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The 3DIT model is a novel approach to language-guided 3D-aware image editing that builds upon previous work in scene rearrangement and image generation. The model incorporates a diffusion process to render object transformations and uses a novel shadow generation module and occlusion-aware rendering module to maintain consistency with scene lighting and object occlusions.\n\n- The authors conducted human preference evaluations to measure geometric and lighting consistency, and found that 3DIT outperformed relevant baselines in both categories. This demonstrates the effectiveness of the model in producing high-quality, visually consistent image edits.\n\n- The authors trained 3DIT on a new benchmark dataset called OBJECT and found that it generalized well to images in the CLEVR dataset as well as the real world. This suggests that the model is robust and versatile, and could be applied to a wide range of real-world scenarios.\n\n- The potential applications for this technology are vast, including virtual and augmented reality, gaming, and e-commerce. The model could be used to create personalized avatars for virtual reality environments, or to generate realistic product images for e-commerce websites. Additionally, the model could be extended to support more complex scenes and interactions, such as object physics and collision detection.\n", "weaknesses": "- The ablation study is relatively weak. It's unknown which component contributes most to the final performance and which is effective.\n\n- The shadow of the box did not follow the rotation action in the shown GIF. And some artifacts are obvious.\n\n- How did the method choose which box to be moved? Or did it need a handcrafted mask as the selection?\n\n- What is the number of samples used for calculating FID? Normally FID is not reliable of the number of samples is small.\n\n- The comparing table did not include the previous method. I believe several important baselines [a] methods are missing.\n\nReferences:\n[a] Editable free-viewpoint video using a layered neural representation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to read our submission and for providing valuable feedback. We will now respond to the highlighted questions and concerns.\n\n**Lack of ablation**\nWe have extensively evaluated our model's capability using multiple metrics across tasks, single/multitask models, and seen/novel objects. In addition, we now include an ablation of model initialization. To test the effect of the pre-trained initialization, we train two additional versions of the multitask model with different initializations. One is initialized with image-conditioned StableDiffusion [1] and the other with Zero123-XL [2]. \n\nIn the table below, we find both Zero-123 based initializations to outperform Stable Diffusion demonstrating the benefits of Zero-123’s novel-view synthesis training by helping the model develop a better implicit understanding of 3D geometry of objects. Initializing with Zero-123 trained on the larger Objaverse-XL dataset shows diminishing returns with regards to object-level pretraining. However, the scaling limits of our 3D-aware editing training are yet to be explored and would be an interesting topic for future work.\n\n|                   | Seen Objects |       |       |       | Unseen Objects |        |       |       |\n|-------------------|--------------|-------|-------|-------|----------------|--------|-------|-------|\n| Model             | PSNR         | SSIM  | LPIP  | FID   | PSNR           | SSIM   | LPIP  | FID   |\n| Task: Translation |              |       |       |       |                |        |       |       |\n| SD                | 14.373       | 0.264 | 0.51  | 0.101 | 14.351         | 0.253  | 0.51  | 0.102 |\n| Zero123           | 15.21        | 0.3   | 0.472 | 0.244 | 15.2           | 0.292  | 0.477 | 0.253 |\n| XL                | 15.121       | 0.294 | 0.477 | 0.252 | 15.052         | 0.286  | 0.478 | 0.239 |\n| Task: Rotation    |              |       |       |       |                |        |       |       |\n| SD                | 15.074       | 0.368 | 0.43  | 0.089 | 14.558         | 0.359  | 0.438 | 0.095 |\n| Zero123           | 16.859       | 0.382 | 0.429 | 0.248 | 16.279         | 0.366  | 0.447 | 0.236 |\n| XL                | 15.433       | 0.381 | 0.42  | 0.241 | 15.008         | 0.3783 | 0.429 | 0.243 |\n| Task: Insertion   |              |       |       |       |                |        |       |       |\n| SD                | 13.22        | 0.253 | 0.57  | 0.108 | 13.131         | 0.255  | 0.572 | 0.1   |\n| Zero123           | 13.63        | 0.263 | 0.551 | 0.222 | 13.088         | 0.261  | 0.568 | 0.214 |\n| XL                | 13.481       | 0.264 | 0.557 | 0.274 | 13.094         | 0.259  | 0.566 | 0.258 |\n| Task: Removal     |              |       |       |       |                |        |       |       |\n| SD                | 23.882       | 0.576 | 0.263 | 0.117 | 23.352         | 0.542  | 0.27  | 0.115 |\n| Zero123           | 24.98        | 0.585 | 0.249 | 0.236 | 24.661         | 0.568  | 0.26  | 0.24  |\n| XL                | 24.775       | 0.585 | 0.255 | 0.247 | 24.83          | 0.568  | 0.253 | 0.215 |\n\n**Inconsistent shadow in GIF**\nAs mentioned in the common statement, the original version of the OBJECT dataset used a single directional light source which was very different than lighting found in CLEVR (the dataset used for making the GIFs) and real images. Hence, learning to model lighting and shadows on the OBJECT dataset had limited generalization to out-of-distribution images found in CLEVR. In-line with reviewer suggestions, we improved the dataset with a more realistic 3-point lighting (see the Figures B for examples of the improved dataset) that reduced the sim-to-real gap as well as the domain gap to CLEVR. As can be seen in figure H, the new shadow of the blue cube more accurately moves in response to the rotation.\n\n**Artifacts in the GIF**\nLike other SOTA image editing diffusion models, the model often makes small changes outside the region of interest. These artifacts are often difficult to notice in isolated edited images but become clear when making a sequence of edits and compiling them into a video or GIF. Techniques for improving temporal consistency in video-editing literature might be applicable, but our current benchmark and method focuses on single atomic edits as an important first step. \n\n**How did the method choose which box to be moved? **\nWe do not need to provide a mask for the box because our method is conditioned on a language description of the object. We identify the box with the prompt “a blue box” for the rotation GIF. Please see the supplementary material of our submission for the rest of the prompts in the GIFs.\n\n**Number of samples for FID**\nWe calculated FID with 1024 samples. \n\n**Additional baseline**\n\nIn this work, we proposed a novel problem of 3D-aware image editing. While there aren't existing methods that solve this exact problem, we do compare to relevant baselines that involve chaining off-the-shelf models together. The suggested baseline, from the paper “Editable free-viewpoint video using a layered neural representation”, is interesting, but requires video from 16 different cameras. Our intention is to allow users to edit photos from a single RGB image, making this baseline out of the scope of our problem formulation. However, we will add this paper to the discussion in the related work.\n\nReferences\n[1] High-Resolution Image Synthesis with Latent Diffusion Models\n[2] Objaverse-XL: A Universe of 10M+ 3D Objects\n\n"}, {"review_id": "jJFiPaSYBX", "reviewer": "Reviewer_Rpi6", "summary": "The authors propose a large dataset of  3D aware image edits along with editing instructions built on the objaverse dataset. They also introduce a model finetuned on Zero-1-to-3 for 3D aware editing tasks which include object insertion, removal, translation and rotation. Comparisons are provided against state of the art models for each task and performance improvement is demonstrated. ", "questions": "1. For translation and insertion, what are the location? are they provided in pixel space? Since they are 3D aware, does it make more sense to provide edit instruction in camera coordinates? ( In pixel coordinates, how is depth interpreted?)\n2. Is it easier to provide control inputs as a spatial map (say as gaussians/ keypoints on a 2D image) rather than as a pixel locations to allow for more local specificaition?", "limitations": "Adequate treatment of limitations have been provided.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. **Clarity**: The paper is well written with attention to detail. All the necessary details particularly with regards to the dataset creation have been adequately explained.\n2. **Interesting dataset**: The 400k dataset of images along with edit instructions would serve as an interesting training and benchmarking dataset for the task of 3D aware editing. \n3. **Quantitative metrics**: A number of qualitative comparisons and user studies are provided to demonstrate the geometric consistency of the edits and lighting consistency.\n", "weaknesses": "1. **Novelty**: Although the proposed dataset represents an important contribution, the proposed approach relies on zero-1-to-3 and finetuning on a new dataset.\n2. **Need for zero-1-to-3**: The approach finetunes a model on top of zero-1-to-3 to incorporate edit instructions. Can the finetuning be done on top of base SD?. Adding an ablation to this effect will be helpful to demonstrate the need for the 3 stage curriculum. \n3. **Related work**: Several related work that may provide important context are missing. The authors might find some of the following works relevant and interesting [1,2,3,4]. Although some of these works are pre-prints and do not warrant strict comparisons, incorporating them into the related work section would place the proposed work appropriately w.r.t the landscape of current literature \n4. **Changes in the edited image** : There are certain global changes in the edited image, that dilutes some of the claims w.r.t editing. Particularly, for the CLEVR dataset, the provided supplm examples show changes in the color of certain objects upon insertion/ removal\n\n\n[1] ControlNet\n[2] InstructPix2Pix\n[3] InstantBooth\n[4] GLIGEN", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to read our submission and for providing valuable feedback. We will now respond to the highlighted questions and concerns.\n\n**Novelty**\n\nThere are 3 main novel contributions in our work:\n1. We propose the task of language-guided 3D-aware image editing.\n2. Given the challenges involved in creating supervised training data for this task in the real world,  we propose a large-scale procedurally generated benchmark as a means for learning and evaluating this new task.\n3. We train a model to edit scenes in context and observe real-world transfer.\n\nIdentifying a simple approach (adapting Zero123 and finetuning on OBJECT) that works well for a challenging new task may seem obvious in hindsight, but we respectfully argue that this simplicity is a strength and a technical contribution of this work. As per the reviewer’s suggestion, the comparison of model weights presented next, also confirms the value of initializing with Zero123 as compared to finetuning from Stable Diffusion. Finally, it is not obvious how to construct a training dataset that would allow a Zero123-like model to edit real scenes. As the first work to do so, we consider this a significant and novel contribution. We leave more thorough exploration of model design for future research.\n\n**Comparison of initialization**\n\nThis is a great suggestion! We provide a study of different initialization schemes comparing model weight initialization using: (i) Stable Diffusion which was trained for text-to-image; (ii) Zero-123 trained on image-to-image novel-view synthesis on Objaverse (current initialization scheme); (iii) Zero-123 trained on a larger Objaverse-XL [1]. \n\nIn the table below, we find both Zero-123 based initializations to outperform Stable Diffusion demonstrating the benefits of Zero-123’s novel-view synthesis training by helping the model develop a better implicit understanding of 3D geometry of objects. Initializing with Zero-123 trained on the larger Objaverse-XL dataset achieves similar performance as Zero-123 because Objaverse is already a massive scale pretraining dataset and further scaling Zero-123 style novel-view synthesis pretraining has diminishing returns if any. Scaling limits of our 3D-aware editing training are yet to be explored.  \n\n|                   | Seen Objects |       |       |       | Unseen Objects |        |       |       |\n|-------------------|--------------|-------|-------|-------|----------------|--------|-------|-------|\n| Model             | PSNR         | SSIM  | LPIP  | FID   | PSNR           | SSIM   | LPIP  | FID   |\n| Task: Translation |              |       |       |       |                |        |       |       |\n| SD                | 14.373       | 0.264 | 0.51  | 0.101 | 14.351         | 0.253  | 0.51  | 0.102 |\n| Zero123           | 15.21        | 0.3   | 0.472 | 0.244 | 15.2           | 0.292  | 0.477 | 0.253 |\n| XL                | 15.121       | 0.294 | 0.477 | 0.252 | 15.052         | 0.286  | 0.478 | 0.239 |\n| Task: Rotation    |              |       |       |       |                |        |       |       |\n| SD                | 15.074       | 0.368 | 0.43  | 0.089 | 14.558         | 0.359  | 0.438 | 0.095 |\n| Zero123           | 16.859       | 0.382 | 0.429 | 0.248 | 16.279         | 0.366  | 0.447 | 0.236 |\n| XL                | 15.433       | 0.381 | 0.42  | 0.241 | 15.008         | 0.3783 | 0.429 | 0.243 |\n| Task: Insertion   |              |       |       |       |                |        |       |       |\n| SD                | 13.22        | 0.253 | 0.57  | 0.108 | 13.131         | 0.255  | 0.572 | 0.1   |\n| Zero123           | 13.63        | 0.263 | 0.551 | 0.222 | 13.088         | 0.261  | 0.568 | 0.214 |\n| XL                | 13.481       | 0.264 | 0.557 | 0.274 | 13.094         | 0.259  | 0.566 | 0.258 |\n| Task: Removal     |              |       |       |       |                |        |       |       |\n| SD                | 23.882       | 0.576 | 0.263 | 0.117 | 23.352         | 0.542  | 0.27  | 0.115 |\n| Zero123           | 24.98        | 0.585 | 0.249 | 0.236 | 24.661         | 0.568  | 0.26  | 0.24  |\n| XL                | 24.775       | 0.585 | 0.255 | 0.247 | 24.83          | 0.568  | 0.253 | 0.215 |\n\n\n**Related work**\n\nThank you for these recommendations! We already cited ControlNet, but will add the other three suggestions and adequately discuss them in the related work.  At a high-level, a key difference between these works and ours is that our work focuses not on changing the style, texture or attributes of objects in the scene but rather on enabling 3D-like object rearrangement capabilities via 2D image editing.\n\n\n**Changes in edited image**\n\nSince our method is based on Zero123 and SD, we inherit some of the flaws of these approaches, like creating global changes while targeting local ones.\n\n**Specifying object location**\n\nThey are provided in pixel space. Our reason for doing this is to support an intuitive interaction interface (e.g. on a phone) for editing images where a user can simply click on the region of the image they want to be edited. That being said, our modeling framework can easily be extended to incorporate camera coordinates or spatial maps as well if available. \n\nReferences\n[1] Objaverse-XL: A Universe of 10M+ 3D Objects\n"}, {"review_id": "TsYGfgnafX", "reviewer": "Reviewer_yCLA", "summary": "The paper formulates a task of 3D aware editing using the language guidance. The task aims to insert, remove, translate or rotate objects in a scene (2D images) by maintaining the details like shadows, 3D consistency of the object, changes in the object sizes due to perspective projections etc. The model is based on Stable diffusion, Zero 1-to-3 method and fine tuning on the given dataset having editing information and text to describe the edit. The paper promises to release the dataset OBJECT derived from Objaverse, which the authors use to train their model on.  The results in the teaser figure and others show that the model is able to perform the given edits, while preserving the semantics of the image. For examples, the objects are translated and it respects the perspective projection, shadows and placement on the surface. The authors also claim that the method is generalizable to the real images.", "questions": "Please refer to the weakness section.", "limitations": "The authors do not discuss the limitations of their work in detail. Please add a detailed section on where the method fails.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The paper is able to show that the manipulations possible with the method can preserve the 3D properties of the scene including localization of the objects, scaling, shadows and consistency of the inpainted regions. \n\n2) The paper compares with the image based baselines, for examples uses SAM to segment the images and translates the objects in the scene. It also compares with the a 3D baseline for rotation using Zero 1-to-3.\n\n3) The paper shows quantitative and qualitative results of their method comparing scores such as PSNR, SSIM, LIPIPS and FID between the original and edited images. The authors also conduct a user study to access the quality of lighting and geometry.", "weaknesses": "1) The method to train on the given dataset is not clear to me. There is no pipeline figure to explain the stages of the training. The first two steps are previous works. The contribution which is in the third step is not explained properly in the paper. Is this fine-tuning stage similar to Zero-1-to-3? How was the editing sequence fed to the network? How is it 3D aware besides the Zero-1-to-3 training? Is the method's 3D consistency (for example in rotation) upper-bounded by Zero-1-to-3?\n\n2) While the images shown in the paper show that some properties are preserved as the editing operations are done, the results in the gifs show some obvious flickering artifacts which do not respect the properties like shadows. This does not go well with the objective of the paper. Besides the problems with Stable Diffusion , where do the problems arise?\n\n3) How is the quality drop if number of sequential operations are done? For example once can perform insertion-> rotation ->translation etc for the same/ different objects in the scene. How does the quality drop compare with the Stable diffusion image editing methods? This is more interesting to me. A 3D aware editing  framework should be able to handle multiple sequential edits with consistent results.\n\n4) Another baseline would to use a monocular depth estimation model (eg Zoedepth, Midas) to extract the surface and perform the edits using Zero-1-to-3. This can handle the perspective projection of the objects, and/or even lighting and shadows. Did the authors try similar more strong baselines? How do they compare with the current method?\n\n5) The paper claims to generalize to the real scenes. This is a significant claim and needs to be evaluated. The issue of synthetic and real image domain gap is an active research area. How does the current method solve that in this particular task? Were real images considered in Table 2?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to read our submission and for providing valuable feedback. We will now respond to the highlighted questions and concerns.\n\n**Method clarifications**\n\nOur approach extends zero123 with a CLIP text encoder (the same as the one used in the original StableDiffusion). The model is finetuned starting from the zero123 checkpoint in exactly the same way as zero123. A sample in a minibatch consists of the original image, instruction, and the target image. The denoising network is a  diffusion model trained to recover the target image from a noise version of the target image conditioned on the original image and the instruction. The instruction is always a single edit from one of the four supported tasks. We do not train on sequences. Please see Figures E and F in the attached PDF for further clarification.\n\nWe clarify that 3D awareness has to do with the task definition which requires models to edit objects in complex scenes as if performing 3D transformations in the real world but while only operating in the pixel space. Zero-123 is a model that takes an image and generates novel viewpoints of the object. In doing so, Zero-123 has developed an implicit understanding of the 3D geometry of objects. However, Zero-123 does not take into account scene lighting and occlusions with other objects in the scene. Therefore, our model trained on our 3D-aware editing benchmark learns more about the geometry, lighting, and surface-contact than Zero123.\n\nWhile our current method may be bounded by the limitations of Zero123, the benefit of using this approach is that, unlike approaches that involve explicit 3D geometry, our approach can be improved with scale [1]. Extensions of our framework can develop improved datasets with more objects [1], language annotations [2], or more complex environments [3], for example. \n\n**Flickering artifacts**\n\nThe reviewer should also note that our approach is primarily geared towards editing images and not videos and hence the frames generated when creating a video visualization of the results are not guaranteed to be temporally consistent. Ensuring this temporal consistency is an active area of research [4] and would require several innovations in video modeling. Current state of the art video models still struggle with temporal consistency despite being explicitly trained for this [5]. Our newest model has improved in its ability to synthesize shadows for out of domain objects. Please see Figure I in the PDF. \n\n**Sequential operations**\n\nPerforming a sequence of edits is an interesting idea! We provide an example of performing multiple edits on the same object in the attached PDF. Please see Figure H. \n\n**Additional baseline**\n\nZero123 is an image-to-image model conditioned on a camera pose. Zero-123 doesn’t produce a 3D model nor is it conditioned on one. Therefore any 3D surface extracted with a depth estimation model would have to be converted to an image before being processed by Zero123. The baseline reported in our paper uses Zero123 to rotate an object extracted from a segmentation model and then harmonizes the resulting edit with StableDiffusion inpainting. However, in the spirit of your suggestion, to test the maximum possible performance of a method involving Zero123, we ran an additional version of this baseline where the edited object is extracted with its ground truth mask before being rotated by Zero123.  The results are reported below. \n\n|                 | Seen   |        |       |       | Unseen |       |       |       |\n|-----------------|--------|--------|-------|-------|--------|-------|-------|-------|\n|                 | PSNR   | SSIM   | LPIP  | FID   | PSNR   | SSIM  | LPIP  | FID   |\n| Ours multitask  | 16.859 | 0.382  | 0.429 | 0.248 | 16.293 | 0.372 | 0.445 | 0.28  |\n| Oracle Baseline | 15.613 | 0.3845 | 0.422 | 0.248 | 14.552 | 0.366 | 0.439 | 0.001 |\n\n**Real-world evaluation**\n\nTo address this concern, we have conducted a human evaluation of real world images. We compare our latest model to the baselines and also compare our latest model to the original one.  Please see the tables below.\n\n| Task        | Geometric consistency |             |     | Lighting consistency |             |     |\n|-------------|-----------------------|-------------|-----|----------------------|-------------|-----|\n|             | Baseline              | 3DIT (Ours) | Tie | Baseline             | 3DIT (Ours) | Tie |\n| Translation | 25%                   | 60%         | 15% | 15%                  | 70%         | 15% |\n| Rotation    | 10%                   | 80%         | 10% | 10%                  | 80%         | 10% |\n| Insertion   | 35%                   | 55%         | 10% | 35%                  | 55%         | 10% |\n| Removal     | 10%                   | 75%         | 15% | 5%                   | 80%         | 15% |\n\n| Task        | Geometric consistency |           |     | Lighting consistency |           |     |\n|-------------|-----------------------|-----------|-----|----------------------|-----------|-----|\n|             | Old model             | New model | Tie | Old model            | New model | Tie |\n| Translation | 15%                   | 60%       | 25% | 15%                  | 65%       | 20% |\n| Rotation    | 30%                   | 55%       | 15% | 15%                  | 60%       | 25% |\n| Insertion   | 30%                   | 60%       | 10% | 20%                  | 50%       | 30% |\n| Removal     | 10%                   | 60%       | 30% | 10%                  | 60%       | 30% |\n\nReferences\n[1] Objaverse-XL: A Universe of 10M+ 3D Objects\n[2] Scalable 3D Captioning with Pretrained Models\n[3] Habitat: A Platform for Embodied AI Research\n[4] TokenFlow: Consistent Diffusion Features for Consistent Video Editing\n[5] VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation \n\n\n"}, {"review_id": "aW0LA37HiY", "reviewer": "Reviewer_eZ2Z", "summary": "This paper constructs a dataset containing 400K examples, which is used for the task of language-guided 3D-aware image editing. This paper also proposes a model, named 3DIT, to solve this task. The model is based on 2D diffusion model, which first goes through the pre-training of text-to-image generation and Zero-1-to-3, and then is fine-tuned on the dataset for the 3D-aware image editing task. The model is evaluated on the proposed dataset and achieves state-of-the-art performance.", "questions": "Please see the weakness section.", "limitations": "The authors have discussed the limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper constructs a large-scale dataset, which is a good contribution. And language-guided 3D-aware image editing is an important task.\n2. The paper is well written and easy to understand.\n3. The experiments show that the proposed model trained on the proposed dataset have a reasonable ability on the task of language-guided 3D-aware image editing.", "weaknesses": "My biggest concern is that the performance of the proposed model is not very good.\n\n1. The GIF results in the supplementary material exhibit incorrect results on the shadow.\n2. The lighting and shadow of experimental results on real-data is not realistic. The quality of edited images degrade a lot.\n\nDue to the poor performance, I am not sure if the quality and diversity of the proposed dataset is good enough.\n\nThe illumination and background of presented real data is too simple. It would be great to add more challenging test data to see the upper bound of the proposed model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to read our submission and for providing valuable feedback. We will now respond to the highlighted questions and concerns.\n\n**Incorrect shadow behavior in GIF, dataset realism and diversity**\n\nThe original dataset at the time of submission used a single directional light source leading to unrealistic lighting in our training dataset. Therefore, learning to model shadows on OBJECT dataset showed limited transfer to CLEVR. As discussed in the common statement, we have improved the realism of the dataset. Please see Figure B for a visualization of the lighting improvements. Specially, with our new 3-point and environmental lighting, the scenes are now lit in a much more realistic manner leading to noticeably better shadow rendering behavior in our models. See the Figures A and I in the pdf for comparison between models trained on old and new datasets.\n\n**Difficulty of real data**\n\nFigure G in the attached PDF shows the model editing more complex real world scenes, as well as some failure cases. For example, when removing the can, we see some noticeable infilling artifacts. Our model often struggles with scenes of higher complexity in the real world. We also conducted an analysis of the model's failure modes and report 4 major categories: incorrect texture, incorrect localization, unintended global modifications and incorrect geometry. Figure C in the attached PDF shows the frequency of each error type among a sample of errors annotated from our test set. Figure D shows a visualization of each error category.\n"}, {"review_id": "HPLdCNPbVI", "reviewer": "Reviewer_SoaB", "summary": "The paper studies the problem of object-centric image editing. The authors first curate a dataset based on Objaverse by selecting high-quality textured samples, and then simulate+render them on a plane. The objects can be manipulated in 3D and rendered correspondingly, which generates the groundtruth for training learning-based object editing models. The paper further presents a diffusion-based object editing model (3DIT) based on zero-123, where the major difference is the addition of editing prompt conditioning. Results in quantitative and qualitative experiments show 3DIT outperforms baselines based on foundation models.", "questions": "Please see my questions in the weaknesses. Overall, I think the pros of this paper outweigh the cons -- the curated dataset would be useful for future exploration along this direction and the proposed method can serve as a strong 2D baseline. I believe having sim2real comparison and systematic failure case analysis will further solidify the paper.", "limitations": "More analysis on failure cases and limitations will be beneficial.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The dataset curation using 3D simulation for the object-centric image editing task makes sense. This task intrinsically requires understanding of the 3D world and 3D-aware image formation process. The proposed dataset is guaranteed to be 3D-correct, and would be useful for research along this direction. The proposed method also achieved great performance compared to the baselines. Besides, the paper is well-written and easy to follow.", "weaknesses": "1. The realism of the generated dataset is still limited: \na) The single directional light (why not multiple lights or Image-based lighting?) which makes the shadow and shading's distribution not diverse/realistic enough.\nb) The size range of the object is quite limited (0.8 ratio threshold), in practice, there are a lot of cases where different sized objects are placed nearby (also the true size seems not kept, so chairs and lamps appears to be of similar scale).\nThe authors indeed show some qualitative results of direct transfer to real data, but no 1) quantitative evaluation of sim2real or 2) comparison to baselines' generalization have been provided.\n\n2. Although the data is generated with 3D, the method follows a 2D design. Such a model does not exhibit a great understanding of 3D objectness -- for example, the identity of the object is sometimes not well-kept during rotation or translation (e.g. the change of coke/headphone texture in Fig 1), or the cast shadow or the shading do not make full sense w.r.t. the whole scene if we look close enough (e.g. first 3 rows of clevr in Fig 1). Having some systematic analysis of the failure cases here would have been very helpful for future research.\n\nMinor questions/comments:\n1. Why is the background always black? This seems to be creating domain gap, as usually we don't have pure black background in real life.\n2. Wrong number highlight in Table 1, 3DIT(Multitask) - insertion - LPIP, 0.585 is worse than the other two.\n3. The multi-task model is much worse in terms of FID (Table 1), but not other metrics. Are there any specific reason for this?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to read our submission and for providing valuable feedback. We will now respond to the highlighted questions and concerns.\n\n**Realism of dataset**\n\nAs per reviewer suggestions, we have improved the realism of our dataset in 3 ways: (i) In line with common practices in professional film and photography, we implemented a 3-point lighting system that automatically aligns itself with the viewpoint of the camera, thereby shading objects in a way that better reveals their true 3D form; (ii) We added real-world environmental lighting into our dataset with 360-degree HDRI captures from both indoor and outdoor scenes under a variety of lighting conditions. Not only does this give scenes in our dataset realistic backgrounds, but the light emanated from these captures is integrated into the ray-tracing process during rendering, so that all aspects of the scene benefit from a more realistic lighting distribution; and (iii) We replaced the ground textures in our dataset with more realistic ones that have normal, roughness and displacement maps. Please see figure B in the dataset for a visualization of the improvements.\n\nWe also retrained our models on this more realistic dataset and find that the in-domain performance follows similar trends as reported in the paper, but the models generalize better to out-of-domain data like CLEVR and the real world images. We will update the draft with all recomputed metrics for our new models on this improved dataset but include PSNR here for brevity.\n\n|                  | PSNR (seen objects) | PSNR (unseen objects) |\n|------------------|---------------------|-----------------------|\n| Translation      |                     |                       |\n| Baseline         | 13.699              | 14.126                |\n| 3DIT (1-task)    | 14.546              | 14.4                  |\n| 3DIT (Multitask) | 15.21               | 15.2                  |\n| Rotation         |                     |                       |\n| Baseline         | 13.179              | 12.848                |\n| 3DIT (1-task)    | 16.828              | 16.293                |\n| 3DIT (Multitask) | 16.859              | 16.279                |\n| Insertion        |                     |                       |\n| Baseline         | 12.297              | 12.542                |\n| 3DIT (1-task)    | 13.469              | 12.974                |\n| 3DIT (Multitask) | 13.63               | 13.088                |\n| Removal          |                     |                       |\n| Baseline         | 12.494              | 12.123                |\n| 3DIT (1-task)    | 24.937              | 24.474                |\n| 3DIT (Multitask) | 24.98               | 24.661                |\n\n**2D design**\n\nOur goal is to see if it is possible to perform 3D-aware edits while working solely in the pixel space. This hypothesis is motivated by the success of approaches like Zero-123 which generate plausible novel views of a single object given only one image. Importantly, by using a large pretrained image generator, Zero123 outperforms comparable methods that use explicit 3D geometry. Our results further provide evidence to support that performing 3D-aware edits while solely operating in 2D is possible.\n\n**Failure cases analysis**\n\nWe conducted an analysis of the model's failure modes and report 4 major categories: incorrect texture, incorrect localization, unintended global modifications and incorrect geometry. Figure C in the attached PDF shows the frequency of each error type among a sample of errors annotated from our test set. Figure D shows a visualization of each error category.\n\n**Human eval**\n\nWe also provide quantitative human-evaluation on real images. We find that human evaluators overwhelmingly prefer results from our method over baselines. Among all tasks, insertion is the most challenging for our model. \n\n| Task        | Geometric consistency |             |     | Lighting consistency |             |     |\n|-------------|-----------------------|-------------|-----|----------------------|-------------|-----|\n|             | Baseline              | 3DIT (Ours) | Tie | Baseline             | 3DIT (Ours) | Tie |\n| Translation | 25%                   | 60%         | 15% | 15%                  | 70%         | 15% |\n| Rotation    | 10%                   | 80%         | 10% | 10%                  | 80%         | 10% |\n| Insertion   | 35%                   | 55%         | 10% | 35%                  | 55%         | 10% |\n| Removal     | 10%                   | 75%         | 15% | 5%                   | 80%         | 15% |\n\nBelow we compare the model trained on the new improved dataset and the old dataset. Training on the newer more realistic dataset leads to stronger models. \n\n| Task        | Geometric consistency |           |     | Lighting consistency |           |     |\n|-------------|-----------------------|-----------|-----|----------------------|-----------|-----|\n|             | Old model             | New model | Tie | Old model            | New model | Tie |\n| Translation | 15%                   | 60%       | 25% | 15%                  | 65%       | 20% |\n| Rotation    | 30%                   | 55%       | 15% | 15%                  | 60%       | 25% |\n| Insertion   | 30%                   | 60%       | 10% | 20%                  | 50%       | 30% |\n| Removal     | 10%                   | 60%       | 30% | 10%                  | 60%       | 30% |\n"}]

Author rebuttal:
# Common statement

We are encouraged by all the positive comments and thank all of the reviewers for their valuable feedback. Reviewers found our model to be “a novel approach to language guided 3D-aware image editing” (Reviewer 1T99), “the manipulations possible with the method can preserve the 3D properties of the scene” (Reviewer yCLA), and that our dataset is “guaranteed to be 3D-correct, and would be useful for research” (Reviewer SoaB), among other positive comments which we are grateful for.  In this section, we will address the questions and concerns that were shared in common by multiple reviewers. 

**Reviewers eZ2Z and SoaB expressed concerns that the real-world performance of our model may be limited due to the realism of the dataset.** 

First, we note that in spite of the less realistic training data, we were surprised by the generalization of our models to real images. This is a significant finding that establishes training on simulated 3D-aware image editing examples as a promising direction for this challenging object-centric editing task. 

Second, in line with reviewers suggestions, we have improved the realism of our dataset in 3 ways: (i) As per common practices in professional film and photography, we implemented a 3-point lighting system that automatically aligns itself with the viewpoint of the camera, thereby shading objects in a way that better reveals their true 3D form; (ii) We added real-world environmental lighting into our dataset with 360-degree HDRI captures from both indoor and outdoor scenes under a variety of lighting conditions. Not only does this give scenes in our dataset realistic backgrounds, but the light emanated from these captures is integrated into the ray-tracing process during rendering, so that all aspects of the scene benefit from a more realistic lighting distribution; and (iii) We replaced the ground textures in our dataset with more realistic ones that have normal, roughness and displacement maps. Please see Figures A, B, and D in the attached pdf for examples from the new dataset.

Third, we retrained our models on this more realistic dataset and find that the in-domain performance follows similar trends as reported in the paper, but the models generalize better to out-of-domain data like CLEVR and the real world images. For results, please see a table comparing human evaluation on the old and new model in response to Reviewer yCLA. Please also see figures A, G, H and I for qualitative results. We will update the draft with all recomputed metrics for our new models on this improved dataset. For brevity, we report a table with PSNR metrics which can be found in the response to Reviewer 1T99.


**Reviewers 1T99, Rpi6, yCLA, eZ2Z and SoaB raised concerns about the limitations of our method.** The reviewers correctly identified some of the failure modes of our approach which we address below:
 1. **Unintended global changes / artifacts in GIF:** Like other SOTA diffusion based editing / inpainting models, our model may introduce minor artifacts outside the region of interest specially for out-of-distribution images such as CLEVR and real images. These artifacts are often difficult to notice in isolated edited images but become clear when making a sequence of edits and compiling them into a video or GIF as pointed out by Reviewers 1T99, yCLA, Rpi6, and eZ2Z. Techniques for improving temporal consistency in video-editing literature might be applicable, but our current benchmark and method focuses on single atomic edits as an important first step. 

 2. **Inaccurate lighting and shadows:** The original dataset at the time of submission used a single directional light source leading to less realistic lighting in our training dataset. With the improvements to the training data, the scenes are now lit in a much more realistic manner leading to noticeably better shadow rendering behavior in our models. Please see the table in the response section of Reviewer yCLA for a human evaluated comparison between the new and old models. We show frames of the GIFs from our old and new model for comparison in Figure I. 

However, we emphasize that our main contribution is to introduce the challenging and novel task of object-centric 3D-aware image editing and to create a benchmark for training and evaluation. Our benchmark and model will be useful for future research in this area.

## Additional experiment requests

**Initialization Ablation.** In response to Reviewers 1T99 and Rpi6’s suggestions for model ablations, we provide a study of different initialization schemes comparing model weight initialization using: (i) Stable Diffusion which was trained for text-to-image; (ii) Zero-123 trained on image-to-image novel-view synthesis on Objaverse (current initialization scheme); (iii) Zero-123 trained on a larger Objaverse-XL [1]. 

In the table listed in the response to Reviewer 1T99, we find both Zero-123 based initializations to outperform Stable Diffusion demonstrating the benefits of Zero-123’s novel-view synthesis training by helping the model develop a better implicit understanding of 3D geometry of objects. Initializing with Zero-123 trained on the larger Objaverse-XL dataset achieves similar performance as Zero-123 because Objaverse is already a massive scale pretraining dataset and further scaling Zero-123 style novel-view synthesis pretraining has diminishing returns if any. Scaling limits of our 3D-aware editing training are yet to be explored.

**Human-Evaluation on Real Images.** We also provide quantitative human-evaluation on real images as requested by Reviewers yCLA and SoaB. We find that human evaluators overwhelmingly prefer results from our method over baselines. Among all tasks, insertion is the most challenging for our model. Please see the tables in the section of Reviewer yCLA.

References
[1] Objaverse-XL: A Universe of 10M+ 3D Objects


Meta-review: Evenly the scores are not that high, all reviewers have consistent opinion and stand on the positive side. Although the performance is limited, all reviewers and the AC agree on the value of the proposed dataset.
Decision: Accept (poster)