SlowGuess/NeurIPS_2023_OCR huggingface dataset:
OBJECT 3DIT:
Language-guided 3D-aware Image Editing
Oscar Michel
Ranjay Krishna<sup>1,3</sup>
Anand Bhattad
Aniruddha Kembhavi
Eli VanderBilt
Tanmay Gupta<sup>1</sup>
$^{1}$ Allen Institute for Artificial Intelligence,  $^{2}$ University of Illinois Urbana-Champaign,  $^{3}$ University of Washington
Figure 1: We present 3DIT, a model to edit individual objects in the context of a rich scene with language conditioning. 3DIT is able to effectively edit objects while considering their scale and viewpoint, is able to add, remove and edit shadows to be consistent with the scene lighting and is able to account for object occlusions. Training on our new benchmark OBJECT, 3DIT remarkably generalizes to images in the CLEVR dataset as well as the real world.
Existing image editing tools, while powerful, typically disregard the underlying 3D geometry from which the image is projected. As a result, edits made using these tools may become detached from the geometry and lighting conditions that are at the foundation of the image formation process. In this work, we formulate the new task of language-guided 3D-aware editing, where objects in an image should be edited according to a language instruction in context of the underlying 3D scene. To promote progress towards this goal, we release OBJECT: a dataset consisting of 400K editing examples created from procedurally generated 3D scenes. Each example consists of an input image, editing instruction in language, and the edited image. We also introduce 3DIT: single and multi-task models for four editing tasks. Our models show impressive abilities to understand the 3D composition
of entire scenes, factoring in surrounding objects, surfaces, lighting conditions, shadows, and physically-plausible object configurations. Surprisingly, training on only synthetic scenes from OBJECT, editing capabilities of 3DIT generalize to real-world images. More information can be found on the project page at https://prior.allenai.org/projects/object-edit.
In today's visually-oriented society, the art of image editing has become an indispensable necessity. With the proliferation of camera phones and influences from social media platforms, amateur photographers want to transform ordinary snapshots into visual masterpieces. Unfortunately, the process of image editing is still in its infancy. Professional tools such as Photoshop allow pixel-level edits that can adjust lighting, insert objects, remove clutter, and introduce new shadows; however, these tools, with their steep learning curves are often daunting for novices. With the hopes of pulling image editors out from the minutiae of painstaking pixel-level edits, generative models have been heralded as a promise for object-level edits [57, 58, 31, 52].
Unfortunately, object-centric editing—translating or rotating an object while preserving the 3D geometry of the original photograph—is out of reach for generative models [29, 25, 73, 47, 70]. Although recent strides can take a segmented object and rotate and translate it, they typically operate on objects in isolation and often disregard any scene and lighting context [47, 54, 70]. Others require multiple viewpoints to reconstruct an object in 3D [6, 25, 73]. There is a need for models that can edit objects from a single image while preserving the structure of 3D objects and re-render shadows for the edited scene with the original lighting conditions.
To enable 3D-aware editing of objects in an image, we introduce OBJECT, Objaverse Editing in ContextT, a large-scale benchmark to train and evaluate language-conditioned models that edit objects in images. We develop OBJECT by combining Objaverse [14], a recent 3D asset library, and Blender [12], a 3D rendering engine. OBJECT contains 400k editing examples derived from procedurally generated 3D scenes. Scenes consist of up to four objects, chosen from 59k unique objects, placed on a flat textured surface with an environment lighting map, a three-point lighting system that moves with the camera, and a directional light. As shown in Figure 1, we support four types of object edits: (a) translation across the surface; (b) rotating around the axis orthogonal to the surface; (c) inserting new objects; and (d) removing existing ones. Our 3D rendering engine ensures that all edits are physically plausible and the generated images capture realistic changes in 3D geometry, illumination, and shading resulting from the underlying edit. For instance, rotation and translation require maintaining contact with the surface; inserting new objects requires identifying stable supported poses for new objects; and removing objects often requires rendering occluded objects. Each image contains a language instruction describing one of the four edits and a resulting ground truth edited image. Edited images are evaluated using quantitative metrics that capture realism and faithfulness to the ground truth.
We also introduce 3DIT (3D-aware Diffusion Image-editing with Text), a model which supports each of the four manipulation tasks with language conditioning. 3DIT is initialized with the Zero-1-to-3 [47] diffusion model (which was trained to perform novel view synthesis) and finetuned on the OBJECT dataset for object-centric image editing. The resultant model has effectively been obtained using a three-stage learning curriculum, starting with massive stable diffusion's web-scale pre-training on image-text pairs, followed by Zero-1-to-3's pre-training stage to enhance the model's understanding of 3D objects, and finally with fine-tuning on OBJECT to enable object-centric edits.
On OBJECT's test images, 3DIT outperforms baselines across all four tasks on metrics that capture the faithfulness of the scene edit. Given the known limitations of automatic quantitative metrics, we also provide a human evaluation study, where 3DIT's outputs are preferred to the baselines over  $70\%$  of the time. Edits produced by 3DIT tend to preserve the original scene's structure and not just the edited object. 3DIT preserves the scale and viewpoint of objects, it removes and adds appropriate shadows wherever necessary, and even infills previously occluded portions of the image when the occluder is translated or removed. A multi-task variant of 3DIT performs well despite having to support all four transformations using a single set of parameters. Finally, 3DIT generalizes surprisingly well to new image domains such as CLEVR, a popular synthetic dataset for visual
reasoning, as well as real-world images (see Figure 1). This highlights 3DIT's remarkable capability given that OBJECT is a synthetic, procedurally generated dataset.
Image editing with generative models: The goals of editing objects and semantic regions in images with language have been active for over a decade [45]. Back then, productizable edits were limited to simple changes like cropping, colorization and resizing to complex procedures such as object removal, addition, and rearrangement [59, 30, 18, 46, 16, 85, 22, 4, 71, 53]. Traditionally, these tasks were performed manually using tools like Adobe Photoshop. However, the origin of Generative Adversarial Networks (GANs) [23] revolutionized the field, propelling significant strides toward automation. StyleGAN [36, 37, 35] notably facilitated intricate modifications to the synthesized images, paving the way for sophisticated GAN-based editing techniques with greater control and flexibility [66, 84, 11, 60, 10, 5, 1, 67]. Since then, advancements in generative image architectures have been marked by the emergence of diffusion models [17]. When coupled with the availability of large-scale image-text datasets [64], these models have facilitated the generation of high-fidelity, diverse scenes [51, 58, 62, 63, 34]. Concurrent with these developments, a new wave of image editing methodologies utilizing these large-scale diffusion models have been introduced [83, 50, 38, 29, 43, 19, 72]. Despite these advancements, models lack the 3D awareness necessary for maintaining geometric and lighting consistency. Our dataset, OBJECT, aims to bridge this gap by enhancing existing methods and serving to evaluate future methodologies.
3D image editing: A host of recent research, including StyleNeRF [25], ViewGen [3], EG3D [6], SJC [73], DreamFusion [54], Zero-1-to-3 [47], and Make-It-3D [70], has explored lifting 2D images to 3D. By contrast, our model—3DIT—comprehensively considers the entire scene, not just the object of interest, encompassing geometry, lighting, and other salient attributes of the background.
Scene rearrangement: Current research in scene rearrangement tasks primarily involve solving rearrangement from robotic manipulation and embodied agents [39, 55, 56, 48] to provide more intuitive and human-like commands for scene manipulation and navigation. Specific attempts have also been made to apply these techniques to room rearrangements [74, 79, 78] using datasets like AI2-THOR [41], Habitat [69], Gibson [80], 3D-FRONT [20] or fine-tuning diffusion models with convex decomposition [72]. For instance, LegoNet [78] focuses on room rearrangements without the need to specify the goal state, learning arrangements that satisfy human criteria from professionally arranged datasets provided by 3D-FRONT [20]. Distinct from these works, our research introduces a unique perspective. We focus on object-level rearrangements with a primary emphasis on 3D-aware image editing using language instructions. 3DIT is trained with OBJECT to edit scenes with a high degree of realism and 3D coherence.
3D asset datasets: A diverse set of 3D asset dataset such as ShapeNet [7] and the recent Objaverse [14] have played a pivotal role in 3D computer vision. ShapeNet provides a richly-annotated, large-scale dataset of 3D shapes that has found numerous applications in object recognition, scene understanding, and 3D reconstruction. Objaverse has offered a large collection of 3D objects that are semantically segmented and paired with natural language descriptions. Objaverse has been instrumental in the construction of OBJECT and also advancing several other related research areas, including generating textured meshes [8, 21, 27] zero-shot single image 3D generation [47] and enriching simulators [42, 15] for Embodied AI.
Synthetic datasets for vision models: Diagnostic datasets such as CLEVR [33] and CLEV-ERER [81] provide a rigorous test bed for the visual reasoning abilities of models. They contain synthetically generated images of 3D scenes with simple primitives and associated questions that require an understanding of the scene's objects, attributes, and relations to answer correctly. Kubric [24] is an image and video dataset generation engine that can model physical interactions between objects. In a similar vein, OBJECT offers procedurally generated scenes of commonly occurring natural objects derived from ObjVerse [14] with configurable 3D objects and associated language instructions.
Benchmarks for image editing: There is currently a scarcity of benchmarks to evaluate generative models [32], especially for 3D scene editing. Existing ones, including light probes [76], repopulating street scenes [77], GeoSim [9] and CADSim [75] are not publicly available. Our presented OBJECT benchmark will be made publicly available.
3 OBJECT: A benchmark for Object Editing in Context
Our goal is to design and evaluate image editing models capable of editing objects in scenes. To enable training and evaluation of such models, we develop OBJECT. OBJECT contains scenes with multiple objects placed on a flat textured surface and illuminated with realistic lighting. These edits are described to the model using a combination of language and numerical values (e.g. pixel coordinates and object rotation angle). All edits result in structural changes to the scene which in turn affect illumination changes such as inter-object reflections and shadows. The model does not have access to the underlying 3D scene (including object segmentations, locations, 3D structure, and lighting direction); it must infer these from the input pixels.
3.1 Object editing tasks
OBJECT supports four fundamental object editing tasks: Each of the following manipulations targets a single object within a scene that may contain multiple objects. We now describe each task and the capabilities required from an image editing model to succeed at the task. For specifying locations in an image, we use a coordinate system where  $(0,0)$  represents the bottom-left corner and  $(1,1)$  the top-right corner. Objects are specified in each task using their crowdsourced descriptions.
Translation: Given the x-y coordinates of a target location, a specified object is moved from its original location in the scene to the target location while preserving its angular pose and surface contact. Since the camera is fixed relative to the scene, a change in object location requires to model to synthesize newly visible portions of the object. The model is required to change the object's scale in the image due to perspective projection i.e. the objects should appear smaller when moved further away from the camera and vice-versa. The new location may also result in drastically different illumination of the object.
Rotation: A specified object is rotated counterclockwise around the vertical axis passing through the object's center of mass and perpendicular to the ground by a given angle. To succeed, the model must localize the object, extrapolate the object's shape from a single viewpoint, and re-imagine the scene with the rotated object. Rotating objects leads to intricate changes to the shadow projected on the ground plane which are challenging to accurately produce.
Insertion: Given a language description, an object matching the description is added to the scene at a designated x-y location. The model must perform object generation at the desired location with stable pose and surface contact. Besides modeling the object shape, the model also needs to understand the interaction of the geometry with scene lighting to generate a realistic shadow for the object.
Removal: A specified object is removed from the scene. The model must not only be able to locate and segment the object, but also in-paint the object region using scene context. This often requires inpainting an object that was previously partially or fully occluded.
Figure 2: Scene generation in OBJECT depicting camera constraints, directional lighting (environment and three-point lighting not shown), and the resulting object shadows.
3.2 Benchmark curation
Paired image- & text data is plentiful on the internet and large corpora are commonly used to train text-to-image models. However, there is a lack of image editing data consisting of initial and edited image pairs, with a description of the edit. Gathering such a dataset at scale from the real world requires significant manipulation and annotation effort. Our key insight is that while object manipulation data is difficult to acquire, it is much easier to synthesize large volumes of this data leveraging the latest advances in photorealistic rendering and large 3D asset libraries. Therefore, OBJECT contains procedurally generated 3D scenes rendered with objects from these asset libraries.
Object source. OBJECT scenes are constructed using one to four 3D objects from the Objaverse dataset[14]. The entire Objaverse dataset contains more than  $800\mathrm{k}$  assets. Since objaverse contains objects with errors (some objects are not fully rendered or contain no texture), we filter the objects down to a set of  $59\mathrm{k}$  via a combination of Sketchfab metadata-based filtering and crowdsourcing. The resulting objects all have various textures, are easily recognizable, are of high quality and resolution, are free of copyrighted material, are in isolation (as opposed to a single asset with multiple objects), and are free floating (so that they may be placed on any surface in the generated scenes).
Each of these assets is annotated with one of 1613 unique semantic categories using crowdsourcing. Workers were shown a rotating 3D rendering of a particular object and asked to apply a category label; they were provided with a handy autocomplete list of roughly 1400 categories sourced from LVIS [26]categories. If, however, workers were unable to find an appropriate category, they had the option of generating a new category. After this they were asked to write a sentence that describes the object, pointing out any interesting or noteworthy details that would distinguish it from other objects in the same category. Finally, category names were cleaned up to remove spelling errors; we removed unusual or rare categories.
We randomly choose 1513 categories to be seen during training while holding out the remaining 100 as unseen categories for validation and testing. This category split helps quantify the generalization gap in editing previously seen vs novel objects. We use a library of 17 texture maps obtained from [2] to simulate wooden, cobblestone, and brick flooring for the scenes.
Scene construction. We limit all scenes to a minimum of one and a maximum of four objects. To identify a natural resting pose for these objects, we perform a physical simulation in Blender where we drop each object onto an XY ground plane and record its resting pose. Then to identify object placements, we sample a bounding box of the same x-y aspect ratio as the object and uniformly scale the object to lie in this bounding box. We ensure that objects, when rotated, do not intersect each other: bounding boxes who's circumscribed circles intersect are rejected. To avoid tiny objects being placed in the same scene as very large objects, we enforce the ratio between the smallest and longest largest side of each bounding box to be greater than 0.8. We randomly place the camera in the upper hemisphere surrounding the plane and point it towards the origin which lies on the ground plane. We further constrain the camera elevation angle from the ground between  $40^{\circ}$  to  $80^{\circ}$  to ensure that the viewing angle is neither too close to the ground nor completely vertical which are both relatively unnatural. In each scene, there is a designated object that is manipulated. If this object is not visible from the camera, we move the camera away from the origin until the object is visible both before and after the manipulation.
Scene lighting. We use several light sources to realistically illuminate the scene. First, we add a random environment lighting map, which are special images that capture the light in a real-world scene from all directions, giving the impression that our constructed scenes are imbedded in various indoor and outdoor locations in the real world. We download 18 of these environment maps with CC0 licences from https://polyhaven.com/. Next, we add a three-point lighting system that automatically adapts to the camera view. This involves placing the key light for primary illumination, the fill light to soften key light shadows, and the back light to distinguish the subject from the background. These lights serve to effectively shade the objects in the front of the camera so that their 3D form is apparent. Finally, the scene is illuminated with directional lighting with the direction randomly sampled within a conical neighborhood around the negative-z direction to simulate an overhead light source. This consists of parallel rays emitted by a single light source infinitely far away and therefore can be specified by intensity and direction without specifying a source position.
We generate 100k training examples for each task, and 1024 scenes for validation and testing. The 3D scenes are automatically generated using Blender and its Cycles ray tracer for rendering each scene. We also render segmentation masks that denote object instances, plane and background pixels for all scenes.
4 3DIT: a scene-aware editing model
Task setup. Consider a 3D scene,  $S$ , filled with multiple objects. Let  $x_{1} \in \mathbb{R}^{H \times W \times 3}$  represent an image of this scene produced by a rendering function  $f$ . Let  $l$  represent the text description of the edit, and  $v$  represent the task-specific numeral values (i.e. angle for the rotation task and x,y coordinates for removal, insertion, and translation) to describe the desired edit to the scene  $S$ . In
this paper, we consider object-centric manipulations including rotating, translating, inserting, and removing objects. Manipulating the objects in  $S$  can yield a new image  $x_{2} = f(M(S,l,v))$ , where  $M$  applied the transformation  $l, v$  in 3D.
Our goal is to produce  $x_{2}$  without access to the 3D scene  $S$  and instead, directly editing the source image  $x_{1}$ . Importantly, we have no explicit information about the scene (including scene geometry and layout), no explicit information about the lighting (such as its location and intensity), and no access to the camera parameters. All this information must be implicitly inferred from the single source image  $x_{1}$ . Concretely, we wish to produce the target image  $x_{2} = \hat{f}_{\theta}(x_{1}, l, v)$ , where  $\hat{f}$  is a learned function with parameters  $\theta$ .
Background. Diffusion models [61] have recently shown spectacular results in generating images conditioned on text descriptions. These models consist of an encoder  $\mathcal{E}$  that maps an image  $x$  into a latent code  $z = \mathcal{E}(x)$ , a decoder,  $\mathcal{D}$  that can map a latent code back to image space, and a U-Net  $\epsilon_{\theta}$  with learned parameters  $\theta$  used for denoising. Some diffusion models are trained on large training corpora such as LAION-5B [65] and are able to produce high-quality high-resolution images that faithfully represent input text descriptions. The recently proposed Zero-1-to-3 model[47] finetunes image-conditioned Stable Diffusion[44] on the task of generating an image of a single object from a novel viewpoint, conditioned on an input view and a relative camera transformation.
3DIT. Our model, 3DIT, builds upon Zero-1-to-3. We design  $\hat{f}_{\theta}(\cdot)$  using the same base architecture but make changes to its conditioning module  $c_{\theta}(\cdot)$ . Our changes enable the conditioning module to accept edit instructions in the form of language and location information to precisely define the desired edit. In the cross-attention conditional module, Zero-1-to-3 uses a CLIP image encoding to represent the initial image, followed by concatenating a four-dimensional vector encoding camera pose information. This 772-dimensional vector gets passed through a multi-layered perceptron (MLP) to map it back down to a size of 768 dimensions. Similarly, we encode the source image  $x_{1}$  using the same CLIP image encoder. We encode  $v$  and concatenated the vector with the image representation and feed it into the MLP. Next, we append the MLP outputs with edit text tokens  $l$ , which are extracted using CLIP's text encoder.
We finetune our model from the 16,500-step checkpoint of Zero-1-to-3. During training, the network takes a noised latent encoding of  $z_{t}$ , timestep  $t$  and conditioning information  $c(x_{1},l,v)$ , where  $z_{t}$  is the latent representation of the target image at time step  $t$ . and produces a denoising score estimate  $\epsilon_{\theta}(z_{t},t,c(x_{1},l,v))$  where  $c(\cdot)\in \mathbb{R}^{768\times N}$  outputs a sequence of conditional embedding vectors. We finetune the network with the standard diffusion loss [31, 62]:
$$
\min _ {\theta} \mathbb {E} _ {z \sim \mathcal {E} _ {\theta} (x _ {1}), t, \epsilon \sim \mathcal {N} (0, 1)} | | \epsilon - \epsilon_ {\theta} (z _ {t}, t, c (x _ {1}, l, v)) | |.
$$
We now present experiments to evaluate our 3DIT model. First, we evaluate single task variants of 3DIT, i.e. one model for each of the four tasks - object rotation, translation, insertion and removal. For each of these tasks, we evaluate the performance of the model on novel scenes with objects seen at training time, and with objects unseen at training time. We also provide evaluations for a multi-task model - trained to perform all four tasks.