SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For each of the four tasks, we create strong baselines inspired by recent approaches like VisProg [28] and Socratic models [82] that chain multiple foundation models together to create performant systems for various tasks including image editing.
Removal: We first use SAM [40] in the generation mode to get candidate masks for the entire scene and select the mask that contains the point and occupies no more than a third of the area of the entire image. If no such mask is found, we attempt to get a mask by directly using the point as input to SAM to get a mask. Then, we use Stable Diffusion (SD) to inpaint the masked region using the prompt "a rendering of an uncluttered textured floor with no objects".
Insertion: This baseline uses SD and the target location to re-imagine the scene with an object of the provided category. The final image is generated by using the prompt "a 3D rendering of category on
Figure 3: Generated examples from 3DIT as well as baselines for each of the four tasks in the OBJECT benchmark.
a textured floor" conditioned on the initial image and a fixed-size  $(200 \times 200)$  square mask around the target location.
Translation: Translation requires localizing the object given the category name, removing it from the initial location, and inserting it at the target location. We use OWL-ViT [49] to localize the object given the category name. The detected bounding box is fed into SAM to generate the object segmentation mask which is then used for inpainting similar to the Removal baseline. Finally, the segmented object is composited at the target location.
Rotation: Here we use Zero-1-to-3 [47] as a baseline which requires the object to be tightly centered in the image with a white background. So, we first localize the object using OWL-ViT, crop the localized region, and segment it using SAM to create the appropriate input for Zero-1-to-3 for performing the rotation. The rotated object is composited back onto the image and the remaining unfilled regions are inpainted using SD.
We follow Zero-1-to-3 and use four metrics to automatically evaluate the quality and accuracy of the edited image - PSNR, SSIM, LPIPS, and FID. The first 3 directly compare the prediction to the ground truth image, while FID measures the similarity between the predicted and ground truth sets of images. Instead of computing the metrics for the whole image, we focus on the region where the edits are targeted. To do this, we simply use the ground truth segmentation mask to crop the targeted rectangular region of interest prior to computing the metrics. Since our model, as well as our baselines, can generate multiple solutions for each input, our evaluation considers the best-of-four prediction as per the SSIM metric to compute the final scores for all metrics. This considers the
Table 1: Quantitative evaluation using generated samples. For each method, four samples per test image were generated. The best image according to the PSNR metric is selected to represent each sample, and these values are averaged across samples. To ensure that the metrics focus on the transformed object and not the background which mostly remains unchanged, metrics are computed using the region around the transformed object's mask.
Model	Seen Objects	Unseen Objects
PSNR ↑	SSIM ↑	LPIP ↓	FID ↓	PSNR ↑	SSIM ↑	LPIP ↓	FID ↓
Task: Translation
Baseline	13.699	0.309	0.485	0.942	14.126	0.326	0.467	0.968
3DIT(1-task)	14.546	0.273	0.494	0.254	14.400	0.262	0.498	0.261
3DIT(Multitask)	15.21	0.300	0.472	0.244	15.200	0.292	0.477	0.253
Task: Rotation
Baseline	13.179	0.269	0.540	0.997	12.848	0.270	0.538	1.693
3DIT(1-task)	16.828	0.386	0.428	0.291	16.293	0.372	0.445	0.280
3DIT(Multitask)	16.859	0.382	0.429	0.248	16.279	0.366	0.447	0.236
Task: Insertion
Baseline	12.297	0.269	0.594	0.969	12.542	0.275	0.584	1.325
3DIT(1-task)	13.469	0.267	0.549	0.254	12.974	0.261	0.566	0.233
3DIT(Multitask)	13.630	0.263	0.551	0.222	13.088	0.261	0.568	0.214
Task: Removal
Baseline	12.494	0.383	0.465	0.801	12.123	0.379	0.459	1.047
3DIT(1-task)	24.937	0.588	0.254	0.241	24.474	0.561	0.260	0.258
3DIT(Multitask)	24.980	0.585	0.249	0.236	24.661	0.568	0.260	0.240
typical use case for editing applications where a user has the flexibility to pick from a range of generated solutions. We report metrics separately for seen and unseen object categories.
Table 1 presents quantitative evaluations for 3DIT in comparison to the baselines. 3DIToutperforms the baselines for all four tasks at the metrics PSNR, SSIM and LPIP. Notably, the multi task model does well in comparison to the single task variant, in spite of having to learn 4 tasks using the same number of learnable parameters. The FID scores for the baseline models tend to be higher. This is because the baselines tend to cut/paste objects in the image (for e.g. in the translation task), which retains image fidelity, even if the scale of the object is incorrect. 3DIT on the other hand does not explicitly cut/paste segments and instead must render them using the diffusion process, and is thus prone to a poorer fidelity. On the contrary, our model is able to properly account for a variety of challenging changes to the underlying 3D scene when editing images, as shown in Figure 4. Its worth noting that the automatic evaluation metrics have limitations and often do not capture editing nuances encompassing geometry, lighting, and fidelity to the instruction. This motivates the need for human evaluation studies.
5.3 Human evaluation studies
We conduct human preference evaluations between 3DIT and the relevant baseline by showing two images and asking annotators to select the one that best matches the ground truth image. We measure (1) Geometric consistency – This requires humans to consider the geometric correctness of the transformed object, including the scale, positioning of the object on the ground plane and its relationship to other objects. It also requires humans to consider the correctness of other objects in the scene which may get occluded or unoccluded as a result of the transformation. The source caption. (2) Lighting consistency – This requires humans to consider the lighting correctness of the transformed object, including the direction and scale of the shadow as a result of the directional lighting. It also requires humans to consider the correctness of the shadows of other objects in the scene which may get occluded or unoccluded as a result of the transformation. Both evaluations also allow a third option (Tie) to be selected. Each pairwise evaluation is carried out for 30 test samples for OBJECT evaluation and 20 samples for real world evaluation.
Perspective change
Figure 4: The figure shows the ability of 3DIT to handle various challenges of 3D-aware image editing such as: (a) (Top left) perspective size changes; (b) (Top right) synthesizing novel view points; (c) (Bottom left) generating occluded regions; (d) (Bottom right) accounting for scene lighting while rendering objects and their shadows.
Table 2 presents a human evaluation study of the 3DIT model (in a single task setting) in comparison to the corresponding baseline for all four tasks for images from OBJECT and the real world. 3DIT is heavily favored by humans, consistently obtaining preference scores of  $70\%$  and more across all four tasks for geometric as well lighting consistency. The tied scores refer to instances where both models did exceedingly poorly and where both models did a close to perfect job.
For the translation task, 3DIT is able to scale the object appropriately, as well rendering the shadow correctly. The baseline, in particular, does a poor job of the shadow and gets the scale wrong, leading to a physically implausible image. For the rotation task, 3DIT performs a rotation consistent with the ground plane and also renders a superior shadow. For the removal task, 3DIT tends to inpaint occluded objects well, and correctly adjusts their shadows. It also does well at removing the entire extent of the correct object in contrast to the baseline.
5.4 Real-world transfer qualitative
While we train our models on simulated data, we test the model's ability to transfer to real-world images qualitatively. Figure 5 shows our model's output for different prompts for the same input image for all four tasks. We find these preliminary results encouraging as the outputs not only respect the task description but also look reasonably photo-realistic with appropriate shadows despite never seeing real-world editing examples during training.
6 Limitations and Broader Impact
Our work explores the use of synthetic data for training physically plausible and scene-aware image editing models. Given that even training on scenes with limited realism and complexity results in models that transfer well to the real world, there is tremendous potential to significantly improve performance by using more advanced photo-realistic simulators. We give an analysis of the typical failure cases of our model in the appendix. Finetuning on a small set of hand-crafted real-world editing examples may also improve transfer to real-world images and enable compelling editing applications. Our work leads the way towards easy-to-use and increasingly powerful image editing
Table 2: Outcome of the human evaluation. The table illustrates the evaluators' preferences for 3DIT assessed on geometric accuracy and 3D lighting consistency. Baseline methods rarely gained preference due to their limited capacity to maintain geometric quality and lighting consistency.
Task	OBJECT evaluation
Geometric consistency	Lighting consistency
Baseline	3DIT(Ours)	Tie	Baseline	3DIT(Ours)	Tie
Translation	20.0 %	73.3 %	6.6 %	3.3 %	80.0 %	16.6 %
Rotation	3.3 %	80.0 %	16.6 %	6.6 %	73.3 %	20.0 %
Insertion	13.3 %	70.0 %	16.6 %	10.0 %	73.3 %	16.6 %
Removal	3.3 %	86.6 %	10.0 %	0.0 %	86.6 %	13.3 %
Real world evaluation
Task	Geometric consistency	Lighting consistency
Baseline	3DIT(Ours)	Tie	Baseline	3DIT(Ours)	Tie
Translation	25.0 %	60.0 %	15.0 %	15.0 %	70.0 %	15.0 %
Rotation	10.0 %	80.0 %	10.0 %	10.0 %	80.0 %	10.0 %
Insertion	35.0 %	55.0 %	10.0 %	35.0 %	55.0 %	10.0 %
Removal	10.0 %	75.0 %	15.0 %	5.0 %	80.0 %	15.0 %
Figure 5: 3DIT is able to generalize to the real world while only being trained on a synthetic dataset. Here we show varying prompts for each of the four editing tasks.
capabilities for the broader society in the near future. Like any generative model, our work could also potentially be misused for propagating misinformation.
This work presents 3DIT, a model capable of editing individual objects within images, given a language instruction. 3DIT is trained on a new dataset, OBJECT, consisting of 400k 3D scenes procedurally generated using Objaverse objects. 3DIT performs well across on OBJECT and shows promising generalization to CLEVR as well as the real world.
[1] R. Abdal, P. Zhu, N. J. Mitra, and P. Wonka. Styleflow: Attribute-conditioned exploration of stylegan-generated images using conditional continuous normalizing flows. ACM Transactions on Graphics (ToG), 40(3):1-21, 2021.
[2] O. G. Art. OpenGameArt.org. https://opengameart.org/, 2023. [Online; accessed 16-May-2023].
[3] A. Bhattachad, A. Dundar, G. Liu, A. Tao, and B. Catanzaro. View generalization for single image textured 3d models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[4] A. Bhattachad and D. Forsyth. Cut-and-paste object insertion by enabling deep image prior for reshading. In 2022 International Conference on 3D Vision (3DV). IEEE, 2022.
[5] A. Bhattad and D. A. Forsyth. Stylitgan: Prompting stylegan to produce new illumination conditions, 2023.
[6] E. R. Chan, C. Z. Lin, M. A. Chan, K. Nagano, B. Pan, S. De Mello, O. Gallo, L. J. Guibas, J. Tremblay, S. Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16123-16133, 2022.
[7] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015.
[8] D. Z. Chen, Y. Siddiqui, H.-Y. Lee, S. Tulyakov, and M. Nießner. Text2tex: Text-driven texture synthesis via diffusion models. arXiv preprint arXiv:2303.11396, 2023.
[9] Y. Chen, F. Rong, S. Duggal, S. Wang, X. Yan, S. Manivasagam, S. Xue, E. Yumer, and R. Urtasun. Geosim: Realistic video simulation via geometry-aware composition for self-driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[10] M. J. Chong and D. Forsyth. Jojogan: One shot face stylization. arXiv preprint arXiv:2112.11641, 2021.
[11] M. J. Chong, H.-Y. Lee, and D. Forsyth. Stylegan of all trades: Image manipulation with only pretrained stylegan. arXiv preprint arXiv:2111.01619, 2021.
[12] B. O. Community. Blender - a 3d modelling and rendering package, 2018.
[13] M. Deitke, R. Liu, M. Wallingford, H. Ngo, O. Michel, A. Kusupati, A. Fan, C. Laforte, V. Voleti, S. Y. Gadre, et al. Objverse-xl: A universe of  $10\mathrm{m} + 3\mathrm{d}$  objects. arXiv preprint arXiv:2307.05663, 2023.
[14] M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi. Objverse: A universe of annotated 3d objects. arXiv preprint arXiv:2212.08051, 2022.
[15] M. Deitke, E. VanderBilt, A. Herrasti, L. Weihs, J. Salvador, K. Ehsani, W. Han, E. Kolve, A. Farhadi, A. Kembhavi, et al. Procthor: Large-scale embodied ai using procedural generation. Conference on Neural Information Processing Systems, 2022.
[16] A. Deshpande, J. Lu, M.-C. Yeh, M. Jin Chong, and D. Forsyth. Learning diverse image colorization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6837-6845, 2017.
[17] P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
[70] J. Tang, T. Wang, B. Zhang, T. Zhang, R. Yi, L. Ma, and D. Chen. Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior. arXiv preprint arXiv:2303.14184, 2023.
[71] D. Ulyanov, A. Vedaldi, and V. Lempitsky. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[72] V. Vavilala, S. Jain, R. Vasanth, A. Bhattachad, and D. A. Forsyth. Blocks2world: Controlling realistic scenes with editable primitives. CoRR, abs/2307.03847, 2023.
[73] H. Wang, X. Du, J. Li, R. A. Yeh, and G. Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. arXiv preprint arXiv:2212.00774, 2022.
[74] H. Wang, W. Liang, and L.-F. Yu. Scene mover: Automatic move planning for scene arrangement by deep reinforcement learning. ACM Transactions on Graphics, 39(6), 2020.
[75] J. Wang, S. Manivasagam, Y. Chen, Z. Yang, I. A. Bársan, A. J. Yang, W.-C. Ma, and R. Urtasun. Cadsim: Robust and scalable in-the-wild 3d reconstruction for controllable sensor simulation. In 6th Annual Conference on Robot Learning, 2022.
[76] Y. Wang, B. L. Curless, and S. M. Seitz. People as scene probes. In European Conference on Computer Vision, pages 438-454. Springer, 2020.
[77] Y. Wang, A. Liu, R. Tucker, J. Wu, B. L. Curless, S. M. Seitz, and N. Snavely. Repopulating street scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[78] Q. A. Wei, S. Ding, J. J. Park, R. Sajnani, A. Poulenard, S. Sridhar, and L. Guibas. Lego-net: Learning regular rearrangements of objects in rooms. arXiv preprint arXiv:2301.09629, 2023.
[79] L. Weihs, M. Deitke, A. Kembhavi, and R. Mottaghi. Visual room rearrangement. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5922-5931, 2021.
[80] F. Xia, A. R. Zamir, Z.-Y. He, A. Sax, J. Malik, and S. Savarese. Gibson env: real-world perception for embodied agents. In Computer Vision and Pattern Recognition (CVPR), 2018 IEEE Conference on. IEEE, 2018.
[81] K. Yi, C. Gan, Y. Li, P. Kohli, J. Wu, A. Torralba, and J. B. Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442, 2019.
[82] A. Zeng, A. S. Wong, S. Welker, K. Choromanski, F. Tombari, A. Purohit, M. S. Ryoo, V. Sindhwani, J. Lee, V. Vanhoucke, and P. R. Florence. Socratic models: Composing zero-shot multimodal reasoning with language. ArXiv, abs/2204.00598, 2022.
[83] L. Zhang and M. Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
[84] J. Zhu, Y. Shen, D. Zhao, and B. Zhou. In-domain gan inversion for real image editing. In Proceedings of European Conference on Computer Vision (ECCV), 2020.
[85] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, 2017.
8.1 Training and Inference Details
We closely follow the training procedure established by [47], with a few modifications. Our approach uses an effective batch size of 1024, which is smaller than the batch size of 1536 used by Zero-1-to-3. This adjustment was necessary because of the additional memory requirements caused by the reintroduction of the CLIP text encoder. This batch size is achieved by using a local batch size of 64 across 40GB NVIDIA RTX A6000 GPUs, along with two gradient accumulation steps. Similar to Zero123, we train on images with a resolution of  $256 \times 256$ , resulting in a latent spatial dimension of  $32 \times 32$ . Following their protocol, we utilize the AdamW optimizer, with a learning rate of 1e-4 for all parameters of the model except for those of the concatenation MLP, which uses a learning rate of  $1e - 3$ . Our training process runs for a total of 20,000 steps. We then select the best checkpoint based on our metrics computed from an unseen object validation set. As was the case in StableDiffusion, we freeze the CLIP text encoder during training. For inference, we generate images with the DDIM [68] sampler using 200 steps. We do not use classifier-free guidance, i.e. the CFG term is set to 1.0.
8.2 Robustness to severity of transformation
We analyze the robustness of our method by measuring the performance of the single task rotation model as the complexity of the scene and severity of transformation changes. In Figure 6, we show the average of our Mask PSNR metric as the number of objects in the scene varies from 1 to 4, where a slight drop in performance occurs as the number of objects increases. In Figure 7, we show average Mask PSNR for rotations in a given angle range on a pie chart, where it can be seen that the model does better with smaller angle deviations.
Figure 6: Average Mask PSNR of the single-task rotation model as the number of objects in the scene varies.
Figure 7: Average Mask PSNR of the single-task rotation model for rotation angles falling within a slice.
8.3 Model failure analysis
Here we analyze the most common failure model of 3DIT.
Incorrect geometry: The model incorrectly changes the object's geometry.
Unintended global modifications: The model changes parts of the scene that are not intended to be edited.
Incorrect texture: The model changes details on the surface of the object, failing to preserve its identity.
Incorrect localization: The model performs the requested task, but on an object that is different from the intended one.
Figure 8: Qualitative examples of the four must common failure cases produced by the model.
Figure 9: Frequency of most common failure cases. We examine 50 random outputs of the model and bucket the model errors by failure type
8.4 Initialization ablation
To test the effect of the model's initialization, we run an ablation training our model on three different initializations: Image-conditioned Stable Diffusion [44], Zero123 [47] (our standard initialization), and the recently released Zero123-XL [13], which is a Zero123 model trained on a much larger version of the Objaverse dataset with 10M assets. We see that although Zero123-XL was a more performant base model for the task of rotating a single object, it does not cause an improvement overall on the tasks we consider here.
Table 3: Quantitative results for training our single-task model across three different ablations: Image-conditioned Stable Diffusion, Zero123, and the recently released Zero123-XL.
Model	Seen Objects	Unseen Objects
PSNR ↑	SSIM ↑	LPIP ↓	FID ↓	PSNR ↑	SSIM ↑	LPIP ↓	FID ↓
Task: Translation
SD	14.373	0.264	0.510	0.101	14.351	0.253	0.510	0.102
Zero123	15.210	0.300	0.472	0.244	15.200	0.292	0.477	0.253
Zero123-XL	15.121	0.294	0.477	0.252	15.052	0.286	0.478	0.239
Task: Rotation
SD	15.074	0.368	0.430	0.089	14.558	0.359	0.438	0.095
Zero123	16.859	0.382	0.429	0.248	16.279	0.366	0.447	0.236
Zero123-XL	15.433	0.381	0.420	0.241	15.008	0.378	0.429	0.243
Task: Insertion
SD	13.220	0.253	0.570	0.108	13.131	0.255	0.572	0.100
Zero123	13.630	0.263	0.551	0.222	13.088	0.261	0.568	0.214
Zero123-XL	13.481	0.264	0.557	0.274	13.094	0.259	0.566	0.258
Task: Removal
SD	23.882	0.576	0.263	0.117	23.352	0.542	0.270	0.115
Zero123	24.980	0.585	0.249	0.236	24.661	0.568	0.260	0.240
Zero123-XL	24.775	0.585	0.255	0.247	24.830	0.568	0.253	0.215
8.5 Additional qualitative results
In Figure 10, we show qualitative results from our multitask model on each of the four editing tasks.
Table 4: Summary of key statistics of the OBJECT dataset.
Total objects	62950
Total categories	1613
Object per category median	6.0
Object per category mean	39.03
Object per category std	138.00
8.6 Dataset Analysis
In this subsection, we provide some details about the composition and statistical makeup of our dataset. In Table 4, we show a statistical overview of the dataset, including total number of objects and categories, as well as the mean, median, and standard deviation of objects per category. We also visualize the distribution of objects across categories, as can be show in the histogram in Figure 11. Finally, we visualize the frequency of category names in the wordcloud in Figure 12.
Figure 10: Generated examples by the 3DIT multitask model.
Figure 11: Object categories from seen and unseen splits sorted by frequency, in log scale.
Figure 12: A wordcloud visualizing the frequency of various object category names.