Generating Images with 3D Annotations Using Diffusion Models
Authors: Wufei Ma
Qihao Liu
Jiahao Wang
Angtian Wang
Xiaoding Yuan
Yi Zhang
Zihao Xiao
Guofeng Zhang
Beijia Lu
Ruxiao Duan
Yongrui Qi
Adam Kortylewski
Yaoyao Liu
Alan Yuille
Venue: ICLR 2024 spotlight (2024)
Keywords: Synthetic Data
Transfer Learning
Diffusion Models
3D
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We propose 3D Diffusion Style Transfer (3D-DST), which incorporates 3D control into diffusion models. With 3D-DST, we can change the 3D structures of the objects in the generated images and obtain ground-truth 3D annotations automatically.

Abstract: Diffusion models have emerged as a powerful generative method, capable of producing stunning photo-realistic images from natural language descriptions. However, these models lack explicit control over the 3D structure in the generated images. Consequently, this hinders our ability to obtain detailed 3D annotations for the generated images or to craft instances with specific poses and distances. In this paper, we propose 3D Diffusion Style Transfer (3D-DST), which incorporates 3D geometry control into diffusion models. Our method exploits ControlNet, which extends diffusion models by using visual prompts in addition to text prompts. We generate images of the 3D objects taken from 3D shape repositories~(e.g., ShapeNet and Objaverse), render them from a variety of poses and viewing directions, compute the edge maps of the rendered images, and use these edge maps as visual prompts to generate realistic images. With explicit 3D geometry control, we can easily change the 3D structures of the objects in the generated images and obtain ground-truth 3D annotations automatically. This allows us to improve a wide range of vision tasks, e.g., classification and 3D pose estimation, in both in-distribution (ID) and out-of-distribution (OOD) settings. We demonstrate the effectiveness of our method through extensive experiments on ImageNet-100/200, ImageNet-R, PASCAL3D+, ObjectNet3D, and OOD-CV. The results show that our method significantly outperforms existing methods, e.g., 3.8 percentage points on ImageNet-100 using DeiT-B. Our code is available at <https://ccvl.jhu.edu/3D-DST/>

Published as a conference paper at ICLR 2024
GENERATING IMAGES WITH 3D ANNOTATIONS USING DIFFUSION MODELS
Wufei Ma Qihao Jiahao Wang', Angtian Wang', Xiaoding Yuan', Yi Zhang', Zihao Xiao', Guofeng Zhang', Beijia Lu', Ruxiao Duan', Yongrui Qi', Adam Kortylewski Yaoyao Liu 18, Alan Yuille'
Johns Hopkins University University of Freiburg Max Planck Institute for Informatics, Saarland Informaties Campus
ABSTRACT
Diffusion models have emerged powerful generative method, capable of produc- ing stunning photo-realistic images from natural language descriptions. However, these models lack explicit control over the 3D structure in the generated images. Consequently, this hinders our ability to obtain detailed 3D annotations for the gen- erated images or to craft instances with specific poses and distances. In this paper, we propose 3D Diffusion Style Transfer (3D-DST), which incorporates 3D geome- try control into diffusion models. Our method exploits ControlNet, which extends diffusion models by using visual prompts in addition to text prompts. We generate images of the 3D objects taken from 3D shape repositories (e.g., ShapeNet and Objaverse), render them from a variety of poses and viewing directions, compute the edge maps of the rendered images, and use these edge maps as visual prompts generate realistic images. With explicit 3D geometry control, we can easily change the 3D structures of the objects in the generated images and obtain ground-trute 3D annotations automatically. This allows us to improve wide range of vision tasks, e.g., classification and 3D pose estimation, in both in-distribution (ID) and out- of-distribution (OOD) settings. We demonstrate the effectiveness of our method through extensive experiments on ImageNet-100/200, ImageNet-R. PASCAL3D+, ObjectNet3D, and OOD-CV. The results show that our method significantly outper- forms existing methods, c.g., 3.8 percentage points on ImageNet-100 using DeiT-B. Our code is available at https: /tt:////cvvvi /ccll.cl...//ooo//occc//e..ce.ll jhu edu 3D-DS/
INTRODUCTION
Understanding the underlying 3D world of 2D images essential numerous computer vision tasks. The utilization of 3D modeling opens up the possibility of addressing a significant portion of the variability inherent natural images, which could potentially enhance the overall understanding and interpretation of images (Wu et al., 2020). For example, 3D-aware models show high robustness and generalization ability under occlusion or environmental changes (Liu al., 2022a). However, is expensive and time-consuming to obtain ground-trutt 3D annotations for 2D images. This training data shortage becomes a main obstacle training large-scale 3D-aware models.
Recently, diffusion models (Ho et al., 2020) have shown impressive performance in generating photo-realistic images, which can used to solve the training data shortage. These models allow us to produce high-quality images from various conditional inputs, e.g., natural language descriptions, segmentation maps, and keypoints (Zhang et al., 2023). This facilitates generative data augmentation, e.g., He et al. (2023) use diffusion models to augment ImageNet (Deng et al., 2009) and significantly improve the classification results.
Despite their success, diffusion models still lack explicit control over the underlying 3D world during the generation process. As a result, they still face two challenges that hinder their use in augmenting data for 3D tasks. The first challenge is the inability to control the 3D properties of the object in
Equal contribution.
Corresponding author: Yaoyao Liu (yy1 1uccs. ..l.c....h.ee. -.u...uee edu).

---

Published as a conference paper at ICLR 2024
Baseline
w/ T Text=to-lmmee Imame
w/ 3D-DST (ours)
90
88
48
Accuacy (96)
86
84
44
82
80
40
CAD Model
Generated Images
in-distribution (II) out-oo--o disttiiiiuutonn (OOD)
(a) Visualization of our 3D-DST
(b) Performance comparison
Figure (a) Visualization of our 3D-DST. Our proposed solution, 3D-DST, leverages both 3D visual prompts and large language model (LLM) text prompts to generate diverse images from CAD model. The use 3D visual prompts enables explicit control over the 3D structure of the object within the generated images, such varying 3D poses and distances. On the other hand, LLM text prompts facilitate the automatic generation of images with diverse backgrounds, weather conditions, and colors. (b) Performance comparison. Our model can be utilized to generate data enhance both in-distribution (ID) and out-of-distribution (OOD) performance. We report results of DeiT-S on ImageNet-100 (Tian et al., 2020) and ImageNet-R (Hendrycks al., 2021). "Text-to-Iman denotes using diffusion models without 3D control to augment the data (He et al., 2023).
the generated images, such as pose and distance. The second challenge is the difficulty in obtaining ground-truth 3D annotations of objects automatically.
To tackle the above challenges, we propose a simple yet effective framework, 3D Diffusion Style Transfer (3D-DST), which enables to incorporate knowledge about 3D geometry structures. Our method exploits ControlNet (Zhang et al., 2023), which extends diffusion models by using visual prompts in addition repositories (e.g., ShapeNet (Chang et al., 2015) and Objaverse (Deitke et al., 2023b)), render them from variety of viewing directions and distances, compute the edge maps of the rendered images, and use these edge maps visual prompts to generate realistic images. With explicit 3D geometry control, we can easily change the 3D structures of the objects in the generated images and obtain corresponding 3D annotations automatically.
prompts. We generate images of 3D objects taken from 3D shape
To enhance the diversity of the generated images, we apply the following strategies. Firstly, we vary the viewing directions in which the 3D objects are rendered. This generates wide range of edge maps and allows us to produce multiple distinct images for each 3D object. Secondly, we introduce a novel prompting technique to improve diversity further. We input essential information about the 3D objects into large language models (LLM) (Touvron et al., 2023) to obtain reasonable descriptions, such as background and color. Then, use these descriptions as the text prompts for diffusion models. This not only enables us to make the most of the vast potential of the diffusion models but also helps us avoid generating images that too similar. These two strategies allow to generate diverse images that can be used to improve the nut-of-distribution (OOD) robustness of AI models. As shown in Figure 1, our 3D-DST effectively generates images with wide range of viewpoints, distances, colors, and backgrounds. These generated images have proven valuable in enhancing performance across both in-distribution (ID) and OOD scenarios.
Our 3D-DST allows us to transfer recently released large-scale 3D object datasets, e.g., Objaverse and Objaverse-XL, into 2D datasets enriched with comprehensive 3D annotations, e.g., 3D poses, key points, and depths. Objaverse is repository of 800K CAD models, while Objaverse-Xo (Deitke al., 2023a) expands to more than 10 million 3D objects. With the capabilities offered by our 3D-DST, we are able to harness the potential of these vast datasets to enhance performance across various tasks, including 3D-aware classification,
pose estimation, OOD classification, and more.
We conducted extensive experiments to show the effectiveness of our method. Firstly, we show that our method can be directly used as a data augmentation method for classification. For example, our method improves the ImageNet-100 accuracy by 3.8 percentage points DeiT-B (Touvron et al., 2021). Secondly, we demonstrate that our method is able to boost the performance of 3D-aware models. With automatically produced 3D annotations, pre-traine pre-training improve the 3D pose estimation benchmark, PASCAL3D+ (Xiang et al., 2014), by 3.9 and 2.4 percentage points in ID and OOD settings, respectively (accuracy @ f.)
generated images can
2

---

Published as a conference paper at ICLR 2024
To summarise, we make three contributions:
Automatic generation of 3D annotations. We propose a simple yet effective pipeline that allows to add 3D) conditional control to diffusion models. This enables us acquire 3D annotations for the generated images through the rendering process.
Generating images with multiple viewpoints. Our method generates images from diverse viewpoints, including those that are rarely encountered in typical scenarios. They can be used as training to improve the model's 's robustness in the OOD setting.
Diverse text prompts by LLM. We propose novel strategy for text prompt generation using LLM. This strategy effectively prevents the generation of redundant or similar images.
2 RELATED WORK
Synthetic data augmentation. Synthetic data has gained significant attention in generating la- beled data for vision tasks that require extensive annotations (Rombach et al., 2022; Zhang et al., 2023). The synthetic data augmentation methods can be categorized into two groups: 1) 2D-based methods employ recent generative models like GANs and diffusion models to create photo-realistic images (Baranchuk et al., 2022; Liu et al., 2021a; Dosovitskiy et al., 2015; Sun et al., 2021), and other dense prediction tasks. Despite their effectiveness, these methods lack 3D structures, making it challenging to acquire 3D annotations for the generated images. 2) 3D-based methods leverage simulation environments with physically realistic engines to render 3D models and generate im- ages (Greff et al., 2022; Zheng et al., 2020). However, the generated images' diversity is limited as they heavily rely on the existing textures of the 3D models. In our work, we investigate the integration of 3D control into diffusion models. This allows us to generate diverse images using the appearance produced by diffusion models while obtaining 3D annotations through the 3D structure conditions.
Diffusion models operate by incrementally degrading the data through introducing Gaussian noise gradually, and subsequently learn to restore the data by reversing this noise infusion process (Ho al., 2020; Singer et al., 2023; Villegas et al., 2023). They have shown remarkable success in generating high-resolution photo-realistic images from various conditional inputs, c.g., natural language descriptions, segmentation maps, and keypoints (Ho et al., 2020; 2022; Zhang et al., 2023). Recently, text-to-image diffusion models have also been used to augment training data. Trabucco et al. (2023) investigate various strategies for augmenting individual images with the help of a pre-trained diffusion model, showcasing considerable enhancements in few-shot learning scenarios. Azizi et al. (2023) present evidence that the usage of class names as text prompts can guide the diffusion models to generate images that subsequently improve performance in ImageNet classification tasks. Despite their advancements, diffusion models still face limitations in explicitly controlling the 3D structure of the images they generate. Our research contributes to overcoming this challenge by integrating 3D geometry conditional inputs into diffusion models. This enhancement empowers us with precise control over the 3D structure of the object within the produced image, and facilitates effortless acquisition of 3D annotations, such as 3D pose key points.
Large language models (LLM).. The field of natural language processing has witnessed transfor- mative shift in recent years, spurred by the advent of large language models such as PaI.M (Chowdhery et al., 2023), and LLaMA (Touvron et al., 2023). These large language models have showcased remarkable proficiency in zero-shot and few-shot tasks, as well as more intricate assignments like mathematical problem-solving and commonsense reasoning. Their impressive performance can be attributed to the extensive corpora they trained on and the intensive computational resources dedicated to their training. In our study, we utilize LLaMA to enhance the quality of text prompts. By automatically generating deseriptive prompts pertaining backgrounds, color, and weather conditions, we successfully improve the diversity of our generated images.
3 3D DIFFUSION STYLE TRANSFER (3D-DST)
As illustrated in Figure 2, our 3D-DST enhances diffusion models by incorporating both 3D visual prompts and diverse text prompts. In Section 3.1, we describe how existing diffusion models incorporate 2D visual and simple text conditions for generative data augmentation. In Section 3.2, we introduce how to produce visual prompts based on 3D geometry conditions by graphics-based rendering. In Section 3.3, we show how to create text prompts with LLM to enhance diversity. Algorithm summarizes how to generate images with 3D annotations using our 3D-DST.
3

---

Published as a conference paper at ICLR 2024
3.1 BACKGROUND: DIFFUSION MODELS WITH TEXT AND 2D VISUAL PROMPTS
Diffusion models (Ho et al., 2020; Rombach et al., 2022) achieve great success in conditional image generation using and 2D visual prompts. In the following, we first introduce the original diffusion models without conditional inputs. Then, we show how to add text prompts by augmenting the underlying U-Net backbone of the diffusion models with a cross- -toss-tttnntion mechanism. Moreover, we elaborate on how to integrate 2D visual prompts using ControlNet.
Diffusion model pipeline. Diffusion models learn to generate images by learning sequence of denoising U-Nets. Starting from random noise 27, the denoising process is as follows,
2 E(2,, t) ==
(I)
where e(z,, t) denotes the denoising U-Nets. 2 is denoise version of the input 2₁. In the final step, zo is input pre-trained decoder D get the generated image Iinal, i.c., Arnal D(zo).
Adding text prompts via cross-attention. To enable lext-conditioning and generate images adhered to specific content requirements, previous works (Zhang et al., 2023; Azizi et al., 2023; Rombach et al., 2022) incorporate text prompts into diffusion models using the cross-attention mechanism (Vaswani al., 2017), following Rombach et al. (2022). Specifically, text prompts are preprocessed with pre-trained text encoder 0, which are then integrated to intermediate layers of the U-Net via cross-attention layers:
QKM Vioi-1
V, Q - w 01-1, K= w O(T), V = V= O(T),
softmax
(2)
where Φ; denotes the i-th layer intermediate representation of the U-Net e, and o; = zt w w((1)
and ( are learnable projection matrices. M a hyperparameter of the cross-attention.
Adding 2D visual prompts via ControlNet. To integrate 2D visual prompts into diffusion models, we utilize the ControlNet (Zhang et al., 2023) architecture, which incorporates 2D visual prompts without retraining the entire diffusion model. During Step t of the denoising process, the 2D visual prompts EDD are added to the latent variable 2,, and the resulting sum is inputted into the ControlNet. Then, the outputs of the ControlNet will be integrated into the denoising U-Net c, i.e.
E(E,, T, ControlNet(z; + E2D))
Limitations of existing methods. Although existing approaches have achieved notable success, they still encounter the following challenges. Firstly, these methods lack 3D geometry control, making it arduous to explicitly modify the 3D structure of objects in generated images and acquire corresponding 3D annotations, such as 3D pose key points, Secondly, existing methods frequently depend on simple text prompts guide the image generation process when augmenting datasets with diffusion models. Consequently, it is difficult to create images for OOD scenarios, which is very important for training robust models.
In order to address these challenges, in the following, we demonstrate the generation of 3D visual prompts through graphics- based rendering and augment the text prompts with LLM, effectively enhancing the diversity of the generated images.
3.2
3D VISUAL PROMPTS BY GRAPHICS-BANE BASED RENDERING
In order to effectively integrate 3D geometry control into diffusion models, it is essential to ensure that the visual prompts meet the following two specific requirements.
Firstly, the visual prompts must encompass sufficient information to depict the 3D geometry structure of the objects accurately. Without this information, it would be challenging to achieve explicit control over the 3D structure of the objects (e.g., shape and pose) in the generated images, consequently hindering the generation of corresponding 3D annotations.
Secondly, the visual prompts should be compact and concise, enabling diffuusion models to compre- hend and effectively process them. Utilizing overly complex visual prompts, such as the vertices
4

---

Published as a conference paper at ICLR 2024
render camera parameters &, K
Canny edge
3D Visual Prompt
rendered image render
CAD model C
visual prompt Eap
Image Generation
ControlNet
image decoder
U-Net E
3D shape and pose
D
2T
2
2
Zn
text encoder
"airplane"
L.M
very altitude
θ
Text
class name W
generated descriptions
Prompt
"MeDonnell Douglts acruplane
text prompt
keywords k
Our 3D-DST comprises three essential steps. (1) 3D visual prompt generation. We generate images of 3D objects taken from 3D shape repository (e.g., ShapeNet and Objaverse), render them from variety of viewpoints and distances. compute the edge maps of the rendered images, and use these edge maps as 3D visual prompts. (2) Text prompt generation. Our approach involves combining the class names of objects with the associated lags or keywords of the CAD models. This combined information forms the initial text prompts. Then, we enhance these prompts by incorporating the descriptions generated by LLaMA. (3) Image generation. We generate photo- realistic images with 3D visual and text prompts using Stable Diffusion and ControlNet.
Figure 2:
and meshes of a CAD model, would prove impractical and unfeasible for the diffusion models to handle efficiently. Therefore, it is crucial to strike a balance and employ visual prompts that are both informative and manageable within the diffusion model framework.
To satisfy the affrrementioned requirements, our proposed approach involves generating visual prompts through graphics- based rendering. We leverage 3D Computer Aided Design (CAD) models, which can be easily obtained from existing 3D shape repositories such as ShapeNet, Objaverse, and Objaverse-XL. XL. Then, render the CAD models from diverse viewpoints and distances. compute edge maps based on the rendered images, and utilize these edge maps as visual prompts. This approach enables of the objects. The rendered images provide essential details such as the viewing directions and distances, facilitating explicit control over the 3D structure during the generation of images. By extracting edge maps, we create compact visual prompts that are suitable for integration within diffusion models. In the following, we will elaborate on the detailed steps.
encapsulate the necessary information regarding the 3D geometry structure
Craphics-based rendering. We render CAD models with various viewing directions and distances and then obtain the rendered sketch images. Given a CAD model C, we generate the rendered sketch images Lender as follows.
/render R,C,E,K)
where R denotes off-the-shelf renderer, i.e. (Community, 2018), K is the camera intrinsic matrix, E represents a camera extrinsic matrix computed from a randomized viewing direction, and distance the Perspective camera with focal length
follows predefined distribution. In practice, 35mm to render the images. Complex backgrounds are not necessary here because the rendered sketch images used to extract edge maps.
5

---

Published as a conference paper at ICLR 2024
Edge map computation. Next, we extract edge maps from the rendered sketch images. The primary objective of this process is to enhance the compactness and conciseness of the rendered images while preserving the underlying 3D geometry structure. To achieve this, we apply classical edge detection methods, e.g.. Canny edge (Canny, 1986). Given a rendered sketch images renders we obtain the edge map &DD as follows,
EDD = Cannnyddgeennnn (ereedggeererderr Tender).
(5)
3.3 DIVERSE TEXT PROMPTS BY LLM
Algorithm Generating images using our 3D-DST
Text prompts play crucial role in guid- ing diffusion models to generate diverse im- ages. However, existing diffusion-modee based data augmentation methods often rely on overly simplistic text prompts. For exam- ple, Azizi et al. (2023) utilize class names directly as text prompts, while Zhang al. (2023) employ a default prompt such as "a high-quality, detailed, and professional im- age." These methods fail to fully harness the rich appearance information stored within diffusion models.
1: Input: 3D shape repository {C, ws kj.
Output: Images with 3D annotations (Jemat: Bsp)
for iterations do
Get a CAD model C from the 3D shape repository;
Load the class name and keyword & of C: Load the camera intrinsic matrix K;
Generate randomized camera extrinsic matrices (E): for different camera extrinsic matrix & do
9: // 3D visual prompt generation
10: Render sketch image leader using & and K by Eq.4;
11: Produce 3D annotations ysD of nenderi
To address the above issue, we present
12: Compute edge map Enp using Canny Edge by Eq.
novel strategy for text prompt generation.
13:
// Text prompt generation
Our approach forms the initial text prompts using the class names of objects with the asso-
14: Create text prompts T using us and by Eq. 6;
15:
Image generation
ciated tags or keywords of the CAD models.
16:
Initialize random noise 2T;
Then, we enhance these prompts by incor- porating the descriptions generated by LLM. The final text prompts T of CAD model C are created as follows,
17:
for in T...... do
18:
Denoise 9, to #1-1 using T and Eap by Eq. 3;
19:
Generate images /inal D(z0).
20:
Collect the generated dataset (Jimat, MSD
T {t, w, k. LLM(t, w. k)),
(6)
where represents prompt template, such as "a photo of ..... corresponds to the class name of the CAD model C, while le denotes the tags or keywords associated with C in the 3D shape repository (e.g.. Objaverse-XL). LLM is large language model capable of generating rich and coherent descriptions (e.g.. backgrounds, colors) when provided with the initial text prompts (t, w. k.).
4 EXPERIMENTS
4.1 IN-DISTRIBUTION (ID) AND OUT-OF-DISTRIBUTION (OOD) IMAGE CLASSIFICATION
Datasets. We consider three datasets: ImageNet-100 ImageNet-200, and ImageNet-R. The first two datasets are ID datasets, while the third one is an OOD dataset. All datasets are derived from ImageNet (Russakovsky et al., 2015), where ImageNet-100 (Tian et al., 2020; Liu et al., 2023; 2024) and ImageNet-200 are 100-class and 200-class subsets from random sampling. ImageNet- (Hendrycks et al., 2021) contains 30, 000 images with various artistic renditions of 200 classes of the original ImageNet. It is widely used to evaluate OOD performance.
Implementation details. Network architectures. We use representative network architectures, i..., DeiT (Touvron al., 2021), ConvNeXt (Liu et al., 2022b), and Swin Transformer (Swin) (Liu et al., 2021b). For DeiT and ConvNeXt, we follow the official codebases and experimented on tiny (DeiT only), small, and base settings. No distillation is used for DeiT. For Swin Transformer and MAE, we experimented on the small setting following the training strategies released in the official implementations. Data generation. We collect around 30 CAD models from ShapeNet and Objaverse for each object class. Then, run 3D-DST and generate 2, 500 images for each class.

---

Published as a conference paper at ICLR 2024
In-distribution (ID)
Methods
DeñT-Ti
DeïT-S
DeïT-B
ConvNeXt-S ConvNeXt-B Swin-S
MAE-S
Buseline
83.84
84,56
84,28
91.34
91.50
90.78
88,69
w/ Text2mmg (2023)
84.58 (10.74)
84,80 (224))
84.10 (10.18)
91.26
(J008)
91.28 (10.2)
90.42 (10.36)
88.78 (10.0)
w/ 3D-DST (ours)
87.36 (13.52)
88.96 (1444)
88.11
(13.8)
92.18 (10.8)
92.44 (10.94)
91.50 (10.72)
90.47 (11.8)
Out- of distribution (OOD)
Methods
DeïT-Ti
DeiT-S
DeiT-B
ConvNeXI-S ConvNeXt-B Swin-S
MAE-S
Baseline
49.96
49.61
50,53
67.19
66.40
54.99
65.18
w/ Tex22mmg (2023)
52.58
(12.2)
50.83
(12))) 49.61
(10.92)
67.50 mo.1
67.15 (10.75)
56.34 (11.35)
67.28 (12.00)
w/ 3D-DST (ours) 56.12
(10.16)
56.65
(7004)
56.74 (16.21)
69.69 (2250)
69.21 (?2.1)
59.97 (14.98)
68.42 (1324)
Table 1: Image classification accuracy (%) on ImageNet-100 (ID) and ImageNet-R (OOD) using representative network architectures, ResNet and ViT. We compare the performances when models are (1) trained purely on the target dataset, (2) pre-trained on Text2Img (He et al., 2023) data, which does not have 3D control, and then finetuned on the target dataset, (3) pre-trained on 3D-DST data, and finetuned on the target dataset. Experiments show that our 3D-DST data can help boost the classification accuracy of both models on both ID and OOD cases by a large margin.
ID results on ImageNet- 100 and ImageNet-200. In Ta- ble 1, we show the ID classification results ImageNet-100.. "Baseline" (Line 1) is to train models purely on ImageNet- 100 for 600 epochs. "w/ Tex12Img" (Line 2) is to pre- train the models on the images generated by a text-to-image (Text2Img) model (He et al., 2023) without 3D control for 300 epochs and finetune on ImageNet-100 for 300 epochs. "ww/ 3D-DST" (Line 3) is to pre-train the models on our 3D-DST synthetic data first for 300 epochs and finetune on ImageNet-100 for 300 epochs. For MAE in various settings, we follow the official implementation and pretrain the model for 800 epochs and then finetune ImageNet-100 for 100 epochs. Comparing results on Line and Line 1, we can observe with the help of 3D-DST data, the Top-1 accuracies increase by more than 3.50 percentage points for DeiT models and average of 1.07 percentage points for other models with an accuracy of more than 90 percentage points. As a comparison, pretraining on Text2Img (Line 2) yields mixed results with no evident improvements. In Table 2, we provide the results on ImageNet-200. We can see that using our 3D-DST can still achieve significant improvements, i.e., 3.31 percentage points compared the baseline.
Methads
Accuracy (%)
Baseline
81.50
w/ / Tex12Img (2023)
83.45 (t15
/ /D3DDDST (ours)
84.81 (13.31)
Table 2: Image classification accuracy
on ImageNet-200 using DeiT-S.
OOD results on ImageNet-R. Table also demonstrate the OOD classification results on ImageNet- R. For "Baseline", "w/ Text2Img", and "w/ 3D-DST". we apply the similar settings as the ID experiments. Comparing results on Line 2 with Line 1, we can observe that Text2Img improves the performance on ImageNet-R by small marge with an average of 1.06 percentage points, while pretraining models on our 3D-DST data bring a consistent and significant improvement across all models, with average of 4.70 percentage points.
Choice of different data generation methods. To demonstrate the advantages of our proposed 3D-DST over, we consider the following data generalization baselines and present the quantitative results in Table 6. (i) Diffusion-based generation conditioned on edges obtained from ImageNet images. (ii) Rendering images with random 2D image backgrounds. (iii) Rendering images with random 3D environment backgrounds. Please refer Section D where we discuss the detailed implementations and present qualitative examples from each method.
Ablation results. In Table 3, we provide the ablation results. Line shows the baseline results without pre-training. Line and Line report the results that pre-train on 3D-DST data with and without LLM prompts. As we can see, using LLM can significantly improve the performance in-distribution and -nttrf-distribution data. Note that the LLM prompts are generated with our novel generation strategy introduced in Section 3.3, effectively improving the diversity of the produced prompts. As a result, L.M prompts achieve larger improvements in OOD compared to ID.
both
4.2 ROBUST CATEGORY-LEVE, 3D POSE ESTIMATION
Datasets and evaluations. Datasets. We consider three datasets for 3D pose estimation: PAS- CAL3D+, ObjectNet3D,, and OOD-CV. The PASCAL3D+ dataset (Xiang al., 2014) contains
7

---

Published as a conference paper at ICLR 2024
3D-DST LLM
In-distribution (ID)
Out-of-distribttion (OOD)
No.
(pre-train)
DeiT-Ii
DeiT-S
ConvNeXi-S
DeiT-Ti
DeiT-S
ConvNeXI-S
83.84
84.56
91.34
49.96
49.61
67.19
86,52 (12.6))
87.82 (13.26)
91.96 (10.22)
54.51 (14.55)
55.95 (16.4))
68.29 (11.0)
3
87.36 (13.52)
88.96 (14.40)
92.18 (10.84)
56.12 (16.16) 56.65 (17.04)
69.69 (12.50)
Table 3: Ablation Study (%) in the ID and OOD settings. Numbers in brackets show rise/fall compared to the baseline. For ID, we use ImageNet-100. For OOD, we use ImageNet-R. Line I shows the baseline results without pre-training. Lines and 3 report the results that pre-train on 3D-DST data with and without LLM prompts.
In-distribution (ID)
Out-of-distribttion (OOD)
Methods
Acc@n /
Acc@
Acc@T/6
Acc@ //18
ResNet
82.33
52.60
50.38
23.38
ResNet w/ AugMix (2020)
82.72 (10.39)
53.89 (11.29)
51.77 (11.39)
24.57 (11.19
ResNet w/ 3D-DST (ours)
84.22 (89)
56.52 (13.92)
52.75 (12.37)
25.70 (12.32)
NeMo (2021)
82.23
57.12
55.31
26.57
NeMo w/ AugMix (2020)
83.11 (10.88)
58.22 (11.10)
56.38 (11.07)
26.63 (70.00)
NeMo w/ 3D-DST (ours)
85.70 (13.47)
62.51 (15.39)
58.81 (13.50)
26.44 (10.13)
Table 4: Robust 3D pose estimation on ID (PASCAL3D+ & ObjectNet3D (Xiang al., 2016; 2014))
and OOD (OOD-CV (Zhao et al., 2022)). We experiment with classification- pose estimation method, ResNet, and 3D compositional model, NeMo (Wang et al., 2021). Experimental results demonstrate that our DST synthetic data can effectively improve 3D pose estimation performance on both ID and OOD benchmarks.
11, 045 training images and 10, 812 validation images with category and object pose annotations. The OOD-CV dataset (Zhao et al., 2022; 2023) includes OOD examples from PASCAL3D+ and a benchmark evaluate OOD robustness to individual nuisance factors, including pose, shape, appearance, context, and weather. The ObjectNet3D dataset is another 3D pose estimation benchmark that contains 100 categories with 17, 101 training samples and 19, 604 testing samples. Follow- ing Wang et al. (2021); Zhao et al. (2022), we evaluate pose estimation models on 10 categories from PASCAL3D+ and subset of ObjectNet3D with 10 categories. Evaluations. Following Zhou et al. (2018); Wang al. (2021), we measure the 3D pose prediction with the pose estimation error between the predieted rotation matrix and the ground truth rotation matrix A A(Rpredd Rgg [llegm( (R Byllle We report results under thresholds and I8. following Zhou et al. (2018); Wang et al. (2021).
V2
ID Results on PASCAL3D+ and ObjectNet3D. In Table 4, we show the ID pose estimation results on PASCAL3D+ and ObjectNet3D. The first block (Lines 1-3) shows ResNet results, i.e., extending a ResNet model with a pose classification head (Zhou et al., 2018). The second block (Lines 4-5) shows the results based on the state-of-the--e ahe 3D pose estimation method, NeMo (Wang et al., 2021). "w/ 3D-DST" denotes using the models pre-trained on 3D-DST. We also pre-train the model with a strong data augmentation method, AugMix (Hendrycks al., 2020), which mixes augmented images and enforces consistent embeddings of the augmented images. "w/ AugMix" shows its results. We can see that using the model pre-trained on our synthetic data can effectively improve the 3D pose estimation results. For example, comparing Line 3 with Line 1, we can observe that "w/ 3D-DST" improves the 3D pose estimation results by 1.89 and 3.47 percentage points with a threshold of " and by 3.92 and 5.39 percentage points for 18. on ResNet. As a comparison, "w/ AugMix brings limited improvements compared to our 3D-DST when evaluated on the test data.
OOD Results on on OOD-CV. We also provide OOD pose estimation results on the OOD-CV dataset. For the baseline, "w/ AugMix", and "w/ 3D-DST", we apply the similar settings as the ID experiments. Comparing Line 3 with Line 1, our 3D-DST data can effectively improve the model's performance OOD data, e.g., with a gap of 2.37 and 3.50 percentage points for # This shows that our approach with 3D-DST can introduce diverse data not present in the training data and improve the robustness of models to various domain shifts.
8

---

Published as a conference paper at ICLR 2024
Real vs. Synthetic (3D-DST)
Diverse Appearances
CAD models
Real
3D-DST
Figure 3: Left:
Visualizations of 3D-DST synthetic data for image classification and 3D pose estimation. Right: Objects with various appearances can be generated from only a limited number of CAD models by conditioning on diverse textual prompts.
4.3 3D OBJECT DETECTION
Training: Canny Inference: Canny
To show the flexibility our 3D-DST approach and its po- tential for wide range of 3D tasks, we present qualitative and quantitative results on 3D object detection from scenes in ARKitScenes dataset (Baruch et al., 2021; Brazil al.,
Training: MiDaS Depths; Inference: MiDaS Depths
2023). We present experimental settings in Section and show that with 3D-DST we can effectively improve 3D object detection on both AP2D and AP3D metrics. This demonstrates that our 3D-DST is also applicable com- plex scenes with multiple objects from multiple categories.
Training: MiDa8 Depths; Inference: Rendered Depths
4.4 VISUALIZATIONS
Figure 4: Qualitative examples of us- ing different types of 3D control. We experimented with three different types of 3D control: edge maps (top), MiDaS predicted depth (middle), and Blender rendered depth (bottom), using the same 3D model and text prompts. Qualitative results show that using edge maps as 3D control gives overall better outputs.
Figure 3 shows the visualized samples generated by our 3D-DST. can observe that 3D-DST can generate re- alistic images with diverse appearances. In Figure 4, we provide the comparisons among three types of 3D controls, edge maps, MiDaS (Ranftl et al., 2020) predicted depths, and rendered depths. Results show that using edge maps control gives the best outputs visually, while MiDaaS depths and rendered depths demonstrate reasonable results but are limited in the realism and clarity of the foreground object or the background scene.
4.5 ANALYSES FAILURE CASES AND LIMITATIONS
We conduct a thorough analysis of the failure cases by our generation pipeline by collecting inputs from human evaluators. We further present K-fold consistency filter (KCF) that can automatically detect and remove failed images. By analyzing the failure modes of our generation pipeline, we identify a limitation of our model to be the images with uncommon viewpoints (e.g., looking at cars from below). Please refer to Section F for detailed results and discussions.
CONCLUSION
In this work, we introduce simple yet effective framework, 3D-DST, which incorporates 3D geometry control into diffusion models. This empowers us with explicit control over the 3D structure of the objects in the generated images. As result, annotations for the produced 2D images. To boost the diversity of the images, we adjust 3D poses and distances and employ LLM for creating dynamic text prompts. Our empirical results reveal that the images generated by our method can significantly enhance performance across range of vision tasks, including classification and 3D pose estimation, in both ID and OOD settings.
conveniently acquire ground-truth 3D

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
Alan Yuille acknowledges the Army Research Laboratory award W911NF2320008 and ONR N00014- 21-1-2812. Adam Kortylewski acknowledges support for his Emmy Noether Research Group funded by the German Science Foundation (DFG) under Grant No. 468670075.
REFERENCES
Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David Fleet. Synthetic data from diffusion models improves imagenet classification. arXiv preprint arXiv:2304.08466, 2023.
Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label- efficient semantic segmentation with diffusion models. In ICLR, 2022.
Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, and Elad Shulman. ARKitscenes a diverse real-workl dataset for 3d indoor scene understanding using mobile RGB-d data. NeurIPS, 2021.
Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, and Georgia Gkioxari. Omni3D: large benchmark and model for 3D object detection in the wild. In CVPR, pp- 13154-13164 2023.
John Canny. A computational approach to edge detection. TPAMI, 8(6):699-66) 698, 1986.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint 1rxiv:1512.03012, 2015.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi. Sasha Tsyyashchenko, Joshua Maynez, Abhishck Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan. Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy
Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. JMLR, 24: 240: 140:1-240:1 13, 2023.
Blender Online Community. Blender 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. URL http://// www. blender. org.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPR Workshops, pp. 702-703, 2020.
Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan
Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, Eli VanderBilt, Aniruddha Kembhavi. Carl Vondrick, Georgia Gkioxari, Kiana Ehsani, Ludwig Schmidt, and Ali Farhadi. Objaverse-xl: A universe of 10m+ 3d objects. In NewrIPS, 2023a.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: universe of annotated 3d objects. In CVPR, pp. 13142-13153, 2023b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255 2009.
10

---

Published as a conference paper at ICLR 2024
Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In ICCV, pp. 2758-2766 2015.
Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, et al. Kubric: A scalable dataset generator. In CVPR, pp. 3749-3761, 2022.
Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition? In ICLR, 2023.
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. In ICLR, 2020.
Dan Hendryeks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, pp. 8340-8349, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), NeurlPS, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. JMLR, 23(47):1-33, 2022.
Jizhizi Li, Jing Zhang, Stephen Maybank, and Dacheng Tao. Bridging composite and real: towards end-to-end deep image matting. IJCV, 130(2):246-))) 2022.
Qihao Liu, Yi Zhang, Song Bai, and Alan L. Yuille. Explicit occlusion reasoning for multi-person 3d human pose estimation. In ECCV, pp. 497-517, 2022a.
Yaoyao Liu, Qianru Sun, Xiangnan He, An-An Liu, Yuting Su, and Tat-Seng Chua. Generating face images with attributes for free. TNNLS, 32(6): 2021a.
Yaoyao Liu, Yingying Li, Bernt Schiele, and Qianru Sun. Online hyperparameter optimization for dlas--nncrementa learning. In AAAI, pp. 3906-8913. 2023.
Yaoyao Liu, Yingying Li, Bernt Schiele, and Qiannu Sun. Wakening past concepts without past data: Class-incremental learning from online placebos. In WACV, pp. 2226-2235, 2024.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pp. 10012-10022, 2021b.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.
A convnet for the 2020s. CVPR, pp. 1 11966-1199 11966, 2022b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Wufei Ma, Angtian Wang, Alan Yuille, and Adam Kortylewski. Robust category-level 6d pose estimation with coarse-to-fine rendering of neural features. In ECCV, pp. 492-508, 2022.
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. TPAMI, 44(3): 1623-1637, 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 115(3):211-25) 2015. doi: 10.1007/s 0158061--
11

---

Published as a conference paper at ICLR 2024
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-videst Text-to-video generation without text-video data. In ICLR, 2023.
Deqing Sun, Daniel Vlasic, Charles Herrmann, Varun Jampani, Michael Krainin, Huiwen Chang, Ramin Zabih, William T Freeman, and Ce Liu. Autoflow: Learning a better training set for optical flow. CVPR, pp. 10093-10102, 2021.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, PP. 776-794, 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers distillation through attention. In ICML, volume 139, pp. 10347-10357, July 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 2023.
Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effective data augmenta- tion with diffusion models. arXiv preprini arXiv:2302.079900 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, pp. 5998-6008, 2017.
Ruben Villegas, Mohammad Babacizadeh, Picter-Jan Kindermans, Hernan Moraldo. Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In ICLR, 2023.
Angtian Wang, Shenxiao Mei, Alan L. Yuille, and Adam Kortylewski. Neural view synthesis and matching for semi-supervised few-shot learning of 3d pose. In NeurIPS, pp. 7207-7219, 2021.
Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi. Unsupervised learning of probably sym- metric deformable 3d objects from images in the wild. In CVPR, pp. 1-10, 2020.
Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese. Beyond pascal: A benchmark for 3d object detection the wild. In WACV, pp. 75-82, 2014.
Yu Xiang, Wonhui Kim, Wei Chen, Jingwei Ji, Christopher Choy, Hao Su, Roozbeh Mottaghi, Leonidas Guibas, and Silvio Savarese. Ohjectnet3d: large scale database for 3d object recogni- tion. In ECCV, pp. 160-176, 2016.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh. Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, pp. 6023-6032,, 2019.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, pp. 3813-3824, 2023.
Bingchen Zhao, Shaozuo Yu, Wufei Ma, Mingxin Yu. Shenxiao Mei. Angtian Wang, Ju He, Alan Yuille, and Adam Kortylewski. Ood-cv: A benchmark for robustness to out-of out-of-distributton shifts of individual nuisances natural images. ECCV, pp. 163-180. 2022.
Bingchen Zhao, Jiahao Wang, Wufei Ma, Artur Jesslen, Siwei Yang. Shaozuo Yu, Oliver Zendel, Christian Theobalt, Alan Yuille, and Adam Kortylewski. Ood-ev-v2: An extended benchmark for robustness to out-of-distribution shifts of individual nuisances in natural images. arXiv preprint 10X2:22304.10266. 2023.
Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou. Structured3d: A large photo-realistic dataset for structured 3d modeling. In ECCV, pp. 519-535, 2020.
12

---

Published as a conference paper at ICLR 2024
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmenta- tion. In AAAI, 2020.
Xingyi Zhou, Arjun Karpur, Linjie Luo, and Qixing Huang. Starmap for category-agnostic keypoint and viewpoint estimation. In ECCV, pp. 318-334, 2018.
13

---

Published as a conference paper at ICLR 2024
A ADDITIONAL RESULTS
demonstrate how diverse prompts from LLM can help to generate diverse outputs in textures and backgrounds, we show some qualitative comparisons in Figure between 3D-DST data with and without diverse prompts from LLM (when not using LLM, we only use the class names from ImageNet and descriptions from the CAD models as simple prompts).
Qualitative examples of LLM prompts.
Error bar. It would be too computationally expensive report the error bars for all the experiments. However, we compute the error bar for main experiment in the paper, i.e., the top-1 classification accuracy of the three models trained on ImageNet- 100 3D-DST, by only changing the random seeds and keeping all the other settings the same. The standard deviations of the classification accuracy of DeïT-S, ConvNeXt-S are 0.12%, 0.06%, respectively.
Generating balanced data from 3D controllability. Another advantage of our 3D-DST dataset is to generate balanced data with respect to 3D viewpoint, object 2D location, or size, thanks to the 3D controllability our generation pipeline. To visualize the 3D viewpoint distribution in different synthetic datasets, randomly select 200 car/bus image samples from Text2lmg and our 3D-DST, and run a state-of-the-ar 6D pose estimation model to obtain the 3D viewpoint, NeMo6D (Ma et al., 2022). Then, manually go the estimated poses remove samples with poorly estimated poses and obtain 120 well-annotated samples for each dataset. We visualize the distribution of the azimuth angles for 120 objects in Text2Img (left) and 3D-DST (right). Synthetic data in Tex(2mmg show strong pose biases propagated from the image diffusion model, while our 3D-DST dataset alleviates this issue by introducing explicit 3D controls.
B MODEL TRAINING
Image classification. We follow the implementation from the DeiT Touvron et al. (2021) codebase for all models. During training, the input image size is 224 224, the batch size set to 512, and we use AdamW Loshchilov & Hutter (2019) as the optimizer. The initial learning rate is le - 4 and we use cosine scheduler for learning rate decay. We use Rand-Augnn Augment Cubuk et al. (2020), random erasing Zhong et al. (2020), Mixup Zhang et al. (2018) and Cutmix Yun et al. (2019) for data augmentation. We use two NVIDIA Quadro RTX 8000 GPUs for each training. For other settings, we refer to Touvron et al. (2021).
3D pose estimation. For the classification- based model, ResNet50, we use the released implemen- tation from Zhou et al. (2018) and train the model for 90 epochs with learning rate of 0.01. For NeMo, we adopt the publicly released code Wang al. (2021) and trained the NeMo models for 800 epochs on both the synthetic and real data. Each NeMo model is trained on four NVIDIA RTX A5000 for 10 hours.
C MORE DETAILS ABOUT GENERATING 3D-DST SYNTHETIC DATA
The 100 classes we used in main classification experiments (Table 1) follow the 100 classes in (Tian et al., 2020). To demonstrate the scalability of our approach, we randomly select another 100 classes and report 200-class classification results in Table 2. We collect 30 CAD models for each class from the ShapeNet dataset (Chang et al., 2015) and the Objeverse dataset (Deitke et al., 2023b), and 2,500 images were rendered for each class. We sample the object viewpoint with uniform distribution over the azimuth angle and Gaussian distributions over the elevation and theta angles. The viewpoint sampling rules detailed in Table 5. For pose estimation, we estimate the mean and variance of the object viewpoints from the training data of PASCAL33++ (Xiang et al., 2014) and ObjectNet31) (Xiang al., 2016) and sample from the fitted Gaussian distributions.
D ABLATION STUDY EXPERIMENTS ON DATA GENERATION
Besides the results in Table and Table 4, we consider the following ablation study experiments on different data generation methods. In the following introduce different data generation procedures.
14

---

Published as a conference paper at ICLR 2024
Sample images for each data generation approach are shown in Figure 8. Experimental results presented in Table 6.
Diffusion-based generation conditioned on edges obtained from ImageNet images ("ImageNet edges"). Instead of using edges produced from our 3D visual prompt module, the images are generated with the same diffusion model but conditioned on edges computed from ImageNet images with the Canny edge detector (Canny, 1986).
Rendered images with random 2D image backgrounds (frendering BG2D"). We overlay the rendered objects to a randomly sampled background image. The backgroumd images come from the training split of BG-20k (Li et al., 2022) dataset with no salient objects.
Rendered iamges with random 3D environment backgrounds ("rendering BG3D"), We collected 100 HDRIs from polyhaven and used them as the world environment in Blender rendering. The 100 HDRIs cover wide range of scenes, including both indoor and outdoor, natural and urban, and different lighting conditions (day or night).
E EXPERIMENTAL RESULTS ON 3D OBJECT DETECTION
To show the flexibility of our 3D-DST approach and its potential for wide range of 3D tasks, present qualitative and quantitative results on 3D object detection from 3D scenes on ARKitScenes dataset (Baruch et al., 2021; Brazil et al., 2023). In the following, we start by presenting our DST-3D data generation for 3D scenes with multiple objects. Then we visualize qualitative examples in Figure 9. Finally report the quantitative results in Table 7. Results show that (i) our DST-3D approach can generate synthetic scene images with multiple objects from different categories, and (ii) state-of-the-arr models pre-trained on DST-3D data can achieve improved performance for 3D object detection in terms of both AP2D and AP3D.
DST-3D data generation for 3D scenes. Unlike 2D image classification or 3D pose estimation, generating 3D scene images for 3D object detection consists of the generation of multiple objects from multiple categories in reasonable 3D scene structure. Therefore, we adopt the scene structures from the ARKitScenes dataset (Baruch al., 2021) and extend our 3D visual prompt module (see Figure 2) follows. In ARKitScenes, each 3D scene is represented by a list of 3D bounding boxes, each associated with a category label. We start by sampling CAD model from the target category and putting it in the Blender scene the target location and with the target rotation. The synthetic 3D scene possesses a reasonable scene structure (object location and rotation) with randomness inherent in the CAD models (object styles and shape ratios). Finally, sample different camera locations and rotations to generate a series of images of the same scene but from different viewpoints.
Qualitative examples of 3D synthetic scenes generate with our DST-3D. Figure we visualize several qualitative examples of scene images produced by our DST-3D. We can see that our DST-3D approach can generate scene images with multiple objects from multiple categories with diverse viewpoints.
Quantitative results of 3D object detection on ARKitScenes dataset. We choose CubeR- CNN (Brazil et al., 2023) from CVPR 2023 as the baseline model and report the 3D object de- tection performance ARKitScenes dataset (Baruch et al., 2021) in Table 7. Results show that by pretraining 3D object detection models on synthetic DST-3D scene images, we achieve improved performance in terms of both AP2D and AP3D metrics.
F ANALYZING THE QUALITY OF THE AUTOMATIC 3D ANNOTATIONS
In Table 4 and Section E, we showed that improved performance on both in-distribution data and diitt---iittibttion data can be achieved by pretraining 3D models our DST-3D data. However, it remains unclear how much of the synthetic data has accurate 3D annotations or how much noise was introduced from our DST-3D data generation.
15

---

Published as a conference paper at ICLR 2024
In this section, we first collect feedback from human evaluators and analyze the failure cases of our generation pipeline. Then, we introduce two automatic metrics to analyze the quality of the automatic 3D annotations in our DST-3D data. Lastly, we propose K-fold consistency filter (KCF) built state-of-the-arr pose estimation method that can effectively filter noisy images with inaccurate 31) annotations from our DST-3D data, hence improving the overall quality. We also present quantitative and qualitative results to support our argument.
Human evaluations. We collect feedback from human evaluators to analyze the quality of our generated dataset. Specifically, we developed Gradio app and presented the human evaluator with the textual prompt, the visual prompt, and our 3D-DST image. Then, for each triplet presented, the human evaluator would determine if the 3D-DST image is consistent with the textual and visual prompts. We visualize the collected results Figure 5 and find that around 75% of our 3D-DST images are consistent with both the textual and visual prompts.
Moreover, by analyzing the failure samples, we identify a limitation of our model to be images with challenging and uncommon viewpoints, such looking cars from below guitars from the side. Some failure samples are demonstrated in Figure 10.
Measuring the quality of 3D annotations. 3D annotations in a synthetic 3D dataset.
Two metries were considered to measure the quality of
1. Real-to-synthetic performance. Given a powerful 3D pose estimation model (Ma et al., 2022) trained on real data, we evaluate the model the synthetic dataset. A good real-to- synthetic performance indicates the consistencies between the real data and the synthetic data, well as the general quality of the 3D annotations in the synthetic dataset. However, a critical limitation of this metric is the domain gap between the synthetic and real. For instance, a synthetic dataset with higher diversity than real data, e.g., our DST-3D data with diverse LLMs, could lead to a lower real-to-synan to-stthhetttcc performance due to unseen textures, weather, poses, etc.
2. Pose consistency score (PCS). By randomly splitting the synthetic data into training sets and validation sets, the pose consistency score (PCS) is given by a model's average performance the validation sets after fitting the corresponding training sets. Since the training data and validation data share the same source domain, synthetic data with noisy 3D labels would yield a lower PCS as the model cannot recover the noises added from the image generation module. It should be noted that the absolute value of PCS is also affected by the fitting capabilities of existing 3D models and the diversity of the synthetic data.
K-fold consistency filter (KCF). We propose a simple yet effective approach to filter out noisy samples with inaccurate 3D annotations. We follow the K-fold splits and for each train-val split, we train a state-of-the-art 3D pose estimation model (Ma et al., 2022) and evaluate it the validation split. For each sample in the validation split, we regard as a noisy sample with possibly inaccurate 3D annotation if the confidence score given the 3D annotation is lower than a threshold.
Quantitative and qualitative examples of KCF To show the efficacy of the K-fold consistency filter (KCF), we present quantitative results on four categories: bus, car, and microwave. Results in Table 8 show that KCF can effectively remove the noisy samples with inaccurate 3D annotations with increases in both R2S and PCS. We further visualize samples with high confidence scores (possibly accurate) and low confidence scores (possibly noisy) in Figure 10. We find that KCF is working as expected, and most samples removed by KCF are indeed noisy and have inaccurate 3D annotations. However, we observe few samples mistakenly filtered by KCF We conjecture this is due to the diverse image space explored by DST-3D and the limited robustness of existing 3D pose estimation methods, on which KCF is built.
16

---

Published as a conference paper at ICLR 2024
200
Good Failed
Count
100
50
109220 Poraae Pon
99
G
608 580000
L oz
EG1 BOTTEEO
ELE0692 zou
PEZESEOO PEPE ESSE90E DO
tooou
PTy0u
no
oo for nos
Category
Figure 5: Human evaluation results. We collect feedback from human evaluators and find that about 75% of the generated 3D-DST images are consistent with both the textual and visual prompts. By analyzing the failure cases, we identify a limitation of our model to be images with challenging and uncommon viewpoints.
photo a taxicab speeding down a winding mountain road
photo of a park bench covered in fallen leaves
A photo of airliner parking in the airport ready for departure
Figure 6: Qualitative examples demonstrating the effectiveness of LLM prompts. We conduct ablation study experiments by comparing the quality of the synthetic images obtained from the same 3D visual prompts with (top row) or without (bottom row) LLM prompts. We show that LLM prompts provide extra information about the appearance of the foreground object or the background scene and can effectively improve the diversity and quality of synthetic images.
17

---

Published as a conference paper at ICLR 2024
90
lat-g--mmge OST (Ours)
150
30
150
30
25
180
180
210
330 210
130
Figure 7: Distribution of azimuth angles of objects in Text2Img (left) and 3D-DST (right), We visualize the distribution of the azimuth angles for 120 objects in Text2lmg (left) and 3D-DST (right). Synthetic data in Text2Img show strong pose biases propagated from the image diffusion model, while our 3D-DST dataset alleviates this issue by introducing explicit 3D controls.
Azimuth
Elevation
Theta
Classes
all
all
NO,, x x/18
airliner, beach wagon, cab, coffee mug, dining table, piano bicycle, pillow, police van, pot, school bus, warplane, bottle bench, birdhouse, ambulance, trolleybus cellular phone, laptop, mailbox, microwave, remote control washer, bag keyboard, table lamp, trash can, bathtub, couch, soup bowl printer, stove
front
all
N(0, x/18)
all
top
(00, x18)) 7/18/118
front
top
Table 5: Viewpoint sampling rules for 3D-DST generated for image classification.
In-distribution (ID)
Out-of-distribution (OOD)
Methods
DeiT-S
ConvNeXt-S
DeiT-S
ConvNeXt-S
Baseline
84.56
91.34
49.61
67.19
/ Text2Img (2023)
84.80 (10.24)
91.26 (10.08)
50.83 (11.22) 67.50 (10.3))
/ ImageNet edges
85.20 (70.64)
91.08 (10.26)
48.69 (30.92) 69.29 (12.10)
/ Rendering + BG2D
84.96 (10.40)
91.50 (10.16)
50.92 (11.31) 68 42 (11.23
w / Rendering + BG3D
87.50 (12.94) 91.66 (10.32)
54.81 (15.20) 68.1 18.17 (10.98)
w 3D-DST (ours)
88.96 (14.40) 92.18 (10.84)
56.65 (17.04) 69.69 (12.50)
Table 6: Ablation study of data generation methods on image classification. We report top-1 accuracy (%) 1m0 (ID) and ImageNet-R (OOD) using DeiT-S and Net-S. We compare the performances when models are trained purely on the target dataset or pre-trained on synthetic dataset with different data generation methods. Please refer to Section D for specifics of each data generation approach.
18

---

Published as a conference paper at ICLR 2024
pirate ship
pickup truck
park bench
keyboard
w/ ImageNet edges
Rendered BG2D
Rendered BG3D
DST-3D (ours)
Figure 8: Qualitative examples of different data generation methods for ablation study. Please refer to Section for specifics of different methods.
Figure 9: Qualitative examples of scene images produced by our DST-3D. We extend our 3D visual prompt module with 3D scene structures from ARKitScenes dataset (Baruch et al., 2021). We can see that our DST-3D can produce synthetic images with multiple objects from multiple categories with diverse viewpoints. State-of-thh-an 3D object detection models pretrained on these scene images can achieve improved performance (see Section E).
19

---

Published as a conference paper at ICLR 2024
Methods
AP2D
AP3I)
CubeRCNN (Brazil et al., 2023)
41.50
41.65
w/ DST-3D (ours)
42.34
(10.84)
42.74
(71.09)
w/ DST-3D + camera aug (ours) 42.86
(11.36)
43.19 (71.54)
Table 7: Quantitative results of 3D object detection on ARKitScenes dataset (Baruch et al., 2021). We adopt the state-of-the-an 3D object detection model CubeRCNN (Brazil et al., 2023) from CVPR 2023 as the baseline model and compare the performance with without DST-3D pretraining. Results show that by pretraining 3D object detection models on synthetic DST-3D scene images, we achieve improved performance in terms of both AP2D and AP3D metrics.
Data
R2S (%)
PCS (%)
DST-3D (bus)
65.8
59.5
w / KCF
67.1 (11.3)
66.0 (76.5)
DST-3D (car)
83.2
36.3
w / KCF
83.9 (10.7)
43.8 (17.5)
DST-3D (microwave)
87.6
61.3
KCF
88.1 1 (10.5) 66.5 (15.2)
Table 8: Analyzing the quality of 3D annotations with teal-to-synthetic performance (R2S) and pose consistency score (PCS). We show that with KCF effectively filter noisy samples with inaccurate 3D annotations from our DST-3D data, hence improving the overall 3D annotation quality. Please refer Section F for more details.
20

---

Published as a conference paper at ICLR 2024
KCF: Good
KCF: Bad
bus
car
microwave
Failed of KCF
Figure 10: Qualitative examples of K-fold consistency filter (KCF). For each class visualize samples predicted as good or bad by KCF We can see KCF effectively filter out noisy samples with inaccurate 3D annotations. However, we also observe cases where KCF filters our images with accurate 3D annotation (bottom row). We conjecture this is due to the diverse image space explored by DST-3D and the limited robustness of existing 3D pose estimation methods, on which KCF is built.
21


Reviews:
[{"review_id": "3hO5sdQdD2", "reviewer": "Reviewer_Xyyh", "summary": "This paper presents a novel method for adding 3D geometry control to diffusion models such that recognition models pre-trained on diffusion models generated synthetic data and then trained on target datasets have performance gains on classic tasks like 2D image classification and 3D pose estimation.", "questions": "see weakness", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea in this paper is neat, simple yet effective.\n- The idea is also very novel.\n- The empirical improvements on ImageNet classification and pose estimations are solid, significant, and surprising.", "weaknesses": "- In table 4, why the baseline result NeMo w/ AugMix is missing?\n- Could you discuss or ablate using other rendering types other than canny edges? Does canny edges work the best and why?\n- There is no discussion for limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aEvidPVvdB", "reviewer": "Reviewer_5ugh", "summary": "A method to add 3D geometry control into the image generation process. To achieve the goal, three key techniques are leveraged, including 2D edge maps generation with 3D annotations via rendering, text prompt generation for improving the diversity, and conditional image generation from edge maps and text descriptions. The method can be utilized as a data augmentation strategy for many downstream tasks such as image classification. Experiments can show its promising application potential.", "questions": "- Evaluations on whether the generated images are faithful to images and text conditions. \n- It would be better if more potential applications could be discussed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Adding 3D geometry control via 2D edge maps and text descriptions is interesting and reasonable. This way the generative model only needs to deal with controlling information represented in 2D images and texts. Then many existing powerful techniques can be leveraged for controllable generation. \n- The proposed method is reasonable. It can achieve plausible controllable and diverse generation results. Generated images are of good quality and well-related to edge prompts and text conditions. \n- The method can serve as a promising data augmentation strategy for many downstream tasks. It is a promising way to generate diverse 2D images with 3D information annotation.", "weaknesses": "- The technical significance is relatively limited. The problem of generating 2D edge maps from 3D models and generating text prompts from 3D CAD models can be solved by existing techniques. Though the idea is interesting, no new techniques are proposed. The overall method is rather like an application-guided strategy. Though with promising application potential, it is hard to say what general principles that can guide the research in other domains can be distilled from the paper. \n- It is not sure whether the generated images are very faithful to the edge maps conditions. For example, there is no good guarantee that the objects in the generated images are consistent with the geometry described via the edge maps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TLVcBUSG6w", "reviewer": "Reviewer_B6MH", "summary": "- The author introduces a method named 3D-DST, aimed at enhancing the comprehension of 3D objects by diffusion models.\n- This method comprises two modules: the \"3D Visual Prompt\" module, which utilizes edge maps as prompts derived from rendering, and the \"Text Prompt\" module, which extends prompt words through LLM.\n- Through experiments, the author demonstrates that the images generated by the proposed method, along with paired labels, serve as an effective approach for data enhancement or pre-training. This leads to improved performance in tasks such as image classification and 3D pose estimation across multiple baselines.", "questions": "How to define camera extrinsic matrix and whether to use class-level canonical-space as the extrinsic matrix of identity. If this is the case, there are many symmetric objects whose poses are ambiguous (this issue has been extensively discussed in the work on object 6dof estimation). How to define objects with multiple symmetry axes such as round tables? In addition, how to align the definition of extrinsic coordinate systems between different classes?\nIt is counterintuitive to claim that edge maps are superior to depth maps because depth maps provide more 3D information, such as occlusion relationships, which goes beyond the 2D representation of edge maps. The conclusions presented in the author's paper are difficult to support with only a few selectively chosen qualitative examples.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is easy to understand.\n- The paper presents an approach that incorporates edge maps as additional prompts to enhance the performance of the diffusion-based method.", "weaknesses": "- The framework is mainly inherited from Controlnet, so the technical contributions are limited and incremental.\n- The idea of 3D Visual Prompt via CG rendering and LLM Prompt is more like a combination of multiple previous effective techniques.\n- The author's excessive focus on introducing background knowledge of known technologies like diffusion or cross-attention is unnecessary if the method utilized in this article relies on off-the-shelf approaches. It is not recommended to extensively discuss these technologies in the main text.\n- The second challenge, \"simple text prompts,\" seems to be less directly relevant to the paper's introduction on adding 3D geometry control to diffusion models.\n- The experimental part of the paper lacks details on training the network.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HkAf7DCsjE", "reviewer": "Reviewer_eSpe", "summary": "This paper investigate how to use generated images from diffusion models to improve discrimitive tasks in both in-distribution and out-of-distribution settings. The authors propose to render 3D data to 2D edge maps, fine-tune the large-scale diffusion model via ControlNet approach with the prompt augmented form LLM. After training, the generated images naturally have all the 3D object annotations. These generated data can be subsequently used as data augmentation for downstream tasks. The paper demonstrate the effectiveness of the proposed method on image classification and 3D pose estimation tasks.", "questions": "What is the prompt to LLM for enriching the description?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed framework is simple and straightforward to use. The main technical contribution seems to be how to better use 3D data, ControlNet and LLM for data augmentation.\n- Quantitative improvement looks promising. On the evaluated task (image classification and pose estimation), the quantitative improvement seems quite obvious, espcially for the OOD settings.", "weaknesses": "- The main method aims to produce (image, 3D annotation) pairs. Then is 2D image classification a good task for evaluation? The corresponding 3D ground truth is not used anywhere in this task. And you don't need 3D data to create (image, label) pairs. Even though experiments indeed show the improvement, I doubt this could be achieved without using 3D data.\n- As the main purpose is to use the generated data for downstream tasks, I think the paper needs to carefully examine the data quality and show the necessity of the proposed approach. From this aspective, some necessary ablations are missing. \n  - One simplest baseline to use 3D data is to just use the rendered image with background (e.g., random environment map). Would this kind of synthetic data improve the evaluated tasks? I think this baseline is needed to prove the necessity of using a image generative model.\n  - For image classification, a simpler approach is to just use the imagenet images, extract the edge map and then generate new image conditioned on the LLM prompt. Would this kind of synthetic data also give big improvement? This baseline is needed to show the necessity of using the 3D data, as least for the image classification task.\n- The title is a bit misleading. It seems to suggest a method that enable 3D control of the diffusion model (e.g., changing view points), but it's not. The proposed method merely use the 3D data and diffusion model to create (image, 3D annotation) pairs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)