Scalable 3D Captioning with Pretrained Models
Authors: Tiange Luo
Chris Rockwell
Honglak Lee
Justin Johnson
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: 3D Captioning
Text-to-3D
Multi-Modal Learning
TL;DR: We introduce Cap3D, an automatic approach for generating descriptive text for 3D objects, by leveraging pretrained models in captioning, alignment, and language to consolidate multi-view information.

Abstract: We introduce Cap3D, an automatic approach for generating descriptive text for 3D objects. This approach utilizes pretrained models from image captioning, image-text alignment, and LLM to consolidate captions from multiple views of a 3D asset, completely side-stepping the time-consuming and costly process of manual annotation. We apply Cap3D to the recently introduced large-scale 3D dataset, Objaverse, resulting in 660k 3D-text pairs. Our evaluation, conducted using 41k human annotations from the same dataset, demonstrates that Cap3D surpasses human-authored descriptions in terms of quality, cost, and speed. Through effective prompt engineering, Cap3D rivals human performance in generating geometric descriptions on 17k collected annotations from the ABO dataset. Finally, we finetune Text-to-3D models on Cap3D and human captions, and show Cap3D outperforms; and benchmark the SOTA including Point·E, Shape·E, and DreamFusion.

Scalable 3D Captioning with Pretrained Models
Tiange Luo! Chris Rockwell
Honglak Lee 1:27 Justin Johnson
'University Michigan
2LG Al Research
Abstract
We introduce Cap3D, an automatic approach for generating descriptive text for 3D objects. This approach utilizes pretrained models from image captioning, image-text alignment, and LLM consolidate captions from multiple views a 3D asset, completely side-stepping the time-con time-consuming and costly process of manual annotation. We apply Cap3D to the recently introduced large-scale 3D dataset, Objaverse, resulting in 785k 3D-text pairs. Our evaluation, conducted using 41k human annotations from same dataset, demonstrates that Cap3D surpasses human-ane -uthord descriptions in terms of quality, cost, and speed. Through effective prompt engineering. Cap3D rivals human performance generating geometric de- scriptions 17k collected annotations from the ABO dataset. Finally, we finetune text-to-3D models on Cap3D and human captions, and show Cap3D outperforms; and benchmark the SOTA including Point-E, Shap-E, and Dream Our data, code, and finetuned models can be found at https:s:/// / cap3d-up gi thub. io/
3D model sakura soft with purple and yellow gradient, Japanese writing, and purple flowers.
model of blue grand pismo with spikes and sharp teeth resembling shark mouth.
model of yellow Pikachu- themed Pokémon ball with black and gold stripe and lightning bolt.
3D model of metal cuhe featuring a skull, pizza, and various stickers.
3D model of Notre Dame Cathedral, Gothic cathedral with spires Paris.
Loki bust 3D model featuring green and yellow horned helmet.
3D model featuring basketball hoop, ball, racque, bowling ball, stand, and pin.
model of purple and green Halloween spider bowl metal stand, containing purple liquid.
3D model an anmored character with purple homs and spikes on the back.
model of a robotic scorpion with multiple arms and
cluster of five glass sphere light bulbs suspended single thin wire.
L-shaped sectional sofa chaise, U-shaped backrest, curved armrests, and a footstool on one side.
Figure 1: Cap3D provides detailed descriptions of 3D objects by leveraging pretrained models in captioning, alignment, and LLM to consolidate multi-view information. Two views of 3D objects are shown here, Cap3D uses eight. Additional examples are available in Appendix B
joint first authorship: equal advising
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Table 1: Cap3D is better, cheaper, and faster than crowdsourced annotation. Use 36k responses across 22k objects for A/B testing 8A40s on a cloud platform for speed and cost computations.
A/B Human Testing Win % (Tie %)
Cost per
Annotation Speed
Ik Objects Cost Breakdown
Method
Ik Objects
BLIP2
$3.79
CLIP
$0.38
Human
37.8% + 0.5% (9.5%) 52.3% % + 0.5% % (9.5% ()
$87.18 $8.35
1.4k / day 65k / day
GPT4
$4.18
Cap3D
Cup3D Total Cost
$8.35
Introduction
Text-conditioned 3D synthesis [H33 could revolutionize the creation process of 3D assets, impacting
various sectors, including 3D design, virtual reality 4 film [3], robotics 66 77, and autonomous
driving (8). However, challenges persist, namely the high cost of 3D asset creation and the scarcity of high-quality captions for 3D assets. Objaverse [9] takes a step towards this as the first public large- scale 3D object dataset. Unfortunately, while objects contain paired metadata, these do not serve as informative captions, as shown in Table 3. contrast with 3D, a plethora of high-quality text-image paired data is publicly available [10.14] This data has led to incredible recent progress in image-text
learning [15 text-conditioned image synthesis [19 and image captioning [9]
In this work, we present Cap3D, method to automate 3D object annotation. Our key insight is to leverage the abundance of knowledge in pretrained image-text models remedy the lack of existing 3D-text data. The core of our data collection process is to apply an image captioning model (BLIP2 [29]] ) to a of 3D asset renders, use an image-text alignment model (CLIP [66 to filter captions, and apply language model (GPT4 (30) to fuse the filtered captions across views. Critically, the models we apply are pretrained on varied and large-scale text-image [T 33)
and text (34). data; and approach complementary problems. As a result, each model adds additional value to the framework, as we show in Table
Cap3D is agnostic to 3D asset sources and can be effectively scaled to larger extents with increased 3D assets and computational resources. In this paper, we apply it primarily to Objaverse, gathering dataset of 785k 3D-text pairs. Through object rendering and captioning, we enable ethical filtering of 3D objects via both image and text, as detailed in $ 3.2 We publicly release all of our collected data including automated and numan-annotatie Rendered Images, at /huggingface. co/ htt//aaaaets//eee ange/ CCap3D The dataset is released under ODC-By 1.0 license. We also released trained models and code for replicating the benchmark table.
captions, along with associated Point Clouds and
We validate our collection approach by collecting over 50k crowdsourced captions on 40k objects. We conduct human evaluations and show on Objaverse that our automated captions are superior crowdsoureed captions in quality, cost, and speed (Table [ details in Appendix (A). Specifically, it is preferred 35% more often by humans, costs more than 10 times less, and is over 40 times faster, assuming only 8A40 GPUs. We also test the limits of automated captioning. We consider a separate task of captioning geometry (as shown in Figure [bottom-right) using ABO, a dataset of 3D models with complex geometries [35). Shown in Table 4 our automated captioning underperforms humans. However, by formulating description as a question answering task (detailed in I. we show stronger performance compared to crowdsoureed workers. This result shows the ability of our method to adapt beyond traditional captioning and still be highly competitive.
Finally, high-quality gathered 3D-text dataset enables to train and validate large-scale text- to-3D models. In $5.3] we evaluate several sate-of-thhe-rt methods on Objaverse out-of-the box, including Point-E, Shap-E, DreamFields, and DreamFusion. Finetuning on our data typically shows meaningful improvements, demonstrating the value of the collected dataset. In addition, we show our automatically collected captions yield better finetuning performance than human captions even at the same scale. At full scale, finetuning is further boosted.
2 Related Work
Obtaining 3D-text pairs at scale is challenging, and we take inspiration from image-text datasets and methods when approaching this task.
2

---

Image-Text Data and Modeling, Early image captioning 36 38 and text-image representation
learning methods [39 41 were built using CNNs 42 and LSTMs 45]
46 leveraging human- 33 47 Text-to-image methods used similar datasets, and relied on GANs [48]
annotated datasets 331
[49] and VQVAEs [9]
The advent of semi-automated image-text collection has enabled successful scaling of datasets [T0+14] and models [2))28] Transformer-basee
50.
architectures [16
53 [4] and diffusion models (55H60] have scaled best to large data; we employ transformer-based
methods through our captioning process and adopt diffusion models for text-to-3D experiments.
Training models upon large datasets and using the corresponding trained models to filter larger data has led to datasets of rapidly increasing size 13 14]. In addition to filtering, trained models have
been used annotate new data with high-quality [61] We take this approach, captioning rendered
views with BLIP2 [29]. refining with CLIP [6 62 and summarizing with GPT4 [63]; all of which
are trained on large datasets, including
33
Concurrent works [64 666 use automated
captioning on 2D images using an older system
67 or based upon metadata [66 68
3D-Text Data and Modeling. Until recently, 3D data was of relatively small scale (~ 50k objects) 769-72). Labeled 3D-text data was scarce, relying human annotation, and typically limited to ShapeNet [69] chairs [73] or tables and chairs (74 and ScanNet (76] (7).. This enabled prior work
undertake the task of 3D captioning (78-80 or text-to-3D 74
78 844 small scale. Methods
that approached text-to-3D would sometimes avoid 3D supervision entirely 85/77, leading to slow generation due to many optimization steps. We annotate small-scale dataset containing 3D furniture, ABO 35 to evaluate the ability of Cap3D to specify fine- ome-grained geometry.
Objaverse 9 introduced diverse set of objects over 10 times the size of the prior largest public 3D dataset (69. This data is our primary captioning focus, and we associate a single caption with each object in Objaverse after filtering. Concurrent works [66 [8 gather text associated with Objaverse, but do not fuse captions across views 88 or rely upon metadata [66]. and do not approach text-to-3D.
The concurrent studies 3DGen learns text and image to 3D on Objaverse: Point-E 89 and Shap-E [90] learn text-to-3D models on a large-scale 3D dataset, but none have fully disclosed their code data. Point-E involves two variants and released text-to-3D model and coxt-to-image-uee model by finetuning GLIDE [3] and training an image-to- poino cloud diffusion model [9]]. Other recent works 992 93 also focus scaled image-3D generation. We show finetuning our captions improves Point-E performance despite having already been trained on large amounts of Internet data.
3 Method
3.1
Captioning Process
Our task to produce a single descriptive caption given 3D asset. Our proposed method, Cap3D, employs a four-step process. First, we render set of 2D views for each 3D object. Next, apply image captioning to achieve preliminary descriptions. As these captions may contain inaccuracies, an image-text alignment model, CLIP, is introduced in the third step rectify errors. Finally, an LLM is employed to unify captions from various perspectives, creating a comprehensive caption. This process is shown in Figure 2 and detailed below.
We render using Blender 512x512 from M 8 high-information camera angles rotating horizontally around the object, with two slightly below and the rest slightly above the object, to cover all the object details. The reason we prefer multiple views a forward-facing view may miss self-occluded object details (e.g. Figure [ro 1) or face strange appearance and/or lighting. In contrast, multiple views will see much of the object from different viewpoints, increasing the number of chances for captioning model to predict objects in detail. For instance, in Figure 2 the back view 1 identifies the "yellow handle", which is barely visible in forward view M.
Object Rendering:
Image Captioning: We use BLIP2 29 for captioning, selecting the largest pretrained model
adapting VÝT-G [54] 94 image encoder and FlanT5XXL (95] text encoder. We generate N captions per rendered image using nucleus sampling [96]. By generating multiple captions, we increase the likelihood of generating correct details (e.g. "black and yellow toy bomb" in Figure view M caption 1). Incorrect captions, such as "scissors" in Figure 2 view M caption N, can be filtered in later stages. To generate captions containing fine-grained geometry details (in our ABO experiments), employ two-stage question- question-answerts answering instead of captioning. The first stage
3

---

View
A yellow black bomb with yellow handle
View 1. yellow and black homb with yellow handle
Prompt:
descriptions about the same ID object ... distill these descriptions into one concise caption:
BLIP2
: N. 3d model of bomb with yellow it
View a yellow and black toy ball with straw sticking out of it
GPT4
View M
1. A black and yellow toy bomb background
View M. A black and yellow toy bomb grey background
AD model of yellow und black bomb with a handle and a straw
Input Object
BLIP2
:
CLIP
N. 3d model of bomb with scissons
Output Caption
Figure 2: Overview of Cap3D. Left to Right: (1) Render 3D objects from M = 8 camera angles to capture object details (2) Generate N = 5 image captions per rendered image using BLIP2; (3) Select one caption for each image based on its similarity to the image encoding using CLIP; (4) Use GPT4 to consolidate all selected captions into a final, summary of the object.
generates one answer to prompt asking what object is pictured. The answered object is passed into a second prompt, which asks its structure and geometry, and generates answers.
While BLIP2 often generates high-quality captions, not uncommon for samples contain mistakes, particularly in non-forward facing views such as "yellow cup", in Figure 2 view 1, caption N. To reduce the frequency of mistakes, we compute CLIP [16] ViT-B/32 (54]
Caption Selection:
encodings from each of captions and the associated image, and select the caption maximizing cosine similarity. CLIP tends to select good captions for each view, e.g. Figure 2 view 1, BLIP2 caption 1 and view M, caption CLIP complementary to BLIP2 as not only does it have different training details and architecture, but it trains on different data. While BLIP2 is trained upon COCO [31].
Visual Genome [32]. CC3M M.] CC12M [12) SBU 33 and LAION400M [3]: CLIP trained upon dataset of 400M images based on frequent text occurrence in Wikipedia.
Accumulating information across viewpoints form complete picture of 3D objects is challenging, but crucial. We find prompting of GPT4 6 to summarize the M captions results in good parsing of the details across captions. By applying GPT4 the final summary step, it can both include significant details and remove unlikely ones. For example, the final caption in 2 filters the incorrect information, from view "toy ball", while keeping key details, including "handle" and "straw". The alternative order of GPT4 followed by CLIP would result in (1) GPT4 having to make sense of more incorrect input details and (2) CLIP simply selecting between aggregate captions instead of being able to error-correct small mistakes. The effectiveness of introducing GPT4 verified in ablations (Table 3).
Caption Consolidation:
3.2 Ethical Filtering
Captions generated and images rendered by Cap3D enhance the identification and mitigation of legal and ethical issues associated with large-scale 3D object datasets, including identifiable information and NSFW content.
We manage two datasets: Objaverse and ABO. In Objaverse, our main responsibility involves dealing with artist-created assets. These include identifiable elements such as human face seans and NSFW objects. Objaverse contains approximately 800k objects, which makes the manual verification of each asset impractical. The ABO dataset, on the other hand, is smaller and mostly consists of furniture. We manually ensure the ethical integrity of this dataset.
We exclude objects that lack sufficient camera information for rendering, leaving with 785k objects. These objects encompass a mix of non-commercial and commercial licenses, such as CC BY-NC-SA. CC B BY-NC, CC BY, CC BY-SA, and CCO. We then applied ethical filtering exclusively to objects with commercial-fe -riieddl licenses (i.e., CC BY, CC BY-SA, and CCO), resulting in a count of 680k objects.
We next follow prior work 10 and use face detector 97 and NSFW classifier 98 99 on forward-
facing object renders and filter detected objects with score >= 0.9. The face detector filters out 18.6k

---

Metadata
Human
24
Metadata
Flower.
A plant with white flowers
% captions
18
Human
Cap3D
Cap3D
3D model of branch with white flowers and green leaves.
0
5
10
15
20
25
30
Number words in caption
Figure 3: Objaverse Caption Comparison. Human captions and Internet metadata frequently contain limited detail. Cap3D captions typically have longer length and more detail. Vocabular size comparisons among Cap3D), BLIP2, and human captions are included in Appendix
objects, and the NSFW classifier filters out 217 objects. Text is also carefully processed. Our final cap- tions are the output GPT4, which has been trained to filter out inappropriate or harmful content [63 We run standard blocklist [TOO] its output, remov- ing any object-caption pairs including blocked words. This filters 226 objects. After all the filtering, are left with 661k objects in the Objaverse dataset. We manually estimate detection precision and recall in Ta- ble [ To summarize, our process detects over 19k objects, of which nontrivial amount is accurately re- moved. We estimate roughly 1k face and less than Ik NSFW are missed, using conservative standard (e.g. missed faces are typically sports cards), resulting in 660k subset.
Table 2: Ethical Filtering Analysis. We manually detect faces and NSFW content validate automated filtering. 16 of 17 missed face detections were sports cards.
Detected Precision
Missed dets.
(Filtered)
5k (%)
10k
680k
Faces NSFW
18.6k
790
16%
17
#1k
217
102
47%
12
<1k
Language
226
1: String match filtering is deterministic.
4 Dataset
collect captions in two distinct settings: Objaverse, large and varied dataset of artist-created assets; and ABO, small dataset of real products, typically furniture.
4.1 Objaverse Captions
Objaverse 9 features roughly 800k 3D object assets across 2 classes designed by over 100k artists. is of significantly larger scale than prior work: the paper shows this size enables more diversity by generative 3D models trained upon it. It is released under the ODC-By 1.0 license, permitting subsequent researchers to curate new data from Metadata paired with many assets, however as seen in Figure 3 (right), metadata caption length is frequently short or empty. We collect two caption datasets on Objaverse. First, an automated set of one caption for each of 785k objects using Cap3I> (a total of 785k captions). Second, crowdsourced set of 41 .k4 captions spanning 39.7k objects for evaluating generated captions. Captions are collected using thehive.ai a crowdsourced platform similar to AMT. Workers given instructions with gold-standard sample captions, see the same views as models during captioning, and are routinely monitored. Poor captioning performance results in ban and deletion of the worker's captions. Crowdsoureed captions also filtered using the blocklist in $3.2) Figure (left) shows human captions provide more detail than metadata, but automated captions tend to be most descriptive.
4.2
ABO Geometry Captions
ABO 35 is a collection of 3D models of Amazon products and is primarily furniture. ABO serves as an important contrast to Objaverse as it consists of a small number of classes varying primarily in geometry. Captioning, therefore, needs to focus more on structure as opposed to semantic category. To emphasize this focus, we consider the task of captioning the geometric structure of objects without color or texture (seen in the bottom right of Figure T). Like Objaverse, ABO contains metadata that is
5

---

Metadata
Human
24
A three seater sofa with low backrest the height the urmrests.
Metadata
Bed.
captions
18
Human
Cap3D
12
Cap3D
Cap3D
Cap3D (QA)
Thrre-seater sofa with a slender, curved backrest and armrests.
3D rendering of a couch.
0
0
5
10
15
20
25
30
Number words caption
Figure 4: ABO Automated Geometric Description. Left: Human descriptions provide more detailed geometry than automated captions. With careful prompting. Cap3D (QA) match human- level detail. Right: The high peak of Metadata is cropped, which otherwise obscures other curves.
typically quite short (Table 4) resulting in limited detail. We collect three sets of captions on the 6.4k ABO splits of 78: crowdsourced (a total of 17.2k captions), captions generated by Cap3D (a total of 6.4k captions), and captions generated by Cap3I) (QA) which uses the two-stage prompt captioning (a total of 6.4k captions). Crowdsourced captions follow similar detail to Objaverse with the exception instructions and examples are focused on geometric structure. We compare alternatives Figure 4 In contrast to Objaverse, human geometric descriptions on ABO are more detailed than captioning. With prompting (QA), the Cap3D pipeline can rival human descriptions.
Experiments
In this section, we first validate the quality of Cap3D captions against metadata and human-authored captions on both Objaverse and ABO. To verify Cap3D captions are helpful in practice, we next compare text-to-3D models finetuned on both >30k as crowdsourced captions). Finally, we evaluate captions and Cap3D (using the same captions at scale to measure if finetuning on our captions can improve performance.
text-to-3D models on our
5.1 3D Captioning Objaverse
Dataset. We evaluate caption quality on three subsets Objaverse: (1) a random set of 22k objects containing human caption, (2) random split of 5k objects containing human caption, and (3) random 5k split across the entire dataset.
Baselines. In data splits (1) and (2), we compare the caption generated by Cap3D with human- authored annotations, Human, and existing Objaverse metadata, Metadata, described in $ 4.1 Split (1) is used for A/B testing of Cap3D vs. Human, as shown in Table [ at scale. Collecting A/B comparison is expensive, so we compute more extensive experiments on the smaller set (2) in Table 3
In data split (3), ablate the main components of Cap3D into BLIP2 and +GPT4, BLIP2 uses only the image captioning component of our method, taking a front-view rendering and producing single output caption. +GPT4 uses the same image captioning process of our method, producing 5 captions for each of 8 views. However, instead of using CLIP to filter 5 captions from each view, it directly summarizes all 40 captions into final caption.
Metrics. Our primary metric is human judgment A/B tests, where ask workers to select between two captions on a scale of 1-5, where is a tie. Workers are carefully monitored and each comparison has at least 10k observations across 5k objects. We report mean score, along with the percent each method is preferred (i.e. scores a or 5). We use automated metrics CLIPScore [66 62]. I the cosine similarity of CLIP encodings with input images; and ViLT Image and Text Retrieval, which ranks likely image-text pairs, from which one computes precision.
We emphasize CLIPScore is not our primary metric since our captioning model utilizes CLIP. BLIP2 utilizes ViT-I/14 and ViT-g/14, while our filtering uses ViT-B/32, so following previous work (85) compute CLIP score using different model reduce bias (ViT-B/16). However, we report it

---

Table 3: Objaverse Captions Evaluations. Cap3D outperforms human and Metadata, BLIP2, GPT4, and CLPP are all important to performance. We report 95% confidence interval and use 5k objects.
User A/B Study vs. Cap3D
CLIP Score
ViLT Img Retr.
ViLT Text Retr. R@5
Method
Score (1-5)
Win %
Lose %
R@5
R@ 10
R@ 10
Metadata
1.74+-00026 2.86+-0.026
10.7 + 0.7
83.8 + 0.8
66.8
4.3
6.3
6.1
8.5
Human
37.0+1.0
46.1+1.00 1.0
72.5
21.2
29.0
18.5
24.9
Cap3D
88.4
35.7
46.3
34.7
44.2
BLIP2
2.87+ 0.019
41.0% 0.7
50.61 0.7
83.1
24.7
32.3
21.9
29.3
GPT4
2.94+ 0.015
35.2+ 0.6
40.8+ 0.6
86.3
31.9
39.9
30.2
38.4
CL.IP (Cap3D)
86.9
31.1
40.2
30.3
38.6
BLIP2
GPT4 A 3D model of house with
24
A 3D model of a house gray background.
BLIP2
roof, garage, grass, trees, and
captions
18
GPT4
green field, featuring knifé and pair of scissors.
CLIP (Cap3D)
CLIP (Cap3D)
6
model house with a garage, roof, grass and trees.
0
0
5
10
15
20
25
30
Number of words in caption
Figure 5: Objaverse Caption Ablations. GPT produces longer and more detailed captions than BLIP2; CLIP tends prune incorrect details and reduces length slightly.
has shown a higher correlation with human judgments than other automated metrics 62
ViLT 101
trained on different data and a different architecture than CLIP, providing an orthogonal metric.
Results. We report large scale A/B testing (1) against Human in Table [. which shows Cap3D is better across metrics, with high confidence. The top three rows of Table 3 use the smaller human- captioned split (2), and demonstrate Cap3D's superior performance over Objaverse metadata and human-authored captions across A/B studies and automated metrics. The bottom three rows of Table [ studied across random split of the full dataset (3), reveal that while BLIP2 effective, incorporating multiple views with +GPT4 enhances performance. As shown in Figure [3 GPT4 adds detail by consolidating view-specific information. Filtering using + +CIJP (Cap3D) mitigates false details by purging subpar captions from GPT input. In addition to reducing errors, utilizing CL.P also reduces GPT input captions from cost reduction from $15.33 to $4.18.
effectively decreasing token numbers and facilitating a
5.2 Geometry 3D Captioning on ABO
comparing Cap3D captions for each object against a maximum two human-authored ones. To emphasize geometric focus, images used for model input and human assessment are cexture-free and colorless.
Dataset. We evaluate geometric captioning on 6.4k object split from ABO 35 78
Baselines and Metrics. We use two automated variants from $3.1 Cap3D and Cap3D (QA), which a two-stage prompt captioning ask more about the input 3D geometry; and compare to crowdsourced human descriptions, Human, detailed in 44. and ABO metadata, Meta.
Our primary metric of comparison is similar human A/B testing to $..1 since automated metrics such as CL.PPcore do not accurately represent the distance between fine-grained captions and images shown in [78).
Results. In stark contrast to Objaverse, Human captions beat automated (Cap3D) in Table 4 Automated captions alone contain little geometric detail (e.g., Figure 4. making Cap3D unsuited for this setting. However, by using the two-stage prompt engineering, Cup3D (QA) is preferred to Human. Shown Figure Cap3D (QA) produces significant fine- fine-grained geometric detail as well as longer captions in general. In contrast, Metadata clearly the weakest bascline.
7

---

Table 5: Text-to-3D: Human Captions. Cap3D
captions are better than human on the 30k set.
Table 4:
ABO Fine-Grained Geometry Cap- tions. Cap3D (QA) performs best; crowd-
Finetuning on Cap3D full set performs best.
Finetune
CLIP
CLIP
Precision (2k)
sourced beats captioning alone.
Dataset
FID
Score
R@
R@S
R@ 10
Method
A/B
A/B
A/B Lose %
Pretrained
36.1
72.4
6.0
16.2
22.4
Score (1-5)
Win %
Stup Poin
30k (Human)
34.6
74.4
8.2
21.3
29.1
Human v. Cap3D
3.09:00.02
47.3+:%%
41,4=19
30k (Cap3D)
33.7
75.0
10.4
24.3
32.1
Cap3D(QA) v. Human
3.08:=0.02
50.24:%%
44.0= 1%
350k (CapāD)
32.8
75.6
12.4
28.1
36,9
Cap3D(QA) Cap3D
3.274-0.2
56.0+:%
37.4+1%
Pretrained
37.2
80.4
20.3
39.7
48.7
Cap3D(QA) v. Meta
4.27:10.02
88.2::.%
10.0 1%6
30k (Human)
36.0
79.6
18.6
36.3
45.3
30k (Cap3D)
37.2
79.4
19.1
37.5
46.1
350k (Cap3D)
35.5
79.1
20.0
38.8
47.3
5.3 Large-Scale Text-to-3D Generation
Dataset. We evaluate text-to-3D generation on three subsets of Objaverse: (1) a 30k split of objects containing human-authored captions, measure if finetuning on Cap3D captions outperform human- authored ones; (2) a 350k split of Objaverse objects paired with Cap3D captions, for finetuning state-of-the-an text-to-3I methods- obtaining high-density point cloud and latent codes to finetune Point-E and Shap-E E for all 785k objects is prohibitively expensive (20k GPU days); and (3) a 300 object split for optimization- based baselines, which typically take >30 mins per object to optimize. Pretrained and Finetuned models are evaluated on 8 views across a held-out test set of 2k objects.
Methods. We consider several recent SOTA methods in three general categories: text-to-3D diffusion, We use the direct [03]
3D cascaded text-to- toiagee-mmee then image-to -t- diffusion, and optimization-base text-to-3D variant of Point-E E (89, as well as two variants of Shap-E E [90]: STF [002] and NeRF
We use Stable Diffusiio cascaded with Point-E: (Im-to-3D), adapting ControlNet [64] and LoRA 104
for Stable Diffusion finetuning. We use aptimization-basee baselines DreamField (85), the publicly available implementation of DreamFusion 3. Stable DreamFusion
[105]; and 3()Fus [06], using
their implementation based on Karlo 24
Metrics. We use standard metries from prior work 3
90] to evaluate. Primarily, these are CLIP Score and CLIP R-Precision. CLIP R- R-Precision ranks rendered image against all text pairs in the test set by CLIP cosine similarity, and computes precision upon true text-image correspondence. Since we have ground truth images, we calculate the FID [108] of 3D rendered images against ground truth images, as well as assess CLIP Score on these reference images. We also use ViLT Retrieval R- R-Precision, used in 5.1 which has the same evaluation procedure as CLIP R-Precision with different model.
85
Results. Table 5 lists the results of finetuning using numan-authoree and Cap3D captions. Point-E improves after finetuning upon human captions. However, performance is further improved using our captions on the same dataset; and improved most by training upon the full dataset. This result strongly defends Cap3D captioning at scale. Shap-E E does not improve on CLIP metrics after finetuning in any dataset, but performs the least bad on the full dataset using our captions; and FID improves most.
Table 6 presents results from several state-of-the-art pretrained and finetuned models using Cap3D- generated captions. The models finetuned on our captions generally outperform pretrained models under the FID metric. For CLIP-related metrics, the finetuned models of Point-E E (Text-to-3D) and StableDiffusson + Point-E (Im-to-3D) also beat their pretrained counterparts. Point-E and Stable Diffusion have been trained on massive datasets, so improvement from finetuning is strong evidence Cap3D captions are effective. The observed downturns in Shap-E E could be attributed at least two factors. First, our replication of their loss during finetuning. We restart from earlier checkpoints upon crashing, but the result alone is concerning. Second, we exclusively finetune the diffusion model in Shap-E's two-stage approach.
train code is unstable, often resulting in NaN
Qualitative results in Figure 6 validate quantitative findings. Point-E and Stable Diffusion baselines show large improvements from finetuning, while Shap-E can better fit the Objaverse data distribution (corresponding to improved FID).
8

---

Table Text-to-3D on Objaverse. Finetuning improves FID over pretrained performance across
models. CLIP metrics of Stable Diffusion increase; CLIP metrics of Point-E E increase significantly.
Pretrained
Finetumed on Cap3I>
CLIP Score
CLIP R-Precision (2k) R
CLIP
CLIP R-Precision (2k)
FID
FID.
R@1
R@ 10
Score
R 1
R
R@ 10
Ground Truth Images
81.6
32.7
55.1
64.3
81.6
32.7
55.1
64.3
Point-E (Txt-to-3D)
89
36.1
72.4
6.0
16.2
22.4
32.8
75.6
12.4
28.1
36,9
(CNet)
73.6
53.3
74.6
12.4
26.2
S. Dift. 22
(LoRA) 104
19
(Im-to-3D)
54.7
73.6
11.0
23.4
30.0
53.7
74.4
11.6
24.6
31.4
Shap-E (STF) 102
37.2
80.4
20.3
39.7
48.7
35.5
79.1
20.0
38.8
47.3
Shap-E
(NeRF)
48.7
79.4
19.0
37.7
48.2
78.1
18.3
35.1
43.5
3D model of green teapot with horns.
A 3D white skateboard ramp model.
Finetuned Pretrained
Point-E S.Dif((LoRA)
S.Diff(CNet) Shap-E NeRF
Shap-E E STF
(Text-to-3D) Point-E E (Im-to-3D) Point-E (Im-to-3D)
Figure 6: Text-to-3D results. Finetuning on Cap3D) captions can significantly improve results.
Additional examples available in Appendix
Optimization baselines, shown in Table 7 per- form very well upon CLIP--based metrics, con- sistent with prior work [90]. In fact, DreamField outperforms ground truth images in CLIP met- rics. This demonstrates DreamField overfits to the CLIP metric, which is the standard protocol for text-to-3D evaluation. We propose also consider ViLT precision (see $5.1). This helps mitigate the bias of CLIP, though DreamField performance on this metric is still strong.
Table 7: Text-to-3D: Optimization Baselines. Overfitting via CLIP leads to higher CLIP--ased scores than ground truth; ViLT score is more fair.
FIDI
CLIP
ViLTT
Score
R@!
R@5
R@1
R@5
True Images
83.2
53.2
77.8
41.3
69.0
D. Field 65
106.1
83.7
61.8
83.6
32,3
$6.0
Fusion
127.8
72.4
28.4
45.3
3DFuse 106
91.1
77.0
38.6
58.5
26.3
$3.0
Limitations and Future Works
As described in 3 Cap3D consists four steps: (1) 3D objects rendering: (2) captioning via BLIP2: (3) filtering captions via CLIP; (4) consolidate multiview information via GPT4. To effectively capture comprehensive information through 2D renderings, cameras are strategically placed above or below objects. However, this occasionally results in unconventional 2D views, making BLIP2 susceptible to errors that CLIP fails to rectify. This, in turn, hampers GPT4's ability to merge variegated information across views, culminating in vague and verbose descriptions, as illustrated in Figure 7 requiring robust image-captions incorporations beyond the current eight.
The system also falters with certain complex indoor 3D scans, as depicted in Figure 8 thus
models (30 and potentially benefiting from additional view
Our method's provision of extensive 3D-Caption pairs for Objaverse 9 could foster the advancement of 3D-LLM models [109 110), facilitating 3D-caption centrie tasks like captioning, dialog, and language-based navigation. Additionally, the geometric descriptions generated for ABO [35) enable

---

compositional structure analysis of fine line-grained 3D objects 11 112). Our developed method assists in scaling up of 3D-text pairs for expansive 3D datasets I3
Prompt: Given of descriptions about the same object descriptions into concise caption:
model dragon on rock
3d model of fommation with dragon it
3d model of dragon mek
ad model of rock yellow blobioo it
GPT4
Output Caption
3D model of blue dragon rock formation with
surrounding elements like
a 3d model buiterlly rock
3d model of dragon rocks and waller
3d model of rock with sitting
3d rendering of rock with flowers on
butterfly, girl, water.
Figure 7: An failed case. The caption under each rendered image are generated by BL.P2 filtered by CLIP. The inaccurate content are highlighted with colors. GPT4 CLIP cannot fix the error generated by BLIP2 and result in fuzzy description.
Prompt: Given
descriptions about the same 3D object distill descriptions caption:
model of window
3d model of middle of field.
model of mud house
house with hole
GPT4
Output Caption
3D model featuring house window,
in field, a mud house, dump rusted car.
model of gray background
3d model of
3d model of gray
3d model a torn piace
background
Figure 8: An failed case. The caption under each rendered image are generated by BLIP2 filtered by CLIP. The inaccurate content highlighted with colors. The various views contain inaccurate information. The associated details, roughly described, fail to accurately depict the indoor scene.
Conclusion
In this work, we collect (1) object captions at scale, creating the largest publicly available high- quality 3D-text by order of magnitude. To do so we propose Cap3D, automated pipeline leveraging several models pretrained on large datasets, and show design choices are important to performance. In addition, we collect (2) dataset of geometric captions upon fine-grained 3D objects. This helps analyze shortcomings automated captioning and study the potential question answering, while yielding geometric descriptions for 3D assets of real objects paired with real images. These datasets serve as benchmarks for text-to-3I tasks (1) at scale and (2) geometric detail.
10

---

Acknowledgments and Disclosure of Funding
This work is supported by two grants from LG Al Research and Grant #1453651 from NSF. We greatly thank Kaiyi Li for his technical support. We thank Mohamed El Banani, Karan Desai, and Ang Cao for their helpful discussions. Thanks Matt Deitke for helping with Objaverse-related questions. Thanks to Haochen Wang for helping notice some incorrect rendering.
References
[1] Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint S.kiv:2305.0246.. 2023.
[2] Anchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas Oğuz. 3dgen: Triplanc latent diffusion for textured mesh generation. arXiv, 2023.
[3] Ben Poole, Ajay Jain, Jonathan Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion, arXiv, 2022.
[4] Yuk Ming Tang and Ho Lun Ho. 3d modeling and computer graphics in virtual reality. In Mixed Reality and
Computer Graphics. IntechOpen, 2020.
[5] Rick Parent. Computer animation: algorithms and techniques. Newnes, 2012.
[6] Afsoon Afzal, Deborah Katz, Claire Le Goues, and Christopher Timperley. A study on the challenges of using robotics simulators for testing. arXiv preprint arXi::2004.07366 2020.
[7] Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín-Martin, Chen Wang. Gabrael Levine, Michael Lingelbach. Jiankai Sun, al. Behavior- lk: A benchmark for embodied with 1,000 everyday activities and realistic simulation. Conference on Robot Learning. pages 80-93. PMLR, 2023.
[8] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference robot learning, pages 1-16. PMLR, 2017.
[9] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe annotated 3d objects. 2023.
[10] Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. Redcaps: Web-curated image-text data created by the people, for the people. NeurIPS, 2021.
[11] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-tcxt datasct for automatic image captioning. In ACL, 2018.
[12] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 3558-3568, 2021.
[13] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaezmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv, 2021.
[14] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Laion-5b 5b: An open large-scale dataset for training next generation image-text models. 2022.
[15] Mohamed El Banani, Karan Desai, and Justin Johnson. Learning Visual Representations via Language- Guided Sampling. In CVPR, 2023.
[16] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In /CML, 2021.
[17] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language- image pre-training. In ECCV, 2022.

---

[18] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene. Arthur Mensch, Katherine Millican, Malcolm Reynolds, al. Flamingo: visual language model for few-shot learning. NeurIPS, 2022.
[19] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Iya Sutskever. Zero-shot text-to-image generation. In ICMI., 2021.
[20] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. In ECCV, 2022.
[21] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorcalistic text-to- image diffusion models with deep language understanding. 2022.
[22] Robin Rombach, Andreas Blattmann. Dominik Lorenz. Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[23] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, llya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. CoRR, 2021.
[24] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv, 2022.
[25] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. ICLR, 2022.
[26] Xiujun Ii, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV, 2020.
[27] Pengehuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang. Lei Zhang, Lijuan Wang. Yejin Choi, and Jianfeng Gao. VinvI: Revisiting visual representations in rision-language models. In CVPR, 2021.
[28] Zhizhong Han, Chao Chen, Yu-Shen Liu, and Matthias Zwicker. Shapccaptioner: Generative caption network for 3d shapes by learning mapping from parts detected in multiple views to sentences. In ACM MM, 2020.
[29] Jumnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. arXiv, 2023.
[30] OpenAI. Gpt-4 technical report, 2023.
[31] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[32] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJC,, 2017.
[33] Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned photograhs.. NeurlPS, 2011.
[34]
https://ccomoncralllorgg////
[35]
Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas Yago Vicente, Thomas Dideriksen, Himanshu Arora, Matthicu Guillaumin, and Jitendra Malik. Abo:
Dataset and benchmarks for real-world 3d object understanding. CVPR, 2022.
[36] Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, 2018.
[37] Steven Rennie, Etienne Marcheret, Youssef Mrouch, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. CVPR, 2017.
[38] Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Neural baby talk. In CVPR, 2018.
[39] Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular attention network for referring expression comprehension. In CVPR, 2018
12

---

[40] Aoxue Li. Tiange Luo, Zhiwu Lu, Tao Xiang, and Liwei Wang. Large-scale few-shot learning; Knowledge transfer with class hierarchy. In Proceedings the ieee/cyf conference on computer vision and pattern recognition, pages 7212-7220, 2019.
[41] Kuang-Huei Lce, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In ECCV, 2018.
[42] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
[43] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, 2012.
[44] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI 2015, 2015.
[45] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1997.
[46] Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. transactions on Signal Processing, 1997.
[47] Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In ICCV, 2019.
[48] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. 2014.
[49] Tero Karras, Samuli Laine, Mïika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, 2020.
[50] Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. NeurIPS, 2017.
[51] Patrick Esser, Robin Rombach, and Bjom Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
[52] Ming Ding, Zhuoyi Yang. Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, al. Cogview: Mastering text-to-image generation via transformers. NeurIPS. 2021.
[53] Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations. In CVPR, 2021.
[54] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer. Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[55] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurlPS, 2021.
[56] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurlPS, 33, 2020.
[57] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. NeurlPS, 2022.
[58] Alexander Quinn Nichol and Prafulla Dhariwal, Improved denoising diffusson probabilistic models. In ICML, 2021.
[59] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. NewrIPS, 2019.
[60] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In JCML, 2015.
[61] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg. Wan- Yen Lo. et al. Segment anything. arXiv, 2023.
[62] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv, 2021.
[63] OpenAL Gpt-4 technical report. arXiv, 2023.
13

---

[64] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to texl-to-image diffusion models. arXiv, 2023.
[65] Justin N. M. Pinkney, Pokemon blip captions. httes:///wngginoaaccccoo/aassssd
pokemon- bwwpb captionst 2022.
[66] Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su. Openshape: Scaling up 3d shape representation towards spen-world understanding. arXiv, 2023.
[67] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip:
Bootstrapping language-image pre- pre-training for unified Asion-language understanding and generation. In ICML, 2022.
[68] Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martin Jiajun Wu, Caiming Xiong, Ran Xu, Juan Car- los Niebles, and Silvio Savarese. Ulip: Learning unified representation of language, image and point cloud for 3d understanding. CVPR, 2023.
[69] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li. Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, al. Shapenet: An information-res -rcch 3d model repository. arXiv, 2015.
[70] Xingyuan Sun, Jiajun Wu, Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang. Tianfan Xue, Joshua Tenenbaum, and William T Freeman. Pix3d: Dataset and methods for single-image 3d shape modeling. In CVPR, 2018.
[71] Joseph Lim. Hamed Pirsiavash, and Antonio Torralba. Parsing ikea objects: Fine pose estimation. In ICCV, 2013.
[72] Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao. 3d-future: 3d furniture shape with texture. IJCV, 2021.
[73] Panos Achlioptas, Judy Fan, X.D. Robert Hawkins, D. Noah Goodman, and J. Leonidas Guibas. Shape- Glot: Learning language for shape differentiation. CoRR, 2019.
[74] Kevin Chen, Christopher Choy, Manolis Savva, Angel Chang. Thomas Funkhouser, and Silvio Savarese. Text2shape: Generating shapes from natural language by learning joint embeddings. In ACCV, 2019.
[75] Rao Fu, Xiao Zhan, Yiwen Chen, Daniel Ritchie, and Srinath Sridhar. Shapecrafter: A recursive lext-conditioned 3d shape generation model. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https:
[76] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias NieGner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017.
[77] Dave Zhenyu Chen, Angel Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In ECCV, 2020.
[78] Tiange Luo, Honglak Lee, and Justin Johnson. Neural shape compiler: A unitied framework for transforming between text, point cloud, and program. Transactions on Machine Learning Revearch, 2023. ISSN 2835-8856. URL RRRL/hhtps://ppnrreeee https////..eeeeeeeeeeeeee////nna nett/forummidrgeeuyyyttt
[79] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In ICML, 2015.
[80] Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X. Chang. Scan2eap: Context-aware dense captioning in rgb-d seans. CVPR, 2021.
[81] Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin-Y Cheng, Marco Fumero, and Ka- mal Rahimi Malekshan. Clip-forge: Towards zero-shot text-to-shape generation. In CVPR, 2022.
[82] Paritosh Mitall Yen-Chi Cheng. Maneesh Singh, and Shubham Tulsiani. Autosdf: Shape priors for 3d completion, reconstruction and generation. In CVPR, 2022.
[83] Jiacheng Wei, Hao Wang, Jiashi Feng. Guosheng Lin, and Kim- Hui Yap. Taps3d: Text Text-guided 3d textured shape generation from pseudo supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16805-1666 16815, 2023.
14

---

[84] Biao Zhang, Jiapeng Tang, Matthias Niessner. and Peter Wonka. 3dshape2vecset: A 3d shape representa- tion for neural fields and generative diffusion models. arXiv preprint arXiv:2301.11445, 2023.
[85] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. In CVPR, 2022.
[86] Christina Tsalicoglou, Fabian Manhardt, Alessio Tonioni, Michael Niemeyer, and Federico Tombari. Textmesh: Generation of realistic 3d meshes from text prompts. arXiv, 2023.
[87] Chen-H Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 300-309 2023.
[88] Le Xue. Ning Yu, Shu Zhang, Jumnan Li, Roberto Martin-Martnn, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip-2: Towards scalable multimodal pre-training for 3d understanding, arXiv, 2023.
[89] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv, 2022.
[90] Alex Nichol and Heewoo Jun. Shap-e: Generating conditional 3d implicit functions. arXiv, 2023.
[91] Lingi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5826-5835, 2021.
[92] Chao-Yuan Yuan Wu, Justin Johnson, Jitendra Malik, Christoph Feichtenhofer, and Georgia Gkioxari. Multiview compressive coding for 3d reconstruction. arXiv, 2023.
[93] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object. arXiv, 2023.
[94] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. CVPR, 2023.
[95] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction finuuction-finetueee language models. arXiv, 2022.
[96] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In /CLR, 2020.
[97] Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. Retinaface: Single- shot multi-level face localisation in the wild. In CVPR, 2020.
[98] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
[99] Gant Laborde. Deep nn for nsfw detection. https://giiuub.cmm/atttaaaaassss accessed 7-May-2023).
[Online;
[100]
https://github. [Online; accessed 7-M3y-2023).
[101] Wonjae Kim, Bokyung Son, and Iodoo Kim. Vilt: Vision-and-language transformer without convolution region supervision. In ICML, 2021.
[102] Jun Gao, Tianchang Shen, Zian Wang. Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojeic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. NewrIPS, 2022.
[103] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. ECCV, 2020.
[104] Edward Hu, Yelong Shen, Phil Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
[105] Jiaxiang Tang
Stable-dreanfusion https://gituu..com/// http://tww...o////////////sbblea
Text-to-3d with stable-diffusion.
2022.
15

---

[106] Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Jaehoon Ko, Hyeonsu Kim, Junho Kim, Jin-Hwa Kim, Jiyoung Lee, and Seungryong Kim. Let 2d diffusion model know 3d-consistency for robust text-to-3d generation. arXiv, 2023.
[107] Donghoon Lee. Jiseob Kim, Jisu Choi, Jongmin Kim, Minwoo Byeon, Woonhyuk Back, and Sachoon Kim. Karlo-v1.0.alpp on coyo-100m and cc15m. ht thub. cww/ comm/aakaobriinkkrrloo 2022.
[108] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. NeurlPS, 2017.
[109] Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empow- ering large language models to understand point clouds. arXiv preprini arXiv:2308. 16911, 2023.
[110] Yining Hong. Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du Zhenfang Chen, and Chuang Gan. -lm Injecting
world into large language models. arXiv preprint arXiv:2307.1249 2023.
[111] Tiange Luo, Kaichun Mo, Zhiao Huang, Jiarui Xu, Siyu Hu, Liwei Wang, and Hao Su. Learning group: bottom-up framework part discovery in unseen categories. arXiv preprint arXiv:2002.0647 2020.
[112] Kaichun Mo, Shilin Zhu, Angel Chang. Yi, Subarna Tripathi, Leonidas Guibas. and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 909-918. 2019.
[113] Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte. Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-al xl: universe of 10m+ 3d objects, arXiv preprint arXiv:2307.05633, 2023.
16

---

Appendix A
Price Breakdown Details
This section provides our details computation for Table Using single A40 GPU, BLIP2 runs 2700 iterations per hour, enabling it to process around ~ 337. 337.5 objects hourly given the eight-run requirement for generating captions for 8 rendering views. This translates about 2.96 hours to process 1k objects, costing 2.96 x $1.28
$3.79 with the rate $1.28/ha on the cloud platform, CoreWeave On the same A40 GPU, CLIP operates at ~ 27000 iterations per hour, incurring a cost of $0.38. Importantly, utilizing eight A40s costs the same using one, due to the parallel processing capacity across multiple GPUs for multiple rendering views.
We compute our GPT4 cost by averaging input token numbers, OpenAl GPT4 API (8k context) costs 0.03/ 1k tokens, Our input prompt is: "Given distill these descriptions into one concise caption. The descriptions are as follows: 'captions' Avoid describing background, surface, and posture. The caption should be:", which consists of (1) text prompt and (2) captions generated by BLIP2 or BLIP2 + CLIP. Without CLIP's filtering, our input prompt contains 40 captions which have 511.1 tokens on average, cost 511.1 511.1/1000 0.03 1000 $15.33 for 1 objects. With CLIP, our input prompt contains 8 captions which have 139.3 tokens on average, cost 139.3/ 139.3/100 3/100 0.03 x 1000 $4.15 for 1 objects.
of descriptions about the same 3D object,
The average cost per l objects for human-authored annotation is computed as the average expenditure on the crowdsoureing platform, Hive The human annotation speed is computed by averaging the annotation progress across our whole annotation process.
We do not report the average cost of Cap3D (QA) in the main paper, only use it on ABO. For completeness, we report it here. The one distinction is BLIP2 run twice instead once for the two-stage question answering (QA). The cost of BLIP2 thus doubles, from $3.79 to $7.58; and total cost increases from $8.35 to $12.14 per 1k objects.
Appendix B Additional 3D Captioning Results
3D low poly model of woldier/warrio with a sword shield.
A 3D model of belmet with red, yellow, and black gus respitator and light.
3D model of red and white molocule.
A model of and white mushroom.
3D rendering of spiral staircase with railing in white room with open
3D model of Rubik's Clbe featuring blue, orange, rod, green, yellow squares.
suitcase with resembling an old medical bag or aid box.
A model of robotic with and spikes.
3D model green and yellow metal holes and cross beam.
A 3D model house with fence, carousel.
printed figurine Santa Claus with reindeer antlers, bolding a stick and standing on rock.
L-shaped sectional
chaise, U-shaped backrest, curved armrests, and a footstool on
Figure 9: Random 3D captioning examples generated by Cap3D. Two views of 3D objects (Obja- verse shown here, Cap3D uses eight.
17

---

UFC card featuring shirtless man in red and white design.
3DD white submarine model gun and pole
model stone mask sculpture resembling broken piece of ice.
3D Wooden Barrel Model
3D omnate picture frame with light bulb it.
3D rendering a house with roof
3D white satellite model with antenna
A set of three colorful 3D model houses with roofs.
3D model Earth circle shape.
A white figurine of bird top shoe.
3D model of pink plastic
Star Wars Boba Fetl 3D Model
Figure 10: Random 3D captioning examples generated by Cap3D, Two views of 3D objects (Obja-
verse 99] are shown here, Cap3D uses eight.
model of green, armored lizard wearing crown and holding sword.
model large rock, possibly marble granite, with flag on it.
A 3D model of statue af man with hole the ground.
3D model of white robot.
A 3D model of small flying robos- shbbt-spaceship hybrid with extended arms, featuring alien and it.
rendering vending machines and various colored buxes.
3D printed kookabura model silling on branch.
wooden model
22
A 3D model of a white cylindrical object features resembling radiator, vase, and light bull.
model of Transformers Optimus Prime blue and red truck
yellow table cup coffce top.
3D illustraioon of small yellow flower with
Figure 11: Random 3D captioning examples generated by Cap3D. Two views of 3D objects (Obja-
verse [9] are shown here, Cap3D uses eight.
18

---

rendering house with roof structure featuring pink lines.
white daily floral pattern and circular ociling light with a flower design.
3D rendering of white square object
3D model of wooden park bench with leaves on it.
3D model of modern blue leather sofa with metal legs and gray motal shelf with two hooks.
A 3D white skull model with red eyes, resembling combination of animal, squid, starfish features.
3D mudel of a green pod and propeller with two peas.
3D model a marble nativity scene figurine.
A 3D scene featuring destroyed house, building, plane, and car, with flying bird.
White Nintendo Wii console with power outlet and USB
yellow gold ring with pink sapphire stone.
3D model of cow
port.
Figure 12: Random 3D captioning examples generated by Cap3D, Two views of 3D objects (Obja-
verse 19] are shown here, Cap3D uses eight.
A white model of teddy with outstretched.
3D model of tall high-rise building.
A 3D model of robot with table and satellite.
3D model of sworlddagger
A 3D model of twisted, colorfual object resembling with purple, pink, blue squiggles and wires.
3D model a green and yellow hole
White Aston Martin DB5 3D Model
3D model basehall bat
3D model featuring pillars, a fountain with statues, three vases, pots table, valves, pipe, and light fixtures.
A 3D model of red shipping container blue and white logo white label on it.
Brown bag with coins green string, featuring coin falling out.
3D model of gravestume
Figure 13: Random 3D captioning examples generated by Cap3D. Two views of 3D objects (Obja- verse [9] are shown here, Cap3D uses eight.
19

---

3D model of wooden door with a dosign, featuring weoden sign, curved wall, and clock.
model colorful, multi-colored hoat with flag, oars, and paddles.
model featuring bird,
A purple and yellow toy lantem with handle.
twigs, fishing poles,,
dragonfly, and skis with poles.
of a white and blue toy robat leddy bear.
3D blue box with a hole and sand on it.
3D model of cricket bat, royalty-free vector illustration.
A 3D rendering of row of yellow poles.
White 3D spiral staircase model
3D model of white axe
Wooden coat rack in 3D model.
A 3D model of green plant
an eye.
Figure 14:
Random 3D captioning examples generated by Cap3D, Two views of 3D objects (Obja-
verse [9] are shown here, Cap3D uses eight.
model amnored hand wearing gloves with leather strap and metal cuff.
3D model of white cloud.
White plastic ring
building with sluirs, featuring cube and ceiling light fixture.
3D model small bost and house yellow platform.
3D model of wooden power pole with
3D model of a white plastic bottle with
A knife and sliced bread on wooden catling bourd.
model two-story house with roof structure.
A 3D rendering fish Rubik's cube.
model grassy field with a road in front.
model of flower black box with pupple container and blue cormer shelf.
Figure 15: Random 3D captioning examples generated by Cap3D. Two views of 3D objects (Obja-
verse [9] are shown here, Cap3D uses eight.
20

---

Human Cap3D
model of boy in black outfit bolding a sword with outstretched.
3D model a Quntas Airbus A300 airplane.
sneakers with blue logo and chameter, fenturing soles.
Uptempo
basketball sncaker 3D model.
cartoon kid with clothes black
cartoon commercial sirplane 3d
grey
grey coloured sports shoe with undulations its surface.
Adidas scanned by Thank3D bandheld scanner, sampling ratio 40%. For more information, please kindly check as below, Whatsapp:...
Metadata
First Character 3Ds max, Student Work
bong 787
Figure 16: Comparative Analysis: Cap3l) Generated Caption vs Human-Annotated Caption vs. Objaverse Metadata [ Two views of 3D objects are shown here, Cap3D and human use eight.
Cap3D
3D rendering of grey Champion oweatpants red and black logo.
3D model of jur with a green
rusty, train engine.
model white deer.
Metadata Homan
three layer structure with green and white middle part also having brown hase.
a 3d model of train old engine.
a carton made antelope w homs borns each other
pants
named Jar of jam Saved from Ioly by
This is hacksp of Poby Asset
pua lequoe nade Chargtoo Lestuato
meniing enterel (eroosee boudd brokent
inquene Move
Old industrial diesel locomotive from Hungary.
Google Provicw without
still Download
con on
2u6
413,00
Figure 17: Comparative Analysis: Cap3D Generated Caption Human-Annotatee Caption vs
Objaverse Metadata [9] Two views of 3D objects are shown here, Cap3D and human use eight.
Appendix C Additional Text-to-3D Results
In this section, provide several text-to-3D results for all of our compared methods. We include Shap-E and Point-E pretrained models and the models finetuned on our data, as well as optimization baselines, including DreamFusion, DreamField, and 3D Fuse.
21

---

A 3D model of green icapot with homs.
Pretrained
Reference
Vinetuned
Point-E
S.Dif(LLRA)+
S.Diff(CNet) Point-E (Im-to-31)
Shap-F.
Shap E STF
Dream Fusion
Dream Field
(Text-to-aD)
Point- (Im-to-30)
Figure 18: Text-to-D results. The top text prompt and "Reference" are from our test set. We fine-tune the left 5-column methods on Cap3D-generated captions. The detailed setting and methods
described in $53
lamp
Pretrained
3D Fuse
Reference
Finetuned
Point-E
S.Dif(LoRAA+
S.DiffCNet)+ +
Shap-E NeRF
Shap- E STF
Dream Fusion
Dream Field
(Text-to-3D)
(Im-to-3D)
(Im-to-3D)
Figure 19; Text-to-DD results. The top text prompt and "Reference" are from our test set. We fine-tune the left 5-column methods Sap3D-gennratte captions. The detailed setting and methods are described $3.3
A model of featuring dragon hend, sward, and long, colorful hundle.
Pretrained
T
3D Fuse
Reference
Fine Tinetuned
Point-F. S.Di((LORA)+
S.Diff(CNet)+ Point-E (1m--o-3D)
Shap-E. NeRF
Shup
(Text-to-3D)
Point-E (In-to-3DD
Figure 20: Text-to-3D results. The top text prompt and "Reference" are from our set. We fine-tune the left 5-column methods on Cap3D-generated captions. The detailed setting and methods described $5.3
22

---

37 model af Five Nights at Froliy's 's fox churacter with ouistratched arms, weiring and holding gum.
Pretrained
Reference
Finetuned
Point E (Text-to-aD)
L.Dif(LaRAA+
S.Diff(CNet) Point-R (Im-to-31)
Shap-F.
Shap E STF
Dream Fusion
Dream Field
Point-B (Im-to-30)
Figure 21: Text-to-D results. The top text prompt and "Reference" are from our test set. We fine-tune the left 5-column methods on CapnD-generated captions. The detailed setting and methods described in $5.
model range with green grass
Pretrained
3D Fuse
Reference
Finetuned
Point-E
S.Dif((LoRA)
S.Diff(CNet)+ (Im-to-3D)
Shap-E NeRF
Shap E STF
Dream Fusion
Dream Field
(Text-to-3D)
(Im-to-3D)
Figure 22: Text-to-DD results. The top text prompt and "Reference" are from our test set. We fine-tune the left 5-column methods are described $3.
captions. The detailed setting and methods
model witch hats lantems hanging chain.
Pretrained
3D Fuse
Reference
Fine tuned
S.Dif(LLRA))
S.Diff(CNet)+ Point-E (1m-to-3D)
Shap NeRF
Shup
(Text-to-3D)
Point-E (In--o-33D
Figure 23: Text-to-3D results. The top text prompt and "Reference" are from our set. We fine-tune the left 5-column methods on Cap3D-generated captions. The detailed setting and methods described $5.3
23

---

Appendix D ABO Captioning: Automated Metrics
In 5.2 we report human A/B judgments ABO. We do not report automated metrics, which are poor measures of performance for at least two reasons. First, ABO contains a large number of objects that are very similar, meaning it would be challenging for captions to distinguish their differences. Thus, retrieval metries such as ViLT Image or Text Retrieval will show very poor scores across metrics. Second, we show automated captioning performs poorly at describing geometry well, meaning it is likely automated image-caption alignment will not align based on geometry well. For completeness, report automated metrics in Table 8 As expected, all retrieval scores very low. Automated captioning scores best across automated metrics, however we caution against drawing conclusions from this result. Human studies in Table 4 suggest the opposite, and qualitative results agree with this finding, e.g. 4
Table 8: ABO Automated Caption Evaluations. Automated captions are poor measure of performance on ABO as (1) many objects are similar, making retrieval difficult; (2) automated captioning does not describe geometry well, we should not expect automated image-caption alignment describe geometrically correct captions well.
CLIP
ViLT Img Retr.
ViLT Text Retr.
Method
Score
R@5
R@ 10
R@5
R@ 10
Meta
61.9
0.8
1.7
0.8
1.7
Human
75.2
2.6
4.4
2.3
4.2
Cap3D
89.9
4.2
7.2
3.2
5.6
Cap3D(QA)
82.7
2.9
5.3
2.4
4.3
In contrast with A/B tests, which take place on the full 6.4k objects of ABO, this table is computed on a random 5k object subset of ABO to follow standard retrieval benchmarks (performance drops considerably dataset size increases. Using 5k instead of the full 6.4k makes it much casier to contextualize retrieval numbers). A/B performance on this 5k subset is very close to the full 6.4k dataset, meaning the sample is highly representative, and one can compare the results from this table in combination with Table in the main paper.
Appendix E Cap3D, BLIP2, Human Caption Comparisons
comparative analysis on the number of n-grams was conducted to shed light the distinet vocabulary sizes among Cap3D, BLIP2, and human captions for Objaverse [ objects. As elucidated in Table 9 Cap3D exhibits a considerably larger vocabulary as compared to BLIP2. Although human-generuue generated captions encompass a higher number of phrases, the disparity is notably lesser than that observed between BLIP2 and Cap3D. Moreover, hypothesized that the slightly elevated dictionary size in human captions could be attributed to the inclusion of typographical errors, typically arising from crowdsourced platforms.
Table 9: N-gram Comparison among Cap3D, BLIP2, and Human Captions
Occ. in 5k captions
Unigrams
Bigrams
Trigrams
BLIP2
1,928
6,616
10,899
Cap3D
3.108
15,274
25,883
Human
3,762
17,818
27,316
24

---

Appendix F Additional Details
E.
Prompt used in Cap3D
The two prompts used for BLIP2 used in Cap3D (QA) are (1) "Question: what object is in this image? Answer:" and (2) "Question: what is the structure and geometry this <object>>" where <object> is replaced with the response to prompt (I).
For the prompt used in GPT4, used "Given set of descriptions about the same 3D object, distill these descriptions into one concise caption. The descriptions are as follows: 'captions". Avoid describing background, surface, and posture. The caption should be:". We did several prompt engineering and considered prompt with more context, like "Below you will find a set of descriptions, each one is originating from various renderings of an identical 3D object. The level of accuracy in these descriptions ranges significantly: some might not correspond to the 3D object at all, others could be entirely accurate, while few may only partially represent the object. Your task involves scrutinizing these descriptions and distilling them into a single, holistic depiction. The descriptions are as follows: 'captions'. Note: Please avoid using the phrases 'gryy background", 'gray background', and gray surface' in your consolidated depiction. The synthesized description of the 3D object should be:". However, with those longer prompt with more context, we noticed GPT4 sometimes would generate its reasoning process which led to confusing output captions. Also, for the sake of cost, we hope to make our prompt as short as possible.
F.2 Rendering Details
We use Blender to render 3D objects in Objaverse [9] and ABO (35] normalize them into a unit cube and recenter origin. Then, we place 8 different cameras surrounding the object with cameras slightly below the object to capture the bottom of the object. Three area lights are placed and function key light, fill light, and rim light, respectively. The detailed parameters are listed in our rendering script, provided in our Github
For each object, we first
In Objaverse, filter out objects that fail in rendering, resulting subset of 785k objects for rendering and captioning. In ABO, we exclude categories with simple geometry to concentrate on ge- ometrical captioning. including "BLANKET", "RUG", "WALL_AR"s "PLACEMAT", "CURTAIN", "MOUSE PAD". This resulting a final subset of 6.4k objects for rendering and captioning.
F.3 Human Captioning Split
Human captions collected on manually selected subset of Objaverse with good renders of nontrivial but decipherable objects. These objects are likely to be the most sensible for captioning and A/B testing. For instance, some Objaverse objects essentially simple rock with little texture; in others it can be difficult for a human to describe an object (e.g. abstract art, no clear object visible, or 3D scans with hard-to- hard-to-distinguish details). These excluded objects are generally not effective samples to use for human A/B testing, as the correct caption may not be clear or may be trivial. We also exclude furniture, which is suitable for captioning, but we measure this with more focus on ABO. Human captions on ABO follow the split of (78).
Appendix G
Crowdsourced Captioning Details
We use Hive for crowdsourced captioning. Workers are given instructions for the task including gold-standard examples. Captioning instructions are shared below for Objaverse in Figure 24jand ABO Figure 25 Workers are persistently monitored. are promptly banned from captioning, and their previous captions are discarded. Workers are paid approximately $50 per Ik tasks. We do not have access to their captioning rates; assuming a rate of objects per minute, this would result in $9 per hour. Across Objaverse and ABO we spend total of S7k on captioning.
a worker produces bad captions they
25

---

Please describe the structure of the 3D object Each answer should answer the following three questions.
pieces up object? (e g. back, seat, legs, window)
(2) What shape are these pieces? (eig round, flat, square, long, narrow, wide, large)
How these pieces connected? (e.g. left, right, above, below, connected, held inside has, contains)
"Please not describe color texture (texture color is purposely left renders)."
"Captions must be in English. you do not provide real answers you will promptly be permanently banned; we actively monttor annotations"
arage
lnge
Capaae vag Moter
aoaue and
Figure 25: ABO Caption Instructions.
26

---

Please describe the object pictured the image in terms appearance and geometry in al least two sentences of detail
"Caplions must in English. you do not provide real answers will promptly be permanently banned; we actively monitor annotations"
Example
Caption A cartoon donut that vanilla with pink frosting and white sprinkles. The donut round with a hole in the middle. +> Note: describe in detail both appearance (cartoon donut, vanilia, pinkk frosting, white sprinkles) and geometry (round).
Example Image:
Caption: A yellow cartoon duck with white beak and 3 dark hairs coming out of its head. It is the Pokemon Psyduck, and has large round torso with short arms and
w>> Note: describe in detail both appearance (yellow cartoon duck, white beak, dark hairs) geometry (large round torso, short arms and legis, 3 head). don't know specific names Pokemon Psyduck) problemt you better!
Example
Image:
Caption An airplane model, small the frame made of wood and the wings made beige material. two wheels the front and one in the back in the center.
-> Note: we describe in detail both appearance (airplane model, frame wood, wings beige) and geametry (very small, two wheels in front, one wheel in back in center).
Example 4
Image:
big eyes Note: we describe detail both appearance (light-green toy pony, colorful on head and tail, stars checks) and geometry (big eyes, fluffy hair and tail).
Caption: A light: green toy pony
fluffy
Figure 24: Objaverse Caption Instructions.
27

---

Appendix H Crowdsourced A/B Testing Details
We use Hive for crowdsourced A/B testing. Specifically, workers are given an image and two captions, and select which is better on a scale from to 5, where 3 is tie. So would be "left much better", and 2 would be "left better". Workers are given instructions for the task along with gold standard examples. Workers are informed to prioritize accuracy, then informative detail, then brevity. Left/right order between methods was randomized for each instance. A/B Testing instructions are shared below for Objaverse in Figure 27 and ABO in Figure 26
Workers are automatically banned by the platform if they miss too many gold-standard examples. However, we found some workers would successfully pass the handful of gold-standard examples while scamming on the rest of the examples. The most common scam cases were always picking the same number, or always picking the shorter or longer caption. We thus manually search through all workers and ban workers who meet these scamming criteria and discard their judgments. Unfortu- nately, discarding judgments leads to uneven numbers of observations for each individual experiment. Nevertheless, in all cases, enough observations are available to draw conclusive findings.
The size of each experiment's 's data after discarded judgments is below.
Objaverse Split (I) takes place random set upon which human captions are available. Cap3D vs. Human has 36k observations across 22k objects.
Objaverse Split (2) takes place on a random object set upon which human captions are available. Cap3D Human has 10k observations across 4.7k objects. Cap3D Metadata has 7k observations across 4.7k objects (less than the target 10k), though given the extremely poor rating of Metadata, results conclusive.
Objaverse Split (3) takes place on random object set upon the entire Objaverse dataset. Cap3D vs. BLIP2 has 20k observations across 5.0k objects and Cap3D vs. +GPT4 has 29k observations across 5.0k objects.
ABO takes place on the full ABO object set. Human Cap3D has 21k observations across 6.4k objects, Cap31) (QA) vs. Human has 17k observations across 6.4k objects, Cap3D (QA) vs. Cap3D) has 1 13k observations across 6.4k objects, and Cap3D (QA) vs. Meta has 12k observations across 6.4k objects.
Workers are paid approximately $20 per Ik tasks. We do not have access to their captioning rates; assuming a rate of 7.5 A/B tests selected per minute, this would result $9 per hour. Across Objaverse and ABO we spent a total of $1.88 8k on A/B testing.
28

---

In this task, object will be pictured from views. There will be two captions: ("left" caption and "righ" caption).
Please select the caption best describing the pictured object terms type, appearance, and structure.
Priontize captions informative object
Select captions
Ignore caption details about the background (these are nelther good nor the task)
NOTE: There is only one object. appears times since it shown from 8 views
--> If caption says there are multiple objects and there is only one object in each of views, the caption is wrong
Select from the followlng options:
Much Better caption describes the object with much accuracy, informative detail, avoiding many unnecessary details
caption describes object
accuracy,
detail, avoiding unnecessary
Tie: both captions have similar accuracy and detail
Right Better: right caption describes the object with more accuracy, informative detail, or avoiding unnecessary details
Right Much Better: right caption describes the object with much more accuracy, informative detail, or avolding many unnecessary details
Eemeple
right
Capa dut kC ceder conaiia asscuasia Dhs reepee reseen
GeDNA and MMAA COMAMMAAAAAAA CORITT
COMMETOL teaets
Colemtoo Letteete
wilie oer syt, se det
Duenall Tir lo desr Intomati
Ceeett oveczon sech 2018
shid tpdinn Stayiing
by
paitor
Lighte
match
Class HRgTTMU 2012
Mileeddeer giet
Covert fetente bete
cepirr
codiee cone
beccion lig
de yman retooven asl oetets our aiti altovo eetry loco encent me thom by 1e cocc wilor
Figure 26: A/B Instructions: Objaverse Captions.
29

---

task, an object will pictured views. There two captions ("left' caption "right caption).
Please select the caption best describing the structure of the pictured object
Language describing the color or texture of the object or the background should be ignored only judge based on object geometry.
Priortizee captons that make accurate statements about the object
Select captions with informative object details but not ones with unrelated repeated (redundant) details.
NOTE: There is only object appears times since shown from views
caption says
multiple and there only object
caption wrong
following options:
Much Better: left caption describes the object geometry with much more accuracy, informative detail, avolding many unnocessary details
Left Better: left caption describes the object geometry with more accuracy, informative detail, or avoiding unnecessary details
Tie: both captions have similar accuracy and detail
Right Better: right caption describes the object geometry with more accuracy, Informative detail, or avolding unnecessary details
Right Much Better, right caption describes the object geometry with much Informative detail, avolding unnecessary
details
cuppert
Indariica
Fvipp
desgest cww.era AADEAAA
eup ceeee Baw Garad
Figure 27: A/B Instructions: ABO Captions.
30

---

Appendix I
Additional Experimental Details
we perform one full-scale evaluation run for all captioning experiments; 95% confidence interval for mean is presented. Metrics are overviewed in $5.1 A/B testing is detailed further in ş CLIP Score takes about 5 minutes, while ViLT R-Precision takes about hours using an A40 for test set of 5k object-caption pairs. Crowdsourced A/B testing takes about 12 hours for 10k responses across 5k objects.
Captioning:
Text-to-3D, finetuning: for finetuning experiments, we used one train and evaluation run using a small overfitting experiment the train set. Training took about 3 days
learning rate validated the full set and I day on the small (human) set. We used AdamW optimizer and CosineAmeellnggL. scheduler with initial learning rate le 5 for finetuning both Point-E and Shap-E. We adopted batch size 64 and 256 for Shap-E and Point-E, respectively. However, for Shap-E, we found it usually outputs NaN and needed re-start from saved checkpoints, which could be one of the reaons why our finetune did not bring improvements. For LoRA, we use A optimizer and Cosine/Anealingg scheduler with initial learning rate le and batch size of 3. For ControlNet, we use Adam optimizer and constant learning rate of le 5 and batch size of 8. Experiments use 4 A40s to train except LoRA, which fails upon multi-gpu training due to a HuggingFace internal DDP error. Notably single-gpu gpu training still yields improvement. Evaluation takes the following time (in seconds) per iteration, which includes rendering:
PointE (text-to-3D): 37sec 28sec (text-to-3D) + 9sec (render)
LoRA + PointE(im-to-3D) 1 114sec = 5sec + 100sec (im-to-3D) + 9sec (render)
ControlNet PointE(im-to-3D 124sec 15sec + 100sec (im-to-3ID)- 9sec (render)
ShapE (NeRFy: 193sec (text-to-3D render)
ShapE (stf): 16sec (text-to-3D render)
Note publicly available PointE (im-to-3D) is IB param, making it slower than the largest publicly available PointE (text-to-3D) of 40M. Evaluation metrics are detailed in $5.33
Text-to-3D, optimization: For one object, optimization plus final rendering takes 40 minutes for 3DFuse, 95 minutes for Stable DreamFusion, and 35 minutes for DreamField; using A40 GPU. We use default parameters for all methods and run them once.
31


Reviews:
[{"review_id": "sG3e9G7vGa", "reviewer": "Reviewer_WJz6", "summary": "", "questions": "", "limitations": "It is not that fair of a baseline, but to me, this paper largely presents a workaround to caption objects with LLMs, given that LLMs largely don't have the ability to take in image input. It seems like once GPT-4 + vision is public, or other LLMs in this space that have the ability to consume images as input, the approach might not be that useful, and the captions that come out of GPT-4 + vision might be better than the captions that come out of Cap3D alone. Nevertheless, I don't think this should go against this work, as these models are not currently fully public.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "Overall, I am quite positive on the paper.\n\nI think the proposed captioning strategy is quite neat. The captions all manually seem quite sensible and strong (much stronger than just captioning a single image, using the metadata, and often human captions alone).\n\nThe approach is quite sensible, simple, and scalable as they can automatically be obtained on compute time alone without any human intervention. It nicely leverages other powerful models built by the community (i.e., BLIP, CLIP, GPT-4).\n\nThey provide nice evaluations and the paper is well written and easy to follow.\n\nThere is a need for strong text annotations for 3d, and this paper presents probably the best result I've seen in this space.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "uQo3mkeY3F", "reviewer": "Reviewer_bhAA", "summary": "", "questions": "", "limitations": "Yes,  the authors have adequately addressed the limitations and potential negative societal impact of their work", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. A large-scale 3D text-object paired dataset is in short supply and important for 3D reasoning study. The authors generate such a dataset and publicly release it.\n\n2. The proposed automatic annotation method is reasonable and straightforward. The automatically generated caption surpasses human-authored descriptions in terms of quality, cost, and speed.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "ZGj2XJmufA", "reviewer": "Reviewer_UuMn", "summary": "", "questions": "", "limitations": "1 This work only uses rendered images information from 3D objects as input, which may result in the loss of some 3D information, such as geometry details obscured by occlusion, part functions, and accurate position information. \n\n2 The pipeline designed in this work is not end-to-end, but rather a combination of multiple existing solutions. While this approach might achieve the desired goal, an end-to-end solution would potentially streamline the process, reducing complexity and improving efficiency.\n\n3 The comparison solely based on human-authored captions and their associated classes might have limitations. Human descriptions can be subjective and might not comprehensively capture all properties of the 3D objects. Including a qualitative comparison that evaluates various aspects like categories, colors, textures, shapes, components, and functions could provide a more comprehensive assessment of the model's performance", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1.\tThis work addresses an important problem by focusing on generating descriptive text for 3D objects, effectively bridging the gap between 3D objects and text modalities.  \n2.\tBy conducting fine-tuning on several text-to-3D models, the dataset facilitates meaningful improvements in the performance of these models. This dataset plays a critical role in evaluating and advancing text-to-3D generation.\n3.\tThe evaluation results presented in the paper showcase the significant performance of the proposed method when compared to tedious and low-quality human annotations.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Gwns9uW2pA", "reviewer": "Reviewer_Dwxs", "summary": "", "questions": "", "limitations": "The technical contribution is a littl weak.The idea of generating text-3D paired data using LLMs is not novel. Compared with Ref.[86], the submission additionly conducts the fusion precedure using GPT-4, does much ethical filtering work, and conducts text-to-3D tasks on the generated dataset. ", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The submission proposes to utilize GPT-4 to fuse captions generated from multi-view rendered images, which is a novel idea. Besides, the submission does much ethical filtering work. Experiments on Objaverse and ABO dataset demonstrate advantages over metadata and human-authored captions, and effectiveness in finetuning text-to-3D models on the generated dataset.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Five experts reviewed this paper with all accepted recommendations. The area chairs agree that this work makes a very important contribution by introducing a tool for 3D Captioning. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes. It would be also nice if the authors could include some future work discussions on how to further make progress on this challenge by leveraging the recent success of concurrent works  (e.g. [1,2]) to further enhance the capability in the final version. 
 
[1] 3D-LLM: Injecting the 3D World into a Large Language Model. arXiv:2307.12981

[2] PointLLM: Empowering Large Language Models to Understand Point Clouds. arXiv:2308.16911
Decision: Accept (Poster)