SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tiange Luo $^{1,*}$  Chris Rockwell $^{1,*}$  Honglak Lee $^{1,2,\dagger}$  Justin Johnson $^{1,\dagger}$
$^{1}$ University of Michigan  $^{2}$ LG AI Research
We introduce Cap3D, an automatic approach for generating descriptive text for 3D objects. This approach utilizes pretrained models from image captioning, image-text alignment, and LLM to consolidate captions from multiple views of a 3D asset, completely side-stepping the time-consuming and costly process of manual annotation. We apply Cap3D to the recently introduced large-scale 3D dataset, Objaverse, resulting in 785k 3D-text pairs. Our evaluation, conducted using 41k human annotations from the same dataset, demonstrates that Cap3D surpasses human-authored descriptions in terms of quality, cost, and speed. Through effective prompt engineering, Cap3D rivals human performance in generating geometric descriptions on 17k collected annotations from the ABO dataset. Finally, we finetune text-to-3D models on Cap3D and human captions, and show Cap3D outperforms; and benchmark the SOTA including Point-E, Shap-E, and DreamFusion. Our data, code, and finetuned models can be found at https://cap3d-um.github.io/.
A 3D model of a blue grand piano with spikes and sharp teeth resembling a shark mouth.
A 3D model of a metal cube featuring a skull, pizza, and various stickers.
3D model of a sakura soft drink can with purple and yellow gradient, Japanese writing, and purple flowers.
Loki bust 3D model featuring a green and yellow horned helmet.
A 3D model featuring a basketball hoop, ball, racquet, bowling ball, stand, and pin.
3D model of a yellow Pikachu-themed Pokémon ball with a black and gold stripe and lightning bolt.
3D model of a purple and green Halloween spider bowl on a metal stand, containing purple liquid.
3D model of Notre Dame Cathedral, a Gothic cathedral with spires in Paris.
3D model of an armored character with purple horns and spikes on the back.
Figure 1: Cap3D provides detailed descriptions of 3D objects by leveraging pretrained models in captioning, alignment, and LLM to consolidate multi-view information. Two views of 3D objects are shown here, Cap3D uses eight. Additional examples are available in Appendix B.
3D model of a robotic scorpion with multiple arms and guns.
A cluster of five glass sphere light bulbs suspended from a single thin wire.
L-shaped sectional sofa with a chaise, U-shaped backrest, curved armrests, and a footstool on one side.
* joint first authorship; † equal advising
Table 1: Cap3D is better, cheaper, and faster than crowdsourced annotation. Use 36k responses across 22k objects for A/B testing; 8A40s on a cloud platform for speed and cost computations.
Method	A/B Human Testing Win % (Tie %)	Cost per 1k Objects	Annotation Speed	1k Objects Cost Breakdown
BLIP2	$3.79
CLIP	$0.38
GPT4	$4.18
Human	37.8% ± 0.5% (9.5%)	$87.18	1.4k / day
Cap3D	52.3% ± 0.5% (9.5%)	$8.35	65k / day	Cap3D Total Cost	$8.35
Text-conditioned 3D synthesis [1-3] could revolutionize the creation process of 3D assets, impacting various sectors, including 3D design, virtual reality [4], film [5], robotics [6, 7], and autonomous driving [8]. However, challenges persist, namely the high cost of 3D asset creation and the scarcity of high-quality captions for 3D assets. Objaverse [9] takes a step towards this as the first public large-scale 3D object dataset. Unfortunately, while objects contain paired metadata, these do not serve as informative captions, as shown in Table 3. In contrast with 3D, a plethora of high-quality text-image paired data is publicly available [10-14]. This data has led to incredible recent progress in image-text learning [15-18], text-conditioned image synthesis [19-24], and image captioning [25-29].
In this work, we present Cap3D, a method to automate 3D object annotation. Our key insight is to leverage the abundance of knowledge in pretrained image-text models to remedy the lack of existing 3D-text data. The core of our data collection process is to apply an image captioning model (BLIP2 [29]) to a set of 3D asset renders, use an image-text alignment model (CLIP [16]) to filter captions, and apply a language model (GPT4 [30]) to fuse the filtered captions across views. Critically, the models we apply are pretrained on varied and large-scale text-image [11-13, 31-33], and text [34], data; and approach complementary problems. As a result, each model adds additional value to the framework, as we show in Table 3.
Cap3D is agnostic to 3D asset sources and can be effectively scaled to larger extents with increased 3D assets and computational resources. In this paper, we apply it primarily to Objaverse, gathering a dataset of 785k 3D-text pairs. Through object rendering and captioning, we enable ethical filtering of 3D objects via both image and text, as detailed in § 3.2. We publicly release all of our collected data including automated and human-annotated captions, along with associated Point Clouds and Rendered Images, at huggingface.co/datasets/tiange/Cap3D. The dataset is released under ODC-By 1.0 license. We also released trained models and code for replicating the benchmark table.
We validate our collection approach by collecting over 50k crowdsourced captions on over 40k objects. We conduct human evaluations and show on Objaverse that our automated captions are superior to crowdsourced captions in quality, cost, and speed (Table 1, details in Appendix A). Specifically, it is preferred  $35\%$  more often by humans, costs more than 10 times less, and is over 40 times faster, assuming only 8A40 GPUs. We also test the limits of automated captioning. We consider a separate task of captioning geometry (as shown in Figure 1 bottom-right) using ABO, a dataset of 3D models with complex geometries [35]. Shown in Table 4, our automated captioning underperforms humans. However, by formulating description as a question answering task (detailed in § 3.1), we show stronger performance compared to crowdsourced workers. This result shows the ability of our method to adapt beyond traditional captioning and still be highly competitive.
Finally, our high-quality gathered 3D-text dataset enables us to train and validate large-scale text-to-3D models. In §5.3, we evaluate several state-of-the-art methods on Objaverse out-of-the box, including Point-E, Shap-E, DreamFields, and DreamFusion. Finetuning on our data typically shows meaningful improvements, demonstrating the value of the collected dataset. In addition, we show our automatically collected captions yield better finetuning performance than human captions – even at the same scale. At full scale, finetuning is further boosted.
Obtaining 3D-text pairs at scale is challenging, and we take inspiration from image-text datasets and methods when approaching this task.
Image-Text Data and Modeling. Early image captioning [36-38] and text-image representation learning methods [39-41] were built using CNNs [42-44] and LSTMs [45, 46], leveraging human-annotated datasets [31-33, 47]. Text-to-image methods used similar datasets, and relied on GANs [48, 49] and VQVAEs [19, 50-52]. The advent of semi-automated image-text collection has enabled successful scaling of datasets [10-14] and models [25-28]. Transformer-based architectures [16, 53, 54] and diffusion models [55-60] have scaled best to large data; we employ transformer-based methods through our captioning process and adopt diffusion models for text-to-3D experiments.
Training models upon large datasets and using the corresponding trained models to filter larger data has led to datasets of rapidly increasing size [13, 14]. In addition to filtering, trained models have been used to annotate new data with high-quality [61]. We take this approach, captioning rendered views with BLIP2 [29], refining with CLIP [16, 62], and summarizing with GPT4 [63]; all of which are trained on large datasets, including [11-13, 31-33]. Concurrent works [64-66] use automated captioning on 2D images using an older system [67] or based upon metadata [66, 68].
3D-Text Data and Modeling. Until recently, 3D data was of relatively small scale ( $\sim$ 50k objects) [69-72]. Labeled 3D-text data was scarce, relying on human annotation, and typically limited to ShapeNet [69] chairs [73] or tables and chairs [74, 75], and ScanNet [76, 77]. This enabled prior work to undertake the task of 3D captioning [78-80] or text-to-3D [74, 78, 81-84] at small scale. Methods that approached text-to-3D would sometimes avoid 3D supervision entirely [3, 85-87], leading to slow generation due to many optimization steps. We annotate a small-scale dataset containing 3D furniture, ABO [35], to evaluate the ability of Cap3D to specify fine-grained geometry.
Objaverse [9] introduced a diverse set of objects over 10 times the size of the prior largest public 3D dataset [69]. This data is our primary captioning focus, and we associate a single caption with each object in Objaverse after filtering. Concurrent works [66, 88] gather text associated with Objaverse, but do not fuse captions across views [88] or rely upon metadata [66], and do not approach text-to-3D.
The concurrent studies 3DGen [2] learns text and image to 3D on Objaverse; Point-E [89] and Shap-E [90] learn text-to-3D models on a large-scale 3D dataset, but none have fully disclosed their code or data. Point-E involves two variants and released a text-to-3D model and a text-to-image-to-3D model by finetuning GLIDE [23] and training an image-to-point cloud diffusion model [91]. Other recent works [92, 93] also focus on scaled image-3D generation. We show finetuning on our captions improves Point-E performance despite having already been trained on large amounts of Internet data.
3.1 Captioning Process
Our task is to produce a single descriptive caption given a 3D asset. Our proposed method, Cap3D, employs a four-step process. First, we render a set of 2D views for each 3D object. Next, we apply image captioning to achieve preliminary descriptions. As these captions may contain inaccuracies, an image-text alignment model, CLIP, is introduced in the third step to rectify errors. Finally, an LLM is employed to unify captions from various perspectives, creating a comprehensive caption. This process is shown in Figure 2 and detailed below.
Object Rendering: We render using Blender at  $512 \times 512$  from  $M = 8$  high-information camera angles rotating horizontally around the object, with two slightly below and the rest slightly above the object, to cover all the object details. The reason we prefer multiple views is a forward-facing view may miss self-occluded object details (e.g. Figure 1 row 1) or face strange appearance and/or lighting. In contrast, multiple views will see much of the object from different viewpoints, increasing the number of chances for a captioning model to predict objects in detail. For instance, in Figure 2, the back view 1 identifies the "yellow handle", which is barely visible in forward view  $M$ .
Image Captioning: We use BLIP2 [29] for captioning, selecting the largest pretrained model adapting ViT-G [54, 94] image encoder and FlanT5XXL [95] text encoder. We generate  $N = 5$  captions per rendered image using nucleus sampling [96]. By generating multiple captions, we increase the likelihood of generating correct details (e.g. "black and yellow toy bomb" in Figure 2 view  $M$  caption 1). Incorrect captions, such as "scissors" in Figure 2 view  $M$  caption  $N$ , can be filtered in later stages. To generate captions containing fine-grained geometry details (in our ABO experiments), we employ a two-stage question-answering instead of captioning. The first stage
Figure 2: Overview of Cap3D. Left to Right: (1) Render 3D objects from  $M = 8$  camera angles to capture object details (2) Generate  $N = 5$  image captions per rendered image using BLIP2; (3) Select one caption for each image based on its similarity to the image encoding using CLIP; (4) Use GPT4 to consolidate all selected captions into a final, summary of the object.
generates one answer to a prompt asking what object is pictured. The answered object is passed into a second prompt, which asks its structure and geometry, and generates 5 answers.
Caption Selection: While BLIP2 often generates high-quality captions, it is not uncommon for samples to contain mistakes, particularly in non-forward facing views such as "yellow cup", in Figure 2 view 1, caption  $N$ . To reduce the frequency of mistakes, we compute CLIP [16] ViT-B/32 [54] encodings from each of 5 captions and the associated image, and select the caption maximizing cosine similarity. CLIP tends to select good captions for each view, e.g. Figure 2: view 1, BLIP2 caption 1 and view  $M$ , caption 1. CLIP is complementary to BLIP2 as not only does it have different training details and architecture, but it trains on different data. While BLIP2 is trained upon COCO [31], Visual Genome [32], CC3M [11], CC12M [12], SBU [33] and LAION400M [13]; CLIP is trained upon a dataset of 400M images based on frequent text occurrence in Wikipedia.
Caption Consolidation: Accumulating information across viewpoints to form a complete picture of 3D objects is challenging, but crucial. We find prompting of GPT4 [63] to summarize the  $M$  captions results in good parsing of the details across captions. By applying GPT4 as the final summary step, it can both include significant details and remove unlikely ones. For example, the final caption in Figure 2 filters the incorrect information, from view 2, "toy ball", while keeping key details, including "handle" and "straw". The alternative order of GPT4 followed by CLIP would result in (1) GPT4 having to make sense of more incorrect input details and (2) CLIP simply selecting between aggregate captions instead of being able to error-correct small mistakes. The effectiveness of introducing GPT4 is verified in ablations (Table 3).
3.2 Ethical Filtering
Captions generated and images rendered by Cap3D enhance the identification and mitigation of legal and ethical issues associated with large-scale 3D object datasets, including identifiable information and NSFW content.
We manage two datasets: Objaverse and ABO. In Objaverse, our main responsibility involves dealing with artist-created assets. These can include identifiable elements such as human face scans and NSFW objects. Objaverse contains approximately 800k objects, which makes the manual verification of each asset impractical. The ABO dataset, on the other hand, is smaller and mostly consists of furniture. We manually ensure the ethical integrity of this dataset.
We exclude objects that lack sufficient camera information for rendering, leaving us with 785k objects. These objects encompass a mix of non-commercial and commercial licenses, such as CC BY-NC-SA, CC BY-NC, CC BY, CC BY-SA, and CC0. We then applied ethical filtering exclusively to objects with commercial-friendly licenses (i.e., CC BY, CC BY-SA, and CC0), resulting in a count of 680k objects.
We next follow prior work [10] and use a face detector [97] and NSFW classifier [98, 99] on forward-facing object renders and filter detected objects with score  $> = 0.9$ . The face detector filters out 18.6k
Figure 3: Objverse Caption Comparison. Human captions and Internet metadata frequently contain limited detail. Cap3D captions typically have longer length and more detail. Vocabulary size comparisons among Cap3D, BLIP2, and human captions are included in Appendix E.
objects, and the NSFW classifier filters out 217 objects. Text is also carefully processed. Our final captions are the output of GPT4, which has been trained to filter out inappropriate or harmful content [63].
We run a standard blacklist [100] on its output, removing any object-caption pairs including blocked words. This filters out 226 objects. After all the filtering, we are left with 661k objects in the Objaverse dataset. We manually estimate detection precision and recall in Table 2. To summarize, our process detects over 19k objects, of which a nontrivial amount is accurately removed. We estimate roughly 1k face and less than 1k NSFW are missed, using a conservative standard (e.g. missed faces are typically sports cards), resulting in a 660k subset.
Table 2: Ethical Filtering Analysis. We manually detect faces and NSFW content to validate automated filtering. 16 of 17 missed face detections were sports cards.
Detected (Filtered)	Precision	Missed dets.
5k	(%)	10k	680k
Faces	18.6k	790	16%	17	≈1k
NSFW	217	102	47%	12	<1k
Language †	226	-	-	-	-
†: String match filtering is deterministic.
We collect captions in two distinct settings: Objaverse, a large and varied dataset of artist-created 3D assets; and ABO, a small dataset of real products, typically furniture.
4.1 Objverse Captions
Objaverse [9] features roughly 800k 3D object assets across 21k classes designed by over 100k artists. It is of significantly larger scale than prior work; the paper shows this size enables more diversity by generative 3D models trained upon it. It is released under the ODC-By 1.0 license, permitting subsequent researchers to curate new data from it. Metadata is paired with many assets, however as seen in Figure 3 (right), metadata caption length is frequently short or empty. We collect two caption datasets on Objaverse. First, an automated set of one caption for each of 785k objects using Cap3D (a total of 785k captions). Second, a crowdsourced set of 41.4k captions spanning 39.7k objects for evaluating generated captions. Captions are collected using thehive.ai, a crowdsourced platform similar to AMT. Workers are given instructions with gold-standard sample captions, see the same 8 views as models during captioning, and are routinely monitored. Poor captioning performance results in a ban and deletion of the worker's captions. Crowdsourced captions are also filtered using the blocklist in § 3.2. Figure 3 (left) shows human captions provide more detail than metadata, but automated captions tend to be most descriptive.
4.2 ABO Geometry Captions
ABO [35] is a collection of 3D models of Amazon products and is primarily furniture. ABO serves as an important contrast to Objaverse as it consists of a small number of classes varying primarily in geometry. Captioning, therefore, needs to focus more on structure as opposed to semantic category. To emphasize this focus, we consider the task of captioning the geometric structure of objects without color or texture (seen in the bottom right of Figure 1). Like Objaverse, ABO contains metadata that is
Figure 4: ABO Automated Geometric Description. Left: Human descriptions provide more detailed geometry than automated captions. With careful prompting, Cap3D (QA) can match human-level detail. Right: The high peak of Metadata is cropped, which otherwise obscures other curves.
typically quite short (Table 4), resulting in limited detail. We collect three sets of captions on the  $6.4\mathrm{k}$  ABO splits of [78]: crowdsourced (a total of 17.2k captions), captions generated by Cap3D (a total of 6.4k captions), and captions generated by Cap3D (QA) which uses the two-stage prompt captioning (a total of 6.4k captions). Crowdsourced captions follow similar detail to Objaverse with the exception instructions and examples are focused on geometric structure. We compare alternatives in Figure 4. In contrast to Objaverse, human geometric descriptions on ABO are more detailed than captioning. With prompting (QA), the Cap3D pipeline can rival human descriptions.
In this section, we first validate the quality of Cap3D captions against metadata and human-authored captions on both Objaverse and ABO. To verify Cap3D captions are helpful in practice, we next compare text-to-3D models finetuned on both human-authored captions and Cap3D (using the same  $>30k$  set as crowdsourced captions). Finally, we evaluate state-of-the-art text-to-3D models on our captions at scale to measure if finetuning on our captions can improve performance.
5.1 3D Captioning on Objaverse
Dataset. We evaluate caption quality on three subsets of Objaverse: (1) a random set of 22k objects containing a human caption, (2) a random split of 5k objects containing a human caption, and (3) a random 5k split across the entire dataset.
Baselines. In data splits (1) and (2), we compare the caption generated by Cap3D with human-authored annotations, Human, and existing Objaverse metadata, Metadata, described in § 4.1. Split (1) is used for A/B testing of Cap3D vs. Human, as shown in Table 1, at scale. Collecting A/B comparison is expensive, so we compute more extensive experiments on the smaller set (2) in Table 3.
In data split (3), we ablate the main components of Cap3D into BLIP2 and  $+GPT4$ . BLIP2 uses only the image captioning component of our method, taking a front-view rendering and producing a single output caption.  $+GPT4$  uses the same image captioning process of our method, producing 5 captions for each of 8 views. However, instead of using CLIP to filter 5 captions from each view, it directly summarizes all 40 captions into a final caption.
Metrics. Our primary metric is human judgment A/B tests, where we ask workers to select between two captions on a scale of 1-5, where 3 is a tie. Workers are carefully monitored and each comparison has at least 10k observations across 5k objects. We report mean score, along with the percent each method is preferred (i.e. scores a 4 or 5). We use automated metrics CLIPScore [16, 62], the cosine similarity of CLIP encodings with input images; and ViLT Image and Text Retrieval, which ranks likely image-text pairs, from which one computes precision.
We emphasize CLIPScore is not our primary metric since our captioning model utilizes CLIP. BLIP2 utilizes ViT-L/14 and ViT-g/14, while our filtering uses ViT-B/32, so following previous work [85] we compute CLIP score using a different model to reduce bias (ViT-B/16). However, we report it as it
Table 3: Objverse Captions Evaluations. Cap3D outperforms human and Metadata; BLIP2, GPT4, and CLIP are all important to performance. We report  $95\%$  confidence interval and use 5k objects.
Method	User A/B Study vs. Cap3D	CLIP Score	ViLT Img Retr.	ViLT Text Retr.
Score (1-5)	Win %	Lose %	R@5	R@10	R@5	R@10
Metadata	1.74±0.026	10.7 ± 0.7	83.8 ± 0.8	66.8	4.3	6.3	6.1	8.5
Human	2.86±0.026	37.0±1.0	46.1±1.0	72.5	21.2	29.0	18.5	24.9
Cap3D	-	-	-	88.4	35.7	46.3	34.7	44.2
BLIP2	2.87±0.019	41.0±0.7	50.6±0.7	83.1	24.7	32.3	21.9	29.3
+ GPT4	2.94±0.015	35.2±0.6	40.8±0.6	86.3	31.9	39.9	30.2	38.4
+ CLIP (Cap3D)	-	-	-	86.9	31.1	40.2	30.3	38.6
BLIP2 A 3D model of a house on a gray background.
$\rightarrow$  +GPT4 A 3D model of a house with a roof, garage, grass, trees, and a green field, featuring a knife and a pair of scissors.
Figure 5: Objverse Caption Ablations. GPT produces longer and more detailed captions than BLIP2; CLIP tends to prune incorrect details and reduces length slightly.
+CLIP (Cap3D) A 3D model of a house with a garage, roof, grass and trees.
has shown a higher correlation with human judgments than other automated metrics [62]. ViLT [101] is trained on different data and is a different architecture than CLIP, providing an orthogonal metric.
Results. We report large scale A/B testing (1) against Human in Table 1, which shows Cap3D is better across metrics, with high confidence. The top three rows of Table 3 use the smaller human-captioned split (2), and demonstrate Cap3D's superior performance over Objaverse metadata and human-authored captions across A/B studies and automated metrics. The bottom three rows of Table 3, studied across a random split of the full dataset (3), reveal that while BLIP2 is effective, incorporating multiple views with  $+GPT4$  enhances performance. As shown in Figure 5, GPT4 adds detail by consolidating view-specific information. Filtering using  $+CLIP$  ( $Cap3D$ ) mitigates false details by purging subpar captions from GPT input. In addition to reducing errors, utilizing CLIP also reduces GPT input captions from 40 to 8, effectively decreasing token numbers and facilitating a cost reduction from $15.33 to $4.18.
5.2 Geometry 3D Captioning on ABO
Dataset. We evaluate geometric captioning on a 6.4k object split from ABO [35, 78], comparing Cap3D captions for each object against a maximum of two human-authored ones. To emphasize geometric focus, images used for model input and human assessment are texture-free and colorless.