SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The two prompts used for BLIP2 used in Cap3D (QA) are (1) "Question: what object is in this image? Answer:" and (2) "Question: what is the structure and geometry of this <object>?" where <object> is replaced with the response to prompt (1).
For the prompt used in GPT4, we used "Given a set of descriptions about the same 3D object, distill these descriptions into one concise caption. The descriptions are as follows: 'captions'. Avoid describing background, surface, and posture. The caption should be:". We did several prompt engineering and considered prompt with more context, like "Below you will find a set of descriptions, each one is originating from various renderings of an identical 3D object. The level of accuracy in these descriptions ranges significantly: some might not correspond to the 3D object at all, others could be entirely accurate, while a few may only partially represent the object. Your task involves scrutinizing these descriptions and distilling them into a single, holistic depiction. The descriptions are as follows: 'captions'. Note: Please avoid using the phrases 'grey background', 'gray background', and 'gray surface' in your consolidated depiction. The synthesized description of the 3D object should be:". However, with those longer prompt with more context, we noticed GPT4 sometimes would generate its reasoning process which led to confusing output captions. Also, for the sake of cost, we hope to make our prompt as short as possible.
F.2 Rendering Details
We use Blender to render 3D objects in Objaverse [9] and ABO [35]. For each object, we first normalize them into a unit cube and recenter to origin. Then, we place 8 different cameras surrounding the object with 2 cameras slightly below the object to capture the bottom of the object. Three area lights are placed and function as key light, fill light, and rim light, respectively. The detailed parameters are listed in our rendering script, provided in our Github.
In Objaverse, we filter out objects that fail in rendering, resulting a subset of 785k objects for rendering and captioning. In ABO, we exclude categories with simple geometry to concentrate on geometrical captioning, including "BLANKET", "RUG", "WALL_ART", "PLACEMAT", "CURTAIN", "MOUSEPAD". This resulting a final subset of 6.4k objects for rendering and captioning.
F.3 Human Captioning Split
Human captions are collected on a manually selected subset of Objaverse with good renders of nontrivial but decipherable objects. These objects are likely to be the most sensible for captioning and A/B testing. For instance, some Objaverse objects are essentially a simple rock with little texture; in others it can be difficult for a human to describe an object (e.g. abstract art, no clear object visible, or 3D scans with hard-to-distinguish details). These excluded objects are generally not effective samples to use for human A/B testing, as the correct caption may not be clear or may be trivial. We also exclude furniture, which is suitable for captioning, but we measure this with more focus on ABO. Human captions on ABO follow the split of [78].
Appendix G Crowdsourced Captioning Details
We use Hive for crowdsourced captioning. Workers are given instructions for the task including gold-standard examples. Captioning instructions are shared below for Objaverse in Figure 24 and ABO in Figure 25. Workers are persistently monitored. If a worker produces bad captions they are promptly banned from captioning, and their previous captions are discarded. Workers are paid approximately  $50 per 1k tasks. We do not have access to their captioning rates; assuming a rate of 3 objects per minute, this would result in$ 9 per hour. Across Objaverse and ABO we spend a total of $7k on captioning.
Please describe the structure of the 3D object. Each answer should answer the following three questions.
(1) What pieces make up the object? (e.g. back, seat, legs, wheel, window)
(2) What shape are these pieces? (e.g. round, flat, square, long, narrow, wide, large)
(3) How are these pieces connected? (e.g. left, right, above, below, connected, held up by, on, inside of, has, contains)
**Please do not describe the color or texture (texture / color is purposely left out of renders).**
**Captions must be in English. If you do not provide real answers you will promptly be permanently banned; we actively monitor annotations**
Caption: fully upholstered londn armchair with four thick legs, two straight trapezoidal legs at the front and two curved legs at the back
Caption: long, rectangular coffee table with a lower shelf, legs in each corner and a designed piece at each end.
Figure 25: ABO Caption Instructions.
Caption: corner I-shaped sofa with three seats connected to a rectangular shaped bed.
Caption: the bed is has a wide, round backrest. it has two pillows and no visible legs.
Please describe the object pictured in the image in terms of appearance and geometry in at least two sentences of detail.
Caption: A cartoon donut that is vanilla with pink frosting and white sprinkles. The donut is round with a hole in the middle.
$\rightarrow$  Note: we describe in detail both appearance (cartoon donut, vanilla, pink frosting, white sprinkles) and geometry (round).
Caption: A yellow cartoon duck with a white beak and 3 dark hairs coming out of its head. It is the Pokemon Psyduck, and has a large round torso with short arms and legs.
$\rightarrow$  Note: we describe in detail both appearance (yellow cartoon duck, white beak, dark hairs) and geometry (large round torso, short arms and legs, 3 hairs out of head). If you don't know specific names (e.g. Pokemon Psyduck) that is no problem! But if you do know, even better!
Caption: An early airplane model, it is very small and the frame is made of wood and the wings are made of a beige material. It has two wheels in the front and one in the back in the center.
Note: we describe in detail both appearance (airplane model, frame is wood, wings beige) and geometry (very small, two wheels in front, one wheel in back in center).
Figure 24: Objverse Caption Instructions.
Caption: A light-green toy pony with colorful hair on its head and tail. It has big eyes and stars on its cheeks and fluffy hair and tail.
$\rightarrow$  Note: we describe in detail both appearance (light-green toy pony, colorful hair on head and tail, stars on cheeks) and geometry (big eyes, fluffy hair and tail).
Appendix H Crowdsourced A/B Testing Details
We use Hive for crowdsourced A/B testing. Specifically, workers are given an image and two captions, and select which is better on a scale from 1 to 5, where 3 is a tie. So 1 would be "left much better", and 2 would be "left better". Workers are given instructions for the task along with gold standard examples. Workers are informed to prioritize accuracy, then informative detail, then brevity. Left/right order between methods was randomized for each instance. A/B Testing instructions are shared below for Objaverse in Figure 27 and ABO in Figure 26.
Workers are automatically banned by the platform if they miss too many gold-standard examples. However, we found some workers would successfully pass the handful of gold-standard examples while scamming on the rest of the examples. The most common scam cases were always picking the same number, or always picking the shorter or longer caption. We thus manually search through all workers and ban workers who meet these scamming criteria and discard their judgments. Unfortunately, discarding judgments leads to uneven numbers of observations for each individual experiment. Nevertheless, in all cases, enough observations are available to draw conclusive findings.
The size of each experiment's data after discarded judgments is below.
- Objaverse Split (1) takes place on a random set upon which human captions are available. Cap3D vs. Human has 36k observations across 22k objects.
- Objaverse Split (2) takes place on a random object set upon which human captions are available. Cap3D vs. Human has 10k observations across 4.7k objects. Cap3D vs. Metadata has 7k observations across 4.7k objects (less than the target 10k), though given the extremely poor rating of Metadata, results are conclusive.
- Objaverse Split (3) takes place on a random object set upon the entire Objaverse dataset. Cap3D vs. BLIP2 has 20k observations across 5.0k objects and Cap3D vs. +GPT4 has 29k observations across 5.0k objects.
- ABO takes place on the full ABO object set. Human vs. Cap3D has 21k observations across 6.4k objects, Cap3D (QA) vs. Human has 17k observations across 6.4k objects, Cap3D (QA) vs. Cap3D has 13k observations across 6.4k objects, and Cap3D (QA) vs. Meta has 12k observations across 6.4k objects.
Workers are paid approximately  $20 per 1k tasks. We do not have access to their captioning rates; assuming a rate of 7.5 A/B tests selected per minute, this would result in$ 9 per hour. Across Objaverse and ABO we spent a total of $1.8k on A/B testing.
In this task, an object will be pictured from 8 views. There will be two captions ("left" caption and "right" caption).
Please select the caption best describing the pictured object in terms of type, appearance, and structure.
- Prioritize captions that make accurate statements.
- Select captions with clear, informative object details
- Ignore caption details about the background (these are neither good nor bad to the task)
- NOTE: There is only one object. It appears 8 times since it is shown from 8 views
----> If a caption says there are multiple objects and there is only one object in each of 8 views, the caption is wrong
Select from the following options:
1. Left Much Better - left caption describes the object with much more accuracy, informative detail, or avoiding many unnecessary details
2. Left Better: left caption describes the object with more accuracy, informative detail, or avoiding unnecessary details
3. Tie: both captions have similar accuracy and detail
4. Right Better: right caption describes the object with more accuracy, informative detail, or avoiding unnecessary details
5. Right Much Better: right caption describes the object with much more accuracy, informative detail, or avoiding many unnecessary details
Here are some examples
Example 1: tie
Correct Selection. 3: The
Correct Selection: 5. Right Much Better
Explanation: The left caption contains many inaccuracies (this is a table, has a spiral staircase with a harp on it, had a square ceiling light). Since the right caption is correct, we select right much better.
Example 2: Missing caption (or caption not in English)
Correct Selection. 5: Right Much Better
Explanation: The left caption is missing, and the right caption is correct, so we select the right caption is much better.
Example 3: Caption unrelated to image
Correct Selection. 5: Right Much Better
Explanation: The left caption is unrelated to the image, while the right caption is correct. So we select right much better
Example 4: one inaccuracy
Correct Selection: 2. Left Better
Example 5: many inaccuracies
Figure 26: A/B Instructions: Objverse Captions.
Explanation: IGNORE INFO ABOUT BACKGROUND. Both captions have similar accuracy and detail (white spaceship), so we select tie
Example 6: one additional clear and informative detail
Correct Selection. 2: Left Better
Explanation: The left caption has the correct and useful additional detail the object shown is white with purple eyes. Not all human skull models are white with purple eyes, so we select left is better.
Example 7: many additional clear and informative details
Correct Selection. 5: Right Much Better
Explanation: The right caption contains a number of additional informative details: the jersey is black and yellow, it has flames, and the word jetmaster on it. These are all clear and informative details, so we select right much better.
Example 8: one additional unclear or uninformative detail
Correct Selection: 2: Left Better
Explanation: Both captions describe this as a blue, yellow and green cube. However the one on the right says this is a cube "and object." It is unclear what this means - whether there are actually two objects in each view or something else. But the caption on the left is clearer.
Example 9: many unclear or uninformative details
Correct Selection: 5: Right Much Better
Explanation: The caption on the right correctly describes this as a red and golf tetrahedral structure with a cross, and that it resembles a molecule or chistmas tree. The caption on the left has many details such as the different colors and types of objects making up the full structure, but does not describe what is important - which is that the shape of the overall structure looks like a Christmas tree / molecule and that it has a cross.
In this task, an object will be pictured from 6 views. There will be two captions ("left" caption and "right" caption).
Please select the caption best describing the structure of the pictured object
- Language describing the color or texture of the object or the background should be ignored -- only judge based on object geometry.
- Prioritize captions that make accurate statements about the object.
- Select captions with informative object details but not ones with unrelated or repeated (redundant) details.
- NOTE: There is only one object. It appears 6 times since it is shown from 6 views
----> If a caption says there are multiple objects and there is only one object in each of 6 views, the caption is wrong
1. Left Much Better: left caption describes the object geometry with much more accuracy, informative detail, or avoiding many unnecessary details
2. Left Better: left caption describes the object geometry with more accuracy, informative detail, or avoiding unnecessary details
3. Tie: both captions have similar accuracy and detail
4. Right Better: right caption describes the object geometry with more accuracy, informative detail, or avoiding unnecessary details
5. Right Much Better: right caption describes the object geometry with much more accuracy, informative detail, or avoiding many unnecessary details
Correct Selection. 5: Right Much Better
Explanation: Left caption is not in English. The right caption is in English.
Example 2: Caption unrelated to image or very inaccurate
Correct Selection. 5: Right Much Better
Explanation: Left caption is not related to the image / very inaccurate. The left caption is correct.
Example 3: tie
Correct Selection. 3 Tie
Explanation: Both captions have a similar amount of correct detail about the object structure: It is a bar stool (or tail chair for bainr早餐), it has a cushion (ophistiled) on the seat, and it has four legs / stained backrest. So we select tie.
Example 4: one inaccuracy
Correct Selection: 2: Left Better
Explanation: Both captions correctly identify this as a sloo, but the right caption says it has a backrest, which is wrong.
Example 6: many inaccuracies
Correct Selection: 5; Right Much Better
Explanation. The right caption correctly identifies this as an office desk. The one on the left says this it has a trapezoid top (wrong), a hinged door (wrong) and a chair (wrong). So we select right much better.
Example 8: one additional informative detail
Correct Selection. 4 Right Better
Explanation: The left caption contains that this is a lamp. The right contains that it is a lamp but also that it has a long round stand. Since this detail is correct, we select right better:
Example 7: many additional information details
Correct Selection. 1: Left Much Better
Explanation: The right caption is correct in describing this as a living room chair, but the left caption has many more correct details about its structure: It has a wing-shaped back, stained seat, backrest and armrests
Example 8: one unrelated or repeated (redundant) detail
Correct Selection: 4. Right Better
Explanation: The left caption says this is a source of light that stands of a table or any piece of furniture. While this is true, it is not relevant to describing the object structure. The right caption has more detail - that the table lamp has a shade on it.
Example 9: many irrelevant or repeated (redundant) details
Correct Selection: 5. Right Much Better
Explanation. The left caption describes this as a wooden coffee table, a round coffee table, a living room/round cocktail table with storage, a sofa table with shelf and decorative tables with carved legs. These are many repeated unnecessary details. Since the right caption describes it
Figure 27: A/B Instructions: ABO Captions.
Appendix I Additional Experimental Details
Captioning: we perform one full-scale evaluation run for all captioning experiments;  $95\%$  confidence interval for mean is presented. Metrics are overviewed in  $\S 5.1$ ; A/B testing is detailed further in  $\S H$ . CLIP Score takes about 5 minutes, while ViLT R-Precision takes about 8 hours using an A40 for test set of 5k object-caption pairs. Crowdsourced A/B testing takes about 12 hours for 10k responses across 5k objects.
Text-to-3D, finetuning: for finetuning experiments, we used one train and evaluation run using a learning rate validated on a small overfitting experiment on the train set. Training took about 3 days on the full set and 1 day on the small (human) set. We used AdamW optimizer and CosineAnnealingLR scheduler with initial learning rate  $1e - 5$  for finetuning both Point-E and Shap-E. We adopted batch size 64 and 256 for Shap-E and Point-E, respectively. However, for Shap-E, we found it usually outputs NaN and needed to re-start from saved checkpoints, which could be one of the reasons why our finetune did not bring improvements. For LoRA, we use AdamW optimizer and CosineAnnealingLR scheduler with initial learning rate  $1e - 4$  and batch size of 3. For ControlNet, we use AdamW optimizer and constant learning rate of  $1e - 5$  and batch size of 8. Experiments use 4 A40s to train except LoRA, which fails upon multi-gpu training due to a HuggingFace internal DDP error. Notably single-gpu training still yields improvement. Evaluation takes the following time (in seconds) per iteration, which includes rendering:
- PointE (text-to-3D): 37sec = 28sec (text-to-3D) + 9sec (render)
- LoRA + PointE(im-to-3D): 114sec = 5sec + 100sec (im-to-3D) + 9sec (render)
- ControlNet + PointE(im-to-3D): 124sec = 15sec + 100sec (im-to-3D) + 9sec (render)
- ShapE (NeRF): 193sec (text-to-3D + render)
- ShapE (stf): 16sec (text-to-3D + render)
Note publicly available PointE (im-to-3D) is 1B param, making it slower than the largest publicly available PointE (text-to-3D) of 40M. Evaluation metrics are detailed in §5.3.
Text-to-3D, optimization: For one object, optimization plus final rendering takes 40 minutes for 3DFuse, 95 minutes for Stable DreamFusion, and 35 minutes for DreamField; using 1 A40 GPU. We use default parameters for all methods and run them once.