SlowGuess/NeurIPS_2023_OCR huggingface dataset:
B-1	B-2	B-3	B-4	METEOR	ROUHE-L	CIDER	EM
VoteNet+MCAN*	28.0	16.7	10.8	6.2	11.4	29.8	54.7	17.3
ScanRefer+MCAN*	26.9	16.6	11.6	7.9	11.5	30	55.4	18.6
ScanQA*	30.2	20.4	15.1	10.1	13.1	33.3	64.9	21.0
LLaVA(zero-shot)	7.1	2.6	0.9	0.3	10.5	12.3	5.7	0.0
ULIPPointMLP+flant5	18.4	7.2	2.7	1.4	7.4	18.1	26.9	7.5
ULIPPointMLP+opt	19.1	7.3	2.7	1.9	7.4	18.2	28.0	8.4
ULIPPointBERT+flant5	29.2	17.9	10.3	6.1	11.6	28.1	50.9	14.5
ULIPPointBERT+opt	28.8	16.9	9.7	5.9	11.3	27.9	50.5	13.8
flamingo-SingleImage	23.8	14.5	9.2	8.5	10.7	29.6	52	16.9
flamingo-MultiView	25.6	15.2	9.2	8.4	11.3	31.1	55	18.8
BLIP2-flant5-SingleImage	28.6	15.1	9.0	5.1	10.6	25.8	42.6	13.3
BLIP2-flant5-MultiView	29.7	16.2	9.8	5.9	11.3	26.6	45.7	13.6
3D-LLM (M2F, flamingo)	30.3	17.8	12.0	7.2	12.2	32.3	59.2	20.4
3D-LLM (M2F, BLIP2-opt)	35.9	22.5	16.0	9.4	13.8	34.0	63.8	19.3
3D-LLM (SAM, BLIP2-opt)	35.0	21.7	15.5	9.5	14.0	34.5	67.1	19.8
3D-LLM (M2F, BLIP2-flant5)	39.3	25.2	18.4	12.0	14.5	35.7	69.4	20.5
3D-LLM (SAM, BLIP2-flant5)	37.5	24.1	17.6	12.9	15.1	37.5	74.5	21.2
Table 1: Experimental results on ScanQA validation set. * Means the models use explicit object representations. B-1, B-2, B-3, B-4 denote BLEU-1, BLEU-2, BLEU-3, BLEU-4 respectively. M2F denotes mask2former, SAM denotes Segment Anything.
Format	test set	Avg.
What	Is	How	Can	Which	Others
Blind test	SQ → A	26.75	63.34	43.44	69.53	37.89	43.41	43.65
ScanQA(w/o s\( ^{\text{txt}} \))	VQ → A	28.58	65.03	47.31	66.27	43.87	42.88	45.27
ScanQA	VSQ → A	31.64	63.80	46.02	69.53	43.87	45.34	46.58
ScanQA+aux task	VSQ → AL	33.48	66.10	42.37	69.53	43.02	46.40	47.20
MCAN	VSQ → A	28.86	59.66	44.09	68.34	40.74	40.46	43.42
ClipBERT	VSQ → A	30.24	60.12	38.71	63.31	42.45	42.71	43.31
Unified QA	VSQ → A	33.01	50.43	31.91	56.51	45.17	41.11	41.00
Unified QA	VSQ → A	27.58	47.99	34.05	59.47	40.91	39.77	38.71
GPT-3	VSQ → A	39.67	45.99	40.47	45.56	36.08	38.42	41.00
GPT-3	VSQ → A	28.90	46.42	28.05	40.24	30.11	36.07	34.57
3D-LLM	VSQ → A	37.05	65.18	45.81	67.46	51.00	49.82	49.79
Table 2: Experimental Results on SQA3D test set. In the Format column, "V" means the 3D visual inputs, "S" means the situation inputs, "Q" and "A" denote questions and answers respectively. Here we use 3D-LLM (SAM, BLIP2-flant5).
Result Analysis We report our results on ScanQA validation set in Table 1. We observe a significant increase in the evaluation metrics. For example, for BLEU-1, our model outperforms the state-of-the-art ScanQA model by  $\sim 9\%$  for validation set. For CIDER, we report a  $\sim 10\%$  gain compared to ScanQA, and much higher than other 3D-based baselines. These results show that by injecting 3D into LLMs, the models can generate answers that are much more similar to the ground-truth answers. Furthermore, 3D-based baselines use object detectors like VoteNet to segment the objects, and then send per-object features into their models, while our inputs are holistic 3D features without explicit object representations. This shows that our model could perform visual reasoning about objects and their relationships even without explicit object representations. We then examine whether 2D VLMs have the same ability. We find that by taking single-view images or multi-view images as inputs, the performances drop much compared to 3D-LLMs. Specifically, multi-view images also contain information about the whole scene. However, they have much lower performances compared to 3D-LLMs, probably because features of multi-view images are disorganized, thus losing 3D-related information.
5.1.2 Experiments on SQA3D
SQA3D [32] requires the tested agent to first understand its situation (position, orientation, etc.) in the 3D scene as described by text, then reason about its surrounding environment and answer a question under that situation. We finetune our pretrained 3D-LLMs on the SQA3D dataset and compare with baseline models. We include all baseline models introduced by the original paper. Specifically, ScanQA+aux task achieves the SOTA performance by adding two auxiliary tasks: prediction the
position and rotation of the agent situation. Table 2 shows the results. We can see that our 3D-LLM outperforms all baseline models a lot, even without training with auxiliary tasks and losses.
Methods	Concept	Counting	Relation	Comparison	Overall
NS-VQA*	59.8	21.5	33.4	61.6	38.0
3D-Feature+LSTM	61.2	22.4	49.9	61.3	48.2
3D-CLR*	66.1	41.3	57.6	72.3	57.7
flamingo-SingleImage	58.7	18.5	38.4	60.1	40.3
flamingo-MultiView	60.0	18.3	40.2	61.4	41.6
BLIP-SingleImage	58.0	20.4	42.3	62.3	43.1
BLIP-MultiView	61.9	21.1	48.0	62.3	47.1
3D-LLM (M2F, flamingo)	68.9	32.4	61.6	68.3	58.6
3D-LLM (M2F, BLIP2-opt)	63.4	30.7	57.6	65.2	54.9
3D-LLM (SAM, BLIP2-opt)	73.4	24.5	63.2	77.6	61.5
3D-LLM (M2F, BLIP2-flanT5)	68.1	31.4	55.1	69.7	54.6
3D-LLM (SAM, BLIP2-flanT5)	76.3	30.2	64.3	80.2	64.0
Table 3: Experimental results on 3DMV-VQA dataset. * denotes using explicit object representations and neuro-symbolic reasoning.
5.1.3 Experiments on 3DMV-VQA
We finetune our pretrained 3D-LLMs on the 3DMV-VQA dataset and compare with baseline models. We include all baseline models introduced by the original paper. Specifically, 3D-CLR [20] is the SOTA achieves the SOTA performance via neuro-symbolic reasoning based on 3D features.
Result Analysis Table 3 shows the performances on 3DMV-VQA. We can see that 3D-LLMs outperform state-of-the-art baseline model in the question types of concept and relation, and also in the overall performance. Our model also outperforms 3D-Feature+LSTM, demonstrating the power of LLMs over vanilla language models with similar 3D features as inputs. Overall, 3D-based methods outshine 2D-based versions of the methods. Our 3D-LLMs outperform their corresponding 2D VLMs with image input, further demonstrating the importance of 3D representations for 3D-LLMs.
Tasks	Models	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	ROUGH-L
3D Captioning	flamingo-SingleImage	29.0	17.9	12.5	12.1	12.4	28.2
flamingo-MultiView	29.5	18.6	13.7	12.4	14.0	29.0
BLIP2-flant5-SingleImage	30.3	18.3	14.5	12.0	13.1	30.9
BLIP2-flant5-MultiView	34.4	23.9	18.0	14.1	17.5	35.7
3D-LLM (flamingo)	36.1	24.5	18.7	15.6	17.6	35.8
3D-LLM (BLIP2-opt)	35.7	26.7	20.3	15.9	18.7	40.1
3D-LLM (BLIP2-t5)	39.8	31.0	24.7	20.1	17.7	42.6
3D-LLM (SAM, BLIP2-t5)	44.5	38.6	29.5	24.2	22.1	45.4
3D-assisted Dialog	flant5	27.4	16.5	11.1	8.7	9.5	27.5
flamingo-SingleImage	29.4	18.7	11.3	9.4	10.0	26.8
flamingo-MultiView	30.6	21.3	11.9	9.1	10.4	27.9
BLIP2-flant5-SingleImage	28.4	17.3	10.6	9.1	10.2	27.4
BLIP2-flant5-MultiView	32.4	20.9	12.1	9.5	11.0	29.5
3D-LLM (flamingo)	35.0	22.8	15.4	10.6	16.0	34.2
3D-LLM (BLIP2-opt)	39.6	27.5	20.5	16.2	18.4	38.6
3D-LLM (BLIP2-flant5)	39.0	27.8	21.2	16.6	18.9	39.3
3D-LLM (SAM, BLIP2-t5)	40.5	29.4	23.9	21.4	19.6	40.8
Task Decomposition	flant5	25.5	21.1	16.7	6.0	13.9	28.4
flamingo-SingleImage	31.4	23.0	18.8	7.1	15.6	30.6
flamingo-MultiView	33.1	24.7	21.4	7.3	16.1	33.2
BLIP2-flant5-SingleImage	32.2	25.3	18.2	6.9	15.0	31.0
BLIP2-flant5-MultiView	33.1	27.0	20.6	6.9	15.5	34.0
3D-LLM (flamingo)	32.9	25.6	20.2	6.4	16.0	33.5
3D-LLM (BLIP2-opt)	34.1	27.7	20.8	7.6	16.5	35.4
3D-LLM (BLIP2-flant5)	33.9	28.1	20.7	7.4	15.9	37.8
3D-LLM (SAM, BLIP2-t5)	31.6	22.3	17.2	8.8	14.0	38.3
Table 4: Experimental Results on Held-In Datasets. 3D-LLMs outperform 2D VLMs.
5.2 More Extensive Evaluation
Held-In Evaluation We carry out experiments on held-in datasets of three tasks: 3D captioning, 3D-assisted dialog and task decomposition. The baselines include 2D VLMs as for the held-out evaluation. We add one language-only baseline: FlanT5, which examines LLMs' ability to complete these tasks without any visual input. To evaluate the quality of responses, we include BLEU, ROUGE-L, METEOR, CIDEr as our metrics. We report the held-in evaluation performances in Table 4. From the table, we could see that 3D-LLMs could generate high-quality responses, outperforming both 2D VLMs and language-only LLMs.
Qualitative Examples In Figure 4, we show qualitative examples of 3D-LLM's predictions. We can see that our 3D-LLM is able to perform a variety of tasks.
Figure 4: Qualitative examples of 3D-LLM's prediction.
6 Conclusion
In this paper, we propose a new family of 3D-LLMs that can take 3D representations as inputs and generate responses. We introduce a series of 3D-language data generation pipelines to generate a dataset of 1M 3D-language pairs to train our 3D-LLMs. Our 3D-LLMs leverage 2D pretrained VLMs as backbones and a novel 3D localization mechanism. Experiments show that our 3D-LLMs outperform state-of-the-art baseline models on ScanQA datasets, and could perform a diverse set of 3D-related tasks. A limitation is that the 3D feature extractor relies on 2D multi-view images, and thus all 3D scenes need to be rendered so that they can be trained in 3D-LLMs, which introduces an additional rendering process.
7 Acknowledgements
This work was supported by the MIT-IBM Watson AI Lab, DARPA MCS, DSO grant DSOCO21072, and gift funding from MERL, Cisco, Sony, and Amazon. We would also like to thank the computation support from AiMOS, a server cluster for the IBM Research AI Hardware Center.
[41] Q. Sun, Y. Fang, L. Wu, X. Wang, and Y. Cao. Eva-clip: Improved training techniques for clip at scale, 2023.
[42] Z. Sun, Y. Shen, Q. Zhou, H. Zhang, Z. Chen, D. Cox, Y. Yang, and C. Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. arXiv e-prints, pages arXiv-2305, 2023.
[43] P. Wang, A. Yang, R. Men, J. Lin, S. Bai, Z. Li, J. Ma, C. Zhou, J. Zhou, and H. Yang. Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In International Conference on Machine Learning, 2022.
[44] K. Yadav, R. Ramrakhya, S. K. Ramakrishnan, T. Gervet, J. Turner, A. Gokaslan, N. Maestre, A. X. Chang, D. Batra, M. Savva, et al. Habitat-matterport 3D semantics dataset. arXiv preprint arXiv:2210.05633, 2022.
[45] S. Ye, D. Chen, S. Han, and J. Liao. 3D question answering. IEEE transactions on visualization and computer graphics, PP, 2021.
[46] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69-85. Springer, 2016.
[47] P. Zhang, Y. Goyal, D. Summers-Stay, D. Batra, and D. Parikh. Yin and Yang: Balancing and answering binary visual questions. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[48] D. Zhu, J. Chen, K. Haydarov, X. Shen, W. Zhang, and M. Elhoseiny. Chatgpt asks, blip-2 answers: Automatic questioning towards enriched visual descriptions, 2023.