Think-Program-reCtify: 3D Situated Reasoning with Large Language Models
Authors: Qingrong He, Kejun Lin, Shizhe Chen, Anwen Hu, Qin Jin


Reviewer paper summaries: This paper introduces a Think-Program-reCtify (TPC) loop method for 3D situated reasoning tasks. Experiments show the proposed method can outperform the end-to-end models on SQA3D benchmark. The paper also provides ablation study on different 3D visual perception modules and each part of the TPC loop, and breakdown analysis of different models on different question types.
Summary of strengths: - The paper is well-written. 
- The paper conducts comprehensive ablation study and breakdown analysis, which provide insights of the effects of the 3D perception modules, each part of the TPC loop, and different models' performance on different question types.
Summary of weaknesses: - The introduced TPC loop is not novel. This kind of prompting method with LLMs has been widely used in various embodied tasks, like robotic manipulation, and navigation, etc. [1, 2] though this idea hasn't been tried on SQA3D benchmark.
- I am not familiar with the 3D situated reasoning field. But according to the experimental results, the proposed method is inferior to the end-to-end supervised methods without ground truth information which is the more common scenario. Therefore, this paper does not provide a better method to solve the 3D situated reasoning task though there is an ensemble method.

References
- [1] Liang, Jacky, et al. "Code as policies: Language model programs for embodied control." 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.
- [2] Zhou, Gengze, Yicong Hong, and Qi Wu. "Navgpt: Explicit reasoning in vision-and-language navigation with large language models." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 7. 2024.
Comments, suggestions and typos: My main concern is that the proposed method is not novel, and it does not show superior performance than the end-to-end supervised ones. But the provided ablation study and analysis are insightful.
Overall assessment scores: 2.5