SQA3D: Situated Question Answering in 3D Scenes
Authors: Xiaojian Ma
Silong Yong
Zilong Zheng
Qing Li
Yitao Liang
Song-Chun Zhu
Siyuan Huang
Venue: ICLR 2023 poster (2023)
Keywords: 3D vision
scene understanding
visual question answering
embodied AI

Abstract: We propose a new task to benchmark scene understanding of embodied agents: Situated Question Answering in 3D Scenes (SQA3D). Given a scene context (e.g., 3D scan), SQA3D requires the tested agent to first understand its situation (position, orientation, etc.) in the 3D scene as described by text, then reason about its surrounding environment and answer a question under that situation. Based upon 650 scenes from ScanNet, we provide a dataset centered around 6.8k unique situations, along with 20.4k descriptions and 33.4k diverse reasoning questions for these situations. These questions examine a wide spectrum of reasoning capabilities for an intelligent agent, ranging from spatial relation comprehension to commonsense understanding, navigation, and multi-hop reasoning. SQA3D imposes a significant challenge to current multi-modal especially 3D reasoning models. We evaluate various state-of-the-art approaches and find that the best one only achieves an overall score of 47.20%, while amateur human participants can reach 90.06%. We believe SQA3D could facilitate future embodied AI research with stronger situation understanding and reasoning capability.

Published as a conference paper at ICLR 2023
SQA3D:
SITUATED QUESTION ANSWERING IN 3D SCENES
Xiaojian Ma ,ilong Yong!. 1.3g Zilong Zheng', Qing Li Yitao Liang!.4
Song-Chun Zhu Zhu 1,23,4 Siyuan Huang
BBeijing Institute for General Artificial Intelligence (BIGAI) PUCLA TTsinghua University *Peking University
xiacjian ma@ucla.edu, yongz 19maalls.ll.....mm tengglllseaalstttssnhhuuaa edu. {zl zheng, 1 ginin, sezhu, syhuanggehigai.aa titall@kku edu.
Deseription txx Sitting at the edge of the bed and facing the couch.
Answer a: No
Question 4: Can I straight to the coffee table in front of me?
Location (optional): (s pos rot
Scene context $ 3D scan, egocentric video, bird- eye view (BEV) picture. etc.
Figure I: Task illustration of Situated Question Answering in 3D Scenes (SQA3D). Given context S (e.g., 3D sean, egocentric video, bird-eye view picture). SQA3D requires an agent to first comprehend localize its situation (position, orientation, etc.) in the 3D scene from a textual description pt then answer a question under that situation. Note that understanding the situation and imagining the corresponding egocentric view correctly is necessary to accomplish our task. We provide exampic questions in Figure 2.
ABSTRACT
We propose a new task to benchmark scene understanding embodied agents: Situated Question Answering in 3D Scenes (SQA3D). Given (e.g., 3D Scan), SQA3D requires the tested agent to first understand its situation (position, orientation, etc.) in the 3D scene as described by text, then reason about its surrounding environment and answer a question under that situation. Based upon 650 scenes from ScanNet, we provide dataset centered around 6.8k unique situations, along with 20.4k descriptions and 33.4k diverse reasoning questions for these situations. These questions examine wide spectrum of reasoning ca- pabilities for intelligent agent, ranging from spatial relation comprehension to commonsense understanding, navigation, and multi-hop reasoning. SQA3D imposes significant challenge current multi-modal especially 3D reasoning models. We evaluate various state-of-the-an approaches and find that the best one only achieves overall score of 47.20%, while amateur human participants reach 90.06%. We believe SQA3D could facilitate future embodied Al research with stronger situation understanding and reasoning capabilities. Code and data are released at github.b......
context
INTRODUCTION
In recent years, the endeavor of building intelligent embodied agents has delivered fruitful achieve- ments. Robots now can navigate (Anderson al., 2018) and manipulate objects (Liang al., 2019; Savva et al., 2019; Shridhar et al., 2022; Ahn al., 2022) following natural language commands
First authors contributed equally. Correspondence Zilong Zheng and Siyuan Huang.

---

Published as a conference paper at ICLR 2023
Figure 2: Examples from SQA3D. We provide some example questions and the corresponding situations (s and and 3D scenes, The categories listed here do not mean to be exhaustive and a question could fall into multiple categories. The green boxes indicate relevant objects situation description SM while red boxes are for the questions
Embodied activities
Navigation
Common sense
Multihop reasoning
.xx, Stunding in front of the sink and lacing the towels.
stxt, Working by the desk and the window is on right.
sbo,; Just looking for food in the fridge. q: Which direction should I go heat my lunch?
5 Playing computer gamess and the window on my right.
q: Can see myself the mimor?
q: How many thains will pass open the window from other side of the desk?
q: How many monitors are there the desk that the chair on my left is facing?
a: No
a: Right
a: Three
or dialogues. Albeit these promising advances, their actual performances in real-world embodied environments could still fall short of human expectations, especially in generalization to different situations (scenes and locations) and tasks that require substantial, snowledge-intensti To diagnose the fundamental capability of realistic embodied agents, we investigate the problem of embodied scene understanding. where the agent needs to understand its situation and the surround- ings in the environment from dynamic egocentric view, then perceive, reason, and act accordingly, to accomplish complex tasks.
reasoning.
What is at the core of embodied scene understanding? Drawing inspirations from situated cog- nition (Greeno, 1998; Anderson et al., 2000), a seminal theory of embodiment, we anticipate it to be two-fold:
Situation understanding. The ability to imagine what the agent will see from arbitrary situa- tions (position, orientations, etc.) in a 3D scene and understand the surroundings anchored the situation, therefore generalize to novel positions or scenes; Situated reasoning. The ability to acquire knowledge about the environment based on the agents' current situation and reason with the knowledge, therefore further facilitates accomplishing com- plex action planning tasks.
To step towards embodied scene understanding, we introduce SQA3D, new task that reconciles the best of both parties, situation understanding, and situated reasoning, into embodied 3D scene understanding. Figure 1 sketches our task: given a 3D scene context (e.g., 3D scan, ego-centric video, or bird-eye view (BEV) picture), the agent in the 3D scene needs to first comprehend and lo- calize its situation (position, orientation, etc.) from textual description, then answer question that requires substantial situated reasoning from that perspective. We rowdd-soureed the situation de- scriptions from Amazon MTurk (AMT), where participants are instructed to select diverse locations and orientations in 3D scenes. To systematically examine the agent's 's ability in situated reasoning, collect questions that cover wide spectrum of knowledge, ranging from spatial relations to navigation, common sense reasoning, and multi-hop reasoning. In total, SQA3D comprises 20.4k descriptions of 6.8k unique situations collected from 650 ScanNet scenes and 33.4k questions about these situations. Examples of SQA3D can be found Figure 2.
Our task closely connects the recent efforts on 3D language grounding (Dai et al., 2017; Chen et al., 2020; 2021; Hong et al., 2021b; Achlioptas et al., 2020; Wang et al., 2022; Azuma et al., 2022). However, most of these avenues assume observations of 3D scene are made from some third-person perspectives rather than an embodied, egocentric view, and they primarily inspect spa- tial understanding, while SQA3D examines scene understanding with a wide range of knowledge, and the problems have to be solved using an (imagined) first-person view. Embodied QA (Das et al., 2018; Wijmans et al., 2019a) draws very similar motivation as SQA3D, but our task adopts a simplified protocol (QA only) while still preserving the function of benchmarking embodied scene understanding, therefore allowing more complex,
www.lege-intensiiee questions and a much larger scale of data collection. Comparisons with relevant tasks and benchmarks are listed in Table 1.
Benchmarking existing baselines: In our experiments, we examine state-of the-art multi-modal reasoning models, including ScanQA from Azuma al. (2022) that leverages 3D scan data, Clip-
2

---

Published as a conference paper at ICLR 2023
Table I: An overview of the different benchmark datasets covering grounded scene understanding. In general, we consider semantic grounding, language-driven navigation, and question-answering in photo- realistic 3D scenes. In the first row, situated indicates whether the benchmark task is supposed to be completed by a "situated" agent with its egocentric perspective. navigation, common sense, and multi-hop reasoning show whether the task requires certain capability or knowledge level 3D understanding. "Rather than observing a complete 3D sean of the scene, the learner needs to navigate in a simulator to perceive the 3D scene incrementally.
datasct
task
situated?
3D
text collection
navi- gation?
common sense"
multi-hop reasoning?
#scenos
#tasks
Type
ScanNet (Dai ot al., 2017)
seg.
X
scan
n/a
x
X
800 rooms
1.5k
ScanRefer (Chen et al,, 2020)
det.
scan
human
x
800 rooms
52k
Referit3D (Achlioptas et ol., 2020)
det.
scan
human
x
X
707 rooms
41k
ScanQA (Azuma et al., 2022)
q.a.
scan
template
X
X
800 rooms
41k
3D-QA (Ye al., 2021)
qa.
human
X
806 rooms
5 3.8k
CLEVR3D (Yan al,, 2021)
q.a.
scan
template
x
478 rooms
60k
MP3D R2R (Anderson al. 2018)
maw.
"nav.
floors
MP3D-EQA (Wijmans et al., 2019a)
"nav.
template
X
X
146 floors
1.1k
SQA3D (Ours)
q.a.
scan
human
650 rooms
33.4k
BERT (Lei et al., 2021) and MCAN (Yu et al., 2019) that exploits egocentric videos and BEV pictures. However, the results unveil that both models still largely fall behind human performances by large margin (47.2% of the best model vs. 90.06% of amateur human testers). To understand the failure modes, we conduct experiments on settings that could alleviate the challenges brought by situation understanding. The improvement of these models confirms that the current models are indeed struggling with situation understanding, which is pivotal for embodied scene understanding. Finally, we explore whether powerful Large Language Models (LLMs) like GPT-3 (Brown al., 2020) and Unified QA (Khashabi et al., 2020) could tackle our tasks by converting the multi-modal SQA3D problems into single-modal surrogates using scene captioning. However, our results read that these models can still be bottlenecked by the lack of spatial understanding and accurate captions.
Our contributions can be summarized as follow:
We introduce SQA3D, new benchmark for embodied scene understanding, aiming at reconciling the challenging capabilities of situation understanding and situated reasoning and facilitating the development of intelligent embodied agents.
We meticulously curate the SQA3D to include diverse situations and interesting questions. These questions probe wide spectrum of knowledge and reasoning abilities of embodied agents, rang- ing from spatial relation comprehension to navigation, common sense reasoning, and multi-hop reasoning.
We perform extensive analysis on the state-of-the-e art multi modal reasoning models. However, experimental results indicate that these avenues are still struggling on SQA3D. Our hypothesis suggests the erucial role of proper 3D representations and the demand for better situation under- standing in embodied scene understanding.
2 THE SQA3D DATASET
A problem instance in SQA3D can be formulated as a triplet (8, where denotes the scene context, e.g., 3D scan, egocentric video, bird-eye view (BEV) picture, etc.; (s s soot de- notes a situation, where the textual situation description SM (e.g., "Sitting at the edge of the bed and facing the couch" in Figure 1) depicts the position gPas and orientation sddl of an agent in the scene; denotes a question. The task to retrieve the correct answer from the answer set a = {ai...... )N). while optionally predicting the ground truth location (sPos, gooy from the text. The additional pre- diction of location could help alleviate the challenges brought by situation understanding. The following subsections will detail how to collect and curate the data and then build the benchmark.
2.1 DATA FORMATION
The 3D indoor scenes are selected from the ScanNet (Dai et al., 2017) dataset. We notice that some scenes could be too crowded/sparse, overall tiny, making situations and questions collection infea- sible. Therefore, we first manually categorize these scenes based on the richness of objects/layouts and the space volume. We end up retaining 650 scenes after dropping those that failed to meet the requirement. We then develop an interactive web-based user interface (UT) to collect the data. Details of UT design can be found in appendix. All the participants are recruited on AMT.
3

---

Published as a conference paper at ICLR 2023
Situation Edentification
Question Preguration Partiipaats asked write question the staation depicted both T and ,tM.
III. Answer Collection & Human Study
Participnts rolcol pickk pos and write description 1fxt
More participuuts asked 4 given the smtttio depicted only in glt
question
Figure 3: Data collection pipeline of SQA3D, Since our dataset comprises multiple types of annotations (situations and their descriptions, questions, answers, etc.), we found it more managcable to break down a single annotation task into three sub-tasks: i) Situation Identification; ii) Question Preparation; iii) Answer Collection & Human Study, where the participants recruited on AMT only need to focus on a relatively simple sub-task at time.
Compared to counterparts, the annotation load of single SQA3D problem instance could be sig- nificantly heavier as participants need to explore the scene, pick situation, make descriptions, and ask few questions. All these steps also require dense interaction with the 3D scene. To ensure good quality, we introduce a multi-stage collection pipeline, which breaks down the load into more manageable sub-tasks. Figure delineates this process:
I. Situation Identification. We ask the workers to pick 5 situations by changing the location (sPos ) of a virtual avatar in a ScanNet scene S. The workers are then instructed to write descriptions g that can uniquely depict these situations in the scene. We also use examples and bonuses to encourage more natural sentences and the use of human activities (e.g.. "I'm waiting for my lunch to be heated in front of the microwave"). All the collected situations are later manually curated to ensure diversity and the least ambiguity. If necessary, we would augment the data with more situations to cover different areas of the scene.
IL Question Preparation. We collect a set of questions w.r.t. each pair of the 3D scene S, and
the situation description gax (the virtual avatar is also rendered (s s²()).. To help prepare questions that require substantial situated reasoning, we tutor the workers before granting them access to our tasks. They are instructed follow the rules and learn from good examples. We also remove & penalize the responses that do not depend on the current situation, egg. "How many chairs are there in the room?".
III. Answer Collection & Human Study. In addition to the answers collected alongside the ques- tions, we send out the questions more workers and record their responses. These workers are provided with the same interface as in stage II except showing in the scene to ensure consistency between question and answer collection. There is also mandatory scene familiarization in all three steps before the main job starts and we find it extremely helpful especially for more crowded scenes. More details can be found in appendix.
2.2 CURATION, DATA STATISTICS, AND METRICS
Curation. Our multi-stage collection ends up with around k descriptions of 6.8k unique situations and 35k questions. Although the aforementioned prompt did yield many high-quality annotations, some of them are still subject to curation. We first apply a basic grammar check to clean up the lan- guage glitches. Then we follow the practices in VQAv2 (Goyal et al., 2017) and OK-VQA (Marino al., 2019) further eliminate low-effort descriptions and questions. Specifically, we eliminate & rewrite template-ale alike descriptions (e.g.. repeating the same sentence patterns) and questions that are too simple or do not require looking at the scene. We also notice the similar answer bias reported in Marino et al. (2019) where some types of questions might bias toward certain answers. There- fore, we remove questions to ensure a more uniform answer distribution. comparison of answer distribution before and after the balancing can be found in appendix. As a result, our final dataset comprises 20.4k descriptions and 33.4k diverse and challenging questions. Figure demonstrates some example questions in SQA3D.
Statistics. Compared to most counterparts with template-basse based text generation, SQA3D is crowd- sourced on AMT and therefore enjoys more naturalness and better diversity. To the best of our knowledge, SQA3D is the largest dataset of grounded 3D scene understanding with the human- annotated question-answerigg pairs (a comparison the counterparts can be found in Table 1).
4

---

Published as a conference paper at ICLR 2023
coffee table
S
withun taus
ng
Pafonn monnt
table
ee feftt
Halr
heeing
66
standing
clock Direction
a
ing
couch we
the
Senire
locking
desk
g ght
door
rook
chair
yall
Pncrused
What
color
behind
"window
Are
betica
which
Figure 4: Word cloud 8d in SQA3D.
Can
Statistic
Value
How
the
amount
Total 5tt (train/val/tsst)
16,229/ (,929//1,9972,149
tere
Total (train/val/test)
26,623/3.221/3.511
see
Unique g (train/val/test)
20,183/2.87233.066
Total scenes (train/val/test)
518/65/67
the
chairs are
Total objects (train/val/test)
11,723/1,550/1.622
Average g length
17.49
Average y length
10.49
Table 2: SQA3D dataset statistics.
Figure 5: Question distribution in SQA3D
Table 2, Figure 4, and Figure 5 illustrate the basic statistics of our dataset, including the word cloud of situation descriptions and question distribution based on their prefixes. It can be seen that descriptions overall meet our expectations human activities like "sitting" and "facing" are among the most common words. Our questions are also more diverse and balanced than our counterparts, where those starting with "What" make up more than half of the questions and result in biased questions (Azuma et al., 2022). More statistics like distributions over answers and length of the text can be found in appendix.
Dataset splits and evaluation metric. We follow the practice of ScanNet and split SQA3D into train, val, and test sets. Since we cannot access the semantic annotations in ScanNet test set, we instead divide the ScanNet validation scenes into two subsets and use them as our val and test sets, respectively. The statistics of these splits can be found in Table 2. Following the protocol in VQAv2 (Goyal et al., 2017), we provide set of 706 "top-K" answer candidates by excluding answers that only appear very few times. Subsequently, we adopt the "exact match" as our evaluation metric, i.e., the accuracy of answer classification in the test set. No further metric is included find it sufficient enough to measure the differences among baseline models with "exact match".
MODELS FOR SQA3D
Generally speaking, SQA3D can be characterized multi-modal reasoning problem. Inspired by the recent advances transformer-based (Vaswani et al., 2017) vision-language models (Lu et al., 2019; Li et al., 2020; Alayrac et al., 2022), we investigate how could these methods ap- proach our task. Specifically, we study a recent transformer- ScanQA (Azuma et al., 2022), which maps 3D scans and questions into answers. We make a few adaptations to ensure its compatibility with the protocol in SQA3D. To further improve this model, we consider including some auxiliary tasks during training (Ma al., 2022). For other types of 3D scene context, e.g. egocentric video clips and BEV pictures, we employ the corresponding state-of- the-art models. Finally, we explore the potential of recently-introduced LLMs like GPT-3 (Brown et al., 2020) and Unified QA (Khashabi et al., 2020) on solving SQA3D in zero-shot fashion. An overview of these models can be found in Figure 6.
question-answe answering system:
3D model. We use the term 3D model to refer a modified version of the ScanQA model (Azuma et al., 2022), depicted in the blue box of Figure 6. includes a VoteNet (Qi et al., bbsed 3D perception module that extracts object-centric features, LSTM-based language encoders for processing both questions q and situation description S²ا and some cross-attention transformer blocks (Vaswani et al., 2017). The object-centric feature tokens attend to the language tokens of T and q successively. Finally, these features will be fused and mapped to predict the answer. Op- tionally, we can add one head to predict the location (s r of the agent. Since the VoteNet module is trained from scratch, we also employ an object detection objective (not shown in the figure).
5

---

Published as a conference paper at ICLR 2023
3D Model
Video Image Model
Zero-shot Model
Location goo grott
MLP
Answer
Answer
Answer B
Fasion
Question
Language Encoder
Cross- attention
CliBBERT MCCAN
Unifled QA (zcro-shot)
Prompts
Description ylx
Language Encoder
Langunge Kncoder
Vision Encoder
atlention
Captions
[SEP] yat [SEP)
Object features
Egocentric video
Sean
VoteNet
FIEE [SEP]
BEY picture
Scan2Cap
Scan
Figure 6: Potential models SQA3D. split the considered models into three groups: model, video / image model, and zero-shot model. The 3D model is modified from the ScanQA model (Azuma et al., 2022) and maps 3D scan input to the answer. While the video / image models are effectively borrowed from canonical video QA and VQA tasks but we augment them with the additional situation input. The zero-shot model explores the potential of large pre-trained LLMs on our tasks. But they have to work with additional 3D caption model that converts the 3D scene into text.
Auxiliary task. As we mentioned before, situation understanding plays crucial role in accom- plishing SQA3D tasks. To encourage a better understanding of the specified situation, we introduce two auxiliary tasks: the model is required to make predictions about the sPos and 5et of the situation. We use wean--ruare--rror (MSE) loss for these tasks. The overall loss for our problem therefore becomes L Lans + aops + BC where Lanss Epos and Lio depicts the losses of the main and auxiliary tasks, a and B are balancing weights.
Video and Image- model. The orange box in the middle of Figure 6 demonstrates the models for video and image- based input. SQA3D largely resembles a video question answering or visual question answering problem when choosing to represent the 3D scene context S as egocentric video clips or BEV pictures. However, SQA3D also requires the model to take both question q and the newly added situation description gxxt input. We, therefore, follow the practice in the task of context- based QA (Rajpurkar et al., 2018) and prepend s to the question as a context. For the model, we use the state-of-the-art video QA system ClipBERT (Lei al., 2021) and VQA system MCAN (Yu et al., 2019). We adopt most of their default hyper-parameters and the details can be found in appendix.
Zero-shot model. We explore to which extent the powerful LLMs like GPT-3 (Brown et al., 2020) and Unified QA (Khashabi et al., 2020) could tackle our tasks. Following prior practices that apply GPT-3 to VQA (Changpinyo et al., 2022; Gao et al., 2022), we propose to convert the 3D scene into text using an emerging technique called 3D captioning (Chen al., 2021). We provide the caption, giu,, and q as part of the prompt and ask these models to complete the answer. For GPT-3, we further found providing few-shot examples the prompt helpful with much better results. Minor post- processing is also needed to ensure answer quality. We provide more details prompt engineering in the appendix.
EXPERIMENTS
4.1 SETUP
We benchmark the models introduced in Section 3 to evaluate their performances on SQA3D. As mentioned before, we examine three types of scene context S: 3D scan (point cloud), egocentric video, and BEV picture. Both the 3D sean and egocentric video for each scene are provided by ScanNet (Dai et al., 2017). However, we down-sample the video to allow more efficient computa- tion per the requirement of the ClipBERT model (Lei et al., 2021). The BEV pictures are rendered by placing a top-down camera on top of the scan of each 3D scene. We also conduct additional experiments that investigate factors that could contribute to the results, e.g., situation and auxil- iary tasks. In our early experiments, we found that the 3D model overall performs better than the video or image-based models. Therefore we only conduct these additional experiments with the variants of our 3D model due to the limit of computational resources. We use the official imple- mentation of ScanQA, ClipBERT, and MCAN and include modifications for SQA3D. For the zero-shot models, we extract 3D scene captions from two sources: ScanRefer (Chen et al., 2020) and ReferIt3D (Achlioptas al., 2020). Considering the limit on the length of the input prompt,

---

Published as a conference paper at ICLR 2023
test set
S
Format
Avg.
What
Is
How
Can
Which
Others
Blind test
SQ A
26.75
63.34
43.44
69.53
37.89
43.41
43.65
ScanQA (w/o s
3D scan
VQ-M +A
28.58
65.03
47.31
66.27
43.87
42.88
45.27
ScanQA
3D scan
VSQ AA
31.64
63.80
46.02
69.53
43.87
45,34
46.58
ScanQA aux. task
3D scan
33.48
66.10
42.37
69.53
43.02
46.40
47.20
MCAN ClipBERT
BEV
VSQ +A
28.86
59.66
44.09
68.34
40.74
40.46
43.42
Ego. vidco
VSQ-AA
30.24
60.12
38.71
63.31
42.45
42.71
43.31
Unified QA
ScanRefer
VSQ +A
33.01
50.43
31.91
56.51
45.17
41.11
41.00
Unified QALage
ReferIt3D
VSQ-*A
27.58
47.99
34.05
59.47
40.91
39.77
38.71
GPT-3
ScanRefer
VSQ->A
39.67
45.99
40.47
45.56
36.08
38.42
41.00
GPT-3
ReferIt3D
VSQ-*A
28.90
46.42
28.05
40.24
30.11
36.07
34.57
Human (amatcur)
3D scan
VSQ->
88.53
93.84
88.44
95.27
87.22
88.57
90.06
Table 3: Quantitative results on the SQA3D benchmark. Results are presented in accuracy (%) on different
types of questions. In the "Format" column: V = 3D visual input S; S = situation description sh Q = question q: A answer a; L location (s " In ScanQA, aux. task indicates the use of both Eyes and Lxe as additional losses. We use the Large variant as Unified QA (Khashabi et al., 2020) as works better.
these 3D captions are also down-sampled. The Unified QA model weights are obtained from its Huggingface official repo All the models are tuned using the validation set and we only report results on the test set. More details model implementation can be found in appendix.
4.2 QUANTITATIVE RESULTS
We provide the quantitative results of the considered models (detailed in Section 3) on our SQA3D benchmark in Table 3. The findings are summarized below:
Question types. In Table 3, we demonstrate accuracy on six types of questions based on their prefixes. Most models tend to perform better on the "Is" and "Can" questions while delivering worse results on "What" questions, likely due to a smaller number of answer candidates most questions with binary answers start with "Is" and "Can", offering better chance for the random guess. Moreover, we observe the hugest gap between the blind test (model w/o 3D scene context input) and our best model the "What" and "Which" categories, suggesting the need for more visual information for these two types of questions. This also partially echoes the finding reported in Lei et al. (2018).
Situation understanding and reasoning. At the heart of SQA3D benchmark is the requirement of situation understanding and reasoning. As we mentioned in Section 2.1, the model will be more vulnerable to wrong answer predictions if ignoring the situation that the question depends on (e.g- "What in front of me" could have completely different answers under different situations). In Ta- ble 3, removing situation description gaMt from the input leads to worse results, while adding the auxiliary situation prediction tasks boosts the overall performance, especially on the challenging "What" questions. The only exception is "How" questions, where majority of them are about counting. We hypothesize that most objects in each ScanNet scene only have a relatively small number of instances, and the number could also correlate to the object category. Therefore, guess- ing/memorization based on the question only could offer better results than models with the situation as input if the situation understanding reasoning are still not perfect yet. Additionally, we also provide an inspection of the relation between situation understanding and QA using attention visu- alization in Section 4.3.
Representations of 3D scenes. Indeed, SQA3D does not limit the input to be 3D scan only, as we also offer options of egocentric videos and BEV pictures. Compared models with the 3D scan as input, the tested models with other 3D representations (i.e., MCAN and ClipBERT) deliver much worse results, implying that the 3D scan so far could still be better representation for the 3D scene when the reasoning models are probed with questions that require holistic understanding of the scene. On the other hand, MCAN and ClipBERT are general-purpose QA systems, while trade-off could also
ScanQA is designed for 3D-language reasoning tasks. The generalist-speciall partially account for the gap. Finally, the poor results of BEV and egocentric videos based models compared to the blind test could also be due to the additional "vision-bias" when the visual input is provided (Antol et al., 2015). Note that the vision-bias can be mitigated with better visual represen- tations (Wen et al., 2021), implying that ScanQA, which seems suffer less from the vision-bias
7

---

Published as a conference paper at ICLR 2023
fli picking backpask left within chairs sehind
goat ode there
CRC, ennening What coler the bockreck clokk
, I working the uuatie
with cabinsts over leaa the
wall within reach
gi culfiz table fanihce! surounned
If walkd bockwards.
would hit behind
Scargq
a Unkounes X
Black
Irom X
SaarQA
Ose X
Couch
or Bicck
a: Wall
Tao
Cuuch
Wall
Figure 7: Qualitative results. We show the predicted answer and bbox with highest attention for the variants of ScanQA (Azuma et al., 2022) models. We anticipate the bbox to indicate the object that situation description g question q refers We observe that better situation understanding (via comprchension sl auxiliary tasks) could result more reasonable attention over objects, which positively correlates to more robust answer prediction.
than the counterparts using BEV and egocentric videos, is fueled by better visual representations in terms of combating the dataset bias.
Zero-shot training from scratch. The success of pre-trained LLMs like GPT-3 myriads of challenging reasoning tasks (Wei et al., 2022b;a) suggests that these models could possibly also understand embodied 3D scenes with language-only input (Landau & Jackendoff, 1993). However, SQA3D imposes a grand challenge to these models. The powerful Unified QA (Large variant) and GPT-3 both fail to deliver reasonable results on our tasks. Further, we hypothesize the bottleneck could also be on the 3D captions, as the results verify the consistent impact on model performances However, we still believe these
brought by different source of captions (ScanRefer--Refert5l) models have great potential. For example, one zero-shot model (GPT-3 ScanRefer) do pretty well on the challenging "What" questions (39.67%), even better than the best ScanQA variant.
Human vs. machine. Finally, all the machine learning models largely fall behind amateur human participants (47.2% of ScanQA + aux. task vs. 90.06%) Notably, we only offer a limited number of examples for the testers before sending them the SQA3D problems. Our participants promptly master how to interact with the 3D scene, understand the situation from the textual description, and answer the challenging questions. The human performance also shows significant bias for different question types.
4.3 QUALITATIVE RESULTS
Finally, we offer some qualitative results of the variants of our 3D model Figure 7. We primarily focus on visualizing both the answer predictions and the transformer attention over the object-centric feature tokens (bounding boxes) generated by the VoteNet (Qi al., 2019) backbone. We highlight model, in the
the most-attended bounding box among all the predictions by the transformer- hope of a better understanding of how these models perceive the 3D scene to comprehend the sit- uations and answer the questions. In Figure 7, the correct predictions attention over relevant objects in the situation description S and questions. Moreover, in case there multiple instances of the same object category, it is also crucial identify the correct instance. For example, only ScanQA + aux. task makes the correct prediction for the first question and also attends the right chair behind while ScanQA focuses wrong instance. These results
always associated with
8

---

Published as a conference paper at ICLR 2023
firm our findings in Section 4.2 about the critical role of situation understanding. We also provide some failure modes in appendix.
5 RELATED WORK
Embodied AL. The study of embodied AI (Brooks, 1990) emerges from the hypothesis of "ongoing physical interaction with the environment as the primary source of constraint on the design of intel- ligent systems". To this end, researchers have proposed a myriad of Al tasks to investigate whether intelligence will emerge by acting in virtual photo-realistic environments. Notable tasks includ- ing robotic navigation (Das et al., 2018; Anderson et al., 2018; Savva et al., 2019; Chen et al., 2019; Wijmans et al., 2019b; Qi et al., 2020; Deitke et al., 2022) and rision-based manipulation (Kolve et al., 2017; Puig et al., 2018; Xie et al., 2019; Shridhar et al., 2020a;b; 2022). These tasks are made more challenging as instructions or natural-dialogues are further employed conditions. So- phisticated models have also been developed to tackle these challenges. Earlier endeavors usually comprise multi-modal fusion (Tenenbaum & Freeman, 1996; Perez et al., 2018) and are trained from scratch (Wang et al., 2016 Fried et al., 2018; Wang et al., 2019), while recent efforts would employ pre-trained models (Pashevich et al., 2021; Hong et al., 2021a; Suglia et al., 2021). However, the agents still suffer from poor generalization to novel and more complex testing tasks (Shridhar et al., 2020a) compared to results on training tasks. More detailed inspection has still yet to be conducted and it also motivates our SQA3D dataset, which investigates one crucial capability that the current embodied agents might need to improve: emhodied scene understanding.
Grounded 3D understanding. Visual grounding has been viewed key to connecting human knowledge, which is presumably encoded in our language, to the visual world, so as enable the in- telligent agent to better understand and act in the real environment. Is natural extend this ability to 3D data as it offers more immersive representations of the world. Earlier work has examined word-level grounding with detection and segmentation tasks on 3D data (Gupta al., 2013; Song & Xiao, 2014; Dai et al., 2017; Chang et al., 2017). Recent research starts to cover sentence-level grounding with complex semantics (Chen et al., 2020; Achlioptas al., 2020; Chen et al., 2021). More recently, new benchmarks introduce complex visual reasoning to 3D data (Azuma et al., 2022; Ye et al., 2021; Yan et al., 2021). However, these tasks mostly assume a passive, hhird-person's per- spective, while our SQA3D requires problem-solving with egocentric viewpoint. This introduces both challenges and chances for tasks that need a first-person's view, egg. embodied AI.
Multi-modal question answering. Building generalist question answering (QA) systems has long been a goal for AI. Along with the progress in multi-modal machine learning, VQA (Antol et al., 2015; Zhu al., 2016) pioneers the efforts of facilitating the development of more human-like, multi-modal QA systems. It has been extended with more types of knowledge, e.g. common sense (Zellers et al., 2019) and factual knowledge (Marino al., 2019). Recent research has also introduced QA tasks on video (Lei et al., 2018; Jia et al., 2020; 2022; Grunde-McLLughhee et al., 2021; Wu al., 2021: Datta al., 2022), and 3D data (Ye et al., 2021; Azuma et al., 2022; Yan et al., 2021). ). We propose the SQA3D benchmark also in hope of facilitating multi-modal QA sys- tems with the ability of embodied scene understanding. Notably, models for SQA3D could choose their input from 3D scan, egocentric video, or BEV picture, which makes our dataset compatible with a wide spectrum of existing QA systems.
CONCLUSION
We've introduced SQA3D, a benchmark that investigates the capability of embodied scene under- standing by combining the best of situation understanding and situated reasoning. We carefully curate our dataset to include diverse situations and interesting questions while preserving the rela- tively large scale (20.4k situation descriptions and 33.4k questions). Our questions probe wide spectrum of knowledge and reasoning abilities of embodied agents, notably navigation, common sense, and multi-hop reasoning. We examine many state-of-the-art multi-modal reasoning systems but the gap between the best ML model and human performances so far is still significant. Our find- ings suggest the crucial role of proper 3D representations and better situation understanding. With SQA3D, we hope of fostering research efforts in developing better embodied scene understanding methods and ultimately facilitate the emergence of more intelligent embodied agents.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
The authors would like to thank Dave Zhenyu Chen for his insightful ScanRefer project and help on data collection, Wenjuan Han for discussions on data collection and model design. This project is supported by National Key R&D Program of China (2021ZD0150200)
REFERENCES
Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3D: Neural listeners for fine-grained 3D object identification in real-world scenes. Eu- ropean Conference Computer Vision (ECCV), pp. 422-440, 2020. 2, 3, 6,9
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0166
2022.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. arXiv preprint arXiv:2204 14119 2022. 5
John R Anderson, James G Greeno, Lynne M Reder, and Herbert A Simon. Perspectives on learning, thinking, and activity. Educational Researcher, 29(4):11 1-13, 2000.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-languae language navigation: Interpreting navigation instructions real environments. In Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp.
2018. 3,
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In International Conference on Computer Vision (ICCV), pp. 2425-2433, 2015. 7, 15
Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. ScanQA: 3D Question Answering for Spatial Scene Understanding. In Conference Computer Vision and Pattern Recognition (CVPR), pp. 19129-19139, 2022. 2, 3, 5, 6, 8, 9, 16, 20
Rodney A Brooks. Elephants don't play chess. Robotics and autonomous systems, 6(1-2):3-15, 1990. 9
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 33:17771101, 2020. 3, 6
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Nicssner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3D: Learning from rgb-d data in indoor environments. arXiv preprint 1709v:17090666 2017.
Soravit Changpinyo, Doron Kukliansky, Idan Szpektor, Xi Chen, Nan Ding, and Radu Soricut. All You May Need for VQA are Image Captions. arXiv preprint 2xXi:22055001833 2022. 6
Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3D object localization in rgb-d scans using natural language. In European Conference Computer Vision (ECCV), pp- 202-221, 2020. 2, 3, 6, 9,
Howard Chen, Alane Suhr. Dipendra Misra, Noah Snavely, and Yoav Artzi Touchdown: Natu- ral language navigation and spatial reasoning in visual street environments. In Conference on
Computer Vision and Pattern Recognition (CVPR), pp. 12538-12547, 2019. 9
Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X Chang, Scan2cap: Context-aware dense captioning in rgb-d scans. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3193-3203,, 2021. 2, 6,
10

---

Published as a conference paper at ICLR 2023
Angela Dai, Angel X Chang. Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotatee 3D reconstructions of indoor scenes. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3828-5839, 2017. 3, 6. 16
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Em- bodied question answering. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-10, 2018. 2.9
Samyak Datta, Sameer Dharur, Vincent Cartillier, Ruta Desai, Mukul Khanna, Dhruv Batra, and Devi Parikh. Episodic Memory Question Answering. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 19119-19128, 2022. 9
Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Jordi Salvador, Kiana Ehsani, Winson Han, Erie Kolve, Ali Farhadi, Aniruddha Kembhavi, et al. ProcTHOR: Large-Scale Embodied AI Using Procedural Generation. arXiv preprint wxXi:2226.06994 2022.
Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. Speaker-followe models for vision-and- and-language navigation. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018.
Feng Gao, Qing Ping, Govind Thattai, Aishwarya Reganti, Ying Nian Wu, and Prem Natarajan. Natural Language-Centn
Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5067-5077,
2022. 6
Yash Goyal, Tejas Khot, Douglas Summers- Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Conference on Computer Vision and Puttern Recognition (CVPR), pp. 9904-6913, 2017. 4,5
James Greeno. The situativity knowing, learning, and research. American psychologist, 53(1):
5, 1998. 2
Madeleine Grunde-Mel.nnd MeLaaghinn,,,, Ranjay Krishna, and Maneesh Agrawala. AGQA: A benchmark for compositional spatio-temporal reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11287-11297, 2021.
Saurabh Gupta, Pablo Arbelaez, and Jitendra Malik. Perceptual organization and recognition of indoor scenes from RGB-D images. In Proceedings of the IEEE conference on computer vision and pattemn recognition, pp. 564-571, 2013. 9
Yicong Hong, Qi Wu, Yuankai Qi Cristian Rodriguez-Opazo, and Stephen Gould. Vin bert: A recurrent lan-laaugeee bert for navigation. In Conference on Computer Vision and Pattern
Recognition (CVPR), pp. 1643-1653, 2021a. 9
Yining Hong. Qing Li, Song-Chun Zhu, and Siyuan Huang. Vigrammar: Grounded grammar in- duction vision and language. International Conference on Computer Vision (ICCV), 2021b. 2
Baoxiong Jia, Yixin Chen, Siyuan Huang, Yixin Zhu, and Song-chun Zhu. Lemma: A multi-view dataset for learning multi-agent multi-task activities. European Conference on Computer Vision (ECCV), 2020. 9
Baoxiong Jia, Ting Lei. Song- Chun- Zhu, and Siyuan Huang. EgoTaskQA: Understanding Human Tasks in Egocentric Videos. The 36th Conference on Neural Information Processing Systems (NeurIPS 2022) Track Datasets and Benchmarks, 2022.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannanch Hajishirzi. Unifiedqa: Crossing format boundaries with a single qa system. arXiv preprint ww.xiv:2005000000 2020. 3, 5,5,6,7 6,
Eric Kolve, Roozbeh Mottaghi. Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. Ai2-thor: An interactive 3D environment for visual ai. arXiv preprint 1rXiv:1712.05474, 2017.

---

Published as a conference paper at ICLR 2023
Barbara Landau and Ray Jackendoff. "What" and "where" in spatial language and spatial cognition. Behavioral and brain sciences, 16(2):217-23,, 1993.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. Tvqa: Localized, compositional video question answering. arXiv preprint arXiv: 00960199,,,, 2018. 7.9
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and-language learning via sparse sampling. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7331-7341, 2021. 3, 6, 16, 20
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semanties aligned pre-training for vision-language tasks. In European Conference on Computer Vision (ECCV), pp. 121-137. Springer, 2020. 5
Hongzhuo Liang, Xiaojian Ma, Shuang Li, Michael Görner, Song Tang, Bin Fang, Fuchun Sun, and Jianwei Zhang. Pointnetgpd: Detecting grasp configurations from point sets. In International Conference on Robotics and Automation (ICRA), pp. 6629-3635. IEEE, 2019.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- tasks. Advances in Neural Information Processing
guistic representations for
Systems (NeurIPS), 32, 2019.
Xiaojian Ma, Weili Nie, Zhiding Yu, Huaizu Jiang, Chaowei Xiao, Yuke Zhu, Song-Chun Zhu, and Anima Anandkumar. Relvit: Gundedt-guided vision transformer for visual relational reasoning. arXiv preprint arXiv:2204. /667, 2022. 5, 20
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge, In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3195-3300003 -204,, 2019. 4, 9, 15
Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. GECTOR - grammatical error correction: Tag, not rewrite. In Proceedings of the Fifteenth Work- shop Innovative Use of NLP for Building Educational Applications, pp. 163-170, Seattle, WA, USA at' Online, July 2020. Association for Computational Linguistics. URL https:
/ //www. , ac wwwwwb......... org/ antooooyy//22222 bea- 1.16. 15
Alexander Pashevich, Cordelia Schmid, and Chen Sun. Episodic transformer for vision-and- language navigation. In Conference Computer Vision and Pattern Recognition (CVPR), pp.
15942-15952, 2021. 9
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In AAAI Conference on Artificial Intelligence (AAAI), volume 32, 2018. 9
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1494-8502, 2018. 9
Charles Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3D object detection in point clouds. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9277-9286, 2019. 5,8
Yuankai Qi. Qi Wu, Peter Anderson, Xin Wang. William Yang Wang, Chunhua Shen, and Anton van den Hengel. Reverie: Remote embodied visual referring expression in real indoor environ- ments. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9982-9991, 2020. 9
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't 't know: Unanswerable questions for SQuAD. arXiv preprint arXiv: 180688000.30 03822, 2018. 6
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun. Jitendra Malik, et al. Habitat: A platform for embodied ai research. In International Conference on Computer Vision (ICCV), pp. 1339-9347,44 2019. 1,
12

---

Published as a conference paper at ICLR 2023
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Conference Computer Vision and Pattern Recognition (CVPR), pp- 10740-10749, 2020a. 9
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.037666 2020b. 9
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning (CoRI.), pp. 894- 946906..7 PMLR, 2022. 1,9
Shuran Song and Jianxiong Xiao. Sliding shapes for 3D object detection in depth images. In European conference computer vision, pp. 634-651 Springer, 2014.
Alessandro Suglia, Qiaozi Gao, Jesse Thomason, Govind Thattai, and Gaurav Sukhatme. Embodied bert: A transformer model for embodied, seguuge-ggiides visual task completion. arXiv preprint OrXiv:2108.04927 2021. 9
Joshua Tenenbaum and William Freeman. Separating style and content. Advances in Neural Infor- mation Processing Systems (NeurIPS), 9, 1996.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- fion Processing Systems (NeurIPS), 30, 2017. 5
Xin Wang, Wenhan Xiong, Hongmin Wang, and William Yang Wang. Look before you Ieap: Bridg- ing model-free and model-based reinforcement learning for planned-ahead vision-and -ad--aanddddggge navigation. In European Conference on Computer Vision (ECCV), pp. 37-53, 2018.
Xin Wang, Qiuyuan Huang. Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fan Wang, William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and self-supervised im- itation learning for vision-language navigation. In Conference Computer Vision and Pattern
Recognition (CVPR), pp. 6629-6636 6638, 2019. 9
Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang, HUMANISE: conduage-conditiooed Human Motion Generation in 3D Scenes. In Advances in Neural Informa- fion Processing Systems (NeurIPS), 2022.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. Xrxiv preprint arXiv:2206.07682 2022a. 8
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022b. 8
Zhiquan Wen, Guanghui Xu, Mingkui Tan, Qingyao Wu, and Qi Wu. Debiased Visual Question Answering from Feature and Sample Perspectives. 34:3784 3796, 2021. 7
Erik Wijmans, Samyak Datta, Oleksandr Maksymets, Abhishek Das, Georgia Gkioxari, Stefan Lee, Irfan Essa, Devi Parikh, and Dhruv Batra. Embodied question answering in photorealistic envi- ronments with point cloud perception. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6659-6668, 201 9a. 2,
Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames. arXiv preprint arXiv: /9/ 100357, 2019h. 9
Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. STAR: A benchmark for situated reasoning in real-world videos. In Thirty-fitth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. 9
13

---

Published as a conference paper at ICLR 2023
Xu Xie, Hangxin Liu, Zhenliang Zhang, Yuxing Qiu, Feng Gao, Siyuan Qi, Yixin Zhu, and Song- Chun Zhu. Vrgym: A virtual testbed for physical and interactive ai. Proceedings of the ACM Turing Celebration Conference- China, pp. 1-6, 2019. 9
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2288-5296, 2016. 20
Xu Yan, Zhihao Yuan, Yuhao Du, Yinghong Liao, Yao Guo, Zhen Li, and Shuguang Cui. CLEVR3D: Compositional Language and Elementary Visual Reasoning for Question Answer- ing in 3D Real-World Scenes. arXiv preprint arXiv:21122 11691, 1691, 2021. 3,9 9
Shuquan Ye, Dongdong Chen, Songfang Han, and Jing Liao. 3D Question Answering. arXiv preprint arXiv:2112.083550 2021. 3,9
Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 0281-6290, 2019. 3, 6, 16, 20
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition cognition: Visual commonsense reasoning. In Conference on Computer Vision and Pattern Recognition (CVPR),
pp. 6720-6731, 2019. 9
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question an- swering in images. In Conference Computer Vision and Pattern Recognition (CVPR), pp- 4995-5004, 2016. 9
14

---

Published as a conference paper at ICLR 2023
A DATA COLLECTION
A.1 DATA COLLECTION WEB UI
We present the Web UI of our data collection in Figure 8 (Stage 1), Figure 9 (Stage II) and Figure 1 (Stage III) respectively. We developed our UI based on Chen et al. (2020). These UIs share some common components: 3D scene viewer, where the user can drag, rotate, and zoom in/out the scene; clickable objects/tags, where users might click on either the object mesh directly or the tag on the sidebar to highlight it in the scene; and an instruction set that guide the user through the task. Users may also switch between full scene or object mesh only to focus on the tasks. The users are also required to submit multiple responses with the same scene.
Notably, we create detailed tutorials for each stage (not shown in the UT) with examples and animated demonstrations. We found tutorials and instruction sets with clear criteria on rejection and bonus(e.g. Figure 10) helpful with high-quality data. Finally, all the testers need to pass test before the qualification for our task is granted.
A.2 DATA POST-PROCESSING
There two major data post-processing steps in SQA3D: cleaning and balancing. For cleaning, we primarily focus on grammatical correction. We adopt both rule-based cleaning and an ML-based tool called GECToR (Omelianchuk al., 2020) in our grammatical correction pipeline. We adjust the correction threshold based on human judgment over the corrected data samples.
In the balancing step, our goal is to reduce the question-answer bias in the dataset. Therefore follow the practice in Antol et al. (2015); Marino et al. (2019) and re-sample the questions based on their prefixes and answer type, in hope of more balanced answer distribution. We provide answer distribution before and after balancing in Section B.1.
A.3 MORE MTURK DETAILS
We provide the detailed MTuurk job settings below:
Region. We enable access to our tasks in the following countries/regions:
US, DE, GB, AU, CA, SG. NZ, NO, SE, FL. DK, IE
Approval rate & Number of approved jobs. The testers are required to have at least 95% approval rate and have completed more than 1000 tasks. However, we relax this requirement to 90% approval rate for Stage III it is simpler than the other annotation tasks.
Reward. The participants will rewarded $0.5 for each task in Stage I and II, and $0.2 for QA tasks in Stage III, with possibility of a bonus depending the overall quality. We actively monitor the response quality and send bonuses/rejections daily. Note that we collected 5 responses for each task in all three stages.
Task lifetime. We set the lifetime as 10 days for tasks in Stage and 20 days for those in Stage and III. However, found most of the tasks can be completed in less than days.
B DATASET DETAILS
B. MORE STATISTICS
We provide the histogram of the answer distribution before & after balancing in Figure 12 and Fig- ure 13, respectively. It can be seen that we manage to ensure there is no single answer that dominates any type of question (categorized by their prefixes). However, we do acknowledge that prefix- based balancing might still not be sufficient since models could also learn to the n-grams pattern. A more effective avenue is collecting more questions with less- fresuentt answers, which we leave as future work.
15

---

Published as a conference paper at ICLR 2023
S0ADO
List objects
mutrel Annoue
0667) Amant-1
ausal (onwir
Goar Amoue
cn Amoont
Ineikcce (Arwee
Wiliias Aotau
Step
how
Step hok certan location on onteraatcs twe tnam "conteaz") Navigate lndetionan wienauis v. Click and drng the arrowe to move, Corl Shifts rotato..
the picked
Step Desoribe the location and fentation "contentt y picked Step 2 with a sentence, then Submiit desoriction locato unique posit
Antbiguous description
nory vontee
Please repwar Stea 283 for times
Ve subminted resposses. Tonal ingoonees reeded:
Figure 8: Dataset collection Web UI for Stage
In Figure 14a and Figure 14b, show the histogram of the length of situation description sa and question q. Overall most the descriptions and questions are middle-length sentences (10-20 words).
B.2 DETAILS ON EGOCENTRIC VIDEO AND BEV IMAGE
For egocentric videos, uniformly downsample the frames of the original ScanNet (Dai et al., 2017) video by using the first frame of every 20 frames. Afterward, we resize all the frames to 224 224 to create the video used for training ClipBERT(Lei rendering all BEV images. We compute the radius of the bounding sphere of the scene and put the camera at the top of the scene with distance of times the radius the center of the bounding sphere. Images of size 1920 1080 are rendered for clarity while the input to the MCAN(Yu et al., 2019) model the resized version of the images to 224 224.
al., 2021). Blender is used for
C MODEL DETAILS
C.I INPUT PIPELINE
We follow the input pipeline in ScanQA(Azuma et al., 2022) without further modification. As for MCAN, we only transform the images to the ImageNet-pretraine randomly sample 8 clips with each clip consisting of frames of the video to feed into the model as the scene representation. Note that each frame is resized to 1000 x 1000 following the practice of original ClipBERT(Lei et al., 2021).
encoder. In ClipBERT, we
16

---

Published as a conference paper at ICLR 2023
List of objects
bag Amoane
Amont:
battroo Aesur
obe Annour
Hiro Araan:
Anoutt11
lowrd) Arouut
Annwent
DUT the Towel on the counser's to and are lo bestee on the floo ltf.
Question
Answer
dimppla worrd
tiee
You ve subminted responses. Tinal responees needed
Figure 9: Dataset collection Web UI for Stage Il.
Assignment with questions that can be answered without considering the context will be rejected.
How many thair are thore the rooe?
the amount moniors desk the loing ond or event
Assignment with questions that can be answered by merely reading the description (no need to look at the scene) will be rejected.
You may ask at most "simple" questions ((you need to ask 5 questions in total) encourage creative questions; otherwise assignment will be rejected, Questions below are viewed as "simple":
Q usetions about cimple siject coegeryprpeety rting as there chair my 6 criook disctoon What the color table my rights How many chairs are there behine mot?
Gue aestions res the same parten
questione "yee" ansigmment
Figure 10: Additional instruction set to the AMT participants in Stage II.
C.2 HYPER-PARRAMTTER
We provide the hyper-parameters the considered models in Table
C.3 ADDITIONAL DETAILS ON ZERO-SHOT MODELS
We uniformly sample sentences from our 3D caption sources for both models. When testing with the Unified QA model, employ simple greedy sampling method and the following prompt:
17

---

Published as a conference paper at ICLR 2023
Ailss
objects
Amoust
couch Insoort
Jopp call Ursuant
Aemount
Step
Step The lolwiing ceeoiibes your curne statue
Iv uxting on the office chair focing the deaks with the vir dowin my o'dlosk direction and in my oldookk dioctton.
thome Jow naww thene "corieee" tt abons, arawer owing quantoon "Sccent.
dead
Answer www.oooooo simpa
Figure 1I: Dataset collection Web UI for Stage III.
front
Inft
forwa
edd
forward
zero
fiw
brwward
right
backward
green
four
redtangula
right
rght
forward
open1
right
black
closed
WINGOR TV
ractangle
fonward
two
reclangula
bohind
backpa 5c
badkwardd
white
no
round
yes
Window
left
yes
backpapk
couch
backward
door
bed
left
yes
right
add
square
right
brown
dow
even
table
table
which vide
What direction
What object
How should
Which one
Which way
Dows
What appe
Where
there
Which direction
What calor
Can
How many
the
What ie
Figure 12: Answer distribution (organized by question prefixes) before balancing.
Q: (a} A:
18

---

Published as a conference paper at ICLR 2023
back
behind
!width
backward
rightwards
yellow grey
Tive
hi South
worward
retaaguu
Istaped
even
odd
Forward
green
zer0
back right
forward
blue
four
Pigh
red
open
front right
right
oval
closed
front
left
Window
bathnub
rectangle
behinn
fowardd
white
threa threa Pertane gu
backpatk trash can
backoaard
right
right
yes
left
round
left
two
backpark
couch
Goor
edd
6oor
right
bed
backward
yes
door
right
backward
chair
couch
right
black
win dow
oven
chair
Lable
table
Which side
What directioon
What objert
How should
Which ene
Which way
Does
What shane
Where
there
Which direction
What calor
How many
Is the What
Figure Answer distribution (organized by question prefixes) balancing.
3.2-
2.30
3.05
Histogram of situation description gx length.
(b) Histogram question g length.
19

---

Published as a conference paper at ICLR 2023
where { and {q) replaced by the situation description and question. For GPT-3, the text-davinci-0Oe variant and the following prompt:
Context: There is a book on the desk. laptop with green cover is to the left of the book.
Q: I'm working by the desk. What is on the desk beside the book?
A: laptop
Context:
Q: (1)
A:
where 1-shot example to demonstrate the format of our task. Interestingly, found only GPT-3 would benefit from few-shot examples.
C.4 ADDITIONAL DETAILS ON SCANQAMCANICLIPBETT
ScanQA (Azuma et al., 2022). We slightly modify the original ScanQA code base (from http:://ithub..omm//ii wwt.////--///////aa.... DBIIScanQAA)) to make it fit our task better. The original reference branch discarded and the supervision signal for the language classification branch is changed make use of it as regression branch. More specific details can be found below.
The original data loader only outputs the question as whole (meaning that the situation con- catenated before the question), while our version split the two sentences.
The original model takes language as input, while we feed situation and question separately into the model.
The original model uses self-attention block and 1 cross-attention block for the fusion of lan- guage and visual features, while version uses 2 self-attention blocks and cross-attention blocks to treat situations and questions separately.
The original model uses additive operation to fuse language & visual features, while our version uses concatenation for fusion.
To conduct the ablation experiment of blind test, we simply discard the output feature of VoteNet and only feed the situation feature and question feature into the QA head.
To conduct the ablation experiment of w/o S! we replace situation with several (unk) tokens to make fair comparison.
To add auxiliary task into training, we change the supervision of the language classification head from Cross Entropy to MSE Loss to make it regression head.
MCAN (Yu et al., 2019). We use the code base from ReIVIT (Ma et al., 2022) (htts://githu..com//V /oabaab/.o.oo//aa//a/l/i.Ti since its implementation of MCAN could take raw images as input while the original one cannot. The default training setting is kept except for learning rate de- cay. We cancel it to make a fair comparison with the other baselines. We concatenate the situation before the question make them whole and this sentence the question that MCAN requires.
ClipBERT (Lei al., 2021). We use the official repository of ClipBERT (https://github.com httpa//...ooooaaeee///iiieee the format ClipBERT takes. The configuration file for MSR-VTT QA (Xu al., 2016) is used for generality as we find all the configuration files to be almost identical. The evaluated question types are changed since our focus is different from MSR- VTT. We turn off mixed precision training as we observe instability when using it. We concatenate the situation before the question to make them as a whole and use this new sentence as the question that ClipBERT requires.
and follow the instruction to transform our data into
D ADDITIONAL EMPIRICAL RESULTS
We provide additional qualitative results and failure modes in Figure 15 and Figure 16.
20

---

Published as a conference paper at ICLR 2023
(as,; am dreowing iooh into the preen maah zin with the office chair
am sitinng on oce side the
Jamn standing in frunt of
stit; am standing facing the stak
lail facing the arinkos with lhe ciznk
dtor and the hooksself on nggtt side. Whet left agoinst the wall
wall chais to my lull will walhin neach. q: What is the fire obyect in right?
immeddate right.
my 5 clock divection.
Which will lowen the roum?
Witt above the shelf that is Ta'chack??
X
a: Right
a: Window
a: Dresser
Red
Scarg
Right
Lamp
Dresser
Figure 15: Additional qualitative results.
LIMITATION AND POTENTIAL IMPACT
Limitations. One major limitation of SQA3D the selection of 3D scenes. Since our dataset is collected on mostly indoor ScanNet scenes about household environments, it cannot cover outdoor scenes and other types scenes, ex. warehouse. This could limit the application to autonomous driving and warehouse robots, which are likely deployed to the scene types that do not present in SQA3D. Moreover, all the scenes ScanNet are static, i.e. the agent cannot interact with the object, making the exploration in SQA3D limited to hovering over the 3D scenes. However, many embodied tasks also require non-trivial interaction with articulated objects, e.g. drawers. Therefore, the em- bodied scene understanding capability examined by SQA3D also be limited to interactteee scenarios, iee. situation understanding and situated reasoning.
Societal impact. SQA3D offers two sets of annotations: situations (s sPos gooss and QA (q, a)- The situation annotations themselves could enable many exciting applications including building a real-world household assistant robot
of its core capabilities is connecting natural language in- cruuttions/dsccritioons to the situations in the scene, e.g. locations. Moreover, non-trivial common- sense reasoning also required this process. Our annotations with accurate duscription-locatioo pairs and the requirement of commonsense knowledge in text understanding could support these needs. The QA tasks also examine wide spectrum capabilities of embodied agents house- hold domains, making it great benchmark for testing these household assistant robots. Finally, we will also release all the annotation interfaces and meta information, inviting everyone from either academia or industry to develop customized version of QA datasets upon SQA3D and its infras- tructure, which might help with the development the 3D-language-relate research and products.
21

---

Published as a conference paper at ICLR 2023
yts zmnchour window. evey
5T focing right q: amochale
Lbt am focing saile caffee g: couch?
bxt sm facing us and sable and
left within
reach. my
clains
Light X
Black X
a: Lei
a: Fox
Picuze X
a:Forwed X
F:Fou
TV x
Browe
X
Four
Bulletin boand
Right
Figure 16: Failure mode. Models are likely to predict the wrong answers when they do not attend to relevant objects.
22

---

Published as a conference paper at ICLR 2023
Table Hyper-parameters for the considered models.
Parameter
Value
ScanQA(modifted)
Optimizer
Adam
Gradient clipping
1.0
Epsilon
1e 8
Weight decay factor
1e-3
Beta hyperparameters for Adam Learning rate
[0.9, 0.999]
5e
Learning rate schedule
No learning rate schedule
Batch size
Total training epochs
Number of layers for transformer
Number of heads for transformer
8
MLP hidden size in MCAN
256
MCAN flatten output size
512
Model hidden size
256
Number of VoteNet output proposals
256
Position regression loss weight a for auxiliary task
1.0
Rotation regression loss weight /3 for auxiliary task
1.0
MCAN
Optimizer
AdamW
Gradient clipping norm
0.3
Epsilon
le-
Weight decay factor
Beta hyperparameters for Adam Learning rate
[0.9, 0.999]
1e-1
Learning rate schedule
No learning rate schedule
Batch size
16
Total training epochs
12
Number of layers for transformer
6
Number of heads for transformer
8
MLP hidden size in MCAN
512
MCAN flatten output size
1024
Model hidden size
512
ClipBERT
Optimizer
AdamW
Gradient clipping norm
5.0
Epsilon
1e-6
Weight decay factor
le
Beta hyperparameters for Adam
[0.9, 0.98
Learning rate
5
Learning rate schedule
No learning rate schedule
Batch size
16
Total training epochs
10
23


Reviews:
[{"review_id": "gf1tLl9maO", "reviewer": "Reviewer_ZAer", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a new benchmark for situated embodied scene understanding. Mechanical Turkers are given tasks of diverse scenes, which include writing text descriptions of situations and human activities, writing questions given these tasks and their text descriptions, and finally, answering these questions. The dataset collected spans over 650 rooms and 6 types of questions. Experiments with 3 different model variants and qualitative analysis show there exist a large gap between embodied AI models and humans.", "summary_of_the_review": "While there exist several benchmark datasets on embodied QA, to the best of my knowledge, this seems to be one of the largest ones and most diverse wrt questions. Concerns involve whether the standalone nature of such datasets and the evaluation with exact match / top-1 accuracy metrics is sufficient for embodied agent scene understanding.", "strength_and_weaknesses": "Strengths:\n- Table 1 presents a clear overview of related work and the contributions of the proposed SQA3D dataset.\n- Experiments with 3 model variations, 3D, Video-Image and text-based.\n\nWeaknesses:\n- Evaluating GPT-3 on 10% of testing data make results uncomparable. Would be better to replace this with an open-source model, e.g., BLOOM.\n- Assuming results reported in Table 3 are top-1 accuracy, the ablation on the important of situation understanding seems to showcase that there is a marginal decrease in most question types when the situation description is removed from the input.\n- Please further explain: \"No further metric is not included as we find it sufficient enough to measure the differences between baseline models with exact match\".\n\n\nTo my understanding, the dataset is built on a standalone web interface rather than on top of an exciting simulator such as AI2-Thor, which could facilitate the evaluation of embodiedQA agents on completing, for example, object navigation tasks. Another comment is on the lack of limitations and societal impact sections. Perhaps articulating similar limitations would make the work more complete, e.g., on the expected usage of the dataset.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YX1twqmESd", "reviewer": "Reviewer_4eyF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new dataset and benchmark for 3D scene question answering. It is based on ScanNet and it consists of a large number of situations, with descriptions and reasoning questions. These questions test the agent in various ways including relation comprehension, commonsense understanding, and navigation. The authors tested several existing QA methods on this dataset and achieved a very low score in comparison to what humans achieve leaving large room for improvement.", "summary_of_the_review": "Please see the Strengths and Weaknesses. Overall this work provides a useful benchmark to encourage research in 3D question answering. However, more description is needed for how this benchmark compares to existing ones and it would be useful to have a variation of existing Q&A methods that can at least slightly address this benchmark's specific challenges.", "strength_and_weaknesses": "Strengths\n\n- The paper goes into great detail in describing how the dataset SQA3D has been acquired, I appreciate the quality check made in generating the situations.\n- The experiments show a thorough analysis of how the methods perform for different question types. \n- SQA3D seems to be a useful challenging benchmark where current methods for question answering are far off from human performance opening doors for a lot of advancement opportunities\n- I enjoyed reading the detector section as different methods were thoroughly compared in terms of strengths and weaknesses and useful practices like having balanced data were provided.\n\nWeaknesses\n\n-More descriptions need to be given how SQA3D’s challenges differ from existing related benchmarks (Azuma et al., 2022; Ye et al., 2021; Yan et al., 2021). One question to ask, does having a model that performs really well on these existing benchmarks enough to deploy the model, or do we need to test it on SQA3D as well?\n\n- It would be useful to have more detailed descriptions of the dataset challenges and why the scores by the AI algorithms are so slow. One way to reconcile that is to pinpoint several challenges and implement variations of existing methods that can at least slightly address those challenges. This strategy is similar to how some of the challenges of imagenet were addressed using a deep network in its original paper\n\n- No multiple runs of the same experiments in Table 3, it would help to see what  the variance is like if different random seeds were used to run the experiments in order to identify if the differences between the results are significant.\n\n- No code was released to verify and reproduce the results and interact with the dataset to get a deep understanding of how it is structured. Could you add a script that allows us to run one of the methods onto one of the situations? Since this is a benchmark there should be an easy-to-use codebase that makes it smooth to run the baselines and extend them for new algorithms.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "T0d81qACYp6", "reviewer": "Reviewer_cywf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This article proposes a task to benchmark the understanding of specific agents, scenes, situational questions, and answers in 3D scenes (SQA3D). Given the 3D scene background, then answering a question that requires a lot of situational reasoning from this perspective. Based on scenes from ScanNet, it provides a data set centered on 6.8k unique scenes, 20.4k descriptions, and 33.4k different reasoning questions for these scenes for scenario answers in 3D scenes. At the same time, the author studies the most advanced multi-mode reasoning models, and the results show that these two models still lag behind human performance to a large extent. It shows the key\nrole of correct 3D representation and the need for better situational understanding in specific scene understanding.", "summary_of_the_review": "Please refer to the above comments for the pros and cons.", "strength_and_weaknesses": "Strength：  \nDifferent from other embedded scene understanding tasks, SQA3D tasks propose to understand and complete tasks in the first person to answer questions, which is more practical and has a broader range of practical task needs. And it has knowledge-intensive questions and a larger scale of the collection. The paper is well-organized, which is easy to read and understand. The current experiment of baseline performance on SQA3D tasks is relatively complete. \n\nWeaknesses：  \nIt is good to introduce how to control the potential biases in the dataset.  \nThe paper should also refer to several recent Situated Reasoning benchmarks or  Video QA benchmarks as they proposed situated reasoning with QA tasks or similar scenarios. For example, STAR: A benchmark for situated reasoning in real-world videos; Agqa: A benchmark for compositional spatio-temporal reasoning, etc.   \nThe experiment in Table 1 shows that the blind model reached 43.65 performance. So the hints or potential connections in the language part are strong which probably will be easy to correlate to answers.   \nProviding more module-level experiments or ablation studies will be good.   ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KAVksrZ0aY", "reviewer": "Reviewer_neRZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel task, called SQA3D, to benchmark and help improve 3D scene understanding of embodied agents. The embodied agent is provided a visual scene context and a textual description of its own location and state in the scene. The task is to answer an input question which requires both an understanding of the scene and the agent’s location in it. The questions encompass spatial relations, common sense understanding, native and multi-hop reasoning. The authors build the task using scenes from the ScanNet dataset and crowdsourcing with human curation is used to prepare the ground truth. Human benchmark (using amateurs) is prepared on the task yielding an overall score of ~90%. Several state of the art approaches are evaluated with the best reaching 47.2% accuracy. This large gap in performance is expected to facilitate further research on embodied scene understanding. ", "summary_of_the_review": "A good contribution introducing a novel shared task in the general area of embodied AI along with a human study, reasonable baselines based on SOTA identifying the large gap between the two. The paper quality is acceptable. I have some concerns regarding reproducibility and the analysis of the baselines could’ve been better. My lower rating reflects these concerns.\n\nSubject to the above, this task is expected to have a positive impact on the research in the area.", "strength_and_weaknesses": "### Strengths\n\n- The paper addresses creates a novel shared task (challenge) for the emerging field of embodied AI, in particular embodied 3D scene understanding. \n- The task is similar to recent work in 3D language grounding and embodies QA but consists of a larger diversity of context-dependent, knowledge-intensive questions on a much larger dataset. \n- Human benchmarking is performed. \n- Several recent multimodal reasoning approaches are evaluated revealing a large performance gap of ~43%. The choices of these seem reasonable and provide a good coverage of different approaches.\n- The large, labeled SQA3D dataset (650 scenes, 20.4k descriptions of ~6.8k situations and 33.4k questions) and the identified performance gap should spur further research on this important topic.\n\n### Weaknesses\n\n- __(W.1)__ I suppose the authors are planning on releasing the code, data and metadata related to the task:  the data cuts used for training their models, the trained models employed to get the results in the paper, the code which changed the SOTA models etc. I might have missed it if the authors explicitly state this. Kindly confirm. \n\n- __(W.2)__ There are some surprising, quantitative results which need further probing and a discussion. \n\n- __(W.2.a) 3D scans__: The VSQ setting should be better than the max of SQ and VQ. This goes to the heart of this task. The improvements seem low on {Is, How, Can} questions. What seems to be going on? It will be good to have human benchmarks in these settings. \n\n- __(W.2.b)  BEV, Ego. videos__: While I agree with the general observations in the paper, low performance on {Is, How, Can, Others} with respect to the blind test is disconcerting. This means that those models are not ‘working’ at all.\n\n - __(W.2.c) Zero Shot__: Clearly captioning is the bottleneck but for a ‘reasonable’ system, the performance shouldn’t degrade below the ‘Blind test’. The low performance on {Is, How, Can, Others} mirrors the above. Can the authors clarify?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster