EQA-MX: Embodied Question Answering using Multimodal Expression
Authors: Md Mofijul Islam
Alexi Gladstone
Riashat Islam
Tariq Iqbal
Venue: ICLR 2024 spotlight (2024)
Keywords: multimodal representation learning
visual-language models
embodied question answering
Primary area: datasets and benchmarks
TL;DR: We present EQA-MX, a dataset and benchmark tasks for embodied multimodal QA, and VQ-Fusion model, enhancing visual-language alignment that outperforming existing models by 13%.

Abstract: Humans predominantly use verbal utterances and nonverbal gestures (e.g., eye gaze and pointing gestures) in their natural interactions. For instance, pointing gestures and verbal information is often required to comprehend questions such as "what object is that?" Thus, this question-answering (QA) task involves complex reasoning of multimodal expressions (verbal utterances and nonverbal gestures). However, prior works have explored QA tasks in non-embodied settings, where questions solely contain verbal utterances from a single verbal and visual perspective. In this paper, we have introduced 8 novel embodied question answering (EQA) tasks to develop learning models to comprehend embodied questions with multimodal expressions. We have developed a novel large-scale dataset, EQA-MX, with over 8 million diverse embodied QA data samples involving multimodal expressions from multiple visual and verbal perspectives. To learn salient multimodal representations from discrete verbal embeddings and continuous wrapping of multiview visual representations, we propose a vector-quantization (VQ) based multimodal representation learning model, VQ-Fusion, for the EQA tasks. Our extensive experimental results suggest that VQ-Fusion can improve the performance of existing state-of-the-art visual-language models up to 13% across EQA tasks.

Published as a conference paper at ICLR 2024
EQA-MX: EMBODIED QUESTION ANSWERING USING MULTIMODAL EXPRESSION
Md Mofijul Islam* Alexi Gladstone', Riashat Islam', Tariq Iqbal & University of Virginia, 8 Amazon GenAI, McGill University, Mila, Quebec Al Institute
EQA-MX Dataset with Nonverbal Gestures and Multiple Visual & Verbal Perspectives
VQA Dataset
EQA Dataset
@: What is the color of teat objecr?
Q: What color the hydrant?
Q: What calor is the car?
Figure 1: Compared to the QA tasks in existing VQA (Antol al. 2015) and EQA (Das et al. 2018a) datasets, the models to answer EQA tasks in our EQA-MX dataset require the reasoning of questions with multimodal expressions (verbal and nonverbal gestures).
ABSTRACT
Humans predominantly use verbal utterances and nonverbal gestures (e.g.. eye gaze and pointing gestures) during natural interactions. For instance, pointing gestures and verbal information is often required to comprehend questions such as "what object is that?" Thus, this question question-anssering (QA) task involves complex reasoning of multimodal expressions (verbal utterances and nonverbal gestures). However, prior works have explored QA tasks in non-embodied settings, where questions solely contain verbal utterances from single verbal and visual per- spective. In this paper, we have introduced 8 novel embodied question answering (EQA) tasks to develop learning models to comprehend embodied questions with multimodal expressions. We have developed a novel large-scale dataset, EQA- MX, with over 8 million diverse embodied QA data samples involving multimodal expressions from multiple visual and verbal perspectives. To learn salient multi- modal representations from discrete verbal embeddings and continuous wrappings of multiview visual representations, we propose a vector- custo--uantization (VQ) based multimodal representation learning model, VQ-Fusion, for EQA tasks. Our exten- sive experimental results suggest that VQ-Fusion can improve the performance of existing visual-language models up 13% across EQA tasks.
INTRODUCTION
Understanding human instructions is crucial for autonomous agents to effectively collaborate with humans (Chen et al. 202 Kratzer et al. 2020 Islam et al. 2022b a). To develop models for instruc- tion comprehension, several tasks have been designed, such as referring expression comprehension Yang et al. 2019b Yu et al. 2016 Kamath al. 2021 Akula et al. [22 Chen et al. (0200a), spa- tal relations grounding Yang et al. 2019at Viethen Dale [2088] Achlioptas et al. 2220 Liu et al,
2022 and visual question answering (Antol et al. [015]] 2015|Gao
2015 Zhu al. 2016 Krishna et al.] 2017 Kafle et 2018 Gurari et al. (008). Notably, VQA has gained sig- nificant attention due to its complex reasoning demands, such as answering questions about object
Gao et al. 2015 Yu et al.
presence and category using visual cues (Antol et al. 20155 Goyal et al. 2017 Lee et al. 2022).
Numerous synthetic and real-world datasets exist for VQA. yet their sole focus verbal questions is a crucial limitation, contrasting with natural multimodal expressions (verbal utterances and nonver- bal gestures) used in inquiries. Studies affirm that nonverbal gestures often provide complementary information for understanding verbal questions (McNeill
2012. Corkum & Moore, 1998: Butter-
worth et al. 2002 Scaife& Bruner 1975 Colonnesi et 2010 Iverson & Goldin-Meadon 2005)
Kita 2003 Liszkowski et al. 2004
Chen et al. 2021
For example, in scene with two differently
'Work does relate position Amazon.

---

Published as a conference paper at ICLR 2024
Table I: Comparison of the QA datasets. Existing VQA and EQA datasets do not contain nonver- bal gestures (NV), multiple verbal (V) perspectives (MVP), contrastive (C) and ambiguous (A) data samples. Embodied (E) interactions refer to humans interacting using multimodal expressions. Embodied interactions refer to an agent navigating in an environment. Please check the supple- mentary for detailed comparison with other related datasets.
Views
No. of No. of Samples
Datasets
NV
E
EQA
MVP
Exo
Ego
Top
Images
VQA (Antol ct al. 2015)
X
X
X
X
2044
614k
KB-VQA (Wang ct al. 2015
X
X
X
X
/
X
X
X
X
0.7k
5k
FBQA (Wang et al. 2017
X
X
X
X
V
X
X
X
2k
5k
VQA-MED Hasan
2018)
X
X
X
V
X
2k
6k
Doc Mathew et al. 2021
X
X
X
J
X
12k
50K
GRiD-3D Lee et al. 2022
X
X
X
X
J
X
X
X
X
8k
445k
VIMA Jiang et al,
2022
X
X
X
X
J
X
X
X
X
650k
650k
EQA Das el al.
2018a
X
/
/
X
5k
Sk
MT-EQA
(Das et al.
2018a
X
5
5
X
X
X
X
X
19k
19k
EQA-MX
750k
8,243k
colored balls, a pointing gesture can clarify questions like "what is the color of that ball?" The absence of nonverbal interactions in prior VQA datasets makes them less suitable for developing models to comprehend investion-nsswent
(QA) tasks in embodied settings.
Following VQA, embodied question-answering -nswering (EQA) tasks have recently been studied in the liter- ature (Yu et al. 2019; Luo et a [2019] Gordon et al., 2018 Tan et al. (020). EQA can be bifurcated based on embodied interactions: the first centers on an agent, like a virtual robot, navigating to answer verbal questions (Das et al. 2018a), solely incorporating verbal queries. The second encom- passes multimodal expressions, where humans interact with the environment using verbal utterances and gestures (Chen et al. 2021 Islam et al. 2022a). Adopting the latter definition, we designed EQA tasks to comprehend questions using multimodal expressions (verbal uttrances and nonverbal ges- tures) in embodied settings. For instance, an EQA task may involve pointing to an object and asking "what is that object?" requiring reasoning over multimodal expressions answer the question.
A notable limitation in many existing VQA and EQA datasets is the singular perspective (either speaker observer) of verbal utterances, unlike real-world interactions where where people use both perspective interchangeably. For instance, a speaker's question, "What is the object to the right of the red mug?" could be interpreted as left of the red mug from an observer's perspective. This lack of multiple perspectives in existing datasets hinders the development of robust QA models.
Similarly, existing VQA and EQA models answer questions from single visual perspective LLi iet al.) 2019; Kim et al. 2021: Lu et al., 2019). Multiple views provide complementary informa- tion. and varying camera angles capture interactions differently. Aligning visual representations before merging with verbal ones can aid in developing generalized representations and robust com- prehension across perspectives. Moreover, the inconsistency of embedding structures, particularly continuous visual and discrete verbal representations, can lead to sub-optimal representations.
To address the shortcomings of existing VQA and EQA datasets, we have extended an embodied simulator to develop a large-scale novel dataset, EQA-MX, for comprehending EQA tasks (Table We have addressed the limitations of existing multimodal fusion approaches and developed a mul- timodal learning model for EQA tasks, VQ-Fusion, using vector quantization (VQ). The VQ-based bottleneck plays a key role in disentangling the continuous visual representations into discrete em- beddings and enables salient fusion with discrete verbal representations. We shared codebook in VQ to align multiview representations and learn the unified concept shared among multiple views. We highlight our key contributions below:
We developed a large-scale dataset (EQA-MX) with multimodal expressions from various verbal and visual perspectives to reduce perspective bias and enhance model generalizability.
We designed 8 new EQA tasks blending multimodal questions (verbal and gestural) to be ad- dressed using visual context in an embodied setting.
We designed a VQ-based multimodal fusion method to align continuous visual and discrete verbal representations and extract salient representations across multiple visual and verbal perspectives. Our extensive experimental analyses indicate that our proposed model, VQ-Fusion, can help to improve the performance of EQA tasks up to 13%.
2

---

Published as a conference paper at ICLR 2024
Task Distributions EQA-MM
Ego-view
Exp-view
Top-wiew
10.%
15.h
oC
Task Name
Verbal Questions
Answer Set
Existence Prediction (EP]
QU there plant in the scena? Yes
Wea/No
Object Grounding
What the name that object? A: Illender
All 52 objects
Perspectiv-Awwan
Considering obsenvers perspective,
the
objects
Grounding [POd]
thing
Object Counting [OC]
yellow object? A:7 What is color of thing?
many objects
(0-6)
Object Attribute Query [DAO]
colors
Objeci Attribute Compare [DAC]
the lhet object the seme color kettlle? No
Wea/No
Pempective Grounding [PG)
Q: blerder to the right of the book. from which perspeccive is the
Speellnn/Oneevee/eeeerrr
object described? A: Speaker
Relation Grounding (RG)
Q: Considering the observer's perspective, the yellowe blenderrbeeerrrs to the right of bread, the object referred to aporopriataly? A:
Ves/No
Figure 2: EQA tasks for sample data from EQA-MX. Top-row: data distribution for each task in
EQA-MX (left) and an embodied interaction with multiple visual perspectives (right). Bottom-row: name of the task (left), example questions and answers for the given task based on the visual scene above (middle), and the set of possible answers (right).
2 RELATED WORK
Visual Question Answering:
Many datasets have been developed to study visual question-
answering tasks (Gao et al.] 2013:
Zhu al., [2016] Liu et al., 2019 Krishna et al.) [017]
Wang
ct al.
2015 2017; Kembhavi et al.
2016| et al. 2017 Kafle et al. |018| Gurari et al. 2018
Hasan al. 2018 Huang et al.
2018 Andreas al.) 2016 Chou et al., 2020: Hudson & Manning
2019 Mishra et al. 2019). These datasets primarily involve answering verbal questions using the
visual scene as context. For example, Antol al. (2015) developed a VQA dataset and introduced
QA tasks involving an image and verbal questions about the image. This dataset contains both real-world images from the MS-COCO dataset (Lin al. 2014) as well as synthetic virtual scenes containing clipart. |enn et al. (2015) generated synthetic QA pairs using an algorithm that converts image descriptions into QA form. Recently, few datasets have been developed containing multi- modal expressions (Schauerte & Fink 2010 Islam et al. (022b). For example, Chen et al. (202I) developed dataset for referring expression comprehension tasks embodied settings, where a human uses multimodal expressions to refer to object.
Several visual- language (VL) models have been developed for VQA tasks and were consequentially evaluated on these datasets (Radford et al., 2021 Lu et al. 2019 Tan & Bansal, 2019 Chen al.) 2020a). For example, Liunian et al. (Li et al.) 2019) developed VisualBERT to answer a question using the visual context by learning multimodal representations from visual and verbal embeddings. Kim et al. (Kim et al.] (021) designed a VL Transformer model (ViLT) with monolithic processing of visual inputs learn VL rspresttattions without regional supervision of object detection.
Embodied Question Answering (EQA): EQA tasks are often designed as agents (c.g.. virtual robots) navigating in an environment answer verbal question. For example, Das et al. (0 Sa) developed a synthetic dataset, where a virtual robot navigates the environment and gathers visual in- formation from an egocentric view answer verbal question. Yu et al. (2019) extend this dataset and include questions with multiple targets, such as finding multiple objects through navigation. However, some works have used embodied interactions to refer to comprehending referring expres- sions (Islam et al. 2022b: Chen et al.] 2021). We follow this definition of embodied interaction.
Several models have been developed for existing EQA tasks. For instance, Das et al. (20T8h) (8)) troduced modular model for learning a policy to navigate and answer verbal questions, while Gao cet al. (2021) utilized a transformer-basee clues. These models aim to develop a navigation policy for answering verbal questions.
model to generate scene memory tokens as exploration
Most current VQA and EQA studies focus on understanding solely verbal questions, contrasting goal of comprehensively understanding multimodal expressions (verbal utterances and gestures) in embodied settings. Moreover, existing models fuse disparate embedding structures (continuous visual and discrete verbal representations), potentially leading sub-optimal VI. repssenttttins
3

---

Published as a conference paper at ICLR 2024
3 EMBODIED QUESTION ANSWERING TASKS
We have created 8 novel EQA tasks: Ex- istence Prediction (EP), Object Grounding (OG), Perspective-Aware Object Ground- ing (POG), Object Counting (OC), Object
Obwerver
33.3%
33.3%
Attribute Query (OAQ), Object Attribute Compare (OAC), Perspective Grounding (PG), and Relation Grounding (RG). Sim- ilar tasks have been developed in prior works (Antol et al. 2015 Lee et al. 2022
33.3%
Nentral
(a) Questions length in words.
(b) Verbal perspective (PG).
Ego view
Exo view
Too view
Wang al. 2017 Yu et al.) 2016 Ren Zhu et al.J 2016), however, those tasks involve only verbal questions. We are the first to design QA tasks em- bodied settings where human avatar asks questions using verbal utterances and non- verbal gestures in a virtual environment. Each task has multiple sub-ten sub-templates for variation (described further in the supple-
ct al. 2015
Left
Right
(c) Object locations with respect to spatial relations.
Figure 3: EQA-MX Dataset Analysis: (a) demon- strates varied question lengths in the EQA-MX dataset, indicating differing contextual information across EQA tasks. (b) presents data sample ratios of different verbal perspectives for the perspective grounding (PG) task. (c) depicts object locations in relation to different spa- tial relations, showing the EQA-MX dataset's 's non-bias towards verbal and visual perspectives due to insepa- rable object locations. For detailed analysis, refer to the supplementary materials.
mentary materials). In Fig. 2 we provide samples of these EQA tasks.
Existence Prediction (EP): The EP task involves determining whether the scene contains particular object with specific attributes (e.g., color, location). Complet- ing this task requires knowledge of object appearances as well as holistic under- standing of the scene.
Object Grounding (OG): In the OG task, the object category is determined based on the question by utilizing multimodal expressions. This task also involves understanding which perspective the question is asked from (i.e., speaker, observer, neutral).
Perspective-Awart Object Grounding (POG): Similar the OG task, the POG task involves determining which object is being referred to. However, this task also includes the verbal perspective in the question (speaker, observer, neutral). This done intentionally to determine how much of an impact verbal perspective has on performance.
Object Counting (OC): In the OC task, the number of objects scene asked based on different spatial relations. To understand this, different objects in the visual scene must be attended to and spatial relations given in the verbal question must be used determine whether or not certain objects have that attribute prior to counting.
Object Attribute Query (OAQ): The OAQ task involves determining the color of a given object that is queried for, which can be helpful in scenarios where humans are interested in learning partic- ular characteristics of an object. The spatial location and the color of the object must be determined using the given verbal and nonverbal expressions.
Object Attribute Compare (OAC): The OAC task entails comparing two objects' attributes, in- volving pointing to object and querying their similarity in attributes.
Perspective Grounding (PG): Understanding human verbal perspective is crucial for effective human-Al communication, humans describe objects from varying perspectives. We simulate this by employing three perspectives neutral, speaker, and observer, tasking the model with identifying the perspective of given question.
Relation Grounding (RG): The RG task involves determining whether a verbal utterance and nonverbal gestures refer the same object. As referring expression can be interpreted differently from different visual and verbal perspectives, understanding the RG task requires complex reasoning of perspective and spatial relations.
4

---

Published as a conference paper at ICLR 2024
4 DATASET GENERATION WITH EQA SIMULATOR
In this work, we have extended the CAESAR simulator (Islam et al. 2022b) to generate data for different EQA tasks. CAESAR is used to randomly generate environments where an actor simulates nonverbal expressions through a pointing gesture and gaze in a scene (Fig. 33. Verbal expressions are created based the visual scene. To increase the dataset's generalizability, we have used multiple environments. These environments differ in terms of camera views, object locations, and nonver- bal/verbal expressions. In each visual scene, we generated four different situations, 1) situation with no human and therefore no nonverbal expressions, 2) a situation with a human head gaze, 3) a situation with human pointing gesture, and 4) situation involving human using a head gaze and a pointing gesture.
Generated nonverbal expressions consist of a pointing gesture and gaze. Pointing gestures are pro- cedurally generated using inverse kinematics through the Unity engine. We create these pointing gestures based on random noise added onto real-world data of human pointing gestures captured us- ing an Optitrack motion capture system (opt). Similarly, we have simulated human head gazes using inverse kinematics and an object location within the scene as a target. Verbal questions are generated based on different templates for each EQA task. The nonverbal and verbal expressions may describe the same object, or be contrastive, meaning the nonverbal and verbal expressions describe different objects. We use these contrastive instructions for the Relation Grounding task. Additionally, the ab- sence of nonverbal gestures in situations with no humans generates ambiguous data samples. Please check the supplementary document for additional details the data generation process.
DATASET ANALYSIS
Table 2: EQA-MX dataset splits for 8 EQA tasks.
Splits EP
OG
POG
OC
OAQ
OAC
PG
RG
We have generated novel large- scale dataset, EQA-MX, containing
Train 1060k
1060k
1060k
1060k
1060k
218k
785k
349k
Valid 126k
126k
126k
126k
126k
27k
93k
41k
8, 243, 893 samples across the tasks
Test 126k
126k
126k
126k
126k
28k
93k
42k
described in Sect. 3 The training, validation, and test set splits for each of these tasks shown in Table
We removed some data
samples to generate balanced dataset splits for the OAC, PG, and RG tasks.
Our designed EQA tasks vary in terms of the goals (Fig. 2 and visual-verbal contextual information in the questions. This is made apparent by the variance in question lengths in words (Fig. ((a)) Questions are as short as words for the EP task and as long as 34 words for the OG task. Addition- ally, one of the main focuses of the EQA-MX dataset is to introduce data that varies in verbal and visual perspectives. Fig. 3) demonstrates the PG task's outcome of different verbal perspectives.
Similarly, Fig. 3 shows the location of objects based on spatial relations in questions from verbal perspectives. Fig. (e also demonstrates how objects being referred as on the left (blue) and right (red) are not linearly separable through the use of spatial relations, different verbal perspectives use different relations to describe an object. For example, consider a speaker describing the red table lamp in Fig. 2 The speaker could state "the red lamp on the left". However, from the observer's perspective (exo view) the table lamp is on the right. Thus, given the verbal perspectives, spatial relations are non-separable in EQA-MX (Fig. b)). This reduced verbal and visual perspective biases in EQA-MX dataset can help train robust models for comprehensively comprehending EQA tasks. Please check the supplementary materials for more detailed data analysis.
6 VQ-FUSION: VQ-BASED MULTIMODAL FUSION
We develop vector quantization-baser multimodal fusion approach, VQ-Fusion, to learn visual- language representations. As EQA tasks in EQA-MX involve multiple visual views, VQ-Fusion extracts visual representations from multiple visual views (X ego- Xezo- and Xtop) and verbal ques- tions (X,) for different EQA tasks (Fig. 4 and Sect. Following the existing adapter-based based learning
models (Beck al.) 2022 Ansell et al. 2021 Rücklé et al.) 2021: Pfeiffer et al.) [2000a] b 202
we design VQ-Fusion as an adapter model that can be used in existing models without significantly changing the existing model architecture.
Visual and Language Representation Learning: At first, VQ-Fusion extracts visual and lan- guage representations using a state-of-the-art visual encoder (e.g., ResNet (He et al. 2016) and
5

---

Published as a conference paper at ICLR 2024
VIT (Dosovitskiy et al. 2020 and language model (e.g., BERT (Devlin al. 2018)). VQ- Fusion uses shared models to extract the visual representations from muluple views indepen- dently: Em = Fm (XX)), E (ego, ezo, top, verbal). Here, F., is the visual or verbal encoders, Em E ERDm, and Dm is the representation dimension of modality m.
Discretization and Multimodal Fusion: Language models create discretized repre- sentations, whereas visual encoders pro- duce continuous representations of visual scenes. Fusing these representations with different embedding structures can lead to sub-optimal multimodal representations [Liang et al. (022). For this reason, we discretize the visual representations before fusion.
Lrask
EQA Task
Discrete Multimodal Fusion
Discrebzation
Factor-1
Factor-1
Factor-1
Avqgattion
Factor-2
Factor-2
Factor-2
Factor N
Factor N
Factor-N
Vector Quantization
In VQ-Fusion, we adopted the vec- tor quantization (VQ) method from VQ- VAE (Van Den Oord et al.) (017)) and Discrete- Valuue Neural Communica- tion (Liu et al.] (022)) works to dis- cretize multiview visual representations, Fm E (Fego, Fezzo Frop). Previ- ous works use VQ to discretize a rep- resentation using codebooks, whereas use shared codebooks discretize and align multiview representations to learn unified concepts across visual views for extracting salient multimodal repre- sentations. First, VQ-Fusion divides each Em into G continuous segments (s(m,1): (...))....... ((,,))) where Em = CONCAT(S (mm,)) ((m.2)..... (m,,)) and ((m.)) E RDm/G Second, VQ-Fusion independently maps continuous segment ((,,) to dis- crete latent code cy E RR.. (P../))).))) using shared codebooks C, where L is codebooks size (i.e., number of categorical codes in each codebook). We can find the optimal code for each contin- uous segment ((m,)) from the codebooks in the following way: ((m,os)) FD (s),s)) o arg max([Is(m.)) - Gyll Here, FD is the discretization (D) method. Finally, we concate- jEI...
Paprosentatoon
Repreamtatoo
Peprosentatio
Visual Encoders
Language
(Shared)
Model
Codebooks (Sharad)
What
yellow object?
Figure 4: VQ-Fusion: Vector Quantization (VQ) based multimodal learning model architecture. VQ-Fusion extracts multiview visual representations using visual encoders, which are then discretized using shared codebooks. The shared codebooks' bottleneck allows the model to learn unified concepts across multiple views. Finally, discretized visual representations are fused with discrete verbal representations to produce multimodal representation.
nate the discretized codes to produce discretized visual representation ED in the following way: FD (s(m,))))
ED (st..)))
Following the training procedure in (Liu al. 2021 and Van Den Oord et al. 2017), we calculate VQ loss for learning the codebooks: LvQ.align gradient operator blocking gradients to Co and 3 is a hyperparameter controlling reluctance to change the code. We train the discretization module learn codebooks using gradient descent with the other parts of VQ-Fusion. As VQ-Fusion employs shared codebooks to discretize visual representation for multiple views, LvQ align loss aids in aligning multiview representations and learning unified concepts across views. This shared codebooks approach allows aligning multiview representation to answer the question with multimodal expressions effectively.
[ss sg(co, 1e. Here, sg is the stop-
Finally, VQ-Fusion fuses these discretized visual and verbal representations using a self-attention approach to produce task representation:
Efused
Σ am Here, Om - exp(Tm) and
exp()
mEM
mEM
Ym = (W)T Em. m E M Here, M the modality list (ego, exo, top, verbal), W is a learnable parameter, and am is the attention score which is calculated using a ID-CNN with filter size of 1.
Task Learning: We use the fused representation. Efuseds to learn different EQA tasks T3 yr, Fr, (Ffusod). Here, Fry, is the task learning module, which can be designed based on the EQA task

---

Published as a conference paper at ICLR 2024
Table Comparisons of VL models performance for EQA tasks. The results suggest that incorpo- rating VQ-Fusion in VL models can improve the performance of EQA tasks. /: VL models with VQ-Fusion, and X: VL models without VQ-Fusion.
EP
OG
POG
OC
Models
Dual Encoder
53.46
55.78
48.31
49.96
83.91
84.28
12.28
12.38
CLIP
53.17
54.72
54.06
65,49
70.92
82.70
09.65
13.14
VisualBERT
50.00
54.51
53.39
54.50
86.09
87.09
14.09
14.35
ViLT
90.24
91.50
59.74
61.04
86.10
87.42
11.14
12.54
OAQ
OAC
PG
RG
Models
X
Dual Encoder
63.71
66.90
57.92
61.45
66.72
66.77
75.78
89.36
CLIP
70.85
74.32
58.59
70.59
66.64
66.99
85.84
89.93
VisualBERT
51.43
54.45
58.56
59.98
66.37
79.11
89.13
89.26
ViLT
55.96
59.47
58.93
60.16
80.36
81.23
87.36
88.68
properties. For example, please check the supplementary for further details). Moreover, Ltask,Ty is used to train the model for task Tk: Leask,T, Ty (yT, (r,))
multi-layer perceptron for the object existence task (Sect. R and
El= V(T,.5) .) log ((T.,)). Finally, combine the task learning loss (Leask,T, with the VQ loss (LvQ atign) using task learning weights (Wvq and Wrask) to train WhaskeCaskt
the VQ-Fusion model: & WvQLvq.ee
Variations of VQ-Fusion: VQ-Fusion allows use existing VL models (e.g., VisualBERT Li al. 2019) & ViLT (Kim et al. (02T)) to extract these representations. As the architecture of these VL transformer models limited processing a single visual and verbal input, we need pair the verbal question to each visual view and pass through these models to extract multiview visual and verbal representations. We use these representations VQ-Fusion to discretize and fuse to produce multimodal representations. Please check the supplementary materials for further details.
EXPERIMENTAL ANALYSIS
In this section, we have presented experimental analyses on our EQA-MX dataset to evaluate the impact of VQ-Fusion in VI. models for EQA tasks. We have included additional ablation studies and experimental analyses for another task in the supplementary to evaluate the significance of VQ- Fusion for multimodal representation learning.
Existing visual-lagguage (VL) models for QA tasks are designed answer question using single visual context. Since our proposed EQA tasks involve three visual views, we extend four VL models learn multiview representations: Dual-Encoder (ViT+BERT) (Dosovitskiy
Baseline Models:
et al. 2020 Devlin et al. (018). CLIP (Radford et al. 2021), VisualBERT eLi al. 2019), and ViLT (Kim et al. 2021). For the Dual-Encoder (ViT+BERT) model, we independently extract visual representations for each view using shared VIT model and verbal representations using BERT model. fuse these visual and verbal representations to produce task representations. For CLIP models, pair each visual view to verbal question and pass this through the model to extract multiple visual and verbal representations and fuse them to produce task representations. For VisualBERT and ViLT, use ResNet-10 101 (He et al. (016) to extract visual representations that passed through the model with verbal embeddings to produce task representations. Please check the supplementary materials for further details.
7.1 COMPARISON OF MULTIMODAL LEARNING MODELS
We evaluated fate-of-the-ar visual-language (VL) models with and without our VQ-Fusion to learn
VL representations for 8 EQA tasks. We varied the number of codebooks to {2, 4, 16) in VQ for each task and reported the best performance. We trained and evaluated these models independently EQA-MX dataset. We used data samples with varying nonverbal gestures: gaze and pointing gestures, only gaze, and only pointing gestures. All the visual views (ego, exo, and top) and verbal perspectives (speaker, observer, and neutral) used
for each task single-task model

---

Published as a conference paper at ICLR 2024
to train models and evaluate whether the models can learn generalized representation from diverse data. We report macro-accuracy across all tasks to accurately gauge whether models can effectively understand EQA tasks and are not biased toward particular class (Table
The results in BableBssugger that incorporating VQ-Fusion in VL models helps to success- fully fuse extracted salient multiview representations with verbal representations, and thus improves model performance on EQA tasks. For example, the CLIP model without VQ-Fusion achieves 54.06% accuracy in the object grounding task (OG), whereas incorporating VQ-Fusion in the CLIP model increases the OG task's performance to 65.49%. Similarly, VQ-Fusion improved the CLIP model's performance on the object attribute query task (OAQ) by 12%, the VisualBERT model's performance on the perspective grounding task by 12.74%. the ViLT model's performance on the object attribute comparison (OAC) task by 3.5%, and the DualEncoder model's performance on the relation grounding task (RG) by 13.58% These performance improvements validate the sig- nificance of VQ-Fusion in extracting salient multimodal representations from multiple visual and verbal perspectives for effectively learning EQA tasks.
Results:
Discussion: The primary reasoning behind the performance improvement by incorporating VQ- Fusion in VL models lies in its discretization of multiview representations before fusion with discrete verbal representations. VQ-Fusion uses codebooks to discretize and align the visual representations with the discrete structure of verbal representations. Conversely, existing VL models extract con- tinuous monolithic visual representations and fuse them with discrete verbal representations. This structural mismatch leads to sub-optimal multimodal fusion, adversely affecting the extraction of salient task representations and subsequently degrading task performance.
Moreover, as VQ-Fusion uses shared codebooks in the VQ information bottleneck to learn multi- modal representations, this codebook sharing enables models to align the multiview representations and learn unified concepts. Learning unified concepts from multiple views is crucial, as multiple views capture the same interaction. Existing models are designed to learn visual and language rep- resentations from a single visual perspective. Thus, these models do not have any mechanisms to extract unified concepts from multiple visual views. VQ-Fusion enables these models to learn this unified concept using shared codebooks-basee VQ.
Our experimental results also indicate that incorporating additional perspective-relee relatted information can help models to successfully ground objects. This is made apparent by the model performance on the perspective-awar object grounding (POG) task being consistently higher then the model perfor- mance on the object grounding (OG) task. This is particularly notable as the only difference between these tasks is the presence of the question's 's verbal perspective (Fig. 2 Thus, these results suggest models need to understand verbal perspective for successfully grounding objects in situations with multiple verbal perspectives.
Although the VL models presented can achieve considerable performance for most of the EQA tasks, these models perform slightly better than random- gueessinggingggg for the object counting (OC) task. As these models do not use object location-specific information, the models suffer at locating and counting objects given a spatial relation. One possible extension of these models to improve perfor- mance for the OC task is learning mechanisms to push VL models to learn object locations. The EQA-MX dataset contains rich annotations of object locations, which can easily be incorporated in developing models more capable of understanding spatial locations.
7.2 IMPACT OF NONVERBAL GESTURES (ABLATION STUDY)
We evaluated the impact of nonverbal gestures on learning EQA tasks. We evaluated VQ-Fusion with CLIP models and 8 codebooks on the different splits of EQA-MX dataset: data samples with gaze and gestures, only gaze, only gestures, and without gaze and gestures (this data split contains visual scenes without human).
The results in Table suggest that the model performs is worse for EQA tasks if we train the model using data without nonverbal gestures. For example, the model trained using data without nonverbal gestures achieved only 26.65% accuracy for the object grounding (OG) task, whereas the model trained using data with gaze and pointing gestures achieved 68.61% % accuracy for the OG task. This is a trend for all other tasks where the performance improved when gaze and/or pointing gestures were incorporated compared to when it only relied on the verbal
Results and Discussion:
8

---

Published as a conference paper at ICLR 2024
message. The performance degradation indicates that the models must learn nonverbal gestures to answer questions with multimodal expressions for EQA tasks.
7.3 IMPACT OF VQ CODEROOKS (ABLATION STUDY)
We evaluated VQ-Fusion with the CLIP model for 8 EQA tasks by varying the number of codebooks in VQ: {2, 4, 8, 16). We evaluated these models on our EQA-MX with varied nonverbal gestures (gaze and pointing gestures, only gaze, and only pointing gestures). We trained these models with multiple visual and verbal perspectives.
Results and Discussion: The sults in 5 suggest that different codebooks help the model achieve the highest performance for differ- ent tasks. For example, VQ-Fusion with 8 codebooks can achieve the highest performance in existence pre- diction (EP), object grounding (OG), and object attribute compare (OAC) tasks, whereas VQ-Fusion with 2 codebooks can achieve the highest V V 54.38
Table 4: Impact of gaze (G) and pointing gestures (PG) in learning EQA tasks. The results suggest that incorporating gestures improves EQA task performance. G (X) and PG (X) indicate visual scenes that do not include humans.
EQA Tasks
PG
EP
OG
POG
OC
OAQ OAC
PG
RG
X
51.03
26.65
52.79 09.94 24.01 51.22 48.95
56.75
X
V 53.87
60.66
71.08
11.51 64.69
60.63 66.31
90.01
V x
53.51
63.49
70.90
12.29
69.43
61.25 66.67 87.23
68.61
79.68
11.86
72.62 60.74 66.68 89.59
performance for perspective-awart object grounding (POG) and object counting (OC) tasks. The number of codebooks depends on the task complexity of how many concepts need to be learned. As the OG task requires learning verbal perspective, the model requires more codebooks perspective-relatee concepts. On the other hand, as perspective is already given in the POG task, VQ-Fusion requires fewer codebooks. Our results also show similar phenomena, where VQ-Fusion achieves 82.700 70% accuracy for the POG task with only 2 codebooks, whereas it achieves 65.49% accuracy for the OG task with 8 codebooks.
learn
However, increasing codebooks more than optimal leads to decreasing task performance. For example, the object attributes compare (OAC) task accu- racy degrades if we increase the num-
Table 5: Impact of the number of VQ codebooks (VQ CBs) in VQ-Fusion with the CLIP model in learning EQA tasks.
VQ
EQA Tasks
ber of codebooks to more than 4. As the OAC task involves whether two
CBs
EP
OG
POG
OC
OAQ
OAC
PG
RG
2
53.46
64.86
82.70
13.14
61.39 57.43
61.39
88.24
4
52.15
61.12
73.94
11:35
69.42
70.59
60.30
89.93
objects have the same attribute, the
8
54.72
65.49
73.97
11.92
70.85
60.68
66.82
88.23
model can learn these simple con-
16.
53.19 55.12
71.32
11.43
69.35
60.37
66.99
84.36
cepts using fewer codebooks. In-
creasing the number may lead sparsity in codebooks, i.e., many codes are left unutilized, hin- dering models from extracting salient representations. On the other hand, using a few codebooks for complex tasks, such as OG and OAQ, leads tight bottlenecks, which deters models from learning salient concepts. This results in lower task performance. These results indicate that each task has a different optimal number of codebooks.
CONCLUSION
To develop models for comprehending embodied interactions, we designed 8 novel EQA tasks re- quiring comprehension of questions with multimodal expressions (verbal and nonverbal gestures). To train and diagnose models for these EQA tasks, we developed a novel large-scale dataset, EQA- MX, which contains questions with multimodal expressions from multiple verbal and visual per- spectives. Moreover, we developed vector quantization-base model. VQ-Fusion, to learn salient multimodal representation from multiple visual and verbal per- spectives. Our extensive experimental analyses suggest that VQ-Fusion effectively fuse continu- ous multiview visual and discrete verbal representation, which helps to improve the visual-languarl model's performance for all EQA tasks up to 13%.
multimodal representation learning

---

Published as a conference paper at ICLR 2024
REFERENCES
Optitrack. https:://opittrcck com/ Accessed: 2022-06-03.
Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas J. Guibas. ReferIt3D: Neural listeners for fine-graid meneggained 3d object identification in real-world scenes. In 16th European Conference on Computer Vision (ECCV), 2020.
Arjun Akula, Varun Jampani, Soravit Changpinyo, and Song-Chun Zhu. Robust visual reasoning via language guided neural module networks. Advances in Neural Information Processing Systems, 34, 2021.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings the the IEEE conference computer vision and pattern recognition, pp. 39-48, 2016.
Alan Ansell, Edoardo Maria Ponti, Jonas Pfeiffer, Sebastian Ruder, Goran Glavaš, Ivan Vulić, and Anna Korhonen. Mad-g: Multilingual adapter generation for efficient cross-lingual transfer. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 4762-4788 2021.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV), 2015.
Tilman Beck, Bela Bohlender, Christina Viehmann, Vincent Hane, Yanik Adamson, Jaber Khuri, Jonas Brossmann, Jonas Pfeiffer, and Iryna Gurevych. Adapterhub playground: Simple and flexi- ble few-shot learning with adapters. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 61-75, 2022.
George Butterworth, Fabia Franco, B McKenzie, Lida Graupner, and Brenda Todd. Dynamic aspects of visual event perception and the production of pointing by human infants. British Journal of Developmental Psychology, 20(1): 2002.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision, pp. 104-120. Springer, 2020a.
Yixin Chen, Qing Li, Deqian Kong, Yik Lun Kei, Song-Chur Chun Zhu, Tao Gao, Yixin Zhu, and Siyuan Huang. Yourefit: Embodied reference understanding with language and gesture. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 1385-1395, 2021.
Zhenfang Chen, Peng Wang. Lin Ma, Kwan Yee K Wong, and Qi Wu. Cops-ref: A new dataset and task on compositional referring expression comprehension. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10086-10095, 2020b
Shih-Ho Han Chou, Wei-Lun Chao, Wei-Sheng Lai, Min Sun, and Ming- Yang. Visual question answering on 360deg images. In Proceedings the the IEEE/CVF winter conference on applications
of computer vision, pp. 1607-1616, 2020.
Cristina Colonnesi, Geert Jan JM Stams, Irene Koster, and Mare J Noom. The relation between pointing and language development: A meta-analysis. Developmental Review, 30(4):352-366, 2010.
Valerie Corkum and Chris Moore. The origins of joint visual attention in infants. Developmental psychology, 34(1):28, 1998.
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruy Batra. Embod- ied question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-10, 2018a.
Abhishek Das, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Neural modular control for embodied question answering. In Conference Robot Learning, pp. 53-62 62. PMLR, 2018b.
Hann De Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville. Guesswhat?! visual object discovery through multi-modia dialogue. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5503-5512, 2017.
10

---

Published as a conference paper at ICLR 2024
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprins arXiv:1810.0480) 018.800000400, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Reprexentations, 2020.
WA Falcon. Pytorch lightning. GitHub. Note: http://ggttubb.oom//yyyyoellhhit lightming, 2019.
Chen Gao, Jinyu Chen, Liu, Luting Wang, Qiong Zhang, and Wu. Room-and-objec aware knowledge reasoning for remote embodied referring expression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3064-3073, 2021.
Haoyuan Gao, Junhua Mao, Jie Zhou, Zhiheng Huang, Lei Wang, and Wei Xu. Are you talking to a machine? dataset and methods for multilingual image question. Advances in neural information processing systems, 28, 2015.
Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. Iqa: Visual question answering interactive environments. Proceedings of the IEEE 2018.
conference on computer vision and pattern recognition, pp.
Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 20177
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3608-3617, 2018.
Sadid A Hasan, Yuan Ling, Oladimeji Farri, Joey Liu, Henning Müller, and Matthew Lungren. Overview of imageclef 2018 medical domain visual question answering task. Technical report, 10-14 September 2018, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Li-Chi Huang, Kuldeep Kulkarni, Anik Jha, Suhas Lohit, Suren Jayasuriya, and Pavan Turaga. Cs-vqa: visual question answering with compressively sensed images. In 2018 25th IEEE Inter- national Conference on Image Processing (ICIP), pp. 1283-1287. IEEE, 2018
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real- world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Md Mofijul Islam and Tariq Iqbal. Hamlet: hierarchical multimodal attention based human activ- ity recognition algorithm. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 10292- 2020. doi: 10.1 (U09/RROS45443.20000... 90499999.997...
Md Mofijul Islam and Tariq Iqbal. Mumu: Cooperative multitask learning- based guided multi- modal fusion. Proceedings of the AAAI Conference Artificial Intelligence, 36(1),0033-1051, URL httpt::////ootttttt aaai www.:oorgiiiio..... /wngg/inee.../// php/
Jun. 2022. doi: 10.16099aaii.v36ii AAAI
Md Mofijul Islam, Alexi Gladstone, and Tariq lqbal. PATRON: Perspective-awart awaree multitask model for referring expression grounding using embodied multimodal cues. 2022a.
Md Mofijul Islam, Reza Manuel Mirzaiee, Alexi Gladstone, Haley N Green, and Tariq Iqbal. CAESAR: A multimodal simulator for generating embodied relationship grounding dataset. In NeurlPS, 2022b.
11

---

Published as a conference paper at ICLR 2024
Jana M Iverson and Susan Goldin-Meadow. Gesture paves the way for language development. Psychological science, 16(5):367-377 2005.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang. Yonggiang Dou, Yanjun Chen, Li Fei- Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv, 2022.
Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visual- izations via question answering. In Proceedings of the IEEE conference on computer vision and pattem recognition, pp.
2018.
Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. Figureqa: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300, 2017.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Car- ion. Mdetr-modulated detection for end-to-end multi-modal understanding. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1780-1790, 2021.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. ReferltGame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pp. 787-798, Doha, Qatar, Oc- tober 2014. Association for Computational Linguistics. doi: 10.3115/v1/D14-0066 URL
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. In European conference on computer vision, pp. 235-251. Springer, 2016.
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: ssiion-and-languaae transformer without convo- lution region supervision. In International Conference on Machine Learning, pp. 5583-5594. PMLR, 2021.
Sotaro Kita. Pointing: Where language, culture, and cognition meet. Psychology Press, 2003.
Quan Kong, Ziming Wu, Ziwei Deng, Martin Klinkigt, Bin Tong, and Tomokazu Murakami. MMAct: large-scale dataset for cross modal human action understanding. In IEEE Interna- fional Conference on Computer Vision (ICCV), pp. 8658 2019.
Philipp Kratzer, Simon Bihlmaier, Niteesh Balachandra Midlagajni, Rohit Prakash, Mare Toussaint, and Jim Mainprice. Mogaze: A dataset of full-body motions that includes workspace geometry and eye-gaze. IEEE Robotics and Automation Letters, 6(2):367-373, 2020.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsourced dense image annotations. International of computer vision, 123(1):32-73, 2017.
Jae Hee Lee, Matthias Kerzel, Kyra Ahrens, Cornelius Weber, and Stefan Wermter, What is right for me is not yet right for you: A dataset for grounding relative directions via multi-task learning. arXiv preprint orXiv:2205.0267/ 2022.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. In Advances in Neural Information Processing Systems, 2019.
Paul Pu Liang, Amir Zadeh, and Louis Louis-Philippe Morency. Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions. arXiv preprint arXiv:2209.0344 03330, 2022.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
12

---

Published as a conference paper at ICLR 2024
Ulf Liszkoskki, Malinda Carpenter, Anne Henning, Tricia Striano, and Michael Tomasello. Twelve- month-olds point to share attention and interest. Developmental science, 7(3):297-307, 2004.
Dianbo Liu, Alex Lamb, Kenji Kawaguchi, Anirudh Goyal, Chen Sun, Michael Mozer, and Yoshua Bengio. Eiscrete-valuee neural communication in structured architectures enhances generaliza- tion. 2021.
Fangyu Liu, Guy Edward Toh Emerson, and Nigel Collier. Visual spatial reasoning. ArXiv, 2022.
Runtao Liu, Chenxi Liu, Yutong Bai, and Alan L Yuille. Clevr-reft: with referring expressions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4185-4194, 2019.
Diagnosing visual reasoning
Xiang Long, Chuang Gan, Gerard De Melo, Xiao Liu, Yandong Li, Fu Li, and Shilei Wen. Mul- timodal keyless attention fusion for video classification. In Proceedings of the AAAI Conference Artificial Intelligence, 2018.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2017.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguis- tic representations for ssioo-and--anguaal tasks. In Advances in Neural Information Processing Systems, 2019.
Haonan Luo, Guosheng Lin, Zichuan Liu, Fayao Liu, Zhenmin Tang, and Yazhou Yao. Segeqa: Video segmentation based visual attention for embodied question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9667-9676, 2019.
Mateusz Malinowski, Marcus Rohrbach, and Mario Fritz. Ask your neurons: A deep learning approach to visual question answering. International Journal of Computer Vision, 125(1):110- 135, 2017.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11-20, 2016. doi: 10.1 1..110//CPR.2016.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200-2209, 2021.
David McNeill. How language began: Gesture and speech in human evolution. Cambridge Univer- sity Press, 2012.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pp. 947-952. IEEE, 2019.
Ferda Ofli, Rizwan Chaudhry, Gregorij Kurillo, René Vidal, and Ruzena Bajcsy, Berkeley mhad: A comprehensive multimodal human action database. In WACV, pp. 53-60. IEEE, 2013.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In Advances Neural Information Processing Systems, pp. 8024-8035, 2019.
Jonas Pfeiffer. Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing: System Demonstrations, pp. 46-54, 2020a.
Jonas Pfeiffer, Ivan Vulié, Iryna Gurevych, and Sebastian Ruder. Mad-x: An adapter- based frame- work for multi-task cross-lingual transfer. arXiv preprint arXiv:2005.0000 00052, 2020b.
13

---

Published as a conference paper at ICLR 2024
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapter- fusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 487-503, 2021.
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phras correspondences for richer smage-to-sentence models. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2641-2649, 2015. doi: 10.1 109//CCCV.0155.03.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Mengye Ren, Ryan Kiros, and Richard Zemel. Exploring models and data for image question answering, Advances in neural information processing systems, 28, 2015.
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. Adapterdrop: On the efficiency of adapters in transformers. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 7930-7946, 2021.
Michael Scaife and Jerome Bruner. The capacity for joint visual attention in the infant. Nature, 253(5489):265-266 1975.
Boris Schauerte and Gernot A. Fink. Focusing computational visual attention in multi-modal human- robot interaction. International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction, ICMI-MLMI '10, New York, NY, USA, 2010. Association for Computing Machinery. ISBN 9781450304146.
Boris Schauerte, Jan Richarz, and Gernot A. Fink. Saliency-based identification and recognition of pointed-at objects. In 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4638 4643, 2010. doi: 10. (00//ROS..01055449000
Dadhichi Shukla, Ozgur Erkent, and Justus Piater. Probabilistic detection of pointing directions for human-robot interaction. In 2015 International Conference on Digital Image Computing: Techniques and Applications (DICTA), pp. 1-8, 2015. doi: 10.1109/DICTA, 2015.7371299
Dadhichi Shukla, Özgür Erkent, and Justus Piater. A multi-view hand gesture rgb-d dataset for human-robot interaction scenarios. In 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pp. 1084-1091, 2016. doi: 10.1109/OMMA.
2016.7745243.
Hao Tan and Mohit Bansal. LXMERT: Learning cross-modality encoder representations from transformers. Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5100-5111, Hong Kong, China, November 2019. Association for Com- putational Linguistics. doi: 10. 00.18653/1/DD1-11 15114444. URL htt.ps::/acaantholl.o...o///
D19-1514
Sinan Tan, Weilai Xiang. Huaping Liu, Di Guo. and Fuchun Sun. Multi-agent embodied question answering in interactive environments. In European Conference on Computer Vision, pp. 663- 678. Springer, 2020.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Jette Viethen and Robert Dale. The use of spatial relations in referring expression generation. In Pro- ceedings of the Fifth International Natural Language Generation Conference, pp. 59-67, 2008
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In European Conference on Computer Vision (ECCV), 2016.
14

---

Published as a conference paper at ICLR 2024
Peng Wang, Qi Wu, Chunhua Shen, Anton van den Hengel, and Anthony Dick. Explicit knowledge- based reasoning for visual question answering. arXiv preprint arXiv:1511.02570, 2015.
Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. Fvqa: Fact-based visual question answering. IEEE transactions on pattern analysis and machine intelligence, 40
(10) 2017.
Kaiyu Yang. Olga Russakovsky, and Jia Deng. Spatialsense: An adversarially crowdsourced bench- mark for spatial relation recognition. In Proceedings of the IEEE/CVV International Conference Computer Vision, pp. 2051-2060, 2019a.
Sibei Yang. Guanbin Li, and Yizhou Yu. Cross-modal relationship inference for grounding refer- ring expressions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4145 4154, 2019b.
Licheng Yu, Eunbyung Park, Alexander C Berg, and Tamara Berg. Visual madlibs: Fill in the blank image generation and question answering. arXiv preprint arXiv:1506.0025 2015.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expressions. Computer Vision ECCV 2016, pp. 69-85, Cham, 2016. Springer International Publishing.
Licheng Yu, Xinlei Chen, Georgia Gkioxari, Mohit Bansal, Tamara L Berg, and Dhruv Batra. Multi- target embodied question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6309-6318, 2019.
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answer- ing in images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4995-5004, 2016.
15

---

Published as a conference paper at ICLR 2024
TECHNICAL APPENDIX
A RESOURCES
The EQA-MX dataset, souree code for the CAESAR simulator with our modifications, benchmark learning models, trained model checkpoints, and docker for computing environment can be accessed through the following links. will publicly release these resources with the camera-ready version of our paper. For double-blind reviewing purposes, we are sharing these resources anonymously with the reviewers:
datasset Source code of VQ-Fusion, benchmark models, dataset processing, and dataset anal- yses:
EQA-MX dataset (162 GB):
bttps::/bbit.ly/eeaa-1
Source code the EQA simulator with new extensions (3.1 GB): https:
se/eqaasiimllee
Trained model checkpoints of ViLT with VQ-Fusion for existence prediction task (1.3 GB):
Docker for training models (8.59 GB): We built a docker to facilitate easy reproducing of our experimental settings and training environment. We cannot currently share the docker hub link to maintain anonymity. We plan to share that docker link upon publication of the paper. For this reason, we are sharing the singularity container built from the same docker we used for our experimentation: https: //bittbittt ..e..lly/.o 1y/multimodal imodal docker
B BROADER IMPACT
Our dataset contains rich annotations of visual scenes, such object locations, spatial relations, and multiple visual and verbal perspectives. These can be used to design new tasks to robustly comprehend embodied interactions. Moreover, our EQA-MX dataset can be used for diverse tasks in embodied settings, such as scene segmentation and conversational human-AI interactions with multimodal expressions. Additionally, our dataset can be used develop and evaluate models that can be transferred to robots for comprehending embodied human instructions in real-world settings. Lastly, our experimental analysis provides valuable insights that can be used designing robust VI. models, such as using similar embedding structures for fusing continuous and discrete representations leading to performance improvements.
C ADDITIONAL EXPERIMENTAL ANALYSES
C.I IMPACT OF MULTIPLE VISUAL PERSPECTIVES AND MODALITIES
In real-world settings, robots are typically equipped with multiple camera views. Several studies have emphasized the significance of multiview data in accurately comprehending human actions and instructions(Kong et al., 2019 Islam Iqbal 2022). To further validate the importance of multimodal data (nonverbal gestures captured through visual views and verbal utterances) in under- standing embodied question answering (EQA) tasks, we conducted extensive ablation studies with varying visual views (ego, exo and top) and verbal utterances (verbal utterance templates described Table TTp.
In the first setting, we used only verbal utterances for all eight EQA tasks (Table Top). We used BERT (Devlin et al.) 2018) for learning the EQA tasks. The results suggest models using only verbal modality can not effectively learn these EQA tasks. Conversely, if we utilized both verbal and nonverbal data, then the performance of these EQA tasks improved (Table This degraded performance using only verbal data emphasizes the importance of utilizing both verbal and nonverbal data modalities for appropriately learning EQA tasks. Additionally, it also indicates that our proposed FQA-MX dataset less biased towards verbal data for comprehending EQA tasks.
16

---

Published as a conference paper at ICLR 2024
Table 6: We trained CLIP models with VQ-Fusion using different combinations of modalities on the 8 tasks described in Figure 2 in the paper. Top Table: only verbal questions. Bottom Table: different visual modalities and verbal questions. The results suggest that multimodal models outperform those using only verbal data (Top Table), Additionally, training models with multiview data leads to robust performance, while using a subset of views results in performance degradation if the views change during testing (Bottom Table). Existence Prediction (EP), Object Grounding (OG), Perspective- Aware Object Grounding (POG), Object Counting (OC), Object Attribute Query (OAQ), Object Attribute Compare (OAC), Perspective Grounding (PG), Relation Grounding (RG).
Only
Verbal
EP
OG
POG
OC
OAQ
OAC
PG
RG
40.64
8.90
45.46
7.45
7.69
29.49
45,23
44.82
Train
Test
EP
OG
POG
OC
OAQ
OAC
PG
RG
Ego
Ego
53.86
59.92
70.98
10.60
68.56
61.86
64.41
87.54
Ego
Exo
52.61
17.28
62.45
8.96
15.06
56.62
63.39
82.33
Εxo
Exo
53.67
39.46
69.96
11.24
56.76
60.20
66.39
88.58
Exo
Ego
52.84
21.39
69.70
10.78
25.03
58.68
64.49
88.20
ALL
ALL
54.72
65.49
82.70
13.14
74.32
70.59
66.99
89.93
ALL
Ego
54.32
60.63
82.31
12.22
69.84
60.89
66.71
89.03
ALL
Exo
54.17
59.14
78.02
12.55
61.71
62.25
66.53
89.26
In the second setting, we used verbal utterances and nonverbal gestures to learn EQA tasks. We varied the visual perspectives during training and testing through the use of different camera views (ego, exo, and top) to capture the nonverbal interactions. We used CLIP model to learn EQA tasks involving verbal utterances and visual views. The results suggest that models trained using multiple visual perspectives perform better than models trained using a single visual perspective (Table Bottom). The reasoning behind this performance improvement is that models using multiple vi- sual views can learn generalized multiview representations, which can improve the performance at inference time when visual views are varied.
C.2 COMPARISON OF SINGLE AND MULTITASK MODELS
We evaluated the impact of learning multiple tasks in visual-language model. We conducted this experimental analysis in two settings. In both settings, we used verbal utterances and multiple visual modalities to learn EQA tasks. In the first setting, we trained CLIP models for each EQA task separately. In the second setting, we trained CLIP models for a subset of EQA tasks. Finally, we used the extracted representation in each EQA task head, where these task heads are designed using an MLP.
The results in Table 7 suggest that the performance of models learning multiple tasks degrades com- pared to the models learning these tasks separately. As these tasks have different characteristics, learning these tasks together can compete the representation learning space and degrades these tasks' performance. For example, training the CLIP model for the Existence Prediction (EP) and Object Grounding (OG) tasks together degrades the Object Grounding task performance to 40.76% compared to accuracy of 65.49% for a separately trained CLIP model for OG task. Previous studies have observed similar performance degradation when learning multiple competing tasks. The primary reason behind the performance degradation is that the competing tasks have conflicting gradients among different tasks that introduce negative knowledge transfer and thus degrade these tasks' performance. Thus, exciting future research direction would be design novel multi- task model architectures and training approaches where training on multiple tasks using multiple modalities improves the performance of every task in shared model.
C.3 GENERALIZABILITY OF VQ-FUSION
To evaluate the generalizability of VQ-Fusion for another task involving multimodal representation learning, incorporate VQ-Fusion in existing multimodal learning model (HAMLET (Islam & Iqbal, (020)) for human activity recognition tasks with multimodal sensor data (RGB videos, accel- eration, gyroscope, and orientation). We have evaluated this modal the MMAct dataset (Kong
17

---

Published as a conference paper at ICLR 2024
Table We train CLIP models with VQ-Fusion in single task (ST) and multitask (MT) settings. We reported accuracy of these tasks. Tasks trained in an MT setting are grouped together. The results suggest that the performance of these models with multiple tasks degrades compared models learning these tasks separately. Existence Prediction (EP), Object Grounding (OG), Perspective- Aware Object Grounding (POG), Object Counting (OC), Object Attribute Query (OAQ), Object Attribute Compare (OAC), Perspective Grounding (PG), Relation Grounding (RG).
EP
OG
POG
OC
OAQ
OAC
PG
RG
ST
54.72
65.49
82.70
13.14
74.32
70.59
66.99
89.93
EP
OG
EP
POG
EP
PG 49,86
MT
53.25
40.76
52.68
73.90
52.62
EP
OAQ
OG
EP
PG
OAQ
PG
EQ
OAQ
MT
54.24
68.70
55.56
53.17
66.92
66.61
66.80
53.26
69.01
et al. 2019). The MMAct dataset comprises 37 common daily life activities, each performed by 20 individuals and repeated five times. The dataset includes seven modalities, ranging from RGB data to acceleration and gyroscope measurements. Our experiments focused on utilizing two available viewpoints of RGB videos, well acceleration, gyroscope, and orientation data. Notably, the MMAct dataset also includes visually occluded data samples, providing an opportunity to evalu- the effectiveness of multimodal learning approaches in extracting complementary features for activity recognition.
In our experimental analyses, we adhered the original session-based evaluation settings and ported the Fl-score. We have used eight codebooks to discretize the multimodal representations. The results indicated that the HAMLET model, which utilizes our proposed VQ-Fusion approach, outperformed all existing state-of-the-art multimodal human activity recognition (HAR) approaches session-based evaluation settings on the MMAct dataset (Table &). Specifically, the inclusion of VQ-Fusion enabled HAMLET to improve its F1-score by 4.2%, resulting in the highest reported Fl-score of 87 69. (Table 8). These findings suggest that VQ-Fusion can effectively aid exist- ing models in extracting salient multimodal representations, thereby enhancing the performance of downstream tasks the field of HAR.
D TRAINING ENVIRONMENT
We developed all the models using the Pytorch (version: 1.12.1ccul13) (Paszke et al. 2019) and Sytorch-Lightning (version: 1.7.1) (Falcon 2019) deep learning frameworks. We also used Hug-
Table 8: Cross-session performance comparison (F1-Score) of multimodal learning methods on MMAct dataset
Method
FI-Score (%)
SVM+HOG (Ofli et al. 2013)
46.52
TSN (RGB) (Wang al. 2016)
69.20
TSN (Optical-Flow Wang et al.
2016)
72.57
MMAD Kong et al. 2019)
74.58
TSN (Fusion) Wang al.
2016
77.09
MMAD (Fusion) (Kong et al. 2019
78.82
Keyless (Long al. 2018
81.11
MAMLETT(slam) & Igbal
2020)
83.89
MuMu (Islam Iqbal
2022)
87.50
VQ-Fusion
87.69
(Q-FusioonMMuMu)
87.83
18

---

Published as a conference paper at ICLR 2024
Table Comparison of the QA datasets. Existing VQA and EQA datasets do not contain nonverbal human gestures (NV), multiple verbal perspectives (MV), contrastive (C) and ambiguous (A) data samples. Embodied (E) interactions refer to humans interacting with multimodal expressions. Embodied interactions refer to an agent navigating in an environment. V: Verbal and MT: Multitasks.
Views
Datasets
VN
E
EQA
MT
MV
C
A
Exo
Ego
Top
PointAt (Schauerte et al.
2010)
X
X
X
X
X X
ReferAt Schauerte & Fink 2010)
/
V
X
X
/
X
X
X
IPO (Shukla al. 2015)
X
V
X
X
/
X
X
X X
IMHF (Shukla et al. (2016)
X
V
X
X
X
V
X
X
X
Reflt (Kazemzadch al. 2014)
X
X
X
X
V
X
X
X X
RefCOCO Yu et al. (0166)
X X
X
X
J
X
X
X
RefCOCO+ Yuu a.. 2016)
X
X
X
J
X
X
X
X
RefCOCOg (Mao al. 20166)
X
X
X
X
X
Flickr30k (Plummer et al. 2015
X
X
X
X
X
X
Guess W hat?? (De Vries et al. 2017)
X
X
X
X
X
X
X
Cops-Ref (Chen et al. 2020b)
X
X
X
X
X
X
CLEVR- Rev+ (Liu et al. 2019)
X
J
X
X
DAQUAR (Malinowski et al. 2017
X
Y
X
X
FM-IQA (Gao al.] 2015)
X
X
J
X
Visual Madlibs Yu et al. 2015
X
X
J
X
Visual Genome (Krishna al. 2017)
X
X
X
J
X
X
X
DVQA (Kafle et al. 2018)
X
X
X
J
X
X
X
VQA (COCO) (Antol al.
2015
X
X
X
J
X
J
X
X
VQA (Abs.) (Antol et al. 2015
X
X
X
J
X
X
Visual 7W (Zhu et al. 2016
X
X
X
J
X
J
KB-VQA (Wang et al. 2015
X
X
J
FBQA WWan et al. (017)
X
X
J
VQA-MED (Hasan al. 2018
X
J
Doe VQA Mathew et al. 2021
X
X
X
YouReflt Chen et al. 2021
GRiD-3D Lee et al.
2022
X X
*
EQA (Das et al.
201 8a)
MT-EQA Das al.
(018a)
CAESAR Islam et al. (022b)
CAESAR XL (Islam al.
2022b)
BQA-MX
/
gingFace library (version: 4.21.1) for pre-trained models (BERT Devlin al. 2018). ViT Doso-
vitskiy et al. 2020), VisualBERT Li et al. 2019), Dual Encoder ViLT Kim et al. 202 and (Radfodd et al. 2021)) For the Dual-Encode and CLIP models, we used an embedding size of 512, and for VisualBERT and ViLT, we used an embedding size of 768. We train models using the Adam optimizer with weight decay regularization (Loshchilov Hutter (017) and cosine anneal- ing warm restarts an initial learning rate: 3e-1 cycle length (7p): 4, and cycle multiplier (Tmwtt)
CLIP (Radford
We used batch size 128 and trained models for 8 epochs. We used the same fixed random seed (33) for all the experiments to ensure reproducibility. Lastly, all models are trained in distributed GPU clusters, where each node contains 8 A.00 GPUs.
docbbrtt
bert
19

---

Published as a conference paper at ICLR 2024
No. of
No. of
Object Categories
Avg.
Datasets
Images
Samples
Words"
PointAt Schauerte et al.
2010)
220
220
28
ReferAt SChauerte Fink
2010
242
242
28
IPO (hukkaa et al. 2015)
278
278
10
IMH (Shukla al. 201 6)
1716
1716
28
Reflt (Kazemzadeh et al.
20144
19,894
130,525
238
3.61
RefCOCO Yu al. 2016))
19,994
142,209
80
3.61
RefCOCO+ Yu et al.
2016)
19,992
141,564
80
3.53
RefCOCOg (Mo al. 20T6)
26,711
104,560
8.43
Flickr30k Pluummer al.
2015
31,783
158,280
44,518
Whats (De Vries et al. 2017
66,537
155,280
Cops-Ref Ref (Chen et al. 20206)
75,299
148,712
508
14.40
CLEVR- ReIR (Liu et al. 2019
99,992
998,743
3
22.40
DAQUAR (Malnowski et al. 2017
1449
124,68
37
11.5
FM-IQA (Gao et al. 2015)
157,392
316,193
7.38
Visual Madlibs Yu al. 2015)
107,38
360,001
6.9
Visual Genome Krishna et al.
2017
108,000
1,445,332
37
5.7
DVQA (Kafle al.
2018)
300.000
3,487,194
VQA (COCO) (Antol et al. 2015
204,721
614,163
80
6.2
VQA (Abs.) (Antol al. 2015
50,000
150,000
100
6.2
Visual 7W (Zhu et al. 2016)
47,300
327,939
36,579
6,9
KB- VQA [Wang et al. 2015)
700
5826
23
FBQA (Wang et al. 2017
2190
5826
32
9.5
VQA-MED (Hasan et al. 2018
2866
6413
Doc (Mathew ct
2021
12,767
50,000
YouReflt Chen et al. 2021
497,348
4,195
395
3.73
GRiD-3D Lee
2022
8,000
445,000
28
EQA Das et al. 20 88))
5,000
5,000
50
MT-EQA (Das et al.
2018a) 8a
19,287
19,287
61
CAESAR-L Isslam et 2022b
11,617,626
124,412
61
5.56
CAESAR XL (Islam et al.
2022b)
841.620
1,367,305
80
5.32
EQA-MX
750,849
8,243,893
52
11.45
Table 10: Comparison of the QA datasets. EQA-MX has more samples than all other previous QA datasets. "Average number of words in questions.
Ego-view
Exo-view
Top-view
Figure 5: Sample data demonstrating the shelf environment vs. the table environment
20

---

Published as a conference paper at ICLR 2024
Table 11:
Templates for all 8 tasks in the EQA-MX dataset, The answers for these templates are based on the environment in the first row of 5
Existence Prediction
Template
Task Example
Is there any/a/an <object name?> in the scene?
Is there cucumber in the scene?
Is there any/a/an <object color> <object name?> in the scene?
Is there a green cucumber in the scene?
Object Grounding
Template
Task Example
What the of that object/thing?
What is the name of that object?
What is the name of the <object color> object/thing?
What is the name of that yellow thing?
What is the name of that <object absolute location
What is the name of that right yellow object?
<object color> object/thing?
What is the name of that <selected object absolute location
<selected object color> object/thing to the <'spatial relation of the <rrelational object absolute location>>
What is the name of that right yellow object to the right of the yellow cheese?
<relational object color> <relational object name?>?
Perspective-Aware Object Grounding
Template
Task Example
Considering the <observer's s/speaker's> perspective, what is the name of that object/thing?
Considering the observer's perspective, what is the name of that object?
Considering the <observer's s/speaker's> s>> perspective, what the name the <object color> object/thing?
Considering the observer's perspective, what
is the name of that yellow thing?
Considering the <observer' <observer's/speaker s> perspective, what is the name of that <object absolute location,> <object color>> object/thing?
Considering the speaker's perspective, what the name of that right yellow object?
Object Counting
Template
Task Example
How many objects are <sspatial relation> of the object/thing?
How many objects are above the object?
How many objects <spatial relation>> of the <object color> object/thing?
How many objects are left of the yellow thing?
Object Attribute Query
Template
Task Example
What it the color of that object/thing?
What it the color of that object/thing?
What is the color of the <object name>?
What is the color of the hand soap dispenser?
Object Attribute Compare
Template
Task Examplc
the color of that object/thing
Is the color of that thing
the same color as the <rrelational object name>>?
the same color as the cheese?
the color of that <selected object name>>
Is the color of that hand soap dispenser
the same color as the <relational object name>?
the same color as the soda bottle?
Perspective Grounding
Template
Task Example
<Referrngg expressions using the templates from CAESARS.
The hand soap dispenser above the soda bottle.
From which perspective is the object described?
From which perspective is the object described?
Relation Grounding
Template
Task Example
<Referring expressions using templates from CAESAR>, is the object referred to appropriately?
The hand soap dispenser above the cucumber, is the object referred to appropriately?
Considering the observer's 's perspective, <Referring expressions using the templates from CAESAR>, the object referred to appropriately?
Considering the observer's 's perspective, the hand soap dispenser below the cucumber,
the object referred to appropriately? Considering the observer's 's perspoctive, the hand soap next to the coffee maker. is the object referred to appropriately?
Considering the speaker's 's perspective, <Referring expressions using the templates from CAESAR>, is the object referred to appropriately?
E EMBODIED QUESTION ANSWERING TASK AND DATASET ADDITIONAL INFORMATION
We include additional information the EQA-MX dataset compared to previous EQA datasets in
Tables 9 and 10
21

---

Published as a conference paper at ICLR 2024
All 8 EQA tasks created described in the paper catered towards specific scenarios that would benefit models in real-world human interaction scenarios.
Existence Prediction (EP): Naturally, humans able determine what objects present in given scene. scenarios where humans are interacting and an actor mistakenly references an object not in the scene, this allows observers to request more information. Created to mimic this situation, the existence prediction task involves determining whether the scene contains a particular object with some specific attributes, such color.
Object Grounding (OG): Understanding which objects a human refers to using verbal and non- verbal cues is key successful human-Al interaction. A model successfully able ground objects has use-cases such as assisting surgeons during procedure by handing surgeons the correct tools. Thus, we design the object grounding task around this scenario, where models must identify the name of the object being referred to by verbal and nonverbal expressions.
Perspective-Aware Object Grounding (POG): Similar the object grounding task, the Perspective-Awarc Object Grounding involves determining which object is being referred to, but this task includes the verbal perspective (either ego, exo, or neutral). Although real-world human- Al interactions will not always contain the perspective of given relation, including the perspective allows for to determine whether or not understanding perspective can help grounding objects.
Yes
Yes
Yes
50.0%
50.0%
50.0%
50.0%
50.0%
50.0%
No
No
No
(a) Existence Prediction Task (b) Object Attribute Compare Task (b) Relation Grounding Task
Figure 6: Distributions of task outputs in the existence prediction (EP), object attribute compare
(OAC), and relation grounding (RG) tasks. All these tasks have balanced binary outputs
hair
orange
loaf
corner
bread
bottom
Dink
cyan
small
perspectio
cup
Loo
mam
Jeer
blue
serseettive
green
name
red
corn
iron
large
six
color
Veft
diie
front
bag
hat
right
back
referred hanger
sale
yellow
book
theting
(a) Wordeloud all verbal expr.
(b) Dist. of outputs for OG task
(c) Dist. of outputs for POG task
Figure 7: A verbal expression Wordcloud for the EQA-MX dataset, as well as the output distribution for the object grounding (OG) and perspective-awar object grounding (POG) tasks. In the Word- cloud the size of words represents the frequencies that they occur in the verbal utterances. Therefore, the most frequent words describe general properties of objects or are general words inside questions such color, perspective, and spatial relations/locations In the diagrams for object frequencies for the object grounding and perspective-aware object grounding tasks, the most referred objects all have the same frequencies (these tasks have the same object distributions). Expr. expression, Dist. distribution.
Object Counting (OC): As understanding what object human is referring to in scene involves interpretation the different number of objects inside that scene, understanding the number of objects in scene can serve auxiliary task for the object grounding task. If models are able
22

---

Published as a conference paper at ICLR 2024
to create salient multimodal representations to attend to all the objects in given scene, is is likely they will be able to ground particular objects better. Thus, in the object counting task the number of objects in scene is asked based different spatial relations.
Object Attribute Query (OAQ): It is often important in human -umaan interactions to identify particular attributes of objects. Additionally, this information can be used auxiliary information for tasks such as the Object Grounding task, where the goal to identify objects. We design the Object Attribute Query task around this particular situation, where the color of given object is queried for.
Object Attribute Compare (OAC): Humans often exchange information throughout conversations through the use of comparison of different object attributes. This exchange of information can assist in understanding the different objects an actor referring to. Thus, we design the object attribute compare task, where the attributes of two different objects in the scene are compared.
Perspective Grounding (PG): Understanding human verbal perspective integral to successful human-Al communication, as humans interchangeably describe objects from their perspective as well as the perspective of others. We simulate this in the perspective grounding task using three different perspectives neutral, egocentric (speaker), and exocentric (observer).
Relation Grounding (RG): As described Islam et al. (002b), the relation grounding task in- volves determining whether the supplied verbal and nonverbal signals align with respect to describ- ing the same object. Understanding whether or not a human is accurately verbally and nonverbally referring to object can enable the identification of human mistakes. We add complexity to this task through the variation verbal perspective in the question.
One
Below
Light Green
Navy Blue
Grange
16.5%
Two
15.3%
9.3%
In front of
Above
15.6%
Zero
16.6%
16.7%
16.0%
7.4%
13.2%
Nine
16.8%
16.8%
Three
15.5%
5.0%
Eight
7.5%
Behind
Right
6.8%
Seven
Lavender
16.6%
12.3%
7.5%
Six
Cyan
Left of
Four
Five
Pink
Green
(a) OC task spatial relations
(b) Dist. of OC task output
(c) Dist. of OAC task output
Figure 8: Distribution of task outputs in the object counting and object attribute compare tasks. Both distributions are not completely even due to different observed scene probabilities. For the object counting (OC) task, lower numbers have higher probabilities of occurring due the number of objects in the scene ranging from 4 - 10, hence the imbalance in distributions. Similarly, in the object attribute compare task different object colors are queried for, and since the colors of objects is not completely balanced, the task distribution is imbalanced. Dist. Distribution.
E. EQA TASK TEMPLATES
In this work we presented 8 EQA tasks. Each of these tasks has multiple sub-templates, which we present in more detail in Table I1 Each sub-template has multiple degrees of freedom from which to vary, ensuring generated embodied questions are diverse. For example, since most sub-templates use the absolute location of object, this absolute location can often times be described from either the observer or speaker perspective.
E.2 NEW ENVIRONMENTS IN EQA-MX
To increase dataset generalizability. we have added a shelf environment into the CAESAR simulator, and thus into the EQA-MX dataset. We visualize the three views (ego, exo, and top) for this and the table environment in Fig. 5 Because the exo and ego views in the table environment are on different sides of the table, the verbal perspectives differ. However, in the shelf environment, the and ego views are aligned meaning the verbal perspective is aligned. We created this environment this way ensure models have differing situations with regards views and perspective. Additionally,
23

---

Published as a conference paper at ICLR 2024
Ego-view
Exo-view
Top-view
Ego-view
Exo-view
Top-view
Bottorn
Front
Back
(a) Object locations for "Left" and "Right" spatial locations,
(b) Object locations "Front" and "Back" spatial locations.
Ego-view
Exo-view
Top hop-iew
Ego-view
Exo-view
Top-view
Center
Not Center
Front
Front Right
Back
Back Right
(c) Object locations "Center" spatial locations (Table-top).
(d) Object locations for "Corner" spatial location.
Object locations visualized for different spatial relations/locations across the EQA-MX dataset. The object locations are not easily separable based spatial relations/locations that vary based perspectives. (a & b) demonstrates how the shelf environment has more non-separable locations/relations due the fact that verbal perspective in the shelf environment does vary based on visual perspective. с is generally linearly separable, expected, as the center of given scene is objective. d demonstrates how opposing corners (i.e. front left and back right) are non- separable due to varying based verbal perspectives).
Figure 9:
since the shelf has objects below/on top one another, adds diversity with respect spatial relations/lccation, ensuring models understanding these relations/locations in all dimensions.
F ADDITIONAL DATASET ANALYSES
We have thoroughly analyzed the different aspects of data samples in our dataset, EQA-MX. We visualize the output distribution for all EQA tasks, well as the object locations with respect to different spatial relations/locations, and the most frequent words found in our dataset.
F.1 TASK OUTPUT DISTRIBUTIONS
As shown in Figs. we balance outputs of our task distributions where possible in order to ensure the EQA-MX dataset not biased. For the OG and POG tasks, the output distribution of all 52 categories balanced to ensure models do not bias a particular object. Additionally, binary tasks (EP, OAC, and PG) contain 50/50 split between yes and no answers. Because the CAESAR simulator randomly generates scenes populated with objects, the OC and OAC tasks do have even task distributions. This can be explained by these tasks involving observed character- istics in scenes where some characteristics are more common than others. For example, since the max number of objects that can be generated in a scene is 10, the probability of an object have objects to the left of it is much lower than the probability of an object having 2 objects to the left of it. Similarly, certain colors are more common objects inside the CAESAR simulator. These distributions made more apparent in Fig. (we report macro accuracy for models trained these tasks).
Fig. 5 all
F.2 OBJECT LOCATIONS ANALYSES
We visualize object locations inside the EQA-MX dataset to show how different spatial relations have/don't have bias (Fig. 99. Particularly. since one of our contributions is the creation of the shelf environment, we show how since visual views are aligned certain visual cues have bias.
24


Reviews:
[{"review_id": "LHv23mVzi5", "reviewer": "Reviewer_7hsY", "summary": "This paper proposes a new dataset for the task of embodied question answering using multiple expressions (gaze and pointing) and textual questions called EQA-MX. It further provides a simulator for generating a large dataset containing simultaneous multimodal expressions (gaze and gesture) and verbal questions from multiple viewpoints. It defines 8 different EQA tasks to evaluate the performance of verbal and non-verbal gestures at embodied question answering. Additionally, it proposes a vector quantization (VQ)-based method to quantize dense visual information from various visual views with discrete textural information for EQA. The paper shows the effectiveness of using the proposed VQ model at combining information from multiple views and the utility of using gaze and gesture information at the task of question answering through several empirical tests.", "questions": "I would like to hear from the authors about the questions/concerns that I raised in the \"Weaknesses\" section of my review.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper addresses a new and interesting problem of embodied question answering which is key to enabling human-robot interactions in their natural environments. Inspired by the observation that communication between humans is a combination of both verbal and non-verbal gestures (gaze and pointing), for example, it examines for the first time whether incorporating such multiple cues can be useful for embodies question answering tasks well. Lastly the paper also proposes a new and effective approach to fusing visual information from multiple different viewpoints/perspectives into a single semantic visual code using VQ and combines it with verbal information for various EQA tasks.\n\nThe work is of broad and significant impact. All aspects of the proposed methodology are technically sound.\n\nThe paper is extremely well presented, written and organized.", "weaknesses": "1. The only criticism/question that I have is whether the proposed methodology and experimental conclusions would generalize to real-world datasets? Synthetic datasets contain a significant domain gap to reality. Real scenes have more noise, clutter, lighting variations, etc and experiments purely in simulations and the conclusions obtained from them cannot be guaranteed to hold in the real world as well.  This work would have been a lot stronger and more impactful had the authors also evaluated their proposed methodology on real world datasets/scenarios and provides results of it.\n\n2. While the authors show the value of VQ in Table 3, the value of using multiple views has not been ablated. How much does having visual information from multiple views help in EQA? It seems obvious that it should, especially if some views are likely to have occlusions and ambiguities, while other are not, but it would have been nice to see its effect quantitatively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LtJd0HxFJc", "reviewer": "Reviewer_9x3E", "summary": "The paper introduces Embodied Question Answering (EQA) tasks that require agents to answer questions using both verbal and nonverbal gestures in an embodied setting. The authors present a large-scale dataset, EQA-MX, that contains diverse multimodal expressions from various verbal and visual perspectives. The work also introduces a Vector Quantization-based multimodal fusion approach (VQ-Fusion), which discretizes visual representations using shared codebooks before fusing them with discrete verbal representations. Additionally, the paper explores the importance of nonverbal gestures (gaze and pointing)  in learning EQA tasks.", "questions": "In terms of the dataset, which is one of the main contributions of this work, there are aspects that need further explanations:\n\n- Figure 3(a) is also difficult to read due to resolution issues and the same colors used for all lines. What is the y-axis, i.e., how are datapoints ordered?\n\n- How many of the tasks involve non-verbal gestures? From Fig. 2 and Table 6, it seems that {OG, POG, OAQ, and OAC} (4/8) is the subset of tasks that involves gestures, how about the rest? Do OAQ, POG, and OAC focus on color only? Object Attribute Compare in Table 6 mentions \"<Referring expressions using the templates from CAESAR>. From which perspective is the object described?\" which seems very different from the example in Fig. 2, please explain.\n\n-  To my understanding, experiments in the ablation study on Table 4 (only gaze, only gestures, and without gaze and gestures) involve different splits of the EQA-MX dataset? It seems that it would make more sense to keep the splits the same and just remove the non-verbal information (gaze, gesture, etc). Keeping the splits consistent ensures that the experimental conditions are comparable, and any performance differences can be more directly attributed to the presence or absence of non-verbal information.\n\nA couple of minor questions:\n- In supplementary section 3.2, could you provide references for the sentence \"Previous studies have observed similar performance degradation when learning multiple competing tasks\"?\n- In supplementary section 3.3, what is the performance of the combination of VQ-fusion on MuMu (SoTA model)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "-  The paper presents eight EQA tasks designed for understanding questions using multimodal expressions (verbal utterances and nonverbal gestures) in embodied settings. \n- The introduction of the vector quantization-based fusion model addresses the structural mismatch between the discrete verbal representations and the continuous visual representations from the three available visual views. Experimental results show that incorporating VQ-Fusion in the existing baseline models improves their performance across EQA tasks. \n- The paper consists of extensive ablation studies on the number of codebooks, the use of multiple views, the generalizability of VQ-fusion, learning single vs. multiple tasks, etc.", "weaknesses": "- The paper focuses on a synthesized dataset, which, while important for controlled experiments, may not fully capture the complexities and nuances of real-world interactions. This concern is partially due to the way view perspectives are included in the data. \"The blender to the right of the book. From which perspective is the object described?\" may not be something you'd commonly encounter in everyday (embodied) conversations/questions. Same for the example shown in Fig. 2 for RG.\n- Table 1 is a bit unclear, e.g., why is EQA not available for all datasets, when some of the datasets referred to in the table correspond embodied QA benchmarks (EQA, MT-EQA)? Also, to the best of my knowledge, some of the datasets have egocentric views and not exocentric ones. \n- There is a body of work in pointing gesture recognition - most involving human communication in interactive embodied task-completion environments [2,3,4]. Similarly, the exists work that focuses on view perspectives [5]. It would be good to include and differentiate this work from the existing literature.\n- While the reviewer appreciates the performance improvements of VQ-Fusion on 4 baselines, it would be beneficial to include more advanced baselines, such as a Multimodal Transformer model that extracts visual representations across all visual views and the verbal representations together, as well as more recent Multimodal LLMs (Otter, MiniGPT, UniLM, GPT4, InstructBLIP, LLaVA, etc.) that potentially can offer competitive fusion capabilities for these tasks.\n- Based on Table 5 and Table 3, results in Table 3 for VQ-Fusion models are reported with varying codebooks, tuned for each task. This can improve performance, but it also raises questions about the generalizability of the approach. In some cases, the worst-case setting of the number of codebooks (e.g., 4 codebooks for EP or PG) can hurt the performance of the baseline CLIP model, which might impact the model's ability to perform well on new unseen tasks.\n\nMinor edits:\n- If the paper focuses on embodied AI, it would make sense to categorize the works into embodied and not embodied and then embodied multimodal prompting and language-guided embodied. VIMA [1], PATRON, and CAESAR could also be moved to Table 1 from Table 4 as similar embodied multimodal prompting tasks, especially since the Table 1 format and most of the capabilities originate from the use of CAESAR as the underlying simulator. \n\n[1] Jiang, Yunfan, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. \"VIMA: General Robot Manipulation with Multimodal Prompts.\" In ICML 2023\n\n[2] Constantin, Stefan, Fevziye Irem Eyiokur, Dogucan Yaman, Leonard Bärmann, and Alex Waibel. \"Interactive Multimodal Robot Dialog Using Pointing Gesture Recognition.\" In European Conference on Computer Vision 2022\n\n[3] Wan, Yanming, Jiayuan Mao, and Josh Tenenbaum. \"HandMeThat: Human-Robot Communication in Physical and Social Environments.\" Advances in Neural Information Processing Systems 35 (2022)\n\n[4] Constantin, Stefan, Fevziye Irem Eyiokur, Dogucan Yaman, Leonard Bärmann, and Alex Waibel. \"Multimodal Error Correction with Natural Language and Pointing Gestures.\" In Proceedings of the IEEE/CVF International Conference on Computer Vision 2023.\n\n[5] Shi, Cheng, and Sibei Yang. \"Spatial and Visual Perspective-Taking via View Rotation and Relation Reasoning for Embodied Reference Understanding.\" In European Conference on Computer Vision, pp. 201-218. Cham: Springer Nature Switzerland, 2022.\n\n\n\nWhile the dataset introduced tackles an important embodied task, the execution, motivation, and realism of some of the subtasks could be better explained. The choice of baselines and the generalizability of the VQ-fusion approach raise concerns about its applicability to more advanced multimodal models that have been recently more prevalent in the E-AI literature. Additionally, the paper could benefit from providing a clearer differentiation between embodied and non-embodied works in the field, as well as a more comprehensive review and comparison with recent advancements in multimodal language models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L1Nx73JLg2", "reviewer": "Reviewer_4CbJ", "summary": "The paper presents a large-scale Embodied Visual Question Answering dataset along with 8 benchmarks, and extensive experiments of various vision-language model representations are studied and benchmarked in the task. In addition, the paper proposes a VQ-fusion model that discretizes the representations of VL models, which improves the corresponding VL models by large margins in some tasks.", "questions": "See weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. The motivation of the paper is clear, and aims to solve the referring expression issues in VQA by explicitly proposing an embodied VQA task where the objects being asked are referred not only by verbal expressions but also by gestures or utterances.\n\n2. The scale of the dataset is quite large and diverse, and each question category is balanced. Overall, I feel like the quality of the dataset is really good to serve as a reliable benchmark for different approaches.\n\n3. The paper proposes an vq-fusion model (discretizing the multi-view encoded representations) that outperforms the ones without discretizations.", "weaknesses": "1. The proposed method described seems lack of implementation details, I think a section should be devoted to describe each component of the model in very details. I don't understand why multiple codebooks are used in the paper, as read from the figure 4. I understand that each vector representation is divided into different segments along its feature dimension, so the question: is each segment belong to different codebooks? What is the Factor-1, Factor-2, ..., Factor-N in the figure, standing for N codebooks?\n\n2. More qualitative studies regarding the samples answered by VQ-fusion and without VQ-fusion should be included in the paper to help understand why the VQ-fusion works and brings more insight to the proposed baseline approach.\n\n3. Studies of more powerful models like GPT-4V could be added as strong baseline, I am wondering if GPT-4V could solve this problem already?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "au1ADUl1WT", "reviewer": "Reviewer_BG1y", "summary": "The paper presents a large-scale dataset for embodied question answering (EQA) in 3D scenes. The dataset combines verbal questions with visual perspectives, such as gaze directions and pointing gestures, to enrich EQA with multimodal tasks. To solve these tasks, the paper also proposes a VQ-based multimodal fusion network to discretize the visual features and align them with the discrete verbal features. The authors benchmark their dataset and show the benefits of their proposed method through multiple dataset comparisons, experiments, and ablation studies.", "questions": "1. On Page 4, the descriptions of object grounding (OG), perspective-aware object grounding (POG), and perspective grounding (PG) appear to be slight variations of a single composite task, POG. In other words, solving POG seems to imply solving both OG and PG. Is this correct? Or could the authors please clarify the differences between these tasks further, maybe with some examples?\n\n2. Looking at Fig. 2, is the current method capable of locating up to 10 objects?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The proposed dataset is a natural extension of VQA tasks into EQA, providing novel, multimodal interactions that are closer to how humans interact in the real world. The dataset is likely to be of interest to researchers in related areas, including scene understanding, human-object interaction understanding, and robot navigation.\n\n2. The proposed approach of discretizing visual features through vector quantization (VQ) and aligning them with discrete verbal features obtained from natural language is technically sound and overall well-explained.\n\n3. The experiments are well-planned, extensive, and conclusive. They clearly highlight the benefits of the proposed dataset and the approach.", "weaknesses": "I did not find any major weaknesses in the paper. However,\n\n1. There seem to be some details missing on the processing of non-verbal gestures. For example, how is the pointing direction determined? Does the network take in absolute 3D joint positions, relative joint rotations, or some other representation? How is gaze represented?\n\n2. Do the authors consider any situations with multiple similar objects? For example, in Fig. 1, what if there are two red lamps next to each other? By extension, what other sources of ambiguity can exist in the dataset (for example, specific pointing with a finger vs. general pointing with the palm facing upwards, pointing to an object that is behind another object in the gaze direction, etc.)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)