reviews: 
Comments: The paper introduces Embodied Question Answering (EQA) tasks that require agents to answer questions using both verbal and nonverbal gestures in an embodied setting. The authors present a large-scale dataset, EQA-MX, that contains diverse multimodal expressions from various verbal and visual perspectives. The work also introduces a Vector Quantization-based multimodal fusion approach (VQ-Fusion), which discretizes visual representations using shared codebooks before fusing them with discrete verbal representations. Additionally, the paper explores the importance of nonverbal gestures (gaze and pointing)  in learning EQA tasks.
Strengths: -  The paper presents eight EQA tasks designed for understanding questions using multimodal expressions (verbal utterances and nonverbal gestures) in embodied settings. 
- The introduction of the vector quantization-based fusion model addresses the structural mismatch between the discrete verbal representations and the continuous visual representations from the three available visual views. Experimental results show that incorporating VQ-Fusion in the existing baseline models improves their performance across EQA tasks. 
- The paper consists of extensive ablation studies on the number of codebooks, the use of multiple views, the generalizability of VQ-fusion, learning single vs. multiple tasks, etc.
Weaknesses: - The paper focuses on a synthesized dataset, which, while important for controlled experiments, may not fully capture the complexities and nuances of real-world interactions. This concern is partially due to the way view perspectives are included in the data. "The blender to the right of the book. From which perspective is the object described?" may not be something you'd commonly encounter in everyday (embodied) conversations/questions. Same for the example shown in Fig. 2 for RG.
- Table 1 is a bit unclear, e.g., why is EQA not available for all datasets, when some of the datasets referred to in the table correspond embodied QA benchmarks (EQA, MT-EQA)? Also, to the best of my knowledge, some of the datasets have egocentric views and not exocentric ones. 
- There is a body of work in pointing gesture recognition - most involving human communication in interactive embodied task-completion environments [2,3,4]. Similarly, the exists work that focuses on view perspectives [5]. It would be good to include and differentiate this work from the existing literature.
- While the reviewer appreciates the performance improvements of VQ-Fusion on 4 baselines, it would be beneficial to include more advanced baselines, such as a Multimodal Transformer model that extracts visual representations across all visual views and the verbal representations together, as well as more recent Multimodal LLMs (Otter, MiniGPT, UniLM, GPT4, InstructBLIP, LLaVA, etc.) that potentially can offer competitive fusion capabilities for these tasks.
- Based on Table 5 and Table 3, results in Table 3 for VQ-Fusion models are reported with varying codebooks, tuned for each task. This can improve performance, but it also raises questions about the generalizability of the approach. In some cases, the worst-case setting of the number of codebooks (e.g., 4 codebooks for EP or PG) can hurt the performance of the baseline CLIP model, which might impact the model's ability to perform well on new unseen tasks.

Minor edits:
- If the paper focuses on embodied AI, it would make sense to categorize the works into embodied and not embodied and then embodied multimodal prompting and language-guided embodied. VIMA [1], PATRON, and CAESAR could also be moved to Table 1 from Table 4 as similar embodied multimodal prompting tasks, especially since the Table 1 format and most of the capabilities originate from the use of CAESAR as the underlying simulator. 

[1] Jiang, Yunfan, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. "VIMA: General Robot Manipulation with Multimodal Prompts." In ICML 2023

[2] Constantin, Stefan, Fevziye Irem Eyiokur, Dogucan Yaman, Leonard Bärmann, and Alex Waibel. "Interactive Multimodal Robot Dialog Using Pointing Gesture Recognition." In European Conference on Computer Vision 2022

[3] Wan, Yanming, Jiayuan Mao, and Josh Tenenbaum. "HandMeThat: Human-Robot Communication in Physical and Social Environments." Advances in Neural Information Processing Systems 35 (2022)

[4] Constantin, Stefan, Fevziye Irem Eyiokur, Dogucan Yaman, Leonard Bärmann, and Alex Waibel. "Multimodal Error Correction with Natural Language and Pointing Gestures." In Proceedings of the IEEE/CVF International Conference on Computer Vision 2023.

[5] Shi, Cheng, and Sibei Yang. "Spatial and Visual Perspective-Taking via View Rotation and Relation Reasoning for Embodied Reference Understanding." In European Conference on Computer Vision, pp. 201-218. Cham: Springer Nature Switzerland, 2022.



While the dataset introduced tackles an important embodied task, the execution, motivation, and realism of some of the subtasks could be better explained. The choice of baselines and the generalizability of the VQ-fusion approach raise concerns about its applicability to more advanced multimodal models that have been recently more prevalent in the E-AI literature. Additionally, the paper could benefit from providing a clearer differentiation between embodied and non-embodied works in the field, as well as a more comprehensive review and comparison with recent advancements in multimodal language models.
Recommendation: 8: accept, good paper
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 2 fair