Kati8/ICLR_2023_2025 huggingface dataset:
authors: Md Mofijul Islam
authors: Alexi Gladstone
authors: Riashat Islam
authors: Tariq Iqbal
reviews: 
Comments: The paper presents a large-scale dataset for embodied question answering (EQA) in 3D scenes. The dataset combines verbal questions with visual perspectives, such as gaze directions and pointing gestures, to enrich EQA with multimodal tasks. To solve these tasks, the paper also proposes a VQ-based multimodal fusion network to discretize the visual features and align them with the discrete verbal features. The authors benchmark their dataset and show the benefits of their proposed method through multiple dataset comparisons, experiments, and ablation studies.
Strengths: 1. The proposed dataset is a natural extension of VQA tasks into EQA, providing novel, multimodal interactions that are closer to how humans interact in the real world. The dataset is likely to be of interest to researchers in related areas, including scene understanding, human-object interaction understanding, and robot navigation.

2. The proposed approach of discretizing visual features through vector quantization (VQ) and aligning them with discrete verbal features obtained from natural language is technically sound and overall well-explained.

3. The experiments are well-planned, extensive, and conclusive. They clearly highlight the benefits of the proposed dataset and the approach.
Weaknesses: I did not find any major weaknesses in the paper. However,

1. There seem to be some details missing on the processing of non-verbal gestures. For example, how is the pointing direction determined? Does the network take in absolute 3D joint positions, relative joint rotations, or some other representation? How is gaze represented?

2. Do the authors consider any situations with multiple similar objects? For example, in Fig. 1, what if there are two red lamps next to each other? By extension, what other sources of ambiguity can exist in the dataset (for example, specific pointing with a finger vs. general pointing with the palm facing upwards, pointing to an object that is behind another object in the gaze direction, etc.)?
Recommendation: 8: accept, good paper
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 4 excellent
Clarity: 4 excellent
reviews: 
Comments: The paper presents a large-scale Embodied Visual Question Answering dataset along with 8 benchmarks, and extensive experiments of various vision-language model representations are studied and benchmarked in the task. In addition, the paper proposes a VQ-fusion model that discretizes the representations of VL models, which improves the corresponding VL models by large margins in some tasks.
Strengths: 1. The motivation of the paper is clear, and aims to solve the referring expression issues in VQA by explicitly proposing an embodied VQA task where the objects being asked are referred not only by verbal expressions but also by gestures or utterances.

2. The scale of the dataset is quite large and diverse, and each question category is balanced. Overall, I feel like the quality of the dataset is really good to serve as a reliable benchmark for different approaches.

3. The paper proposes an vq-fusion model (discretizing the multi-view encoded representations) that outperforms the ones without discretizations.
Weaknesses: 1. The proposed method described seems lack of implementation details, I think a section should be devoted to describe each component of the model in very details. I don't understand why multiple codebooks are used in the paper, as read from the figure 4. I understand that each vector representation is divided into different segments along its feature dimension, so the question: is each segment belong to different codebooks? What is the Factor-1, Factor-2, ..., Factor-N in the figure, standing for N codebooks?

2. More qualitative studies regarding the samples answered by VQ-fusion and without VQ-fusion should be included in the paper to help understand why the VQ-fusion works and brings more insight to the proposed baseline approach.

3. Studies of more powerful models like GPT-4V could be added as strong baseline, I am wondering if GPT-4V could solve this problem already?
Recommendation: 8: accept, good paper
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 2 fair