EMABench: Evaluating Multimodal Large Language Models as Embodied Agents
Authors: Zhili Cheng, Yuge Tu, Yang Shi, Jinyi Hu, Ran Li, Shengding Hu, Shiqi Dai, Tianyu Yu, Lei Shi, Zhiyuan Liu, Maosong Sun


Reviewer paper summaries: This paper introduces EMABENCH, a benchmark for evaluating the embodied capabilities of Multimodal Large Language Models (MLLMs).
EMABENCH involves ego-centric, discrete interaction and navigation to complete question-answering and execution tasks.
The evaluation methodology employs a multiple-choice format, where the agent selects each action from a set of options defined at the beginning of each episode. 
Experimental results on various MLLMs demonstrate limited embodied capabilities.
Summary of strengths: 1. The paper introduces a dataset and evaluation process to test MLLMs' embodied capabilities in a zero-shot manner.
2. The paper describes a mostly automatic methodology to create scenes and generate the evaluation tasks (excluding some manual post-generation scene and task editing). 
3. The evaluation process is applied to 13 MLLMs, including open-source and API-based models trained with single or multiple images. The results show that current models have limited embodied capabilities.
Summary of weaknesses: Missing details
1. Options: It is not clear how the options for each task are selected, particularly how the distractor options are selected. Moreover, from the figures, it seems that there are 8 options per trajectory, but it should be clarified if it's the same number for all tasks (including for Task Decomposition).
2. Embodied capabilities evaluation: The targeted capabilities defined in Section 3.2 are quite broad, and there isn't sufficient information to ensure that the task generation process is systematic:
 
a) Is each of the targeted capabilities evaluated with both types of tasks (QA and execution)? If not, it would be useful, to provide a breakdown. 

b) It is mentioned that GPT-4 is prompted to generate diverse tasks in batches. However, how does this ensure that the generated tasks test for each targeted capability? This could be clarified by including more information about the seed tasks included in the prompt.

c) Lines 364-371: It is mentioned that some of the manually added tasks target the model's world knowledge. However, that doesn't fit within the defined capabilities.

3. Data Statistics: It would be informative to provide some additional data analyses: How many objects are included in the selected scenes and what is their distribution? What is the vocabulary distribution (related to the claim in Lines 244-250)? How many navigation and interaction steps are needed on average per task / capability?
4. Limited object interactions that include only pick and place actions
Comments, suggestions and typos: Typos:
- Process 1 does not fully capture the case of a correct answer for a QA task (which does not rely on a predicate function)
- Lines 226-234: The "look up" action that is shown is the figures in not listed with the other navigation actions 
- Line 258: Missing citation for Objaverse
- Tables 2 and 5 refer to Mobile Manipulation, while Section 3.2 refers to Object Interaction
- Line 151: "multiply-choice" instead of "multiple-choice"
Overall assessment scores: 2.5