MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agents

## Abstract

Abstract MLLM agents demonstrate potential for complex embodied tasks by retrieving multimodal task-relevant trajectory data.
However, current retrieval methods primarily focus on surface-level similarities of textual or visual cues in trajectories, neglecting their effectiveness for the specific task at hand.
To address this issue, we propose a novel method, M LLM A s R e T riever ( MART ), which enhances the performance of embodied agents by utilizing interaction data to fine-tune an MLLM retriever based on preference learning, such that the retriever fully considers the effectiveness of trajectories and prioritize them for unseen tasks.
We also introduce Trajectory Abstraction, a mechanism that leverages MLLMs’ summarization capabilities to represent trajectories with fewer tokens while preserving key information, enabling agents to better comprehend milestones in the trajectory.
Experimental results across various environments demonstrate our method significantly improves task success rates in unseen scenes compared to baseline methods.
This work presents a new paradigm for multimodal retrieval in embodied agents, by fine-tuning a general-purpose MLLM as the retriever to assess trajectory effectiveness. All benchmark task sets and simulator code modifications for action and observation spaces will be released.

## 1 Introduction

Embodied agents interacting with complex environments require understanding both the current context and task-specific domain knowledge to perform effectively (Wang et al., ; Lifshitz et al., ). Recently, Multimodal Large Language Models (MLLMs), which are capable of processing both textual and visual data, have shown promise in various embodied tasks – *e.g., *table manipulation (Handa et al., ; Chen et al., ), robot navigation (Zhang et al., ; Shah et al., ), and 3D games (Wang et al., ; Tan et al., ; Jiang et al., ).
However, such models typically lack effective grounding in the embodied environments in which agents operate, greatly limiting their performance in embodied tasks (Long et al., ; Wang et al., ).

To mitigate this limitation, providing additional task-relevant grounding information is essential to better leverage the general capabilities of MLLMs. Trajectory data, consisting of sequences of actions and observations, can be easily available and provide valuable insights into task execution (Zheng et al., ; Zhao et al., ), therefore serving as a good information source for grounding. By using trajectory data in prompting an MLLM, the embodied agent can readily leverage previous experiences to better guide agents through similar tasks in new situations or environments (Zhang et al., ; Lee et al., ). However, retrieving the most effective trajectories — those that can significantly enhance task performance — remains a challenge, particularly when multiple trajectories appear similar in both textual and visual modalities (Jeurissen et al., ).

Existing retrieval methods mainly focus on surface-level textual or visual similarities of trajectories, often neglecting key aspects critical for task effectiveness, *e.g., *a trajectory with a similar task instruction but in a different scene, or one in the same scene but with a different layout. In such cases, these trajectories fail to provide useful information for the current task and can mislead the agent.
As shown in Figure 2, relying solely on similarity is not effective in retrieving useful trajectories, as similarity does not directly correlate with success rate.
To better support agents in embodied tasks, a trajectory retriever model needs to consider the effectiveness of trajectories for a given task.

Figure: (a) AI2-THOR
Refer to caption: x2.png

To achieve a better retriever, we propose a new paradigm that integrates interactive learning with the retriever. Firstly, we consider expert trajectories of training scenarios as prompt for an MLLM agent, and let the agent interact with the environment to collect different success rates for different such reference trajectories. This interactive feedback data is then organized into preference pairs, which are used to fine-tune an MLLM – LLaVA (Liu et al., ) in our case – with a Bradley-Terry model (Bradley & Terry, ), such that the fine-tuned retriever model is capable of prioritizing more effective trajectories for unseen tasks. Combining this functionality with the inherent general capabilities of MLLM allows embodied agents to operate more effectively in unseen environments by leveraging their most useful past experiences.

We also introduce a new Trajectory Abstraction mechanism, which uses MLLMs’ summarization capabilities to represent trajectories in a reduced number of tokens, while preserving key information and enabling agents to better understand such information in the trajectory (*e.g., *key relevant overarching actions). This mechanism is especially important in long-horizon tasks, both reducing the required context window length and removing distracting information from trajectory samples.

Combining the aforementioned components, we present our approach – MART (MLLM As ReTriever) – which adapts embodied agents in unseen scenarios by fine-tuning MLLM through preference data.
To assess the benefits of our method, we conduct empirical experiments across diverse environments. The experimental results show that MART achieves significantly higher task success rates compared to baselines, demonstrating its effectiveness. With this approach, we present a new paradigm for multimodal retrieval in embodied agents, fine-tuning a general-purpose MLLM as a retriever capable of considering trajectory effectiveness. Our contributions can be summarized as follows:

- •
To the best of our knowledge, MART is the first approach that integrates interactive learning with a retriever and uses interactive feedback to fine-tune an MLLM retriever in evaluating trajectory effectiveness, combining its inherent general capabilities with the ability to assess the task-guiding effectiveness of trajectories.
- •
We introduce Trajectory Abstraction, a new mechanism that utilizes MLLM capabilities to significantly condense trajectories. This method reduces the token number while retaining essential information, allowing agents to effectively use this condensed knowledge in novel situations and provide guidance for long-horizon tasks.
- •
The effectiveness of MART is empirically validated through comprehensive experiments in various environments, demonstrating significant performance improvements on unseen tasks. MART consistently surpasses baselines by over 10% across different environments.

## 2 Related Work

### 2.1 Embodied Agents Based on Large Models

Recently, there have been several attempts to utilize the general-purpose capabilities of large models for complex embodied tasks.
These efforts can be broadly categorized into two types: VLA models and LLM/MLLM-based agents.
1) VLA models, including PaLM-E (Driess et al., ), RT-2 (Brohan et al., ), Gato (Reed et al., ), VIMA (Jiang et al., ), and MOO (Stone et al., ), rely on trajectory data to train a Transformer-based VLM for action planning, without explicitly constructing a memory.
However, their generalization capabilities are limited due to the inherent issue of catastrophic forgetting in neural networks.
2) LLM/MLLM-based agents, like Voyager (Wang et al., ) and DEPS (Wang et al., ) for Minecraft, Cradle (Tan et al., ) for RDR2, LLM-Planner (Song et al., ) for ALFRed, and Code-as-Policies (Liang et al., ) for real-world embodied control, do not involve directly training new models. Instead, they leverage the general-purpose capabilities of LLM/MLLM primarily through prompt engineering.
Most of these agents build and maintain comprehensive memory systems to assist in task completion.
However, memory retrieval mostly focuses on surface-level similarity, overlooking actual effectiveness in completing complex tasks.

### 2.2 Memory Retrieval in Agents

Agents can continuously learn and improve by recalling task-related experiences (Zhang et al., ; Xi et al., ).
During interactions with the environment, two main types of information are stored in memory.
1) Semantic information:
Early LLM agents faced limitations due to input token constraints, leading to reliance on short-term memory and greedy strategies (Chen et al., ; Zhang et al., ; Abdelnabi & et al, ; Wang et al., ). However, summarizing memory over short periods risked information loss (Light et al., ; Kaiya & et al, ).
While storing comprehensive memory (semantic, episodic, procedural) can provide great value, its effective utilization for decision-making remains challenging (Sumers et al., ).
More recent approaches try to prioritize relevant memories based on embedding similarity or LLM-based relevance identification (*e.g., *(Park & et al., ), (Hong & et al., ), (Lin et al., ), (Wang et al., ), and (Xu et al., )).
2) Image information: Recent advances in MLLM agents have greatly enhanced their grounding abilities by allowing image memory retrieval during actions. As illustrated in Figure 1, Similarity-Based Retrievers for agents, *e.g., *(Zhou et al., ; Wang et al., ; Li et al., ), leverage image embedding. while Groot (Cai et al., ) encodes visual and temporal information from video frames for guiding actions. But neither considers direct task effectiveness.

### 2.3 Multi-modal Information Retrieval

Recent advances in multimodal retrieval have developed various methods for encoding, fusing, and measuring similarities across different modalities. ViLT (Kim et al., ) directly embeds image patches with text using a Transformer, while ALIGN (Li et al., ) and MURAL (Jain et al., ) use dual-encoder architectures with EfficientNet and BERT to align modalities through contrastive learning. IMAGEBIND (Girdhar et al., ) extends this by creating joint embeddings for six modalities, using ViT and Transformer models. Furthermore, (Changpinyo et al., ) integrates users’ mouse trace interactions for refined image retrieval, while ReViz (Luo et al., ) employs advanced encoding mechanisms for visual question answering.
Building on these encoding strategies, retrieval augmentation further enhances multimodal generation (*e.g., *RAG (Lewis et al., )).
FLMR (Lin et al., ) addresses RA-VQA limitations by combining multi-dimensional embeddings from ColBERTv2 and ViT-based models for accurate knowledge retrieval.
RA-CM3 (Yasunaga et al., ) enhances image captioning and text-to-image generation by using a pre-trained CLIP model to augment inputs for a CM3 Transformer. Similarly, UniRAG (Sharifymoghaddam et al., ) integrates retrievals using UniIR’s (Wei et al., ) CLIP Score Fusion and BLIP Feature Fusion, improving performance in MLLMs like LLaVA. Lastly, MuRAG (Chen et al., ) introduces a retrieval-augmented transformer for KB-VQA, employing T5 and ViT for multimodal encoding and retrieval from a large-scale memory bank.
In contrast, our method prioritizes the effectiveness of retrieved information by employing interactive learning, ensuring that the information contributes directly to task completion.

## 3 Interactively Learning Multimodal Retrieval

Figure: Figure 3: Overview of MART. Our approach interactively learns a multimodal retriever to score expert trajectories and retrieve most effective trajectory to guide an agent in novel situations. By considering trajectories with higher success rates as positive samples and those with lower success rates as negative trajectories, we obtain the preference pairs, which are used to fine-tune an MLLM retriever to score trajectory effectiveness for a specific task.
Refer to caption: x6.png

### 3.1 Problem Formulation

In this study, we investigate interactions of a retrieval-augmented MLLM agent with an environment to complete embodied tasks drawn from a specific distribution.
Figure 3 provides an overview of our MLLM As ReTriever approach – MART. The agent is assigned a task instruction $\ell^{c}$ sampled from the task instruction distribution $p(\ell)$, and operates over a finite horizon $H$.
At each timestep $t\in{1,2,...,H}$, the agent selects an action $a_{t}$ from the action space $A$ based on the current observation $o_{t}^{c}$ from the observation space $O$ and a reference trajectory $\tau^{e}$ retrieved from the expert trajectory memory $\mathcal{M}$.
The memory contains a set of multimodal expert trajectories $\mathcal{M}=\{\tau_{1}^{e},\tau_{2}^{e},...,\tau_{n}^{e}\}$, where each trajectory $\tau_{i}^{e}$ includes task instructions $\ell_{i}^{e}$ sampled from the same task instruction distribution $p(\ell)$. The trajectory $\tau_{i}^{e}$ also contains observation sequences $\vec{o_{i}^{e}}=\{o_{i_{1}}^{e},o_{i_{2}}^{e},...,o_{i_{H}}^{e}\}$, and action sequences $\vec{a_{i}^{e}}=\{a_{i_{1}}^{e},a_{i_{2}}^{e},...,a_{i_{H}}^{e}\}$.

The agent follows a frozen policy $\pi(a|\ell^{c},\tau^{e},o^{c})$, implemented as a Multimodal Large Language Model (MLLM), and the reference trajectory $\tau$ plays a significant role in grounding the agent within the embodied environment, supporting task accomplishment.
This reference trajectory is retrieved through an MLLM retriever $q_{\theta}$.

We fine-tune our MLLM retriever on training task distribution $p_{\textrm{train}}(\ell)$ and evaluate its performance on test task distribution $p_{\textrm{test}}(\ell)$, which has no overlap with training tasks. This retriever aims to identify and retrieve a trajectory $\tau^{e}$ from the expert memory pool that is most effective for the current task $\ell^{c}$, *i.e., *which can help ground the MLLM agent with a specific embodied task and enable its effective completion.
It is worth noting that we have different memories $\mathcal{M^{\textrm{train}}}$ and $\mathcal{M^{\textrm{test}}}$, which corresponds to training tasks and test tasks, respectively.

### 3.2 Memory

To enable trajectory retrieval for task execution and fine-tuning our MLLM retriever, we first construct memory databases containing expert trajectories from previous successful executions for tasks both from $p_{\textrm{train}}(\ell)$ and $p_{\textrm{test}}(\ell)$. For each trajectory,
$\tau_{i}^{e}=\{\ell_{i}^{e},\vec{o_{i}^{e}},\vec{a_{i}^{e}}\}$,
represents a task $\ell_{i}^{e}$ completed in $H_{i}$ steps and comprises the sequence of observations $\vec{o_{i}^{e}}$, and corresponding actions $\vec{a_{i}^{e}}$.

In multi-modal environments, such as AI2-THOR (Kolve et al., ) and LEGENT (Cheng et al., ), each timestep observation $o_{i}$ includes an egocentric image. Moreover, in the AI2-THOR environment, we assign numerical IDs (e.g. Cup 1) to all objects in the current visible field of view to identify target objects for interaction. These IDs appear in the environment feedback output in natural language, which is also part of the observation.

Expert trajectories are collected via a planner-based method (Hoffmann & Nebel, ). Storing these trajectories allows the agent to later leverage past experience when facing new task instances. Trajectory data is collected independently for the training and test sets. For each task, we initialized a task instance and used a planner-based method to collect expert trajectory data. It is worth mentioning that the initialization position and orientation in each task is randomly chosen.

Since each task is directly corresponds to one unique task in the task set, the size of our memory used for experiments is relatively small.
For example, in the experiments performed in the LEGENT environment, the memory pool consisted of 40 trajectories for training, and distinct 32 trajectories during testing. More details are available in the experimental settings (Section 4.1).

### 3.3 Multimodal Retriever

The core of MART is the innovative use of interactive learning to train the trajectory retriever. For an embodied task, different trajectories stored in memory can be provided as references to the MLLM agent, leading to varying effects on the completion of the current task, depending on the degree of grounding with the environment they provide. Even if a trajectory has text instruction similar to the current task or an image sequence similar to the initial egocentric observation of the task, it does not guarantee that this trajectory can provide effective grounding. This is due to plain similarity alone not being able to reflect the effectiveness of the trajectory for the embodied task.
For instance, a failed trajectory for a related task could have high textual and visual similarity to the target task.

In order to retrieve the trajectory that can provide the most benefit (*i.e., *effective grounding) for the current task from the trajectory memory, we propose an interactive learning method for the MLLM retriever.
Specifically, for each task in training set, we sample $K$ trajectories from the training memory $\mathcal{M^{\textrm{train}}}$, and feed them as prompt for MLLM agent to execute the embodied task respectively. After that, based on the induced success rates of task execution, we can get the effectiveness of each trajectory for the embodied task.
We can then obtain a partial order list based on success rate comparisons, producing $\binom{K}{2}$ pairs through pairwise comparison, where the trajectory with a higher success rate is treated as the positive item, and the one with a lower success rate as the negative item.
In this way, these preference pairs from interactive feedback are arranged as a positive-negative pair dataset $D$, which we use to fine-tune the MLLM according to the Bradley-Terry (Bradley & Terry, ) Reward Modeling loss to enhance its critiquing ability, as in Equation 1:

$$ $\mathcal{L}(\theta)=-\mathbb{E}_{(\ell_{1}^{c},o_{1}^{c},\mathcal{A}(\tau_{w}^ {e}),\mathcal{A}(\tau_{l}^{e}))\sim D}\left[\log\left(\sigma\left(q_{\theta}( \ell^{c},o_{1}^{c},\mathcal{A}(\tau_{w}^{e}))-q_{\theta}(\ell^{c},o_{1}^{c}, \mathcal{A}(\tau_{l}^{e}))\right)\right)\right].$ (1) $$

In particular, we use LLaVA-7B (Liu et al., )
as base MLLM and add a Bradley-Terry score head
based on hidden states of base model output. The score head is a one-layer MLP that takes as input the last token in the hidden state and outputs a scalar score. The input to the MLLM retriever includes the trajectory abstraction result of expert trajectory $\tau_{i}^{e}$, $\mathcal{A}(\tau_{i}^{e})$, current observation $o_{1}^{c}$, current task instruction $\ell^{c}$, and a prompt for it to judge the effectiveness of this trajectory for the current task and state. Upon inference, the MLLM outputs the score of the trajectory, which indicates its effectiveness for the current task. We select the trajectory with the highest score as the reference for the embodied agent to complete the current task.

### 3.4 Trajectory Abstraction

A complete multimodal trajectory often has dozens of timesteps, which correspond to dozens of observations, actions, and feedback, including redundant or irrelevant information. Furthermore, inputting all trajectory tokens, especially image tokens, to the MLLM retriever and agent will likely lead to confusing the models/agent or even exceeding their context windows.

We thus use another MLLM (in our experiments, GPT-4o) to automatically create an abstract trajectory in zero-shot manner. The initial input to the MLLM is the trajectory $\tau^{e}$ (consisting of task instruction $\ell^{e}$, observation sequence $\vec{o^{e}}$, and action sequence $\vec{a^{e}}$) as well as the current task instruction $\ell^{c}$, and we let the MLLM find whether each observation contained in the trajectory $\tau^{e}$ is helpful for the current task $\ell^{c}$. If it is considered to be useful, we will keep the observation into the resulting trajectory abstraction $\mathcal{A}(\tau^{e})$.

To be more specific, we let the MLLM comprehend the tasks accomplished in the given trajectory $\tau^{e}$, and then identify important observations in the trajectory as milestones and preserve them into the resulting trajectory abstraction.
These milestones are steps that are essential for accomplishing the trajectory task of $\ell^{e}$, such as steps where important decisions are made, goals are achieved, or notable changes in the environment or state occur. Also if the target object of current task instruction $\ell^{c}$ appears in the trajectory feedback, then the step where it appears is also considered to be a significant milestone as it is strongly related to current task $\ell^{c}$.
The milestone output format consists of: 1) a description of the milestone; 2) the corresponding image ({image x}); 3) the corresponding feedback ({feedback x}); and 4) the overarching actions taken between this milestone and the next one.

The summarized trajectory manages to remove redundant information without affecting relevance for the current task $\ell^{c}$, so that the agent can better receive grounding information contained in the trajectory. Quantitatively, for the tasks in the test set, the input trajectories of trajectory abstraction have an average of 11.51 steps, while the output milestones have an average count of 3.13.

## 4 Experiments

### 4.1 Experimental Setup

Figure: (a) AI2-THOR
Refer to caption: x7.png

#### 4.1.1 Environments

To validate the effectiveness of our method in various environments, we perform evaluations on multiple scenarios in two environments, AI2-THOR (Kolve et al., ) and LEGENT (Cheng et al., ). Unlike LLM agents, which use only text as input and simplify the action space by employing teleportation actions (*e.g., *“move directly to target"), we believe that MLLM agents should undertake more challenging tasks as they have access to visual input and are no longer limited to a ‘blind’ mode of operation. Therefore, both target environments are multimodal, whose observations are egocentric images, and both make use of fine-grained control actions.

AI2-THOR simulates embodied household tasks supporting natural language instructions and egocentric visual observations, where agents must navigate and interact with various household items within realistic 3D environments, including kitchens, living rooms, and other indoor spaces. it allows agents to perform fine-grained navigation actions including ‘move ahead’, ‘turn left/right x degrees’, ‘look up/down’, and interactive atomic action including ‘pick up object A’, ‘put object A on/in object B’, ‘open/close object A’, ‘toggle on/off object A’.

LEGENT is designed to imitate human activities and tasks in home environments, including cross-room navigation. The action space is similar to AI2-THOR’s, including fine-grained movement actions. It also includes a ‘speak’ action, which sends a message to the user.

Table 4 and Figure 4 show a comparison between the two environments. Besides having many more objects in its scenes, AI2-THOR supports more complex object hierarchy relationships, such as “inside" (*e.g., *“inside a cabinet", which require open/close interaction to complete a task, or “a cup inside a sink", *i.e., *an open container), not supported in LEGENT (^1^11Although there are cabinets in LEGENT, they do not need to be opened/closed to complete tasks.).

#### 4.1.2 Task Settings

Notably, in all tasks, the initial position and orientation of the agent is chosen randomly. Each task is tested 5 times to reduce the impact of random errors.

AI2-THOR.
To better assess the fine-grained control ability of MLLM agents to complete real-world embodied tasks, we integrate characteristics of two AI2-THOR-based benchmarks – ALFWorld (Shridhar et al., ) and ALFRed (Shridhar et al., ) – and built an environment setting that is more suitable for the MLLM agent. Since tasks are long-horizon, we follow the method in ALFRed and apply task decomposition to divide them into sub-tasks before execution. Each sub-task is then provided to the agent, and it determines sub-task success based on environmental feedback by itself (details in Appendix B.2 and  B.3
). Once a sub-task is successfully completed, the agent proceeds to the next sub-task. Task types include pick_and_place, pick_clean_then_place, pick_cool_then_place, and pick_heat_then_place. Completing these tasks requires dozens of steps of navigation, as well as interaction with objects. There are 45 tasks comprising a total of 260 sub-tasks in training set, and 28 tasks including 158 sub-tasks in testing set.

LEGENT.
Tasks for the MLLM agent are categorized into two types: ‘Come Here’ and ‘Where Is’. Each task is further divided into ‘One-room’ and ‘Two-room’ types, based on whether it requires traversing between rooms. In LEGENT, task decomposition is not performed as task instructions are simpler and do not contain combinations of sub-tasks; *i.e., *the granularity of tasks is similar to that of sub-tasks in AI2-THOR. To train the retriever, we use 40 tasks (10 tasks for each task type) and we use 32 tasks, also covering all task types, as test set.

#### 4.1.3 Memory Construction

Memory initialization follows the procedure described in Section 3.2; with randomized starting positions.

AI2-THOR. Once trajectories are collected, we decompose them into sub-task trajectories (following the task decomposition procedure in ALFRed) and treat each sub-task level trajectory as a expert trajectory into memory. Similar redundant trajectories are then filtered out, accounting for about one-third of total, resulting in a collection of 170 memory trajectories for the training memory and 118 trajectories for the testing memory.

LEGENT. As decomposition is not necessary due to the simpler tasks in this environment, the training memory is initialized with 40 trajectories, and the test memory with distinct 32 trajectories; one trajectory per task.

#### 4.1.4 Task Evaluation

To evaluate the effectiveness of the retrieval-augmented embodied agents, we assess their performance using two metrics: Success Rate (SR) and Average Steps (AS).

Success Rate denotes the percentage of tasks attempts successfully completed by the agent. In AI2-THOR  it indicates the percentage of completed full tasks, and we additionally use SR-Sub to represent the percentage of completed sub-tasks.

Average Steps represents the average number of steps the agent takes to complete a task. In AI2-THOR  it indicates the number of steps to complete a full task, and AS-Sub represents the step average to complete a sub-task.

#### 4.1.5 Baselines

It is worth noting that MART is the first work to retrieve multimodal trajectories as references for embodied MLLM agents and let the agent directly output fine-grained control actions. We compare MART against three baseline methods to explore the performance of our approach:

Plain-Agent (PA) is an embodied MLLM agent without making use of reference trajectories, *i.e., *without any memory. In our experiments, we use GPT-4o (2024-05-13 version).

LLaVA-Plain (LP) is a pre-trained LLaVA with no modified head and no finetuning. We use the probability of special token generation to represent the score. Its input is the same as MART, and it is prompted to output only Yes/No tokens, and the final score is calculated based on the probability of token generation (more details and limitations in Appendix A.3).

Similarity+LLaVA (SL) is a reasonable retrieval+ranking approach. Such approach is common in the text retrieval field – *e.g., *Sun et al. (; ); Dong et al. () – and it can take into account both similarity and effectiveness. We use similarity to choose the top-K candidate trajectories, and then choose the most likely effective one using a plain LLaVA model (*i.e., *same as LP).

RAP (Kagaya et al., ) performs retrieval based on plain similarity per modality and is the most similar setting in literature. It mainly targets text modality experiments in the ALFWorld (Shridhar et al., ) and WebShop (Yao et al., ) (treated as text-only) environments, and simple tasks in the multimodal environments Franka-Kitchen (Gupta et al., ) and Meta-World (Yu et al., ).

### 4.2 AI2-THOR

We firstly demonstrate the effectiveness of MART over test tasks in the AI2-THOR environment. The experimental results demonstrate the effectiveness of our approach compared to the baselines. As shown in Table LABEL:tab:ai2thor_results, MART surpasses all baselines over 10% in Success Rate, and reaches best performance across all metrics.

**Table 2: Performance comparison of different methods in AI2-THOR.**
|  | PA | LP | SL | RAP | MART |
| --- | --- | --- | --- | --- | --- |
| SR $\uparrow$ | 0.18 | 0.26 | 0.24 | 0.22 | 0.40 |
| SR-Sub $\uparrow$ | 0.63 | 0.69 | 0.68 | 0.67 | 0.75 |
| AS $\downarrow$ | 159.66 | 144.18 | 147.48 | 147.03 | 123.19 |
| AS-Sub $\downarrow$ | 44.65 | 39.88 | 40.79 | 40.67 | 34.07 |

### 4.3 LEGENT

We then conduct experiments in the LEGENT environment. This environment includes tasks involving crossing between rooms, thereby enriching the experimental space and demonstrating the effectiveness of our method. The experimental results, shown in Table 3, demonstrate MART greatly surpasses all baselines in all four task types.

**Table 3: Performance comparison of different methods in LEGENT.**
|  | PA | LP | SL | RAP | MART |
| --- | --- | --- | --- | --- | --- |
| SR $\uparrow$ | 0.70 | 0.69 | 0.75 | 0.75 | 0.87 |
| AS $\downarrow$ | 23.62 | 25.01 | 20.92 | 20.62 | 13.81 |

### 4.4 Ablations

In this section, we use a set of ablation studies to examine the contribution of key components in MART. More specifically, we aim to answer the following questions.

Q1. Does Trajectory Abstraction indeed improve embodied agent performance?

We compare the full MART approach and the ablation of removing its Trajectory Abstraction module (w/o Abstraction). As shown in Table 4, MART consistently reaches the best results across settings. Even if in AI2-THOR sub-tasks the average success rate is comparable, the improvement margin leads to a 9 percentage points improvement in full task success rate.

Q2. How does the MART approach compare against a typical retrieve and rank approach, even if it uses a fine-tuned ranking model for trajectory usefulness?

Table 4 shows decomposing the unified MART approach into separate retrieval and ranking steps (*i.e., *similarity-based retrieval and MART’s MLLM as ranker – Sim.+FTM) decreases success rates across settings. It is also interesting to note that Sim.+FTM outperforms both SL and RAP (Tables LABEL:tab:ai2thor_results and 3), further illustrating MART’s trajectory utility scoring.

**Table 4: Ablation studies of MART in the AI2-THOR and LEGENT environments.**
| Environment | Metric | w/o Abstraction | Sim.+FTM | MART |
| --- | --- | --- | --- | --- |
| AI2-THOR | SR $\uparrow$ | 0.31 | 0.34 | 0.40 |
| SR-Sub $\uparrow$ | 0.73 | 0.74 | 0.75 |  |
| AS $\downarrow$ | 130.26 | 125.20 | 123.19 |  |
| AS-Sub $\downarrow$ | 36.03 | 34.63 | 34.07 |  |
| LEGENT | SR $\uparrow$ | 0.77 | 0.77 | 0.87 |
| AS $\downarrow$ | 18.48 | 18.83 | 13.81 |  |

Figure: Figure 5: Comparison between similarity-based retriever and MART.
Refer to caption: x11.png

### 4.5 Case Study

We present two case studies for more in-depth discussion of MART’s capabilities handling challenges of the MLLM agent setting. The first case (Figure 5) demonstrates the effective handling of a very long-horizon trajectory. Given a 73-step task trajectory – “navigate to DishSponge, put it in the Plate, and place the Plate on the Cabinet" – Trajectory Abstraction identifies 5 key milestones. MART achieves an 80% success rate with an average of 28 steps, while both the similarity-based method and the agent without memory reach only 40% success rate, averaging 69.6 and 74.6 steps, respectively.

The second case (Figure 6) shows how MART extracts implicit rules for long sequence tasks. For the task “put the Potato into the microwave, heat it, and pick it up”, Trajectory Abstraction analyzes the agent’s actions (including exploration, attempts, and success) and generates an abstract set of inferred rules, pruning non-contributory and redundant actions, reducing 13 transitions to just 6. MART achieves an 80% success rate with an average of 30.2 steps, while other methods have a 0% success rate.

Figure: Figure 6: Showcase of the significance of the Trajectory Abstraction mechanism.
Refer to caption: x12.png

## 5 Conclusion and Limitations

We propose MART, a new paradigm for trajectory retrieval incorporating interactive learning, to enhance embodied agents’ performance by providing them with task-relevant trajectory data. Our approach utilizes interaction-based feedback to identify the most effective trajectories, and constructs preference pairs based on the comparisons between trajectories.
An MLLM retriever is fine-tuned through these preference pairs, effectively prioritizing the trajectories that improve task performance.
We also introduce Trajectory Abstraction in MART, a novel mechanism that leverages MLLMs’ summarization capabilities, to abstract trajectories, *i.e., *reduce the required number of tokens to represent them, while preserving key information and enabling agents to better understand relevant information. Experimental results in different environments demonstrate that our method significantly enhances task success rates in unseen tasks, compared to multiple baselines. This work helps bridge the gap between general-purpose MLLMs and the specific requirements of embodied tasks, offering a new paradigm for multimodal trajectory retrieval for embodied agents.

As the context window is limited for the MLLM we used, it cannot accurately process a large number of images, so we only retrieve one trajectory at a time to input to the agent in the manner of one-shot learning. If multiple trajectories can be used as input at a time, as in few-shot learning, an agent could splice the skills corresponding to different sub-trajectories, which has the potential to further improve performance in more complex and longer-horizon tasks. Such setting, and also methods for retrieving trajectory sub-segments instead of full trajectories, are left as future work.

## Appendix A Experimental setup details

In this appendix we provide more low-level details on the implementation of MART experiments.

### A.1 Multiple image input in LLaVA

The LLaVA architecture itself is compatible with using multiple images as input, but the released model weights do not have the ability to handle multiple images. In other words, when you input multiple images, it will only focus on the contents of the first image. Therefore, we fine-tune LLaVA through multi-image captioning data to enable it to understand multiple images.

We utilized LLaVA’s single-image perception capabilities to enhance its multi-image perception. Specifically, we sampled images collected from the environments and employed the pre-trained LLaVA to describe the content of each image. We organized the responses into multi-image captioning data, which we then used to fine-tune LLaVA. After our verification, the fine-tuned LLaVA demonstrated the ability to perceive multiple input images. After fine-tuning with multiple image datasets, LLaVA serves as the basez model for our subsequent retriever training.
We will release all the code.

### A.2 Parameter settings

The specific parameter settings in the experiment are shown in Table 5.

**Table 5: Hyperparameters of LLaVA fine-tuned by LoRA**
| Hyperparameters | Value |
| --- | --- |
| LLaVA_version | llava-v1.6-mistral-7b |
| train_batch_size | 32 |
| eval_batch_size | 8 |
| gradient_accumulation_steps | 8 |
| learning_rate_AI2THOR | 2e-5 |
| mm_projector_lr_AI2THOR | 2e-5 |
| learning_rate_LEGENT | 3e-6 |
| mm_projector_lr_LEGENT | 3e-6 |
| lora_r | 16 |
| lora_alpha | 32 |
| warmup_ratio | 0.05 |
| model_max_length | 32768 |
| lr_scheduler_type | cosine |
| vision_tower | clip-vit-large-patch14-336 |

### A.3 Details of LLava-Plain

Building on existing work (Asai et al., ) (Sun et al., ) that employs LLM for text retrieval, we use the generation probability of a special token to represent the score for LLaVA-Plain.
In detail, the effectiveness score $s_{i}$ is measured by the probability of LLaVA-Plain to generate the special token ‘Yes’ and ‘No’, as in Equation 2, where $p\left(\mathrm{Yes/No}\right)$ denoted the probability of LLaVA-Plain to generate Yes or No.
E.

$$ $s_{i}={\frac{p\left(\mathrm{Yes}\right)}{p\left(\mathrm{Yes}\right)+p\left( \mathrm{No}\right)}}$ (2) $$

In detail, the prompts we use for LLaVA-Plain and MART are presented in prompt 1 of Appendix E.

## Appendix B AI2-THOR environment specifics

### B.1 Setting differences

Two popular benchmarks, ALFRED (Shridhar et al., ), and ALFWorld (Shridhar et al., ) – derived from ALFRED – are both built on AI2-THOR. However, none of them are directly suitable for benchmarking MLLM agents performing real-world tasks.

ALFRED is a multimodal benchmark in AI2-THOR that uses fine-grained navigation actions. However, it requires pixel-level masks to specify objects for interaction actions. MLLM lacks the capability to generate such pixel-level masks, making ALFRED incompatible with MLLM agents without adaptation at either side.

ALFWorld adapts ALFRED for LLM agents (*i.e., *text-only) by simplifying it. Firstly, it provides text feedback as observation, detailing objects in the agent’s field of view along with their corresponding IDs. Secondly, it simplifies the action space by replacing all navigation actions with the teleportation action ‘go to’ and composite high-level actions like ‘heat’, ‘clean’, and ‘cool’, each involving multiple atomic interactions. For example, the “cool object a" action is equivalent to: ‘open the refrigerator’, ‘put object A inside the refrigerator’, ‘close the refrigerator’, ‘open the refrigerator’, and ‘pick up the object A’. These modifications significantly reduce task difficulty, while allowing LLM agents to perform in ALFWorld.

To better evaluate the fine-grained control abilities of MLLM agents in real-world tasks and longer-horizon more realistic tasks, we reject ALFWorld’s setting approach, which uses teleportation and composite high-level actions, opting instead for fine-grained navigation actions and finer-grained actions. However, unlike in ALFRED, since MLLMs cannot generate pixel-level masks by default, we allow interaction actions to reference objects using a numerical ID (*e.g., *cup 1) provided by the environment’s feedback, instead of a pixel-level mask.

All selected task sets and modified simulator code for adapting the MLLM agent will be released upon paper publication.

### B.2 Task Decomposition

We adopt the ALFRed (Shridhar et al., ) method to decompose the entire task into multiple sub-tasks.
In ALFRed, tasks are decomposed as follows:
First, they encode agent and object states, along with high-level environment dynamics, into Planning Domain Definition Language (PDDL) rules (Aeronautiques et al., ).
Next, they define task-specific PDDL goal conditions, such as a heated potato resting on a tabletop.
The planner assumes a fully observable environment with perfect knowledge of world dynamics.
Consequently, each task is decomposed into several sub-tasks, with instructions provided for each subtask by human labelers.

We follow the ALFRed method but adjust the decomposition results.
PDDL-based decomposition can result in inconsistent sub-task difficulty.
For example, a sub-task like ‘pick up object A’ or ‘put object A on object B’, which often follows a navigation sub-task.
If the previous navigation sub-task is executed successfully, the current sub-task can be completed in one step according to the PDDL decomposition.
We adjusted the task decomposition by merging sub-tasks that can be completed in one step with adjacent sub-tasks to balance their difficulty.
We will release the benchmark including our tasks and modified sub-tasks.

### B.3 Success Detection

In AI2-THOR, we provide the agent with the environment’s metadata and feedback in the Success Detection module. After each step is executed, the agent then determines whether the current sub-task is completed using a few-shot approach. For specific prompts, please refer to Appendix E.

### B.4 Hierarchy examples

Figure 7 illustrates multiple cases of the object hierarchy relationship *Inside*, only available in the AI2-THOR environment.

Moreover, Figure 8 illustrates the different types of rooms in our LEGENT experiments, showing connectivity, but simpler *On* hierarchies than AI2-THOR.

Figure: (a) Microwave.
Refer to caption: x13.png

Figure: (a) Where-2room.
Refer to caption: extracted/5901958/images/legent/where-2room.png

## Appendix C Full step count results

Due to space limitations, we present the full tables with Success Rate (SR) and Average Steps (AS) results here.

Table LABEL:tab:legent_results_complete shows the performance comparison of different types in LEGENT, with full step count results. While, Table 7 shows the performance comparisons of the ablation studies in both AI2-THOR and LEGENT environments, with full step count results.

**Table 6: Performance comparison of different types in LEGENT, with full step count results.**
| Metric | Type | PA | LP | SL | RAP | MART |
| --- | --- | --- | --- | --- | --- | --- |
| SR $\uparrow$ | Where-2room | 0.65 | 0.60 | 0.60 | 0.73 | 0.88 |
| Where-1room | 0.78 | 0.80 | 0.80 | 0.89 | 0.91 |  |
| Come-2room | 0.63 | 0.55 | 0.63 | 0.46 | 0.73 |  |
| Come-1room | 0.75 | 0.82 | 0.96 | 0.93 | 0.98 |  |
| Average | 0.70 | 0.69 | 0.75 | 0.75 | 0.87 |  |
| AS $\downarrow$ | Where-2room | 25.23 | 28.98 | 28.30 | 26.23 | 14.03 |
| Where-1room | 10.40 | 16.49 | 15.36 | 9.82 | 5.07 |  |
| Come-2room | 32.05 | 34.95 | 28.80 | 35.33 | 26.90 |  |
| Come-1room | 26.80 | 19.62 | 11.22 | 11.09 | 9.24 |  |
| Average | 23.62 | 25.01 | 20.92 | 20.62 | 13.81 |  |

**Table 7: Ablation studies of MART in the AI2-THOR and LEGENT environments, with full step count results.**
| Environment | Metric | w/o Abstraction | Sim.+FTM | MART |  |
| --- | --- | --- | --- | --- | --- |
| AI2-THOR | SR $\uparrow$ | 0.31 | 0.34 | 0.40 |  |
| SR-Sub $\uparrow$ | 0.73 | 0.74 | 0.75 |  |  |
| AS $\downarrow$ | 130.26 | 125.20 | 123.19 |  |  |
| AS-Sub $\downarrow$ | 36.03 | 34.63 | 34.07 |  |  |
| LEGENT | SR $\uparrow$ | Where-2room | 0.78 | 0.72 | 0.88 |
| Where-1room | 0.74 | 0.89 | 0.91 |  |  |
| Come-2room | 0.63 | 0.55 | 0.73 |  |  |
| Come-1room | 0.95 | 0.91 | 0.98 |  |  |
| Average | 0.77 | 0.77 | 0.87 |  |  |
| AS $\downarrow$ | Where-2room | 17.36 | 15.96 | 14.03 |  |
| Where-1room | 12.36 | 11.51 | 5.07 |  |  |
| Come-2room | 32.75 | 34.13 | 26.90 |  |  |
| Come-1room | 11.47 | 13.73 | 9.24 |  |  |
| Average | 18.48 | 18.83 | 13.81 |  |  |

## Appendix D Algorithm

MART’s Agent Execution Pseudocode is shown in Algorithm 1.

UTF8gkai

Figure: algorithm 1 MART Agent Execution Pseudocode

## Appendix E Agent prompts

Figure: Prompt 1: Retriver Prompt to LLaVA.

Figure: Prompt 2: Trajectory Abstraction Prompt.

Figure: Prompt 3: Self-Reflection Prompt.

Figure: Prompt 4: Action Planning Prompt.

Figure: Prompt 5: Success Detection Prompt.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See above.
soundness: 3
strengths: 1. The paper proposes a unique method for enhancing multimodal retrieval in embodied agents by leveraging preference learning and trajectory abstraction. This approach addresses a critical gap in current retrieval methods that often focus on surface-level similarities.
2.  The introduction of Trajectory Abstraction is a valuable contribution. It effectively reduces the complexity of trajectories while maintaining essential information
3. The experimental results are comprehensive and demonstrate the effectiveness of the proposed method across various environments. The improvements in task success rates in unseen scenes are particularly noteworthy.
summary: The paper introduces a novel method, MLLM As ReTriever (MART), which enhances the performance of embodied agents by utilizing interaction data to fine-tune a multimodal language model (MLLM) retriever. The method aims to improve the effectiveness of retrieved trajectories for specific tasks, and it introduces Trajectory Abstraction to summarize trajectories while preserving key information. The experimental results show significant improvements in task success rates in unseen scenes compared to baseline methods.
weaknesses: 1. While the paper demonstrates the effectiveness of MART in several environments, the scope of experiments could be expanded to include more diverse and challenging scenarios to further validate the robustness of the method.
2. The paper does not provide a detailed analysis of the computational complexity and resource requirements of the proposed method. This information is crucial for practical implementation and comparison with existing methods.
3. The paper mentions the use of interaction data but does not delve into the specifics of how user interactions are collected and processed. More details on this aspect would enhance the clarity and reproducibility of the method.
4. Provide more detailed implementation guidelines and discuss potential limitations and solutions. This would make the method more accessible to other researchers and practitioners.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: You can find all my inquiries detailed in the Weaknesses section.
soundness: 2
strengths: 1. This paper proposes a new retrieval-augmented MLLM agent, which finds the most matched expert reference trajectory to enhance current trajectory planning. To compress these multimodal trajectories with lots of images, actions, and feedback, Trajectory Abstraction (GPT-4o) is introduced to identify important trajectory milestones.

2. The experimental results demonstrate the effectiveness of MART in improving task success rates across different environments.
summary: The paper presents a novel method, MLLM As ReTriever (MART), which aims to enhance the performance of embodied agents in complex environments by improving the retrieval of multimodal, task-relevant trajectory data. It uses expert trajectories as prompts for an MLLM agent, collects interactive feedback in the form of success rates, and organizes this data into preference pairs. These pairs are then used to fine-tune the MLLM retriever, enabling it to prioritize more effective trajectories for unseen tasks. The authors conducted experiments across two environments, demonstrating MART's success rates in unseen scenes.
weaknesses: I am not an expert in this field, but I have several problems:

---

1. This work aims to tackle the challenge of grounding in environments. However, the literature review focuses more on embodied agents, memory retrieval in agents, and MLLM. These areas encompass a wide range of topics and, as a result, **overlook numerous specific and significant studies concerning how past embodied grounding approaches have tackled this issue**. In summary, I am unable to identify key references within this manuscript, and the innovation it presents remains obscure to me.

--- 
2. The proposed model is evaluated on AI2-THOR and LEGENT, with performance comparisons against Plain-Agent, LLaVA-Plain, Similarity+LLaVA, and RAP. However, it's worth noting that these comparative methods are predominantly variations of Multimodal Language Models (MLLM) or retrieval-based approaches, effectively serving as ablation studies. **This suggests that the absence of key comparisons with prior grounding techniques raises concerns regarding the efficacy of the proposed method's performance.**


---
3. What are the implementation details including model and training details?

---

confidence: 4
contribution: 2
initial_rating: 8
presentation: 3
questions: 1. It would be better if the authors could further explain the specific innovations in the pipeline compared to prior works LLM-Planner and P-RAG, highlighting how the proposed approach diverges or improves upon these method.
2. Could the authors provide an explanation for HOW Trajectory Abstraction contributes to performance improvements? 
3. If incorrect trajectories are retrieved, could this lead to catastrophic performance loss?
soundness: 3
strengths: 1. The problem addressed in this paper—decision-making in embodied tasks—is a crucial area.
2. The writing in this paper is clear and well-structured, with each paragraph presenting ideas in a logical, cohesive manner. The authors effectively communicate complex concepts, making the paper accessible and easy to follow.
3. This paper introduces an innovative multimodal retrieval tool and selects appropriate baselines to validate the effectiveness of the proposed method.
summary: This paper introduces MART, a novel trajectory retrieval paradigm leveraging interactive learning to enhance embodied agents’ task performance. MART utilizes interaction-based feedback to identify the most effective trajectories and constructs preference pairs based on trajectory comparisons. These preference pairs are used to fine-tune a Multimodal Large Language Model (MLLM) retriever, prioritizing trajectories that improve task outcomes. Additionally, MART introduces Trajectory Abstraction, a mechanism using MLLMs’ summarization capabilities to condense trajectory representations by reducing token counts while preserving essential information, enabling agents to better understand task-relevant details. Experimental results across diverse environments demonstrate that MART significantly improves success rates in unseen tasks compared to various baselines. This work bridges the gap between general-purpose MLLMs and embodied task requirements, offering a new paradigm for multimodal trajectory retrieval for embodied agents.
weaknesses: 1. The methodology in this paper shows some similarities with previous works, such as LLM-Planner (arXiv:2212.04088 ) and P-RAG(	arXiv:2409.11279). While there are innovations in the retriever design, the overall novelty of the approach is somewhat reduced.
2. The paper lacks some detailed analysis and deeper insights. For instance, the ablation study does not include experiments explaining HOW Trajectory Abstraction contributes to performance improvement

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Is the High Cost of Training an Additional MLLM Justified?
Could you verify whether the benefits of training and using an additional MLLM outweigh the high costs in terms of additional inference time and required training resources? Are there specific scenarios where the MLLM retriever significantly outperforms conventional similarity retrieval methods to justify its use?

2. How Did You Overcome Stability and Hallucination Issues?
Since language models can be unstable and prone to hallucinations, what methods did you employ to mitigate these issues in your MLLM retrieval system? Have you verified the effectiveness of your approach across multiple evaluation methods to ensure reliable performance?

3. Clarification on Evaluation Metrics in Table 4:

	a) What Exactly Does AS Represent? Could you clarify the meaning of AS (Average Steps) in your evaluation? Does it represent the number of iterations required for the agent to succeed once?

	b) Correlation Between AS and SR: Does Average Steps and Success Rate correlate? How do you interpret the results where AS shows minimal difference but SR differs more significantly?

	c) Fairness of Using AS as a Metric:
Considering the additional computational cost of incorporating an MLLM, would it be more appropriate to use the average inference time of the entire system instead of AS to ensure a fair comparison between methods?

4. Can You Provide Balanced Case Studies?
To ensure a fair comparison, could you provide both the success and failure cases for each method (MLLM Retrieval and similarity retrieval) in your case studies? This would help understand each approach's strengths and weaknesses more comprehensively.
soundness: 2
strengths: 1. Innovative Retrieval Method: The paper introduces a novel method by fine-tuning an MLLM as a retriever using interactive preference learning, moving beyond traditional similarity-based retrieval methods. This allows the retriever to assess trajectory effectiveness more holistically.
2. Trajectory Abstraction Mechanism: The introduction of Trajectory Abstraction is a significant contribution. By condensing trajectories into shorter representations while retaining crucial information, agents can process and utilize past experiences more efficiently.
3. Demonstrated Performance Improvements: The method substantially improves task success rates across various environments and unseen tasks, indicating its effectiveness and potential for broader applications in embodied AI.
4. Advancement in Embodied Agent Capabilities: By enhancing how agents retrieve and utilize past trajectories, MART contributes to developing more intelligent and capable embodied agents capable of handling complex, long-horizon tasks.
5. Commitment to Open Science: The release of benchmark task sets and simulator code modifications promotes transparency and allows other researchers to build upon this work, fostering collaboration in the community.
summary: The paper introduces MLLM As Retriever (MART), a novel approach designed to enhance the performance of embodied agents in complex tasks by improving multimodal retrieval of task-relevant trajectory data. Traditional retrieval methods often rely on surface-level similarities of textual or visual cues, which may not effectively capture the nuances required for specific tasks. MART addresses this limitation by fine-tuning a Multimodal Large Language Model (MLLM) retriever through interactive preference learning, allowing it to evaluate and prioritize trajectories based on their effectiveness for unseen tasks. Additionally, the paper presents Trajectory Abstraction, a mechanism that leverages MLLMs’ summarization capabilities to represent trajectories with fewer tokens while preserving essential information, enabling agents to better understand key milestones. Experimental results across various environments demonstrate that MART significantly improves task success rates on unseen tasks compared to baseline methods, consistently surpassing them by over 10%. The authors commit to releasing all benchmark task sets and simulator code modifications to facilitate further research.
weaknesses: 1. Efficiency Issues: The proposed method of using a Multimodal Large Language Model (MLLM) as a retriever may not be as efficient as conventional similarity retrieval methods, known for being fast and low-cost. Given that complex language model reasoning can already achieve high-quality planning decomposition and performance, the added cost of training and deploying an additional MLLM may not be justified. There is a need for verification to determine if the high cost is worth the potential benefits, including considerations of additional inference time and required training resources.

2. Potential Stability and Hallucination Problems: Replacing simple similarity calculations with judgments based on a language model introduces potential instability and hallucination issues inherent to language models. The MLLM’s retrieval process might be inconsistent, leading to unreliable action selection. It’s unclear how these issues are overcome in the system and whether effectiveness has been verified across multiple evaluation methods. Since MLLM retrieval effectively acts as a terminal evaluator to judge the best action, stability is critical.

3. Experimental Confusion: In Table 4, the ablation study comparing MLLM Retrieval and similarity retrieval methods shows minimal differences in Average Steps (AS) but noticeable differences in Success Rate (SR). Questions arise about the correlation between these two evaluation metrics. Specifically:

    a) Clarification of AS: What does AS (Average Steps) represent? Is it the number of iterations the agent system takes to succeed once?

    b) Difference in Metrics: Why is there a significant difference in SR but not AS between the two methods?

    c) Fairness of AS: Given the introduction of an additional language model, using AS may not be fair. It might be more reasonable to replace AS with the average inference time of the entire system to reflect the true cost and efficiency.

4. Unfair Comparison in Case Study: The case study presents success cases of the proposed MLLM Retrieval method alongside failure cases of the similarity retrieval method. This comparison may not be fair or sufficient to demonstrate the advantages of the proposed method. Providing success and failure cases for each method would offer a more balanced and comprehensive evaluation.

Meta-review: **Summary**

The paper proposes to use multi-modal large language models (MLLM) to retrieve trajectories for embodied tasks by fine-tuning the MLLM to judge the effectiveness of the retrieved trajectory.  To capture important information in the trajectory, the paper introduces a mechanism (using MLLMs) to create a trajectory abstraction (e.g. summarization) that extract key milestones from the complete trajectory and augment it with additional information such as a description, feedback, and actions taken between the current milestone and the previous milestone.  Experiments are conducted on two environments (AI2-THOR, LEGENT) to compare the effectiveness of the proposed approach to alternatives.

The main contributions of the work are the proposed model (MART) that fine-tunes the MLLM, the notion of the trajectory abstraction, and the experiments evaluating the effectiveness of the proposed approach.

**Strengths**

Reviewers noted the following strengths of the work:
1. Decision making in embodied area is important [QSXR]
2. Proposed model of retrieval-augmented MLLM agent and the idea of trajectory abstraction seems to be useful and effective [CWhu,EV8c,QSXR,7byC]
3. Sufficient experiments demonstrating the effectiveness of the proposed approach [CWhu,EV8c,QSXR,7byC]
4. Clear and well-structured paper [QSXR]

**Weaknesses**

During the review process, the reviewers noted the following weaknesses.
1. Scope of experiments is relatively limited 
   - More challenging scenarios [CWhu]
   - Additional baselines [EV8c]
2. Lack of information on computational and resource costs [CWhu,7byC]
3. Limited information on how user interactions are collected and processed [CWhu]
4. Additional details about implementation [CWhu, EV8c], evaluation [7byC] and discussion of limitations [CWhu,7byC]
5. Limited discussion of how prior owrk tackled the embodied grounding problem [EV8c] and clarifications of innovations [QSXR]

Many of these where addressed by the authors during the rebuttal period, causing the reviewers to increase their scores.


**Recommendation**

The reviewers are overall positive on the work, and their concerns were addressed during the author response period.  The AC agrees with the assessment of the reviewers that the work is interesting and well-presented, and recommends acceptance.

Decision: Accept (Poster)