MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agents
Authors: Junpeng Yue
Xinrun Xu
Börje F. Karlsson
Zongqing Lu
Venue: ICLR 2025 Poster (2025)
Keywords: multimodal retrieval
interactive learning
MLLM embodied agent
Primary area: foundation or frontier models, including LLMs

Abstract: MLLM agents demonstrate potential for complex embodied tasks by retrieving multimodal task-relevant trajectory data. However, current retrieval methods primarily focus on surface-level similarities of textual or visual cues in trajectories, neglecting their effectiveness for the specific task at hand. To address this issue, we propose a novel method, MART, which enhances the performance of embodied agents by utilizing interaction data to fine-tune an MLLM retriever based on preference learning, such that the retriever fully considers the effectiveness of trajectories and prioritize them for unseen tasks. We also introduce Trajectory Abstraction, a mechanism that leverages MLLMs' summarization capabilities to represent trajectories with fewer tokens while preserving key information, enabling agents to better comprehend milestones in the trajectory. Experimental results across various environments demonstrate our method significantly improves task success rates in unseen scenes compared to baseline methods. This work presents a new paradigm for multimodal retrieval in embodied agents, by fine-tuning a general-purpose MLLM as the retriever to assess trajectory effectiveness. All the code for benchmark tasks, simulator modifications and the MLLM retriever is available at https://github.com/PKU-RL/MART.

Published as a conference paper at ICLR 2025
MLLM AS RETRIEVER: INTERACTIVELY LEARNING MULTIMODAL RETRIEVAL FOR EMBODIED AGENTS
Jumpeng Yue"; Xinrun Xu², Börje F. Karlsson", and Zongqing Lu
School of Computer Science, Peking University
Institute of Software, Chinese Academy of Sciences
BBejjing Academy of Artificial Intelligence
ABSTRACT
MILM agents demonstrate potential for complex embodied tasks by retrieving multimodal task-relevant trajectory data. However, current retrieval methods pri- marily focus on surface-level similarities textual or visual cues trajectories, neglecting their effectiveness for the specific task hand. To address this issue, we propose a novel method, MLLM As ReTrieever (MART), which enhances the performance of embodied agents by utilizing interaction data to fine-tune MLLMM retriever based on preference learning, such that the retriever fully consid- the effectiveness of trajectories and prioritize them for unseen tasks. We also introduce Trajectory Abstraction, a mechanism that leverages MILMss summa- rization capabilities to represent trajectories with fewer tokens while preserving key information, enabling agents to better comprehend milestones in the trajec- tory. Experimental results across various environments demonstrate our method significantly improves task success rates in unseen scenes compared to baseline methods. This work presents a new paradigm for multimodal retrieval in embod- ied agents, by fine-tuning general-purpose MLLM the retriever to assess tra- jectory effectiveness. All the code for benchmark tasks, simulator modifications and the MLLM retriever is available at http://gthobbcom//KUUU//MMM
(1) Similarity-Base Based Retriever
(2) MLLM As Retriever
of
Of
i
of
0
00
0
of
CLIP
Sentence Transformer
MLLM As ReTriever (MART)
Max Similarity
Similarity
Trajectory Effectiveness Score
Current Task
Expert Trajectory
Trajectory Embedding Similarity
Figure 1: Similarity- Based Retriever vs. MART. Traditional multimodal retrieval methods (1) de- pend on calculating weighted sums of image and text embedding similarities, while our approach (2) introduces interactive learning to assess the relevance between the current and expert trajectories.
INTRODUCTION
Embodied agents interacting with complex environments require understanding both the current context and task-specific domain knowledge perform effectively (Wang et al., 2023c; Lifshitz et al., 2023). Recently, Multimodal Large Language Models (MLLMs), which are capable of pro- cessing both textual and visual data, have shown promise in various embodied tasks egg. table manipulation (Handa et al., 2023; Chen et al., 2022a), robot navigation (Zhang et al., 2024b; Shah al., 2022), and 3D games (Wang et al., 2024a; Tan et al., 2024; Jiang al., 2024). However, such
"Junpeng Yue and Xinrun Xu work as interns at BAAL "oorrsponndecce Zongqing Lu <zongqing. www.uuu..k..............

---

Published as a conference paper at ICLR 2025
models typically lack effective grounding in the embodied environments in which agents operate, greatly limiting their performance in embodied tasks (Long et al., 2024; Wang et al., 2024c).
To mitigate this limitation, providing additional task-relevant grounding information is essential to better leverage the general capabilities of MLLMs. Trajectory data, consisting of sequences of ac- tions and observations, can be easily available and provide valuable insights into task execution (Zheng et al., 2024; Zhao et al., 2024), therefore serving as a good information source for ground- ing. By using trajectory data in prompting an MLLM, the embodied agent can readily leverage previous experiences to better guide agents through similar tasks in new situations or environments (Zhang et al.. 2024a; Lee et al.. 2024). However, retrieving the most effective trajectories those that can significantly enhance task performance remains challenge, particularly when multiple trajectories appear similar in both textual and visual modalities (Jeurissen et al., 2024).
Existing retrieval methods mainly focus on surface-level textual or visual similarities of trajectories, often neglecting key aspects critical for task effectiveness, e.g., trajectory with a similar task in- struction but in different scene, or one in the same scene but with different layout. In such cases, these trajectories fail to provide useful information for the current task and can mislead the agent. As shown in Figure 2, relying solely on similarity is not effective in retrieving useful trajectories, as similarity does not directly correlate with success rate. To better support agents in embodied tasks, trajectory retriever model needs to consider the effectiveness of trajectories for given task.
1.0
0.8
0.8
0.8
Success
0.6
0.6
0.4 Success
0.6
0.6
0.4-
0.4
0.2
-70 -15
0.9
Embedding Similanty
Effectiveness Score
Embedding Similarity
Effectiveness Score
(a) A12-THOR
(b) LEGENT
Figure 2: Scatter plots illustrating the relationship between success rate and embedding similarity (left) or effectiveness score (right) in two environments. The red line indicates a linear fit to the data.
To achieve better retriever, we propose a new paradigm that integrates interactive learning with the retriever. Firstly, we consider expert trajectories of training scenarios as prompt for an MLLM agent, and let the agent interact with the environment to collect different success rates for different such reference trajectories. This interactive feedback data is then organized into preference pairs, which are used fine-tune tune an MILM LLaVA (Liu et al., 2023) in our case with a Bradley-Terry model (Bradley & Terry, 1952), such that the fine-tuned retriever model is capable of prioritizing more effective trajectories for unseen tasks. Combining this functionality with the inherent general capabilities of MLLM allows embodied agents to operate more effectively in unseen environments by leveraging their most useful past experiences.
We also introduce new Trajectory Abstraction mechanism, which uses MLLMs' summarization capabilities to represent trajectories in a reduced number of tokens, while preserving key informa- tion and enabling agents better understand such information in the trajectory (e.g., key relevant overarching actions). This mechanism is especially important in long-horizon tasks, both reducing the required context window length and removing distracting information from trajectory samples.
Combining the aforementioned components, we present our approach MART (MLLM As ReTriever) which adapts embodied agents in unseen scenarios by fine-tuning MILMM through preference data. To assess the benefits of our method, we conduct empirical experiments across diverse environments. The experimental results show that MART achieves significantly higher task success rates compared to baselines, demonstrating its effectiveness. With this approach, we present a new paradigm for multimodal retrieval in embodied agents, fine-tuning a general- -arepol-purpose MILM as a retriever capable of considering trajectory effectiveness. Our contributions can be summarized as follows:
To the best of our knowledge, MART is the first approach that integrates interactive learn- ing with retriever and uses interactive feedback to fine-tune an MILM retriever in eval-
2

---

Published as a conference paper at ICLR 2025
uating trajectory effectiveness, combining its inherent general capabilities with the ability to assess the task-guiding effectiveness of trajectories.
We introduce Trajectory Abstraction, a new mechanism that utilizes MLLM capabilities to significantly condense trajectories. This method reduces the token number while retaining essential information, allowing agents to effectively use this condensed knowledge novel situations and provide guidance for long- monghoorioon tasks.
The effectiveness of MART is empirically validated through comprchensive experiments in various environments, demonstrating significant performance improvements on unseen tasks. MART consistently surpasses baselines by over 10% across different environments.
2 RELATED WORK
2.1 EMBODIED AGENTS BASED ON LARGE MODELS
Recently, there have been several attempts to utilize the general-purpose capabilities of large mod- els for complex embodied tasks. These efforts can be broadly categorized into two types: VLA models and LLM/MLLM-
agents. 1) VLA models, including PaLM-E (Driess et al., 2023), RT-2 (Brohan al., 2023), Gato (Reed et al., 2022). VIMA (Jiang al., 2022), and MOO (Stone et al., 2023), rely on trajectory data to train a Transformer- explicitly constructing memory. However, their generalization capabilities limited due to the inherent issue of catastrophic forgetting in neural networks. 2) LLM/MLMM-
VLM for action planning, with-
agents, like Voyager (Wang et al., 2024a) and DEPS (Wang al., 2023b) for Minecraft, Cradle (Tan al., 2024) (Song et al., 2023) for ALFRed, and Code-as-Policies (Liang et al., 2023)
for RDR2, for real-world embodied control, do not involve directly training new models. Instead, they leverage the general-purpose capabilities of LLM/MLLM primarily through prompt engineering, Most of these agents build and maintain comprehensive memory systems to assist in task completion. How- ever, memory retrieval mostly focuses surface-level similarity, overlooking actual effectiveness in completing complex tasks.
2.2 MEMORY RETRIEVAL IN AGENTS
Agents can continuously learn and improve by recalling task-related experiences (Zhang et al., 2024c; Xi et al., 2023). During interactions with the environment, two main types of information are stored in memory. 1) Semantic information: Early LLM agents faced limitations due input token constraints, leading to reliance on short-term memory and greedy strategies (Chen et al., 2024; Zhang et al., 2023; Abdelnabi et al, 2023; Wang et al., 2023a). However, summarizing memory over short periods risked information loss (Light et al., 2023; Kaiya & et al, 2023). While storing comprehensive memory (semantic, episodic, procedural) can provide great value, its effective uti- lization for pocisiionmmking remains challenging (Sumers et al., 2024). More recent approaches try to prioritize relevant memories based on embedding similarity or LLLM-asedd relevance identifica- tion (e.g., (Park & et al., 2023), (Hong & et al., 2023), (Lin et al., 2023a), (Wang et al., 2024b), and (Xu et al., 2023)). 2) Image information: Recent advances in MILM agents have greatly en- hanced their grounding abilities by allowing image memory retrieval during actions. As illustrated in Figure 1, Similarity-Base Based Retrievers for agents, e.g., (Zhou et al., 2024; Wang et al., 2023c; Li al., 2024), leverage image embedding. while Groot (Cai et al., 2023) encodes visual and temporal information from video frames for guiding actions. But neither considers direct task effectiveness.
3 INTERACTIVELY LEARNING MULTIMODAL RETRIEVAL
3.1 PROBLEM FORMULATION
In this study, we investigate interactions a retrieval-augmented MLLM agent with an environment to complete embodied tasks drawn from a specific distribution. Figure 3 provides overview of our MILM As ReTriever approach MART. The agent is assigned a task instruction E" sampled from the task instruction distribution p(f), and operates over finite horizon H. At each timestep t E 1, 1.2,.. 2...... . H, the agent selects an action a, from the action space A based on the current observa- tion of from the observation space O and reference trajectory 7f retrieved from the expert trajectory
3

---

Published as a conference paper at ICLR 2025
Observation
Task Instruction Navigate
Expert Trajectories
One-Shot Retrieval
cup.
An Expert Trajectory
Component
Trajectory Abstraction
Feedback teeee you do ... you see
Instruction: Menepate
up
Vision
Vision Projector
Identify significant milestones related to current task progress.
Action: LockDown, Rotate
Encoder
Trajectory instruction - Abstract Feedback
Text Tokenizer
Self Reflection
Bradley- Reward Head
Analyze if last successful, providing detailed reasoning.
T2
15
2
Milestone: Openingthe Overarching Action: Open extenet ft and take bettle
Action Planning
Analyze current situation determine the next action, providing
Success Detection
Determine the task successful,
reasaning.
the information.
(Pos, Neg, Pairs
Figure 3: Overview of MART. Our approach interactively learns a multimodal retriever to score expert trajectories and retrieve most effective trajectory guide agent in novel situations. By considering trajectories with higher success rates as positive samples and those with lower success negative trajectories, obtain the preference pairs, which are used fine-tune MLLM retriever to score trajectory effectiveness for a specific task.
memory M. The memory contains of multimodal expert trajectories M {of, -.,,. : where each trajectory of includes task instructions f sampled from the same task instruction dis- tribution p()). The trajectory of also contains observation sequences of = {or ....... og }. and action sequences af {ai, ".,,,.,,,
The agent follows a frozen policy (aae", r",o"), implemented as a Multimodal Large Language Model (MILM), and the reference trajectory plays significant role in grounding the agent within the embodied environment, supporting task accomplishment. This reference trajectory is retrieved through an MLLM retriever qa.
We fine-tune our MILM retriever on training task distribution Prain and evaluate its performance on test task distribution (f), which has no overlap with training tasks. This retriever aims to identify and retrieve trajectory 7 from the expert memory pool that is most effective for the current task f", i.e., which can help ground the MLLM agent with a specific embodied task and enable its effective completion. is worth noting that we have different memories M and M which corresponds to training tasks and test tasks, respectively.
3.2 MEMORY
To enable trajectory retrieval for task execution and fine-tuning our MLLM retriever, first struct memory databases containing expert trajectories from previous successful executions for tasks both from Prain (f) and Prea (f). For each trajectory, of = { ; of. of represents a task & completed in H, steps and comprises the sequence of observations of. and corresponding actions a².
In multi-modis environments, such as AI2-THOR (Kolve et al., 2017) and LEGENT (Cheng al., 2024), each timestep observation includes an egocentric image. Moreover, in the A12-THOR environment, we assign numerical IDs (e.g. Cup 1) to all objects in the current visible field of view to identify target objects for interaction. These IDs appear in the environment feedback output in natural language, which is also part of the observation.
Expert trajectories collected via planner-based method (Hoffmann & Nebel, 2001). Storing these trajectories allows the agent to later leverage past experience when facing new task instances. Trajectory data is collected independently for the training and test sets. For each task, we initial- ized task instance and used a planner-baal based method to collect expert trajectory data. It is worth mentioning that the initialization position and orientation in each task is randomly chosen.
4

---

Published as a conference paper at ICLR 2025
Since each task is directly corresponds one unique task in the task set, the size of memory used for experiments is relatively small. For example, in the experiments performed in the LEGENT environment, the memory pool consisted of 40 trajectories for training, and distinct 32 trajectories during testing. More details are available in the experimental settings (Section 4.1).
3.3
MULTIMODAT. RETRIEVER
The core of MART is the innovative use of interactive learning to train the trajectory retriever. For an embodied task, different trajectories stored in memory can be provided references to the MILMM agent, leading to varying effects on the completion of the current task, depending on the degree of grounding with the environment they provide. Even if trajectory has text instruction similar the current task or an image sequence similar to the initial egocentric observation of the task, it does not guarantee that this trajectory can provide effective grounding. This is due to plain similarity alone not being able to reflect the effectiveness of the trajectory for the embodied task. For instance, failed trajectory for a related task could have high textual and visual similarity to the target task.
In order to retrieve the trajectory that can provide the most benefit (i.e., effective grounding) for the current task from the trajectory memory, we propose an interactive learning method for the MILM retriever. Specifically, for each task in training set, we sample K trajectories from the training mem- ory Mirrai and feed them as prompt for MLLM agent to execute the embodied task respectively. After that, based on the induced success rates of task execution, we get the effectiveness of each trajectory for the embodied task. We can then obtain a partial order list based on success rate comparisons, producing (K) pairs through pairwise comparison, where the trajectory with a higher success rate is treated as the positive item, and the one with a lower success rate as the negative item. In this way, these preference pairs from interactive feedback are arranged as a positive-negativy pair dataset D, which we use fine-tune the MLLM according to the Bradley-Terry (Bradley & Terry, 1952) Reward Modeling loss to enhance its critiquing ability, as in Equation 1:
(0) = -E(e, oG AfrA)
[log (a (go(E", A(==()=)) - go("", of A(T))))
(1)
In particular, we use VLaVA-7B (Liu et al., 2023) as base MILM and add Bradley-Terry score head based on hidden states of base model output. The score head one-layer MLPP that takes as input the last token in the hidden state and outputs a scalar score. The input to the MLLM retriever includes the trajectory abstraction result of expert trajectory T; A(T") current observation of. current task instruction f", and prompt for to judge the effectiveness of this trajectory for the current task and state. Upon inference, the MLLM outputs the score of the trajectory, which indicates its effectiveness for the current task. We select the trajectory with the highest score as the reference for the embodied agent to complete the current task.
3.4 TRAJECTORY ABSTRACTION
A complete multimodal trajectory often has dozens of timesteps, which correspond to dozens of observations, actions, and feedback, including redundant or irrelevant information. Furthermore, inputting all trajectory tokens, especially image tokens, to the MLLM retriever and agent will likely lead to confusing the models/agent or even exceeding their context windows.
We thus use another MLLM (in our experiments, GPT-4o) to automatically create an abstract tra- jectory in zero-shot manner. The initial input the MLLM is the trajectory 7 (consisting of task instruction f", observation sequence d², and action sequence a²) as well as the current task instruc- tion é and we let the MLLM find whether each observation contained in the trajectory 7" helpful for the current task g If it is considered to be useful, we will keep the observation into the resulting trajectory abstraction A(r*).
To be more specific, we let the MILM comprehend the tasks accomplished in the given trajectory 7", and then identify important observations in the trajectory as milestones and preserve them into the resulting trajectory abstraction. These milestones are steps that are essential for accomplishing the trajectory task of é" such as steps where important decisions are made, goals are achieved, or notable changes in the environment state occur. Also if the target object of current task instruction te appears in the trajectory feedback, then the step where appears is also considered to be a significant milestone as it is strongly related to current task ge. The milestone output format consists
5

---

Published as a conference paper at ICLR 2025
of: 1) description of the milestone; 2) the corresponding image (fimage x)); 3) the corresponding feedback (ffeedback x)); and 4) the overarching actions taken between this milestone and the next one.
The summarized trajectory manages to remove redundant information without affecting relevance for the current task E that the agent better receive grounding information contained in the trajectory. Quantitatively, for the tasks the test set, the input trajectories of trajectory abstraction have average of 1..51 steps, while the output milestones have an average count of 3.13.
4 EXPERIMENTS
4.1
EXPERIMENTAL SETUP
Table 1:
Environment complexity comparison. A12- THOR contains more than four times the number of inter- active objects per scene, and more complex object hier- archy, compared to LEGENT. E.g., A12-THOR supports relationships such as "inside" (e.g.. inside microwave (Figure 10a), or inside an open container, like sink (Fig- ure 10c)).
(a) A12-THOR
A12-THOR
LEGENT
Avg. Objects/Scene Object Hierarchy Layout Complexity
47.30
11.13
Inside, On
On
High
Low
(b) LEGENT
Task decomposition Observation
Yes
No
Image, Feedback
Image
Figure 4: Environment comparison.
4.1.1 ENVIRONMENTS
To validate the effectiveness our method in various environments, we perform evaluations on mul- tiple scenarios in two environments, A12-THOR (Kolve et al., 2017) and LEGENT (Cheng et al., 2024). Unlike L.M agents, which use only text as input and simplify the action space by employing teleportation actions (e.g. "move directly to target"), we believe that MILM agents should under- take more challenging tasks as they have access to visual input and are no longer limited a 'blind' mode of operation. Therefore, both target environments are multimodal, whose observations egocentric images, and both make use fine-grained control actions.
A12-THOR simulates embodied household tasks supporting natural language instructions and ego- centric visual observations, where agents must navigate and interact with various household items within realistic 3D environments, including kitchens, living rooms, and other indoor spaces. it al- lows agents to perform fine-grained navigation actions including 'move ahead', "turn left/right x degrees', 'look up/down', and interactive atomic action including 'pick up object A', 'put object A on/in object B', "open/close object A', 'toggle on/off object A'.
LEGENT is designed imitate human activities and tasks in home environments, including cross- room navigation. The action space is similar to AI2-THOR's, including fine-grained movement actions. also includes 'speak' action, which sends a message the user.
Table and Figure 4 show comparison between the two environments. Besides having many more objects in its scenes, A12-THOR supports more complex object hierarchy relationships, such as "inside" (e.g.. "inside a cabinet", which require open/close interaction to complete task, or "a cup inside a sink", i.e., an open container), not supported in LEGENT
4.1.2 TASK SETTINGS
Notably, in all tasks, the initial position and orientation of the agent is chosen randomly, Each task tested times to reduce the impact of random errors.
Although there cabinets in L.GENT, they do not need to be opened/closed complete tasks.

---

Published as a conference paper at ICLR 2025
A12-THOR. To better assess the fine-grained control ability of MLLM agents to complete real- world embodied tasks, we integrate characteristics of two A12-THOR -based benchmarks ALF- World (Shridhar al., 2021) and ALFRed (Shridhar et al., 2020) and built environment setting that more suitable for the MILM agent. Since tasks are long-horizon, we follow the method in ALFRed and apply task decomposition tion. Each sub-task is then provided to the agent, and it determines sub-task success based on environmental feedback by itself (details in Appendix F.2 and F3 ) Once a sub-task success- fully completed, the agent proceeds to the next sub-task. Task types include pick Lack_clean Saan__the__llaa_
divide them into sub-tasks before execu-
lack_and__laa_e sick_cool____ee aechen_ppace__ pla,, and pick Becck_hhhat__ee what_ethnn_ppaeteeeeee
Completing these tasks requires dozens of steps of navigation, well as interaction with objects. There are tasks comprising a total of 260 sub-tasks in training set, and 28 tasks including 158 sub-tasks in testing set.
LEGENT. Tasks for the MILM agent are categorized into two types: 'Come Here' and 'Where Is'. Each task is further divided into 'One-room' and 'Two-room' types, based on whether it requires traversing between rooms. In LEGENT, task decomposition not performed as task instructions simpler and do not contain combinations of sub-tasks; ie., the granularity of tasks is similar to that sub-tasks in A12-THOR, To train the retriever, we use 40 tasks (10 tasks for each task type) and we 32 tasks, also covering all task types, test set.
4.1.3 MEMORY CONSTRUCTION
Memory initialization follows the procedure described in Section 3.2; with randomized starting positions.
A12-THOR. Once trajectories are collected, we decompose them into sub-task trajectories (fol- lowing the task decomposition procedure in ALFRed) and treat each sub-task level trajectory as a expert trajectory into memory. Similar redundant trajectories then filtered out, accounting for about one-third of total, resulting in collection of 170 memory trajectories for the training memory and 118 trajectories for the testing memory.
LEGENT. As decomposition is not necessary due to the simpler tasks in this environment, the training memory is initialized with 40 trajectories, and the test memory with distinct 32 trajectories; one trajectory per task.
4.1.4 TASK EVALUATION
To evaluate the effectiveness of the retrieval-augmentee embodied agents, we assess their perfor- mance using two metries: Success Rate (SR) and Average Steps (AS).
Success Rate denotes the percentage of tasks attempts successfully completed by the agent. In A12-THOR it indicates the percentage of completed full tasks, and we additionally SR-Sub to represent the percentage of completed sub-tasks.
Average Steps represents the average number of steps the agent takes complete task. In A12- THOR it indicates the number of steps to complete a full task, and AS-Sub represents the step average complete sub-task. Notably, for failed cases, their steps counted the step limit.
4.1.5 BASELINES
It is worth noting that MART is the first work to retrieve multimodal trajectories as references for embodied MLLM agents and let the agent directly output fine- reee-rained control actions. We compare MART against three bascline methods to explore the performance of our approach:
Plain-Agent (PA) is an embodied MLLM agent without making use reference trajectories, i.e., without any memory. In experiments,
GPT-4o (2024-05-13 version).
LLa LLaVA-Plain (LP) is a pre-trained LLaVA with no modified head and no finetuning. We use the probability of special token generation to represent the score. Its input is the same MART, and it is prompted to output only Yes/No tokens, and the final score calculated based on the probability of token generation (more details and limitations in Appendix E.3).

---

Published as a conference paper at ICLR 2025
Table 2: Performance comparison of different methods in A12-THOR.
PA
LP
SL
RAP
MART
SR ?
0.18
0.26
0.24
0.22
0.40
SR-Sub
0.63
0.69
0.68
0.67
0.75
AS
159.66
144.18
147.48
147.03
123.19
AS-Sub 4
44.65
39.88
40.79
40.67
34.07
Table Performance comparison of different methods in LEGENT.
PA
LP
SL
RAP
MART
SR
0.70
0.69
0.75
0.75
0.87
AS
23.62
25.01
20.92
20.62
13.81
Table Ablation studies of MART in the A12-THOR and LEGENT environments.
Enviromment
Metric
w/o Abstraction
SIM.+FTM
MART
SR f SR-Sub
0.31
0.34
0.40
0.73
0.74
0.75
A12-THOR
130.26
125.20
123.19
AS-Sub 4
36.03
34.63
34.07
f
0.77
0.77
0.87
LEGENT
AS
18.48
18.83
13.81
Similarity+LLLa (SL) is reasonable retrievallrannking approach. Such approach is common in the text retrieval field - e.g., Sun et al. (2023b;a); Dong et al. (2024) and it can take into account both similarity and effectiveness. We use similarity choose the top-K candidate trajectories, and then choose the most likely effective one using a plain LLaVA model (i.e., same as LP).
RAP (Kagaya al., 2024) performs retrieval based plain similarity per modality and is the most similar setting in literature. It mainly targets text modality experiments in the ALFWorld (Shridhar al., 2021) and WebShop (Yao et al., 2022) (treated text-only) environments, and simple tasks in the multimodal environments Franka- Franka-Kitchen Kitchen (Gupta et al., 2019) and World (Yu et al., 2019).
4.2 A12-THOR
We firstly demonstrate the effectiveness of MART over test tasks the A12-THOR environment. The experimental results demonstrate the effectiveness of our approach compared to the baselines. As shown in Table 2, MART surpasses all baselines over 10% Success Rate, and reaches best performance across all metrics.
4.3 LEGENT
We then conduct experiments in the LEGENT environment. This environment includes tasks involv- ing crossing between rooms, thereby enriching the experimental space and demonstrating the effec- tiveness of our method. The experimental results, shown in Table 3, demonstrate MART greatly surpasses all baselines in all four task types.
4.4 ABLATIONS
of ablation studies to examine the contribution of key components in MART. More specifically, we aim to answer the following questions.
In this section,
Q1. Does Trajectory Abstraction indeed improve embodied agent performance?
We compare the full MART approach and the ablation of removing its Trajectory Abstraction mod- ule (w/o Abstraction). As shown in Table 4, MART consistently reaches the best results across settings. Even if in AI2-THOR sub-tasks the average success rate is comparable, the improvement margin leads to percentage points improvement in full task success rate.
8

---

Published as a conference paper at ICLR 2025
Navigate the Plate, pick up the Plate.
Current Task
2008
Navigate to the Kettle, and pick up
Similarity- based Retrieval
Navigate the Plate, up the Plato.
Experimental Result
Navigate Disháponge, then Plate on
MART Retrieval Abstraction
Navigate the Plate, pick up the Plato.
Experimental Result
Figure 5: Comparison between similarity- -based retriever and MART.
Q2. How does the MART approach compare against typical retrieve and rank approach, even if it uses fine-tuned ranking model for trajectory usefulness?
Table 4 shows decomposing the unified MART approach into separate retrieval and ranking steps (i.e., similarity-based retrieval and MART's MLLM as ranker Sim.+FTM) decreases success rates across settings. It is also interesting note that Sim.+FTM outperforms both SL and RAP (Tables and 3), further illustrating MART's trajectory utility scoring.
4.5
CASE STUDY
We present two case studies for more in-depth discussion of MART's capabilities handling chal- lenges of the MLLM agent setting. The first (Figure 5) demonstrates the effective handling of a very long-horizon trajectory. Given 73-step task trajectory "navigate to DishSponge, put it in the Plate, and place the Plate the Cabinet" Trajectory Abstraction identifies key milestones. MART achieves 80% success rate with an average of 28 steps, while both the similarity- baased method and the agent without memory reach only 40% success rate, averaging 69.6 and 74.6 steps, respectively. We provide more detailed and balanced case studies in Appendix D.
The second case (Figure 6) shows how MART extracts implicit rules for long sequence tasks. For the task "put the Potato into the microwave, heat it, and pick it up", Trajectory Abstraction analyzes agent's actions (including exploration, attempts, and success) and generates an abstract set of inferred rules, pruning non-contributory and redundant actions, reducing 13 transitions to just 6. MART achieves an 80% success rate with an average 30.2 steps, while other methods have 0% success rate.
CONCLUSION
We propose MART, a new paradigm for trajectory retrieval incorporating interactive learning, to enhance embodied agents' performance by providing them with task-relevant trajectory data. Our approach utilizes interaction-base feedback to identify the most effective trajectories, and con- structs preference pairs based on the comparisons between trajectories. An MLLM retriever is fine- tuned through these preference pairs, effectively prioritizing the trajectories that improve task per- formance. We also introduce Trajectory Abstraction in MART, novel mechanism that leverages

---

Published as a conference paper at ICLR 2025
the micrawave, heatit, then pick up.
Current Task
potato into miorowave, it, pickit up.
MART
Retrieval
mug into the microwave, heat than pick
Experimental Result
Put the microwave, heat it, then pick up.
TAC
Trajectory Abstraction
Put the microwave, heot pick up.
Experimental Result
Figure 6: Showcase of the significance of the Trajectory Abstraction mechanism.
MLLMs' summarization capabilities, to abstract trajectories, i.e., reduce the required number of to- kens represent them, while preserving key information and enabling agents to better understand relevant information. Experimental results in different environments demonstrate that our method significantly enhances task success rates in unseen tasks, compared to multiple baselines. This work helps bridge the gap between general-purpose MLLMs and the specific requirements of embodied tasks, offering a new paradigm for multimodal trajectory retrieval for embodied agents.
DISCUSSIONS AND FUTURE WORKS
Our study has a few limitations. First, the MILM's restricted context window limits its ability to process multiple images simultaneously, restricting to one-shot learning with single trajectory inputs. Future work will explore few-shot learning to combine skills from multiple trajectories for enhanced performance in complex tasks.
Second, the lack of detailed ablation study will be addressed by evaluating each component's contribution, including self-reflection mechanisms, prompt designs tailored specific functions in action planning, and retrievers based on diverse base models.
Third, ensure fair comparisons, will construct large-scale, high-quality datasets specific the household domain and fine-tune general feature extractors on these datasets to provide a more fair and direct comparison with similarity retrieval methods.
Fourth, model transfer stability across environments within the same domain needs validation. There are significant differences when comparing the household domain with the open-ended sandbox game domain and the web domain, including variations in task nature, operational frequency, ob- ject morphology, and textual input complexity. Therefore, we will train retrievers for the web, open-ended sandbox game, and household domains, respectively, to evaluate their generalization capability in novel environments and scenarios within each domain.
Lastly, the experimental scenarios in this study are somewhat limited, experiments in the household domain. In future work, we plan to extend our approach to the web and open-ended sandbox game domains by collecting preference data through interactions and training retrievers for each domain. Subsequently, we will evaluate the generalization capabilities within the same domain, but in unseen environments and scenarios.
have only conducted
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by NSFC under Grant 62450001 and 62476008. The authors would like to thank the anonymous reviewers for their valuable comments and advice.
REFERENCES
Sahar Abdelnabi and Amr Gomaa et Cooperation, Competition, and Maliciousness: LLM-- Stakeholders Interactive Negotiation. WxXiv:2309.7225: 2023.
Constructions Aeronautiques, Adele Howe, Craig Knoblock, ISI Drew McDermott, Ashwin Ram, Manuela Veloso, Daniel Weld, David Wilkins Sri, Anthony Barrett, Dave Christianson, et al. Pddl- the planning domain definition language. Technical Report, Tech. Rep., 1998.
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(334))324-345, 1952.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choroman- ski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. R1-2: Vision-language-action models transfer web knowledge robotic control. arXiv preprint arXiv:2307. 158188 2023.
Shaofei Cai, Bowei Zhang, and et al. Groot: Learning to follow instructions by watching gameplay videos. FrXiv:2310008235: 2023.
Soravit Changpinyo, Jordi Pont-Tuset, Vittorio Ferrari, and Radu Soricut. Telling the what while pointing to the where: Multimodal queries for image retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 12136-12148 2021.
Jiaqi Chen, Yuxian Jiang, Jiachen Lu, and Li Zhang. S-agents: self-organizing agents in open-ended environment. arXivv2402.0445 240.0404..04 2024.
Tao Chen, Jie Xu, and Pulkit Agrawal. A system for general in-hand object re-orientation. In Conference on Robot Learning. pp. 297-307. PMLR, 2022a.
Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William W. Cohen. Murag: Multimodal retrievalaaugmented generator for open question answering over images and text. Conference on Empirical Methods Natural Language Processing, EMNLP, 2022b.
Zhili Cheng. Zhitong Wang, Jinyi Hu, Shengding Hu, An Liu, Yuge Tu, Pengkai Li, Lei Shi, Zhiyuan Liu, and Maosong Sun. LEGENT: Open platform for embodied agents. In Yixin Cao, Yang Feng, and Deyi Xiong (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume: 3: System Demonstrations), 2024.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.0 240.00400,. 2024.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multi- modal language model. arXiv preprint arXiv:2303.03376 2023.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space bind them all. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp. 15180-15190, 2023.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving horizon tasks via imitation and reinforcement learning. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), 3rd Annual Conference on Robot Learning, CoRL 2019, volume 100 Proceedings of Machine Learning Research, pp. 1025- 1037. PMLR, 2019.
11

---

Published as a conference paper at ICLR 2025
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning, pp. 18..8866 1861-1870. PMLR, 2018.
Ankur Handa, Arthur Allshire, Viktor Makoviychuk, Aleksei Petrenko, Ritvik Singh, Jingzhou Liu, Denys Makoviichuk, Karl Van Wyk, Alexander Zhurkevich, Balakumar Sundaralingam, et al. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp- 5977-5984. IEEE, 2023.
Jörg Hoffmann and Bernhard Nebel. The ff planning system: Fast plan generation through heuristic search. Journal of Artificial Intelligence Research, 14:253-302, 2001.
Sirui Hong and Mingchen Zhuge et MetaGPT: Meta programming for multi-agent collaborative framework. arXiv:2308.0000 00352,, 2023.
Aashi Jain, Mandy Guo, Krishna Srinivasan, Ting Chen, Sneha Kudugunta, Chao Jia, Yinfei Yang. and Jason Baldridge. MURAL: Multimodal, multitask representations across languages. In Find- ings of the Association for Computational Linguistics: EMNLP, Punta Cana, Dominican Repub- lic, 2021.
Dominik Jeurissen, Diego Perez-Liebana, Jeremy Gow, Duygu Cakmak, and James Kwan. Playing nethack with 1lms: Potential & limitations as zero-shot agents. arXiv:2-03.006900 2024.
Haobin Jiang, Junpeng Yue, Hao Luo, Ziluo Ding, and Zongqing Lu. Reinforcement learning friendly vision-language model for minecraft. European Conference on Computer Vision (ECCV), 2024.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yonggiang Dou, Yanjun Chen, Li Fei- Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:2210.0300 2022.
Tomoyuki Kagaya, Thong Jing Yuan, Yuxuan Lou, Jayashree Karlekar, Sugiri Pranata, Akira Ki- nose, Koki Oguri, Felix Wick, and Yang You. Rap: Retrieval-augmented planning with contextual memory for multimodal Ilm agents. arXiv preprint arXiv:2402.03610, 2024.
Zhao Kaiya and Michelangelo Naim al. Lyfe agents: Generative agents for low-cost real-time social interactions. arXiv:2310.02172, 2023.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference empirical methods natural language processing (EMNLP), pp. 787-798, 2014.
Taewoong Kim, Cheolhong Min. Byconghwi Kim, Jinyeon Kim, Wonje Jeung, and Jonghyun Choi. Realfred: An embodied instruction following benchmark in photo-realistic environments. In European Conference Computer Vision, pp. 346-364. Springer, 2025.
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-add-langeee transformer without convo- lution or region supervision. In International conference on machine learning, pp. 5583-5594.
PMLR, 2021.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:
2017.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9579-9589, June 2024.
Sunjae Lee, Junyoung Choi, Jungjae Lee, Munim Hasan Wasi, Hojun Choi, Steven Y. Ko, Sangeun Oh, and Insik Shin. Explore, select, derive, and recall: Augmenting Ilm with human-like memory for mobile task automation, 2024.
12

---

Published as a conference paper at ICLR 2025
Patrick Lewis, Ethan Perez. Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih. Tim Rocktäschel, et al. Retrieval- -aggeennd genera- tion for wwwlledge-intensiv nlp tasks. Advances in Neural Information Processing Systems, 33: 9459-9474, 2020.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems, 34:9694-9705, 2021.
Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, and Liqiang Nie. Optimus-
1: Hybrid multimodal memory empowered agents excel in long-horizon tasks. arXiv preprint 1rXiv:2408.03611 2024.
Jacky Liang Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. 2023 IEEE International Conference Robotics and Automation (ICRA), pp. 9493-9500, IEEE, 2023.
Shalev Lifshitz, Keiran Paster, Harris Chan, Jimmy Ba, and Sheila A. Mcllraith. STEVE-1: A generative model for text-to-behavior minecraft. In Thirty-seventh Conference on Neural In- formation Processing Systems (NeurlPS), 2023.
Jonathan Light, Min Cai, Sheng Shen, and Ziniu Hu. Avalonbench: Evaluating Ilms playing the game of avalon. In NeurlPS 2023 Foundation Models for Decision Making Workshop, 2023.
Jiaju Lin, Haoran Zhao, Aochi Zhang, Yiting Wu, Huqiuyue Ping, and Qin Chen. Agentsims: An open-source sandbox for large language model evaluation. arXiv:2308.04400 2023a.
Weizhe Lin, Jinghong Chen, Jingbiao Mei, Alexandru Coca, and Bill Bymee Fine-grained late- interaction multi-modal retrieval for retrieval augmented visual question answering. In Thirty- seventh Conference on Neural Information Processing Systems (NeurIPS), 2023b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems, 36, 2023.
Yuxing Long, Xiaoqi Li, Wenzhe Cai, and Hao Dong. Discuss before moving: Visual language navigation via multi-expert discussions. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 17380-17387, 2024.
Man Luo, Zhiyuan Fang, Tejas Gokhale, Yezhou Yang, and Chitta Baral. End-to-end knowledge retrieval with multi-modal queries. Association for Computational Linguistics (ACL), 2023.
Varun K Nagaraja, Vlad I Morariu, and Larry S Davis. Modeling context between objects for referring expression understanding. In Computer Vision-ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pp. 792-807. Springer, 2016.
Joon Sung Park and Joseph O'Brien et al. Generative agents: Interactive simulacra of human be- havior. In The 36th UIST, 2023.
Scott Reed, Konrad ZoIna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06/73 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1=X.vv1707.0634: 2017.
Dhruv Shah, Blazej Osinski, Brian Ichter, and Sergey Levine. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. In Karen Liu, Dana Kulic, and Jeffrey Ichnowski (eds.), Conference on Robot Learning, CoRL 2022, volume 205 of Proceedings of Machine Learning Research, pp. 492-504. PMLR, 2022.
Sahel Sharifymoghaddam, Shivani Upadhyay, Wenhu Chen, and Jimmy Lin. Unirag: Universal retrieval augmentation for molli--odal large language models. arXiv preprint arXiv:240.. 10311, 2024.
13

---

Published as a conference paper at ICLR 2025
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In Proceedings of the International Conference Learning Representations (ICLR), 2021.
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. Lim-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), October 2023.
Austin Stone, Ted Xiao, Yao Lu, Keerthana Gopalakrishnan, Kuang-Huek Lee, Quan Vuong, Paul Wohlhart, Brianna Zitkovich, Fei Xia, Chelsea Finn, et al. Open- object manipulation using pre-trained vision-language models. arXiv preprint arXiv:2303. 00905, 2023.
Theodore R. Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L.. Griffiths. Cognitive archi- tectures for language agents. Transactions on Machine Learning Research (TMLR), 2024.
Weiwei Sun, Zheng Chen, Xinyu Ma, Lingyong Yan, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Instruction distillation makes large language models efficient zero-shot rankers. arXiv preprint arXiv:2311.015555 2023a.
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang. Pengjic Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is chatgpt good at search? investigating large language models as re-ranking agents. arXiv preprint arXiv:2304. 3xX::2304409542, 2023b.
Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Jun- peng Yue, Jiechuan Jiang, Yewen Li, Ruyi An, Molei Qin, Chuqiao Zong. Longtao Zheng, Yujie Wu, Xiaoqiang Chai, Yifci Bi, Tianbao Xie, Pengjie Gu, Xiyun Li, Ceyao Zhang, Long Tian, Chaojie Wang. Xinrun Wang, Börje F. Karlsson, Bo An, Shuicheng Yan, and Zongqing Lu. Cradle: Empowering Foundation Agents Towards General Computer Control. arXiv preprint 12Xi::2403.03166 2024.
Guanzhi Wang. Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Transactions on Machine Learning Research, 2024a.
Noah Wang, Z.y. Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, Man Zhang. Zhaoxiang Zhang, Wanli Ouyang, Ke Xu, Wenhao Huang. Jie Fu, and Junran Peng. RoleLLM: Benchmarking, eliciting, and enhancing role-playing abilities large language models. Findings of the Association for Computational Linguistics (ACL), 2024b.
Shu Wang, Muzhi Han, Ziyuan Jiao, Zeyu Zhang, Ying Nian Wu, Song-Chun Chun Zhu, and Hangxin Liu. Lim3:large language model-basee task and motion planning with motion failure reasoning, 2024c.
Zhilin Wang, Yu-Ying Chiu, and Yu Cheung Chiu. Humanoid agents: Platform for simulating human-like generative agents. In Yansong Feng and Els Lefever (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP. pp. 167-176. As- sociation for Computational Linguistics, 2023a.
Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. In ICML, 2023b.
Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. JARVIS-1: Open-world multi- task agents with memory- memory-augmente augmented multimodal language models. In Second Agent Learning in Open-Enndedness Workshop, 2023c.
14

---

Published as a conference paper at ICLR 2025
Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir Training and benchmarking universal multimodal information retrievers. arXiv preprint arXiv:2311.17136, 2023.
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.077 077844 2023.
Weiye Xu, Min Wang. Wengang Zhou, and Houqiang Li. P-rag: Progressive retrieval augmented generation for planning on embodied everyday task. In Proceedings of the 32nd ACM Interna- tional Conference on Multimedia, pp. 6969-6978, 2024.
Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Exploring large language models for communication games: An empirical study on werewolf. arXiv preprint Ttt1::3000046555 2023.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real- world web interaction with grounded language agents. Sanmi Koyejo, S. Mohamed, Agar- wal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, 2022.
Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. Retrieval-augmente multimodal language modeling. International Conference on Machine Learning (ICML), 2023.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta- world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL). 2019.
Haoqi Yuan, Zhancun Mu, Feiyang Xie, and Zongqing Lu. Pre-training goal-based models for sample-efficient reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024.
Ceyao Zhang, Kaijie Yang, Siyi Hu, Zihao Wang, Guanghe Li, Yihang Sun, Cheng Zhang, Zhaowei Zhang, Anji Liu, Song-Chun Zhu, Xiaojun Chang, Junge Zhang, Feng Yin, Yitao Liang, and Yaodong Yang. Proagent: Building proactive cooperative agents with large language models. a30iv:2308. 1 11339, 2023.
Jiayi Zhang, Chuang Zhao, Yihan Zhao, Zhaoyang Yu, Ming He, and Jianping Fan. Mobileexperts: A dynamic tool-enabled agent team in mobile devices, 2024a.
Jiazhao Zhang, Kunyu Wang, Rongtao Xu, Gengze Zhou, Yicong Hong, Xiaomeng Fang, Qi Wu, Zhizheng Zhang, and He Wang. Navid: Video-based vim plans the next step for vision-and- language navigation, 2024b.
Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. A survey on the memory mechanism of large language model based agents. arXiv preprint 2xXiv2244.13301, 2024c.
Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang, Expel: Llm agents are experiential leamners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19632-19642, 2024.
Longtao Zheng, Rundong Wang, Xinrun Wang. and Bo An. Synapse: angjectory-as-xxeppe prompting with memory for computer control. In The Twelfth International Conference on Learn- ing Representations (ICLR), 2024.
Enshen Zhou, Yiran Qin, Zhenfei Yin, Yuzhou Huang, Ruimao Zhang, Lu Sheng, Yu Qiao, and Jing Shao. Minedreamer: Learning to follow instructions via chain-of- io-iiiinatiinnninnnn for simulated- world control. arviv preprint arXiv:2403. 12037, 2024.
15

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
In this section, we provide more implementation details about the model, training process and im- plementation pipeline.
A.1 MODEL AND TRAINING DETAILS
We transform a generative language model (MLLM) into trajectory scoring model by replacing the language model head with a Bradley-Tere score head. In particular, both the original language model head and our proposed Bradley-Terry scoring head are single-layer MLPs. However, there are notable differences: the language model head processes all hidden states as input and generates a probability distribution over the vocabulary for each token, facilitating token sequence generation through sampling. In contrast, hidden state as input and outputs a single floating -ointt score. Using this approach, our model generates only one new token time (i.e., by setting 'max new tokens' 1). In comparison to conventional MLLM training, which generates hundreds or even thousands of new tokens per iteration, our model's training significantly more computationally efficient. All the code for the MLLM retriever model, training process, benchmark tasks and simulator modifications is available https://github.comm wwt.s/coo///./M/////
Bradley-Terry scoring head relies solely the last non-zero
During training, we firstly fine-tune LLaVA to enable it to understand multiple images, details in E. I. After that, we replace the language model head with the Bradley-Tery score head, and fine-tune the model with Iora. The parameter settings are listed in 6.
A.2 DETAILS IMPLEMENTATION PIPELINE
In this section, we provide more details about the whole pipeline of implementation, including the data collection and retrieval in downstream tasks. The implementation pipeline is as follows:
1. Construct memory databases containing expert trajectories via the planner-bas based method, details in 3.2.
2. Collect the pairwise comparison data via interactive feedback to train the retriever model.
(a) Specifically, for each task in the training set, sample K trajectories from the train- ing memory Mitain and feed them as prompts for MLLM agent to execute the em- bodied task respectively.
(b) After that, based on the induced success rates task execution, we can get the effec- tiveness of each trajectory for the embodied task. We can then obtain partial order list based success rate comparisons, producing (K) pairs through pairwise com- parison, where the trajectory with higher success rate is treated as the positive item, and the one with lower success rate the negative item.
(c) In this way, these preference pairs from interactive feedback are arranged as a positive- negative pair dataset D, which we use to fine-tune the MILLM according to the Bradley-Tery (Bradley Terry, 1952) Reward Modeling loss to enhance its critiquing ability, in Equation 1. Details are listed in Appendix 3.3 line 249-259.
3. Train the modified retriever model using the preference data. Details are listed in A.I.
4. Evaluation unseen tasks. For each unseen task, we first retrieve trajectory with highest score for current task. Then the retrieved trajectory will be simplified through Trajectory Abstraction module. After that, the MLLM agent will execute the task with the help of abstraction of retrieved trajectory. Details are listed in A Algorithm 1.
B EXTENSION OF RELATED WORKS
B. EMBODIED GROUNDING
Grounding is a critical challenge in embodied agents, referring to the alignment between the agent and its environment.
16

---

Published as a conference paper at ICLR 2025
The grounding problem can be categorized into visual grounding and embodied grounding. Visual grounding (Lai et al., 2024; Kazemzadeh et al., 2014; Nagaraja et al., 2016) addresses the problem at the perception level by identifying the most relevant object region in image based on language query, whereas embodied grounding focuses on the effects of actions on environmental dynamics and how an agent generates action sequences accomplish a given task. Approaches to addressing embodied grounding can be categorized into the following types:
Reinforcement learning (RL) trains an agent's policy through interaction with the envi- ronment, making the agent inherently grounded in the environment, such as PPO (Schulman et al., 2017) and SAC (Haamoja al.. 2018). However, RL typically requires extensive interaction with environments and often suffers from instability, making it unsuitable for MLLMs.
1. RL:
VLA: These methods focus on fine-tuning vision-language models (VLMs) using expert datasets collected from embodied environments, such as PaLM-E (Driess et al., 2023) and RT-2 (Brohan et al., 2023). These methods demand a significant amount of high-quality trajectory data for training-
3. LLM as Planner: These methods leverage Large Language Models (LLMs) or Multimodal Large Language Models (MLLMs) to generate high-level plans, which are then translated into ex- ecutable action sequences by low-level controllers, such as LLLM-Planner (Song et al., 2023) and P-RAG (Xu et al., 2024). A key limitation of these methods their reliance on predefined skill library, which restricts the scope of the agent's capabilities. Besides, acquiring skill library might require additional RL or IL training or prior knowledge about the environment (Lifshitz et al., 2023; Yuan et al., 2024).
This category involves integrating task trajectory data into the prompts provided to MLLMs, such as RAP (Kagaya et al., 2024). These trajectory data, rich in grounding information about the environment, enable agents to perform tasks effectively. Retrieval- aerrenel-augmmented methods usually demonstrate greater sample efficiency compared to RI. and VLA, thanks to the use of an explicit memory buffer. Our work falls into this category.
4. Retrieval-Augmented MILM Agent:
B.2 MULTI-MODAL INFORMATION RETRIEVAL
Recent advances in multimodal retrieval have developed various methods for encoding, fusing, and measuring similarities across different modalities. ViLT (Kim al., 2021) directly embeds image patches with text using a Transformer, while ALIGN (Li et al., 2021) and MURAL (Jain et al., 2021) use dual-encoder architectures with EfficientNet and BERT to align modalities through con- trastive learning. IMAGEBIND (Girdhar et al., 2023) extends this by creating joint embeddings for six modalities, using ViT and Transformer models. Furthermore, (Changpinyo al., 2021) in- tegrates users' mouse trace interactions for refined image retrieval, while ReViz (Luo et al., 2023) employs advanced encoding mechanisms for visual question answering. Building on these encod- ing strategies, retrieval augmentation further enhances multimodal generation (e.g.. RAG (Lewis et al., 2020)). FLMR (Lin et al., 2023b) addresses RA-VQA limitations by combining multi- dimensional embeddings from ColBERTv2 and ViT-based models for accurate knowledge retrieval. RA-CM3 (Yasunaga et al., 2023) enhances image captioning and text-to-image generation by using pre- treitra CLIP model to augment inputs for a CM3 Transformer. Similarly. UniRAG (Sharify- moghaddam et al., 2024) integrates retrievals using UnilR's 's (Wei et al., 2023) CLIP Score Fusion and BLIP Feature Fusion, improving performance in MLLMs like LLaVA. Lastly, MuRAG (Chen et al., 2022b) introduces a retrieval-augmenter transformer for KB-VQA, employing T5 and ViT for multimodal encoding and retrieval from large-scale memory bank. In contrast, our method prior- itizes the effectiveness of retrieved information by employing interactive learning, ensuring that the information contributes directly task completion.
C EXTENSION EXPERIMENTS
We also evaluate our method on ReALFRED (Kim et al., 2025) environments, which provides re- alistic 3D-captured and multi-room scenes, as shown in Figure 7. The action space is similar with AI2-THOR, including fine- grainer movement actions, e.g. 'move ahead', 'turn left/right x degrees', 'look up/down', and interactive atomic action including 'pick up object A', 'put object A on/in object B', 'open/close object A', 'toggle on/off object A'. Since completing tasks often involves navigat- ing across rooms, and the scenes within these rooms closely resemble real-world environments, this
17

---

Published as a conference paper at ICLR 2025
setting offers more diverse and challenging scenario. The chosen task type is 'pick_and.place which requires the agent to first navigate to the target object, pick it up, and then transport it to the designated location for placement. There are 30 tasks comprising total 60 sub-tasks in training set, and 20 tasks including 40 sub-tasks in testing set.
Figure Image examples of ReALFRED.
The experimental results demonstrate the effectiveness of our approach compared to the baselines. As shown in Table MART surpasses all baselines 10% in Success Rate, and reaches best perfor- mance across all metrics.
Table 5: Performance comparison of different methods in ReALFRED.
PA
L.
SL
RAP
MART
SR t
0.25
0.20
0.26
0.27
0.37
SR-Sub f
0.50
0.44
0.52
0.53
0.58
AS J.
101.70
87.93
89.79
87.89
78.48
AS-Sub &
28.43
24.32
24.84
24.31
21.71
D DETAILED CASE STUDY
We present more detailed case studies, encompassing both success and failure cases for each method, along with simplified reasoning processes for clarity.
In the first detailed case, as shown in Figure the MART Retriever successfully retrieved trajec- tory containing the target object's location, while Trajectory Abstraction effectively compressed 73-step trajectory into only significant milestones, preserving crucial information. For the success- ful trial, the agent identified the target object's 's location (the plate on the table) through the retrieved trajectory. After some exploration, it successfully found the target object and completed the task. For the unsuccessful trial, during exploration, the agent made mistakes (highlighted in purple) and failed to complete the task within the step limit.
In the second detailed case, shown in Figure 9, the similarity-based retriever retrieved trajectory that appeared similar but lacked useful information. As result, the agent had to explore indepen- dently. By chance, the agent located the target object, but the overall success rate remained low.
E EXPERIMENTAL SETUP DETAILS
In this appendix, provide more low-level details on the implementation of MART experiments.
E. MULTIPLE IMAGE INPUT IN LLAVA
The LLaVA architecture itself is compatible with using multiple images as input, but the released model weights do not have the ability to handle multiple images. In other words, when you input multiple images, will only focus on the contents of the first image. Therefore, we fine-tune LLaVA through multi-image captioning data enable it to understand multiple images.
18

---

Published as a conference paper at ICLR 2025
Navigate to the Plate. and pick up the Plate.
Current Task
Navigate to DishSpange, put the Plate, put Plate the Cabinet.
MART Retrieval
Abstraction
Unsuccessful Trajectory
Successful Trajectory Step 01:
Step 001:
Action: RotateRight_55 90
Action: RotateRight_ 90
Reasoning:
Reasoning:
trajectory suggests looking around and exploring, so rotating to see more of area.
Reference trajectory tells plate location and
provides general procedure for interacting.
Step 02:
Step 024:
Action: RotateRight, 90
Reasoning: Plate found. trajectory suggests continue rotating explore the kitchen.
Action: MoveAhead
Reasoning: visible;
exploring find potential plate locations.
Step 03:
Step 054-057:
Action: RotateRight, 90 Reasoning: Plate not yet found, rotate right for broader view.
Step 04:
Open cabinet Open drawer 3, Open drawer 9, Open drawer 7, Open cabinet
Action:
Action: Take plate 1
Reasoning: Plate not found; Reference trajectory suggests continue exploring nearby containers/surface
Reasoning: Plate located, pick up.
Step 05:
Step 068:
Finished.
Action:
Take
Reasoning: Plate visible open cabinet; try to pick the plate.
Step 074-100:
Action:
Open drawers cabinets
Reasoning:
Plate not found; continue inspecting remaining
drawers.
Figure 8: Comparison between unsuccessful and successful trajectory with MART.
We utilized LLaVA's single-image perception capabilities to enhance its multi-image perception. Specifically, sampled images collected from the environments and employed the pre-trained LLaVA to describe the content of each image. We organized the responses into multi-image cap- tioning data, which then used fine-tune LLaVA. After our verification, the fine-tuned LLaVA demonstrated the ability to perceive multiple input images. After fine-tuning with multiple image datasets, LLaVA serves the basez model for our subsequent retriever training.
19

---

Published as a conference paper at ICLR 2025
Navigate to the Plate, and pick up the Plate.
Current
Task
8006 Similarity-
Navigate Kettle, and pick the Kettle
based Retrieval
14 steps
Unsuccessful Trajectory
Successful Trajectory Step 01:
Step 001:
Action: RotateRight, 90
Action: LookDown
Reasoning: Reference trajectory's steps don't involve process navigating the plate. Rotating to explore the environment and locate the plate.
Reasoning: Current observation and reference trajectory do not reveal a plate.
Looking down first adjust
Step 024:
Step 04:
Action: Take plate
Action: RotateRight_
Reasoning: Plate visible the table, try to pick up the plate.
Reasoning: Plate not visible; rotating view to inspect the countertop.
Step 21:
Step 034:
Action: RotateRight,
Action: Open cabinet
Reasoning: Having failed to find the plate cabinet drawers; Checking other nearby locations, like the sink.
Reasoning: visible; trajectory suggests opening containers.
Step 22:
Step 054:
Action: RotateRight
Action: Open drawer
Reasoning: Plate not found. Since the nearby drawer is empty. should check
Reasoning: Reference trajectory provides the same logical approach of exa swamining storage places remains valid. The logical step inspect the other drawers.
other nearby storage spaces.
Step 23:
Step 064:
Action:
Take
Action: MoveAhead
Reasoning: Plate found. Since the nearby drawer empty, should check other nearby storage spaces.
trajectory shows systematic rotation and movements help locate objects. Therefore, moving ahead might provide a better view even reveal another storage could contain the plate.
Reasoning:
Step 24:
Step 100:
Task Finished.
Action: LookUp
Reasoning: Reference trajectory indicates moving around the kitchen Searching for the plate by inspecting other potential storage areas looking around.
Figure Comparison between unsuccessful and successful trajectory with similarity- based re- trieval.
E.2 PARAMETER SETTINGS
The specific parameter settings in the experiment shown in Table 6.
20

---

Published as a conference paper at ICLR 2025
Table Hyperparameters of LLaVA fine-tuned by LoRA
Hyperparameters
Value
LLaVA version
llava-vI 6.-mmsttal-m
train hii_hbatch_sze
32
eval_batch_stt
8
gradient.accumulltion...
8
Carning-rate_AI2THHB
2e-5
mm-projector IITTTTTOO AI2THOO
2e-5
learning rate LEGENT
3e-6
projector Ir LEGENT
3e-6
lora_r
16
lora_alpha
32
warmup ratio model max length Ir tr.scheduler.tye type vision
0.05
32768
cosine
tower vision_tower
clip-vit-large-patch144 14-336
E.3 DETAILS OF LLAVA-PLAIN
Building on existing work (Asai al., 2024) (Sun et al., 2023b) that employs LLM for text retrieval, we use the generation probability of special token to represent the score for LLaVA-Plain. In detail, the effectiveness score s; is measured by the probability of LLaVA-Plain to generate the special token "Yes" and 'No', as in Equation 2, where (Yes/No) denoted the probability of LLaVA-Plain generate Yes or No. I.
p Yes)
8; (Yes) + (No)
(2)
In detail, the prompts use for LLaVA-Plain and MART are presented in prompt I of Appendix I.
F A12-THOR ENVIRONMENT SPECIFICS
F.1 SETTING DIFFERENCES
Two popular benchmarks, ALFRED (Shridhar et al., 2020), and ALFWorld (Shridhar et al., 2021) derived from ALFRED both built on A12-THOR. However, none of them are directly suitable for benchmarking MILM agents performing real-world tasks.
ALFRED is multimodal benchmark in A12-THOR that uses fine-grained navigation actions. How- ever. it requires pixel-level masks to specify objects for interaction actions. MLLM lacks the ca- pability to generate such pixel-level masks, making ALFRED incompatible with MLLM agents without adaptation at either side.
ALFWorld adapts ALFRED for LLM agents (i.e., text-only) by simplifying it. Firstly, provides text feedback as observation, detailing objects in the agent's 's field of view along with their corre- sponding IDs. Secondly, simplifies the action space by replacing all navigation actions with the teleportation action 'go to' and composite high-level actions like 'heat', 'clean', and 'cool', each involving multiple atomic interactions. For example, the "cool object action equivalent "open the refrigerator', 'put object A inside the refrigerator'. 'close the refrigerator', 'open the frigerator", and 'pick up the object A'. These modifications significantly reduce task difficulty, while allowing LLM agents perform in ALFWorld.
To better evaluate the fine- onained control abilities of MLLM agents in real-world tasks and longer- horizon realistic tasks, reject ALFWorld's setting approach, which uses teleportation and composite high-level actions, opting instead for fine-grained navigation actions and finer- grained actions. However, unlike in ALFRED, since MILMS cannot generate pixel-level masks by default,
21

---

Published as a conference paper at ICLR 2025
allow interaction actions to reference objects using numerical ID (e.g., cup 1) provided by the environment's 's feedback, instead of a pixel-level mask.
F.2 TASK DECOMPOSITION
We adopt the ALFRed (Shridhar et al., 2020) method to decompose the entire task into multiple sub-tasks. In ALFRed, tasks are decomposed as follows: First, they encode agent and object states, along with high-level environment dynamics, into Planning Domain Definition Language (PDDL) rules (Acronautiques et al., 1998). Next, they define task-specific PDDL goal conditions, such as a heated potato resting tabletop. The planner assumes fully observable environment with perfect knowledge of world dynamics. Consequently. each task decomposed into several sub-tasks, with instructions provided for each subtask by human labelers.
We follow the ALFRed method but adjust the decomposition results. PDDL-based decomposition can result in inconsistent sub-task difficulty. For example, sub-task like 'pick up object A' or 'put object A on object B', which often follows navigation sub-task. If the previous navigation sub-task executed successfully, the current sub-task can be completed in one step according to the PDDL decomposition. We adjusted the task decomposition by merging sub-tasks that can be completed in one step with adjacent sub-tasks balance their difficulty. We will release the benchmark including our tasks and modified sub-tasks.
F.3 SUCCESS DETECTION
In A12-THOR, we provide the agent with the environment's metadata and feedback in the Success Detection module. After each step is executed, the agent then determines whether the current sub- task completed using a few-shot approach. For specific prompts, please refer to Appendix
F.4 HIERARCHY EXAMPLES
Figure 10 illustrates multiple cases of the object hierarchy relationship Inside, only available in the A12-THOR environment.
Moreover, Figure 11 illustrates the different types of rooms in our LEGENT experiments, showing connectivity, but simpler On hierarchies than A12-THOR.
(a) Microwave.
Cabinet.
(c) Sink.
(d) Coffee machine.
Figure 10: Image examples of object hierarchy in A12-THOR.
G FULL STEP COUNT RESULTS
Due to space limitations, present the full tables with Success Rate (SR) and Average Steps (AS) results here.
Table shows the performance comparison of different types in LEGENT, with full step count results. While, Table shows the performance comparisons of the ablation studies in both A12- THOR and LEGENT environments, with full step count results.
22

---

Published as a conference paper at ICLR 2025
(a) Where-2room.
(b) Where-Irone
(c) Come-2room.
(d) Iroom.
Figure 11: Image examples of different types of tasks in LEGENT.
Table 7: Performance comparison of different types in LEGENT, with full step count results.
Metric
Type
PA
LP
SL
MART RAP
Where-2room
0.65
0.60
0.60
0.73
0.88
Where-Iroom
0.78
0.80
0.80
0.89
0.91
SR
Come-2room
0.63
0.55
0.63
0.46
0.73
Come-Iroom
0.75
0.82
0.96
0.93
0.98
Average
0.70
0.69
0.75
0.75
0.87
Where-2room
25.23
28.98
28.30
26.23
14.03
Where-Iroom
10.40
16.49
15.36
9.82
5.07
AS
Come-2room
32.05
34.95
28.80
35.33
26.90
Come-Iroom
26.80
19.62
11.22
11.09
9.24
Average
23.62
25.01
20.92
20.62
13.81
H ALGORITHM
MART's Agent Execution Pseudocode is shown in Algorithm 1.
algorithm 1 MART Agent Execution Pseudocode
Input: Expert Trajectory Memory M, Retriever qa, Policy æ, Task & Horizon H. Initial Observa- tion of. Preference Pairs
Output: the Success status of task execution
1: Fine-tune retriever qu with Preference PairsD
2: Retrieve reference trajectory 7f <- qo (e of
3: TrajectoryAbssructionn to simplify e
4: for 1 to do
5: i! = then
6:
Ft f SelfReflection (a+ of 18-1) 7: Select action t (ee, 70, r1)
8:
end
9:
if (tt=11 1 then
10:
Select action a₂ - x(E,, - of) end if
11:
12:
(of:f2 + + ActionExxcction(a if SuccessDDtectton ( then
13:
14:
return True else if t H then
// Task successfully completed
16:
return False // Task failed after reaching horizon
end if
18:
end for
23

---

Published as a conference paper at ICLR 2025
Table 8: Ablation studies of MART in the A12-THOR and LEGENT environments, with full step count results.
Environment
Metric
w/o Abstraction
Sim.+FTM
MART
SR
0.31
0.34
0.40
SR-Sub f
0.73
0.74
0.75
A12-THOR
AS AS-Sub 1
81.22
78.09
78.48
22.47
21.60
21.71
Where-2room
0.78
0.72
0.88
Where-Iroom
0.74
0.89
0.91
SR
Come-2room
0.63
0.55
0.73
Come-Iroom
0.95
0.91
0.98
Average
0.77
0.77
0.87
LEGENT
Where-2room
17.36
15,96
14.03
Where-Iroom
12.36
11.51
5.07
AS +
Come-2room
32.75
34.13
26.90
Come-Iroom
11.47
13.73
9.24
Average
18.48
18.83
13.81
24

---

Published as a conference paper at ICLR 2025
I AGENT PROMPTS
Prompt 1: Retriver Prompt to LLaVA.
You are a highly intelligent vision language assistant agent situated in a virtual environment.
You have been given a task instruction that need to complete.
Additionally, you are provided with reference trajectory, which includes previous task instructions, actions, and egocentric observations from the sane virtual environment.
This reference trajectory represents a successful completion of task and is intended to guide you in performing the current task. The current task instruction is: Icurrent task)
The task instruction Among these input images, the 1st image is your current observation,
reference trajectory
[memory task).
while the other images are milestones of observations from the reference trajectory.
The abstraction of the reference trajectory is:
<Descripticn of Milestone 0>, <Image 0>, <Feedback 0>, <Action 0>;
<Description of Milestone 1>, <Image 1>, <Feedback 1>, <Action 1>;
Your should thoroughly understand the current task and the reference trajectory. Then, anal yze whether the reference trajectory can assist in executing the current task by answering "Yes' 'No' You should only respond in the format described below, and you should not output comments or other information: Answer: Yes or No.
25

---

Published as a conference paper at ICLR 2025
Prompt 2: Trajectory Abstraction Prompt.
You are highly intelligent visicn-language assistant agent placed
within virtual environment. You are provided with trajectory
consisting of:
Trajectory Task Instruction: (Task Instruction)
A sequence of first-person perspective observations (<Inage x>)
A sequence of environment teedbacks (<Feedback x>)
A sequence of actions (<Action x>)
Another Task Instruction: (Current Task Instruction)
Your Tasks:
1. Fully comprehend the tasks accomplished in the trajectory.
2. Identify the significant milestcnes in the trajectory that essential for accomplishing the task. These are points where important decisions are made, goals are achieved, or notable changes in the environment or state occur. Do not treat every image as significant milestone, If the target object of Another Task Instruction appears
the trajectory feedback, then the point where it appears is also a significans milestone because it very important to Another Task Instruction.
3. For each significant milestone, provide:
description of the milestone.
The corresponding image (<Image x>)
The corresponding feedback (<Feedback x>)
The sequence of actions taken between this milestone and the next one.
<Image 02, <Feedback 0%, <Action 0>;
<Image 12, <Feedback 1>, <Action 1>;
Response Format (do not include any comments additional information) (Description of significant milestone 1): <Image a>. <Feedback a>.
1.
Actions: (Actions taken between this significant milestone and the next fignificant fictnt. mi milestone) (such as <Action a>, <Action b>).
2. (Description of signi ficant mi lestone 2): <image c>. <Feedback c>. (Actions taken between this significant milestone and significant milestone:
Actions:
Notes:
Ensure that the number <Image x> and <Action x> matches the provided trajectory.
Only include the specifiee fied information in your response. A significam milestone is peint in the trajectory where key part of the task is accomplished. If the target object Another Task Instruction appears it appears is also a significant milestone because it is very important to complete Another Task
the trajectory feedback, then the point where
26

---

Published as a conference paper at ICLR 2025
Prompt 3: Self-Reflection Prompt.
You are a vision language assistant agent with high intelligence.
You are placed inside virtual environment, equipped to handle wide
range of tasks in the virtual environment.
Your acvanced capabilities enable you to process and interpret egocentric observation screenshots and environment feedback.
Your task is to examine these inputs, interpret the envirenmental feedback, and determing ne whether the executed action takes effect.
Current task:
inakkiintttttionn
Previous actions:
(previous_action_]]) (previous_action 2),
Previous environment feedback:
(previous_feccbackk
Reasoning for the previous actions: (previiou_rreasmnnn___l [previous_reassmng__]]
Previous observations:
iprevious_image 11, Iprevious image_ 21,
Reasoning: You need to answer the fellowing questions step by step to get some reasoning based on the previous actions and sequential images of the execution of the previous actions.
1. What is the last executed acticn?
2. Was the last executed action successful? Give reasons. You should refer to the following rules:
If the action involves movement of position, change of view, interaction with an object, it considered unsuccessful when the image you currently observe remains unchanged as the previous frame. 3. If the last action is not executed successfully, what is the most probable cause? You should give only one cause and refer to the following rules:
it is an interaction actien, the most probable cause was that the object id of the interaction was wrong.
If it is movement action, the mest probable cause was that were blocked by seen unseen obstacles.
4. the last action executed successfully, Does the previous action sequence promote the progress of the current task?
5. If the answer to Reasoning Question 4 is No, how should you adjust to promote the progress of the current task?
You should only respond in the format deseribed below, and you should not output comments or other information:
Reasoning:
4.
5.
27

---

Published as a conference paper at ICLR 2025
Prompt 4: Action Planning Prompt.
You are a rebot working in a household environment. You can move and interact with the objects you see. The actions you can perform include:
1. "Movehhead Move one step forward.
2. SeaateLLeit_degree Turn to the left by the specified number of degzees, ranging from 0 to 180 degrees.
3. sottateeightt_seereee Turn to the right by the speci fied number of degrees, ranging from 0 180 degrees.
4. Lookkpp: Look up 30 degrees.
5. LookDown': : Look down 30 degrees.
6. Take $cbjectID': You can take any object in your line of sight. The SobjectID can be obtained from the environment feedback.
7. Put SobjectID on/in StargetID': You can put the object in your hand onto/into the target receptacle, The StargetID can be obtained from the cnvirenment feedback, and $objectID refers to the object in your hand.
"Open $objectID':
open any openable object in your line of sight. The $object can be obtained from the environment feedback.
9. "cose ScbjectID's you can close any open object in your 1ine of sight The $objectID can be obtained from the environment feedback.
10. 'ToggleOn $objectID's You sucn as faucet or microwave. The $objectID can be obtained from the environment feedback.
toggle the switen of the object,
11. ToggleOOff $object You can toggle off the switch of the object, such faucet microwave. The $objectID can be obtained from the environment feedback.
12. 'slice SabjectID'; You can slice any object in your line of sight. The SobjectID can be obtained from the environment feedback.
Examples of actions: RotateLeft_30. KoveAhead; Take mug 1; Open fridge 1; ToggleOn microwave 1; close fridge 1;...
You need to follow the task instructions to complete the task. Here is some helpful information
Current task: (task)
Current environment feedback: (obs)
Previous environment feedback: (previous_cbs)
Previous action and reasoning: pprevious_action]
Current observation <image>
Previous observation <image>
Reference trajectory abstraction: The reference trajectory a successful trajectory, which is used to guide you to complete the current task. The task of reference trajectory is (current task).
<Description of Milestone 0>, <Image 0>, <Feedback 0>, <Action 0>; <Description of Milestone 1>, <Image 1>, <Feedback 1>, <Action 1>;
Based the above information, you should first analyze the current situation, and provide the reasoning for what you should for the
28

---

Published as a conference paper at ICLR 2025
next step to complete the task. Then, you should output the action you want execute the simulator.
Roasoning: You should think step by step and provide detailed reasoning to determine the next action executed on the current state of the task. You need
the following questions step by step.
1. Does reference trajectory abstraction exist? If the answer is no, ignore the questions from number 2 to number 4,
2. What process does reference trajectory abstraction describe?
3. Consider what is your current task. Hased the Observation
previous step and Current Observation, which waypoint has the current task reached?
4. Based on the answer of the question number 3, you should consider how the current waypoint and the parts after that in the reference trajectory abstraction can help you with your current task. The help provided by the reference trajectory abstraction can be knowing the locatien of the target object or knowing the execution flow of a combined action.
5. Based on the completion progress of the current task and the question number 4, what should you do for the next step?
6. Why do you take this action next step?
Action:
The best action to execute next to progress in completing the task. You should pay nore attention to the following action rules:
1. Given the current situation and task, you should only choose the most suitable action from the valid action set. You cannot use actions that are not in the valid action set to control the application, especially 'waait Next Task'.
2. the Action of the previous step fails, you should net continue trying but should consider adjusting your position to get closer to the target object.
3. You MUST NOT match imitate the reference trajectory. You should think about how complete the current task based on the answer Reasoning Question 4.
You should only respond in the fermat described below, and you shoutd d not output
or other information:
Reasoning:
1.
2.
3.
4.
5.
6.
Action:
29

---

Published as a conference paper at ICLR 2025
Prompt 5: Success Detection Prompt.
You are highly intelligent visicn-language assistant agent. You are situated in a virtual envirenment, equipped to handle a diverse array of tasks.
Your advanced capabilities allow yeu process interpret egocentric observation screenshots, eavironmental feedback and environmental metadata.
Your task is to examine these inputs, understand the envi ronmental metadata, and assess the success of the current task.
Current task: <task>
Environmental Metadata <environment metadata>
Environmental Feedback
cenvironment feedback>
Current Inventory <inventory>
You neec to refer to the follewing rules:
1. If the current task contains multiple tasks, it is considered successful only when each task succeeds.
2. the task is navigation task, you need to check the environmental metadata. The navigation task succeeds when the target object view and the distance is less than 1m.
3. If the task is pickup task, then according to the environmental feedback, the pickup task succeeds when the target object is in your inventery.
4. the task is put down task (put object a on/in object b), the put down task succeeds when the envi envi ronmental. feedback from the envi ronment includes You put object on/in the object b successfully ly
5. the task cleaned objects in environmental metadata include the target object and the same target object is also in inventory.
clean task, the clean task succeeds when the
6. If the task is cool task, the cool task succeeds when the target object and the same target object also inventory. 7. If the task is heat task, the heat task succeeds when the Seated_objecte in rovmenttetall metadata includes the target object and the same target object is also in inventory.
cooled_objects in environmental metadata includes
You should only respond in the format described below, and you should not output comments or other information:
Answer: True False.
30


Reviews:
[{"review_id": "PTyJrZdTlV", "reviewer": "Reviewer_CWhu", "summary": "The paper introduces a novel method, MLLM As ReTriever (MART), which enhances the performance of embodied agents by utilizing interaction data to fine-tune a multimodal language model (MLLM) retriever. The method aims to improve the effectiveness of retrieved trajectories for specific tasks, and it introduces Trajectory Abstraction to summarize trajectories while preserving key information. The experimental results show significant improvements in task success rates in unseen scenes compared to baseline methods.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a unique method for enhancing multimodal retrieval in embodied agents by leveraging preference learning and trajectory abstraction. This approach addresses a critical gap in current retrieval methods that often focus on surface-level similarities.\n2.  The introduction of Trajectory Abstraction is a valuable contribution. It effectively reduces the complexity of trajectories while maintaining essential information\n3. The experimental results are comprehensive and demonstrate the effectiveness of the proposed method across various environments. The improvements in task success rates in unseen scenes are particularly noteworthy.", "weaknesses": "1. While the paper demonstrates the effectiveness of MART in several environments, the scope of experiments could be expanded to include more diverse and challenging scenarios to further validate the robustness of the method.\n2. The paper does not provide a detailed analysis of the computational complexity and resource requirements of the proposed method. This information is crucial for practical implementation and comparison with existing methods.\n3. The paper mentions the use of interaction data but does not delve into the specifics of how user interactions are collected and processed. More details on this aspect would enhance the clarity and reproducibility of the method.\n4. Provide more detailed implementation guidelines and discuss potential limitations and solutions. This would make the method more accessible to other researchers and practitioners.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SzjTF2VM7K", "reviewer": "Reviewer_EV8c", "summary": "The paper presents a novel method, MLLM As ReTriever (MART), which aims to enhance the performance of embodied agents in complex environments by improving the retrieval of multimodal, task-relevant trajectory data. It uses expert trajectories as prompts for an MLLM agent, collects interactive feedback in the form of success rates, and organizes this data into preference pairs. These pairs are then used to fine-tune the MLLM retriever, enabling it to prioritize more effective trajectories for unseen tasks. The authors conducted experiments across two environments, demonstrating MART's success rates in unseen scenes.", "questions": "You can find all my inquiries detailed in the Weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper proposes a new retrieval-augmented MLLM agent, which finds the most matched expert reference trajectory to enhance current trajectory planning. To compress these multimodal trajectories with lots of images, actions, and feedback, Trajectory Abstraction (GPT-4o) is introduced to identify important trajectory milestones.\n\n2. The experimental results demonstrate the effectiveness of MART in improving task success rates across different environments.", "weaknesses": "I am not an expert in this field, but I have several problems:\n\n---\n\n1. This work aims to tackle the challenge of grounding in environments. However, the literature review focuses more on embodied agents, memory retrieval in agents, and MLLM. These areas encompass a wide range of topics and, as a result, **overlook numerous specific and significant studies concerning how past embodied grounding approaches have tackled this issue**. In summary, I am unable to identify key references within this manuscript, and the innovation it presents remains obscure to me.\n\n--- \n2. The proposed model is evaluated on AI2-THOR and LEGENT, with performance comparisons against Plain-Agent, LLaVA-Plain, Similarity+LLaVA, and RAP. However, it's worth noting that these comparative methods are predominantly variations of Multimodal Language Models (MLLM) or retrieval-based approaches, effectively serving as ablation studies. **This suggests that the absence of key comparisons with prior grounding techniques raises concerns regarding the efficacy of the proposed method's performance.**\n\n\n---\n3. What are the implementation details including model and training details?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s1kAmZwqGH", "reviewer": "Reviewer_QSXR", "summary": "This paper introduces MART, a novel trajectory retrieval paradigm leveraging interactive learning to enhance embodied agents’ task performance. MART utilizes interaction-based feedback to identify the most effective trajectories and constructs preference pairs based on trajectory comparisons. These preference pairs are used to fine-tune a Multimodal Large Language Model (MLLM) retriever, prioritizing trajectories that improve task outcomes. Additionally, MART introduces Trajectory Abstraction, a mechanism using MLLMs’ summarization capabilities to condense trajectory representations by reducing token counts while preserving essential information, enabling agents to better understand task-relevant details. Experimental results across diverse environments demonstrate that MART significantly improves success rates in unseen tasks compared to various baselines. This work bridges the gap between general-purpose MLLMs and embodied task requirements, offering a new paradigm for multimodal trajectory retrieval for embodied agents.", "questions": "1. It would be better if the authors could further explain the specific innovations in the pipeline compared to prior works LLM-Planner and P-RAG, highlighting how the proposed approach diverges or improves upon these method.\n2. Could the authors provide an explanation for HOW Trajectory Abstraction contributes to performance improvements? \n3. If incorrect trajectories are retrieved, could this lead to catastrophic performance loss?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The problem addressed in this paper—decision-making in embodied tasks—is a crucial area.\n2. The writing in this paper is clear and well-structured, with each paragraph presenting ideas in a logical, cohesive manner. The authors effectively communicate complex concepts, making the paper accessible and easy to follow.\n3. This paper introduces an innovative multimodal retrieval tool and selects appropriate baselines to validate the effectiveness of the proposed method.", "weaknesses": "1. The methodology in this paper shows some similarities with previous works, such as LLM-Planner (arXiv:2212.04088 ) and P-RAG(\tarXiv:2409.11279). While there are innovations in the retriever design, the overall novelty of the approach is somewhat reduced.\n2. The paper lacks some detailed analysis and deeper insights. For instance, the ablation study does not include experiments explaining HOW Trajectory Abstraction contributes to performance improvement", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WIAstIWmek", "reviewer": "Reviewer_7byC", "summary": "The paper introduces MLLM As Retriever (MART), a novel approach designed to enhance the performance of embodied agents in complex tasks by improving multimodal retrieval of task-relevant trajectory data. Traditional retrieval methods often rely on surface-level similarities of textual or visual cues, which may not effectively capture the nuances required for specific tasks. MART addresses this limitation by fine-tuning a Multimodal Large Language Model (MLLM) retriever through interactive preference learning, allowing it to evaluate and prioritize trajectories based on their effectiveness for unseen tasks. Additionally, the paper presents Trajectory Abstraction, a mechanism that leverages MLLMs’ summarization capabilities to represent trajectories with fewer tokens while preserving essential information, enabling agents to better understand key milestones. Experimental results across various environments demonstrate that MART significantly improves task success rates on unseen tasks compared to baseline methods, consistently surpassing them by over 10%. The authors commit to releasing all benchmark task sets and simulator code modifications to facilitate further research.", "questions": "1. Is the High Cost of Training an Additional MLLM Justified?\nCould you verify whether the benefits of training and using an additional MLLM outweigh the high costs in terms of additional inference time and required training resources? Are there specific scenarios where the MLLM retriever significantly outperforms conventional similarity retrieval methods to justify its use?\n\n2. How Did You Overcome Stability and Hallucination Issues?\nSince language models can be unstable and prone to hallucinations, what methods did you employ to mitigate these issues in your MLLM retrieval system? Have you verified the effectiveness of your approach across multiple evaluation methods to ensure reliable performance?\n\n3. Clarification on Evaluation Metrics in Table 4:\n\n\ta) What Exactly Does AS Represent? Could you clarify the meaning of AS (Average Steps) in your evaluation? Does it represent the number of iterations required for the agent to succeed once?\n\n\tb) Correlation Between AS and SR: Does Average Steps and Success Rate correlate? How do you interpret the results where AS shows minimal difference but SR differs more significantly?\n\n\tc) Fairness of Using AS as a Metric:\nConsidering the additional computational cost of incorporating an MLLM, would it be more appropriate to use the average inference time of the entire system instead of AS to ensure a fair comparison between methods?\n\n4. Can You Provide Balanced Case Studies?\nTo ensure a fair comparison, could you provide both the success and failure cases for each method (MLLM Retrieval and similarity retrieval) in your case studies? This would help understand each approach's strengths and weaknesses more comprehensively.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Innovative Retrieval Method: The paper introduces a novel method by fine-tuning an MLLM as a retriever using interactive preference learning, moving beyond traditional similarity-based retrieval methods. This allows the retriever to assess trajectory effectiveness more holistically.\n2. Trajectory Abstraction Mechanism: The introduction of Trajectory Abstraction is a significant contribution. By condensing trajectories into shorter representations while retaining crucial information, agents can process and utilize past experiences more efficiently.\n3. Demonstrated Performance Improvements: The method substantially improves task success rates across various environments and unseen tasks, indicating its effectiveness and potential for broader applications in embodied AI.\n4. Advancement in Embodied Agent Capabilities: By enhancing how agents retrieve and utilize past trajectories, MART contributes to developing more intelligent and capable embodied agents capable of handling complex, long-horizon tasks.\n5. Commitment to Open Science: The release of benchmark task sets and simulator code modifications promotes transparency and allows other researchers to build upon this work, fostering collaboration in the community.", "weaknesses": "1. Efficiency Issues: The proposed method of using a Multimodal Large Language Model (MLLM) as a retriever may not be as efficient as conventional similarity retrieval methods, known for being fast and low-cost. Given that complex language model reasoning can already achieve high-quality planning decomposition and performance, the added cost of training and deploying an additional MLLM may not be justified. There is a need for verification to determine if the high cost is worth the potential benefits, including considerations of additional inference time and required training resources.\n\n2. Potential Stability and Hallucination Problems: Replacing simple similarity calculations with judgments based on a language model introduces potential instability and hallucination issues inherent to language models. The MLLM’s retrieval process might be inconsistent, leading to unreliable action selection. It’s unclear how these issues are overcome in the system and whether effectiveness has been verified across multiple evaluation methods. Since MLLM retrieval effectively acts as a terminal evaluator to judge the best action, stability is critical.\n\n3. Experimental Confusion: In Table 4, the ablation study comparing MLLM Retrieval and similarity retrieval methods shows minimal differences in Average Steps (AS) but noticeable differences in Success Rate (SR). Questions arise about the correlation between these two evaluation metrics. Specifically:\n\n    a) Clarification of AS: What does AS (Average Steps) represent? Is it the number of iterations the agent system takes to succeed once?\n\n    b) Difference in Metrics: Why is there a significant difference in SR but not AS between the two methods?\n\n    c) Fairness of AS: Given the introduction of an additional language model, using AS may not be fair. It might be more reasonable to replace AS with the average inference time of the entire system to reflect the true cost and efficiency.\n\n4. Unfair Comparison in Case Study: The case study presents success cases of the proposed MLLM Retrieval method alongside failure cases of the similarity retrieval method. This comparison may not be fair or sufficient to demonstrate the advantages of the proposed method. Providing success and failure cases for each method would offer a more balanced and comprehensive evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)