MLLM as Retriever: Interactively Learning Multimodal Retrieval for
  Embodied Agents
Authors: name: Junpeng Yue, name: Xinru Xu, name: Börje F. Karlsson, name: Zongqing Lu
Organization: Beijing Academy of Artificial Intelligence

MLLM agents demonstrate potential for complex embodied tasks by retrieving
multimodal task-relevant trajectory data. However, current retrieval methods
primarily focus on surface-level similarities of textual or visual cues in
trajectories, neglecting their effectiveness for the specific task at hand. To
address this issue, we propose a novel method, MLLM as ReTriever (MART), which
enhances the performance of embodied agents by utilizing interaction data to
fine-tune an MLLM retriever based on preference learning, such that the
retriever fully considers the effectiveness of trajectories and prioritize them
for unseen tasks. We also introduce Trajectory Abstraction, a mechanism that
leverages MLLMs' summarization capabilities to represent trajectories with
fewer tokens while preserving key information, enabling agents to better
comprehend milestones in the trajectory. Experimental results across various
environments demonstrate our method significantly improves task success rates
in unseen scenes compared to baseline methods. This work presents a new
paradigm for multimodal retrieval in embodied agents, by fine-tuning a
general-purpose MLLM as the retriever to assess trajectory effectiveness. All
benchmark task sets and simulator code modifications for action and observation
spaces will be released.