FindingDory: A Benchmark to Evaluate Memory in Embodied Agents

## Abstract

Abstract Vision-Language models (VLMs) have recently demonstrated impressive performance in planning and control tasks, driving interest in their application to robotics. Yet their deployment in embodied settings remains limited by the challenge of incorporating long-term experience, often spanning multiple days and represented by vast image collections. Current VLMs typically handle only a few hundred images at once, underscoring the need for more efficient mechanisms to manage long-term memory in embodied contexts.
To meaningfully evaluate these models for long-horizon control, a benchmark must target scenarios where memory is essential. Existing long-video QA benchmarks neglect embodied challenges like object manipulation and navigation, which require low-level skills and fine-grained reasoning over past interactions. Moreover, effective memory integration in embodied agents involves both recalling relevant historical information and executing actions based on that information, making it essential to study these aspects together.
In this work, we introduce FindingDory , a new benchmark for long-range embodied tasks in the Habitat simulator. FindingDory evaluates memory-centric capabilities across 60 tasks requiring sustained engagement and contextual awareness in an environment. The tasks can also be procedurally extended to longer and more challenging versions, enabling scalable evaluation of memory and reasoning. We further present baselines that integrate state-of-the-art closed-source and fine-tuned open-source VLMs with low-level navigation policies, assessing their performance on these memory-intensive tasks and highlighting key areas for improvement. 1 1 1 Website: https://findingdory-benchmark.github.io/

## 1 Introduction

Figure: Figure 1: FindingDory: We propose a benchmark for evaluating agents’ memory by constructing scenarios that require reasoning over previously collected experiences. In these scenarios, an agent receives logs of prior interactions in indoor environments and must complete navigation, pick, and place instructions. The figure shows a sample episode: first, a privileged agent with full environment access executes randomly sampled navigation and manipulation tasks. Then, the memory-evaluating agent is placed in the same environment and tasked with completing instructions that rely on the privileged agent’s logs to decide where to navigate or which object to retrieve.
Refer to caption: figures/teaser.png

Memory is a fundamental capability for intelligent agents that allows them to recall past experiences, adapt to dynamic environments, and make informed decisions over extended timescales. In humans and animals, it plays a crucial role in navigation, reasoning, and goal-directed behavior. As we strive to develop more capable embodied systems, equipping them with robust memory mechanisms is essential, particularly in settings where agents must process high-dimensional multimodal inputs to interact with their surroundings.
The ability to store and retrieve relevant information is the key to unlocking sophisticated behaviors, from household assistance to autonomous exploration.

Recent advances in Vision-Language Models (VLMs) have significantly improved high-level reasoning and planning for embodied tasks. These models leverage large-scale multimodal training to exhibit strong scene understanding and action-guided perception. However, most existing VLM applications focus on short-term or static tasks, such as image captioning or Visual Question Answering (VQA), which do not require sustained memory usage. Extending these models to long-horizon embodied control introduces new challenges: embodied agents must integrate observations over time, recall past experiences, and act accordingly within their environment.

While recent efforts have sought to extend long-context understanding in VLMs, primarily through video-based QA or long-document comprehension (Song et al., 2023; Xiong et al., 2025; Ma et al., 2024), these approaches fail to fully capture the complexities of long-term memory in decision-making. Many existing QA benchmarks either rely on multiple-choice formats, which enable guessing, or require extensive human annotation to curate question-answer pairs that require models to attend to long video sequences. In contrast, we propose embodied memory tasks—such as “navigating to a soft toy that you did not rearrange yesterday” or “find the second object picked up earlier” that demand fine-grained recall and multi-hop reasoning over past observations. These tasks provide a more rigorous test of memory capabilities, requiring agents to track and respond to environmental changes rather than passively perceive static information.

In this paper, we introduce FindingDory, a benchmark designed to evaluate long-horizon memory in highly photorealistic simulated environments. Our benchmark consists of 60 diverse navigation tasks, which we categorize along various axes of memory requirements, that require both long-range temporal and spatial reasoning while discouraging heuristic shortcuts or random guessing. To ensure that performance is directly tied to memory utilization rather than simple perception, we curate scenarios where agents must recall past interactions to succeed. Additionally, our benchmark features dynamic environments - where agents modify the scene - making it essential to reason over changing contexts. Furthermore, FindingDory is procedurally extensible, allowing tasks to scale in complexity as VLMs improve, ensuring continued relevance as embodied AI progresses.

Unlike standard QA datasets that rely on subjective human annotations, our benchmark leverages photorealistic simulation to enable automatic evaluation of memory-based navigation. These tasks go beyond static recall since they require agents to perform complex spatio-temporal reasoning over past interactions. For instance, tasks such as “navigate to any object that you interacted with yesterday” demand not only accurate memory retrieval, but also strategic decision-making to identify and navigate to the nearest valid target from where the agent is currently positioned.
In FindingDory, we introduce metrics that quantify task completion efficiency and not just successful recall.

Our key contributions are:

- 1.
A benchmark for evaluating long-horizon memory in embodied decision-making, featuring 60 diverse navigation tasks in highly realistic indoor environments.
- 2.
A comprehensive evaluation of VLM based high-level policies combined with low-level navigation policies, analyzing their performance and limitations in memory-intensive tasks.
- 3.
A systematic and extensible evaluation framework with metrics that enable future advancements in memory-efficient embodied agents.

## 2 Related Work

In this section, we review related work on long-video question answering and embodied AI benchmarks that incorporate historical experience in their problem formulation. We defer the discussion of different approaches for augmenting policies with memory-like mechanisms to [section C.1](https://arxiv.org/html/2506.15635v2#A3.SS1).

### 2.1 Video QA benchmarks

Recent analyses of video QA benchmarks (Mangalam et al., 2023; Buch et al., 2022) show that many tasks can be solved by attending to key frames, not full video sequences, suggesting that current benchmarks may favor keyframe selection over evaluating long-term memory.

**Table 1: Comparison of FindingDory with popular memory-focused embodied AI benchmarks.**
| Benchmark | Photorealism | Semantic Reasoning | Task Specification | Template Categories | # Resulting Tasks | Memory Length ($\pi^{*}$) | Isolates Memory |
| --- | --- | --- | --- | --- | --- | --- | --- |
| MemoryMaze (Pašukonis et al., 2023) | ✗ | ✗ | One-hot | ✗ | $1$ | 500-1000 steps | ✗ |
| MemoryGym (Pleines et al., 2023) | ✗ | ✗ | One-hot | ✗ | $3$ | 128-512 steps (Extensible) | ✗ |
| MultiON (Wani et al., 2020) | ✓ | ✗ | One-hot | ✗ | $1$ | 500 steps | ✗ |
| SIF (Min et al., 2024) | ✓ | ✓ | Language | $3$ | $240$ (val/test) | ✗ (Uses semantic map) | ✗ |
| OpenEQA (Active) (Majumdar et al., 2024) | ✓ | ✓ | Language | ✗ | $1600$ | 75 steps | ✗ |
| GoatBench (Khanna* et al., 2024) | ✓ | ✓ | Language + Image | ✗ | $1800-3600$ (val) | 500 steps (Static Env) | ✗ |
| Excalibur (Zhu et al., 2023) | ✓ | ✓ | Language | $11$ | 21 | Requires exploration | ✗ |
| Ours | ✓ | ✓ | Language | $11$ | $\approx 6000$ (val) | 500-3500 steps (Extensible) | ✓ |

Long-context evaluations for foundation models also focus on “needle-in-a-haystack” tasks, evaluating retrieval of specific details rather than integrating information across many segments (Zhang et al., 2024). While recent video benchmarks feature more challenging questions (Mangalam et al., 2023; Li et al., 2023; Wang et al., 2024; Fu et al., 2024; Song et al., 2023; 2024; Li et al., 2023; Rawal et al., 2024; Yang et al., 2024; Zhou et al., 2024; Majumdar et al., 2024), they adopt multiple-choice QA formats which are not suitable for evaluating fine-grained decision-making over long histories. They may also allow models to succeed through guessing and are not extensible since they require substantial human annotation effort.

### 2.2 Embodied AI benchmarks

Various benchmarks have been proposed to evaluate multi-modal language models in embodied settings where agents must leverage their experience in environments to improve future decision-making (Wani et al., 2020).
Many focus on embodied question answering (EQA), such as HM-EQA (Ren et al., 2024), which requires active exploration, and Open-EQA (Majumdar et al., 2024), which tests VLMs in a passive setting with pre-collected episode histories.

Notably, Open-EQA found that a *blind* LLM baseline—one that ignores visual input—still performed well, suggesting that multiple-choice evaluation can introduce biases and reduce the need for genuine memory recall.
Other benchmarks study the situated instruction following setting where agents must gather information to interpret ambiguous instructions (Sashank Dorbala et al., 2024; Ma et al., 2023; Min et al., 2024). However, many allow agents to sidestep memory challenges by precomputing relevant objects and locations based on recent history, effectively turning potentially memory-intensive tasks into structured instruction-following problems. Memory-Maze and Memory-Gym (Pašukonis et al., 2023; Pleines et al., 2023) focus on partially observable decision-making with history-based state modeling, but their simplistic visuals and small state spaces limit their relevance to complex real-world memory settings.

In [table 1](https://arxiv.org/html/2506.15635v2#S2.T1), we compare our proposed benchmark to the most related benchmarks within embodied AI and RL research with respect to the following desired properties:

- 1.
Photorealism: Measures the visual complexity of environments, affecting the meaningful evaluation of foundation models trained on real-world data.
- 2.
Scope of reasoning needed over memory: Distinguishes between benchmarks that require recalling diverse semantic information v.s. a narrow set of attributes (e.g. object locations).
- 3.
Task Specification: Evaluates whether tasks are defined using flexible language-based descriptions, enabling richer memory challenges.
- 4.
Memory horizon length: Adapting the definition from Ni et al. (2023), this measures the temporal scope over which memory must be maintained for optimal task performance. Unlike static benchmarks requiring localized frame recall, dynamic environments demand reasoning over events distributed across extended observation sequences. This metric becomes difficult to quantify when tasks require substantial exploration phases.
- 5.
Memory isolation: Whether benchmark performance isolates memory evaluation from confounding factors, particularly exploration requirements, which represent orthogonal challenges to memory recall and reasoning.

Our benchmark isolates memory evaluation by avoiding confounding related to exploration and focuses on careful and extensible task curation with automated metrics, requiring agents to recall and reason over long-horizon environmental observations.

## 3 FindingDory

In this section, we introduce our embodied memory benchmark, FindingDory, built on top of the Habitat simulator Savva et al. (2019). Habitat offers photorealistic visuals, diverse household scenes, and object-rich environments, making it a practical and transferable testbed for evaluating memory-augmented embodied agents. Since many vision-language models (VLMs) are pretrained on real-world image distributions, they can directly interpret observations from Habitat, enabling realistic and scalable evaluation of long-term decision-making.

We first describe the overall structure of FindingDory and the suite of tasks it comprises, then provide implementation details.

### 3.1 Benchmark Design and Tasks

The goal of FindingDory is to evaluate an agent’s ability to retain and utilize past experiences to efficiently complete future tasks. To achieve this, we design a two-phase setup: an experience collection phase, where scripted oracle agents autonomously gather experiences in the environment, and an interaction phase, where agents are evaluated on their ability to recall and act upon that history. This decomposition isolates memory from exploration, enabling standardized evaluation.

During experience collection, oracle agents pick up and place objects onto designated receptacles (e.g., a table, chair, or bed), introducing meaningful state changes into the environment. Episodes span 400–3500 frames with 2–11 pick-and-place interactions (see [section B.2](https://arxiv.org/html/2506.15635v2#A2.SS2)). Once collection ends, the interaction phase begins: the agent is given access to its recorded history (images, pose, past actions) and tasked with following instructions that require integrating long-term memory into decision-making.

Tasks are instantiated from a set of templated instructions designed to probe diverse memory challenges. These cover spatial, temporal, and semantic reasoning: e.g., recalling which object was moved, when it was moved, or how it was interacted with; identifying rooms visited or farthest locations; and reasoning over attributes such as shape, color, or material. We also introduce time-usage memory tasks (e.g., longest room exploration, object with longest manipulation). A smaller subset of attribute-based tasks involves referring to objects indirectly via observed properties (e.g., “go to the striped object you interacted with previously”), helping distinguish methods that can flexibly reconstruct fine-grained visual details from those relying on static summaries. In total, 60 templates are used to generate scene-specific task instances (see LABEL:tab:all_tasks).

Single vs. Multiple Goals.
Most tasks involve single-goal instructions, but some require reaching multiple destinations in sequence. Even single-goal tasks can be challenging, demanding agents recall what was *not* interacted with or trace object movements across rearrangement states.

Extensibility.
The task suite is procedurally extensible: adjusting the complexity of experience collection (e.g., number of interactions) scales the temporal reasoning space and memory dependencies. We further group tasks into interpretable categories for analysis (see [table 2](https://arxiv.org/html/2506.15635v2#S3.T2)), making FindingDory one of the most visually complex and extensible embodied memory benchmarks to date.

**Table 2: Task categories with example prompts and challenges. See [appendix G](https://arxiv.org/html/2506.15635v2#A7) for more examples.**
|  | Task Category | Example Instructions | Memory Requirement |
| --- | --- | --- | --- |
| Spatial | Object Recall | Navigate to a {category}. | Object was previously seen or needs exploration. |
|  | Interaction | Navigate to any object that you did not interact with yesterday. | Differentiate between interacted & uninteracted objects. |
|  | Conditional Interaction | Navigate to a {receptacle_category} you picked an object from. | Object categories, object-receptacle relationships. |
|  | Object Attribute | Navigate back to a {target_color} colored object that you interacted with yesterday. | Object attributes other than category. |
|  | Spatial Relationship | Navigate to the object which you interacted with which is the farthest from your current location. | Spatial relationships to objects. |
|  | Room Visitation | Navigate to a room that you did not visit yesterday. | Room boundaries and past interactions. |
| Temporal | Interaction Order | Navigate to the object you interacted with immediately after {object_category}. | Sequence of objects interactions. |
|  | Time-Based | Navigate to the object that you interacted with at {HH:MM} yesterday. | Time of interaction & final position of objects. |
|  | Duration Tracking | Navigate to the object which took the longest time to rearrange. | Duration of interactions. |
| Multi-Goal | Unordered Revisitation | Revisit all the receptacles you picked objects from yesterday. | Past locations of multiple objects and best visitation order. |
|  | Ordered Revisitation | Revisit all the objects you interacted with yesterday in specific order. | Similar to unordered revisitation but with strict sequencing. |

### 3.2 Implementation details

Simulation Setup and Dataset Composition.
We build FindingDory using the Habitat simulator (Savva et al., 2019; Szot et al., 2021) and scenes from the Habitat Synthetic Scenes Dataset (HSSD) (Khanna* et al., 2023). Our benchmark uses 107 scenes from the training split and 30 from the validation split, with 1,478 training and 100 validation episodes. We populate these scenes with the same object sets used in OVMM (Yenamandra et al., 2023).
Across these episodes, we include 839/247 object instances spanning 84/72 distinct categories in the train/val splits, respectively. Objects are rearranged during the experience collection phase to induce temporally grounded memory challenges. Additionally, agents interact with 17/16 distinct receptacle categories in the train/val splits (e.g., tables, chairs, beds).
To support attribute-based tasks (see [table 2](https://arxiv.org/html/2506.15635v2#S3.T2)), we generate semantic descriptions of object instances using GPT-4o. We prompt the model with multi-view images of each object to extract attributes such as shape, color, material, and functionality. These descriptions are used to populate the object attribute task templates (5 total), which are limited to the validation set and undergo manual quality review. In total, our benchmark includes 79,213 training and 5,876 validation task instances. More details in [appendix B](https://arxiv.org/html/2506.15635v2#A2).

Agent Setup. We adopt the Hello Robot Stretch embodiment (Kemp et al., 2022) in our simulation.
The agent is equipped with RGB and depth sensors (resolution $640\times 480$) and a GPS+Compass sensor. The agent can execute one of four discrete navigation actions: MOVE_FORWARD ($0.25$m), TURN_LEFT / TURN_RIGHT (by $10^{\circ}$) and a STOP action. During the experience collection phase, the agent can also manipulate its arm joints (extension, lift, yaw, pitch, roll) and activate a manipulation_mode for pick-and-place actions, which rotates the base ($90^{\circ}$) and directs the head camera toward the gripper.

Procedural Generation and Verification using PDDL. We operate over a large set of diverse task instructions (see  [table 2](https://arxiv.org/html/2506.15635v2#S3.T2)) by encoding them into templates using the PDDL specification system  (Aeronautiques et al., 1998; Szot et al., 2024).
Specifically, each template specifies entities (objects and receptacles) with properties such as interaction order, target color, and start/goal receptacle.
These entities are combined using predicates that map to pythonic functions which can be evaluated against any simulator state.
Goal conditions compose these predicates with quantifiers (e.g., EXISTS, FOR_ALL_UNORDERED) to express complex logical relationships between multiple predicates.
The defined instructions templates are procedurally expanded by sampling valid entity bindings based on the entity properties which serve as constraints.
The bindings are finally substituted into the goal conditions to generate the FindingDory dataset and enable scalable, automated verification for each episode without hand-coding low-level rules.

## 4 Experiments

We evaluate several baseline approaches on FindingDory and assess their ability to complete memory-intensive tasks using a combination of vision-language reasoning and navigation. Below, we describe the baseline architecture, implementation details, and evaluation metrics.

### 4.1 Evaluated Approaches

To establish strong baselines, we adopt a hierarchical architecture ([fig. 15](https://arxiv.org/html/2506.15635v2#A5.F15)) comprising two modules: (1) a high-level goal selection module, which processes the interaction history to select a sequence of goal frames, and
(2) a low-level navigation policy, which executes actions to reach the selected frame. This architecture enables reasoning over long-horizon multimodal experience while delegating fine-grained control to a learned policy. Inspired by prior work (Xu et al., 2024), we avoid directly predicting actions using VLMs due to their poor performance in continuous control, and instead restrict them to high-level goal prediction.

#### 4.1.1 High-Level Goal Selection

Video LM Agent.
This agent receives the full interaction video along with a task instruction. Each frame in the video is annotated with its index and the time of day at which it was captured. The agent is tasked with predicting the index of a frame from the video that corresponds to a viable goal state. Specifically, the predicted frame should represent a viewpoint from which, if the agent were to navigate back to the same camera position, it could successfully complete the task. Although the model outputs a single frame index, selecting the correct one often requires reasoning over multiple temporally scattered observations, including frames where the agent may not be actively interacting. Since different VLMs are capable of handling different numbers of frames in context, we subsample the frames passed to each model based on what empirically performs best. We evaluate both proprietary (Gemini-2.0-Flash (Team et al., 2024), GPT-4o) and open-source (Qwen2.5-VL (Bai et al., 2025), Gemma-3 (Team et al., 2025), GLM-4.1V-Thinking (Hong et al., 2025)) vision-language models capable of processing long video contexts. See [sections D.1](https://arxiv.org/html/2506.15635v2#A4.SS1), [D.2](https://arxiv.org/html/2506.15635v2#A4.SS2) and [D.3](https://arxiv.org/html/2506.15635v2#A4.SS3) for more details.

Textual Memory Agent. Similar to (Min et al., 2024; Anwar et al., 2024), we also consider an explicit, memory-building baseline which breaks down the video frames into chunks and uses a VLM to generate relevant textual summaries for each video chunk. After this guided “captioning” of the entire video sequence, an LLM selects goal frame indices by reasoning over the generated textual descriptions. See [section D.4](https://arxiv.org/html/2506.15635v2#A4.SS4) for more details.

Supervised Fine-Tuning (SFT). We additionally explore a supervised fine-tuning approach where the high-level VLM is trained to predict goal frame indices, given the task prompt and interaction history. Since multiple frames may validly represent the goal (e.g., several consecutive frames showing an object being placed), we supervise the model on the full list of acceptable target frames. At test time, a single frame is sampled from the list of predictions and is considered correct if it matches any of these valid frames. This setup enables the model to learn task-specific cues from examples and serves as a stronger baseline than zero-shot prompting, particularly for tasks requiring subtle temporal grounding.
We provide additional details in [section D.5](https://arxiv.org/html/2506.15635v2#A4.SS5).

#### 4.1.2 Low-Level Navigation

To execute the high-level plan, we use a low-level policy that takes the selected goal frame and outputs discrete actions: MOVE-FORWARD, TURN-RIGHT, TURN-LEFT, and STOP. Our primary policy is the LSTM-based controller from OVRLv2 (Yadav et al., 2023), trained to reach image goals from egocentric RGB-D observations (see [section E.1](https://arxiv.org/html/2506.15635v2#A5.SS1)) (ImageNav). We also evaluate a mapping-based policy that deterministically navigates to the location corresponding to the goal frame using a global map, without relying on learned visual features (see [section E.2](https://arxiv.org/html/2506.15635v2#A5.SS2)).

Solvability. The action space of our high-level baseline is constrained to frames observed during experience collection, which is insufficient for some tasks.
For example, in the task “Navigate to a room that you did not visit yesterday” all frames correspond to previously visited locations, making it impossible to complete the instruction.
To establish an upper bound on task difficulty, we quantify the percentage of episodes that could theoretically be solved through optimal frame selection from interaction history, assuming perfect navigation (teleportation) to the selected waypoint.
This upper bound is the dashed line (Oracle) in [fig. 2](https://arxiv.org/html/2506.15635v2#S5.F2).

### 4.2 Evaluation Metrics

We benchmark agent performance on the FindingDory tasks using Low Level Success Rate (LL-SR), assessing whether the agent finds the right target entities, and Low Level Success-weighted-by-Path-Length (LL-SPL), evaluating if the agent selects and navigates to the optimal entity using the shortest possible path. Since we use a hierarchical baseline, we additionally introduce a High-Level Policy Success Rate (HL-SR), and High-Level Policy SPL (HL-SPL) which measures the accuracy and efficiency of the high-level policy in selecting the correct and closest goal frames for navigation. To further analyze failure modes, we also report two relaxed variants: Distance-to-Goal-Only SR (DTG-SR), which relaxes the requirement that the target object must be visible in the selected frame, and Semantic Coverage SR (SC-SR), which ignores spatial proximity.
Additional metrics, success criteria, and threshold specifics are provided in [section B.3](https://arxiv.org/html/2506.15635v2#A2.SS3).

## 5 Results

Figure: Figure 2: FindingDory Task Performance: Proprietary VLMs such as Gemini-2.0-flash and GPT-4o show little success across most FindingDory task categories. Agents struggle especially on multi-goal tasks where multiple subgoals must be achieved sequentially from the interaction videos collected during Phase 1. The Qwen-SFT baseline (see [section D.5](https://arxiv.org/html/2506.15635v2#A4.SS5)), trained to predict ground-truth frame indices, performs best but still reaches only $\approx 50\%$ on high-level tasks.
Refer to caption: x1.png

In this section, we analyze the performance of the previously outlined baselines on FindingDory tasks, presenting results grouped according to the task categories described in [section 3.1](https://arxiv.org/html/2506.15635v2#S3.SS1).

State-of-the-Art VLMs struggle on the FindingDory benchmark. We first evaluate different VLMs on the high-level task and report success rates in [fig. 2](https://arxiv.org/html/2506.15635v2#S5.F2), observing consistently low performance across all task subsets (see [table 4](https://arxiv.org/html/2506.15635v2#A7.T4) for numbers). GPT-4o achieves the highest success at 27.3%, followed by Gemini-2.0-Flash at 25.7%, and GLM-4.1V-Thinking at 23.5%. The non-reasoning open-source models Qwen2.5-VL and Gemma-3 score between 13% and 16%. The text-based agent performs the worst overall, underscoring that textual memory representations alone are insufficient for disambiguating visual goals or reasoning over complex visual histories. Notably, the supervised fine-tuned (SFT) variant outperforms all frozen VLMs, achieving an average improvement of 25% across tasks, with considerable improvement on temporal and multi-goal tasks.

In single-goal tasks, all methods struggle particularly with the spatial relationship subset, with GPT-4o achieving only 14.5% success. These tasks require reasoning about relative distances between entities and the agent, an ongoing challenge noted in prior work (Yang et al., 2024).
In contrast, room identification tasks yield the highest performance, likely due to the small number of distinct rooms and the lenient success criteria of reaching any location within the correct room.
Performance on conditional interaction and object attribute tasks is comparable, suggesting that current VLMs handle both category-based queries (e.g., “apple”) and attribute-based descriptions (e.g., “red food”) with similar effectiveness.

Temporal tasks pose additional challenges to the existing VLMs, with models consistently failing to identify the correct order of interactions, since it requires multi-hop reasoning.
Duration tracking is similarly difficult as it requires inferring elapsed duration based on specific points in the video.
In contrast, for the time-based tasks, we observe that baselines perform better, as these tasks only require single frame retrieval based on the timestamps in the question, without additional reasoning.

Poor performance on multi-goal tasks. From [fig. 2](https://arxiv.org/html/2506.15635v2#S5.F2), we observe that all VLM baselines achieve near-zero performance on most multi-goal tasks. This indicates that VLMs struggle to track and recall multiple object–receptacle interactions. In our multi-goal evaluation, we employ a single-shot prompt, requiring the high-level VLM agent to predict all target frame indices in one response, primarily to avoid repeated inferences over long sequences given the large number of frames. Qualitatively, we find that VLMs often fail to predict even the correct number of targets needed to solve a task. While our trained Qwen model performs somewhat better, reaching around 20% accuracy, it still falls far short of the attainable 99% success rate on this task.

VLMs can identify target receptacles but not precisely localize them.
We analyze VLM performance on tasks where the valid entities are receptacles versus where they are objects. As shown in [fig. 3(a)](https://arxiv.org/html/2506.15635v2#S5.F3.sf1), the Semantic Coverage-SR (SC-SR) metric (see [section B.3](https://arxiv.org/html/2506.15635v2#A2.SS3)) is substantially higher for receptacle-based tasks. This indicates that VLMs often select frames where the receptacle is visible, but the agent remains far from it. In contrast, distance-threshold success (DTG-SR) improves negligibly, suggesting that very few cases exist where the agent is close to a receptacle but not facing it. For smaller objects, no such improvement is observed (see [fig. 3(b)](https://arxiv.org/html/2506.15635v2#S5.F3.sf2)), since small objects are rarely visible from longer distances. Taken together, these results highlight a key limitation of current VLMs: they can semantically detect large targets but fail to construct spatially grounded representations necessary for precise localization.

Figure: (a) Relaxed metrics: Recep.
Refer to caption: x2.png

VLMs struggle with identifying the closest entity that solves the task. In Fig. [3(c)](https://arxiv.org/html/2506.15635v2#S5.F3.sf3) we report the HL-SPL metric ([section 4.2](https://arxiv.org/html/2506.15635v2#S4.SS2)) which captures the efficiency with which the high level VLM-based modules select goal frames.
During the experience collection phase, the agent can interact with multiple object instances of the same category due to which a task such as “Navigate to a squeezer you interacted with” could have multiple valid entities that the agent can navigate to.
We observe that for such tasks there is a wide gap between HL-SR and HL-SPL (upto $50\%$ in case of interaction-based tasks) for all model variants.
This suggests that although the VLMs can correctly recognize a valid entity to solve the task, they still fail to perform a fine-grained spatial analysis of the multiple valid entities being rearranged dynamically in the video. Thus, they cannot spatially locate the valid entity nearest to its current position.

Policy decomposition causes performance degradation. To evaluate the hierarchical agent introduced in [section 4.1](https://arxiv.org/html/2506.15635v2#S4.SS1), we use Qwen2.5-VL-7B as the high-level VLM module and pair it with either an ImageNav policy or a mapping-based policy as the low-level agent. The low-level policy is triggered only when the high-level VLM correctly identifies a subgoal. As shown in [fig. 3(d)](https://arxiv.org/html/2506.15635v2#S5.F3.sf4), the Low-Level-SR (LL-SR) metric drops sharply across all FindingDory tasks for the ImageNav policy. We attribute this decline to a distribution shift in the visual goals provided at evaluation. During training, the policy sees randomly sampled start–end goal poses, whereas VLM-selected goals often correspond to pick–place routines. These frames largely contain background walls or partial receptacle surfaces, offering limited visual cues for navigation. By contrast, the mapping-based navigation policy is more robust, suffering only a 25% relative drop in SR. However, the SPL of both methods remains in the single digits, underscoring the importance of FindingDory in exposing these shortcomings and motivating the development of more reliable memory and navigation strategies.

Low level navigation policy failures. From  [Figure 4(b)](https://arxiv.org/html/2506.15635v2#S5.F4.sf2), we observe that the low-level image-goal navigation policy can fail due to invoking the termination (stop) action incorrectly or a navigation routine timeout (max budget of $1000$ steps including the data collection phase). To further understand this behavior, we analyze if the navigation policy actually looks at the target entity when the stop action is invoked. Interestingly, we find that in several instances the target entity is visible from a slightly larger distance threshold (2.0m) from the chosen task success threshold (1.0 m, in case of receptacles). This often happens when VLMs select goal frames in which the target entity is visible in the 2D image but not necessarily closest in 3D space, and thus the goal frame coordinates are slightly further away than the threshold distance.

Figure: (a) Comparison across different video lengths.
Refer to caption: x6.png

Leveraging additional frames.
We study the effect of input length on high-level performance by evaluating agents on subsampled interaction videos of varying sizes ([fig. 4(a)](https://arxiv.org/html/2506.15635v2#S5.F4.sf1)). This allows us to test whether models benefit from longer histories or merely pick up sparse visual cues. Interestingly, frozen VLMs, although capable of processing long contexts, show little to no improvement with more frames. Their performance often degrades at higher frame counts, suggesting difficulty in extracting relevant signals from densely packed, unfiltered inputs.

In contrast, the fine-tuned model (Qwen SFT) achieves clear gains when trained with longer videos. This shows that with appropriate supervision, models can move beyond shallow matching to exploit richer temporal structure in interaction histories. Nonetheless, even with full finetuning on longer (subsampled) videos, a large gap remains relative to maximum achievable performance (see [section B.4](https://arxiv.org/html/2506.15635v2#A2.SS4)). This underscores the need for future methods that enable VLMs to extract and store in a compressed manner (i.e. via memory mechanisms) task-relevant information more effectively from long video sequences within the context window.

**Table 3: VSI-Bench results. Models trained with FindingDory show consistent gains.**
| Method | MCQ Accuracy ($\uparrow$) | Numerical Accuracy ($\uparrow$) |
| --- | --- | --- |
| Zero-Shot | 33.69 | 21.47 |
| Video-R1-Only SFT | 33.33 | 32.67 |
| Video-R1 + FindingDory SFT | 35.06 | 32.35 |
| Video-R1-Only GRPO | 33.57 | 33.9 |
| Video-R1 + FindingDory GRPO | 35.42 | 34.82 |

Cross-Benchmark Results. In this experiment, we explore whether the spatiotemporal memory capabilities enhanced by training on simulator-based FindingDory data can translate to real world settings.
To evaluate this, we use VSI-Bench  Yang et al. (2024), a challenging benchmark specifically designed to measure spatiotemporal reasoning in VLMs on egocentric real-world videos.
We first augment the Video-R1-CoT-165k dataset introduced in Feng et al. (2025) with $40000$ samples from the FindingDory dataset.
For effective co-training, we construct chain-of-thought traces for the entire training split of the FindingDory dataset (see [section B.5](https://arxiv.org/html/2506.15635v2#A2.SS5)).
We provide specifics about training and evaluation in  [appendix F](https://arxiv.org/html/2506.15635v2#A6).
We then evaluate the trained model on VSI-Bench, showing the results in  [table 3](https://arxiv.org/html/2506.15635v2#S5.T3). We observe that co-training with FindingDory improves the SFT baseline performance by $1.7\%$ (MCQ accuracy).
Furthermore, RL-finetuning with GRPO using combined data from the FindingDory dataset improves performance with a $1.85\%$ increase over the Video-R1-Only RL checkpoint (MCQ accuracy).
Overall, the consistent gains obtained from co-training point to a sim-to-real transfer effect, suggesting that FindingDory can augment scarce real-world data for advancing spatiotemporal understanding in VLMs.

Qualitative Examples. We present multiple example responses from various VLMs in  [appendix G](https://arxiv.org/html/2506.15635v2#A7).
Overall we observe that closed models perform better than open-source counterparts on single-goal tasks (see [figs. 18](https://arxiv.org/html/2506.15635v2#A7.F18), [21](https://arxiv.org/html/2506.15635v2#A7.F21) and [22](https://arxiv.org/html/2506.15635v2#A7.F22)) but all models suffer on multi-goal tasks (see  [fig. 23](https://arxiv.org/html/2506.15635v2#A7.F23)).
We also provide example agent video trajectories at
[https://findingdory-benchmark.github.io/](https://findingdory-benchmark.github.io/).

## 6 Conclusion

We introduced FindingDory, a benchmark for evaluating long-horizon memory in embodied agents within a highly photorealistic indoor simulation.
Our findings reveal limitations in VLM context scaling, particularly in handling long observation sequences required for memory-intensive tasks.
FindingDory provides informative metrics that disentangle different aspects of memory reasoning, allows for procedural task generation, and efficiency-focused failure analysis beyond simple frame selection accuracy.
Its extensibility allows for longer temporal dependencies, supporting research on context scaling and memory efficiency in multimodal models.
Our experiments also highlight challenges with hierarchical policies motivating the need for better architectures that closely integrate long-context VLM capabilities with generalist embodied policies.

## References

- (1)
Explore vision capabilities with the gemini api.
*[https://ai.google.dev/gemini-api/docs/vision?lang=python](https://ai.google.dev/gemini-api/docs/vision?lang=python)*.
- Aeronautiques et al. (1998)
Constructions Aeronautiques, Adele Howe, Craig Knoblock, ISI Drew McDermott, Ashwin Ram, Manuela Veloso, Daniel Weld, David Wilkins Sri, Anthony Barrett, Dave Christianson, et al.
Pddl— the planning domain definition language.
*Technical Report, Tech. Rep.*, 1998.
- Anwar et al. (2024)
Abrar Anwar, John Welsh, Joydeep Biswas, Soha Pouya, and Yan Chang.
ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigation.
*arXiv e-prints*, art. arXiv:2409.13682, September 2024.
doi: 10.48550/arXiv.2409.13682.
- Anwar et al. (2024)
Abrar Anwar, John Welsh, Joydeep Biswas, Soha Pouya, and Yan Chang.
Remembr: Building and reasoning over long-horizon spatio-temporal memory for robot navigation.
In *8th Annual Conference on Robot Learning*, 2024.
- Bai et al. (2025)
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin.
Qwen2.5-vl technical report, 2025.
URL [https://arxiv.org/abs/2502.13923](https://arxiv.org/abs/2502.13923).
- Batra et al. (2020)
Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi, Manolis Savva, Alexander Toshev, and Erik Wijmans.
Objectnav revisited: On evaluation of embodied agents navigating to objects.
*arXiv preprint arXiv:2006.13171*, 2020.
- Buch et al. (2022)
Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles.
Revisiting the “Video” in Video-Language Understanding.
*arXiv e-prints*, art. arXiv:2206.01720, June 2022.
doi: 10.48550/arXiv.2206.01720.
- Chang et al. (2023)
Matthew Chang, Theophile Gervet, Mukul Khanna, Sriram Yenamandra, Dhruv Shah, So Yeon Min, Kavit Shah, Chris Paxton, Saurabh Gupta, Dhruv Batra, Roozbeh Mottaghi, Jitendra Malik, and Devendra Singh Chaplot.
GOAT: GO to Any Thing.
*arXiv e-prints*, art. arXiv:2311.06430, November 2023.
doi: 10.48550/arXiv.2311.06430.
- Feng et al. (2025)
Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, and Xiangyu Yue.
Video-r1: Reinforcing video reasoning in mllms.
*arXiv preprint arXiv:2503.21776*, 2025.
- Fu et al. (2024)
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al.
Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.
*arXiv preprint arXiv:2405.21075*, 2024.
- Guo et al. (2025)
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025.
- Hong et al. (2025)
Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, et al.
Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning.
*arXiv e-prints*, pp. arXiv–2507, 2025.
- Hurst et al. (2024)
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al.
Gpt-4o system card.
*arXiv preprint arXiv:2410.21276*, 2024.
- Kahatapitiya et al. (2024)
Kumara Kahatapitiya, Kanchana Ranasinghe, Jongwoo Park, and Michael S. Ryoo.
Language Repository for Long Video Understanding.
*arXiv e-prints*, art. arXiv:2403.14622, March 2024.
doi: 10.48550/arXiv.2403.14622.
- Kemp et al. (2022)
Charles C Kemp, Aaron Edsinger, Henry M Clever, and Blaine Matulevich.
The design of stretch: A compact, lightweight mobile manipulator for indoor human environments.
In *2022 International Conference on Robotics and Automation (ICRA)*, pp. 3150–3157. IEEE, 2022.
- Khanna* et al. (2023)
Mukul Khanna*, Yongsen Mao*, Hanxiao Jiang, Sanjay Haresh, Brennan Shacklett, Dhruv Batra, Alexander Clegg, Eric Undersander, Angel X. Chang, and Manolis Savva.
Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation.
*arXiv preprint*, 2023.
- Khanna* et al. (2024)
Mukul Khanna*, Ram Ramrakhya*, Gunjan Chhablani, Sriram Yenamandra, Theophile Gervet, Matthew Chang, Zsolt Kira, Devendra Singh Chaplot, Dhruv Batra, and Roozbeh Mottaghi.
Goat-bench: A benchmark for multi-modal lifelong navigation, 2024.
- Kwon et al. (2023)
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica.
Efficient memory management for large language model serving with pagedattention.
In *Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles*, 2023.
- Li et al. (2023)
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan.
Seed-bench: Benchmarking multimodal llms with generative comprehension.
*arXiv preprint arXiv:2307.16125*, 2023.
- Li et al. (2023)
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao.
MVBench: A Comprehensive Multi-modal Video Understanding Benchmark.
*arXiv e-prints*, art. arXiv:2311.17005, November 2023.
doi: 10.48550/arXiv.2311.17005.
- Ma et al. (2023)
Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang.
Sqa3d: Situated question answering in 3d scenes.
In *International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=IDJx97BC38](https://openreview.net/forum?id=IDJx97BC38).
- Ma et al. (2024)
Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, Pan Zhang, Liangming Pan, Yu-Gang Jiang, Jiaqi Wang, Yixin Cao, and Aixin Sun.
Mmlongbench-doc: Benchmarking long-context document understanding with visualizations, 2024.
URL [https://arxiv.org/abs/2407.01523](https://arxiv.org/abs/2407.01523).
- Majumdar et al. (2023)
Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Tingfan Wu, Jay Vakil, et al.
Where are we in the search for an artificial visual cortex for embodied intelligence?
*Advances in Neural Information Processing Systems*, 36:655–677, 2023.
- Majumdar et al. (2024)
Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv Batra, Mrinal Kalakrishnan, Franziska Meier, Chris Paxton, Sasha Sax, and Aravind Rajeswaran.
Openeqa: Embodied question answering in the era of foundation models.
In *Conference on Computer Vision and Pattern Recognition (CVPR)*, 2024.
- Mangalam et al. (2023)
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik.
EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding.
*arXiv e-prints*, art. arXiv:2308.09126, August 2023.
doi: 10.48550/arXiv.2308.09126.
- Min et al. (2024)
So Yeon Min, Xavi Puig, Devendra Singh Chaplot, Tsung-Yen Yang, Akshara Rai, Priyam Parashar, Ruslan Salakhutdinov, Yonatan Bisk, and Roozbeh Mottaghi.
Situated Instruction Following.
*arXiv e-prints*, art. arXiv:2407.12061, July 2024.
doi: 10.48550/arXiv.2407.12061.
- Ni et al. (2023)
Tianwei Ni, Michel Ma, Benjamin Eysenbach, and Pierre-Luc Bacon.
When do transformers shine in RL? decoupling memory from credit assignment.
In *Thirty-seventh Conference on Neural Information Processing Systems*, 2023.
URL [https://openreview.net/forum?id=APGXBNkt6h](https://openreview.net/forum?id=APGXBNkt6h).
- Pašukonis et al. (2023)
Jurgis Pašukonis, Timothy P Lillicrap, and Danijar Hafner.
Evaluating long-term memory in 3d mazes.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=yHLvIlE9RGN](https://openreview.net/forum?id=yHLvIlE9RGN).
- Pleines et al. (2023)
Marco Pleines, Matthias Pallasch, Frank Zimmer, and Mike Preuss.
Memory gym: Partially observable challenges to memory-based agents.
In *International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=jHc8dCx6DDr](https://openreview.net/forum?id=jHc8dCx6DDr).
- Ramakrishnan et al. (2024)
Santhosh Kumar Ramakrishnan, Erik Wijmans, Philipp Kraehenbuehl, and Vladlen Koltun.
Does spatial cognition emerge in frontier models?
*arXiv preprint arXiv:2410.06468*, 2024.
- Rawal et al. (2024)
Ruchit Rawal, Khalid Saifullah, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein.
Cinepile: A long video question answering dataset and benchmark.
*arXiv preprint arXiv:2405.08813*, 2024.
- Ren et al. (2024)
Allen Z. Ren, Jaden Clark, Anushri Dixit, Masha Itkina, Anirudha Majumdar, and Dorsa Sadigh.
Explore until confident: Efficient exploration for embodied question answering.
In *arXiv preprint arXiv:2403.15941*, 2024.
- Sashank Dorbala et al. (2024)
Vishnu Sashank Dorbala, Prasoon Goyal, Robinson Piramuthu, Michael Johnston, Reza Ghanadhan, and Dinesh Manocha.
S-EQA: Tackling Situational Queries in Embodied Question Answering.
*arXiv e-prints*, art. arXiv:2405.04732, May 2024.
doi: 10.48550/arXiv.2405.04732.
- Savva et al. (2019)
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra.
Habitat: A Platform for Embodied AI Research.
*ICCV*, 2019.
- Sethian (1996)
James A Sethian.
A fast marching level set method for monotonically advancing fronts.
*proceedings of the National Academy of Sciences*, 93(4):1591–1595, 1996.
- Song et al. (2024)
Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang.
Milebench: Benchmarking mllms in long context.
*arXiv preprint arXiv:2404.18532*, 2024.
- Song et al. (2023)
Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, et al.
Moviechat: From dense token to sparse memory for long video understanding.
*arXiv preprint arXiv:2307.16449*, 2023.
- Szot et al. (2021)
Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, et al.
Habitat 2.0: Training home assistants to rearrange their habitat.
In *NeurIPS*, 2021.
- Szot et al. (2024)
Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Rin Metcalf, Walter Talbott, Natalie Mackraz, R Devon Hjelm, and Alexander T Toshev.
Large language models as generalizable policies for embodied tasks.
In *The Twelfth International Conference on Learning Representations*, 2024.
URL [https://openreview.net/forum?id=u6imHU4Ebu](https://openreview.net/forum?id=u6imHU4Ebu).
- Team et al. (2024)
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al.
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.
*arXiv preprint arXiv:2403.05530*, 2024.
- Team et al. (2025)
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al.
Gemma 3 technical report.
*arXiv preprint arXiv:2503.19786*, 2025.
- Wang et al. (2024)
Cunxiang Wang, Ruoxi Ning, Boqi Pan, Tonghui Wu, Qipeng Guo, Cheng Deng, Guangsheng Bao, Qian Wang, and Yue Zhang.
Novelqa: A benchmark for long-range novel question answering, 2024.
- Wani et al. (2020)
Saim Wani, Shivansh Patel, Unnat Jain, Angel Chang, and Manolis Savva.
Multion: Benchmarking semantic map memory using multi-object navigation.
*NeurIPS*, 2020.
- Xie et al. (2024)
Quanting Xie, So Yeon Min, Tianyi Zhang, Kedi Xu, Aarav Bajaj, Ruslan Salakhutdinov, Matthew Johnson-Roberson, and Yonatan Bisk.
Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generation.
*arXiv e-prints*, art. arXiv:2409.18313, September 2024.
doi: 10.48550/arXiv.2409.18313.
- Xiong et al. (2025)
Haomiao Xiong, Zongxin Yang, Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Jiawen Zhu, and Huchuan Lu.
Streaming video understanding and multi-round interaction with memory-enhanced knowledge, 2025.
URL [https://arxiv.org/abs/2501.13468](https://arxiv.org/abs/2501.13468).
- Xu et al. (2024)
Zhuo Xu, Hao-Tien Lewis Chiang, Zipeng Fu, Mithun George Jacob, Tingnan Zhang, Tsang-Wei Edward Lee, Wenhao Yu, Connor Schenck, David Rendleman, Dhruv Shah, Fei Xia, Jasmine Hsu, Jonathan Hoech, Pete Florence, Sean Kirmani, Sumeet Singh, Vikas Sindhwani, Carolina Parada, Chelsea Finn, Peng Xu, Sergey Levine, and Jie Tan.
Mobility VLA: Multimodal instruction navigation with long-context VLMs and topological graphs.
In *8th Annual Conference on Robot Learning*, 2024.
URL [https://openreview.net/forum?id=JScswMfEQ0](https://openreview.net/forum?id=JScswMfEQ0).
- Yadav et al. (2023)
Karmesh Yadav, Arjun Majumdar, Ram Ramrakhya, Naoki Yokoyama, Alexei Baevski, Zsolt Kira, Oleksandr Maksymets, and Dhruv Batra.
Ovrl-v2: A simple state-of-art baseline for imagenav and objectnav.
*[https://arxiv.org/abs/2303.07798](https://arxiv.org/abs/2303.07798)*, 2023.
- Yang et al. (2024)
Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie.
Thinking in space: How multimodal large language models see, remember, and recall spaces.
*arXiv preprint arXiv:2412.14171*, 2024.
- Yenamandra et al. (2023)
Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin Wang, Mukul Khanna, Theophile Gervet, Tsung-Yen Yang, Vidhi Jain, Alexander William Clegg, John Turner, et al.
Homerobot: Open-vocabulary mobile manipulation.
*arXiv preprint arXiv:2306.11565*, 2023.
- Zhang et al. (2023)
Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius.
A simple llm framework for long-range video question-answering, 2023.
- Zhang et al. (2024)
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu.
Long context transfer from language to vision.
*arXiv preprint arXiv:2406.16852*, 2024.
URL [https://arxiv.org/abs/2406.16852](https://arxiv.org/abs/2406.16852).
- Zhou et al. (2024)
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu.
Mlvu: A comprehensive benchmark for multi-task long video understanding.
*arXiv preprint arXiv:2406.04264*, 2024.
- Zhu et al. (2023)
Hao Zhu, Raghav Kapoor, So Yeon Min, Winson Han, Jiatai Li, Kaiwen Geng, Graham Neubig, Yonatan Bisk, Aniruddha Kembhavi, and Luca Weihs.
Excalibur: Encouraging and evaluating embodied exploration.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 14931–14942, June 2023.

## Appendix A Limitations

While FindingDory provides a rigorous evaluation framework for long-horizon memory in embodied AI, it has certain limitations that highlight areas for future improvement.

First, our most competitive baseline, which integrates a VLM with a navigation policy, is constrained by its low-level controller. Current memory-guided navigation policies struggle to take the shortest paths in large environments after reasoning over long interaction histories. As a result, models perform poorly on spatial awareness metrics, making it difficult to assess their true memory capabilities beyond basic path efficiency. Future work should explore improved memory-driven navigation policies to better leverage long-term spatial reasoning.

Second, in the experience collection phase, our scripted oracle policy for object interactions (Magic Grasp) executes pick-and-place actions in an unnatural way—objects are abruptly transferred from the receptacle surface to the agent’s gripper without smooth motion. This may confuse VLMs, as the sudden change is visually subtle and may be overlooked if the model does not attend to the specific frames where it occurs. Developing more naturalistic interaction models could improve the realism and interpretability of memory traces.

Third, while we introduce distractor objects to discourage guessing, we do not explicitly control how prominently these objects appear in the experience collection phase. Currently, objects the agent interacts with are often viewed more frequently and closely compared to distractors, which tend to be featured less prominently in the agent’s field of view. Although some tasks mitigate this by explicitly referencing non-interacted objects, future iterations of the benchmark may need stronger distractions by ensuring more balanced exposure between distractors and interacted objects.

Finally, while the experience collection phase involves both navigation and manipulation, the current interaction phase tasks are all navigation-based, as memory reasoning is primarily required for locating objects rather than executing pick-and-place actions, which are Markovian once the goal is known. However, our benchmark is easily extensible to include manipulation-based memory tasks, which would further enrich the evaluation of long-term reasoning in embodied AI.

Despite these limitations, FindingDory establishes a strong foundation for studying long-horizon memory, offering extensible challenges that will evolve alongside advances in vision-language models and embodied learning.

## Appendix B Benchmark Details

### B.1 FindingDory Episode Creation

In this section, we give details about our episode creation pipeline that allows us to procedurally generate data for our benchmark.

Object-Receptacle Pairing. The first step in the episode creation involves generating valid spawning positions for various object-receptacle pairs by sampling entities from the respective split in consideration (train/val). The objective is to identify candidate objects that can be picked up from a start receptacle and placed stably on a goal receptacle. We use the pipeline proposed in Yenamandra et al. (2023) to procedurally generate various object placements on receptacle surfaces by running physics checks for stable placements. To make the generation procedure easier, we assume that each object is picked and placed only once during an episode. Additionally, we constrain the start receptacle category to be distinct from the goal receptacle category. We also ensure that for each candidate object category, a corresponding non-interacted object of the same category is spawned to serve as distractor entities during the interaction phase.  [Figures 5(b)](https://arxiv.org/html/2506.15635v2#A2.F5.sf2) and [5(c)](https://arxiv.org/html/2506.15635v2#A2.F5.sf3) provide a distribution of distractor objects and receptacles in the val episodes. The number of sampled object-receptacle pairings is equal to the number of rearrangements in the particular episode and can be configured through an argument. In essence, this controls the distribution of video frame lengths in the FindingDory tasks(see  [Figure 5(a)](https://arxiv.org/html/2506.15635v2#A2.F5.sf1)).

Object Placement Sequences. Once we have identified the set of candidate objects and the corresponding goal receptacles, we require an oracle policy through which we can “naturally” drop objects on the receptacle surfaces. In practice, one can achieve this by training a “place” skill through end-to-end reinforcement learning or use a heuristic agent that can place objects on receptacles but both frameworks do not guarantee perfect $100\%$ success which is crucial for solvability in FindingDory tasks. To address this, we construct an offline pipeline that produces the set of low-level “oracle” joint actions that lead to stably placing the object on the goal receptacle. We spawn the agent (with the object snapped in its end-effector) at the associated viewpoints of goal receptacles on the navigation mesh. From each viewpoint, we try to execute a heuristic, state-machine based placing policy which performs a set of predefined joint effector actions after identifying a “clutter-free” region in the receptacle pointcloud (using an onboard depth sensor). The output of this routine produces a set of low level joint actions and the corresponding receptacle viewpoint for each goal receptacle that was identified in the previous object-receptacle pairing stage. We reject episodes where the heuristic-based place policy cannot stably place the object from any of the associated goal receptacle viewpoints.

Task Creation. Once we have the final filtered set of episodes, we populate task instructions for each episode using yaml-based instruction templates as used in  Szot et al. (2024).
We employ the instruction templates to procedurally populate various placeholders such as object names, attributes and appearance orders that create specific task instructions grounded in the entities (objects and receptacles) present in the episode in an automated manner.
We present the diverse instructions templates we use for each task category in Table LABEL:tab:all_tasks.
In total, we create $82174/5876$ unique task instructions within the train/val split respectively.

Figure: (a) Distributions of frame counts and interaction counts across episodes.
Refer to caption: x8.png

### B.2 Experience Collection Phase

We now outline the details of the experience collection phase grounded in the episodes created by the pipeline outlined in the previous section. The experience collection phase is designed to collect a “clean” and “natural” video sequence of an agent interacting in an home environment while interacting with various entities. We focus on object rearrangements as the core interaction routine followed by the embodied agent. We construct an OracleAgent agent that uses privileged information from the simulator instance to efficiently generate such interaction routines. We now detail the OracleAgent policy to generate such video sequences for a particular episode.

Nav-to-Pick. The agent begins by being spawned at a random location in the scene associated with a particular episode. We then query a ShortestPath function that leverages privileged simulator information to navigate to the object that has to be picked up. The OracleAgent follows the shortest path to the object and collects images as it navigates in the environment.

Pick. Once the OracleAgent reaches the object location, it executes a hardcoded pick policy sequence which results in the object being magically grasped by the OracleAgent. The pick policy uses the onboard depth sensor to orient itself with the object instance placed on the receptacle infront of the agent. Once the object is in clear view (using the ground truth instance mask from the simulator), the and effector extends outwards to perform a magic_grasp action. In cases when the object is not visible, we re-query the ShortestPath function to navigate to an alternate viewpoint associated with the receptacle that would lead to the object being clearly visible.

Nav-to-Place. After grasping the object, the OracleAgent navigates to the corresponding goal receptacle that was sampled during the episode creation phase. Specifically, the agent navigates to the specific viewpoint associated with the goal receptacle from which the pre-recorded placement action sequence is to be executed.

Place. Once the agent reaches the specific viewpoint, we re-play the end-effector action sequence that was found to lead to a stable object placement during episode creation. This marks the end of a single rearrangement interaction routine and the agent transitions to rearrange the next candidate object in the episode until no more objects are left.

Final Navigation. At the end of all interaction routines within an episode, the agent navigates to a predefined location to ensure that it does not remain in close proximity to the final interacted object or the final goal_receptacle. This ensures that tasks are not solvable by selecting only the last few frames in the video sequence. We ensure that the agent navigates to a distance of $3$ m away from the final interaction location.

The above OracleAgent routine enables us to procedurally create videos of the agent rearranging objects in the environment. Additionally, as the agent navigates between the pick and place location, it collects additional frames detailing the scene environment which can be useful (or serve as distractor frames) when attempting to solve FindingDory tasks. We present an example experience collection trajectory and associated “magic” pick-place sequence in [Figures 16](https://arxiv.org/html/2506.15635v2#A7.F16) and [17](https://arxiv.org/html/2506.15635v2#A7.F17).

### B.3 Evaluations

In this section, we define the primary evaluation metrics employed to benchmark the performance of various baselines on the FindingDory tasks. Following Batra et al. (2020), we focus on metrics that systematically quantify both the success and efficiency of the agent in solving specified instructions. Success focuses on accurately selecting frames from the interaction video that lead to task completion, while efficiency evaluates if the agent selects and navigates to the optimal frame that solves the task using the fewest low-level control actions. Notably, optimizing efficiency poses a significant challenge for current methods (Ramakrishnan et al., 2024; Yang et al., 2024).

High-Level Policy Success Rate (HL-SR). This metric measures the percentage of episodes in which the high-level agent correctly predicts frame indices required to solve the specified task. We define an episode as successful if the frame(s) predicted by the VLM satisfy the following criteria:

- 1.
Distance-to-Goal: The agent is within a specified distance of the target entity upon reaching the predicted frame. We use thresholds of 2.0 m for objects and 0.1 m for receptacles. The larger threshold for objects accounts for the possibility of objects being positioned on elongated receptacles (e.g., couches), potentially affecting reachability.
- 2.
Angle-to-Goal: The agent is oriented within a specified angular distance toward the target entity’s center. We set angle thresholds of $45^{\circ}$ for objects and $90^{\circ}$ for receptacles due to their typically larger dimensions.
- 3.
Semantic Coverage: The semantic mask of the target entity covers at least $0.1\%$ of the pixels in the frame.
- 4.
Room Region Check: For room visitation tasks, we verify if the agent is located within the designated target region.

These thresholds and criteria were qualitatively validated through manual inspection to ensure their appropriateness; During evaluation, these criteria are verified by teleporting the robot to the pose associated with the frame(s) selected by the high-level agent.

Low-Level Policy Success Rate (LL-SR). This metric measures the percentage of episodes where the low-level navigation agent successfully navigates to the target entity. The low-level agent is activated only if the high-level agent correctly predicts the frame indices, meaning LL-SR is evaluated conditionally upon HL-SR being true for each episode. The success criteria match those outlined for HL-SR.

Success-weighted-by-Path-Length. This metric evaluates task completion efficiency, computed per task episode as follows (Batra et al., 2020):

$$ $\text{SPL}_{i}=S_{i}\cdot\frac{l_{i}}{\max(p_{i},l_{i})}$ $$

where $S_{i}$ denotes HL-SR/LL-SR for computing the HL-SPL/LL-SPL respectively, $l_{i}$ is the length of the shortest possible path to the closest successful goal frame (or shortest path to revisit all subgoals), and $p_{i}$ is the actual path length traveled by the agent.

Additionally, we introduce two relaxed success metrics to analyze baseline failure modes:

- 1.
Distance-to-Goal-Only Success Rate (DTG-SR). Computed identically to HL-SR but without the semantic coverage requirement. This metric helps quantify instances where the agent selects frames close to the goal without necessarily capturing it visually.
- 2.
Semantic Coverage Success Rate (SC-SR). Computed identically to HL-SR but without the distance-to-goal requirement. This metric evaluates instances where the agent correctly selects frames showing the target entity but fails to meet proximity criteria.

### B.4 Subsampled Videos Solvability Analysis

Figure: Figure 6: Task Solvability Ratio at the $96$ frame subsampling level.
Refer to caption: figures/task_solvability_96_frames.png

In this section, we analyze the solvability of the tasks proposed in FindingDory suite when all videos are subsampled to enable full-finetuning of VLMs (see [section D.5](https://arxiv.org/html/2506.15635v2#A4.SS5) for details).
Specifically, we conduct a solvability analysis at the $96$-frame video subsampling level.
Full finetuning of the VLM on these videos enables highest task performance of $\approx 52\%$ as shown in  [fig. 4(a)](https://arxiv.org/html/2506.15635v2#S5.F4.sf1).

From  [fig. 6](https://arxiv.org/html/2506.15635v2#A2.F6) we observe that after subsampling the videos to $96$ frames, there is a $\approx 1\%$ drop in solvability across the various low-level task categories.
The largest drop is observed in the Conditional Interaction tasks where a specific object or receptacle is to be revisited based on the original interaction sequence.
We want to highlight that this solvability calculation is solely based on whether a solution frame (from multiple possible frames in the full length video) is left over in the subsampled video.
Such frames are treated as a solution based on whether they satisfy the PDDL goal conditions (see  [section 3.2](https://arxiv.org/html/2506.15635v2#S3.SS2)) corresponding to the task.
But such frames are extremely difficult to select for goal completion for a VLM since the subsampling routine causes a large drop in the relevant context that is needed to make sense of the solution frame.
For instance, consider the task “Navigate to the object which you interacted with which is the farthest from your current
location”: it is possible that a frame with the target object is left over in the subsampled video.
But selecting this solution frame is nearly impossible since the subsampling to $96$ frames eliminates all the spatiotemporal context that is necessary to solve this task.

From  [fig. 6](https://arxiv.org/html/2506.15635v2#A2.F6) we also observe that $3.85\%$ of the tasks are Originally Unsolvable which correspond to tasks focused on distractor entities such as “Navigate to a soft toy that you did not rearrange yesterday” or “Navigate to a room that you did not visit yesterday”.

### B.5 Chain-of-Thought Trace Generation

Figure: Figure 7: Textual Memory Trace corresponding to agent interaction during Experience Collection. The memory trace is subsampled to ensure alignment with the 48-frame video.
Refer to caption: x9.png

In this section, we outline the pipeline that is employed to generate detailed chain-of-thought traces for the FindingDory training data split.
We begin by generating textual memory traces that detail the interactions performed by the robot during the Experience Collection phase (see [Section B.2](https://arxiv.org/html/2506.15635v2#A2.SS2)).
For this, we track all the individual actions that are executed by the robot along with necessary information such as time of day, object-recetpacle info, room names that are necessary for successful task completion.
For the co-training experiments for cross-benchmark analysis, we subsample all videos to $48$ frames and accordingly modify the textual memory traces to ensure alignment between the memory trace and subsampled video frames.
A sample original and subsampled textual memory trace is shown in [Figure 7](https://arxiv.org/html/2506.15635v2#A2.F7).

To generate faithful chain-of-thought traces for all samples in the training split, we leverage a zero-shot, text-based LLM to generate a rationale for each task based on the textual memory history and corresponding keyframe solution.
We hypothesize that the zero-shot LLM would be able to generate more consistent traces if it is given access to the solution keyframes and prompted to generate a rationale using the solution.
This is in contrast to using the LLM to predict the solution keyframes based on the textual memory trace which is a more difficult task.
We use the open-source DeepSeek-R1-Distill-Qwen-32B (Guo et al., 2025) for this purpose and leverage the vLLM library (Kwon et al., 2023) for accelerated inference. We use the prompt structure as shown in  [Figure 8](https://arxiv.org/html/2506.15635v2#A2.F8).
After the initial chain-of-thought trace generation, we observed that $\approx 8\%$ of the generated traces contained references to the solution keyframe list provided in the prompt.
To fix these erroneous traces, we use the DeepSeek-R1-Distill-Qwen-32B model to rephrase the generated traces to remove any references to the solution keyframe list following the prompt shown in [Figure 9](https://arxiv.org/html/2506.15635v2#A2.F9).
After the initial trace generation and rephrasing steps, we are left with $7.8\%$ traces in which the solution keyframes list predicted by the LLM contains a keyframe which does not belong to the actual solution keyframes list.
All these samples are discarded which leads to a total of $72973$ training samples with labeled chain-of-thought traces.

Figure: Figure 8: Prompt used for CoT trace generation with DeepSeek-R1-Distill-Qwen-32B.

Figure: Figure 9: Prompt for CoT Trace Rephrasing with DeepSeek-R1-Distill-Qwen-32B.

### B.6 Sanity Checks

We investigate how a VLM would perform at short-horizon visual comprehension on images from the same episodes used for long-horizon evaluation in our benchmark. This tests if poor performance of the evaluated models on our long-horizon reasoning tasks is due to a domain or sim2real gap. We assess this by sampling frames around object interactions, asking models to list out detected objects, and check accuracy compared to the ground truth category via string matching – a conservative metric that misses correct predictions using synonyms or alternative phrasings. This still results in image sequences of length 50-150, but Gemini achieves an accuracy of 67% which is significantly higher than it’s average high level success rate across tasks in [fig. 2](https://arxiv.org/html/2506.15635v2#S5.F2). This shows that despite clear, close-up views, models struggle to reason over them when temporal sequence comprehension is required.

## Appendix C Extended Related Work

### C.1 An overview of approaches tackling memory in Long VQA and embodied tasks

Many recent works equip large language models with memory-like capabilities through task-specific knowledge bases or semantic maps for navigation planning (Min et al., 2024; Chang et al., 2023). MobilityVLA (Xu et al., 2024) leverages long-context VLMs to process past frames, enabling goal-frame selection via Gemini, while topological mapping guides navigation without explicit pathfinding. ReMEmbR (Anwar et al., 2024) and Embodied RAG (Xie et al., 2024) introduce nonparametric memory trees to store and retrieve past experiences for planning.
For video question answering, Zhang et al. (2023); Kahatapitiya et al. (2024) propose a streaming-based approach that accumulates frame-wise captions for later querying. However, this method constrains memory to the expressivity of vision-language or captioning models, potentially omitting critical details.
We include the hierarchical approach involving long-context VLMs and the approach utilizing textual memory traces as baselines in our evaluations on the benchmark tasks.

## Appendix D Baseline Details

### D.1 Gemini Agent

Figure: Figure 10: Prompt for gemini-2.0-flash model

We experiment with the gemini-2.0-flash model which can process upto 1 million tokens in the context window. We strictly follow the outlined Google Gemini API developer guidelines (gem,) to structure the prompt and perform API calls over the long videos in the FindingDory tasks. We do not use a “structured output” format such as json in the output responses as we empirically observed it failed to produce coherent outputs when supplied with long videos in the context. Since the gemini-2.0-flash model expects the input frames to be sampled at 1 FPS, we prompt the model to directly generate specific timestamps in the recommended MM:SS format to localize the frame corresponding to the target entity. We provide the detailed evaluations prompt in Fig. [10](https://arxiv.org/html/2506.15635v2#A4.F10).

### D.2 Qwen, Gemma and GLM Agent

We run the Qwen2.5-VL-7B (Bai et al., 2025), Gemma3-12B (Team et al., 2025) and GLM-4.1V-Thinking (Hong et al., 2025) locally for evaluations on the FindingDory tasks. The Qwen model uses a maximum of $768$ frames sampled uniformly across the input sequence while Gemma and GLM does not impose a maximum frame limit. For evaluation on 768 frames, we use two A40 GPUs with 48GB of memory. For experiments with shorter context lengths and higher subsampling (24–384 frames), a single A40 GPU suffices. We provide the full task prompt we use for inference with the three agents in Fig. [11](https://arxiv.org/html/2506.15635v2#A4.F11) and utilize structured JSON outputs.

Figure: Figure 11: Prompt for Qwen2.5-VL and Gemma3 model

Figure: Figure 12: Prompt for gpt-4o model

### D.3 GPT Agent

We use the official OpenAI GPT-4o (Hurst et al., 2024) API to run evaluations on the FindingDory tasks with video frames subsampled at $96$ frames to optimize API costs. For running complete evaluation on the entire validation task suite, we incurred a total cost of $\approx 400$ USD. We provide the full system prompt used for GPT-4o evaluations in Fig. [12](https://arxiv.org/html/2506.15635v2#A4.F12).

### D.4 Text Agent

We implement the the text-based VLM agent using the Qwen2.5-VL (Bai et al., 2025) model. For every chunk_size frames in the video, we generate a description using the VLM. We then attach the frame indices and time of day information to the generated summary for the frame window in consideration. We use the prompt shown in Fig. [13](https://arxiv.org/html/2506.15635v2#A4.F13) to generate the structured JSON outputs for each chunk in the video. We then concatenate all the generated JSON outputs and pass it to the Qwen model to perform text-only reasoning over the entire text-based history of the video. The text-based reasoning prompt is provided in Fig. [14](https://arxiv.org/html/2506.15635v2#A4.F14). We note that using the same Qwen model for generating the individual text summaries and performing the final text-based reasoning helps in optimizing inference load as we maintain only a single instance of the VLM through the entire evaluation.

Figure: Figure 13: Prompt to generate text summaries for chunk_size frames with Qwen2.5-VL

Figure: Figure 14: Prompt to perform reasoning over the combined textual summaries with Qwen2.5-VL

### D.5 Supervised Finetuning Baseline

We leverage the FindingDory training split which consists of $82174$ unique tasks across the categories mentioned in [Table 2](https://arxiv.org/html/2506.15635v2#S3.T2).
Each video-instruction pair is coupled with a list of ground-truth frame indices that solve the task. For multi-goal tasks, each subgoal has a corresponding ground-truth frame indices sublist.
Since the FindingDory video sequences can be extremely long, we uniformly subsample all videos to $96$ frames. The ground truth frame indices are also subsampled to match the shorter video lengths to ensure the model chooses from within the subsampled frames.

Training Details. We conduct full-finetuning of the Qwen/Qwen2.5-VL-3B model using the open-source huggingface/trl library.
For hyperparameter optimization, we utilized a smaller representative split of the training dataset with $36274$ samples and conducted a grid search over the learning rate, num_epochs and weight_decay parameters.
The best checkpoints from all runs were selected by evaluating performance on all video-instruction pairs in the full validation split.
This “offline” evaluation is conducted without instantiating the simulator by directly comparing the model’s predicted keyframe indices with the ground-truth keyframe indices using exact string matching (thus, we dont compute metrics proposed in [Section 4.2](https://arxiv.org/html/2506.15635v2#S4.SS2)).
We use the cosine learning rate decay scheduler for all experiments.
For the final training on the larger dataset, we found that a learning rate of $5\times 10^{-6}$ over $5$ epochs with 0 weight decay worked best.
For all experiments, we train in bfloat16 format using 8 A40 GPUs on a single node with a batch size of $1$ and gradient_accumulation of $4$ - providing an effective batch_size of $32$ samples.
To further optimize training, we use gradient_checkpointing and highly optimized flash_attention implementation. In this setup, the full training run takes $\approx 120$ hrs to complete.

Goal Sampling for Online Simulator Eval. The best trained checkpoint is used in online simulator evaluations on the FindingDory task suite.
Since the model is trained to predict the complete list of ground-truth frame indices, we select a single representative index from each generated subgoal solution list to serve as the predicted frame index.
Empirically, we found that choosing either the first or a random index (of a sublist corresponding to a subgoal) yields comparable performance, while using the last index results in slightly worse performance. Therefore, we use the first index of each list as a simple and effective heuristic.

## Appendix E Low-level Policy Details

### E.1 Image-Goal Navigation Agent

Policy Training.
We train the image-goal navigation policy using the architecture and training recipe described in  Yadav et al. (2023). Specifically, we employ an end-to-end reinforcement learning policy trained using DDPPO, which predicts discrete navigation actions conditioned on visual RGB inputs and a goal image. The agent utilizes a ViT based visual encoder along with a 2-layer LSTM backbone. Similar to OVRL-v2 (Yadav et al., 2023), the output patch representations from the ViT are reshaped into a 3D grid and downsampled to a lower dimension using a convolutional layer called the compression layer. However, we slightly deviate from OVRL-v2 by first concatenating the patch representations from the current and goal images before passing them through the compression layer.

The policy training occurs in the training scene split of the HSSD dataset (Khanna* et al., 2023), consisting of $1166$ episodes distributed across $111$ scenes (a subset of the original $125$ training scenes). In the Habitat simulator, the agent is modeled as a Hello Robot Stretch (Kemp et al., 2022) with a height of $1.41$ m and a cylindrical base radius of $0.25$ m. The agent’s RGB sensor is positioned at a height of $1.31$ m, with a resolution of $160\times 120$ pixels and a horizontal field of view of $43^{\circ}$. During training, each episode has a maximum step budget of $1000$ steps, with success defined by invoking the STOP action within $1.0$ m of the goal image position.

As our visual encoder, we use the VC-1-Base (Majumdar et al., 2023) model which was previously finetuned with ImageNav on a smaller training scene split from the HSSD dataset. Empirically, we found that freezing the visual encoder after this targeted finetuning significantly accelerates the overall training performance. During the VC-1 finetuning, we use a smaller learning rate of $1.5\times 10^{-6}$ for the encoder.

We train the agents for a total of $500$M timesteps on 32 A40 GPUs running 32 parallel environments each (1024 envs in total). Following the approach from  (Majumdar et al., 2023), we collect $64$ steps of experience and subsequently perform 2 PPO epochs with a mini-batch size of $2$. The reward function employed is the improved formulation proposed by Yadav et al. (2023), using parameters: success weighting $c_{s}=5.0$, angle success weighting $c_{a}=5.0$, goal radius $r_{g}=1.0$, angle threshold $\theta_{g}=25^{\circ}$, and slack penalty $\gamma=-0.002$. We optimize using AdamW with a learning rate of $2.5\times 10^{-4}$ and weight decay $10^{-6}$.

We select the best-performing policy checkpoint based on a validation success rate obtained by evaluating on $1000$ image-goal navigation episodes from the FindingDory validation set, achieving a success rate of 78% and SPL of 48.21%.

### E.2 Explicit Mapping Agent

Figure: Figure 15: Hierarchical agent architecture when using a VLM for goal frame selection and low level policies for action generation.
Refer to caption: x10.png

We augment the OracleAgent (see [Section B.2](https://arxiv.org/html/2506.15635v2#A2.SS2)) with a top-down occupancy map as it navigates the environment to generate the video sequence. In addition, we also store the 2D poses corresponding to each frame that is collected in the video sequence. The constructed map is employed by a path planning module to generate a deterministic navigation sequence to the 2D map coordinates that correspond to the image-goal location selected by the high-level goal selection agent (see [Section 4.1](https://arxiv.org/html/2506.15635v2#S4.SS1)).

Map Construction. We leverage the onboard depth sensor and project the depth image at each timestep into an egocentric pointcloud using camera intrinsics. The pointcloud is then binned along the z-axis to compute occupancy values on a local 2D grid map centered around the current position of the agent. The pose sensor provides the instantaneous pose of the agent which is employed to register the computed local map in the global map of the environment. In this way, we continuously update the global map as the OracleAgent navigates the environment and visits multiple pick-place targets. We freeze the map updates during the pick-place subroutines as no additional occupancy information is encountered during their execution.

Planner. When the high-level goal agent selects a particular goal frame, we lookup the corresponding target 2D pose in the global map. Similar to the local navigation policy employed in Chang et al. (2023), we use the Fast Marching Method (Sethian, 1996) to generate the shortest path to the goal pose. We use one of four discrete navigation actions (see [Section 3.2](https://arxiv.org/html/2506.15635v2#S3.SS2)) to reach each waypoint on the generated shortest path.

## Appendix F Cross-Benchmark Experiments

In this section, we provide details about the training and evaluation setup for results on the VSI-Bench Yang et al. (2024) dataset.
For evaluation, we follow the exact same evaluation protocol outlined in the original paper.
In  [table 3](https://arxiv.org/html/2506.15635v2#S5.T3), we report results across the multiple-choice ($2490$ in number) and numerical regression ($2640$ in number) QA categories, each of which has four sub-categories.
For all experiments, we use the Qwen2.5-VL-3B as the base model.
Each SFT/RL-finetuning experiment uses a single node of $8$ A40 GPUs with $48$ GB VRAM each.

Supervised Finetuning. We conduct supervised finetuning of model with maximum of $48$ frames in the input context window.
We also use $48$ frames during model inference as that works best between $32$, $48$ and $64$ frame inference for all baselines.
For SFT experiments with the Video-R1-CoT-165k-FindingDory mixture, we experimented with various mixing ratios by changing the amount of FindingDory data ($12.5\%,25\%,37.5\%,45\%$) and found $25\%$ ($40000$ samples) to work best.
All SFT models are trained for one epoch following the hyperparameters from Feng et al. (2025).
We only evaluate the final checkpoint after one epoch for all baselines.
Specifically, we use a learning rate of $1\times 10^{-6}$, gradient_accumulation of 2, max_grad_norm of 5 and train with bfloat16 precision on 8 GPUs
We leverage gradient_checkpointing, flash_attention and deepspeed_zero_2 to optimize training.

RL Finetuning.
For RL finetuning with GRPO, we reduce the context window to $32$ frames to avoid out-of-memory errors encountered with $48$ frames.
Emperically, we observe that inferencing with $64$ frames works better than $32$ frame inference for all baselines.
We mix the full FindingDory training dataset ($\sim 78$k samples) with the Video-R1-260k dataset from Feng et al. (2025).
Because each training step is computationally expensive, we train for only $1000$ gradient steps using a group size of $8$, batch size of $1$ per GPU, and gradient accumulation of $2$.
As a result, the model is exposed to only $\sim 2000$ unique samples during training.
We set the weight decay to $0.01$, maximum gradient norm to $5$, learning rate to $1\times 10^{-6}$, and maximum completion length to $768$.
We evaluate checkpoints at a $100$ step frequency and select the checkpoint with highest average MCQ+Numerical accuracy for each baseline.

For FindingDory samples, we design a custom reward function that rewards the precision of the predicted goal frames. Let $\mathcal{R}=(R_{1},\ldots,R_{n})$, $\mathcal{H}=(H_{1},\ldots,H_{m})$, where $R_{i}\subset\mathbb{Z}$ (reference frame sets) and $H_{i}$ are finite sequences of predicted frames.

Define the score $S(\mathcal{H},\mathcal{R})$ as

$$ $S(\mathcal{H},\mathcal{R})=\begin{cases}0,&\text{if }n\neq m,\[6.0pt] \dfrac{\displaystyle\sum_{i=1}^{n}\sum_{h\in H_{i}}\mathbf{1}\!\left[h\in R_{i}\right]}{\displaystyle\sum_{i=1}^{n}|H_{i}|},&\text{otherwise (assuming }\sum_{i=1}^{n}|H_{i}|>0\text{).}\end{cases}$ $$

## Appendix G Qualitative Examples

In this section, we include representative qualitative examples from the FindingDory task suite. In [Figure 16](https://arxiv.org/html/2506.15635v2#A7.F16), we provide a birds-eye view depiction of the trajectory executed during experience collection (see [Section B.2](https://arxiv.org/html/2506.15635v2#A2.SS2)) in a validation episode involving 5 pick-place interaction routines. For this episode, the collected video consists of $1308$ frames in total. We provide detailed responses (and meta-analysis) from various baselines across the various task categories in [Figures 18](https://arxiv.org/html/2506.15635v2#A7.F18), [19](https://arxiv.org/html/2506.15635v2#A7.F19), [20](https://arxiv.org/html/2506.15635v2#A7.F20), [21](https://arxiv.org/html/2506.15635v2#A7.F21), [22](https://arxiv.org/html/2506.15635v2#A7.F22) and [23](https://arxiv.org/html/2506.15635v2#A7.F23). In the following, we briefly discuss the different failure modes encountered in the generated responses across the 3 high-level task categories defined in [Table 2](https://arxiv.org/html/2506.15635v2#S3.T2).

Single-Goal Spatial Tasks. We observe that for the Object Recall task ([fig. 18](https://arxiv.org/html/2506.15635v2#A7.F18)), the models are able to identify a frame with the target chest of drawers but do not select the closest frames to ensure success. In the Interaction based tasks ([figs. 18](https://arxiv.org/html/2506.15635v2#A7.F18) and [19](https://arxiv.org/html/2506.15635v2#A7.F19)), we observe models fail to accurately list the sequence of interaction events which can be partially attributed to supplying only $96$ subsampled frames from the original video. In one case, GPT-4o instead focuses on a “distractor” target object ([fig. 19](https://arxiv.org/html/2506.15635v2#A7.F19)). All models perform poorly on the Spatial Relationship tasks ([fig. 19](https://arxiv.org/html/2506.15635v2#A7.F19)) as they are unable to coherently reason about the spatial layout of the house in the provided video.

Single-Goal Temporal Tasks. For the Interaction Order task, we observe that the closed-source models are able to accurately identify the target object but the open-source variants fail to do so ([fig. 21](https://arxiv.org/html/2506.15635v2#A7.F21)). Models are generally able to select the correct frame in the Time-Based task as the model only needs to retrieve a single frame corresponding to the correct overlaid timestamp. For the Duration Tracking task, only GPT-4o generates the correct reasoning and frame prediction. Intersetingly, Gemma-3 tries to track the duration spent in each room by matching the floor color observed in the images but eventually fails at the task ([fig. 22](https://arxiv.org/html/2506.15635v2#A7.F22)).

Multi-Goal Tasks. All models fail to perform successfully on tasks under this category. Only GPT-4o gets a correct response for all subgoals. Surprisingly, the reasoning trace has a hallucination but this mistake is omitted in the final frame sequence prediction ([fig. 23](https://arxiv.org/html/2506.15635v2#A7.F23)). The models struggle to coherently list the event sequence and also hallucinate interactions. Furthermore, Gemini-2.0-flash also struggles to adapt the sequence of events it lists into the query revisitation sequence specified in the task ([fig. 23](https://arxiv.org/html/2506.15635v2#A7.F23)).

**Table 4: High-level goal success rate (%) across task categories. The Oracle row is shaded in gray.**
| Method | All<br>Tasks | Object<br>Recall | Inter-<br>action | Conditional<br>Interaction | Object<br>Attributes | Spatial<br>Relation | Room<br>Visitation | Interaction<br>Order | Time-<br>Based | Duration<br>Tracking | Unordered<br>Revisitation | Ordered<br>Revisitation |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Oracle | 95.92 | 100.00 | 98.67 | 92.03 | 99.80 | 85.42 | 80.00 | 99.86 | 99.50 | 100.00 | 99.83 | 99.00 |
| Text Agent | 9.45 | 9.50 | 20.00 | 7.99 | 6.84 | 4.17 | 19.36 | 7.37 | 24.50 | 15.46 | 0.83 | 0.67 |
| Gemma3 | 13.04 | 19.50 | 9.00 | 13.22 | 17.82 | 5.83 | 30.61 | 12.31 | 28.50 | 21.13 | 0.33 | 2.00 |
| Qwen2.5-VL | 15.14 | 15.50 | 35.00 | 15.22 | 17.62 | 4.58 | 31.83 | 9.83 | 40.00 | 12.76 | 1.83 | 1.33 |
| GLM-4-1v | 23.49 | 29.00 | 49.17 | 24.59 | 30.35 | 7.08 | 36.34 | 21.05 | 50.50 | 16.11 | 3.33 | 3.00 |
| Gemini-2.0-Flash | 25.73 | 47.00 | 33.83 | 30.16 | 35.27 | 10.42 | 41.01 | 23.23 | 43.50 | 21.48 | 8.50 | 9.33 |
| GPT-4o | 27.33 | 32.00 | 40.00 | 28.19 | 25.20 | 14.58 | 44.68 | 28.56 | 63.00 | 24.83 | 6.33 | 7.00 |
| Qwen SFT | 52.44 | 64.50 | 70.63 | 46.14 | 42.65 | 24.58 | 53.47 | 68.22 | 88.50 | 53.53 | 29.70 | 20.00 |

Figure: Figure 16: An example experience collection trajectory executed by OracleAgent.
Refer to caption: figures/qualitative_traj.png

Figure: Figure 17: Example “magic” pick-place sequences executed by the OracleAgent for sample episode in Fig. [16](https://arxiv.org/html/2506.15635v2#A7.F16).
Refer to caption: x11.png

Figure: Figure 18: FindingDory Spatial Tasks for the sample episode in [Figure 16](https://arxiv.org/html/2506.15635v2#A7.F16). Each VLM model uses $96$ subsampled frames. Text Agent processes full video as defined in [section D.4](https://arxiv.org/html/2506.15635v2#A4.SS4).
Refer to caption: x12.png

Figure: Figure 19: FindingDory Spatial Tasks for the sample episode in [Figure 16](https://arxiv.org/html/2506.15635v2#A7.F16). Each VLM model uses $96$ subsampled frames. Text Agent processes full video as defined in [section D.4](https://arxiv.org/html/2506.15635v2#A4.SS4).
Refer to caption: x13.png

Figure: Figure 20: FindingDory Spatial Tasks for the sample episode in [Figure 16](https://arxiv.org/html/2506.15635v2#A7.F16). Each VLM model uses $96$ subsampled frames. Text Agent processes full video as defined in [section D.4](https://arxiv.org/html/2506.15635v2#A4.SS4).
Refer to caption: x14.png

Figure: Figure 21: FindingDory Temporal Tasks for the sample episode in [Figure 16](https://arxiv.org/html/2506.15635v2#A7.F16). Each VLM model uses $96$ subsampled frames. Text Agent processes full video as defined in [section D.4](https://arxiv.org/html/2506.15635v2#A4.SS4).
Refer to caption: x15.png

Figure: Figure 22: FindingDory Temporal Tasks for the sample episode in [Figure 16](https://arxiv.org/html/2506.15635v2#A7.F16). Each VLM model uses $96$ subsampled frames. Text Agent processes full video as defined in [section D.4](https://arxiv.org/html/2506.15635v2#A4.SS4).
Refer to caption: x16.png

Figure: Figure 23: FindingDory Multi-Goal Tasks for the sample episode in [Figure 16](https://arxiv.org/html/2506.15635v2#A7.F16). Each VLM model uses $96$ subsampled frames. Text Agent processes full video as defined in [section D.4](https://arxiv.org/html/2506.15635v2#A4.SS4).
Refer to caption: x17.png

**Table 5: Task Categories with all associated prompts**
|  | Navigate to a {target_object_name}. |
| --- | --- |
| Object Recall | Navigate to a {target_receptacle_name}. |
|  | Navigate to any receptacle you interacted with. |
|  | Navigate to any receptacle you did not interact with. |
|  | Navigate to any object that you interacted with yesterday. |
|  | Navigate to any object that you did not interact with yesterday. |
|  | Navigate to any receptacle you picked an object from. |
| Interaction | Navigate to any receptacle you placed an object from. |
|  | Navigate to the {target_object_name} that you interacted with yesterday. |
|  | Navigate to a {target_object_name} that you did not interact with yesterday. |
|  | Navigate to a {target_receptacle_name} you did not interact with yesterday. |
|  | Navigate to a {target_receptacle_name} you picked an object from. |
|  | Navigate to a {target_receptacle_name} you placed an object on. |
|  | Navigate to the receptacle that you picked the {target_object_name} from. |
| Conditional<br>Interaction | Navigate to the object that you picked from the {target_receptacle_name}. |
|  | Navigate back to a {target_shape} shaped object that you interacted with yesterday. |
|  | Navigate back to a {target_color} colored object that you interacted with yesterday. |
|  | Navigate to an interacted object with {target_print_or_design} on it. |
|  | Find an already interacted object that is made of {target_material}. |
| Object Attributes | Go back to an interacted object that is used for {target_functionality}. |
|  | Navigate to the receptacle that you interacted with which is the farthest from your current location. |
|  | Navigate to the receptacle that you did not interact with which is the farthest from your current location. |
|  | Navigate to the receptacle that you picked an object from which is the farthest from your current location. |
|  | Navigate to the receptacle that you placed an object on which is the farthest from your current location. |
| Spatial Relationship | Navigate to the object which you interacted with which is the farthest from your current location. |
|  | Navigate to the room where you picked the {target_appearance_order} object from. |
|  | Navigate to the room where you placed the {target_appearance_order} object in. |
|  | Navigate to the room where you picked the {target_object_name} from. |
|  | Navigate to the room where you placed the {target_object_name} in. |
| Room Visitation | Navigate to a room that you did not visit yesterday. |
|  | Navigate to the {target_appearance_order} object that you interacted with yesterday. |
|  | Navigate to the {target_appearance_order} receptacle that you picked an object from. |
|  | Navigate to the {target_appearance_order} receptacle that you placed an object on. |
|  | Navigate to the receptacle that you picked the {target_appearance_order} object from. |
|  | Navigate to the object that you picked from the {target_appearance_order} receptacle. |
|  | Navigate to the object you interacted with immediately after ending the interaction with {target_object_name}. |
|  | Navigate to the object you interacted with immediately before interacting with {target_object_name}. |
|  | Navigate to the object you interacted with {target_num_counts} interactions after {target_object_name}. |
|  | Navigate to the object you interacted with {target_num_counts} interactions before {target_object_name}. |
|  | Navigate to the object that you interacted with between the interactions with {target_object_name_1} and {target_object_name_2}. |
|  | Navigate to the receptacle that you placed an object on right before you started interacting with {target_object_name}. |
|  | Navigate to the receptacle that you picked an object from right after you finished interacting with {target_object_name}. |
|  | Navigate to the receptacle that you placed an object on {target_num_counts} interactions before you started interacting with {target_object_name}. |
| Interaction Order | Navigate to the receptacle that you picked an object from {target_num_counts} interactions after you finished interacting with {target_object_name}. |
|  | Navigate to the receptacle that you placed an object on between the interactions with {target_object_name_1} and {target_object_name_2}. |
|  | Navigate to the receptacle that you picked an object from between the interactions with {target_object_name_1} and {target_object_name_2}. |
|  | Navigate to the receptacle that you interacted with at {XX:XX} yesterday. |
| Time-Based | Navigate to the object that you interacted with at {XX:XX} yesterday. |
|  | Navigate to the object which took the longest time to rearrange. |
|  | Navigate to the room that you spent the most time in. |
| Duration Tracking | Navigate to the object which took the shortest time to rearrange. |
|  | Revisit all the receptacles you picked objects from yesterday. |
|  | Revisit all the receptacles you placed objects on yesterday. |
|  | Revisit all the {target_receptacle_name} you placed objects on yesterday. |
|  | Revisit all the {target_receptacle_name} you picked objects from yesterday. |
|  | Revisit all the objects you interacted with yesterday. |
| Unordered<br>Revisitation | Revisit all the receptacles you interacted with yesterday. |
|  | Revisit all the receptacles you picked objects from yesterday in specific order. |
|  | Revisit all the receptacles you placed objects on yesterday in specific order. |
| Ordered<br>Revisitation | Revisit all the objects you interacted with yesterday in specific order. |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Nothing additional; please see questions above.
soundness: 3
strengths: - The manuscript is well-written and well-organized
- The manuscript considers compelling tasks and reasoning mechanisms in Embodied AI
- The manuscript provides a reasonable set of initial experiments that provide sufficient headroom for subsequent research
- The paper provides a good amount of experiments
summary: The manuscript proposes a benchmark focused on challenging and evaluating embodied AI agents memory-intensive tasks within a virtual environment. Among the manuscripts claims are that the proposed benchmark is best-suited for evaluating long-horizon tasks, provides a comprehensive evaluation for VLM-based high-level (goal-prediction) policies as well as low-level navigation policies, and provides new effective and extensible metrics for studying/developing memory-efficient agents.
weaknesses: The manuscript is missing a principled discussion of why the tasks were generated in the way that they were. Why were the memory tasks generated according to the templates in Section 3.1, specifically? How was it ensured in the task design that these tasks are meaningful in some way, e.g., resemble naturally-occurring tasks?

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. What exactly is the distance threshold for success determination? In lines 426-427 of the paper, a threshold of 1.0 meter is selected for receptacles; however, in lines 855-856, the thresholds are specified as 2.0 meters for objects and 0.1 meters for receptacles.
2. Although both SPL and SR are expressed as percentages, they describe distinct aspects of navigation performance. Combining them in a single visualization, as done in Figures 3c and 3d, makes the comparison rather unintuitive.
3. It is suggested to add an analysis of failure causes for high-level retrieval, particularly by comparing the performance of Qwen before and after fine-tuning. This analysis would clarify the specific capabilities enhanced through the fine-tuning process.
4. In Figure 4a, the fine-tuned Qwen demonstrates a trend where success rate increases with video length. However, the experimental results are truncated at 96 frames. Could extending the video length further improve the success rate? Additionally, the input limits of different VLMs should be explicitly indicated in this figure.
5. The paper repeatedly mentions the advantage of visual realism. It is recommended to supplement with real-robot experiments to demonstrate that the fine-tuned Qwen exhibits a smaller sim-to-real gap in terms of visual perception.
soundness: 3
strengths: 1. The design of navigation tasks emphasizes that agents rely exclusively on past interaction information for high-level goal selection. This explicitly assesses VLMs' memory retrieval capabilities across diverse dimensions, including single-target spatial tasks, single-target temporal tasks, and multi-target tasks.
2. The integration of high-level and low-level policies reveals that different low-level navigation skills lead to varying degrees of task completion when performing memory-based practical navigation.
3. Tasks can be progressively extended in complexity in accordance with the evolving capabilities of VLMs.
summary: This paper presents FINDINGDORY, a benchmark developed to evaluate the long-horizon memory capabilities of agents within the Habitat simulator. Its primary focus is on assessing long-range spatiotemporal reasoning that relies solely on memory, while mitigating confounding factors particularly associated with exploration. The benchmark comprises 60 diverse navigation tasks, categorized along dimensions of memory requirements. It integrates dynamic environments, which necessitate agents to reason over evolving contexts, and supports procedural extensibility—allowing task complexity to scale in tandem with advancements in embodied agents. The paper’s main contributions are threefold: 1) the creation of 60 diverse navigation tasks that demand spatiotemporal reasoning; 2) the evaluation of high-level policies combined with low-level navigation policies in memory-intensive scenarios; and 3) the derivation of insights to inform the development of memory-efficient embodied agents.
weaknesses: Oracle agents were employed during memory collection, which appears to introduce several stringent assumptions:
1. The memory construction assumes that all consecutive subtasks are successfully completed, and each is executed in the most efficient manner, i.e., via the shortest path. The experiences collected through this method are excessively "clean" and inconsistent with real-world scenarios. This is because completing multiple subtasks is highly challenging—evidenced by a mere 26.4% success rate on the GOAT-Bench [1]. This assumption restricts the approach’s applicability in complex, real-world memory settings.
2. All memories are stored in video format. Due to the upper limit on the number of images that VLMs can process, downsampling becomes necessary during memory retrieval. However, frame-rate-based downsampling has been shown in Mobility VLA [2] experiments to cause information loss, leading to a certain degree of performance degradation—particularly in the processing of small objects. This seems to introduce a level of unfairness in the experimental setup (lines 82-83). A more efficient memory storage method should perhaps be adopted to standardize the input image sets across different VLMs, such as using informative snapshot images to store key information, as implemented in 3D-Mem [3].

[1] GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigation. 
[2] Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphs. 
[3] 3D-Mem: 3D Scene Memory for Embodied Exploration and Reasoning. 

Typo: Line 818: "the and effector" -> "the end effector"

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: (1) Could you please elaborate on the significant performance drop between HL-SR and LL-SR? How can we be confident that the benchmark results reflect the memory limitations of VLMs, rather than the fragility of the chosen low-level controller or the distribution shift between its training data and the VLM-selected goal frames?

(2) How does the FINDINGDORY setup, particularly for the evaluated baselines, differentiate itself from long-video QA tasks that test retrieval from a long context? Have you considered alternative agent architectures that would force the model to build an explicit and evolving memory representation, rather than processing the full history at each step? （If the answer is already in the appendix and I overlook it (this might happen) , please also kindly point it out）
soundness: 2
strengths: (1) The paper addresses a clear and critical gap in current research: the lack of rigorous benchmarks for long-horizon memory in embodied agents. The authors convincingly argue why existing video QA and embodied QA benchmarks are insufficient. The problem it tries to solve is of great significant.

(2) I think the two-phase setup that decouples experience collection from the evaluation phase is a very strong methodological choice. This design effectively isolates an agent's memory and reasoning capabilities from its exploration skills, allowing for a more controlled and focused evaluation.

(3) The paper provides a comprehensive analysis of modern VLMs, revealing their significant shortcomings in long-context embodied reasoning. The finding that performance often degrades with more frames for frozen VLMs, and that all models struggle with multi-goal and fine-grained spatial tasks, offers valuable insights to the community.
summary: This paper introduces a new benchmark designed to evaluate long-horizon memory in embodied agents. 

The benchmark operates in the photorealistic Habitat simulator and is structured in a two-phase process: an initial experience collection phase where an oracle agent performs a series of pick-and-place interactions over a long trajectory, followed by an interaction phase where a test agent is given the complete history (video, poses, actions) and must complete memory-dependent tasks. The paper argues that this setup effectively isolates memory capabilities from exploration challenges. 

FINDINGDORY features 60 diverse, procedurally generated task templates that probe spatial, temporal, and conditional reasoning. The authors evaluate a range of state-of-the-art Vision-Language Models (VLMs) within a hierarchical policy framework, where a high-level VLM selects a goal frame from the history and a low-level policy navigates to it. The results demonstrate that current VLMs, even powerful closed-source models, struggle significantly with these tasks, particularly those involving multi-hop temporal reasoning or multiple sequential goals.
weaknesses: (1) My main reservation is that the evaluation is tightly coupled with a specific hierarchical policy (VLM for goal-frame selection + navigation controller). The paper itself shows in Figure that the performance of this system drops massively when the low-level navigation policy is introduced. This makes it difficult to disentangle the source of failure. For example, is a low success rate on a task due to the VLM's inability to recall the correct information, or is it because the VLM correctly identified a goal, but the chosen frame provided poor visual cues for the navigation policy, causing it to fail? I think this confounding factor complicates the claim that the benchmark purely evaluates memory.

(2) Moreover, I wonder if the current evaluation truly measures memory in an agent-centric sense (i.e., the ability to build and maintain a compressed, internal world state) or if it primarily tests the long-context retrieval capabilities of VLMs on a single, massive prompt. The paper criticizes needle-in-a-haystack tasks but the baseline setup, which feeds the entire video history to the VLM at once, feels conceptually similar.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. How might the "unsolvable" task design and "magic grasp" data artifacts skew the evaluation of an agent's true memory capabilities?
2. Given the significant navigation bottleneck, how could the benchmark better isolate the VLM's high-level memory reasoning from low-level execution failures?
3. Considering the lack of uncertainty reporting and the use of external models for attribute tasks, what steps could improve the benchmark's statistical reliability and reproducibility?
4. Do authors plan to include diverse embodiments beyond the Fetch robot?
soundness: 2
strengths: + The paper addresses a crucial and underexplored area in embodied AI: evaluating long-horizon memory in realistic, interactive scenarios. The introduction persuasively argues for the gap between static video QA benchmarks and the memory requirements of active embodied tasks.
+ The benchmark uses well-specified, multi-faceted metrics, such as HL-SR/HL-SPL for high-level goal selection and LL-SR/LL-SPL for navigation. It also includes relaxed variants like DTG-SR and SC-SR to diagnose specific failure modes. The success criteria are concrete, defined by distance, angle, semantic coverage, and room region, and the evaluation procedures are clearly detailed.
+ A broad, templated task suite is provided, featuring 60 templates that cover spatial, temporal, and semantic reasoning and include multi-goal variations. This task suite is also procedurally extensible, allowing for scalable evaluation.
+ The paper delivers concrete, category-level findings. For example, it shows that VLMs can semantically detect large receptacles but fail to precisely localize them (where SC-SR is much higher than DTG-SR). Other findings include a significant gap between HL-SR and HL-SPL, which highlights VLM failure in selecting the nearest correct entity , and the extreme difficulty models face with multi-goal tasks.
summary: The authors introduce FindingDory, a new benchmark designed to evaluate long-term memory in embodied agents, addressing the limitations of VLMs in handling long-term experiences. Set in the Habitat simulator, this benchmark includes 60 procedurally extensible tasks that require agents to reason over vast image collections from past interactions to solve spatial, temporal, and semantic challenges. The paper evaluates several state-of-the-art VLMs, including GPT-40 and Gemini-2.0-Flash, and finds they struggle significantly, with the best proprietary models achieving success rates below 28%. These models performed especially poorly on multi-goal tasks and temporal reasoning, and their performance often degraded when provided with more video frames, highlighting the need for more efficient memory mechanisms.
weaknesses: + The evaluation treats "unsolvable" instructions as failures. Although the paper quantifies that 3.85% of tasks are "Originally Unsolvable", the agent is not given a way to abstain or declare a task unsolvable and must output frame indices, which diverges from realistic agent requirements.
+ Despite being mentioned by the authors as one of the limitations, data-generation artifacts may confound memory signals; specifically, the "magic grasp"  used by the oracle agent for pick-and-place actions creates abrupt, non-naturalistic transitions. This could weaken temporal cues and confuse VLMs trying to track interactions.
+ Attribute tasks rely on GPT-4o-generated object descriptions and are validation-only. This reduces reproducibility/consistency between train and val and may introduce bias from an external model. 
+ A navigation bottleneck in the hierarchical policy evaluation may mask the VLM's true memory quality. The overall success rate (LL-SR) drops sharply when the high-level Qwen VLM is paired with a low-level ImageNav policy. This is attributed to a distribution shift, as VLM-selected goal frames (e.g., from pick-place routines) often provide poor visual cues compared to those the navigation policy was trained on. This makes it difficult to isolate the VLM's memory contribution in the end-to-end system.
+ The paper reports all key results as point estimates (e.g., Figure 2) without confidence intervals or standard deviations from multiple runs. While the high cost of full evaluations is acknowledged, this lack of uncertainty reporting also extends to smaller-scale analyses like the video length subsampling study and the cross-benchmark transfer experiments, where multi-run statistics might have been more feasible.


Decision: Reject