Title: Kinaema: a recurrent sequence model for memory and pose in motion
Venue: NeurIPS 2025
Accepted: yes
Keywords: Embodied AI

sequence model

agent memory

relative pose estimation
Abstract: One key aspect of spatially aware robots is the ability to "find their bearings", ie. to correctly situate themselves or previously seen spaces. In this work, we focus on this particular scenario of continuous robotics operations, where information observed before an actual episode start is exploited to optimize efficiency. We introduce a new model, "Kinaema" and agent, capable of integrating a stream of visual observations while moving in a potentially large scene, and upon request, processing a query image and predicting the relative position of the shown space with respect to its current position. Our model does not explicitly store an observation history, therefore does not have hard constraints on context length. It maintains an implicit latent memory, which is updated by a transformer in a recurrent way, compressing the history of sensor readings into a compact representation. We evaluate the impact of this model in a new downstream task we call "Mem-Nav", targeting continuous robotics operations. We show that our large-capacity recurrent model maintains a useful representation of the scene, navigates to goals observed before the actual episode start, and is computationally efficient, in particular compared to classical transformers with attention over an observation history.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a new recurrent model termed Kinaema for the Mem-Nav task where the agent first explores the environment and then navigates to the target given a query image. Compared to the quadratic complexity of attention O(N^2), Kinaema is designed to update and query memory in O(1). In addition, gating functions are integrated into Kinaema to decide the speed of updates for each memory item. \n\nExperiments comparing Kinaema with several recent works and classic RNNs demonstrate its advantages in the Mem-Nav task especially for long-term navigation. Ablation studies and visualizations are provided with new insights.\n\nSTRENGTHS AND WEAKNESSES:\n__Strengths__: \n\n__S1__: A new recurrent model is introduced with complexity of O(1) to improve classic RNN and attention’s limitations.\n\n__S2__: Figures and tables are presented in high quality.\n\n__S3__: Experiments are comprehensive including comparison with several recent methods an classic RNN baselines.\n\n__S4__: It is good to demonstrate that Kinaema outperforms GRU in each setting.\n\n__S5__: It is good to include experiments with longer trajectories T=200 T=800 to show the memory’s advantages.\n\n__S6__: The appendix is provided with great detail and includes a demo. \n\n\n__Weakness__: \n \n__P__: Overall, the presentation could be further improved, including __W2-4,7,8__. \n\n__W1__: Differences between Mem-Nav and “Hard” ε-greedy. \n\nThe description of “Mem-Nav” from Line 36-38 sounds like a hard version of the classic ε-greedy algorithm in the Reinforcement Learning [1] textbook: \n\n\"The method that chooses randomly among all the actions with equal probability ε, and greedily with probability 1 − ε, is called ε-greedy.\" \n\nSince this work falls within the scope of deep reinforcement learning, I believe it would be beneficial to more clearly differentiate the task from the ε-greedy framework or to discuss the conceptual connections between them. \n\n[1] Sutton et al. Reinforcement Learning: An Introduction. \n\n__W2__: Line 58 “(ii) “Mem-RPE”, a new task requiring the estimation of relative pose between an image and agent memory; “ \n\nThe term “RPE” appears here for the first time without an explicit interpretation. Providing a clear explanation of the abbreviation “RPE” at its first mention would improve the clarity of the writing. \n\nLine 52-53 in the Appendix provides a good example: “As mentioned in Sec. 4.1 of the main paper, we train all models by jointly optimizing the relative pose estimation (RPE)”. \n\n__W3.1__: The presentation of Table 1 – 4  can be improved. To be concrete, the numbers are left without explanation. Table 5 and Figure 4 are good examples. \n\n__W3.2__: What do the values in green in Table 1 represent? Line 236 mentions two metrics of Success Rate (SR) and SPL, but neither is explicitly referenced in the text or table caption. If the values correspond to Success Rate, it would be helpful to include a percentage sign (“%”), as the numbers appear to range from 0 to 100. Without such clarification or indication of scale, it is difficult for readers to interpret the results. A similar issue is present in Table 4 in the Appendix. \n\n__W3.3__: “SPL” appears for the first time on Line 238 without explanation. Although it is defined in the referenced literature, it would be helpful to also provide the full name within this work for clarity. \n\n__W4__: In Figure 3, the Y-axis is labeled “Mem-RPE Acc. (%)”, but the abbreviation “Acc.” is never used in the main text. Instead, only “Success Rate (SR)” and “SPL” are introduced in Lines 236–238 under the Metrics / Mem-Nav section. It would be helpful to maintain consistent terminology throughout. Similar inconsistencies are also found in Figures 2 and 3 in the Appendix. \n\n__W5__: Figure 4 does not show clear advantages over the standard DEBiT without the introduced modules. It would be helpful to include a discussion or provide new insights to clarify the benefits. \n\n __W6__: It would be great to provide further discussions of the poorer performance “0 11 22” in Seq len 800 in the last row in Table 2 (a). \n\n__W7__: SR and SPL are reported separately, rather than presented together, in Tables 1–4 and Figures 3 and 4 (Assuming Tables 1-4 represent SR results). It would be more comprehensive to report both metrics side by side, in line with common practice in the literature. \n\n__W8__: It is good to include attention visualizations in Figure 6, but they are difficult to interpret without additional context. For instance, appending a goal image could help assess whether the token attention is meaningful. Further explanation of the color coding would also be helpful. \n\n\n__Minor Issues__:\n\n__M__: “but using a GRU; (with h0-injection) is a baseline using the hidden state ht of the original agent to encode the priming sequence with no goal, teacher forced,” \n\nThe notation of “;” looks weird between “GPU” and “(with h0-injection)”.\n\nQUESTIONS:\n__Q1__: Line 60: “(iii) the integration of the sequence model in a navigation agent trained with Reinforcement Learning;” It would be great if the authors can explain where the Reinforcement Learning (RL) part is in the paper. \n\nExcept for Mem-Nav that looks like the hard version of ε-greedy (__W1__) which relates to RL, the main training is about minimizing the discrepancy of the relative pose between a query image and the current agent’s position, denoted as L_{RPE} on Line 199. However, this formulation does not explicitly relate to the basic framework of RL that involving reward, action, and state. In particular, the paper does not describe a reward function, therefore it is unclear how the agent is “trained with Reinforcement Learning” as claimed in the (iii) contribution. \n\n__Q2__: Line 110 “as p = {t, R}, where t = {d, \\theta} is the translation, i.e. distance and bearing angle from the agent to the position depicted in query image. R is the rotation matrix of the goal towards the agent, which is of limited relevance to a navigation task, but which we supervise to increase the learning signal during training.” \n\nThere are two rotation-related notations of R and \\theta, how are they related to each other? Why is only R used for training but not the other one? \n\n__Q3__: What kind of gating functions are used (Line 145)? Does it follow the standard gating function implementation in the literation? Is it Sigmoid or Tanh? It might be beneficial to add such details for completeness.  \n\n__Q4__: What does each color represent specifically in Figure 7?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Kinaema, a novel high-capacity recurrent sequence model designed for embodied agents that operate in long-horizon, continuous visual navigation tasks. The key innovation lies in maintaining a distributed memory composed of multiple embeddings, updated recurrently via a Transformer block and stabilized using gating mechanisms inspired by GRUs. The authors evaluate Kinaema on two newly proposed tasks: Mem-RPE (estimating relative pose between a query image and the agent’s latent memory) and Mem-Nav (navigating to a previously seen target location, using only prior observations). Empirical results on simulated 3D environments (HM3D, Gibson) demonstrate that Kinaema outperforms multiple recurrent baselines (GRU, xLSTM, MooG) in both memorization and navigation efficiency, especially for long sequences. Ablations support the importance of transformer updates, gating, and distributed memory structures.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The paper introduces Mem-RPE and Mem-Nav, two tasks that explicitly target the underexplored and practically important setting of continuous memory-based navigation, aligning more closely with real-world robotics deployment than traditional episodic benchmarks.\n\n2. The Kinaema model is designed to support O(1) memory updates and access, in contrast to the O(N²) cost of standard attention-based transformers. This architectural choice is particularly well-suited for long-horizon tasks where memory scaling is critical.\n\n3. The paper provides a thorough empirical evaluation, including generalization to out-of-distribution sequence lengths, memory scaling studies, and ablation analyses. The inclusion of multiple competitive baselines (GRU, EMA, xLSTM, MooG) allows for a fair and comprehensive comparison.\n\n4. The paper is well-organized and clearly written, with strong exposition of model architecture, training setup, and evaluation procedures. Visualizations such as memory attention maps and cross-attention overlays further aid understanding.\n\n---\n\n## Weaknesses\n\n1. **Streaming history is modeled as random exploration** via varied-length sampled trajectories, rather than through an **active or structured exploration policy**. As a result, there is no analysis of **visual coverage or informativeness** of the memory contents. This limits insights into whether the model benefits from task-relevant scene exploration, which would be expected in real-world deployment.\n\n2. The **model assumes that the goal image comes from a previously observed location**, but the model lacks explicit uncertainty estimation (e.g., recognizing if a goal has not been seen), or out-of-distribution detection.\n\n3. In the **Mem-Nav task**, Kinaema is **used as a frozen plugin module** to DEBiT rather than training a end-to-end navigation policy with recurrent memory. This modular design limits the agent’s capacity that learns to memorize navigation history and learn about the environment lifelong. In another word, **the memory is not updated during navigation**, making it unsuitable for dynamic scenes or tasks that require continual perception.\n\n4. The potential for **lifelong memory** or **multi-episode navigation** is not addressed. Recent work such as **GOAT-Bench** (Khanna et al., CVPR 2024) directly tackles multi-task or long-term memory settings and could provide a more fitting benchmark or discussion context.\n\n5. While comparisons to GRU, xLSTM, EMA, and MooG are helpful, **Mamba (Gu & Dao, 2024)** is notably missing. Mamba’s state-space approach also introduce a distributed memory (D SSM applied to each channel), and would serve as a more **direct architectural comparison** to Kinaema than classical RNNs.\n\n6. Although the model is trained on **sequences of length T=100**, evaluation is extended to sequences of up to T=1000. However, there is **no justification for the limited training horizon**, nor any exploration of **scaling effects** (e.g., whether longer training sequences would improve generalization). The sharp drop in accuracy between T=100 and T=200 suggests potential **brittleness and overfitting** to the training length.\n\nQUESTIONS:\n1. Test Time Training introduces fast weights and could also memorized long sequence with O(1) computational complexity. The authour could add some disscussion on using recurrent transformer compared to TTT.\n\n2. Besides using architecture with O(1) computational complexity, the author could also compare to methods merging KV cache as memory in transformer.\n\nLIMITATIONS:\nThe potential for **lifelong memory** or **multi-episode navigation** is not addressed. The proposed navigation agent is a modular pipeline and **the memory is not updated during navigation**, making it unsuitable for dynamic scenes or tasks that require continual perception.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new model architecture for storing and retrieving long-term memory specifically for the task where a robot needs to estimate the relative pose between a target image and the current pose. While the ideas presented make intuitive sense, I find the writing hard to follow in some cases.\n\nSTRENGTHS AND WEAKNESSES:\nQuality\n\nI like the discussion around the desired behavior of the model and the consideration that go into designing the model architecture at the beginning of section 4. The discussion makes sense and provide a holistic explanation for more technical ideas that follow.\n\nClarity\n\nOn line 144-145, the paper writes this `allowing the model to take decisions (to “gate”) on the speed of updates for each memory item`. What does it mean to take decision on the speed of updates?\n\nOn line 172-173, \"the paper writes the following self-attention transformer contextualizes the embeddings with each other\". Self-attention is an operation. What does it mean to have a self-attention transformer? And what does contextualizes the embeddings with each other mean?\n\nIt is also unclear the relationship between the model described in section 4.1 and background information provided in section 4.\n\nSignificance\n\nI agree with the argument in the paper that the more traditional navigation setting doesn't lend itself well to study longer memory context problem. However, I would have like more description on the task in the background session to be able to understand the experiment better.\n\nOriginality\n\nThe experimental setting that the paper studies and the ideas presented are novel to the best of my knowledge. I don't work in navigation however and I'm not very familiar with the literature.\n\nQUESTIONS:\n1. How does the Kinaema model fit into the steps describe in the background in the beginning of section 4\n\nLIMITATIONS:\nplease see my comments above", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}