Kinaema: a recurrent sequence model for memory and pose in motion
Authors: Mert Bülent Sarıyıldız
Philippe Weinzaepfel
Guillaume Bono
Gianluca Monaci
Christian Wolf
Venue: NeurIPS 2025 poster ()
Keywords: Embodied AI
sequence model
agent memory
relative pose estimation
Primary area: reinforcement_learning
TL;DR: We propose a recurrent transformer for robotics, which can maintain a compact representation of the history of observations useful for continuous robotic operations.

Abstract: One key aspect of spatially aware robots is the ability to "find their bearings", ie. to correctly situate themselves or previously seen spaces. In this work, we focus on this particular scenario of continuous robotics operations, where information observed before an actual episode start is exploited to optimize efficiency. We introduce a new model, "Kinaema" and agent, capable of integrating a stream of visual observations while moving in a potentially large scene, and upon request, processing a query image and predicting the relative position of the shown space with respect to its current position. Our model does not explicitly store an observation history, therefore does not have hard constraints on context length. It maintains an implicit latent memory, which is updated by a transformer in a recurrent way, compressing the history of sensor readings into a compact representation. We evaluate the impact of this model in a new downstream task we call "Mem-Nav", targeting continuous robotics operations. We show that our large-capacity recurrent model maintains a useful representation of the scene, navigates to goals observed before the actual episode start, and is computationally efficient, in particular compared to classical transformers with attention over an observation history.

Kinaema: A recurrent sequence model for memory and pose in motion
Mert Bulent Sariyildiz
Philippe Weinzaepfel Guillaume Bono
Gianluca Monaci Christian Wolf
NAVER LABS Europe
[bulent.sariyildizz www.yalddd/ (esttname)@nave1111 .om
https://europe.
naverlabs htapar//eurrpe..avelllbeee com/kinaema
Abstract
One key aspect of spatially aware robots is the ability "find their bearings", i.e. correctly situate themselves in previously seen spaces. In this work, we focus on this particular scenario of continuous robotics operations, where information observed before an actual episode start is exploited to optimize efficiency. We introduce new model, Kinaema, and agent, capable of integrating stream of visual observations while moving in a potentially large scene, and upon request, processing a query image and predicting the relative position of the shown space with respect its current position. Our model does not explicitly store an observation history, therefore does not have hard constraints on context length. It maintains implicit latent memory, which is updated by a transformer in a recurrent way, compressing the history of sensor readings into a compact representation. We evaluate the impact of this model in a new downstream task we call "Mem-Nav". We show that our sargeccapacity recurrent model maintains useful representation of the scene, navigates to goals observed before the actual episode start, and is computationally efficient, in particular compared to classical transformers with attention over an observation history.
Introduction
Visual input stream
The majority of work in embodied AI. in particu- lar methods based on machine learning, work in episodic settings: the agent begins with clean empty internal representation every start, dealing with every episode as if it was the first one after unpacking the robot after its purchase. This is in stark contrast to realistic robot operations, where
DINO-v2
Enc
Recurrent Transformer
m
Dec
Rel. pose
Memory
we would expect robot to able to exploit infor- mation the scene observed previously.
Enc
Downstream agent
In this work, we propose model capable spa- tially situating previously observed spaces. While applicable to broader class of downstream tasks, we focus navigation and new continuous vari- ant the ImageNav task: an agent is given goal image and is required to navigate to the position shown in this image. In the case when the goal not seen from the starting position, classical solu- tions resort to an exploration strategy, patrolling the scene until the goal is observed. our new
Query image
(navigation task)
Figure 1: We introduce Kinaema. model ca- pable of situating previously observed spaces: recurrent transformer compresses observed sequences into latent memory and estimates rel. pose of goal image w.r.t. its current state.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

setting focusing on continuous operation, "Mem-Nav", the agent can explore the scene before each episode starts to build up a latent representation, but it does not have access to the future goal at this point - it has to build a general representation suitable for any potential future goal.
Previous work addressing continuous navigation mainly focuses on map-based solutions [40, 61], building, maintaining and querying a metric or topological map during operation. We address this problem in purely data-driven way with models and agents which compress a potentially long sequence of visual observations into high-capacity latent representations, denoted m, in Fig. 1. The recurrent nature of our model is a key design feature: given a history of observations of length N, sequence models based on transformers update their representations with O(1), essentially stowing away the imput, but query it in ( due to the quadratic complexity of attention. In contrast, our recurrent model, both, updates and queries memory in ( Classical recurrent networks, on the other hand, suffer from scaling limitations as their network capacity scales quadratically with respect to memory size [28]. We introduce a new recurrent sequence model, which decouples these aspects and maintains memory in the form of a set of embeddings, which are updated with a transformer.
The main skill required for Mem-Nav is the capacity to situate previously observed space, which we directly supervise during a pre-training phase: we train the recurrent model to estimate relative pose between a query/goal image and the current agent position, assuming that the query image depicts a point of the scene which had been previously observed, albeit potentially from a different viewpoint. This text task may be somewhat reminiscent of classical relative pose estimation, but is fundamentally different: compared to classical binocular geometric foundation models comparing pairs of images, eg. DEBiT [9], our model compares single query image to latent agent memory. We combine this pre-training task with a memory based variant of masked-image modeling.
In summary, we introduce the following contributions: (i) a new recurrent sequence model "Kinaema" with distributed memory and transformer-basee update; (ii) "Mem-RPE", new task requiring the estimation of relative pose between an image and agent memory; (iii) the integration of the sequence model in a navigation agent trained with Reinforcement Learning; (iv) new downstream navigation task "Mem-Nar" allowing an agent to access observations collected before the episode start.
2 Related work
Visual navigation has been addressed in robotics for a long time by explicit models [13, 38, 39] based on mapping and localization [12, 34, 56], and explicit planning [32, 52]. ML-based solutions are typically trained on photorealistic simulators [31, 48]. Modular agents [14] decompose the problem in sub-modules, whereas end-to-end trained models directly map input actions with Reinforcement Learning (RL) [27, 42, 57, 68], Imitation Learning (IL) [19], or offlineRRL [54]. Image goal navigation, "ImageNav", adds a skill linked to relative pose estimation, which explicit methods have addressed with local feature matching [33], or by retrieving features from a topological map [7]. End-to-end trained agents compare images by extracting binocular features with [1, 53, 67, 69], potentially directly pre-training for RPE [9]. Modular approaches have also been proposed [18, 65].
Continuous navigation has previously been cast problem where episodes are divided in multiple sub-episodes, where later sub-episodes are supposed to exploit information seen earlier. Common formulations are the K-items scenario [6]. Multi- Object Navigation [40, 61] or the GoathBench [30]. Our "Mem-Nav" task decomposes the problem into a priming sequence of fixed trajectories followed by navigation episodes, which allows to accelerate training by pre-computing priming representations.
Sequence models were early on implemented with recurrence in RNNs, LSTMs [26] and GRUs [16]. Their memory capacity scaling problem had been addressed by external neural memory [22, 55, 70]. but was then eclipsed by transformers [59], which replaced recurrence altogether by attention. While attention over time still dominates multiple fields like NLP and CV, currently recurrence makes inroads again, either through state space models like $4 [24], Mamba [23] and LRU [45] inspired from control theory, or by combining it with attention: XLSTM [5], MooG [58] and Token Turing Machines [47] are prominent examples. We introduce a new transformer- based recurrent model and show that it scales favorably, can be trained for long sequences and generalizes to even longer ones.
Relative pose estimation (RPE) between two images was tackled for decades with pixel-level image matching techniques [25, 43, 49]; before learning- based approaches were proposed [29].
TTis expresses complexity depending on the history size N only.

---

Self-supervision was quickly introduced [36]. More recently, DUSt3R [60] regresses pointmaps, while MASt3R [35] additionally learns descriptor inspired by image matching. Both leverage CroCo [62, 63] for pre-training- Similarly, MicKey [4] regresses pointmaps and supervises relative pose alone with differentiable RANSAC formulations [8, 11]. All these recent methods have led to impressive results for relative pose estimation, even under scenarios with little overlap between input images as in the MapFrre-Relocaliattie benchmark [3]. Compared to the standard formulation, our "Mem-RPE" task requires estimating the pose between an image and the agent's 's memory.
3 The Mem-Nav and Mem-RPE tasks
Mem-Nav We study navigation in photorealistic 3D environments, where an agent is given goal image E xx and is required to navigate from a starting location to the position shown in the goal. At each time step I the agent observes pair of sensor readings o, = (x,; u.), where X2 E x is an RGB image of size 112x112, and u, E R? is an odometry estimate in the form of difference of agent poses between and 1-1.
In contrast to classical navigation tasks in embodied AI, we model a continuous navigation setting by dividing each episode into two different parts: An initial priming sequence of length P. around 200 steps, in which the agent explores the scene and has access to observations {or ) but not yet the future goal. During this initial sequence, the agent cannot chose its actions and follows a predefined path. From step P+1 on, the agent receives the goal image additionally to the observations (or PP.1.... and must navigate by predicting actions a, each step. The action space is the discrete set A A forward 0.25m, turn left 10, turn right 10, stop). An episode is considered successful if the agent calls the stop action within Im of the goal position and within its 1000 steps budget. We use the Habitat simulator [48].
Mem-RPP an intermediate skill Navigating to previously seen positions efficiently requires the capacity to predict where they are, and we train our model for exactly this skill. We introduce the new sub-task of relative pose estimation between memory m, maintained by the agent, and a query image q, as = {t. R., where t = {d. 8} is the translation, i.e. distance and bearing angle from the agent to the position depicted in query image. R is the rotation matrix of the goal towards the agent, which is of limited relevance to a navigation task, but which we supervise to increase the learning signal during training.
agent position at time represented by latent
Situation awareness with latent memory
We designed new sequence model around the following goals targeting continuous robotics:
(G1) Recurrence - Classical auto-regressive models attending over a history of observations are required to revisit every single historical item for each step. Not only is this repetition wasteful in itself, it is further exacerbated by the quadratic computational complexity of transformers of O(N²) given N observed items from the past. We target models maintaining memory representation m, updated at each step given the current observation o, only, and queried directly, without again considering other historical information, leading a complexity of O(1) plus complexities arising from the distributed nature of the memory itself.
While transformers attending over time are currently the dominant models in embodied AI, and we do not claim to argue against their usage, in this paper we argue that the dominance might potentially be an overfit of the scientific process to the short episode lengths from existing benchmarks. While this is suitable for certain application which do not require long-term memory, like manipulation, we think that does hold back research in areas where very long-term memory is necessary. Some applications in robotic navigation fall into this case, where it is advantageous to remember information seen minutes, hours, or even days ago.
(G2) Memory capacity- - Holding actionable information about an entire observed scene requires scaling the memory size. Unfortunately, classical recurrent models like RNNs, LSTMs and GRUs are held back by the direct coupling of memory capacity and network capacity with a quadratic relationship determined by their update matrices. We address this by introducing a distributed hidden state m, of N embeddings of size E, and modeling the update function as a transformer. This ensures that the capacity of the network (transformer) can be chosen independently of the memory capacity by scaling N.
3

---

(G3) Stability - While we advocate for recurrent models for the reasons given above, they come with a shortcoming: training requires back-propggating gradients over memory update chains spanning over the sequence length. This is in contrast to transformers attending over time, where the sequence length is dealt with attention: while the number of attended items as large as for recurrent models, the length of the gradient multiplication chain is not related to the context length, as items are not integrated sequentially, making these models more stable. We address this issue by combining research from classical recurrent models with an architecture from the transformer literature: we add gating functions to the memory update, allowing the model to take decisions (to "gate") on the speed of updates for each memory item.
In what follows, we introduce a new high-capacity recurrent model maintaining latent representation of an observed scene, but as we will compare it to several recurrent baseline models from the literature in the experimental section, we start with quite general equa- tions which fit all tested models. All considered models maintain some form of memory m, over time steps i, and which are characterized by concrete implementations of the following functions:
X (XI)
1/ Eacoda visual Input
m
m
u, EnCodo (us)
// Eacode odometzy input
m, Update (addate(m;:11) (
Xe: üt)
// Update nenory
-1++
Enc
Update
Read
Yt
Yt Read (m,)
// Read out menory
X
Enc
P. = Dyc Encgouu
// Dacoda relative pose
g + Enc
Decode
P
Starting with the commonalities, all models encode the visual input with a Vision Transformer (ViT) [20], (x.) which we initialize with the weights of DINO-v2 [44] ViT-Small/ 14, and finetune the weights during training. Odometry inputs are encoded with an MLP, Encado (nc) All models update memory at each time step and also project memory m, into a set of embeddings y: Reod(m) (m,) through read-out mechanism. Both functions are tailored to each model, given further below. And lastly, we train all models in a supervised manner by predicting relative pose Pa with decoder Dec, which is implemented query image and read out memory, followed by self-attention,
transformer with cross-attention between encoded
P. Alnns- Enc g), K Yt, V = yt)
P. SelfAUu((P.).
(2)
For clarity we omitted residual connections and FF layers from the notation, pose is predicted from an additional CLS token added to the inputs.
4.1 Kinaema memory in motion
We call our model "Kinaema", a neologism from kinema (motion) and mnema (memory). maintains set of N embeddings m, {m,..) of dimension E, and its update is recurrent (Goal G1) and implemented as transformer, ensuring that the memory size can be scaled by increasing N without having to modify the network capacity (Goal G2). This is achieved by modeling the memory update m,- = Update (m,-1, X2, û,) as follows, also shown in Fig. 2. Each memory embedding m is summed with a learned positional embedding et,n combined with the encoded observations (Xe, ü;) through concatenation, and then encoded, resulting in memory embeddings corrected (in a Kalman-like sense) by the observations:
corr m
Linear (mm + eay X1, ür]);
where [...] denotes concatenation over the embedding dimension. The output size of the linear layer is F. While the last operation deals with each embedding independently, the following self-attention transformer contextualizes the embeddings with each other,
m SelfAttn(mm (orr))
The resulting set of embeddings m, corresponds to the update candidates, which are then subject to gating to increase training stability (Goal G3) and to model different speeds of dynamics in memory. The gating block operates on each embedding independently, giving
m Gating Benting(m,-j (m:11,n m.).).
(5)

---

m
m
Embedding wise
Gating
NxEE
NxE
Learned R.E.
Embedding
m
Ut Odometry
cat
-wise
Read Reshope
Enc
Linear
Trans-
former
NxE
Enc
NxE
Updote condidates
yt
DINO-v2
repeat
X Visual observation
N'XE
Figure 2:
Kinaema is a recurrent sequence model maintaining distributed memory m, in the form of N embeddings of size E each. Its previous state m,-1 is first contextualized with observations o, = [x,, Ut) then embedding-wise gated, resulting in new state my.
Gating is inspired by GRUs by adding update gate and forget gate to the model [16], and actually exploit this by implementing the gating block as a GRU cell with weights shared over the N embeddings of m,, This comprises an essential difference between GRUs and Kinaema: in classical gated recurrent networks the gating mechanism is of the same complexity as the actual update function (the parameter matrices for the update and reset gate of size identical the matrix handling the update of the hidden state), which would be intractable for model as large as our model with large distributed state. Kinaema, the actual state update is handled through the transformer given in Eq. (4), whereas the gating is done by block weight-shared over memory embeddings. This choice allows to scale memory capacity easily without impacting the capacity of the gating block. In Sec. 5, we will show that both the transformer block in Eq. (4) and the gating block in Eq. (5) are essential for good performance.
The read-out block projects memory embeddings to a representation useful for the downstream decoders. For our model, this is implemented as a basic reshape, shaping the memory tensor from NxE to N'xE' yı ReshapeNt F (m.). In our experiments, we will show, that it is interesting have fewer memory of embeddings of higher embedding dimension, and read them out into larger number embeddings of lower dimension for decoding.
Training - We train sequences of randomized lengths T between 50 and 100 time steps, and by taking the memory my at the last time step, predicting relative pose for 27 query images of two different types: (i) the T observed images (x, h...r, and (ii) T alternative images {xalt) h...T, which have not been observed but lie in the observed region of the scene. We generate them with the same simulator by slightly disturbing the pose of the observation of the corresponding time step L. These frames prevent the model to learn simple lookup table. We train with supervision,
CRPE = Σ t, + R. - R; where (t,, R2) and (t), R²) are predicted and GT pose for training image respectively. We add auxiliary masked image modeling loss, which reconstructs the same query images after they have been masked. We add a second decoder head querying memory with the same cross-attention mechanisms the RPE decoder in Eq. (2) Appendix B.
4.2 Integration into the downstream navigation agent
We address the Mem-Nav task introduced Sec. by augmenting the DEBiT agent from [9]. which currently achieves state-of-the-art performance on the ImageNav and Instonce-lnageNan tasks:
D obs
BinEnce g)
Binocular encoder get geal direction
h GR p? RN,))) MLP(a.-1))
Recurrent nenory updato
p(a:) = Thi).
// Linear policy
(6)
This agent maintains a recurrent GRU memory h, fed with visual observations x, encoded by ResNet- 18. More importantly, it compares each visual observation x, with the goal image using
5

---

Kinaeama
Pose decoder
Read
y t
m
Kinaema memory
d, 0, R
Enc
BinEnc
Agent
U
EnG
Update
Gool Imoge
obe
mem
EnC
GRU
Z Action
a
Observation
h Agent memory
Figure 3: Integration into the downstream RL-trained agent: The RPE decoder used for pre- training is kept for the downstream task, searching for the goal image in the embeddings yr which are read out from memory m,, while binocular encoder BinFne from DEBIT [9] is used compare the goal to the current observation. Two types memory are updated and queried: (i) the agent maintains its additional recurrent memory h,, and (ii) Kinaema-memory m.-
binocular transformer BinEnnc, pre-trained for relative pose and visibility estimation between pairs of images, to extract information on the goal direction.
The DEBiT' agent is capable of efficiently detecting goals when they visible, but does not handle previously seen goals, in particular when they have been observed before an episode start. This is where our proposed
sequence model comes in i-- is run in parallel, receives the visual observations X, as the main agent, and contributes with goal direction estimates.
As illustrated in Fig. 3, the augmented agent is given
P BinEnc(X;, Xx, g): M using Eqs.(1)
Binocular eccoder get goal dired
11 Kinsena menory update
P
mem
using (qs.(1),
1/ Kinaena get goal direction Recurrent nonory update
h GRU (h, ohs
Pt. mam MLP(a--)))
p(a,) (h,).
// Linear policy
where pm is given by the relative pose decoder of the Kinaema model, denoted as Pt in Eq. (1). Inspired by [9], for both pose estimates, oobs and p!" we provide the latent encoding of the pose taken from the penultimate layers of the respective networks, and not the decoded pose values themselves.
we train the parameters of the policy the recurrent network GRU and the monocular ResNet encoder RN jointly with PPO [50] for 300M steps, with a reward definition in the lines the one proposed by [15] for PointGoal and re-used by [9] for ImageGoal, = K success AGeo X,
Training
where 1010, A Geo is the increase in geodesic distance to the goal, and slack cost 0=0001 encourages efficiency.
We initialize the agent from publicly available trained DEBiT-B model provided by [9], and train the remaining parameters from scratch. Since the size of the GRU was increased by the additional inputs, of the extended agent, this required block-wise initialization of the weight matrix which project GRU input to latent memory space:
W Kinaema-sh
0
W manoc-sh
W monoc-sh pre-trained from
W-
W binoc-th
= W binoc->h pre-trained from
actt>h
Wact-sh pre-trained from
Just after initialization, the extended agent gives the exact same outputs as DEBiT-B [9], We then re-initialize the linear policy layer with default uniform distribution.
5 Experimental results
Experimental setup We trained the models on the HM3D [46] and Gibson [66] datasets and created sound set of splits allowing clean separation of pre-training, training and evalua-

---

Model
Mem
Obs
Seq
200
Seq len 800
size
hist
Im
Im
2m
Im
Im
2m
70
Kiraema/mmo"
GRU1m1O"
1g
90
ape
10"
9ne
ago
Kraemalmmo"
churlmoo
Trunc. Trst..Hist.
41.6k
/
11
28
6
16
Acc.
Kraamaa2m30
GRU22000
MooG
524.3k
X
0
5
14
0
3
9
LRU
3.1k
X
4
18
34
2
9
20
Mem- Mem-RPE
EMA
153.6k
24
XLSTM
2,359.3k
X
8
23
47
5
13
29
GRU [
3.1k
X
12
32
56
4
14
31
Kinaema
61.4k
X
21
41
63
10
21
37
Evaluation sequence length
Figure 4: Generalization to longer sequences,
Table I:
Comparisons of models on Mem-
RPE: Kinaema has N-20 memory embed-
Mem-RPE: GRU and Kinaema, trained for T=100, evaluated on 10000....0000 1000 (RPE-test) )
dings; EMA uses a trainable X (RPE-test). ).
tion: RPE-train used for Mem-RPE training and consisted of sequences sampled from the HM3D/train scenes. RPE-val contains sequences generated from Gibson//rran scenes and was used for checkpoint selection and ablations. RPE-test contains sequences generated from HM3D/val scenes and was used for final evaluation and model comparisons. NAV-train was used for navigation training on new episodes generated from the HM3D/train scenes: starting poses match pre-generated offline priming sequences. A similar arrangement was done for NAV-test, based on HM3D/val scenes. All tables have color-coded backgrounds indicating the splits. More details in Appendix A.
Metrics Mem-RPE - All models have been trained with randomized sequence lengths sampled between 7'=50 and T' 100. We systematically evaluate all models in significant -uito--distribution settings, generalizieg to two different validation sequence lengths of 200 and 7-800, respectively. We provide accuracy of correctly recognized poses with three tolerance margins: less than 1m of translation and 10 of rotation errors, < Im and 90º, and < 2m and 90% We put more emphasis on low translation errors, almost disregarding goal rotation: this is goal rotation towards the agent, irrelevant for navigation. Rotation fowards the goal, a.k.a. "bearing", is part of the translation error.
Metrics Mem-Nav - Navigation performance is evaluated by success rate (SR), i.e., fraction of episodes terminated within a distance of <Im to the goal by the agent calling the stop action, and SPL [2], i.e., SR weighted by the optimality of the path, SPL N Σi S; maxx(,,77f where S; be binary success indicator episode i, L is the agent path length and E the shortest path length.
Baseline models - We implemented the following baseline sequence models, which were adapted to the task by adding read-out mechanism and the same RPE-decoder described in Sec. 4, Eq. (2). More details on them is given in Appendix D.
GRUs [16] model memory m, as a single vector and were implemented with the standard Py implementation, Update - m, Wm,- + UX, where we omitted gating equations from the notation. We explored multiple numbers of layers and we made the memory read-out function non-linear with (non-shared) MLPs, Read(m;) A {MLPa, (m:)):-
EMA [21] models memory update as an exponential average, Update & m, Am;- + Ux,-
with the readout being a simple reshape. They are simple but provide interesting guarantees; more importantly, lacking any learned dynamics, they allow to evaluate the impact of significantly increasing memory capacity without having to deal with side-effects stability.
LLSTSs [5] maintain matrix shaped cell state updated with the covariance update rule [51] resorting to cross- pross-products of K and V projections. We took the official code and projected the models hidden memory to a set of embeddings with the parallel MIP chain given above. This uses XL.SMs "as is". i.e. as a standard sequence model. Potential further integration could be done by opening the black box and adapting the internal querying mechanism to the task.
MooG [58] is probably the model closest us, as it is recurrent with transformer update. However, it has key differences: there is no gating block, updates separate a prediction and correction step, inputs are dealt with patch-wise and cross-attended to 1024 memory embeddings of size 512, learned embeddings are replaced with memory initialization. We re-implemented it and adapted by giving it the same inputs (including u,) as the other models. This re-implementation has the exact architecture and hyper-parameters as described in the paper, but for comparability does not use their specific loss, which separates prediction and correction step.
astpp: // /ttp///gttbb..o//// con/MX AI/xlsta

---

Trunc.Hist. is simple baseline which forwards the last Tunne observations embeddings directly to the decoder. It is not recurrent and has a limited context length.
DEBiT [9] is a natural baseline for the navigation task. It was described in Sec. 4.2 and currently holds SoTA performance on ImageNav and Instance- -inguncee-nneeaa. We used the official code.' 3
Mem-RPE performance - Tab. 1 compares Kinaema with the baselines on the Mem-RPE task. EMA has massive advantage in terms of memory size, which configured to m, 400++8444 113.6kk, 53.6k, but this could not compensate its very simple dynamics modeled as expo- nential decay. As its memory content cannot be re-arranged by the Update function, the burden of organizing it lies with the input encoder placing inputs correctly into the memory values. Compared to the original [21], we made À a trainable vector of size 153.6k, which boosted performance (see Appendix E for the performance of [2 GRU clearly outperforms EMA due to its more expressive handling of process dynamics. Given its small memory capacity, it was crucial encourage the model to compress memory. which we achieved by making Read non-linear. This performed considerable better than the linear variants (see Appendix E). The more recent xLSTM performed less well than GRU, but was used as plug-n-play sequence model. We conjecture that performance could be optimized further by opening the black box and making the internal query mechanism connect to the goal image more directly. MooG was reported to be trained for 7-8 steps only in [58] but we trained the same lengths of 7=100 as the other models in our experiments. It performed very poorly, which we link to the patch-wise handling of visual inputs, which seems to overwhelm the recurrent transformer. Out attempts to switch Kinaema to a similar handling of memory and attention failed similarly. Kinaema has been configured such that each embedding is of the same size as the GRU memory, and the model can leverage its larger memory, making use of its multiple embeddings. It provided the best performance, in particular when needing the generalize to longer sequences. We link this to the combination of large memory size, expressive transformer update, and stable training provided by the gating block. Trunc. was trained with T Tram==100 and evaluated longer sequences by truncation. It does not generalize well, and delegates all the work to the decoder, which lacks capacity.
All models were trained on a max seq. length of T000,, we see that performances drop when they are evaluated significantly larger lengths, 80000. Fig. 4 compares Kinaema with GRU on lengths 1000,.000,, where we see a sharp initial drop increasing T from the in-domain value of T0-100, followed by more shallow further decrease. Comparisons of all models are given in Appendix E.
While the EMA and GRU models were quite stable during training, Kinaema followed bi-modal dis- tribution over seeds: seeds either gave excellent or mediocre performance. Results in Tab. I were given on RPE-test with seeds selected on RPE-val
Num Emb emb dim
Mem
Seq Im
len 200
Seq len 800
size
1m.
2mm
1m
1mo
2m
10a
9g"
90"
10"
9ge
ago
3.1k
3.1k
26
53
3
12
34
5
3.1k
15.4k
14
34
72
7
21
44
10
3.1k
30.7k
14
36
62
9
23
45
20
3.1k
61.4k
24
52
77
13
28
47
Sensitivity study: memory capacity - In Tab. 2a, we studied the impact of the number N of embeddings in m, of size 3072= ~ 3k: The model scales well until roughly 20 embeddings are reached. We conjecture that this is due to training with sequences of length T=100 and longer training could lead to bigger choice for optimal memory usage. In Tab. 2b we studied the compromise between the number of embeddings and their dimensions for given fixed memory size. Fewer and bigger embeddings seem to work better, which we tentatively explain by the factorization of the gating block of the model: each scalar gate value is determined as a function of the values of the same embedding. Increasing the embedding dim increases expressivity.
30
3.1k
92.2k
24
51
73
9
22
40
50
3.1k
153.6k
47
68
1
7
22
(a)
Num Emb
Mem
Seq
len 200
Seq
len 800
emb dim
size
Im
1m
2m
1m
1m
2m
10a
160
384
61.4k
4
19
44
2
13
33
80
768
61.4k
2
12
31
1
7
20
40
1.5k
61.4k
6
26
52
3
14
33
20
3.1k
61.4k
24
52
77
13
28
47
(b)
Table 2:
Kinaema: varying memory
structure, Mem-RPE: (a) keeping memory embedding size constant; (b) keeping total
memory size constant ( (RPE-val) ).
Ablation studies - Tab. 3 ablates the two main blocks of Kinaema's Update block: both the transformer and the gating block are necessary for good performance. In Tab. we ablate training choices. Randomizing sequence length T during training is key design choice. We found that training with constant lengths hindered generalization to longer sequences. We conjecture that leads to models confusing the notion of "state" (in control theory sense) with "layer of abstraction".
hsstgae/// thub. debii
8

---

Transf. Gating block
Seq len 200
Seq 800
Randomize Masked img
Seq len 200
Seq 800
block
1m
1m
2m
Im
Im
2m
seq len
modeling
1m
1m.
2m
1m
1m
2m
10c
age
90"
10"
90"
90
10"
90"
90"
10e
90
90"
11
33
55
3
9
19
8
30
58
15
36
X
11
37
62
4
15
31
28
51
71
4
14
34
24
52
77
13
28
47
24
52
77
13
28
47
Table 3:
Kinaema ablations of update blocks,
Table 4: Kinaema ablations of different losses,
impact on Mem-RPE (RPE-va))
impact Mem-RPE (RPE-val) )
Figure 5: RPE during navigation: exploiting the information from the priming sequence (green), Kinaema can predict (pink) the rel. pose of the goal (red), while the binocular module of [9] only starts providing reliable predictions (orange) when the agent is positioned (cyan) in view of the goal.
Le. using recurrent updates not only to push representations forward in time, but also to make changes in abstraction levels as a neural network would do between layers. The removal of masked image modeling particularly impacts OOD behavior, generalization to longer sequences.
Navigation performance Three variants of the DEBiT agent [9], with different ways to integrate memories, are fine-tuned for 100M steps on NAV-train and compared against the original agent on NAV-test in Tab. 5: (with Kinaema) concatenates the output of the Kinaema for Mem-RPE fed with primer and updated with observations to the input of the agent GRU, as described Sec. 4.2; (with GRU memory) same, but using GRU; (with ho-injection) the original agent to encode the priming sequence with no goal, teacher forced, and injects it into ho episode start with linear adaptation.
baseline using the hidden state h, of
SPL (%)
DIST SPLIT + SEEN SPLIT +
EASY
(2-5m)
MEDIUM
(5-10m)
HARD
(10-22m)
SEEN
UNSEEN
SEEN
UNSEEN
SEEN
UNSEEN
..OEEL
PRIMER
(M=566)
(#=116)
(#=520)
(N=319)
(M=64)
(I==900)
DEBIT (zeroshot)
X
41
41
40
42
47
41
DEBiT (finetune)
X
45
40
42
43
48
44
w/. inject ho
X
37
32
35
37
42
38
/
41 (+4)
40 (+8)
36 (+1)
38 (+1)
41 (-1)
40 (+2)
w/. GRU memory
X
43
42
43
50
45
45
(+2)
48 (+6)
43 (+1)
46 (+3)
50 (+0)
45 (+0)
Kinaema
X
46
(+2) (-1)
48 (+2)
49 (+4)
54 (+10)
(+1)
Table 5:
Downstream navigation performance Mem-Nav (NAV-test)) ): GRU and Kinaema
are integrated into the agent as in Section 4.2. The baseline "ho-injection" uses the non-augmented
DEBiT agent over the priming sequence (teacherffored)) and episode (policy taking decisions) with the same state h, See Fig. 10 in supp. mat. for the corresponding plot.
As shown in Tab. 5, naively adapting hidden state of the original agent (ho injection) represent primer information does not work. Using dedicated memory encoders significantly improves navi- gation performance, indicating that the agent could exploit the information localize the goal and optimize its path. Fig. 10 shows navigation performance, measured by SPL, for different episode difficulties, measured by geodesic distance between start and goal. Kinaema offers a significant ad- vantage compared to other models. While easier episodes can be solved by any model, the additional memory and RPE decoders are helpful when dealing with longer sequences.
In particular, we can observe the advantage of having obtained an initial exploration of the scene through the priming sequence (marked in the 2nd column) and stored in Kinaema memory.
Fig. visualizes Mem-RPE predictions during navigation episodes, indicating that Kinaema can successfully predict the goal position even when it is not in view.

---

Figure 6: Cross-attention memory: each image patch is colored by the memory embedding receiving the highest attention, revealing stable region region-meman memory correspondences over time.
Aug
0.46
0.46
no
130
10
120
118
10)
13
30
17
24
150
200
Figure 7: Occupancy probing of sequence of length T-200 from m., SR*: success rate 10 navigation episodes defined on each GT map, but executed on the probed map.
Visualizing cross-attention between memory and scene patches - As mentioned in Eq. (2), the RPE decoder our Kinaema model contains a cooss-atenttion layer that operates between the patch tokens of the query images and 160 embeddings (20 memory embeddings m, of size 3072 reshaped into 160 embeddings y: of size 384). In Fig. we visualize attention probabilities from this layer on four random selected Gibson episodes, each comprising 200 steps. There, each memory token is given a fixed color. Then for each query image, we associate its patch tokens to the most attended memory token, and overlay the corresponding color on the image. We see that patch tokens from spatially coherent regions tend to map to the same memory token, producing segmentation like effect. This correspondence is stable over time: as the agent moves, the set of patch tokens associated with given memory token "moves" consistently with the viewpoint.
Probing occupancy - In Fig. 7 we show predictions of probing model outputting occupancy BEVs from the frozen memory m, of Kinaema. The probe is able to infer scene structures from Mem-RPE pre-training. While map reconstruction accuracy is the same with and without reconstruction loss (average IoU on the validation set), removing the loss degrades navigation performance in this setting: the avg. SR using probed maps goes from 61.3% to 56.8%. More details are given in Appendix F.
Limitations (i) Kinaema has been trained for relative pose estimation, but does not provide an estimate on whether goal image has been seen in the past. (ii) Training has been limited to 100 steps, increase would likely improve performance. (iii) Dealing with visual inputs single embedding instead a patch-wise representation enabled its high performance, compared to models like MooG [58], but it might limit adding further improvements in the future.
6 Conclusion
We have proposed a new recurrent sequence model maintaining distributed memory updated with transformer. Compared transformers attending over the observation history, it computationally efficient with updates and reads of (1)). We trained the model for new skill "Mem-RRP Fe-m-RP"",,.."" relative pose estimation between goal image and agent memory, and integrated into a new continuous navigation downstream task, "Mem-Nav". We show that the model can spatially situate previously seen spaces and leverage this capability to navigate efficiently continuous operation. The model widcly outperforms other recurrent baselines including recent work using transformer updates.
10

---

References
[1] Z. Al-Halah, S. K. Ramakrishnan, and K. Grauman. Zero experience required: Plug & play modular transfer learning for semantic visual navigation. In /EEE Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[2] P. Anderson, A. X. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, J. Kosecka, J. Malik, R. Mottaghi, M. Savva, and A. R. Zamir. On evaluation of embodied navigation agents. arXiv:1807.06735 2018.
[3] E. Arnold, Wynn, S. Vicente, G. Garcia-Hemaando, Monszpart, Prisacariu, D. Tur- mukhambetov, and E. Brachmann. Map-free visual relocalization: Metric pose relative to single image. In European Conference on Computer Vision (ECCV), 2022.
[4] A. Barroso-Laguna, S. Munukutla, V. A. Prisacariu, and E. Brachmann. Matching 2D) images in 3D: Metric relative pose from metric correspondences. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[5] M. Beck, K. Pöppel, M. Spanring, A. Auer, O. Prudnikova, M. Kopp, G. Klambauer. J. Brand- stetter, and S. Hochreiter. xLSTM: Extended long short-term memory. Conference on Neural Information Processing Systems (NeurIPS), 2024.
[6] E. Beeching, J. Dibangoye, O. Simonin, and C. Wolf. EgoMap: Projective mapping and structured egocentric memory for deep RL. In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2020.
[7] E. Beeching, J. Dibangoye, O. Simonin, and C. Wolf. Learning to plan with uncertain topological maps. In European Conference on Computer Vision (ECCV), 2020.
[8] A. Bhowmik, S. Gumhold, C. Rother, and E. Brachmann. Reinforced feature points: Optimizing feature detection and description for high-level task. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2020.
[9] G. Bono, L. Antsfeld, B. Chidlovskii, P. Weinzaepfel, and C. Wolf. End-to-end (instance)- image goal navigation through correspondence as an emergent phenomenon. In International Conference Learning Representations (ICLR), 2024.
[10] G. Bono, L. Antsfeld, A. Sadck, G. Monaci, and C. Wolf. Learning with a Mole: Transfer- able Latent Spatial Representations for Navigation without Reconstruction. In International Conference on Representation Learning (ICLR), 2024.
[11] E. Brachmann, A. Krull, S. Nowozin, J. Shotton, F. Michel, S. Gumhold, and C. Rother. Dsac- differentiable ransac for camera localization. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2017.
[12] G. Bresson, Z. Alsayed, L. Yu, and S. Glaser. Simultancous localization and mapping: A survey of current trends in autonomous driving. Transactions of Intelligent Vehicles, 2017.
[13] W. Burgard, A. B. Cremers, D. Fox, D. Hähnel, G. Lakemeyer, D. Schulz, W. Steiner, and S. Thrun. The interactive museum tour-guide robot. In AAAI Conference on Artificial Intelli- gence (AAAI), 1998.
[14] D. S. Chaplot, D. Gandhi, S. Gupta, A. Gupta, and R. Salakhutdinov. Learning explore using active neural SLAM. In International Conference on Learning Representations (ICLR), 2020.
[15] P. Chattopadhyay, J. Hoffman, R. Mottaghi, and A. Kembhavi. RobustNav: Towards Bench- marking Robustness in Embodied Navigation. In IEEE International Conference on Computer Vision (ICCV), 2021.
[16] K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, Schwenk, and Y. Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. Conference on Empirical Methods Natural Language Processing (EMNLP), 2014.
11

---

[17] T. Darcet, M. Oquab, J. Mairal, and P. Bojanowski. Vision transformers need registers. In International Conference on Learning Representations (ICLR), 2024.
[18] A. Das, G. Gkioxari, S. Lee, D. Parikh, and D. Batra. Neural modular control for embodied question answering. In Conference on Robot Learning (CoRL), 2018.
[19] Y. Ding, C. Florensa, P. Abbeel, and M. Phielipp. Goal-conditioned imitation learning. In Conference on Neural Information Processing Systems (NeurIPS), 2019.
[20] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
[21] O. Eberhard, M. Muchlebach, and C. Vernade. Partially observable reinforcement learning with memory traces. arXiv:2503. 15200, 2025.
[22] A. Graves, G. Wayne, and I. Danihelka. Neural turing machines. arXiv: 1410.5440.5501,, 2014.
[23] A. Gu and Dao. Mamha: Linear-time sequence modeling with selective state spaces. In First Conference Language Modeling, 2024.
[24] A. Gu, K. Goel, and C. Ré. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations (ICLR), 2022.
[25] R. Hartley and A. Zisserman. Multiple view geometry in computer vision. Cambridge University Press, 2003.
[26] Hochreiter and Schmidhuber. Long short-term memory. Neural Computing. 1997.
[27] M. Jaderberg. V. Mnih, W. M. Czameekk,, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In International Conference on Learning Representations (ICLR), 2017.
[28] C. Jose, M. Cissé, and Fleuret. Kronecker recurrent units. In International Conference on Machine Learning (ICML), 2018.
[29] A. Kendall, M. Grimes, and R. Cipolla. PoseNet: A convolutional network for real-time 6-DOF camera relocalization. In IEEE International Conference on Computer Vision (ICCV), 2015.
[30] M. Khanna, R. Ramrakhya, G. Chhablani, Yenamandra, T. Gervet, M. Chang, Z. Kira, D. Chaplot, D. Batra, and R. Mottaghi. GOAT-Bench: A benchmark for multi-modal lifelong navigation. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[31] E. Kolve, R. Mottaghi, W. Han, VanderBilt, L. Weihs, A. Herrasti, M. Deitke, K. Ehsani, D. Gordon, Y. Zhu, et al. A12-THOR: An interactive 3D environment for visual Al. 1712 2017.
[32] K. Konolige. A gradient method for realtime robot control. In International Conference on Intelligent Robots and Systems (IROS), 2000.
[33] Krantz, T. Gervet, K. Yadav, A. Wang, C. Paxton, R. Mottaghi, D. Batra, J. Malik, Lee, and D. S. Chaplot. Navigating to objects specified by images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[34] M. Labbé and F. Michaud. RTAB-Map an open-source lidar and visual simultancous localization and mapping library for scale and long-term online operation. Journal of Field Roboties, 2019.
[35] V. Leroy, Y. Cabon, and J. Revaud. Grounding image matching in 3D with MASt3R. In European Conference on Computer Vision (ECCV), 2024.
[36] R. Li, S. Wang. Z. Long, and D. Gu. UnDeep Monocular visual odometry through unsupervised deep learning. In IEEE International Conference on Robotics and Automation (ICRA), 2018.
12

---

[37] R. Liu, J. Lehman, P. Molino, F. Petroski Such, E. Frank, A. Sergeev, and J. Yosinski. An intriguing failing of convolutional neural networks and the coordconv solution. Conference on Neural Information Processing Systems (NeurIPS), 2018
[38] S. Macenski, F. Martin, R. White, and G. Clavero. The Marathon 2: A navigation system. In International Conference on Intelligent Robots and Systems (IROS), 2020.
[39] E. Marder-Eppsseinn, E. Berger, T. Foote, B. Gerkey, and K. Konolige. The office marathon: Robust navigation in an indoor office environment. In IEEE International Conference on Robotics and Automation (ICRA), 2010.
[40] Marza, L. Matignon, O. Simonin, and Wolf. Teaching agents how map: Spatial reasoning for multi-object navigation. In International Conference on Intelligent Robots and Systems (IROS), 2022.
[41] F. Milletari, N. Navab, and S.-A. Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In International Conference on 3D Vision (3DV), 2016.
[42] P. Mirowski, R. Pascanu, F. Viola, H. Soyer, A. Ballard, A. Banino, M. Denil, R. Goroshin, L. Sifre, K. Kavukcuoglu, D. Kumaran, and R. Hadsell. Learning to navigate in complex environments. In International Conference on Learning Representations (ICLR), 2017.
[43] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardós. ORB-SLAM: A versatile and accurate monocular SLAM system. Transactions of Robotics, 2015.
[44] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y. Huang, S.-W. Li, I. Misra, M. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research (TMLR), 2024.
[45] A. Orvieto, S. L. Smith, A. Gu, A. Fernando, C. Gulcehre, R. Pascanu, and De. Resurrecting Recurrent Neural Networks for Long Sequences. In International Conference Machine Learning (ICML), 2023.
[46] S. K. Ramakrishnan, A. Gokaslan, E. Wijmans, O. Maksymets, A. Clegg, J. M. Turner, E. Un- dersander, W. Galuba, A. Westbury, A. X. Chang, M. Savva, Y. Zhao, and D. Batra. Habitat- Matterport 3D dataset (HM3D): 1000 large-scale 3D environments for embodied AI. In NeurlPS Datasets and Benchmarks Track, 2021.
[47] M. S. Ryoo, K. Gopalakrishnan, K. Kahatapitiya, T. Xiao, K. Rao, A. Stone, Y. Lu, J. Ibarz, and A. Arnab. Token turing machines. In IEEÉ Conference Computer Vision and Pattern Recognition (CVPR), 2023.
[48] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, Malik, D. Parikh, and D. Batra. Habitat: A platform for embodied Al research. In IEEE International Conference on Computer Vision (ICCV), 2019.
[49] L. Schonberger and J.-M. Frahm. Structure-from-motiie revisited. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[50] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. 10XXv:1707.0633 2017.
[51] J. Sejnowski. Storing covariance with nonlinearly interacting neurons. Journal of Mathemati- cal Biology, 1977.
[52] J. A. Sethian. A fast marching level set method for monotonically advancing fronts. National Academy of Sciences, 1996.
[53] D. Shah, A. Sridhar, N. Dashora, K. Stachowiez, K. Black, N. Hirose, and S. Levine. ViNT: foundation model for visual navigation. In Conference on Robot Learning (CoRL), 2023.
13

---

[54] J. T. Springenberg, A. Abdolmaleki, J. Zhang, O. Groth, M. Bloesch, T. Lampe, P. Brakel, S. Bechtle, S. Kapturowski, R. Hafner, N. Heess, and M. Riedmiller. Offline actor-critic reinforcement learning scales to large models. a:X1v:2402.05546, 2024.
[55] S. Sukhbaatar, A. Szlam, J. Weston, and R. Fergus. End-to-end memory networks. In Conference on Neural Information Processing Systems (NeurIPS), 2015.
[56] S. Thrun, W. Burgard, and D. Fox. Probabilistic robotics. MIT press Cambridge, 2005.
[57] S. Uppal, A. Agarwal, H. Xiong, K. Shaw, and D. Pathak. SPIN: Simultaneous perception, interaction and navigation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[58] S. van Steenkiste, D. Zoran, Y. Yang, Y. Rubanova, R. Kabra, C. Doersch, D. Gokay, J. Heyward, E. Pot, K. Greff, D. A. Hudson, T. A. Keck, Carreira, A. Dosovitskiy, M. S. M. Sajjadi, and T. Kipf. Moving off-the-grid. Scene-grounded video representations. In Conference on Neural Information Processing Systems (NeurIPS), 2024.
[59] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Conference Neural Information Processing Systems (NeurIPS), 2017.
[60] S. Wang, V. Leroy, Y. Cabon, B. Chidlovskii, and J. Revaud. DUS(3R: Geometric 3D vision made easy. In IEÉE Conference on Computer Vision and Puttern Recognition (CVPR), 2024.
[61] S. Wani, S. Patel, U. Jain, A. X. Chang, and M. Savva. Multion: Benchmarking semantic map memory using multi-object navigation. In Conference on Neural Information Processing Systems (NeurlPS), 2020.
[62] P. Weinzaepfel, V. Leroy, T. Lucas, R. Brégier, Y. Cabon, V. Arora, L. Antsfeld, B. Chidlovskii, G. Csurka, and J. Revaud. CroCo: Self-supervised pretraining for 3D vision tasks by cross-view completion. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
[63] P. Weinzaepfel, T. Lucas, V. Leroy, Y. Cabon, V. Arora, R. Brégier, G. Csurka, L. Antsfeld, B. Chidlovskii, and J. Revaud. CroCo v2: Improved cross-view completion pre-training for stereo matching and optical flow. In IEEE International Conference on Computer Vision (ICCV), 2023.
[64] E. Wijmans, M. Savva, Essa, S. Lee, A. S. Morcos, and D. Batra. Emergence of Maps in the Memories of Blind Navigation Agents. In International Conference on Learning Representa- tions (ICLR), 2023.
[65] Q. Wu, J. Wang, J. Liang, X. Gong, and D. Manocha. Image-goal navigation in complex environments via modular learning. IEEE Robotics and Automation Letters (RAL., 2022.
[66] F. Xia, A. R. Zamir, Z. He, A. Sax, J. Malik, and S. Savarese. Gibson env: Real- world perception for embodied agents. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[67] K. Yadav, A. Majumdar, R. Ramrakhya, N. Yokoyama, A. Baevski, Z. Kira, O. Maksymets, and D. Batra. OVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNav. arXiv:2303.07798, 2023.
[68] K.-H. Zeng. Z. Zhang, K. Ehsani, R. Hendrix, Salvador, A. Herrasti, R. Girshick, A. Kemb- havi, and L. Weihs. Poliformer: Scaling on- on-policy RL with transformers results in masterful navigators. In Conference on Robot Learning (CoRL), 2024.
[69] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In IEEE International Conference on Robotics and Automation (ICRA), 2017.
[70] Eukasz Kaiser and I. Sutskever. Neural gpus learn algorithms. In International Conference on Learning Representations (ICLR), 2016.
14

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: the claims are described in the introduction
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
There is limitations paragraph on the last page.
Guidelines:
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All architecture and training details are described in the supplementary material. However, the training meta-data we additionally collected standard datasets will be made public only after acceptance of the paper.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will release code and data, but after acceptance.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: All details are described in the experimental section.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Answer: [Yes]
Justification: This is described in the supplementary material.
9. Code of ethics
15

---

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neur/: httppp///aaarrp.ccccp..c//////// http://pw......eiicccccieeeeeee.
Answer: [Yes]
Justification: comme why would we not respect this?
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
This is described in the supplementary material.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The usage of HM3D and Glbson datasets and the Habitat simulator was cited and licenses respected.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Additional training meta-data will be released after acceptance.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
16

---

Supplementary Material for:
Kinaema: A recurrent sequence model for memory and pose in motion
Contents
Introduction
2 Related work
3 The Mem-Nav and Mem-RPE tasks
Situation awareness with latent memory
Experimental results
6 Conclusion
10
A Data splits and generation
17
More details on training losses
18
Kinaema: architecture details
20
D Baselines: architecture details
22
E Additional experiments
23
Occupancy probing experiments
25
Additional complementary visualizations of memory attention
27
H Computing resources and runtime
28
Broader impact
28
Data splits and generation
A.1 RPE data
Pre-training data for relative pose estimation (RPE) is generated on scenes from the train split of the HM3D dataset. Using a navigating agent configuration close the default one of habitat-sim, we collect a fixed number of frames per scene by chaining random goal pursuits, We split data per connected components of the navigation mesh, such that any sample from any other. We take care of balancing the number of samples with respect to the component area relative the total navigable area of the scene. Compared we use smaller image resolution (112 x 112) and an action space enabling finer motion, which is deliberately different from the action space of the agent used for the validation and testing data, for both Mem-RPE and Mem-Nav. This allows to test our model in
subset can be reached
the default agent configuration.
situations:
Configuration 1: RPE-training
{ {10em forward, 5 turns). This choice corresponds the raw data generated. Additionally, during training frames are not sampled consecutive, but randomly steps between 1 and 8.
Configuration 2: APE-validation-tess Mem-Nav training+test
A = {25cm forward, 10" turns).
This generates significant -iitrf--iitribution behavior between Mem-RPE training and Mem-RPF testing. Note that forward testing step of 25em cannot be expressed as multiple steps of forward training steps of 10cm.
17

---

Additionally, one alternative frame is generated at each step by randomizing most of the sensor specification. We provide details of the parameter distributions in Tab. Algorithm provides more details on the generation procedure. We use a custom storage format based on compressed NumPy archive supporting both dense and sparse values (e.g- for the pursued goal position), and large folders of images compressed in JPEG. Validation and test data are generated on different scenes with the same procedure. We use the train split of the Gibson dataset for validation, from which we select hyper-parameters and models without interfering with the tests used to report metrics. The final metrics are reported on the HM3D val split (the test split is not publicly available). However, as detailed in Tab. 6, agent configuration uses the standard action space instead (25em forward, 10 tums), to match the one of the downstream navigation task.
Input: scene dataset S, #frames per scene Output: Offline dataset trajectories with main and alt cam foreach scene in S do
split scene into connected navigable islands;
select subset I of islands get 80% of scene coverage; foreach island in do
#frames +- area(island) total covered area #frames per scene;
agent state + random navigable position, random orientation;
repeat
select random navigable goal with geodesic dist. constraints;
while goal not reached do
agent state + execute greedy action towards goal; alternative camera < random configuration;
render main and alternative frames;
store(frames, agent position and orientation, camera);
end
until #frames have been generated;
end end
Algorithm RPE data generation procedure.
A.2 MemNav data
For the downstream navigation task, we only use the scenes in the publicly available train and val splits of the HM3D dataset. We use the default agent configuration of habitat-sim except for the frame resolution which we reduce to 112 x 112 to avoid unnecessary rendering and inter-process transfer for frames ending up resized max- max--pooled in the agent model. We first sample few exploration cycles per connected component of the navigable mesh in each scene, to constitute limited offline information used as memory primers for the agent, and for which can pre-compute internal representation. Start poses are then sampled from these cycles with many different goal positions to make navigation episodes. Algorithm 2 provides details of the generation procedure, and Algorithm 7 lists important generation parameters.
More details on training losses
As mentioned in Sec. 4.1 of the main paper, we train all models by jointly optimizing the relative pose estimation (RPE) and masked image modeling (MIM) losses sequences of length T = 100.
Mean- -quare error is used for the MIM loss:
N Σ pET:
CMIM NXP
P:
P
(8)
where p; and P are predicted and GT pixel values for the patch p of the training image i, and N and P are the number of training images and patches in each image, respectively.
18

---

Common
cameras
main
resolution
112 x
field of view
90"
position
125cm above ground straight
orientation
alt
resolution
112 x I12
field of view
4/600 120
aspect ratio
U41,4/3. 16/9, 99, 16/10)
(portrait landscape)
position
125cm above ground
+ =5000mmm in all dir.
orientation
pan Φ ~ 4|
tilt 2 4:30%
roll 0 ~ U +.5%
RPE-train
actions
forward turn L/R
10cm
5
scenes
set
HM3D/train
#
800
frames
#
per scene
100k
total
8OM
sampling rand. intervals
skip
W O, 8 frames
RPE-val
actions
forward
25cm
turn L/R
10
scenes
set
gibson/tras ttrain
#
72
frames #
per scene
1k
total
72k
sampling
contiguous seq.
RPE-test
actions
forward
25cm
turn L/R
10
scenes
set
HM3D/val
#
100
frames #
per scene
1k
total
100k
sampling contiguous seq.
Table 6: Parameters for RPE data generation.
The RPE loss is given in Sec. 4.1 of the main paper, and repeated here for completeness:
t + R; Rt]
LAFE
(9)
N
where (t,, R2) and (t² R²) are predicted and GT pose for the training image i, respectively.
See Appendices C.2 and C.3 on how we make RPE and MIM predictions, respectively.
Generalization to longer sequences We apply types of "drop-out" improve generalization between train and test, and generalization to longer sequences:
1. random sequence sub-sampling, where sequence length is randomly determined each training iteration to lie between a minimum value (Taab) and T.
2. already mentioned in section A.I on data generation, do not sample consecutive steps from the generated training data, but rather use random interval sampling of size 8, where the interval between two consecutive steps can be up 8. This encourages robustness to changes in sampling rate between train and test.
Training hyper-parameters commonly used for all models shared in Tab. 8.
19

---

Input:
scene dataset S, #cycles per island, #frames per cycle, #episodes per scene Output: "navigate from memory" dataset of episodes with primers
foreach scene in S do
split scene into connected navigable islands; select subset Z of islands to get 80% of scene coverage; foreach island I do
for #cycles per island do
while approximate cycle length < #frames per cycle do add random waypoint to cycle; end
solve TSP waypoints;
agent state + arbitrary waypoint in cycle;
foreach waypoint in cycle do
while waypoint not reached do
agent state execute greedy action towards waypoint;
render main camera frame:
ray-trace fog of on occupancy grid;
store(frame, fog, agent position and orientation);
end end
#episodes + area(island) total covered area #cycles x #frames per scene; for #episodes do
start (pos, ornt) random pose along cycle;
goal (pos) +- random navigable point;
memory primer + sub- sample cycle (frames,poses) to fixed #frames per eycle; /* such as memories end at start pose
geodesic distance shortest path length from start to goal;
seen indicator +- approx. goal visibility from cycle (fog or neighbor alignment): store(episode);
end
end
end
end
Algorithm 2: MemNav data generation procedure.
C Kinaema: architecture details
C.1 Main memory model
Inputs: images are of resolution 112x112,
Visual encoders (Encus and Enceoa)): they are implemented ViT-Small with patch size of 14 and 0 registers [17], and they are initialized from the pretrained weights of DINO-v2 [44].
Odometry encoders (ū:): they are implemented as linear layers with an output size of 64.
Update/correction block: equation (3) is Linear function.
Update/transformer block: the transformer block has layers, 24 heads, and an MLP-factor This was optimized over the validation set RPE-val, exploring 0,1,2,3,4 layers and various numbers of heads.
Update/gating block: the gating block is a GRU in the standard PyTorch implementation with layers. We explored 0,1,2,3 and 4 layers, optimized over RPE-val
C.2 RPE modules
All models have Transformer-based decoder to provide estimates for relative pose, but there are slight differences, optimized this for the baselines. In order words,
20

---

common
cameras
main
resolution
112 112
field of view
90
position
125cm above ground straight
orientation
actions
forward
25cm
turn L/R
10"
collision
radius
10em
height
150cm
sliding disabled
cycles
island
2
len
200 frames (aka. steps)
NAV-train scenes
HM3D/train
800
episodes
per scene
10k
total
8M
NAV-test
scenes
set
hm3d/val
100
episodes
scene
20
total
2k
Table 7: Parameters for MemNav data generation.
Hyper-parametar
Value
Length (T): 100
Sequences
Random sequence sub-sampling (Tsub): 50
Random interval dropping: 8
Batch size
32
Training steps
250K
Type: Adam
Optimizer
Weight decay: 5e - 2 (31, (0.9, 0.99)
Min: le 8
Max: 15x10=4xbatch-siin /isiiieee/662 256
Learning rate
Warm-up: Linear, for first 20% of iterations
Schedule: Cosine
Gradient clipping
Over all parameters, max magnitude:1
Data type
AMP with float16
Table Hyper-parameters used for training models.
the choices we made for Kinaema worked best for Kinaema.
the choices we made for the baselines, worked best for the baselines.
The Kinaema decoder employs a cross-attention (CA) layer with no residual connection whose keys and values are provided by the Read function and queries the patch tokens from the goal image encoder Encegaa- The motivation behind this design to make the decoder rely solely on the memory outputs and prevent information leakage from Encgoal into the decoder. learnable token (i.e. CLS token) is attached to the output of the CA layer and given as input to a sequence of 4 standard self-attention (SA) blocks [59]. Finally, the CLS token detached and relative pose of the goal image is estimated by MLP with hidden layer.
Other competitive variants, i.e. GRU and xLSTM, perform better when multiple CA layers interleaved with the MLP and SA layers. More concretely. for those models, we attach the CLS token E outputs, and apply 3 chains of CA-MLP-SA-MML layers, where the first CA layer does not have a residual connection. Relative pose estimations are made similarly to Kinaema.
All decoders have a comparable number of parameters.
21

---

C.3 Masked image modeling modules
These modules provide the prediction for the masked image modeling loss, and they follow the design of the RPE modules from the previous section. For Kinaema, a separate sequence of 4 standard SA blocks is applied after the CA layer between EnCgoot and Read outputs. Then RGB values for each pixel of patch are predicted by linear layer.
On the other hand, for the GRU and xLSTTM models, learned MASK token is inserted the output of Encyoal to the locations of the masked tokens, and they are given as input separate stack of CA-MLP-SA-MA layers, followed by linear layer for predicting RGB values.
D Baselines: architecture details
D.1 GRU
As mentioned in Sec. 5 of the main paper, this variant differs from Kinaema in its Update and Read functions. We implemented the Update function for the GRU variant such that encoded visual and odometry inputs (X, and ú, respectively) are concatenated in the channel dimension and then given to GRU with 4 hidden layers and 3072 units (following the standard implementation in PyTorch). Then the output from the final step of the GRU is given as input to the Read function, which applies a list of 50 MLPs in parallel to produce 50 different memory tokens. Finally, the memory tokens are given to the Dee functions to estimate the relative pose and model the masked image. The hyper-parameten of this variant, i.e. the number of hidden layers and units in GRU and the number of parallel MI.Ps are chosen such that the total number of trainable parameters of this variant is comparable to that of Kinaema. We saw in general that the more the parameters the better the performance on RRE-vvl
D.2 EMA
The EMA models are simple and do not leave much space for configuration choices. The memory vector m, is of size 153.6k, chosen to be multiple of 384. This choice was optimized over the set {19.2k, 38.4k, 76.8k, 153.6k on RPE-val The visual and odometry encoders Encws and Encodo. respectively, are the same as for Kinaema and the other basclines. However, the concatenated encoded inputs [Rc, ür] are additionally projected to dimension 153.6k 6k with a single linear layer into, and added to m, by simple summing, as was also described in the main paper:
Update A m = Am;- Ux+
(10)
The matrix of this layer alone has (4096 64) x 153.6k =~ 640M parameters. The EMA model trades the simplicity of the update function for the complexity in the input projectors.
D.3 XLSTM
This variant closely follows the GRU one described above, except that XLSTM[1:0] (comprised only of mLSTM blocks [5]) is used in the Read function. We took the official code for the implementation mLSTM. To match the trainable parameters of Kinaema, we use 6 mLSTM blocks in total, which are applied sequentially. In each block, the upsampling factor of 1.47 is used to project the concatenated visual and odometry inputs into 30702 dimensions, which produces a cell state of size 768 x 768x4 4= 2, 359, 296. Similarly to the GRU variant, we take the final hidden state (Eq. 21 in [5] and provide it to the Read function.
D.4 MooG
We re- implemented MooG with the architecture described in their original paper, but adapted it to our task by giving it the same inputs (including u,) as the other models. Input images and encoders Enccis and Encado are the same as for Kinaema and the other baselines. The rest of the architecture has been kept as in the original paper (ref. [52] in the main paper). Memory m, is composed of 1024 embeddings of size 512. As in the original paper, inputs are dealt with wisc,- in our case the DINO-v2 patch embeddings. We adapt this be concatenating it with the encoded odometry and then
httpo://githhbs con/MX At/xlsta
22

---

linearly projecting to the memory embedding dimension of 512. Updates separate prediction and correction step.
Update/prediction step: as the original paper, this is implemented as a transformer with 3 layers and 4 heads and self-attention layers.
Update/correction step: as in the original paper, this is implemented as a transformer with 2 layers and 8 heads and cross-attention and self-attention layers.
D.5 Truncated History
We implemented a simple Truncated History (Trune. Hist.) baseline: it directly forwards the concatenated patches embeddings and odometry embeddings to the Dec function.
x
m, (opdate(m--11) Xx, ū.) -
m
ü,
(11)
Yt Read(m) (m,)
m
First value is repeated as much as necessary to pad the resulting m, matrix to fixed Tinme size, eg:
X
X
X1
X2
X3
m2
üg
Üg
ú1
0g
A maximum of Tiuee 100 embeddings is kept (older are discarded), eg:
X141
X115
X212
X213
M243
u144
ü145
u212
Ú243
D.6 Integration into the navigation agent
As shown on Fig. 3, we integrate our Kinaema"menta memory encoder" into the standard agent architecture of [9], which borrows from existing agent baselines models: separate encoders per observation modality, whose outputs are concatenated to be fed to GRU which maintains representation (ic. compressed representation
internal state observations history) from step to step while the agent navigating. This internal state is then given to a linear actor (ic. policy) head to produce a distribution over actions. The originality in [9] a dedicated binocular encoder pre-trained to compare pairs of images, and estimating relative pose and visibility information between them. This exploited to predict the relative pose between goal image and an onboard image observation. We extend this idea by integrating the comparison of the goal to dedicated compressed memory representation, instead of only comparing to last observation and leaving all the recurrent work to the agent GRU and policy.
E
Additional experiments
E.1
EMA w. constant À vs. trainable X
In Table 9 we have explored different variants of the EMA baseline with different values of the A decay factor: X 0.9, À 0.95, and trainable À vector (which is the variant shown in the experiments the main paper). Making À trainable has a positive impact, which can be explained quite easily: conjecture that it allows the model to choose whether certain memory features "fast" or "slow".
E.2 Alternative read outs for the GRU Baseline
In the main paper we had promised further experiments with alternative read outs for the GRU baseline. Unfortunately we seem to have lost the corresponding model checkpoints, we apologize for that. We are re-running these experiments and will have them ready for the rebuttal period.
23

---

Seq
200
Seq
Im
1m
2m
1m
Im
2m
10"
30
90
10c
80
90"
=
0.9
14
29
19
À 0.95
2
9
22
6
15
trainable vector of size 153.6k
6
18
34
11
24
Table 9: EMA: comparisons of À configurations, impact on Mem-RPE (RPE-val)
Mem -REE Acc. (%)
lers trainene 100
Kinaema/lm10"
EMA1m10p LRU/1m10"
GRU/1m10"
MooG/1m10"
Evaluation sequence length
Mem- RPE Acc. (%)
Todamain Jun. Srai Sonmmaallll 100
Kinaema/Im90"
EMA/1m90'
GRU/1m90"
LRU/1m90
MooG/Im90
25
5080
Evaluation sequence length
Mem- RPE Acc. (%)
In-dkmain un ze.zeennn innoaaa - 100
Kinaema/2m90*
EMA/2m90'
GRU/2m900
LRU/2m900
MooG/2m90
Evaluation sequence length
Figure 8: Comparing all models (trained on 7=100) their generalization to longer sequences; Mem-RPE, RPE-test. (a) pose threshold 1m10%; (b) pose threshold Im90%; (c) pose threshold 2m90..
E.3 Generalization to long sequences
As an extension of Figure 3 of the main paper, which compared the Kinaema and GRU models on longer sequences, Figure of this supplementary material we compare Kinaema on RPE-test with all main baselines: GRU, XLSTM, EMA, and MooG. We can see that Kinaema clearly outperforms the baselines and provides the best Mem-RPE predictions over an extremely large range of out-of- distribution sequences lengths, from the in-domain length of 100 frames up 1000 frames. The advantage of Kinaema is also maintained over all metrics.
E.4 Training on different sequence lengths
trained two models on different sequence lengths and compare their generalization Fig. The standard model trained on 7-100 is compared to a variant which has been finetuned sequences of length 7=200 (but limiting backpropagation to the last 7=100 frames of the sequence). both cases, provide the maximun length, for each batch randomly sample sequence lengths. We

---

100
Kinsema/7-1001110 WosemaT=1000--2000/0
Kinsemaa7--0001m30 2T-0001000
Kineema/7-10022990 10emmaf-100-1-20022m0
Mem- RPE Acc.
Evaluation sequence length
the standard model trained T=100 compared to a variant which has been finetuned on sequences of length 70200.. We can see an improvement in precision (the hardest metric) but not necessarily in the overall recall (less hard metrics).
Figure 9: Impact of different training sequence lengths:
can see an improvement in precision (the hardest metric) but not necessarily in the overall recall (less hard metries).
E,5 Visualization of downstream Mem-Nav performances
Tab. 5 provides plot of the results shown in Tab. 5 in the main paper.
Z
DEBiT (zeroshot)
w/. he inject
w/. Kinaema
DEBiT (finetune)
w/. GRU memory
easy
medium
hard
60
seen
40
SPL (%)
20
unseen
60
40
20
1.5
8.3 start - goal distance (m)
15.1
21.8
Figure 10: Navigation efficiency (SPL) for different episode difficulties (start->goal distance), w/ and w/o (E) priming sequence. Numbers correspond to Tab. 5 in main paper.
Occupancy probing experiments
We generate a dataset Mi)=a... of D=185k trajectories of length 100, where m are memory states and M; are corresponding ego-centric 2D metric occupancy maps of size 10mx 10m calculated in simulation. A probing network Φ, inspired by [64, 10], is trained on Gibson training scenes to predict M; = ((m)) minimizing the Dice loss [41] between M; and M.
The network é processes each flattened m; with an MLP with 2 hidden layers of size 512 to produce an output vector of dimension 2304. This vector is reshaped into 3D tensor of size [16, 12, 12] and processed by Coordinate Convolution (CoordConv) layer [37], followed by four CoordConv- CoordUpCon (Coordinate Up-Convolution) blocks. Each such block is composed of:
A 2D Dropout layer with dropout probability 0.05;
25

---

Ground-truth Reconstruction, SR*= 60%
Figure 11: Computation 2D Navigation Success Rate, SR* goal positions In. green, are randomly sampled on the ground-truth occupancy map the left. They are all reachable from the agent position at the center (yellow square). SR is computed as the percentage of goal positions gn. that can be reached the reconstructed map the right, again in green. Unreachable locations highlighted in red.
Kinaema
Kinaema
loss
Kinaema
Kinaema loss
00
58
IOU
SR
lmm
Evaluation sequence length
Evaluation sequence length
Figure 12: Comparing occupancy probing performance of Kinaema trained with and without reconstruction loss with T=100, on sequences of length 70=00 200, 300. (Left) Average Intersec- tion over Union, loU; (Right) Average Navigation Success Rate, SR*
A CoordUpCony layer with kernel size 3, stride 2, padding = 0, that maintains the channel dimension and roughly doubles the spatial dimensions of the feature map;
A CoordConv layer with kernel size = 3, stride 1, padding 0, that halves the channel size while roughly keeping the other dimensions intact;
ReLU activation except for the last block where it is removed.
The result of this process is passed to Conv2D layer to create the output of size [1, 200, 200] representing the unnormalized logits of each map pixel being navigable.
We tested probing performance on 1000 trajectories of length 100, 200 and 300 collected on unseen Gibson validation scenes, evaluating map reconstruction quality with Intersection over Union (IoU) and assessing how useful the probed map is for navigation with 21) Navigation Success Rate (SR") [10]. Fig. 11 shows example of SR" calculation: given pair of ground-truth and predicted maps (M,, M) sample 10 reachable goal locations the ground-trutt -tuut map In. and compute SR* as the percentage of goals 9n that can reached on the reconstructed map M,. See
Fig. 12 shows average probing performance of Kinaema trained with and without reconstruction loss with 100 sequences of length 100, 200 and 300. On the left we display IoU and on the right SR Interestingly, when tested in-domain (T=100), the model without reconstruction loss achieves better performance, while for longer sequences the trend reverses, especially concerning SR This confirms previous observations that the use of reconstruction loss mainly helps the model generalize longer sequences.
Finally, in Fig. 13 we display example predictions occupancy maps obtained by probing the two Kinaema models trained with and without reconstruction loss for sequences of length 100 to 300.
26

---

4
2
Figure 13: Probing of sequences of length 100 (top). 702 (middle). 30330 (bottom). On each figure we display, from top to bottom, ground-truth occupancy map, occupancy reconstructed by probing Kinaema, and occupancy reconstructed by Kinaema without reconstruction loss. Sequences are down-sampled by a factor 10.
G Additional complementary visualizations of memory attention
G.1 Attention over top-down maps
In Fig. 14 we show complementary visualizations of memory attention. While in Fig. 6 of the main paper we link patches of observed images queried memory, in Fig. 14 take single memory m, at time f, read out into embeddings and check how these embeddings are attended given different goal poses. Kinaema primed with priming sequence shown in green. Then, goal poses are sampled from positions on regular grid and four different canonical orientations. Respective icons are color coded, where color represents the choice of read out embedding in {y+) which received highest attention. Figures Fig. 14 (Left) and Fig. 14 (Right) differ in the order in which the priming frames have been visited.
We can see that there are regions of homogeneous color (which identifies the read out embedding) in the map, but also that goal camera orientation potentially plays an important role. This makes sense, as rotation of the goal camera by 90", 180 or 270" will lead to completely different observations. On the other hand, we see quite big differences between the left and the right Figure, which differ by the order in which the priming frames have been seen. This gives evidence that memory embeddings are not plainly selected based on appearance features, a choice which we would have judged as being sub-optimal, as it would have ignored the spatial layout of visited scene. As summary, considering this post-hoc analysis, we judge as positively interesting, that (1) assignments between memory embeddings and goal pose depend the priming sequence, and (2) there are spatial regularities in highly attended memory embeddings. Combined, they provide evidence for a spatially coherent strategy which does not collapse to appearance alonc.
G.2 Global attention patterns
To assess the contribution of memory tokens across diverse goal images, we compute the cross- attention between the patch tokens output by Enc and memory tokens for all 1000 images in
27

---

Figure 14: Visualization of attention to memory queried from different goal poses: Kinaema is primed with the priming sequence shown in green. Then, goal poses are sampled from positions regular and 4 canonical orientations, color coded as the choice of read out embedding in {y:) which received highest attention. Left and right differ in the order in which the priming frames have been visited.
the 200-frame sequences of RPE-val Then we average these probabilities over images and patch tokens, and visualize them in Fig. 15, both per attention head and aggregated across heads.
can be seen that the heads exhibit distinct and complementary patterns. Heads and 6 concentrate sharply on few memory tokens, whereas other heads distribute their attention more evenly. Collec- tively, the heads provide full coverage: the mean attention over heads assigns non-zero weight to every memory token, although certain tokens are consistently emphasized more than others.
H Computing resources and runtime
Model training is performed on a variety of NVIDIA V100, A100 or H100 type of GPUs (up to GPUs). To maintain fixed batch size across all models, implemented the gradient accumulation mechanism. This way, the batch size per GPU can be as low as 1.
Mem-RPE Training Kinaema on H100 GPU takes approximately days.
Mem-Nav RL fine-tuning of the agent w/ Kinaema H100 GPU takes approximately 30 days to reach 300M steps.
Inference of the full agent is lightning fast: processing a full of 2k episodes of NAV-fest takes approximately an hour and half on V100 GPU, which corresponds to roughly 140 frames per second.
I Broader impact
Our paper investigates the scientific question, whether relative pose can be estimated between an image and latent memory, and whether recurrent Transformers can perform this task without requiring to store historical information. Beyond the exciting scientific reasons for exploring work spatial Al and embodied AI, we welcome the potentially high interest for society getting tedious tasks automated. Our contributions advance robotics, and as such, positive and negative impacts are common with other scientific advancements this area. We can provide a few examples, but the impact of robotics is well known: autonomous agents navigating in indoor spaces could be helpful in health care, care for the elderly, but could also be useful for tasks as mundane as guides in offices, museums etc. An intelligent agent could act in the form of textual output, but more importantly, by guiding people to places they have difficulty finding by fetching items. We acknowledge that any progress in robotics can inherently be abused and produce potential harm, for instance, in surveillance or military applications.
"embodied ChatGPT", providing help not only
28

---

Head
10
2
Hcad 2
10
8
6
4
2
0
Head
8
Hcad
8
4
2
Hcad
8
Head 6
10
8
Mean all heads
10
Memory
Attention
8
20
40
80
100
120
140
160
Memory Token Index
Figure 15: Cross-attention scores between goal images and memory tokens. Y-axis represents the attention probability multiplied by 100 for the sake of visualization. Scores for each head sum to 100.
29


Reviews:
[{"review_id": "10VPSwhOgQ", "reviewer": "Reviewer_rguE", "summary": "The paper introduces a new model architecture for storing and retrieving long-term memory specifically for the task where a robot needs to estimate the relative pose between a target image and the current pose. While the ideas presented make intuitive sense, I find the writing hard to follow in some cases.", "questions": "1. How does the Kinaema model fit into the steps describe in the background in the beginning of section 4", "limitations": "please see my comments above", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Quality\n\nI like the discussion around the desired behavior of the model and the consideration that go into designing the model architecture at the beginning of section 4. The discussion makes sense and provide a holistic explanation for more technical ideas that follow.\n\nClarity\n\nOn line 144-145, the paper writes this `allowing the model to take decisions (to “gate”) on the speed of updates for each memory item`. What does it mean to take decision on the speed of updates?\n\nOn line 172-173, \"the paper writes the following self-attention transformer contextualizes the embeddings with each other\". Self-attention is an operation. What does it mean to have a self-attention transformer? And what does contextualizes the embeddings with each other mean?\n\nIt is also unclear the relationship between the model described in section 4.1 and background information provided in section 4.\n\nSignificance\n\nI agree with the argument in the paper that the more traditional navigation setting doesn't lend itself well to study longer memory context problem. However, I would have like more description on the task in the background session to be able to understand the experiment better.\n\nOriginality\n\nThe experimental setting that the paper studies and the ideas presented are novel to the best of my knowledge. I don't work in navigation however and I'm not very familiar with the literature.", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer-rguE for their review, for highlighting that `the ideas presented make intuitive sense`, and for liking `the discussion around the desired behavior of the model and the consideration that go into designing the model architecture`. We provide an answer to the main remaining questions below.\n\n\n> It is also unclear the relationship between the model described in section 4.1 and background information provided in section 4.\n> How does the Kinaema model fit into the steps described in the background in the beginning of section 4.\n\n\nWe directly used goals G1, G2, G3 to design the Kinaema model, these goals were indeed the background for our design choices.\n\n\nMost importantly, we want our model to be recurrent (G1), so that one time step is of complexity $O(1)$ (taking into account the number input time steps in the big-O notation) and thus that it scales to long sequences with low complexity, eg. for robotics applications. This is opposed to transformer based solutions with attention over input tokens, as the standard models used for LLMs or LVAs. These existing solutions require storage of the history observations and attend over them, and are of complexity $O(N^2)$, or, if causal and using KV-caching, $O(N)$.\n\n\nExisting *recurrent* models (RNN/GRU/LSTM/…) do indeed already feature this target complexity of $O(1)$, but they lack network capacity, as their number of parameters is quadratic with respect to memory size. Our design goal (G2) is to scale memory size, which is why we extend memory embeddings from a single embedding to multiple embeddings. Together with the model choice, this addresses (G2), as the update function of our recurrent model is a transformer, which scales over memory size (transformer complexity independent of the number of embeddings, which is in our case the number of *memory* embeddings).\n\n\nLastly, some recurrent models (like classical RNNs) have problems with stability, as the backpropagation chain is complex. This has been addressed by LSTMs and GRUs through gating functions, which better control the speed of individual updates (for more details on this see the answer on gating further below). Stability is our design goal G3, for that reason we take the gating functions from GRUs and adapt them to our transformer-based recurrent model. This required adapting them to the presence of multiple memory embeddings, compared to the model where they originate from, GRUs.\n\n\n**As a summary**, to achieve our 3 goals (G1, G2, G3), we design a transformer-based representation that is *updated recurrently* by a gating mechanism (inspired by classic GRU architectures), such that it can be updated with $O(1)$ and thus scaled to long sequences (G1). Almost all components of our model are based on self- and cross-attention to maintain the capacity and expressive power of modern transformer-based architectures (G2). Finally, we adopt an approach similar to that used in GRUs and LSTMs to address training instability issues of early RNNs, that is, we use a gating mechanism to control the memory update and stabilize training (G3). \n\n\n\n> On line 144-145, the paper writes this allowing the model to take decisions (to “gate”) on the speed of updates for each memory item. What does it mean to take decision on the speed of updates?\n\n\nGating functions are classically used in sequence models such as LSTM or GRU. Each trainable gating function predicts a single scalar between 0 and 1 and does this *for each embedding dimension*. This gating value is then multiplied with the corresponding feature value. Different gates have different purposes: some gates regulate whether a memory value is updated by the model or whether its old value is kept, or rather a continuous value between these two choices.\n\nAs written in the paper, we use the GRU gating functions, which we provide below for completeness (see also Chris Olah’s blog on “Understanding LSTM Networks” for an excellent breakdown including visualizations):\n\n\n$ z_t =\\sigma\\left(W_z \\cdot\\left[h_{t-1}, x_t\\right]\\right) $\n\n\n$ r_t =\\sigma\\left(W_r \\cdot\\left[h_{t-1}, x_t\\right]\\right) $\n\n\n$  \\tilde{h}\\_t = \\tanh (W \\cdot  [r_t * h_{t-1},  x_t ] ) $ \n\n\n$ h_t =\\left(1-z_t\\right) * h_{t-1}+z_t * \\tilde{h}\\_t $\n\n\nFor GRUs, the reset gate $r_t$ controls for each dimension of the GRU memory $h_t$ how much of its value is forgotten when we calculate the updated values. The update gate $z_t$ controls for each memory dimension how much we update it (as opposed to keeping the old value).\n\n\nFor our proposed Kinaema model (L177-187), we use the same gates as above and the same update functions. However, these updates are done for each of the Kinaema embeddings separately. The GRU equations above apply, with the input $x_t$ corresponding to the “update candidates” $\\mathbf{\\tilde{m}}\\_{t,n}$ produced by the transformer block in equation (4). The matrices $W\\_z, W\\_r, W$ above are thus shared over Kinaema memory embeddings: the same gating function is re-used for every Kinaema memory embedding.\n\n\n> On line 172-173, \"the paper writes the following self-attention transformer contextualizes the embeddings with each other\". Self-attention is an operation. What does it mean to have a self-attention transformer? And what does contextualizes the embeddings with each other mean?\n\n\nBy self-attention transformer, we meant a standard transformer block, each composed of multi-head self-attention and an MLP; both including with layer norms, and residual connections; however, we wanted to emphasize the type of attention used, ie. “self-attention” and not “cross-attention”, as this would have been an alternative which is used for similar purposes for instance in MooG [52] (or in our memory decoder in Equation (2)). We will make this clearer. \n\n\nTo “contextualize an embedding” means to update the embedding features with information from its surrounding context, ie. features from the other embeddings of the sequence or set, so that the resulting representation is aware of the broader input or sequence it belongs to. With this we meant the standard operation done by transformers and we borrowed the term from the NLP literature. We will remove it or explain the concept differently."}, {"review_id": "o3CROUFbjk", "reviewer": "Reviewer_3MFD", "summary": "This paper introduces Kinaema, a novel high-capacity recurrent sequence model designed for embodied agents that operate in long-horizon, continuous visual navigation tasks. The key innovation lies in maintaining a distributed memory composed of multiple embeddings, updated recurrently via a Transformer block and stabilized using gating mechanisms inspired by GRUs. The authors evaluate Kinaema on two newly proposed tasks: Mem-RPE (estimating relative pose between a query image and the agent’s latent memory) and Mem-Nav (navigating to a previously seen target location, using only prior observations). Empirical results on simulated 3D environments (HM3D, Gibson) demonstrate that Kinaema outperforms multiple recurrent baselines (GRU, xLSTM, MooG) in both memorization and navigation efficiency, especially for long sequences. Ablations support the importance of transformer updates, gating, and distributed memory structures.", "questions": "1. Test Time Training introduces fast weights and could also memorized long sequence with O(1) computational complexity. The authour could add some disscussion on using recurrent transformer compared to TTT.\n\n2. Besides using architecture with O(1) computational complexity, the author could also compare to methods merging KV cache as memory in transformer.", "limitations": "The potential for **lifelong memory** or **multi-episode navigation** is not addressed. The proposed navigation agent is a modular pipeline and **the memory is not updated during navigation**, making it unsuitable for dynamic scenes or tasks that require continual perception.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "## Strengths\n1. The paper introduces Mem-RPE and Mem-Nav, two tasks that explicitly target the underexplored and practically important setting of continuous memory-based navigation, aligning more closely with real-world robotics deployment than traditional episodic benchmarks.\n\n2. The Kinaema model is designed to support O(1) memory updates and access, in contrast to the O(N²) cost of standard attention-based transformers. This architectural choice is particularly well-suited for long-horizon tasks where memory scaling is critical.\n\n3. The paper provides a thorough empirical evaluation, including generalization to out-of-distribution sequence lengths, memory scaling studies, and ablation analyses. The inclusion of multiple competitive baselines (GRU, EMA, xLSTM, MooG) allows for a fair and comprehensive comparison.\n\n4. The paper is well-organized and clearly written, with strong exposition of model architecture, training setup, and evaluation procedures. Visualizations such as memory attention maps and cross-attention overlays further aid understanding.\n\n---\n\n## Weaknesses\n\n1. **Streaming history is modeled as random exploration** via varied-length sampled trajectories, rather than through an **active or structured exploration policy**. As a result, there is no analysis of **visual coverage or informativeness** of the memory contents. This limits insights into whether the model benefits from task-relevant scene exploration, which would be expected in real-world deployment.\n\n2. The **model assumes that the goal image comes from a previously observed location**, but the model lacks explicit uncertainty estimation (e.g., recognizing if a goal has not been seen), or out-of-distribution detection.\n\n3. In the **Mem-Nav task**, Kinaema is **used as a frozen plugin module** to DEBiT rather than training a end-to-end navigation policy with recurrent memory. This modular design limits the agent’s capacity that learns to memorize navigation history and learn about the environment lifelong. In another word, **the memory is not updated during navigation**, making it unsuitable for dynamic scenes or tasks that require continual perception.\n\n4. The potential for **lifelong memory** or **multi-episode navigation** is not addressed. Recent work such as **GOAT-Bench** (Khanna et al., CVPR 2024) directly tackles multi-task or long-term memory settings and could provide a more fitting benchmark or discussion context.\n\n5. While comparisons to GRU, xLSTM, EMA, and MooG are helpful, **Mamba (Gu & Dao, 2024)** is notably missing. Mamba’s state-space approach also introduce a distributed memory (D SSM applied to each channel), and would serve as a more **direct architectural comparison** to Kinaema than classical RNNs.\n\n6. Although the model is trained on **sequences of length T=100**, evaluation is extended to sequences of up to T=1000. However, there is **no justification for the limited training horizon**, nor any exploration of **scaling effects** (e.g., whether longer training sequences would improve generalization). The sharp drop in accuracy between T=100 and T=200 suggests potential **brittleness and overfitting** to the training length.", "was_revised": true, "final_justification": "From the extra result, I agree that the proposed recurrent sequence model is indeed a solid memory mechanism at the current stage. I would advocate the acceptance of this paper.", "rebuttal": "We thank Reviewer-3MFD for their review, for `having appreciated the introduction of underexplored new tasks`; the `support of O(1) updates well-suited for long-horizon tasks`; the `thorough empirical evaluation`, `generalization, ablations  and multiple competitive baselines`; the `well-organized and clearly written paper`.\n\n\n> 1. no analysis of visual coverage or informativeness of the memory contents. \n\nWe have made new experiments which provide this analysis below, distinguishing between episodes where the goal position has been *SEEN* or *UNSEEN* during the priming sequence, and also whether the priming sequence has actually been presented to Kinaema or not (if not, then SEEN and UNSEEN have the same meaning). \n\nThese are downstream Mem-Nav experiments, similar to Figure 4. We also keep the splits of episode difficulty but restricted it to 3 different difficulty levels to not overcharge the table.\n\n| SPL (%)            | DIST SPLIT → | EASY        | (2-5m)      | MEDIUM      | (5-10m)     | HARD         | (10-22m)   |\n| -- | -- | -- | -- | -- | -- | -- | -- |\n|                    | SEEN SPLIT → | SEEN        | UNSEEN      | SEEN        | UNSEEN      | SEEN         | UNSEEN     |\n| ↓ MODEL            |     PRIMER ↓ | (#=566)     | (#=116)     | (#=520)     | (#=319)     | (#=64)       | (#=290)    |\n| • DEBiT (zeroshot) |            ✘ |   41        |   41        |   40        |   42        |   47         |   41        |\n| • DEBiT (finetune) |            ✘ |   45        |   40        |   42        |   43        |   48         |   44        |\n| • w/. $h_0$ inject |            ✘ |   37        |   32        |   35        |   37        |   42         |   38        |\n|                    |            ✔ |   41   (+4) |   40   (+8) |   36   (+1) |   38   (+1) |   41   (-1)  |   40   (+2) |\n| • w/. GRU memory   |            ✘ |   43        |   42        |   42        |   43        |   50         |   45        |\n|                    |            ✔ |   45   (+2) | **48** (+6) |   43   (+1) |   46   (+3) |   50   (+0)  |   45   (+0) |\n| • w/. Kinaema      |            ✘ |   48        |   47        |   46        |   45        |   44         |   48        |\n|                    |            ✔ | **50** (+2) |   46   (-1) | **48** (+2) | **49** (+4) | **54** (+10) | **49** (+1) |\n\n\nRemarks: these new results are better than what we obtained in the submission as we also RL-trained the downstream agent longer (300M steps instead of 100M). All other parts are identical, in particular the pre-trained Kinaema and GRU models. \n\nAs we can see, \n* the model is capable of exploiting the additional information provided in the priming sequence and obtains gains: (values provided in parentheses). \n* The gains are largest for the hardest sequences (+10p)\n* The gains are not restricted to “seen” episodes. For the “unseen” ones, while the goal object had not been seen in the priming sequence, Kinaema picks it up when it is seen during the actual navigation (Kinaema memory is updated during navigation) and it does not lose it out of mind when it is potentially briefly not seen anymore, in contrast to the DEBiT encoder, which only compares the current observation to the goal.\n\nWe will include these results in the camera ready.\n\n> 2. the model lacks explicit uncertainty estimation (e.g., recognizing if a goal has not been seen)\n\nWe acknowledge this as a minor limitation of the base Kinaema model. However, we would like to point out two ways how the agent can circumvent this limitation, and the results above provide evidence for that:\n\n- (1) if Kinaema has not seen the goal position, its prediction will be OOD and the agent will observe this during RL training. Through RL-training, it can potentially detect OOD patterns.\n\n- (2) the agent also has access to the DEBiT encoder, which compares the currently observed image with the goal image and which has been trained for visibility prediction. The agent can therefore learn to trust DEBiT if it reports to have seen the goal, and to trust Kinaema in the contrary (or to detect its OOD behavior).\n\n> 3. the memory is not updated during navigation\n\nThere seems to be a misunderstanding, **Kinaema memory IS UPDATED during navigation**, and we believe that this is one of its core strengths . The two memories $\\mathbf{m}_t$ and the agent’s own $\\mathbf{h}_t$ are **BOTH** updated in parallel. In the integration Figure 1 of the appendix we omitted this update to remove clutter, we apologize for that. We will make this very explicit in the camera-ready.\n\n> 4. The potential for lifelong memory or multi-episode navigation is not addressed. Recent work such as GOAT-Bench (...)\n\nKinaema has indeed the potential to target multi-episode nav and we actually see our Mem-Nav task as a variant of multi-episode nav: the priming sequence represents the information collection step of a previous episode. There is a difference to GOAT-Bench: during our priming sequence, the agent does not take its own decisions. However, as mentioned in L75, this was a deliberate choice, as it allows us to pre-compute memory representations *before* each RL-episode.\n\nWe would like to stress that this is an important advantage  as it **provides a tremendous speed-up during RL training**. During early training phases, agents tend to make mistakes and early episodes are extremely short. Our Mem-Nav task allows the agents to immediately access useful memory even in the early training phases.\n\nWe would again stress that pre-computations are only done for the observations collected during the priming sequence. During the subsequent RL steps, Kinaema memory continues to be updated.\n\nOur model could in principle also be applied to the GOAT-Bench, up to the described limitations, and by modifying the GOAT-Bench to deal with image goals in each sub-episode.\n\n> 5. comparisons to GRU, xLSTM, EMA, and MooG are helpful, Mamba is notably missing.\n\nAs requested, we have implemented the Mamba baseline, as well as another state space model targeting long sequences, LRU (Orvieto et al., Resurrecting Recurrent Neural Networks for Long Sequences, ICML 2023).\n\n- For LRU, we used the author’s sysid-pytorch-lru repository from forgi86 on github. We have launched multiple runs with different numbers of layers and the same hyperparameters as Kinaema and GRU (emb size=3072, LR, scheduler etc). Training is still in progress due to the short rebuttal time, but we are close to finishing (epoch 147 of 200, and most models do not provide significant gains over the last 30% of training). The training and validation curves are significantly worse than Kinaema. \n\nWe provide preliminary evaluation results (Mem-RPE task, RPE-test set) below, which are comparable to the eval protocol in Table 1. We plan to report the full results during the reviewer/author discussion phase.\n\nThe LRU results are significantly below the results of our model. \n\nSeq-Len 200: Accuracies 3, 19 and 36 (for the 3 different metrics)\n\nSeq-Len 800: Accuracies 2, 12  and 24\n\n(the best model had 4 layers, compared to 3 layers for Kinaema)\n\n- For Mamba, we tried the state-space repositories both on github and huggingface. However, due to large fluctuations of activations within the mamba modules, we haven’t managed to fully train one so far, despite our efforts to provide additional regularization tailored to the Mamba model and which was not necessary for Kinaema or the other baselines. We will do our best to report the results of a stable training in camera ready.\n\n>  6. trained on sequences of length T=100, evaluation is extended to sequences of up to T=1000. \n\nWe kindly refer to Figure 3 of the supplementary material, where we show results training Kinaema (and the GRU baseline) for T=200 steps, which leads to improvements in Mem-RPE performance, in particular in precision (the first metric).\n\nWe would also like to point out that the current recurrent transformer models of the **literature** are trained on **significantly shorter sequences**. MooG [52], for instance, has been trained for T=8 and evaluated to seq of length 16.\n\n> TTT introduces fast weights and could also memorized long sequence with O(1) computational complexity. \n\nTTT is a recurrent model as are RNNs, LSTMs, GRUs, MooG, or our model, and as such it shares some of their properties, like $O(1)$ complexity and the compression of the history of observations into some memory. Instead of compressing the history of observations into an embedding (RNN/LSTM/GRU) or a set of embeddings (MooG/Kinaema), the  TTT model compresses it into theparameters of a small memory-network. Memory updates are therefore gradient descent steps.\n\nIn direct comparison to Kinaema, TTT obtains $O(1)$ by requiring to do memory updates with gradient descent, which comes with its own challenges. Integrating information into a memory this way depends on learning rates and the number of learning steps for each memory write and is generally sensitive to (inference time) optimization hyperparameters and objectives, the requirements of doing some form of batching even during inference time (where sequence batchsize is 1), and the computational downside of doing backward passes on a model during inference.\n\n> The author could also compare to methods merging KV cache as memory in transformer.\n\nMethods caching KV embeddings can decrease the computational complexity for each prediction from the classical transformer complexity of $O(N^2)$ to $O(N)$. This is achieved through causality (embeddings do not attend to the future, so once they are contextualized, they are not touched anymore and can be cached).\n\nWhile this provides a tremendous speedup, it is not O(1) complexity, as our model provides. These methods still need to attend to the full observation history at time step for each query, hence the $O(N)$ complexity. Furthermore, it does not remove the necessity to store the full observation history. The context length problem is not solved, compared to our model."}, {"review_id": "7RTpcqDGtn", "reviewer": "Reviewer_7rrc", "summary": "The authors introduce a new recurrent model termed Kinaema for the Mem-Nav task where the agent first explores the environment and then navigates to the target given a query image. Compared to the quadratic complexity of attention O(N^2), Kinaema is designed to update and query memory in O(1). In addition, gating functions are integrated into Kinaema to decide the speed of updates for each memory item. \n\nExperiments comparing Kinaema with several recent works and classic RNNs demonstrate its advantages in the Mem-Nav task especially for long-term navigation. Ablation studies and visualizations are provided with new insights.", "questions": "__Q1__: Line 60: “(iii) the integration of the sequence model in a navigation agent trained with Reinforcement Learning;” It would be great if the authors can explain where the Reinforcement Learning (RL) part is in the paper. \n\nExcept for Mem-Nav that looks like the hard version of ε-greedy (__W1__) which relates to RL, the main training is about minimizing the discrepancy of the relative pose between a query image and the current agent’s position, denoted as L_{RPE} on Line 199. However, this formulation does not explicitly relate to the basic framework of RL that involving reward, action, and state. In particular, the paper does not describe a reward function, therefore it is unclear how the agent is “trained with Reinforcement Learning” as claimed in the (iii) contribution. \n\n__Q2__: Line 110 “as p = {t, R}, where t = {d, \\theta} is the translation, i.e. distance and bearing angle from the agent to the position depicted in query image. R is the rotation matrix of the goal towards the agent, which is of limited relevance to a navigation task, but which we supervise to increase the learning signal during training.” \n\nThere are two rotation-related notations of R and \\theta, how are they related to each other? Why is only R used for training but not the other one? \n\n__Q3__: What kind of gating functions are used (Line 145)? Does it follow the standard gating function implementation in the literation? Is it Sigmoid or Tanh? It might be beneficial to add such details for completeness.  \n\n__Q4__: What does each color represent specifically in Figure 7?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "__Strengths__: \n\n__S1__: A new recurrent model is introduced with complexity of O(1) to improve classic RNN and attention’s limitations.\n\n__S2__: Figures and tables are presented in high quality.\n\n__S3__: Experiments are comprehensive including comparison with several recent methods an classic RNN baselines.\n\n__S4__: It is good to demonstrate that Kinaema outperforms GRU in each setting.\n\n__S5__: It is good to include experiments with longer trajectories T=200 T=800 to show the memory’s advantages.\n\n__S6__: The appendix is provided with great detail and includes a demo. \n\n\n__Weakness__: \n \n__P__: Overall, the presentation could be further improved, including __W2-4,7,8__. \n\n__W1__: Differences between Mem-Nav and “Hard” ε-greedy. \n\nThe description of “Mem-Nav” from Line 36-38 sounds like a hard version of the classic ε-greedy algorithm in the Reinforcement Learning [1] textbook: \n\n\"The method that chooses randomly among all the actions with equal probability ε, and greedily with probability 1 − ε, is called ε-greedy.\" \n\nSince this work falls within the scope of deep reinforcement learning, I believe it would be beneficial to more clearly differentiate the task from the ε-greedy framework or to discuss the conceptual connections between them. \n\n[1] Sutton et al. Reinforcement Learning: An Introduction. \n\n__W2__: Line 58 “(ii) “Mem-RPE”, a new task requiring the estimation of relative pose between an image and agent memory; “ \n\nThe term “RPE” appears here for the first time without an explicit interpretation. Providing a clear explanation of the abbreviation “RPE” at its first mention would improve the clarity of the writing. \n\nLine 52-53 in the Appendix provides a good example: “As mentioned in Sec. 4.1 of the main paper, we train all models by jointly optimizing the relative pose estimation (RPE)”. \n\n__W3.1__: The presentation of Table 1 – 4  can be improved. To be concrete, the numbers are left without explanation. Table 5 and Figure 4 are good examples. \n\n__W3.2__: What do the values in green in Table 1 represent? Line 236 mentions two metrics of Success Rate (SR) and SPL, but neither is explicitly referenced in the text or table caption. If the values correspond to Success Rate, it would be helpful to include a percentage sign (“%”), as the numbers appear to range from 0 to 100. Without such clarification or indication of scale, it is difficult for readers to interpret the results. A similar issue is present in Table 4 in the Appendix. \n\n__W3.3__: “SPL” appears for the first time on Line 238 without explanation. Although it is defined in the referenced literature, it would be helpful to also provide the full name within this work for clarity. \n\n__W4__: In Figure 3, the Y-axis is labeled “Mem-RPE Acc. (%)”, but the abbreviation “Acc.” is never used in the main text. Instead, only “Success Rate (SR)” and “SPL” are introduced in Lines 236–238 under the Metrics / Mem-Nav section. It would be helpful to maintain consistent terminology throughout. Similar inconsistencies are also found in Figures 2 and 3 in the Appendix. \n\n__W5__: Figure 4 does not show clear advantages over the standard DEBiT without the introduced modules. It would be helpful to include a discussion or provide new insights to clarify the benefits. \n\n __W6__: It would be great to provide further discussions of the poorer performance “0 11 22” in Seq len 800 in the last row in Table 2 (a). \n\n__W7__: SR and SPL are reported separately, rather than presented together, in Tables 1–4 and Figures 3 and 4 (Assuming Tables 1-4 represent SR results). It would be more comprehensive to report both metrics side by side, in line with common practice in the literature. \n\n__W8__: It is good to include attention visualizations in Figure 6, but they are difficult to interpret without additional context. For instance, appending a goal image could help assess whether the token attention is meaningful. Further explanation of the color coding would also be helpful. \n\n\n__Minor Issues__:\n\n__M__: “but using a GRU; (with h0-injection) is a baseline using the hidden state ht of the original agent to encode the priming sequence with no goal, teacher forced,” \n\nThe notation of “;” looks weird between “GPU” and “(with h0-injection)”.", "was_revised": true, "final_justification": "The authors have provided further details and clarity regarding the relationship of Mem-Nav and ε-greedy, the PPO in the context of RL as well as the Gating mechanism used in this work. In addition, new experiments in the rebuttal demonstrate the advantages over the standard DEBiT. Therefore, I raise my score to 4: Borderline accept.\n\nHowever, the idea appears to be a slight modification of a recurrent sequence transformer-based model applied to the Mem-Image navigation task. Since the main advantages were demonstrated only in new experiments provided during the rebuttal, and not in the original submission, I believe a score of 4: Borderline Accept is reasonable instead of a higher score.", "rebuttal": "We thank the reviewer for appreciating the `novelty` (S1), the `quality` (S2,6), and the `relevance` (S3-5) of our contribution. Given the length of this third answer, we formatted it differently from the answers to reviewers rguE and 3MFD.\n\n###  (W1,Q1) Differences between Mem-Nav and ε-greedy, role of RL.\n\nWe do not see a clear link between the Mem-Nav task and the ε-greedy algorithm.\nWhen we said “the agent can explore the scene” to describe one application case in our introduction, we did not mean “exploration” in the same sense as in the ε-greedy algorithm found in classical Reinforcement Learning (RL) literature.\nε-greedy policies provide a way to address the classical problem in any optimization of exploitation vs. exploration during the training of the agent with RL. It is a way to explore the state-action space to get better value estimates and avoid local optima to find better policies.\nIn our Mem-Nav task, the initial sequence of images provided to the agent at the beginning of a navigation episode that we call priming sequence, or primer, is not meant to help optimize its policy, but to build a better representation of its environment. Letting the agent itself “explore” the environment for a few steps without a goal  was only an application example, such a primer could also come from previous experiences in the same environment, from another agent, solving another task, etc.\n\nIn this paper, RL is only involved in the final phase of our training pipeline, the downstream navigation task we call _Mem-Nav_. The main focus of this paper is on the initial pre-training phase, the pretext task we call _Mem-RPE_, which is supervised.\nWe apologize for having delegated most of the task definition, the environment model, and the training algorithm to existing work in the literature for this final RL phase. We will add the following details to the paper:\n\nAs in the standard ImageNav benchmark [8], for each Mem-Nav episode, the agent is positioned at a random start pose, in a random scene (eg. an apartment), and needs to reach a random goal position. \nThe agent receives a single image taken from the goal position, and needs to navigate there using 4 discrete actions: stop, move forward 25cm, and turn left or right 10$^\\circ$, based on images observed from the RGB camera mounted on it. Specific to our new _Mem-Nav_ task, the agent also has access to a static sequence of images taken offline, called primer, or priming sequence.\n\nAs in [8] and prior work in Embodied AI, the agent receives a dense reward $ r = \\Delta d_\\mathrm{goal} + 10. \\mathbb{1}_\\mathrm{success} - 0.01 $, which guides it towards the goal, reinforces success, and encourages shorter paths through slack cost. We train our agent using the Proximal Policy Optimization (PPO) [62] algorithm.\n\nHere the exploration of the state-action space is provided by the PPO algorithm, in particular its entropy regularization term, so there is no need to use an $\\epsilon$-greedy policy. Although the primer is collected by the robot in an application sense, in an RL sense, the primer is not actively collected by the agent, it is just part of the initial observation.\n\nIn new experiments for this rebuttal, we also ran some form of ε-greedy _evaluation_ of the original DEBiT agent, where we replace the static priming sequence by an “active” exploration phase of same length (200 steps) during which random actions are selected with probability 1/10 (greedily following the trained policy otherwise). Results are given in the table below (cf answer to W5). Values are not directly comparable to the ones with primers which are generated from a heuristic designed to maximize coverage of the scene (see Appendix A.2) but are _goal-less_. ε-greedy exploration might have worse coverage but the trained policy will try to go towards the goal. Nonetheless, results are on par if not lower than the zero-shot baseline, which would indicate that the agent was not able to build useful internal states from these ε-greedy explorations.\n\n[62] Proximal Policy Optimization Algorithms - John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov\n\n### (W5) Advantages over the standard DEBiT\n\nIn new experiments, longer fine-tuning (300M steps vs. 100M for the results reported in the initial submission) have revealed a clearer advantage of Kinaema on Mem-Nav task, as shown in the table below:\n\n| SPL (%) | DIST SPLIT → | EASY  | (2-5m) | MEDIUM  | (5-10m) | HARD | (10-22m) |\n|:--- | ---:| --- | --- | --- | --- | --- | --- |\n|  | SEEN SPLIT → | SEEN  | UNSEEN  | SEEN  | UNSEEN  | SEEN  | UNSEEN  |\n| ↓ MODEL | PRIMER ↓ | (#=566) | (#=116) | (#=520) | (#=319) | (#=64) | (#=290) |\n| • DEBiT (zeroshot) | ✘ |   41        |   41        |   40        |   42        |   47         |   41        |\n| • DEBiT (finetune) | ✘ |   45        |   40        |   42        |   43        |   48         |   44        |\n| • DEBiT (ε-greedy) | ✘ |   39        |   34        |   35        |   36        |   38         |   36        |\n| • w/. $h_0$ inject | ✘ |   37        |   32        |   35        |   37        |   42         |   38        |\n|   | ✔ |   41   (+4) |   40   (+8) |   36   (+1) |   38   (+1) |   41   (-1)  |   40   (+2) |\n| • w/. GRU memory | ✘ |   43        |   42        |   42        |   43        |   50         |   45        |\n|  | ✔ |   45   (+2) | **48** (+6) |   43   (+1) |   46   (+3) |   50   (+0)  |   45   (+0) |\n| • w/. Kinaema | ✘ |   48        |   47        |   46        |   45        |   44         |   48        |\n|  |  ✔ | **50** (+2) |   46   (-1) | **48** (+2) | **49** (+4) | **54** (+10) | **49** (+1) |\n\nwhere all baselines have also benefited from the same extended training,\nincluding fine-tuning of DEBiT (imagenav) in hm3d/train.\nParenthesized deltas highlight the capacity of each model to exploit the priming sequences collected offline.\nWe also distinguish between episodes where the goal has been SEEN or UNSEEN in the priming sequence, as requested by reviewer 3MFD. We also kindly refer to comments in the answer to this reviewer.\n\nThese new numbers will be used to update Figure 4 and enrich the corresponding discussion in the text.\n\n### (W6) “Poor performance in last line of Table 2”\n\nTab.2a ablates memory capacity and is commented at L.295-301.  We conjecture that the drop in evaluation over longer (T=800) sequences observed for the maximum capacity of $50$ mem-tokens is caused by a too large memory capacity wrt to the trajectory length, as there is less need for compression — it is widely conjectured that compression is linked to intelligence. For this reason, the compression-less memory handling strategy learned by the model tends to overfit and does not generalize well to longer sequences.\n\n### (Q3) Gating\n\nWe have provided a full explanation of gating with the corresponding equations in the answer to reviewer rguE and we kindly refer to this part for details. In the camera ready, we will update Eq.5 with these details.\n\n###  (P, W2, W3, W4,W7,W8)\n\nThe overall presentation of the paper will be improved as followed:\n\n* Acronyms will be defined at their first appearance:\n    - (W2) l.58:\n    \n    > (ii) **R**elative **P**ose **E**stimation from **Mem**ory (Mem-RPE), a new task [...]\n\n    - (W3.3) l.236-238:\n\n    > **S**uccess **R**ate (SR) [...], and **S**uccess weighted by **P**ath **L**ength (SPL) [2], $SPL=$ [...]\n\n    - (W4): l.232: \n\n    > We provide **Acc**uracy (Acc.) of recognized poses [...]\n\n    y-label of Fig.4 replaced by\n\n    > Accuracy (%)\n\n### (W3.1,3.2,7) Description of Tab.1-4\n\nMetrics used for Mem-RPE evaluation in Tab.1-4 introduced at l.229-233\nwill be explicitly reminded in their captions, eg. Tab.1:\n\n> Mem-RPE **Accuracy** for different models, sequence lengths and (distance,angle) thresholds\n\nImproved captions should also address misunderstanding (W7) where _Mem-RPE_ Accuracies\nhave been confused with _Mem-Nav_ SR, which is reported in Tab.5 only, alongside SPL.\n\n### (W8, Q4) Meaning of colors in Fig.6\n\nAssuming (Q4) referred to Fig.6 instead of Fig.7 which is grayscale, we will clarify caption of Fig.6 and the related discussion l.334-343:\n\nIn this visualization, each image of the sequence has been used as a goal image (which is normally fixed during a navigation episode) to query memory built up from past observations. The purpose of the figure is to highlight attention patterns and coherence along a sequence, colors are arbitrary and only used to identify which mem tokens \"activate\" on each patch (no associated semantic).\n\n### \\(M\\) Inline format of enumeration\n\nThe semicolon \";\" denotes separation of items in the enumeration of variants labelled with the text in parentheses, each corresponding to a row in Tab.5\n\n### (Q2) Confusion between notations $\\theta$ and $R$\n\nSorry for the confusion, we will redefine $\\mathbf{t}$ as a 3D translation vector $\\{x,y,z\\}$ in cartesian coordinates from current agent position to goal position. We had chosen to define it as a 2D vector $\\{d,\\theta\\}$ in polar coordinates, where $d$ is the euclidean distance from agent to goal, and $\\theta$ the angle towards the goal from the direction faced by the agent. This initial choice was motivated by its significance and interpretability for navigation in general, where agents evolve mostly in 2D and need an egocentric view of their goal, but is not what we used for Mem-RPE, where we cared about 3D camera poses and image alignment.\nOn the other hand, $R$ is a 3D rotation matrix describing the relative orientation of the goal camera wrt. the one of the agent. It is not used in the ImageNav and Mem-Nav tasks, and only provides some supervision signal to implicitly align point of views and compare images for Mem-RPE.\n\n### (Q4) Meaning of colors in Fig.7\n\nFig.7 represents top-down occupancy grid where _gray_ is unknown or obstacle and _white_ is free/navigable."}]

Author rebuttal:

Meta-review: In this paper, the authors propose Kinaema, a recurrent transformer sequence model designed for embodied agents that require long-horizon memory. The model maintains a distributed set of memory embeddings updated in O(1) time using a transformer block combined with GRU-style gating, avoiding the quadratic complexity of standard attention while maintaining high capacity. The method is evaluated on two new tasks: Mem-RPE (relative pose estimation from memory) and Mem-Nav (navigation to previously seen goals), showing improvements over recurrent baselines such as GRU, xLSTM, EMA, and MooG, particularly on long sequences.

Initial reviews were mixed (one accept, one borderline accept, one borderline reject). Strengths highlighted by the reviewers included the novelty of introducing an O(1) recurrent transformer memory (R-3MFD, R-7rrc), the introduction of Mem-RPE and Mem-Nav as valuable new benchmarks (R-3MFD, R-7rrc), and the comprehensive empirical comparisons across multiple strong baselines (R-3MFD, R-7rrc). R-rguE, while more negative, also acknowledged the motivation behind the architecture. On the other hand, weaknesses identified were the lack of coverage analysis and questions about whether memory was updated during navigation (R-3MFD), missing baselines such as state-space models (R-3MFD, R-7rrc), unclear distinction of Mem-Nav from ε-greedy exploration (R-7rrc), limited justification for training sequence lengths (R-3MFD), and general clarity issues around gating, notation, and figures (R-rguE, R-7rrc).

The authors provided a detailed rebuttal and discussion that addressed most of these concerns. They added new experiments analyzing SEEN vs. UNSEEN goals and extended training to 300M steps, which showed clearer Mem-Nav gains (addressing R-3MFD, R-7rrc). They made claridfications about Kinaema memory being updated during navigation, introduced state-space baselines (LRU and attempted Mamba), reporting significantly weaker performance for LRU and noting difficulties training Mamba (addressing R-3MFD, R-7rrc). They distinguished Mem-Nav from ε-greedy, provided an ε-greedy baseline, and clarified the PPO training setup (addressing R-7rrc). Finally, they committed to fixing the clarity issues noted by R-rguE and R-7rrc (notation, acronyms, tables/figures, gating explanation). Post-rebuttal, R-3MFD advocated acceptance and R-7rrc raised their score to borderline accept. R-rguE 's rating remained unchanged while the concerns were mainly focused on clairty. 

The core contribution of introducing a scalable recurrent memory mechanism for embodied navigation is sound and well supported by additional experiments and clarifications. Given the novelty of the model, the new tasks introduced, the strong empirical foundation, and the reviewers mostly converging post-rebuttal, I recommend this paper be accepted, and highly recommend the authors incorporate the new results, baselines, and clarity improvements in the final version.
Decision: Accept (poster)