HippoTune: A Hippocampal Associative Loop–Inspired Fine-Tuning Method for Continual Learning
Authors: chenyanxi
Xiuxing Li
Han Yuyang
Zhuo Wang
Qing Li
Ziyu Li
Xiang Li
Chen Wei
Xia Wu
Venue: ICLR 2026 Poster (2026)
Keywords: Associative Memory
Key–Value Memory
Parameter-Efficient Fine-Tuning
Continual Learning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: HippoTune is a continual learning fine-tuning method that embeds hippocampal-inspired iterative retrieval loops into each Transformer layer, boosting accuracy by 5–8 pp while halving training FLOPs.

Abstract: Studies have shown that catastrophic forgetting primarily stems from the difficulty of reactivating old memories; although parameter-efficient fine-tuning can mitigate forgetting while keeping most model parameters frozen, it still falls short in fully reawakening knowledge of prior tasks. In contrast, humans can efficiently retrieve and flexibly integrate existing experiences when learning new tasks, thereby maintaining stable performance on earlier ones. During cognition, the hippocampal EC–DG–CA3–CA1 circuit engages in multiple rounds of associative recall, and its pattern-separation and memory-completion mechanisms excel at activating historical information. Inspired by this mechanism, we propose HippoTune, a latent-space iterative retrieval strategy that embeds a query–retrieve–feedback loop within each Transformer layer. Starting from the hidden state as an initial query, the model performs a few rounds of soft key–value retrieval, projects the retrieved signals back into the query, and updates it iteratively until convergence or a preset iteration limit. Theoretically, we show this process implements a Krylov-style polynomial approximation, equivalent to a differentiable second-order preconditioner, thereby deepening retrieval in a principled way. Empirically, HippoTune outperforms classical buffer-free PEFT-CL methods by 5–8\% in accuracy across three vision benchmarks, while reducing training FLOPs by 50\%, effectively mitigating forgetting under tight compute constraints. 
Code is available at: https://github.com/yan4xi1/HippoTune.

Published as a conference paper at ICLR 2026
HIPPOTUNE: A HIPPOCAMPAL ASSOCIATIVE LOOP-INSPIRED FINE-TUNING METHOD FOR CONTIN-
UAL LEARNING
Yanxi Chen' Xiuxing Li Yuyang Han"
Zhuo Wang!
Qing Li
Ziyu Li
Xiang Li
Chen Wei Xia Wu'
School of Computer Science and Technology. Beijing Institute of Technology
2 School of Artificial Intelligence, Beijing Normal University
Li Auto Inc
[yanxi, xx1, (uxia)bbie moxiaa@bit.edu.co edu.cn
ABSTRACT
Studies have shown that catastrophic forgetting primarily stems from the difficulty reactivating old memories; although parameter-efficient fine-tuning can mitigate forgetting while keeping most model parameters frozen, still falls short in fully reawakening knowledge of prior tasks. In contrast, humans can efficiently retrieve and flexibly integrate existing experiences when learning new tasks, thereby main- taining stable performance on earlier ones. During cognition, the hippocampal EC-DG-CA3-CA circuit engages in multiple rounds of associative recall, and its mechanisms excel at activating histori-
patter--epprattion and memoryy-cmpleed cal information. Inspired by this mechanism, we propose HíppoTune, latent-space iterative retrieval strategy that embeds a querieretrieve- feedback loop within each Transformer layer. Starting from the hidden state as an initial query, the model per- forms a few rounds of soft key-value retrieval, projects the retrieved signals back into the query, and updates iteratively until convergence or preset iteration limit. Theoretically, show this process implements a Krylov-style polynomial approx- imation, equivalent to a differentiable second-order preconditioner, thereby deep- ening retrieval in principled way. Empirically. HippoTune outperforms classical buffer-free PEFT-CL methods by 5-8% accuracy across three vision benchmarks, while reducing training FLOPs by 50%, effectively mitigating forgetting under tight compute constraints. Code is available at:
INTRODUCTION
Deep neural networks excel under independent and identically distributed (i.i.d.) data, yet continual learning (CL) settings where tasks arrive sequentially and distributions shift, they often suffer catas- trophic forgetting: performance on earlier tasks degrades sharply when learning new ones McCloskey & Cohen 1989); Kirkpatrick et al. (0077). Classical approaches such replay, regularization, and structural isolation often require large-scale fine-tuning of the entire network, leading to high compu- tational and storage costs models grow. Moreover, many such methods largely model-agnostic and do not leverage inductive biases of prevalent architectures such as Transformers Rebuffi et al. (2017); Kirkpatrick al. (2017); Mallya & Lazebnik (2018); Vaswani al, (2017).
Parameter-efficient fine-tuning (PEFT) mitigates these costs by inserting a small number of trainable modules (e.g., adapters, LoRA modules, prompts) into otherwise frozen backbone, substantially reducing training overhead Houlssyy et al. (2019): Hu et al. (2022): Lester et al.] (2021): Li & Liang pool" and, at inference time, retrieve and activate a subset of submodules using the sample representation as a query. However, this single-step retrieval can under-activate old-task memories and often requires full backbone forward pass to extract high-level features for the query, introducing additional latency. In contrast, humans
(202I). Recent PEFT-CL methods further maintain "parameter/promme
Corresponding author. I This work done while the author Auto. Now Shopee.

---

Published as a conference paper at ICLR 2026
performing previously learned tasks engage in multiple rounds of associative recall and integration, leading to richer reactivation of historical knowledge. Concretely, sparse cues can trigger multi- recall via the hippocampal EC-DG-CAS CA3 circuit, enabling pattern separation and completion without repeatedly reconstructing high-level semantic representations. This circuit serves as a core pathway for memory formation and retrieval in the brain: information flows from the entorhinal contex (EC), through the dentate gyrus (DG) for pattern separation, is completed via auto-associative recurrence in CA3, and finally integrated in CAI into a coherent memory trace Yassa traceYassa & Stark (201T):
Treves & Rolls 1994).
Inspired by the pattern separation, association, and integration mechanisms of the EC-DG BIG-CA3-CA CAl circuit, we propose Latent Deliberation, layer-internal differentiable, iterative retrieval mechanism. At each Transformer layer, we embed light-weight associative loop: the previous layer's s hidden state serves as the initial query; we perform soft key-value retrieval to activate relevant memories; the retrieved signal is linearly projected and fed back to update the query; the loop continues until convergence maximum number of iterations; finally, we fuse the per-iteration outputs to realize multi-level completion and integration of prior-task knowledge. Operating entirely in latent space avoids repeated construction of high-level features, and exposes practical budget controls via the maximum iteration count, a convergence threshold, and top-k sparsity. This unified view also clarifies relationships prompt-pool methods such L.P Wang et al.] (2022b), DualPrompt Wang al.) (2022a). and CODA-Prompt Smith et al. (2023): these can be seen as single-depth retrieval, whereas our method provides differentiable deepening of retrieval depth increase expressiveness and precision in memory access, See Fig. for the differences between our method and the classic PEFT-CL approaches.
On the theory side, we characterize two key properties. First, near a fixed point, multi-step iteration implements a Krylov subspace polynomial approximation the inverse Hessian, yielding an implicir second-order preconditioner for gradient propagation, achieving curvature correction in a finite number of steps without explicitly computing or storing second-order information Saad (2003); Martens & Grosse (015). Second, we provide convergence and stability conditions based on step sizes and Jacobian spectral bounds, which translate into actionable choices for maximum iteration count, temperature, and entropy regularization; in effect, they operationalize the intuition that "longer deliberatioon/retiivaal leads to better old-tast lask performance" into verifiable and tunable optimization criteria Boyd & Vandenberghe (2004).
We highlight four key contributions:
A unified retrieval perspective for PEFT-CL. We distill existing prompt-pool continual learning methods into a single key-value formulation, clarifying their shared trade-offs and the limits of one-shot retrieval.
Latent Deliberation: hippocampal-inspired iterative retrieval. Drawing on the SC-DG-CA3-CAL circuit, we embed lightweight, multi-step soft bookup-and-feedbeee process within each Transformer layer, deepening memory activation without extra backbone passes.
Krylov-subspace preconditioning theory. We prove that our finite-step loop implements poly- nomial approximation to the inverse Hessian, acting as an implicit second-order preconditioner. We also derive convergence and stability criteria to guide iteration count, temperature, and regulariza- tion.
Strong performance at low compute cost. On three vision benchmarks, Hippo delivers substantial accuracy gains over one-shot PEFT-CL baselines while using only about half the training FLOPs, demonstrating efficiency under tight resource constraints.
2 RELATED WORK
Continual Learning with Garameter-Efficiee Fine-Tuning Within PEFT paradigm, continual learning has evolved into a prominent direction, now marked by the convergence of modularity. routing, and theoretical grounding. Early methods such as L2P et al. (2022b). DualPrompt Wang et al. (2022a), and CoDA-Prompt |mmihh et al. (2023) introduced learnable prompt pools with key-query retrieval to select modules during training and inference, mitigating forgetting without
2

---

Published as a conference paper at ICLR 2026
Classic PRFI-CI Tramework
HippoTune: Latent Deliberation
V
VIT
A 1
Aree
par corameter
posl
parssnecerr
pool
Recurrent Retrieval
Figure 1: Classic PEFT-CL vs. HippoTune. (Left) Standard prompt-basee bosept-based continual learning retrieves a single prompt e()) per ViT layer to compute h()), (Right) Hippo'Tune iteratively retrieves and integrates multiple prompts {ut ...., of') using h(l-1), enabling deeper memory activation and improved retention.
relying replay. Later approaches such LAE Gao et al. (2023a), HiDe Zuo et al. (2023) and MoE- Adapter Yu et al. (2024) have further improved adaptability and efficiency via dynamic expansion, module merging, and expert routing. Theoretical work, including NTK analysis Doan et al. (2021)
and loss landscape studies, has provided insights into how routing reduces gradient interference. However, most methods lack end-to-end optimization and seldom explore fundamental architectural principles for CL., limiting their scalability.
Sespoccaapp--Nooootree Inspired Continual Learning Inspired by the hippocampus- neocortex interplay, continual learning research has proposed the Complementary Learning Systems (CLS) theory: the hippocampus rapidly encodes new experiences, while the neocortex gradually extracts generalized knowledge. Building this, models like FearNet Kemker & Kanan (2018). CLS- ER Arani et al. 2022), and Triple Memory Networks Wang et al. (2021) employ short- and long-term memory modules to balance fast adaptation with long-term retention via experience replay. Key cognitive mechanisms such as hippocampal replay, pattern separation (DG), and pattern completion (CA3) have been abstracted into algorithmic strategies. Some models adopt key-value memory for associative retrieval, while GATE Liu et al. (2025) simulates gated pathways across hippocampal subregions. Despite improving the stability-plasticity trade-off, these brain-inspired methods often architecturally complex, heplaa-dependemt and rarely applied under the PEFT paradigm. In this work, we propose a fine-grained emulation of hippocampal associative memory, aligned with the EC-DG-CA3-CCE circuit. We further validate its biological plausibility and computational efficacy from both theoretical and empirical perspectives.
METHODOLOGY
We unify all PEFT modules into a shared retrieval pool and perform iterative key-value lookups and one-shot fusion at each Transformer layer, mimicking EC-DG- CAG CA1 hippocampal loops to dynamically activate and integrate past-task knowledge. The model is trained end-to-end with classification, orthogonality, and entropy losses, using truncated BPTT align training and inference budgets.
3.1 PROBLEM DEFINITION
In the continual learning (CL) setting, the model exposed to sequence of tasks (T1, T2,..., } each associated with dataset D, (xx, y.)). The goal is to learn a new task T while maintaining performance on previous tasks {Tı.... T 1.. Formally, given a model output f(r; 9), we aim to optimize:
min Σ c (f(z;))),
(I)
where (k) denotes the cross-entropy classification loss for task
3.2 PEFT-CL FRAMEWORK FORMALIZATION
In this subsection, present formalization of the PEFT-CL framework: unify all lightweight modules into a shared retrieval pool, define how to compute relevance scores from a frozen backbone state, and show how aggregate module outputs update the model representation.
3

---

Published as a conference paper at ICLR 2026
We collect all m parameter efficcient modules into single pool
V (((1) G(m)) },
indexed by a learnable key matrix
K 16
k(m))T
E IRa
Each ( parameterizes a small PEFT block that takes layer hidden state as input and produces residual update (for example, an adapter, a prompt-induced projection, or a LoRA-style low-rank block).
Following prior works Gao et al. 2023b); He et al. 2022), we use o(х; (()) as an abstract notation to unify these different PEFT modules; details of the specific form are provided in Appendix A Here
o(;; ((0) denotes the forward mapping of the i-th PEFT module applied to the hidden state
Given frozen-backbone hidden state x E Rdd, we first compute the routing scores
K
g = softmax(s) E A m-1
with optional Top-k truncation. Each module then emits residual A = (z;; B()) R which can be understood as the effect of the i-th PEFT module on this layer's representation. We stack all residuals as
AH [ ...., (m)
Starting from the current backbone state h 2, conceptually update it by mixing all module outputs with the routing weights g:
h h h g AH. AH.
In implementation, this update realized by integrating the PEFT modules into the backbone block so that the layer directly outputs the updated h; the residual formulation above is an equivalent, unified view used for analysis. We provide detailed explanation in Appendix [A on how classical PEFT-CL methods correspond to this framework.
Why this unification matters.
1. Query cost. Using the model's hidden output the retrieval query leverages rich semantic features but incurs extra computation.
2. Retrieval depth. All existing PEFT-CL methods perform only single retrieval; this framework points naturally to deeper, iterative retrieval strategies.
3. Key-gating design. Learning and regularizing K, and choosing temperature, Top-k or entropy penalties, determines which modules activate.
3.3 LATENT DELIBERATION
At each Transformer layer, we extend the standard forward pass into a controllable dynamic process. modeled as an iterative associative loop.
We treat the hidden state from the previous layer (1(1)) E & as the initial query: (1) = (I-1), Each layer maintains a learnable key matrix (1) E Mxd and value matrix v(I) E MM to capture old-task subspaces. This inspired by the EC-DG structure in the hippocampus, where K(!) acts as guide to fixed-point memory.
At step f, the query (*) retrieves from memory via:
( k(t)
S(t) = softmax
( = (11
T
where the temperature T modulates retrieval sharpness. Optionally, Top-k filtering can be applied to (1) to retain only the most relevant memory slots. The top-k hyperparameter is robust within the
4

---

Published as a conference paper at ICLR 2026
ViT(O
7
Pattern separation
Fine-tumed parameters
(DG)
Frozen parameters
and concutenate
Initial
K
Lorth
Memory Integration Fusion
Matrix product
Query
(CAI)
Gating mechunism
Sevding
eent
(EC)
5
p(
CAl
Repeat max sleps until early stopping
EC
p
DG
Recurrent Retrieval (CA3)
The CAI circuit
The ourper of the (I-1) th layer
The 1-th query
5
weight parameter
w0t)
value
p(o
linear tramsformation
Virdo
The I-th layer backbone
Figure 2: comparative illustration of HippoTTune. At each Transformer layer, use the hidden state initial query iteratively perform key-value soft retrieval (with orthogonality and entropy regularization), update the query via projected residual feedback until convergence or max iterations, then fuse all retrievals into memory-enhanced output, enabling selective multi-round activation of parameter-pool submodules.
range of 3 to 10. As this is a standard setting for prompt-based methods, we omit from further discussion.
Then, the query is updated by incorporating the retrieved memory:
(++1) a (1- (v.0).
(5)
where p()) layer-specific linear transformation, and [0, 1] controls the blending. The CA3 region features an auto-associative recurrent mechanism and can be regarded as the core of associative memory. This can be seen as a minimal abstraction of the recurrent CA3 circuit performing memory completion and state integration. The loop terminates when either [|2](t) - -(t-1) or t = maxx.
To avoid repeated forward passes after each retrieval iteration, we adopt one-shot fusion strategy. which integrates retrieved vectors within the latent space unified manner. Specifically, the retrieval vector ((() obtained each iteration / is concatenated along the feature dimension to form an aggregated retrieval vector:
Vcat T 1| (1
|| [ E RTd.
(6)
where || denotes vector concatenation.
Next, the output of the (l- 1)-th layer, denoted as (!-1), is combined with the concatenated retrieval vector Veat and fed into the 1-th layer's ViT block to produce the output of layer l:
1 ViT(") ([A0-D)][/a)))
where (iT()) represents the backbone network of the 1-th layer. This one-shot fusion operation corresponds the CA1 region in the hippocampal circuit, which is responsible for integrating the retrieved information from both DG and C.A3 and producing a complete memory representation.
Importantly, this mechanism enables explicit controllability during inference via hyperparameters such as Tmaxx E, and Top-k, allowing flexible trade-offs between retrieval quality and efficiency. The method framework is shown in Fig. The pseudocode is provided in Appendix B
5

---

Published as a conference paper at ICLR 2026
3.4 END-TO-END TRAINING OBJECTIVE
We design a unified loss to jointly optimize task performance, retrieval sparsity, and module disentan- glement:
L = Lilss + Aorth forth + Aent fent
(8)
Classification Loss Cets: Cross-entropy loss supervising downstream performance:
N
Cele = N
yice log Pi.e.
(9)
==
c=1
where yi is the one-hot label and pi,e the predicted probability.
Orthogonality Regularization Corth: Encourages keys K(O) to be orthogonal, reducing memory interference:
Cotth = Σ| KO -1
(10)
Entropy Regularization L Controls the entropy of retrieval weights S(1,, balancing sharpness and robustness:
T
Lent = -
log 5!
(11)
The weights Anrth and Aont balance these objectives, guiding the model towards disentangled, controllable, and generalizable behaviors.
During training, adopt Truncated Backpropagation Through Time (BPTT), propagating gradients only through the final steps of the retrieval loop. This design aligns with the dynamic budget inference (e.g., Tmax. Top-k), ensuring consistency between training and deployment.
4 THEORETICAL ANALYSIS: MULTI-STEP RECURRENCE AND HIGHER-ORDER PRECONDITIONING
We abstract single-layer "recurrence" as gradient descent smooth potential function d(z):
((+1) 4 yy 1 Vc(("")), 1, Tiax 1,
(12)
with step size 17 0. The outer loss depends only on the final state of (TTmxx), so we denote
OC
Dont
Dy
(13)
Proposition (Krylov Subspace Polynomial Approximation). Suppose & is twice differentiable near a fixed point q², and its Hessian Hl V²o(a") is symmetric positive definite. Further assume the step size satisfies p(1 n/1) < 1. Define
0 (step)
J= nH, P=
bo PT Jout-
(14)
00 4-
Then the leading term of the gradient w.r.t parameters after Thax steps
T muax
VoLT
(JT) bo KInas (H) by
(15)
k=
where
Tunn -1
Krraa (H)
Σ (I-nHTJ
(16)
R=0
is the Krylov series operator, As Tmax Xi, the Neumann series converges and
Kr (H) - (1 (n = VoLo H by.
(17)
The detailed proof is provided in Appendix C
Corollary (Effect of Finite-step Preconditioning). For any funte Tmax. the operator KT (H) is a truncated polynomial approximation of H in the Krylov subspace. In practice Tmax = ~ already yields effective second-order correction at only linear computational cost.

---

Published as a conference paper at ICLR 2026
Method
GFLOPs
Sey CIFARIOD
Seq ImugeNettR
Seq CUB200
Clasical-CI. buffer)
LsFiLa & Holem 2016
16.88
80.29.10.86
87.33:10.73
60.74.0.511
68.55
69.75
80.55-2.088
DER Buzzaga éf (2000
16.88
84.50:11.77
90.16:10.61
54.21
65.26:20.88
77.4240.71
83.61=00.99
PEFT-CL (wle baffer)
L2P Wang et (2222b)
35.20
82.76::1.17
88.48:0.83
71.26:0..4
76.13=00446
68.39:1046
78.29:00.38
DualPromph
2022a
35,38
85.56.10.33
90.33.10.33
0.200
13.39
66.00.1..57
17.92.0.50
KODA-PronipiiSin
2072
35.84
86.28:1::26
91.05
74.05
08.14
72.45:0.51
78.94:0.37
Gao'el
85.83.10.88
HiDe-Prompl /300 el ala (3223
35.25
88.25:10.24
92.69:10.27
74.65#0.14
78.46-0.188
84.27:10.16
88.6410.199
Ours (w/o buffer)
Hippo Tune (ours)
16.92
87.65::021
92.07:10.25
74.85=0.17
79.92=0.22 81,12#0.34
86.63##0.41
Table 1: Comparison of Continual Learning Methods on Seq-CIFAR100, Seq Beg-lmageNet-R, and Seq-CUB200 in terms of Accuracy (Acc) and Average Accuracy Across All Tasks (AAA), along with Training Time.
Practical Guidelines
1. Ensure convergence: Spectrally normalize H or choose sufficiently small so that p(I - H)) 1.
2. Choose maaz: A small constant Tiaa Ro 2-4 balances second-order preconditioning with computational budget.
3. Early stopping: When [|g(++1) () falls below a threshold, the Krylov polynomial has effectively converged and further iterations are unnecessary.
Conclusion. The revised derivation eliminates the erroneous "product series sum" step and uses the chain rule and recursive expansion to rigorously demonstrate how multi-step recurrence implicitly implements Newton or natural-gradiens second-order preconditioning in the gradient.
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUPS
Benchmarks We evaluate on three mainstream vision continual-learinng benchmarks: Seq- et al. (2021) is randomly split by class into 10 subtasks, each with 10 categories; Seq-ImageNet--R Boschini et al. (2022) divided into subtasks of 20 classes each; Seq-CUB200 Wah et al.) (2011); Lomonaco et al. (2021) is split into 10 subtasks, each containing 20 bird species. We conduct evaluations under the cass-Inccrmental Learning.
CIFAR100 CrEARt00Kkizeevsyy (000))/Lomonaai
Compared Methods We evaluate two broad classes of methods. First, classical continual learning methods: LwFLi & Hoiem (2016) is regularization-based approach and does not use any replay buffer, while DER++ [Buzzega
DER+++Buzzeaa al. (2020) is replay-based and in our experiments maintains memory buffer of 1000 images. Second, buffer-free PEFT-CL methods, including 1.2P Wang et al. (2022b), DualPrompt Wang et al. a (2022a). CODA-Prompt Smith et al. (2023), LAE-PreT Gao et al. (2023a), and HiDe-Prompt Zuo al. (2023), freeze the backbone and train only lightweight inserted modules. We include Hippo'l Hipeo'Tune lnne in the buffer-free setting and re-implement all baselines under the same backbone and training regime, using published results for Lre-PreT and HiDe-Promp
G
Implementation Details We adopt VIT-Base/16 as the backbone Dosovitskiy al. (2021). freeze all non-PEFT parameters, and fine-tune only the key-value projection layers of the inserted prompt modules. Training uses the Adam optimizer Kingma & Ba (2015) with base learning rate of 0.01, a batch size of 128, and 5 epochs per subtask. Unless otherwise noted, modules are inserted into layers 1-7 and an orthogonality regularization coefficient of X 1 is applied. All experiments NVIDIA L40S GPUs without any replay buffer, and results are averaged over three random seeds. The detailed hyperparameter settings and code link are provided in Appendix D

---

Published as a conference paper at ICLR 2026
Method
GFLOPs
ImugeNet 5N
ImageNet R (N
ImageNet R (N 20)
AAA
Ace
Acc
Fall Tuning
16.88
64.92:10.87
7..57
60.57.11006
72.31
49.95-1.11
65.32.10.84
PEFT-CL (w/o huffer)
L.2 Wang at. 2022h
35.20
73.04::0.11
76.9440.41
71.26+0.44
76,1340.66
68.97-0.11
0..1640.32
DualPrompt Wing of al
2022a
35.38
69.99:00.57
72.24=0.41
68.22--020
73,81+0.39
65:23=0045
01,30+0,11
CODA- Prompl Small et
2023
35.84
76.63:20.27
80.30=0.28
74.05=00.41
78.14:0.39
69.38=0.33
73.95::0.63
IliDe Prompi Zuo
35.25
74.77.10.25
78.1510.24
74.65.10.14
78.46.10.18
73.59.1.:19
77.93:10.19
Ours (w/o huffer) HippoTune (ours)
16.92
77.16:00.28
81.04=0.37
74,85=00117
79.92+00.2
74.06=0.255
79.33+t0.49
Table 2: Comparison of Continual Learning Methods on ImageNet-R under different task numbers (N), including GFLOPs. Results are reported in terms of final Accuracy (Ace 1) and Average Accuracy Across All Tasks (AAA 1)-
100
100
100
Tro
1-
90
aaers 1-4
Time =3
T- 10
S 8
Time
T- 10
layara 9-12
Time
94
layers 1-7
Acersey
84
Time
T= 10
92
92
90
80
80
80
89
86
84
6
Taak 17
Than 1D
Thak 1D
Figure 3: Further analysis on Seq-CIFAR100 100 for three design choices. (Left) Moderate max iterations (c.g.. Tmax 4) balance early gains and late stability; too few/many degrade results. (Middle) Temperature T' tunes retrieval softness: mid-range (10-1) is best; extremes underperform. (Right) PEFT depth: combining shallow+middle (1-7) beats shallow (1-4), middle (5-8), or deep (9-12), highlighting multi-level memory.
5.2 EXPERIMENTAL RESULTS
In this section, we adopt Acc and AAA as evaluation metrics. Table presents the results of various methods on three benchmarks, along with their respective computational cost (GFLOPs).
Comparison with Classical CL Methods Using Replay Buffers Without relying sample replay, HippoTune achieves Ace improvements of approximately 7.4, 14.1, and 1 11.4 percentage points over LwF on Seq-CIFAR 100, Seq- Bea-llmagNee--R and Seq-C respectively. Compared to DER++ it yields gains of around 3.2, 20.6, and 3.7 points. In terms of AAA, HippoTune also outperforms LwF (+4.7%) and +ER++ (+1.9%), clearly demonstrating that the latent-space iterative retrieval mechanism can effectively suppress forgetting without any additional memory overhead.
Comparison with Other PEFT-CL Methods Compared to typical prompt-tuning methods (L2P, DualPrompt, CODA-Prompt, LAE-PreT), Tune achieves the highest performance on Seq- ImageNet-R with 74.85% Acc and 79.92% AAA. On Seq-CIFAR 100 (87.65%/92.079)) and Seq-CUB200 (81.12% wtt.12%%/6..66.. 63%), it also surpasses most PEFT-CL baselines, second only to HiDe- Prompt with 88. 88.25% 62% and 34.27% 8..., respectively. The superior performance of HiDe-Prompt on these two datasets can be largely attributed to its higher computational budget and more sophisticated multi-step prompting design. Notably, Hippo achieves better results than HiDe-Prompt on Seq- Bog-ImageNet-R (despite using only 16.92 GFLOPs compared to 35.25 GFLOPs), and delivers comparable performance on the other two and competitiveness under limited computational resources.
demonnstr---dmmmsttttttiinnnn its efficiency
Resource Efficiency and Training Speed With computational cost of only 16.92 GFLOPs. approximately half that of most mainstream PEFT-CL methods (around 35 GFLOPs), HippoTune significantly improves training speed and GPU memory efficiency. Under identical hardware
8

---

Published as a conference paper at ICLR 2026
settings, its training time is reduced by approximately 30% average, confirming its practicality for scenarios with constrained computational resources.
5.3 ABLATION STUDY
In the ablation study, constraining the number of iterative retrieval steps to just one (7 max 1) leads to notable performance drop: Acc/AAA Seq-CIFAR 100 declines from 87.65%/92.0079 to 86.81 1..1%%99.66% and on I-ggemeget--RR R from 74.85%/79.99 92% to 73.25% 2...%% See Table f for the detailed results. This highlights the critical role of multi-step retrieval in integrating historical information and mitigating forgetting. Removing orthogonality regularization has a limited effect on Seq-CIFAR 100, but results nearly 1.2-point drop AAA on the more complex Seq- Seq-ImageNet-R, indicating that maintaining the diversity of retrieved vectors is especially important for leveraging prior knowledge in challenging domains. In contrast, removing entropy regularization or adopting fusion strategy that only integrates the last-step retrieval affects overall performance by less than 0.6 points, suggesting their roles are more in stabilizing and fine-tuning the core mechanism. These findings suggest that iterative retrieval and orthogonality regularization are central to preventing catastrophic forgetting, while entropy regularization and fusion strategy can be flexibly adjusted in resource-constrained inference-sensitive settings.
5.4 FURTHER ANALYSIS
ImageNet-R under Varying Task Counts We split ImageNet-R into sequences of N 5, 10, and tasks. Hippo'Tune consistently outperforms leading prompt-basee PEFT-CL meth- ods gaingnninng around 0.5-0.8 points at N = 5 versus CODA-Prompt and 0.2-1.5 points N 10 versus HiDe-Prompt-and even in the hard- est N = 20 setting maintains similar margin. Over the range = 5-+20, its overall accuracy drops by only about 3 points, far less than typical PEFT-CL declines. Crucially, these gains come at just 16.92 GFLOPs, underscoring HippoTune's 's efficiency and resilience to forgetting. See Ta- ble for the detailed results.
Table 3: Ablation Study: Impact of Removing Individual Components of Latent Deliberation on Seq-CIFARIID 100 and Seq-ImageNet-R
Variant
Seq CIFAR 100
Sey ImageNet R
Ace
Full Method
87.65
92.07
74.85
79.92
Baseline
86.28
90.33
72.93
78.16
Orthogonality Regularization wo Entropy Regularization
87.32
91.87
74.09
78:77
87.43
91.30
74.67
79.55
w/o leerttive Retrieval (Tiaxx 1)
86.51
90.63
72.89
78.10
w/o Orthogonality Entropy oop only)
87.24
91.11
74.37
78.53
w/o Iterative Retrieval & Orthogonality Iterative Retrieval Eatropy
86.40
90:43
72.74
77.92
86.32
90.41
72.72
w/o Fusion Strategy (lust-step only)
87.27
91.25
74.13
79.04
wh Early Stopping
87.36
91.39
74.22
79.13
Impact of Iteration Length We sweep Tmax (1,3,4.5,7 on Seq-CIFAR100 (Fig. 3p. In- creasing from Tmax = to 3 yields clear gains on tasks 2-6, while Tmax 4 delivers the best overall accuracy- particulaylll mid-sequence- snndseeeeneee-mprvvinee by about 1-2 points over Tmax = 1. Larger budgets (5 or 7) offer only marginal early-task gains and actually degrade later-task performance, suggesting that excessive iterations introduce noise or redundancy. Thus, Tmax = 4 strikes the right balance between effective memory reuse and stability.
Accuracy Comparison Across All Tasks After Training Figure 4 in Appendix E,4 shows that HuppoTune consistently outperforms both DER++ and DualPrompt throughout the full task sequence. In the early tasks it gains clear lead, demonstrating its ability to recall prior knowledge immediately, This advantage persists in the mid-stage, with baseline methods trailing by a noticeable margin, and even all methods degrade in later tasks, HippoTune's 's drop is the smallest. Overall, iterative retrieval both reinforces early memories and promotes steadier performance across all ten tasks.
We swept the retrieval temperature from 10 10-3 to 10 on Seq-CIFARI00 (Fig. 3). Accuracy peaks T = 10- with mid-phase tasks (3-7) improving by 1-2 points versus T 10 and smoother convergence later. Extremes (T' = 10 or 10 10-3) degrade performance, especially in mid-to-late tasks, indicating that moderate temperature best balances
Impact of Temperature and Insertion Depth

---

Published as a conference paper at ICLR 2026
knowledge sharing and task isolation. Insertion depth experiments (Fig. compare placing the module in shallow (layers 1-4), middle (5-8), deep (9-12), or shallow+middle (1-7) blocks. The 1-7 configuration wins outperfornnn shallow-only and middle-only by 0.5-1 point and showing smaller late-task drops than deep-only. This confirms that early-layer feature retrieval plus mid-layer memory integration yields the strongest continual-leaming gains.
Model Performance in Online Continual Learning Sefting Our method remains highly effective even in the online setting with just one epoch of training. On req-CIFARIO0, it achieves 84.52% accuracy-less than 3% below the offline result- and surpasses the offline performance (epoch = 5) of some competing methods (see Appendix E.T
Effectiveness Diverse Pre-trained Backbones Experiments using DINO and SAM backbones
further demonstrate the strong generalization of Hippo'Tune. As shown in Table (see Appendix E.2) method consistently achieves superior final and average accuracy across both architectures, significantly outperforming baselines like L2P and DualPrompt. Notably, it surpasses CODA-Promp by over 4% on the SAM backbone. These results indicate that the latent iterative deliberation mechanism is architectur--ggostic and adapts well to feature distributions from diverse pre-training objectives, effectively leveraging heterogeneous representations while minimizing forgetting.
Stability and Backward Transfer Analysis As shown in Table 7 (see Appendix E.3 .iin experi- ments on ImageNet-R split into 10 tasks, HippoTune outperforms standard prompt-basee baselines in both accuracy and retention. While LoRA and adapter-based methods such as SD-Lora and EASE achieve high plasticity due to their architectural capacity, they suffer from significant catastrophic forgetting. In contrast. HippoTune maintains Forgetting Mcasure of 4.03%, which is significantly lower than the 6% to 7% range observed in these adapter variants. This demonstrates that our method offers superior stability and effectively mitigates the interference common in high-capacity adapter approaches.
Results in the lnsk-Incrrmental Setting Table 8 (see Appendix E.5 presents the comparative results under the mask-Incrementa Learning (TIL) setting. Hippo consistently outperforms the PEFT-CL baselines across both Seq-CIFAR 100 and ImageNet-R benchmarks. While existing methods like CODA-Prompt already mitigate interference by conditioning on task identities, our approach further pushes the performance boundary, achieving the highest average accuracy and the lowest forgetting measures. This superiority indicates that HlipoTTune effectively leverages task-specific contexts to refine feature representations, ensuring robust learning of new tasks without compronising the stability of previously acquired knowledge.
6 CONCLUSION
We introduced HippoTTune, hinpocamppllinsppreed continual learning method that embeds an iterative retrieval loop into each Transformer layer. By simulating the brain's multi-round associative recall and integration-combnee deepens memory access within PEFT frameworks without incurring repeated backbone passes. Our convergence analysis establishes connection to Krylov-subspace second-order preconditioning. guiding choices of iteration count, temperature, and regularization. Experimentallyy Hippo'Tune delivers buffer-free gains across visual benchmarks, outperforms prompt-pool methods, and halves PEFT-CL's computational cost. Limitations include evaluation on two-level hierarchies and image classification; future work will explore deeper loops, broader modalities, and adaptive retrieval budgets to further bridge biological memory mechanisms and scalable continual learning.
pattern separation (DG) and completion (CA3-CA))-Hipee HuppoTune
ACKNOWLEDGMENTS
This work was supported by the National Science Fund for Distinguished Young Scholars of China under Grant 62325601, and the Beijing Natural Science Foundation under Grant 1.247011.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Elahe Arani, Fahad Sarfraz, and Bahram Zonooz. Learning fast, learning slow: A general continual learning method based on complementary learning system. International Conference on Learning Representations (ICLR), 2022.
Matteo Boschini, Lorenzo Bonicelli, Angelo Porrello, Giovanni Bellitto, Matteo Pennisi, Simone Palazzo, Concetto Spampinato, and Simone Calderara. Transfer without forgetting. In European conference on computer vision, pp. 692-709. Springer, 2022.
Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: A strong, simple baseline. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Thang Doan, Mehdi Bennani, Bogdan Mazoure, Guillaume Rabusseau, and Pierre Alquier. A theoretical analysis of catastrophic forgetting through the NTK overlap matrix. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang, A unified continual learning framework with general parameter-efficient tuning. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pp. 1 11483-1 1493, 2023a.
Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang. A unified continual learning framework with general parameter-efficient tuning. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 11483-1 1493, 2023b. URL htttps://ppencceess.tto
A Unified Continual Learning Framework with General Parameter-Effhiient Tuning ICCV 2023 _at3s__pp_ehmll
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. Towards unified view of parameter-efficient transfer learning. International Conference on Learning Representations, 2022. URL
ICLR 2022 Spotlight.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning (ICML), 2019.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Confer- ence on Learning Representations (ICLR), 2022.
Ronald Kemker and Christopher Kanan. FearNet: Brain-inspired model for incremental learning. In International Conference on Learning Representations (ICLR) Workshop Track, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International
Conference on Learning Representations (ICLR), 2015.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 1 (1((1)::521-3522 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parfmeter-efffcient prompt tuning. In EMNLP, pp. 3045-3059, 2021.
11

---

Published as a conference paper at ICLR 2026
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL, pp. 4582-45855 4597, 2021.
Zhizhong Li and Derek Hoiem. Learning without forgetting. In European Conference on Computer Vision (ECCV), pp. 614-629, 2016.
Yuechen Liu, Zishun Wang, Chen Qiao, and Zongben Xu. GATE: Adaptive learning with work- ing memory by information gating in multi-lamellar hippocampal formation. arXiv preprint arXiv:2501.12611 2025.
Vincenzo Lomonaco, Lorenzo Pellegrini, Andrea Cossu, Antonio Carta, Gabriele Graffieti, Tyler Hayes, Matthias De Lange, et al. Avalanche: An end-to-end library for continual learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2021.
Arun Mallya and Svetlana Lazebnik, Packnet: Adding multiple tasks to a single network by iterative pruning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
James Martens and Roger Grosse. Optimizing neural networks with fronecker-facttored approximate curvature. In International Conference on Machine Learning (ICML), pp. 2408 2407,,2211,111 2015.
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, 24:109-16:5 -65, 1989.
Sylvestre-AIvee Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. iCaRL: Incremental classifier and representation learning, In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2017.
Yousef Saad. Iterative Methods for Sparse Linear Systems. SIAM, 2 edition, 2003.
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf
Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. CODA-Promph Continual decomposed attention-based prompting for rehearsal-free free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Alessandro Treves and Edmund T. Rolls, computational analysis of the role of the hippocampus in memory. Hippocampus, 4(3):374-391, 1994.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-201 1-001, California Institute of Technology. 2011.
Liyuan Wang, Bo Lei, Qian Li, Hang Su, Jun Zhu, and Zhong. Triple memory networks: brain- inspired method for continual learning. IEEE Transactions Neural Networks and Learning Systems, 33(5):1925-1934, 2021.
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang Chen Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Dualprompt: Complementary prompt- ing for rehearsal-free continual learning. In Proceedings of the European Conference on Computer Vision (ECCV), 2022a.
Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Plister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022b.
Michael Yassa and Craig L.. Stark. Pattern separation in the hippocampus. Trends Neuro- sciences, 34(10):515-52 2011.
Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You He. Boosting continual learning of vision- languae models via mixture-of -ffeperrss adapters. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
12

---

Published as a conference paper at ICLR 2026
Yukun Zuo, Hantao Yao, Lu Yu, Liansheng Zhuang, and Changsheng Xu. Hierarchical decomposition of prompt-based continual learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
13

---

Published as a conference paper at ICLR 2026
A DETAILED EXPLANATION OF THE PEFT-CL FRAMEWORK
A. SPECIFIC METHODS WITHIN THE PEFT-CL FRAMEWORK
L2P L2P maintains fixed pool of prompt vectors and, for each example, uses the frozen backbone feature (x) as the retrieval query. It computes the score vector
(x) K
E R
applies a hard top-k operator to obtain sparse, one-hot-like like weight g E {0, 1)m,, and concatenates the selected prompts into the input. In our PEFT-CL framework, this corresponds to Zirop ((x), g(s) = } both train and test time, and a frozen key matrix K, so that
+- h + top-k(s) AH.
DualPrompt DualPrompt extends L2P by maintaining two disjoint pools- "gees-""eneral" and "domain- specific"- and by using a one-hot teacher signal ó, during training to force selection the correct domain prompt. At train time it sets
9 (({(x)K;1) òz,
and at test time uses
s(((x)K) = arg max (ax()(x)<
both yielding a one-hot weight vector. Under our formulation, Trep (x). the branch outputs AH are mixed by these one-hot weights, and K remains frozen thus decoupling routing from the cross-entropy loss so that
h - h AH..
CoDA-Prompy CoDA-Prompt replaces hard, discrete routing with a fully differentiable soft router. Given the frozen feature (xx), it computes the same score (xx) but applies
g(s) softmax(s) A 1-1
to produce dense mixture weight over all m prompt branches. Importantly, both K and the prompt parameters AH are updated via the downstream cross-entropy loss LCE. In unified framework this corresponds to Trep = $ (x), a trainable, differentiable routing function and joint optimization of K, yielding
h h softmax(s AH.
HiDee-rompt HiDe-Prompt refines the query representation itself by learning a lightweight adapter (2x) top of the frozen backbone. It then reuses the DualPrompt routing strategy-one-he teacher 8, during training and arg max at inference-wii while keeping K frozen. Thus, our PEFT-CL notation
one sets Trep = (xx) and
(o());tt) = -d. ((σ(x)K) = arg max (())K).
so that
h - h AH,
14

---

Published as a conference paper at ICLR 2026
with all other design choices- single retrieval, hard routing, frozen key identical to DualPrompt.
A.2 INSTANTIATION OF á FOR COMMON PEFT MODULES.
For completeness, we spell out the concrete functional form of o(z; 0()) for several standard PEFT blocks, where z E Rd denotes the layer hidden state and Ah(1) = o(x; (())
Prefix Tuning. Given query projection W;()) key prefix matrix p(e) and value prefix matrix p(() the residual contributed by the i-th prefix block is
@Preetix (1; (1)) = softmax(xW1 p(") p(
(18)
Adapter. For a bottleneck adapter with down- and up-projection matrices wi() à Adapeer (2; 000) = ReLUU(WWWe down) W up
down and Wi( we have
(19)
LoRA. For low-rank LoRA block parameterized by W
down and Wil the residual update is
ΦLORA (2; ((0)
W
down w
(20)
These instantiations are all special cases of our unified notation An = ((xx0(() used in the main text.
B PSEUDOCODE
Algorithm Latent Deliberation: Iterative Retrieval and Integration
Require: Backbone depth I., max iterations Tmax, tolerance 6, temperature T, blend factor o, (optional) Top-k
1: for (= 1 to L do
2: Input: previous hidden state (1-1) e Rd
3: Initialize query: (1) +- (1--1)
4: Initialize empty list V -
5: for to Tmax do
Compute retrieval weights: S(t) + (o 7: if Top-k enabled then
6:
8:
Keep only top-k entries of (tt), zero others
9:
end if
Retrieve memory: (()) S(t) (()
11:
Append ((() to
12:
Update query: (t-11) x- a (t)) + (1 a) plo (v(1)
ift and [|[(t) (1 then break end if
14:
16:
end for
17: One-shot fusion: Vest f concat(V) E Tmax do
Compute layer output: (1) < ViT || Veat
18:
end for
C PROOF OF PROPOSITION 1
Proof of Proposition I. We regard one gradient-descent "inner" step map
0) q - (-1)) :)
15

---

Published as a conference paper at ICLR 2026
where 9 denotes outer parameters that may affect the step only through some auxiliary operation (e.g. the retrieval projection in our Latent Deliberation loop). Let q" be a fixed point the inner dynamies so that Vo(q") 0. Linearising F (q",0) gives the state Jacobian J aF Bq 4"
T- V² nH and the parameter Jacobian P
OF 3W
Step Recursion on sensitivities. Differentiating the inner recurrence w.r.t. 0 yields
0g((+1)
(7(0)) 00
00
=
+ P, t O...., maxx 1,
with the base term 0q (0)
because the initial hidden state is taken as constant outer optimisation. Solving the first-order, non-homogeneous linear recursion gives
00
Tman)
Tiaa
J.
00
k=0
Step 2: Chain rule for the outer loss. Applying the chain rule,
T
Tmax
VoLTmux
(Og
(imase)
gout
(
Jout
Σ k=0
(J bu.
k=0
where by = PT Dout. This is exactly the Krylov series operator Krmas (H) acting on by.
=kk0
Imase
(I-nH
Step 3: Convergence to the Neumann-series inverse. Because H is symmetric positive definite
and 0 < r < 2/Amax (H), the spectral radius p(J) = p(I nHH < 1. Hence the Neumann series
converges:
lim KT (H) (1 (1 J (1)
Taking the limit in the gradient expression gives VoLoo H bg, which corresponds to exact
second-order (Newton-style) preconditioning.
Step 4: Finite-step interpretation. For any finite T maxx KTnua (H) (oogree-(Tmaae 1) polynomial that approximates H in the Krylov subspace span (bg, H bg..... (H ] bg }. Because the error decays geometrically p(J). practitioners often find T Tmax 2-4 iterations already provide most of the curvature correction at only linear cost.
This completes the proof.
D EVALUATION METRICS AND HYPERPARAMETERS
Our code is available at https:/github.com/OOD00 COUTOO--OOORRRAALREPPPIIIIII and also in- cluded in the supplementary material archive.
Accuracy (Acc) For any evaluation point and task j, the accuracy is simply
Acc(i.j) di,j
i.e. the test accuracy on task j immediately after learning task i.
Average Accuracy Across Tasks (AAA) At the end of task the average accuracy across all seen tasks is
AAA(i)
Σ ai.j
In partieular, AAA(T) summarizes overall performance after the final task.
16

---

Published as a conference paper at ICLR 2026
Forgetting Measure (FM) For each earlier task its forgetting after learning up to task is
max Dt
di.j
Igi
iee. the largest drop from its best-seen aceuracy to its current accuracy. The average forgetting at end of all tasks is then
FM T
Σ f3
smaller FM indicates less catastrophic forgetting.
We introduce the hyperparameters backbone as well as those specific to our HippoTune training procedure.
by one, including key hyperparameters of the VIT
learning_ran rate Initial learning rate used by the optimizer.
batch_size Number of samples processed in each training batch.
epoch Number of full passes over the training dataset.
pretrained Whether load pretrained backbone weights (1: yes; O: no). clip_grad Maximum norm for gradient clipping to prevent exploding gradients. size Number of prompt vectors in the prompt pool.
length Length of each prompt vector (in tokens).
initializer Initialization scheme for prompt values (e.g. uni form for uniform distribution). prompt_key leals_eee____ee___eeeeeee init Initialization scheme for prompt keys (e.g. uni form). batchwise
prompt batchwise_prompt Whether share the prompt across the entire batch (1: yes; 0: per- example).
global pool Pooling method over ViT outputs: token (use class token) or avg (global average pooling).
head _appedtttp____ Input to the classification head: token, gap (global average pooling), prompt, or token+prompt.
freeze List of backbone submodules to freeze during training, c.g. [blocks, patch_embed, cls_token, norm, pos_embed].
Aorth Weight of the orthogonality regularization term on prompt keys.
layer idx Indices of Transformer layers where prompting/retrieva is applied, e.g.
10, 1, 2, 3, 4, 6].
T Softmax temperature for computing attention weights over keys.
delib steps (Tmax) Maximum number of latent deliberation (iterative retrieval) steps. a Query blending ratio each iteration: gt+1 a + (1 (x) P(w)).
eps_stop Early-stop threshold on ||v 1--1l for terminating latent deliberation.
topk Number of top keys to retain for sparse retrieval; None means full softmax.
fuse Method to combine outputs across steps: mean (average) or last.
Aent Weight for entropy regularization on the retrieval distribution.
17

---

Published as a conference paper at ICLR 2026
Dolaset
pretrained
hip__rrd
size
length
inttializer
prompt key,
batchwise prommpt
Ir
buich_aiza
epochs
Seq-CIFAAR100
1.0
uniform
uniform
0.001
Seq-Imoge-No--R
1.0
30
10
uniform
uniform
0.001
128
SuB-CUB200
1,0
30
unicorm
uniform
0.001
128
(a) Basic Training Hyperparameters
Dataset
global_ pooo
head_ head_rype
freeze
Soq-CIFAR100
[blocks, patch_ embed, ts_ko,,, pos_ cos_em [blocks, patcheemeed, cls. token, norm, pos_ embed
Sog-ImageNNt-R
token
token
Seq CUB200
[blocks, patch, embed, Joken, norm, pox, _eueed
(b) ViT-related Hyperparameters
Dataset
Aorth
layer, idx
T
delib stepal (maz))
eps__ko sps_ssop
lopk
fuse
Aent
Seq CIFARIOD
1.0
10.12233.4..66
0.01
4
0.2
le-5
5
Inean
Seq ImageNet
[0.1.2334..6]]]
Seq-CUB200
1.0
(0.12.3345566..
0.01
4
0.2
le-5
5
mean
(c) Hyperparameter Settings for Multi- Key Retrieval and Latent Deliberation
E ADDITIONAL EXPERIMENTS
E.I
ONLINE CONTINUAL LEARNING
Our method remains highly effective in the strictly online setting with single data pass: on Seq- CIFAR100, HippoTune Tune-PreTT achieves 84.52% + 0.23 Acc, 89.09% = 0.21 AAA, and 7.48 + 0.17 FM FM-only ~2.8% below its offline (5-epoch) result and clearly outperforming buffer-free baselines such as L.2P (75. 175.88% + 1.05 Acc) and DualPrompt (80.89% + 0.58 Acc) (Table On Seq-CUB200, it attains 65.99 + 0.24 Acc, 74.52 0.64 AAA, and 3.55 0.35 FM, surpassing CODA-Prompt's 62.63% + 0.34 Acc. The efficient variant Hippo'Tune-Pe Tune-PreT-EE also yields strong performance (84.0%% + 0.28 Acc, 88.62% 0.36 AAA, 8.07 0.15 FM on Seq-CIFAR 100; 64.69% + 0.53 Acc, 72.94% = 0.48 AAA, 4.19 = 0.32 FM prompt allocation (T = 1) outperforms an isolation-favee favored one (T 0.01) by ~0.8% on Seq- CIFAR100 and ~ 11.3% on Seq-CUB200, indicating that emphasizing shared knowledge significantly aids convergence the challenging one-pass online regime.
CCBB00))))), Moreover, sharing-fawe favoree
E.2 EFFECTIVENESS ON DIVERSE PRE-TRAINED BACKBONES
Experiments using DINO and SAM backbones further demonstrate the strong generalization of Hune As shown in Table 6 our method consistently achieves superior final and average accuracy across both architectures, significantly outperforming baselines like L2P and DualPrompt. Notably, it surpasses CODA-Promas by over 4% on the SAM backbone. These results indicate that the latent iterative deliberation mechanism is architecture- agnostic and adapts well to feature distributions from diverse pre-training objectives, effectively leveraging heterogeneous representations while minimizing forgetting.
E.3 STABILITY AND BACKWARD TRANSFER ANALYSIS
As shown in Table 7 in experiments ImageNet-R split into 10 tasks, HippoTTune outperforms standard prompt-based baselines in both accuracy and retention. While LoRA and adapter-based methods such as SD-Lora and EASE achieve high plasticity due to their architectural capacity, they suffer from significant catastrophic forgetting. In contrast, HippoTune maintains Forgetting Measure of 4.03%, which is significantly lower than the 6% to 7% range observed in these adapter variants. This demonstrates that our method offers superior stability and effectively mitigates the interference common in high-capacity adapter approaches.
18

---

Published as a conference paper at ICLR 2026
90
DER++
Dualprompt
HippoTune(ours)
Accuracy (%)
85
80
75
70
1
2 3 4
5
6
7
8 9 10
Task Index
Figure 4: Per-task accuracy comparison across 10 sequential tasks. We report the task-wise
Accuracy (%) of DER++, Dualprompt, and our HippoTTune method on Seq-CIFAR 100. HippoTune consistently outperforms the baselines, especially on early and late tasks, demonstrating its effective latent deliberation mechanism.
E.4 ACCURACY COMPARISON ACROSS ALL TASKS AFTER TRAINING
We provide detailed visualization of the accuracy evolution across all tasks to evaluate the stability of the model throughout the training process. As shown Figure HippoTTune consistently maintains a significant performance advantage over the baseline methods across the entire sequence.
E.5 RESULTS IN THE TASK-INCREMENTAL SETTING
Table 8 presents the comparative results under the Task-Incrementee Learning (TIL) setting. Hippo- Tune consistently outperforms the PEFT-CL baselines across both Seq-CIFAR100 and ImageNet-R benchmarks. While existing methods like CODA-Prompe already mitigate interference by condition- ing on task identities, our approach further pushes the performance boundary, achieving the highest average accuracy and the lowest forgetting measures. This superiority indicates that Hippo Tune effectively leverages task-specific contexts to refine feature representations, ensuring robust learning of new tasks without compromising the stability of previously acquired knowledge.
Table 5: Comparison of Continual Learning Methods on Seq-CIFAR 100 and Seq-CUB200 Terms of Accuracy, Average Accuracy Across All Tasks (AAA), and Forgetting Measure (FM).
Method
Seq CIFAR100
Seq-CUB200
Acc
AAA
FM
Acc
AAA
FM
PEFT-CL (w/o buffer)
L2P
75.38
84.3810.58
10.17=0.62
60.7820.42
69,21
5.37+0.23
DualPrompt
80.8920.58
86.74.0.37
10.3240.55 62.7910.27
71.25+0.53
4.87+0.25
CODA- CODA-Prompt
82.68+0.39
88.0140.46
9.9620.53 62.63+0.34
71.6340.41 4.7940.52
Ours (w/o buffer)
HippoTume Tume
84.52+0.23
89.0940.21
7.4840.17
65.9920.24
74.5240.64
3.55+0.35
19

---

Published as a conference paper at ICLR 2026
Table 6: Performance comparison of PEFT-CL methods and HippoTune on ImageNet-R (N=10) with DINO and SAM backbones (metrics: Acc, AAA, FM).
Method
ImageNet-R (N=10, Dino)
ImageNet-R (N=10, SAM)
Acc
AAA
Ace
AAA
FM
PEFT-CL
L2P
51.12 62.68
1.34
56.93
59.38
6.46
DualPrompt
60.18
66.91
4,40
65.13
70.05
5.29
CODA-Prompt
63.74
69.07
5.43
66.61
71.63
5.72
Ours
HippoTuune
65.62
70.29
3.78
70.92
75.54
5.16
Table 7: Forgetting rates (FM) and backward transfer (bwt).
Metric
12p
dualprompt
codaprompt
sdlora
case
ranpac
hippotune
FM
4.11
5.18
5.39
7.34
6.37
4.62
4.03
-4.03
-5.18
-5.26
-7.24
-6.21
-456
-3.94
Table 8: Task-Incremental Learning (TIL) performance comparison on Seq-CIFAR 100 and ImageNet- R. We report Average Accuracy (Ace), Average After Accuracy (AAA), and Forgetting Measure (FM). Best results are highlighted in bold.
Method
Seq CIFARI00
ImageNet-R
AAA
Acc
AAA
PEFT-CL
L2P
96.84 97.04
0.57
89.45
89.24
0.53
DualPrompt
97.25
97.52
0.62
87.72
88.21
0.71
CODA CODA-Prompt
97.83
98.04
0.50
89.57
89.66
0.52
Ours
HippoTTune
98.54
98.68
0.44 90.43
90.41 0.48
F THE USE OF LARGE LANGUAGE MODELS (LLMS)
In this paper, large language models (LLMs) were used only general-purpose purpose assistive tools for language polishing, improving writing structure, and retrieving and organizing references. L.LMs did not contribute to research ideation, method design, experiment execution, or result analysis, and thus do not constitute substantive scholarly contribution.
20


Reviews:
[{"review_id": "C8q7PKxiXC", "reviewer": "Reviewer_WqnZ", "summary": "This work introduces a biologically inspired approach to mitigate catastrophic forgetting in continual learning. Traditional parameter-efficient fine-tuning (PEFT) methods, such as adapters and prompts, reduce computation and storage costs but still fail to retrieve and integrate previously learned knowledge effectively. Drawing inspiration from the hippocampal EC–DG–CA3–CA1 loop in the human brain, the authors propose HippoTune, a latent deliberation mechanism that iteratively reactivates past representations to enhance memory retention during fine-tuning. In this model, each transformer layer is augmented with a small associative retrieval loop: the input state initializes a query that undergoes pattern separation (DG), auto-associative completion (CA3), and integrative fusion (CA1) across multiple iterations. This process allows the network to refine representations through recursive interaction within a learned latent space, mimicking hippocampal recall dynamics. Theoretically, the iterative update of HippoTune is shown to approximate a second-order preconditioning step in the Krylov subspace, providing curvature-aware optimization without explicit Hessian computation. The method also maintains stability across varying task numbers, demonstrating its scalability and robustness.", "questions": "In, eq5, you mentioned a layer-specific linear transforamtion fuction P, would you mind to elaborate it a littble more?\nHow is the process of eq5 can be seen as a minimal abstraction of the CA3 mechanism? Is there more previous work to enlight us on this issue?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. This work propose HippoTune, a continual learning method that bridges biological insight and machine learning theory by translating hippocampal associative memory mechanisms into a PEFT framework, offering a novel perspective on memory consolidation and retrieval in neural models for continual learning.\n2. The paper presents a relatively novel and biologically inspired analogy that grounds its internal updating mechanism in the hippocampal associative loop. This connection gives the proposed architecture a plausible neurobiological motivation, and the accompanying theoretical analysis adds a layer of methodological soundness.\n3. In addition, the partial update mechanism (Eq. 5) effectively preserves previously acquired knowledge while allowing new information to be injected into multiple parameter layers, achieving a desirable balance between plasticity and stability. Empirically, the method demonstrates strong performance across continual learning benchmarks, surpassing existing PEFT-based baselines by a meaningful margin.", "weaknesses": "However, despite the originality of its biological analogy, it should be acknowledged that the neuroscience of long-term memory formation remains far from fully understood. Thus, while the hippocampal metaphor adds interpretability, it does not constitute a rigorously biomimetic design in a scientific sense. Fundamentally, the proposed HippoTune module still functions as a multi-layer retrieval and partial parameter update process, consistent with mainstream continual learning paradigms rather than a radically new mechanism of memory consolidation. Consequently, the work’s strength lies more in its conceptual integration of biological inspiration and computational practicality than in demonstrating a genuinely new class of biologically faithful learning architecture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AGvxPfslFn", "reviewer": "Reviewer_m6em", "summary": "This paper introduces HippoTune, a continual learning approach inspired by the hippocampal memory circuit (EC–DG–CA3–CA1). Unlike traditional parameter-efficient fine-tuning (PEFT) methods that perform single-step prompt retrieval, HippoTune embeds a latent deliberation loop within each Transformer layer, enabling multi-round associative memory retrieval and feedback to alleviate catastrophic forgetting. The method integrates orthogonal and entropy regularization to stabilize learning, and theoretical analysis shows that the iterative retrieval approximates a second-order preconditioner through a Krylov subspace expansion. Experiments on Seq-CIFAR100, Seq-ImageNet-R, and Seq-CUB200 demonstrate that HippoTune achieves higher accuracy (≈80%) with roughly half the computational cost of prior PEFT-CL baselines, maintaining efficiency and strong memory retention without rehearsal buffers.", "questions": "1. What is the precise novelty over prior PEFT-CL prompt-pool methods beyond adding iterative retrieval; what can HippoTune do that they fundamentally cannot?\n2. What is the exact buffer size used for the classical continual learning baselines (e.g., LwF, DER++)?\n3. Could you clarify what the function $\\phi(x; \\theta)$ specifically represents in your formulation (line 188)?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper’s primary strength is its strong linkage between biological mechanisms and algorithmic design, modeling the hippocampal memory circuit (EC–DG–CA3–CA1) as a multi-round associative retrieval process within Transformer layers.\nThis biologically grounded framework makes the method both conceptually interpretable and practically effective in mitigating catastrophic forgetting.\n2. The authors formalize existing prompt-pool approaches as a single key–value retrieval framework and situate L2P, DualPrompt, CoDA-Prompt, and HiDe-Prompt as special cases, which clarifies trade-offs and motivates deeper, iterative retrieval.\n3. HippoTune  matches or outperforms leading buffer-free PEFT baselines on Seq-CIFAR100, Seq-ImageNet-R, and Seq-CUB200.", "weaknesses": "1. While the paper includes ablation studies, it does not clearly isolate the individual contributions of the recurrent loop, orthogonal regularization, and entropy term to the overall performance gain.\n2. The model introduces several new hyperparameters (loop depth, temperature, $\\alpha$, Top-k), yet lacks an analysis of stability or sensitivity across different settings.\n3. The writing style feels overly formulaic and AI-generated, with excessive use of em dashes and uniform sentence patterns.\n4. All studies use a frozen ViT-B/16 with only PEFT modules trained, which limits adaptability and generalization; including experiments on other architectures such as Swin Transformer as well as partially unfrozen ViT variants, would better demonstrate the method’s robustness and general applicability.\n5. The paper only reports accuracy as the evaluation metric, but does not provide any quantitative measures of forgetting, such as BWT, FWT, or average forgetting over time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IPXt7xeVvs", "reviewer": "Reviewer_RPFD", "summary": "The authors propose a computationally efficient recurrent method for continual learning in the context of parameter-efficient fine-tuning. They draw an analogy between their method and a memory recall mechanism in the hippocampus. Proving properties of the recurrence enables the authors to provide practical guidelines for hyper-parameter selection. Although results are not state of the art in all settings, they are typically very strong vis-a-vis baselines, and the approach is computationally more efficient than the most competitive baselines. The focus here on the (exhaustive) recall of previously learned tasks is refreshing and a novel perspective in the literature.", "questions": "* Is my impression that all evaluations are performed only in a task-incremental learning setting correct?\n* Can you report results in domain- or class-incremental settings in the appendix?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well written, the theoretical analysis is insightful, and practical implications of the analysis are fleshed out. The evaluation is largely thorough. Results are reported on a variety of datasets with a number of baselines. Ablation studies are performed to illustrate the contributions of the (several) components of the method. Complete code appears to have been provided, although this reviewer was not able to review the code thoroughly.", "weaknesses": "It appears to be the case that the evaluations are done using task-incremental learning [1]. Evaluations of the method in domain-incremental or class-incremental settings are not included. The provided code is perhaps too voluminous and is certainly too much for a reviewer to digest. A typical useful repository accompanying a paper enables a reviewer to inspect the implementation of the key details of the method. \n\n[1] https://www.nature.com/articles/s42256-022-00568-3", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hlbmLyRdLw", "reviewer": "Reviewer_r3mR", "summary": "Recently, several continual learning (CL) methods have emerged that utilise pre-trained models through parameter-efficient fine-tuning. One category of these methods is based on soft prompts, which involve training a set of prompts to guide the representations of each input. Typically, these methods select prompts based on a feature vector of the input, computed via an additional forward pass through the model. In this work, the authors address the inefficiency of this process by proposing a new method for selecting prompts for each layer of the model, based on the representations from the previous layer. This approach reduces the number of FLOPs during training by eliminating the need for an initial forward pass to identify prompts. Inspired by the hippocampus, they introduce HippoTune, an iterative retrieval strategy designed for transformer layers. The paper includes results presented across three visual benchmarks, along with a comprehensive ablation analysis.", "questions": "- By concatenating the values of \"v\" from each iteration (Eq. 6), a better representation of what is retrieved is achieved. Although the proposal achieves fewer FLOPs, this concatenation increases the batch size, since each batch element adds several elements to the sequence, resulting in a larger batch size than previous methods.\n    - What percentage of the batch is used by these vectors? How much can this affect scenarios where GPU size may be an issue?\n- Despite showing better results with multiple iterations (T_max >1), can you provide insight into why this improves performance?\n    - Is it related to increasing the concatenated values to v?\n- One problem with prompt-based methods is the low generalisation of vectors to datasets outside the distribution of the pre-training set.\n    - Were experiments on datasets where these types of methods fail conducted?\n    - It would be interesting to see how the proposed method would perform.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper effectively highlights a limitation of several CL methods that use soft prompts: they perform a double forward pass per example (prompt selection and classification). Using the representation of the previous layer for prompt selection is an interesting idea that has been used in other areas, but it is presented in a novel way in the CL scenario.\n- Performing multiple retrieval steps is an interesting idea, showing theoretically and empirically that it works.\n- A comprehensive ablation study of the components of the proposed method is conducted.", "weaknesses": "- The paper discusses a framework for PEFT-CL methods, but it applies only to methods that use soft prompts. LoRA-based methods do not necessarily include a retrieval component, so they cannot be extended to the framework. It would be good to make it clear that the proposed framework applies only to methods that use a key-value selection of prompts.\n    - Also, it is not clear why the proposed framework is necessary. I understand that it helps to explain the method better, but it does not necessarily help to understand the other methods better.\n- There are many hyperparameters to search over (T_max, loss coefficients, T, plus the common ones for this type of method). This can increase the method's complexity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)