Meta-UCF: Unified Task-Conditioned LoRA Generation for Continual Learning in Large Language Models
Authors: ShiLin Xiao
Tianxiang Xu
Canran Xiao
Weihao Luo
Liwei Hou
Chuangxin Zhao
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
Fine-Tuning
Continual fine-tuning
Primary area: foundation or frontier models, including LLMs

Abstract: Large language models are increasingly deployed in settings where newtasks arrive continuously, yet existing parameter-efficient finetuning (PEFT) methods either bloat linearly with the task horizon or sacrifice deep adaptation, leaving catastrophic forgetting unresolved. We aim to achieve memory-constant, on-the-fly adaptation for a frozen LLM facing an unbounded stream of tasks. To this end we propose Meta-Unified Contrastive Finetuning(Meta-UCF), which encodes each task into a lightweight layer-normalised mean embedding and feeds it to a single hypernetwork that instantly generates rank-r LoRA updates for every transformer layer; a meta-contrastive coupled with orthogonality objective further steers task embeddings into near-orthogonal directions, preserving past knowledge without inner-loop gradients. On four benchmark streams—Std-CL 5, Seq-GLUE 7, Long-CL 15 and TRACE-8—Meta-UCF raises average accuracy by up to 2.2 pp and cuts forgetting by 13% relative to the strongest LoRA baseline, while using the parameters of a single adapter. By decoupling continual learning from parameter growth, Meta-UCF provides a practical path toward scalable, low-resource lifelong language modelling.

Published as conference paper ICLR 2026
META-UCF: UNIFIED TASK-CONDITIONED LORA GEN- ERATION FOR CONTINUAL LEARNING IN LARGE LAN- GUAGE MODELS
Shilin Xiao!.5i, Tianxiang Xu Canran Xiao"; Weihao Luo", Liwei Hou", Chuangxin Zhao 1 Nanjjng University Finance & Economics, "Peking University
Shenzhen Campus of Sun Yat-sen University, *College of Science and Technology, Ningbo University
$ "Huaan Airon Tech, "Institute of Automation, Chinese Academy of Sciences
2120233029 nufe. edu 3iaocr30mal sysu. edu. en
ABSTRACT
Large language models are increasingly deployed in settings where new tasks arrive con- tinuously, yet existing aarameter-efficien finetuning (PEFT) methods either bloat linearly with the task horizon sacrifice deep adaptation, leaving catastrophic forgetting unre- solved. We aim to achieve memory-constamt, on-the-fly adaptation for a frozen LLM fac- ing an unbounded stream of tasks. To this end we propose Meta-Unified Contrastive Fine- (Mning(META-- -UFF, which encodes each task into a lightweight snyer-normalieed mean embedding and feeds dates for every transformer layer; a meta-contrastive coupled with orthogonality objective further steers task embeddings into orthogoont directions, preserving past knowl- edge without inner-loop gradients. On four benchmark streams streams-Std-CLe 5, Seq-GLUE 7, raises average accuracy by up 2.2 pp and cuts forgetting by 13 % relative to the strongest LoRA baseline, while using the parameters of a single adapter. By decoupling continual learning from parameter growth, Meta-UCF provides practical path toward scalable, low-resource lifelong language modelling.
single hypernetwork that instantly generates rank-r LoRA up-
Long-CL 15 and PRAACE-&MMea-UUS
INTRODUCTION
Large language models (LLMs) underpin modern NLP systems yet remain costly to personalise for the
continually growing set downstream tasks demanded by real applications (Yao et al. 2023 Zhu et al.)
2024 Kamath et al. 20244 Zhang et al.) 2025b Feng et al. 2026 Zhang et al.) 2025a). Updating a multi-
Dillion-paranee backbone after each task is prohibitive in compute, storage and energy (Ding Shi 2024
Jegham et al.) 2025); nevertheless, accurate and rapid adaptation without forgetting previous skills is crucial
for life-long Al agents deployed at scale(Fawi 2024 Liao et al. 2024).
adapters, aaditteess,,,,terrss prefixes and, most notably, LoRA- shrink per- task overhead from full weights to a few percent (Hu et al., 2022 Houlsby et al. 2019 Li & Liang, 2021; Zhang al. 2024). However, when tasks arrive sequentially existing variants allocate one static slot per task (Wang et al. [2023] Tiwari et al.) 2025 Yang et al. 2025), so model size still grows linearly with the horizon and subspace scheduling becomes brittle. Prompt-retrieval methods (Wang al.
Recent parameter-efficient finetuning (PEFT) techniques-
2022 Song et al. 2023 Bohao et al. 2024) avoid weight growth but leave the backbone frozen, limiting
reasoning transfer.
"Corresponding author
"hese authors contributed equally this work

---

Published as a conference paper ICLR 2026
This work probes a deeper gap in PEFT: current methods treat each incoming task as an isolated patch they either allocate a new low-rank slot or attach a prompt, leaving the backbone untouched-23 yet offer no mechanism to re- organise the knowledge already stored as the task stream grows. Consequently, model size expands linearly, and
Existing LORA-CL Methods
much
Wamiing Essukde
lem
Finctunes LLMs on streom of tasks
task interference is addressed post-hoc with orthogonality heuristies (Wang et al. 2023 Tiwari et al. 2025).
lcarning
TONEY Hare shenel bysreho
the nuris
We close this gap by reframing sequential PEFT as a gen- erative problem, and propose Meta Meta-Unified Unified Contrastive Fine-Tuning Tine-Tuning (META-UCF), Our key idea is to encode ev- ery task into a compact layer-normalised mean vector and feed it to a single hypernetwork that generates rank-r LoRA updates for all transformer layers on the fly(Fi D. A meta-contrastive objective pushes task embeddings lowards traooorthogonllity while a lightweight orthogo- nality penalty prevents their generated directions from col- lapsing. Thus frozen LLM remains both plastic- -via in- stant, conditioned updates and stable-becaute only the hypernetwork learns and its memory footprint constant.
Meto-UCF(ours)
Figure I: Existing approaches keep adding a separate adapter for every new task. Meta- UCF instead trains one shared hypernet- work that, from task embedding, gener- ates the required low-rank update on the fly-elimnaating linear parameter growth.
Our contributions are threefold: (i) We introduce task-conditioned LoRA hypernetwork with an orthogonality meta objective that eliminates linear parameter growth. (ii) We prove expressivity bounds for low-rank hypernetworks and PAC-Bayes generalisation bound over task streams. (iii) Meta- UCF achieves new state-of-the-arr accuracy and forgetting on four benchmarks while using a constant-sized model; ablations reveal robust securacy-latente trade-offs.
2 RELATED WORKS
Parameter-efficier adaptation. Early adapter modules (Houlsby et al. 2019) and prefix/p-tuning (Li & Liang 1 Liu et al. 2021) reduce finetuning cost by inserting tiny task-specific weights. LoRA pushes this idea further by applying low-rank updates directly to attention and FFN matrices (Hu et al. 2022).
recent surge of LoRA variants targets continual scenarios: O-LoRA orthogonalises task subspaces
curb
WeerferenceeWaag et al. 2023), N-LoRA re-parameterises updates to avoid collision
et al.) 2025).
while GRID (iwari et al. 2025) and Adaptive-SVD (Nayak et al.. 2025) compress adapter banks under a shared orthonormal basis. Despite strong empirical gains, these methods allocate a static slot per task, leaving memory proportional to the task horizon and requiring manual scheduling of subspaces. META- UCF replaces the slot bank with single hypernetwork that generates LoRA factors on demand, retaining the footprint of single task regardless stream length.
Continual learning for language models. Classical replay and regularisation ideas (e.g. EWC (irkkaraattrick ata.]20177 GEM(Lopez- & Ranzato (0177) LwF(Li Hoiem] (017)) have been ported transformers but scale poorly when the backbone exceeds billions of parameters. Prompt-based approaches, such as ProgPrompt and L.2(Wang et al., 2022), store small textual or embedding prompts
memory bank;
ConPET (Song et al., 2023). [Bohao al.) (224)) and Continual- Sontinual-TO(Saaae Seialio et al.) 2022) couple such prompts with contrastive objectives. Yet these frameworks depend on explicit prompt retrieval at inference time and cannot modify deeper representations, limiting accuracy reasoning-heavy streams. Meta-UCF instead learns a compact task embedding that drives low-rank updates throughout the network, yielding stronger plasticity while preserving frozen parameters.
2

---

Published as conference paper ICLR 2026
3 METHOD
Meta-Unified Contrastive Fine- Tine-nuning (Meta-UCF) tackles the continualllerninng setting in which a stream of tasks (71, . arrives sequentially and the backbone model must adapt without retaining a separate adapter for each task. Meta-UCF equips frozen LLM backbone with a single hyper-network that generates low-rank LoRA updates on the fly, conditioned on compact task embedding constructed from a small replay buffer. Figure 2 illustrates the training flow: support set drawn from replay memory is encoded by the frozen backbone and averaged to obtain a task embedding exi this embedding fed to the shared hyper-network ge. which instantly generates rank-r LoRA factors for every transformer layer; the backbone augmented with these factors processes the current task's query batch, and the joint task, orthogonality, contrastive, and bias losses vack-propaget to update & alone, leaving the backbone weights unchanged.
Incoming task T
Pretrained
Backbone
Previous Ti. T2...
Prediction
Sampling
Support Set
Query Set
LoRA
Update
(Ar, By
frozen backbone
infer
cond-Conddieoeedd
Corh
Ctask
Overall Loss
Task Embedding dx
Parameter
Backprop
Cetr
R
Generator
feedforward
learning process
frozen
Figure 2: Meta-UCF pipeline: support set from memory yiclds the task embedding eg, the generator 94 produces LoRA updates (A,, B,) for the frozen backbone, and the current task's query batch drives losses (Lusk, Luthh fer, R whose gradient updates only the generator.
3.1 TASK EMBEDDING ACQUISITION
A task embedding should (i) summarise the latent structure of the current task Tx. (ii) be stable under mini-batch sampling noise, (iii) remain parameter-free so that it can be computed on-the-fly fhyyffl deployment time, and (iv) live in the same representation space as the backbone so that geometric notions (e.g. cosine similarity) are meaningful. Formally, let the frozen backbone be a function fe, X Rdd that maps an
input x E to its CLS hidden state h = (z). Given a support set Sş = {z, 1 we would like e; = (h, StS to satisfy
8:3 with (am b) al a
sim e.s ez)
(1)
||||] |b]
so that task embeddings are approximately orthogonal across different tasks.
Layer-normalised mean pooling.
A simple yet powerful choice is the layer-normalised mean:
St
= LN
h
h = (xx):
(2)
3

---

Published as conference paper ICLR 2026
where LN denotes layer normalisation acting on the feature dimension. equation properties:
2
enjoys three favourable
1. Unbiasedness. Let jiu: = Eu (xlee, be the true task mean under the episode distribution Px- Then ([ex)=LN(Hn).
2. Variance decuy. Cov(h,]==S) then Covl + E, hs] "
J ER. i.e. the variance shrinks a rate 3%
3. Scale equivariance. Layer normalisation removes arbitrary feature-wise scale, making e insensi- tive to backbone re-scaling that may occur during pre-training.
equation can be interpreted as the first-order term in a Fisher-kernel expansion. Writing ((2; Oa) for the log-likelihood of under the frozen model, the Fisher score is g(x) Vol(z; Under lineari- sation of the backbone, g(r) is proportional to the hidden state h, hence the average S, Σ. g(x,) yields the same embedding as equation 2 up to constant. From classical theory,
K ((e,, eg) -g. g. By,
(3)
with F the Fisher information matrix, is a kernel that measures task similarity.
Streaming update. During continual training the support set grows; we therefore maintain exponential-
moving-average (EMA) estimates:
& = LN((1-1 p) (t-11) + phh"), h
Σ
(e)
B
ZEB.
with decay P (0,111 1) and mini-batch B, drawn from Sk. yielding (dd) memory overhead irrespective of Sg-
Distance-pressevinng normalisation. Finally, note that applying LN followed by l fg-normalisation projects all embeddings onto the unit hypersphere, so that
sim e₁) = = le; 1
(4)
showing that Fuclidean distance and cosine similarity coincide a useful property for contrastive objectives.
3.2 META-CONDITIONED PARAMETER GENERATOR
The core challenge in continual learning to avoid linear growth in the number of trainable parameters as new tasks arrive. Meta-UCE therefore replaces bank of per-task adapters with a single hyper-network 94 that synthesises LoRA updates for every Transformer layer on demand.
Generator architecture. Let eд Rd be the task embedding from $ 3.1 Z₁ = MLP (ex))) R²,, < d, using a two-layer MLP with GELU activation. For each layer index we retrieve learned positional embedding pa R& and concatenate: ăي.1 2zz Pt). projection heads then generate LoRA factors
We first compute task code
Two low-rank
(Ai, B,) = (reshapexx (WA...). reshape,xd W B&L.)).
(5)
where WA, WB E dr are shared across layers
"Sharing WA. We keeps [b sub-linear in while allowing layer-specific outputs via pa.
4

---

Published as conference paper ICLR 2026
LoRA injection. Given the base weight W E R2dd of layer I, the generator applies a rank-r update
w W + Bi (egs +) ( +), a
+
(6)
The scaling a follows LoRA convention so that the update norm remains comparable across different ranks.
Complexity analysis. The generator's parameters decompose as |MLP task + 2hL + 2hdr, yielding total computational cost O + + Ldr) per forward pass-inneeendent of the number of tasks K.
3.3
META OBJECTIVE
For each episodic task Ti we draw disjoint support S₁. and query Qs sets. All losses are evaluated on Qa after single hyper-network forward pass conditioned on Sy. Concretely, first sample meta-batch of K tasks and, for each task Te. draw disjoint support S₃ and query 2x (Yao et al., 2024). The frozen backbone encodes each support example æ E Se into CLS vector h E IR which layer-normalised and averaged to produce the task embedding maps ER to lower-dimensional code 2 E R*, combines it with the layer embedding Pt, and outputs rank-r LoRA factors Aş E gdxr and Bg IRd for every Transformer layer. These factors are injected into the backbone, which is run once on the query batch Q to obtain predictions and query CLS states; stacking
single meta- teaattrriinng step we
E RR defined in 3.1 The generator then
((k) the latter forms Hs IR Qx|xd on which the task loss iask- the orthogonality penalty Lorth, and the bias regulariser Rg are computed. In parallel, the set of task embeddings {ex J is fed to the contrastive loss Lotr defined below. single backward pass through this graph updates only the generator parameters p while the backbone parameters and layer- normalisomaaiiooioon statisties remain frozen.
Task Accuracy.
Σ (ar,))Ea
(Jer. v).
task
Qk
This is the standard supervised objective that drives the generated adapters to fit the labels of each episode, providing the "plasticity" needed to acquire new tasks.
Orthogonality Penalty. overlap
Let H₂ xd stack each query's 's CLS state. Define the pair-wise Frobenius
Pi 121
H
(8)
12
F
and
Lorth Σ KCJ
2
Intuitively, H2 collects the d-dimensional query representations for task Te. and Ωi measures how much the subspaces spanned by H, and H2 overlap; penalising 22 discourages different tasks from sharing the same dominant directions, improving stability by reducing cross-task interference in the adapted backbone.
Meta-Contrastive Separation. With task embeddings 2₁ Ck, the InfoNCE loss is
1
K
exp exp (im 2x )/T)
Letr
log
(10)
K
exp( 2z (2222222/// 2;)/T)²
where sim b) = a bb/([a[]
[[b]), temperature, and 2 denotes the embedding of independent support minibatch St drawn from the same task Tv as Sv, computed with the same frozen backbone and
5

---

Published as a conference paper ICLR 2026
layer-normalised mean pooling, In other words, (z,,z,) form IID "views" of the same task distribu- tion, giving a simple task-level data augmentation without introducing extra trainable modules. Once the embeddings are ég-normalised, maximising the InfoNCE objective over cosine similarities enforces angu- lar separation between tasks on the unit hypersphere, a standard and numerically stable choice in meta- contrastive learning. In Meta-UCF, Lat shapes this input geometry of the generator by keeping task codes nearly orthogonal, while Lorth regularises the oulpul geometry of the adapted backbone by discouraging overlap between the query subspaces H, and Hj; the two regularisers therefore operate at complementary levels to balance plasticity and stability.
Dynamic Bias Calibration.
For a binary sensitive attribute g {0, 1}, the demographic-pont gap is
Rg
Bza (-0
(11)
Ee- Pl 1oo
Gradients w.r.t. the generator parameters are scaled by o(-3Rx). where the sigmoid and > 0 sensitivity pepame.arameter.
Overall Loss.
K
(Clmd + ApCorth + Achetr + No,
Cmeta
(12)
k=1
Thus Lmeta remains simple episodic objective: for each task, the supervised loss encourages adaptation, the orthogonality and contrastive terms regularise the geometry of task codes and representations, and the bias term gates updates based the demographic-pan gap.
Outer-Loop Optimisation We employ first-order MAML variant with zero inner-loop gradient steps. At each iteration we (a) sample a batch of tasks, (b) construct Sg, 2s for each, (c) compute Lmeta+ and (d) update & via AdamW. Backbone parameters O and layer-norm statistics remain frozen.
Inference During deployment, small support set (S < 16) from previously unseen task is enough to produce Gnew and hence A (Cnew) without optimisation. The frozen backbone combined with the generated adapters executes the downstream prediction, enabling one-mmalel-fo--llltttsss operation with negligible memory overhead.
For the complete implementation pseudocode of Meta-UCR, please refer to Algorithm and Algorithm iin Appendix B.I
4
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Evaluation Benchmarks. To assess the performance of Meta-UCF, we conduct experiments across four distinct task sequences, following the methodology established in continual LoRA fine-tuning: (i) Std-CL A standard five-task sequence used for text classification,, encompassing tasks from AG News, Amazon, Yelp, DBpedia, and Yahoo in that order. (ii) Beq-GLUET 7: A task sequence based on the GLUE benchmark, which includes the tasks CoLA, SST-2, MRPC, QQP. QNLI, RTE, and MNLI, providing a comprehensive test of natural language understanding (NLU) transfer. (iii) Long- CL 15: A more complex sequence of 15 tasks, extending the Std-CL 5 by incorporating tasks from the GLUE, SuperGLUE, and IMDb datasets. This benchmark is evaluated with three distinct task orderings. (iv) TRACE-8: A multi-domain eight-task stream
6

---

Published as a conference paper ICLR 2026
Table I: Overall comparison. Darker shading indicates better performance.
Method
Long- CL 15 Seq-GiLE
TRACE3
Forgetting Ratio
Backward Transter
LaRA
76.3
61.4
75.9
55.6
Method
Sid-5
Long
GLUE-7
TRACES
Sid.5
Long
GLUE-7
TRACE
O-LORA
S0.1
63.4
75.8
57.3
LoRA
12.3
18.3
10.9
21.2
-18
-4.2
-1.3
-5.5
CL-LARA
82.0
64.3
76.7
59.7
O-LoRA
10.4
16.0
9%8
19.5
-1.2
-3.
-1.0
-48
Adapter
82.7
65.9
78.9
59.8
CL-LaRA
9.6
14.9
8.7
17.1
-0.6
-2.5
-0.6
-188
SAPT
83.2
ON ON-Adaaeeer
8.7
14.2
8.3
17.1
-0.7
-2.7
-0.6
-388
N-LORA
835
SAPT
7.3
13.3
73
16.0
-0.3
-2.7
-6.2
-36
1.P
50.0
62.0
75
56,0
Adaptive SVD
7.5
13.0
7.4
159
-2.2
-0.3
-3.3
SPARC
814
61.5
76.4
58.5
N-LoRA
12.4
InCL
80.1
63.2
77.6
58,8
L2P
17.6
10.0
19.7
-.0
EWC
78.8
61.6
19.5
14.3
18.1
-2.4
Replay-LoRA
80.5
63.8
77.1
EWC
17.4
13.5
20.3
MEDA LCF (-8, Top Hall)
META-UCF n=8, All)
88.6
$4.9
70.1
70.7
82.7
82.1
62.9
63.4
LwF
Replay-L.oRA
12.8
9.3
162
15.0
11.6
8.5
18.8
18.2
-2.
-11
-35
3.1
-1.2
0.9
-.3
META-UCF (m4, All
84,3
69.0
81.6
62.0
META UCF
63
11.
63
142
0.2
1.5
0.1
-3
(a) Average Accuracy (%, 1). The two Meta- MCF
(b) Forgetting Ratio (%, and Backward Transfer (BWT,
variants use fewer adapted parameters or layers under comparable budgets.
t). Lower FR and higher BWT indicate better stability.
covering areas like domain-specific question answering. multilingual comprehension, code completion, and mathematical reasoning. All datasets are transformed into the SEQ2SEQ instruction format outlined by Qin et al. (202)) and comprehensive statistics are available in the Appendix.
Evaluation Protocol. We report the standard continual-learminn metries: Average Accuracy (AA), Forget- ting Ratio (F J.R.), and Backward Transfer (BWT). For datasets with multiple metrics (e.g. accuracy & FI) we follow GRID (Tiwari et al. 2025) and average them into single score. All results averaged over three random seeds.
Baselines. We benchmark Meta-UCF against three groups of baselines in parameter-efficient continual adaptation of large language models and NLP systems: (i) adapteriparameter-effiicee
continual learning -LL-LORA ete al. 2025), TreeLoRA (Qian al. (025), OA-Adapter (Wan et al. 2025). N-LoRA Yang
et al..
2025). and SAPT (Zhao et af.] 2024).. (ii) Prompt-based continual adaptation - L2P (Wang et al.,
2022 SPARC (Jayasuriya et al.) 2025), and InsCL (Wang et al., 2024). (iii) Memory / regularisation
strategies - Elastic Weight Consolidation (EWC) (Kirkpatrick et al. 2017), Learning without Forgetting
(LwF) (Li & Hoiem/ 2017), and Replay-LoRA (Pillai 2025)..
Backbone Models. We consider four recent 7-13B checkpoints: LLAMA-3-8B, QWEN- 1.5--B. DEEPSEEK-7B, and MISTRAL-7B. rank-r LoRA adapters are inserted into every qkv and MLP pro- jection.
Optimisation Details. Unless noted, we train each task for a single epoch with Adami ( 0.98), learning-rate 3x10-5 batch 64, sequence length 512, and weight-decay 0.01. Meta-UCF regulariser weights are fixed across streams: A, 0.5, A = 1.0, A = 0.1, bias sensitivity 3= EMA decay 0.2, and support size S₁- 32. All runs fit on a single NVIDIA A100 80G; Long-CL 15 uses ZeRO-2 across four GPUs keep wall-clock under 24 h.
4.2 MAIN RESULTS
The results in Table Ta show that META-UCF delivers the highest average accuracy on all four streams, improving over the strongest prior baseline (N-LoRA) by +1.7.pp STD-CL 5 and +2.2.pp on the het- erogencous TRACE-8. Table [b further indicates that Meta-UCF not only reduces forgetting to a new low (e.g., 6.2% on STD-CL 5) but also turns backward transfer nearly neutral mildly positive, whereas all competing methods remain negative. Together, these gains confirm that task-conditioned LoRA gen- eration crabiee-commined with orthogonality and bias- meta
both superior accuracy and
7

---

Published as conference paper ICLR 2026
Table Combined results. Left: Alpaca pre-tuning effects on MMLU Std-CL Right: single-factor ablations META-UCF on Std-CL 5 and Long-CL 15.
Method
MMLU 5
Std-CL
Long-CI. 15
Variant
w/o CL
Acc. f
FR
Acc. f
FR
LLaMA-7B
34.4
Alpaca-LoRA
37.5
Full Meta-UCF
85.6
6.2
70.7
11.5
w/o Carn
83.9
7.8
68.5
13.2
Alpaca-LoRA-CL
23.3
46.7
w/o Cer
84.1
7.2
68.9
12.7
Apaca--nc-LLoRA-CC..
28.6
33.1
w/o bias calibration
84.6
6.9
69.4
12.0
Mpaca-O1oRA-CI.
33.6
76.8
CLS mean last CLS
82.1
9.5
66.3
15.1
A UCFa---
36.2
80.5
static LoRA (no generator)
80.3
11.1
64.9
17.0
(b) Singlc-factor ablations of META-UCF.
(a) Zero-shot MMLU and downstream Std-CL accu- racy after Alpaca pre-tuning.
markedly improved stability across short, long, and Nomain-diverss continual-learning streams. To ensure that the gains heterogeneous streams not driven by single domain, also compute per-task AA gaps between Meta-UCF and N-LoRA on Long-CL. 15 and TRACE-8, which are reported Appendix D.2
To test the zero-shot performance of the Meta-UCF method, follow the ALoRA(Waang) al. 2023 protocol: first, we instruction-tun tettettiionnneeeeeee LLaMA-7B on the ALPACA dataset using rank-8 LoRA, and then per- form continual training the STD-CL (order 1) stream. As Table 2a shows, META-UCF attains the highest downstream accuracy (80.5%) surpassing the strongest baseline Alpaca-O-LoRA-CL, by +3.7 pp. Crucially, it does so without sacrificing generalisation: the zero-shot MMLU score rises 36.2%. close to the single-task Alpaca-LoRA (37.5%) and considerably above all prior continual variants. These results confirm that caskiconditioned LoRA generation preserves the general knowledge acquired during Alpaca pre-tuning while providing superior resistance to catastrophic forgetting on subsequent tasks.
4.3 ABLATION STUDY
To isolate the impact of each design component, conduct single-factor ablations the two representative streams -STD-TCLLL -L and LONG-CL 15. As shown in Table 2b) removing Loth or Leu lowers accuracy by 1.1-1.9 pp and adds = 1.5 pp forgetting, evidencing their joint role in drift control. Bias calibration is less critical but still helps, especially on longer streams. Replacing the mean-pooled embedding with a single CLS vector costs 3.1 pp STD-CL. 5, and using fixed LoRA slot hurts both metries most, underscoring the need for mask-conditioned generation.
4.4 SENSITIVITY ANALYSIS
Parameter Sensitivity We vary every hyper- paramep that could plausibly influence META-UCP and measure average accuracy (mean + std over three seeds) on STD-CL 5 and LONG-CL 15. Results in Figure indicate that the method is remarkably robust: most settings fluctuate within +11 pp of the default, and no single factor dominates performance.
The generator is rank-efficient shrinking from 8 to 4 costs>>1.5 pp LONG-CU CL 15, while 16 adds no gain. Accuracy rises until S = 32 and then saturates, indicating the mean-pooled task embedding is already stable. Disabling either Lonh Lar drops accuracy by 1-2 pp. confirming both curb drift. Other knobs (p. T) move results by <0.5 pp: an oversized bias scale (3 8) slightly hurts. Thus, META-UCF stays strong across wide hyper-per parameer corridor.
8

---

Published as conference paper ICLR 2026
LoRA Rank D0)
Styport Size (5)
Ort. Weigh(
Coon, Weight (6))
Valoe
Value
Villo
EMA Devir Cal
Tempostuu Ed)
Bla Scale (5)
17.5
715
71.
750
75.0-
12.5
0.0
0.12
0.20
V0laa
ARE
Figure 3: Sensitivity of META-UCF to key hyper-parameters.
FOE SEAL
PCA SEHOLS
Larpo.12
Lorp-2.15
(a) Meta-UCF on different backbone families.
(b) Task embeddings produced by Meta-UCF.
Figure 4: (a) Backbone families vs. Meta-UCF performance; (b) Task embedding geometry. Color encodes
task ID; symbols denote stream order. t-SNE and PCA reveal well-separated, nearly orthogonal clusters.
Generalisability across Backbone Families We apply the default Meta-UCK recipe (rank-8 LoRA, iden- tical hyper-parameters) to four recent 7-13 B checkpoints: LLAMA-3-8B, QWEN-1.5-7DB DEEPSEEK-7B over three seeds; all runs fit on single A100 80 GB with identical training budgets. Across four architecturally diverse backbones, Meta-UCF delivers virtually identical accuracy and forgetting, varying by <0.5 pp on every stream This confirms that its improvements stem from the task-conditioned generator and meta-objectives rather than any model-specific quirk, and suggests practitioners can expect consistent gains when swapping in newer checkpoints without retuning byper-parameters.
and MISTRAL-7B. Figure 4a reports mean
9

---

Published as a conference paper ICLR 2026
Table 3: Dispersion statistics of task embeddings. ( cos f|): mean absolute cosine similarity (lower = better); max cos f/: worst-case overlap; S: average silhouette coefficient (higher better).
Last-CLS (abl.)
Meta-UCF (ours)
Stream
cos 01} 4 max cas 014
S1
cos 013 4 max cas 01
St
S
0.23 a 0.01
0.41 + 0.03
0.52 0.02
0.04 = 0.00
0.12 1 0.01
0.83 0.01
Long CL 15 0.28 a 0.02
0.46 +: 0.02
0.37 a 0.03
0.06 + 0.00
0.15 + 0.01
0.76 + 0.02
Geometry of Task Embeddings To verity that the layer-normalised mean ( eqquatinn] 2 indeed scatters tasks into near-orthogonal directions, we visualise the 32-dimensional embeddings learned on STD-CL 5 and LONG-CL 15. Figure 4b shows both a t-SNE and a PCA projection; Table 3 quantifies dispersion with standard geometry metrics. Meta-UCF compresses each task into compact, almost orthogonal point cloud: the mean cosine similarity drops from 0.23/0.28 to 0.04/0.06, and the silhouette coefficient rises by >0.2 on both streams (Table 3 The scatter plots in Figure corroborate this numerically- clusters are radially separated with minimal overlap--roviding direct evidence that the layer-normalised mean, combined with the meta-contrastive loss, achieves the geometric separation assumed by our objective.
Partial-Layed LoRA Injection Many production systems favour latency over marginal accuracy. We therefore inject LoRA into only a subset of transformer layers and measure the trade-off between speed, memory. and performance on LLAMA-3 rations are compared: (i) All: rank-8 LoRA in every QKV & FFN weight (default); Alt-Layrs every second layer; (iii) Top-Half: upper 50 % layers; (iv) QKV-Only: all layers, but FFN untouched; (v) Last-8: final eight layers only.
Sti-CL 5 (K))
Five configu-
85.0
Tarfarr
84.8
84.6
adapting only the upper half of layers retains > 99% of full accuracy yet raises throughput by 8%. Drop- ping FFN updates (QKV-ONLY) saves extra 5 M parameters but costs another 0.3 pp. The LAST-8 variant delivers the fastest infer- ence while losing 0.8 pp accuracy- acceptable for timeline-critical applications.
Fig. 5 reveal sweet-spot:
155 0 1525 150.0 147.5 145.0 142.5 140.0 Throughout Itokens 5t
Figure 5: Pareto curve of accuracy vs. throughput (STD-CL 5).
5 CONCLUSION
We tackled the long-standing tension between plasticity and memory footprint in continual language model adaptation by introducing META-UCF, hyperettwrk that turns compact task embedding into rank- r LoRA updates, keeping parameter count constant while preventing drift through contrastive and or- thogonality losses. Extensive benchmarks and accompanying theory jointly show that frozen LLM can match-often surpass-thh accuracy of slot-based LoRA stacks while cutting forgetting to single-digit per- centages, suggesting that mos-cconditioned generation is viable alternative to ever-growing adapter banks.
Ethics Statement This work adheres to the ICLR Code of Ethics. Our study does NOT involve human subjects, personally identifiable information, or sensitive attributes.
Reproducibility Statement We structure the paper and supplement for end-to-end reproduction. The full experimental protocol (streams, metrics, basclines, and task orders) is specified in $4.1 dataset statistics, orders, and evaluation rules appear in Appendix $C.1 All corpora converted to uni- fied SEQ2SEQ instruction format with the provided script in Appendix $C.2 We rely only on pub- lic benchmarks (Std-CL 5, Seq-GLUE 7, Long-CL 15, TRACE-8) and document tokenization details
10

---

Published as a conference paper ICLR 2026
and filtering thresholds (max sequence length 512; empty-label removal), Implementation and optimization tertings-including LoRA rank and injection points, genera- tor architecture, objective weights, EMA decay, support size S, and all optimizer knobs are enumerated in }3 and Appendix $C.4 The hyper-parametee search protocol and the chosen defaults are reported in
(Llama-33 3--------tttteeeeteee
Table 5 ZeRO-2 specifics for Long-CL are listed in Appendix $C..4 Computing infrastructure, frame- work/library versions, precision modes, and throughput are provided in Appendix CC.3 Random seeds,
determinism flags, dataloader ordering, and checkpoint verification appear in Appendix C.33 We report averages over three seeds and include the exact task orders used (matching prior work) to resolve order ef- fects. During anonymous review, we cannot release the full training code due to ongoing commercial use; upon acceptance we will (i) open-source the META-UCF reference implementation (training, evaluation, and logging), (ii) release configuration files and seed lists that regenerate every table/figure, and (iii) provide scripts that rebuild all results from the raw public datasets.
REFERENCES
PENG Bohao, Zhuotao Tian, Shu Liu, Ming-Char Yang, and Jiaya Jia. Scalable language model with generalized continual learning. In The Twelfth International Conference on Learning Representations, 2024.
Yi Ding and Tianyao Shi. Sustainable Ilm serving: Environmental implications, challenges, and opportuni- ties. In 2024 IEEE 15th International Green and Sustainable Computing Conference (IGSC), pp. 37-38. IEEE, 2024.
Muhammad Fawi. Curlora: Stable Ilm continual fine-tuning and catastrophic forgetting mitigation. arXiv preprint arXiv:2408. 14572, 2024.
Chen Feng, Minghe Shen, Ananth Balashankar, Carsten Gerner-Beuerle, and Miguel.R.D Rodrigues. Noisy but valid: Robust statistical evaluation of LLMs with imperfect judges. In The Fourteenth Interna- tional Conference on Learning Representations (ICLR), 4 2026. URL htttttt:/ net/
Eorum?id-hhhxeeeal
Gene H Golub and Charles F Van Loan. Matrix computations. JHU press, 2013.
Jiangpeng He, Zhihao Duan, and Fengqing Zhu. CI-lora: Continual low-rank adaptation for rehearsal -free class-incremennal learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 30533-330554,, 2025.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790-2799. PMLR, 2019.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Dinithi Jayasuriya, Sina Tayebati, Davide Ettori, Ranganath Krishnan, and Amit Ranjan Trivedi. Spare: Subspace-awarr prompt adaptation for robust continual learning in 1lms. arXiv preprint 2rXi:2500.002009 2025.
Nidhal Jegham, Marwen Abdelatti, Lassad Elmoubarki, and Abdeltawab Hendawi. How hungry is ai? benchmarking energy, water, and carbon footprint of Ilm inference. arXiv preprint arXiv:2505.09595 2025.
Uday Kamath, Kevin Keenan, Garrett Somers, and Sarah Sorenson. Llm adaptation and utilization. In Large Language Models: A Deep Dive: Bridging Theory and Practice, pp. 135-175. Springer, 2024.
11

---

Published as a conference paper ICLR 2026
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barankkka, et al. Overcoming catastrophic forgetting neural networks. Proceedings of the national academy of sciences, 114(13):3521-355 2017.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 2021.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40( 40(12):2935-2947) 2017.
Chonghua Liao, Ruobing Xie, Xingwu Sun, Haowen Sun, and Zhanhui Kang. Exploring forgetting in large language model pre-training, arXiv preprint 24ti::2410.170111 2024.
Xiao Liu. Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:21 10.07220 2021.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
David A McAllester. Pac-bayesian model averaging. In Proceedings of the twelfth annual conference on Computational learning theory, pp. 164-170, 1999.
Nikhil Shivakumar Nayak, Krishnateja Killamsetty, Ligong Han, Abhishek Bhandwaldar, Prateek Chanda, Kai Xu, Hao Wang. Aldo Pareja, Oleg Silkin, Mustafa Eyceoz, et al. Sculpting subspaces: Constrained full fine-tuning in iims for continual learning. arXiv preprint arXiv:2504.07097, 2025.
Sneh Pillai. Replay to remember: Retaining domain knowledge in streaming language models. arXivv preprint arXiv:2504. 17780, 2025.
Yu- Yang Qian, Yuan-Ze Xu, Zhen-Y Yu Zhang. Peng Zhao, and Zhi-Hua Zhou Treelora: Efficient continual learning via layer-wise loras guided by hierarchical gradient similariy tree. In International Conference on Machine Learning, pp. 100666-5008 PMLR, 2025.
Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. Infobench: Evaluating instruction following ability in large language models. arXiv preprint arXiv:2401.03601 2024.
Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. Fine-tuned language models are continual learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6107-6122, 2022.
Chenyang Song. Xu Han, Zheni Zeng, Kuai Li, Chen Chen, Zhiyuan Liu, Maosong Sun, and Tao Yang. Con- pet: Continual parameter-effiiient tuning for large language models. arXiv preprint urXiv:2309.14700 2023.
Anushka Tiwari, Sayantan Pal, Rohini K Srihari, and Kaiyi Ji. Task agnostii continual prompt tuning with gradient-based selection and decoding, arXiv preprint arXiv:2507. 0it.22207.14755, 2025.
Zhiyi Wan, Wanrou Du, Liang Li, Miao Pan, and Xiaoqi Qin. Budget-adaptive adapter tuning in orthogonal subspaces for continual learning in 1lms. arXiv preprint arXhv:2505.22358; 2025.
Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuan- Jing Huang. Orthogonal subspace learning for language model continual learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 10658-100 10671, 2023.
12

---

Published as a conference paper ICLR 2026
Yifan Wang, Yafei Liu, Chufan Shi, Haoling Li, Chen Chen, Haonan Lu, and Yujiu Yang. Inscl: A data- efficient continual learning paradigm for fine-tuning large language models with instructions. In Pro- ceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 663-677, 2024.
Zifeng Wang. Zizhao Zhang, Chen-Ye Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 139 149, 2022.
Shuo Yang. Kun-Peng Ning, Yu-Yang Liu, Jia-Yu Yao. Yong-Hong Tian, Yi-Bing Song, and Li Yuan. Is parameter collision hindering continual learning in 1lms? In Proceedings of the 31st International Con- ference on Computational Linguistics, pp. 4243-4259, 2025.
Jiawei Yao, Chuming Li, Keqiang Sun, Yingjie Cai, Hao Li, Wanli Ouyang, and Hongsheng Li. Nde- scene: Boost monocular 3d semantic scene completion in normalized device coordinates space. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9421-9431. IEEE Computer Soci- cty, 2023.
Jiawei Yao, Chuming Li, and Canran Xiao. Swift sampler: Efficient learning of sampler by 10 parameters. Advances in Neural Information Processing Systems, 37:59030-59053, 2024.
Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural networks, 94:103-1111 2017.
Haoting Zhang, Haoxian Chen, Donglin Zhan, Hanyang Zhao, Henry Lam, Wenpin Tang, David Yao, and Zeyu Zheng. Socrates: Simulation optimization with correlated replicas and adaptive trajectory evalua- tions. arXiv preprint arXiv:2511.00685, 2025a.
Xiaofeng Zhang, Fanshuo Zeng. Yihao Quan, Zheng Hui, and Jiawei Yao. Enhancing multimodal large language models complex reason via similarity computation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 10203-10211, 2025b.
Yunkai Zhang, Donglin Zhan, Haoting Zhang, Zuo-Jun Max Shen, Zeyu Zheng, and Qing Zhu. Does attention in transformers help wildfire prediction? Available at SSRN 4929315, 2024.
Weixiang Zhao, Shilong Wang, Yulin Hu, Yanyan Zhao, Bing Qin, Xuanyu Zhang. Qing Yang, Dongliang Xu, and Wanxiang Che. Sapt: A shared attention framework for parameter-efficient continual learning of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11641-1166 2024.
Derui Zhu, Dingfan Chen, Xiongfei Wu, Jiahui Geng, Zhuo Li, Jens Grossklags, and Lei Ma. Privauditor: Benchmarking data protection vulnerabilities in Ilm adaptation techniques. Advances in Neural Informa- tion Processing Systems, 37:9668 9689, 2024.
13

---

Published as conference paper ICLR 2026
A THEORETICAL ANALYSIS
A.1 EXPRESSIVITY OF A LORA-HYPERNET
Theorem (Expressivity of LoRA LoRA-HyperNeti, Let 9 R +R2 be a one-hidden-layer ReLU network
ge(e) (21 +b,
whose output is reshaped into ((e)).((e) with rank r <<d. Fix a Transformer layer weight W E Baxd and an embedding e.
(a) Exact realisation of any rank-r adapter. For every rank-r matrix A (A).
B**A* there exists O' such that
- g (c)
(b) Finite-width approximation. With hidden width h, one can choose 4 so that
C(d,r)
[Be))((e - A S
Vh
where C(d,r)= O(vdr))
(c) Full-rank oracle bound. For any full-rank update A fulls let A, be its best rank-r approximation. Then the same & achieves
[B(e) Amull S
|| full
Arllr + C(d.r) Vb.
Proof. Throughout we fix the embedding dimension d, target rank < d, and hidden width of the oae-hidden-layer ReLU hyper-network gæ Rd 12dr defined in $3.2. For an input embedding E Rd the network outputs vector that is reshaped into a pair (A(e),((e) with shapes d x and d respectively,
which in turn induce the rank-r LoRA update A (e) (()) E Idxd in equation 6 We prove parts (a)-() in order.
(a) Exact realisation of any rank-r adapter. Let A B' A' be an arbitrary rank-r matrix with factorisation A E R ddxr, B* E Brxd Choose hidden width h 2 2dr and split the hidden layer into two blocks of size dr each:
h = oo e +b)), h2 W (2) b()))
where (-) = ReLU(-). Set w( = and choose (()) > 0 large enough so that h, = ((1) (all activations positive), then embed vec(A directly by defining ((i) = vcc(A*). Analogously, encode B into h Because h1 hy are constant given e,
Finally set the output weight W, = [Irr Larl and bias
go (e) = (vee(A*), vec(B*)) exactly, concluding part (a).
(b) Finite-width approximation bound. Let c Rd be a compact set that contains task embeddings encountered during training and inference; in practice K. can be chosen the unit Euclidean ball since each eg is lg normalised ($3.I) Define the target mapping
e A for a fixed A E Rdddd
Because F constant on K. Lipschitz with constant O. Applying the uniform approximation theorem for ReLU networks on compacta (c.g. Yarotsky, 2017) yields, for every width h E N, parameters & such that ||o (e) vee( Il
Co/vh for all E K, where Co > 0 depends only on d and the diameter of K. Since each entry of A is approximated up Co/ VTi, summing over the d² entries gives [B(e)A(e)
|| C(d,r)/ with r(d,r) Co d O (ddr), proving (b).
14

---

Published as conference paper ICLR 2026
Technical note. The composition (B,A) BA is bilinear; the Lipschitz constant of the product map upper-bounded by m { [B||= || & |A so the preceding entry-wise bound propagates to
the full matrix product up the same order.
(c) Oracle approximation of full-rank update. Yoo---uuun--ssyyy best rank-r approximation A, arg mitl A A F. achieved by truncating the top-r singular triplets. Applying part (b) to A = A, produces parameters & such that
Let A E Rdxd be arbitrary. By
C(d,r)
[B())((e) -, I
YeEK.
Vh
Using the triangle inequality.
[B())((e - Amille & (((e)A(e) Arlle + 14, Amulle
C(d.r) Vh
A Arlle
which is the desired bound in part (c).
It is worth noting that since Meta-UCF's task embeddings layer-normalised and fz-normalised ($3.1 they lie on the unit sphere gd-1, so the compactness assumption of Theorem is exactly satisfied in our setting.
A.2 PAC-BAYES GENERALISATION
Theorem 2 (PAC-Bayes Generalisation Across a Task Stream). Consider sequence i.dd. tasks (Tk K For each task draw support set Ss. (used only to form the embedding ex) and an independent query set Dj ((E v:) hm Let the empirical and true risks of generator parameter & be
===1
T.k priin 4b) = 1
Σ (I,
ASo.. (x)
Lk ( = (( (
where é E [0, is any bounded loss. Let p))) be hyper-prior and q( the posterior returned by Meta-UCF after observing all tasks. Then, for every 8 (), with probability at least 1 - 8 over the draw of {SS, Dx)
1
KL(gli) + log
k
(g)
<
Σ
Lk
uman
K
2Km
Proef. Recall that each task Te is drawn ii.d. from an (unknown) meta-distribution т, after which we independently sample
support set Sn = (x. Js pS. used only to construct the task embedding en = e(S)) via equation 2. and
query set D (1x) (k) (k) (m))))
Pr on which the empirical loss is evaluated.
Throughout the proof we fix a bounded loss & RR x y + [o. 1]. a prior distribution p(क) over generator parameters, and let q( (b) be the posterior returned by META-UCF after observing all tasks.
15

---

Published as conference paper ICLR 2026
Step 1: Flattening the task stream. by the hierarchical process (T, y) (T) Pr v). Because tasks and examples are sampled i..d., the concatenated query sample D = D U... DDx (ر)ا : K m, is an i.id. draw of size N from P. Thus the task structure can be ignored in the PAC-Bayes analysis (see McAllester 1999 Theorem
Define the mixture distribution P over labelled examples (æ, y)
2).
Step 2: Defining the stochastic classifier. For any parameter realisation c ~ and any task embedding
Cj., the LoRA update is deterministically (eg; ②) via equation 6 and the corresponding predictor )(eut))) Because eμ depends only on Sş (which is independent of Dz), the conditional distribution of given (2, v) P is independent across all N query points. Therefore each random
A(e,)) (x), variable
()
()) A (eacn)))) (zz), (zy)) yi) E [0, j=
is bounded and i.dd when (zg Uj) P. Here ((j) maps the flat index back its task k E (1,...,KK K}.
Step 3: Applying the canonical PAC-Bayes bound. over + be
Let the empirical and true risks of a distribution Q
N
(2) 1
Σ 1=1
(2)
N
R(Q)= := (z Ex~ So. ((e
where is the embedding constructed from independent support set of the same task [ By McAllester's PAC-Bayes inequality (Thm. in McAllester 1999), for any posterior Q and any (0, 1), with probability at least over the draw of D P",
KL(Q|IP) + In
R(Q) & RN(Q)+
(13)
2NV
where P is fixed prior and KL--II)) is the Kullback-Leible divergence. Step 4: Mapping back to task-level notation.
Observe that
Rx(g)=
1
E (z) (K)) Mi (8)
Km
K (q),
L
train
and similarly R(g)= exactly the claimed bound:
K k=
Lest (q). Substituting these equalities and N Km into equation 13 yields
K
KL(glpp) + Im
test
train (g) +
K
K
2Km
k=
Step 5: No extra KL term from LoRA factors. The LoRA update ( +)) is deterministic function the sole random variable + ~ g. Hence the stochastic predictor used in the loss depends only through Φ. Consequently the divergence term in equation 13 remains K LLlll))) with no additional penalty for the crrumeter-gemeratton mechanism, matching the bound stated in the main text.
n
"Idependdncce ensures the conditional distribution of given (x, y) is identical across the population, technical requirement for the mixture flattening in Step 1.
16

---

Published as conference paper ICLR 2026
The PAC-Bayes analysis in $A.2 follows the common meta-learning assumption that tasks are drawn i..d. from meta-distributton. This can be interpreted as an average-cast justification of parameter sharing, showing that single hypernetwork can have controlled average risk K, m grow.
A.3 AUXILIARY LEMMAS AND COROLLARIES
Lemma 1 (ReLU Uniform Approximation with (h2)1/2) Rate). Let K. C. Rd be compact and ft; K R. be a constant function, (x) = IR, For every hidden width N there exists one-hidden-layer ReLU network 9h: Rd -RP with most h hidden units such that
2||c|l
sup #EK
ff
Vh
Proof. Because is constant, approximate each coordinate separately. Following Yarotsky (2017). construct by evenly partitioning K. into h axis- hyperrectangles { [ of equal volume, and assign each block the constant realised by single active ReLU neuron The pointwise error per block is zero; the only mismatch occurs at the interface measure scales like O(h - For d 2 this gives the desired (h-11/2 rate after optimising the partition aspect ratio; see Yarotsky 2017 Lem. 3.2) for details.
internal interfaces. Because has finite perimeter, the
Lemma 2 (Lipschitz Constant of the Bilinear Map). Define 4: Rddxr x Brxd + Rdddd by ( BA. Then for all (A, B),
| op & max | |r, B||r]-
Consequently, if [A|rr F. B S M on a set D, then is M-Lippchiizz over D.
Proof. For perturbations (õA, (B) one has ((A 8A, B 8B) ((A.B) BIA + ABA + 8BAA. Discarding the second-on order term and using X| || S X | F yields
|| < [ ||A ||||| |
so the operator norm of the Jacobian is bounded by max ||| [ [B[]].
Lemma 3 (Eckart Young-Minse Truncation Error). Let Arn E ddxd have singular values di 2 aa 2 0. Its best rank-r approximation under any unitarily invariant norm is
A arg min rankSr
A
Z||
achieved by keeping the top-r singular triplets. Moreover, A Arl/F = Eisr
Proof. Classical; see Golub & Van Loan 2013 Thm. 2.4.8).
Corollary 1 (Frobenius Error for Theorem (b)).
Let the settings of (ehooree hold and assume the generator Then for every E K.
weights are chosen via the construction in Lem
C(d,r)
((c)A(c)- - - <
C(d, r) = 2v dr A max-
Vh
Proof. Apply Lem coordinate-wee wise to approximate the vectorised target vec( (oc(a) Rd with sup-norm error 2|| A |l /h, then invoke Lem 2 with M ||| to translate coordinate enror to matrix-leve Frobenius error.
+A ReLU with weight vector and bias b outputs constant any bounded strictly its positive side.
17

---

Published as conference paper ICLR 2026
Lemma (KL Invariance under Deterministic Transforms). Let random variables & q and Z T(Φ) where T is deterministic. For any prior on & and the induced prior pr on Z,
KL 1p) = =Kl(gr PT),
where qr is the law of Z.
Proof. Because T' deterministic, qr is the push-forward measure of q under T, i... qr q(T- for measurable A. Using the on its image (T' acts as an identity embedding in our setting), the Radon Madon-Nikodym derivatives satisfy dg dgn о T, whence the integrals defining the two KL divergences coincide. dpr
formula and the fact that 7 is injective almost everywhere
Corollary (No Extra Complexity Penalty in Theoren[)). With notation of Theoren the stochastic pre- dictor fe,, Ale:b) induces no additional KL term beyond K L(ali) since A (e; ②) is deterministic map of d; formally,
KL ((o # (fon,((c0))) (L(allp).
Proof. Instantiated from Lem with T( Jeo.A(e::))
B SUPPLEMENTARY TECHNICAL DETAILS
B.1 PSEUDOCODE
Algorithmic overview. Algorithm I details the continual-trainee routine used by Meta-UCF. For each in- coming task Tr, the method first forms task embedding from a memory-basee support set drawn exclusively from previous tasks, by layer-normalised averaging of frozen-CI.S states. This embedding Cupp conditions a shared hyper-network, ga that instantly synthesises low-rank LoRA updates for all transformer layers of the frozen backbone. The current task's query batch is then processed once with the adapted backbone to accumulate (i) a standard prediction loss Lusks (ii) an orthogonality regulariser Conth computed from the batchwise CLS matrix to reduce inter-tast subspace overlap, and (iii) meta- contratont objective Leu that calabbaiiratioon term Rg (demographic-panle
gap) gates gradients via y of (- ). yielding the composite objective L = 7 Lunk Xo Xcheir + A Rg- Cru- cially, only the generator parameters + are updated (backhone frozen), preventing parameter growth with the number of tasks. After convergence on Tg. budgeted exemplar selection step augments the episodic memory for future conditioning.
separates task embeddings against the memory.
Inference path. Algorithm 2 shows the deployment-time procedure. Given a small support set Snew from an unseen task, Meta-UCF computes Cnew via the same layer-normalise mean pooling over frozen CLS features, feeds the trained generator g produce task-specific LoRA adapters A Anew, and performs a single forward pass of the frozen backbone augmented with Ane to obtain the prediction This enables monee--oo-all-eeeooo operation with negligible memory overhead and no test-time optimisation.
C DETAILS OF THE EXPERIMENTAL SETUP
C.I BENCHMARK STATISTICS
Notation. |Du) D |Diel denote train / dev / test sizes after filtering. "Tok." denotes the input length after BPE tokenisation with the Llama- 1llam-3--a 3-83 8B vocabulary. All corpora are lower-cased and stripped of HTML before tokenising.
18

---

Published as conference paper ICLR 2026
Algorithm 1 META-UCK CONTINUAL TRAINING
Require: Frozen backbone u generator Jw: task stream (T= episodic memory M t Ø; memory
budget max: learning rate 7): loss weights (Mos Aes Ab); bias scale B
1: for = to K do
2: while not converged task T& do
3: Ssup f SAMPLEMEMORY (.M) (support set: previous tasks}
4:
2x -- (AM)L {query set: current task]
5:
Esup LN Exes.. CIS(3;;1))
6:
A ( (Esup)
7:
H < Ø {buffer for CLS states}
8:
Lask f-0 0
9:
for all (x, (..y.9) E Qx do
10:
ÿ fe..a(2) (I)
11:
Lusk f- Lask + E(O.y)
12:
H H (CLS(1) Do A)}
13:
end for
R₂ DEMPARITYGAP(Í..) 11
15:
7 a--3Rg
16:
Fonh f- ORTHLLOSS(H)
17:
Lar + INFONCE(esup- M)
18:
4-+--- 7 frss + Xo + Xoheut + 2 Rsg.
19:
$ f- $ V&L
20:
end while
21: M < MM SELECTEXXMPLARS Th, Mmaxx
22:
end for
23:
return +
Algorithm 2 META- UCF INFERENCE
Require: Frozen backbone Oo: trained generator gΦ; support set Snew test example x
1: Enew LN [Secn Ex'eSue CLS(z';
2: Anew f- g (Coew) 3: f fe..A... (z)
4: return y
Table 4: Statistics of the four task streams used in $4
Stream
Dataset
Classes
|Drl
VDaal
DDel
Tok. (avg)
AG News
120 k
7.6 k
7.6
36
Amazon Polarity
3.60 M
200
200 k
84
Std-CL
Yelp Polarity
560 k
38
38
92
DBpedia
14
560 k
70
70
54
Yahoo Answers
10
1.40
60
60
64
CoLA
8.5 k
1k
1
(continued on next page)
Seq-GLUE? 7
19

---

Published as conference paper ICLR 2026
Stream
Dataset
Classes
[Drl
[Dval
[Dtel
Tok. (avg)
SST-2
67 k
872
1.8 k
25
MRPC
3.7 k
408
1.7
QQP
364 k
40
391
44
QNLI
105 k
5.4
5.4k
RTE
2.5k
277
3k
42
MNLI-m/mm
k93
20
20
Std-CL 5 (all)
IMDb
25 k
2k
25
110
SuperGLUU: BoolQ
9.4k
3.3
3.3
68
SuperGLUU: CB
250
56
250
70
SuperGLUE: Copa
400
40
500
41
SuperGLUE: MultiRC
2
27 K
4.5k
4.8
172
SuperGLUE: WiC
5.4
638
1.4k
16
Long-CL 15
GLUE (rest) (remaining tasks identical to Seq-GLUE 7; omilted for brevity)
see above
HotpotQA (abstr.)
90 k
5k
5k
142
XNLI-en
393 k
5k
5k
50
CodeSearch-lava
247 k
8.7k k
10.3
154
GSM8K-synth
76
4k
4k
256
TRACE-8
StackOverflow
20
119k
5k
5k
60
SciQ
11k
1.2K
824
71
WikiSQL
57 K
8k
8
116
AEDiQA-GoldB
34
875
37k
128
Task orders. The orderings used in the main experiments are identical to those in Tiwari al. (2025) for Std-CL 5 and Long-CI. v1; for Seq-GLUE 7 we follow the CoLAA - MNL. curriculum suggested by Qin et al.] (2024). The eight tasks of TRACE-8 are ordered by increasing sequence length to match the mixed-domain setting Bohao et al. (2024).
C.2 INSTRUCTION-FORMAL CONVERSION SCRIPTS
All corpora are converted a unified SEQ2SEQ template compatible with transformers' 1 ForSeg2SeqMM, Listing [ shows the core Python routine (convert.to.seq2aeg used for every dataset; only the dataset-specific buimd.prompt () function differs.
Listing 1: Minimal conversion script.
1 # //usr/bin/env python3
2 # pylint: di disable-invalle-name
3
uun
4 Convert a HP dataset into the uni fied instructi format:
5 <bos> [SYS] You are a helpful assistant. [/SYS]
6
### Input ### toriginal_text) ++# Task ### stask_description)) HH Anawer |||
7
8
9
10
11
<eos>
12
20

---

Published as conference paper ICLR 2026
from datasets import data_dat disable_caching
from pathlib import Path import msgspec, Eqdm, argparse, json
15
16
disable__achimg ng
19
def build_promppe prompt (example: dict, task_nes name: str) -> str:
20 """Task-speiffi prompt construction. """
21
--- Example: AG News ciassification
22
return (f" [SYS] You are helpful assistant. [/SYS] \n"
23
["**# Input ##NNn/cxample texxmma [ext'
24
"## Task ###\n"
25
f"Classify the news article into one of the four categories f"for the AG News \as
26
"### Answer ###")
28
29
def main (args):
30
ds Load dadd_datasetted (args.ht name, split=args.eplit, cache seche_dir-arge dir-args cache)
31
path_out Path (ath wi withhsuffii (",_fffii""mmmpacck
writer - wtw.:eec..maaacc..n.c.... Bncoder .nccode with path_out.open open
33
as fp:
34
for ex in iqdm.tqdm(ds desc="Gerialistng") prompt build prompt (ex, args.hf_name, target ex ex ex""abel"" if "label" in ex else ex("answers") [0]
35
37
fp.write (writer "prommpt"" prompt, "target"; target))) print ("Wrote", path_out)
38
39
40
if
name
"_main_".
41
p
()
42
p.add_argument ("--hf_namet) requi ("--split", detault="train" p.add argument ("--out", required-True) p.add argument ("--cache", default=""/ /"aullt="//cccch//h/t" main (p.parree__aggs
43
44
46
Tokenisation.
After conversion we tokenize the prompt field with
11ama-3 ---------------- 0. the label is left as plain text and compared via string match during evaluation.
Integrity checks. automatically discard examples whose total length exceeds max__eq meax_eeellen=111 limit whose label is empty, leading the slightly smaller sample counts in Table 4 0. % filtered).
C.3 COMPUTING INFRASTRUCTURE
Clusters. All jobs ran on an internal Slurm cluster. Most experiments fit on NVIDIA A100-80GB (PCIe) with single 32-core Intel Xeon Gold 6338 CPU. LONG-CL 15 required 4 x A100 per run (ZeRO-2, stage_of fal.fffaaeddaaaeee No CPU-only training was performed.
OS Drivers. Ubuntu 22.04.3 LTS, CUDA 12.2, cuDNN 8.9, NCCL 2.20, OpenMPI 4.1.6, Slurm 23.02.
Frameworks. PyTorch 2.3.0 + CUDA, Transformers 0.22.0, PEFT 0.10.0, bitsandbytes 0.44.2, Deepspeed 0.14.4 (for ZeRO-2), Accelerate 0.28.0.
Mixed Precision. bfloat16 autocast for all forward passes; gradient accumulation performed bíloat16 with torch. autocast.TF3. disabled to ensure cross-GPU reproducibility.
21

---

Published as conference paper ICLR 2026
Throughput. Under the default configuration (LLAMA-3-8B, rank-8 LoRA, sequence 512, batch 64), me- dian throughput was 285 samples sec on single A100-80GB.
C.4 HYPER-PARAMETER GRID AND SELECTION CRITERIA
Search protocol. For every stream uniformly sampled 20 configurations from the Cartesian product in Table S Each configuration was trained for one epoch on the first two tasks of the stream; the single-epoch dev accuracy on the second task served as proxy objective The top-3 configurations were re-rum on the full stream; the best AA was selected as default. Note that Xo Ae; Ab, 3, share one global configuration across all streams to avoid adaptive therry-pickingg (a priori values bold).
Table 5: Hyper- prrem-ereeeter grid (D=log unnioormm...mm...
Parameter
Grid Values
Default
Learning y
2e-5, 3e-S, Se-5)
3e-5
Batch size B
322, 64, 128 }
64
Rank
{ 8, 16)
Hidden dim h (MLP)
64, 128, 256
128
Weight decay
0.0, 0.01, 0.05}
0.01
Adam 81
fixed 0.9
0.9
Adam 82
fixed 0.98
0.98
Ao (orth.)
{0.25, 0.5, 1.0)
0.5
Ae (contrastive)
{0.5, 1.0. 2.0
1.0
As (bias)
(0.05, 0.2.
0.1
Bias sensitivity
{2, 8)
EMA decay
0.11, 0.2, 0.4)
0.2
Support size S
16, 32, 64
32
Temp. TSNR
0.05, 0.07. 0.1)
0.1
Max seq. len
(256, 512)
512
Each proxy trial consumed < 3 GPU-minutes on an A100; the complete search per stream therefore used < 1.5 GPU-hours.
Validation budget.
ZeRO-2 specifies.
On LONG-CL 15 we retained the same 7). B. r, h but enabled
deepspeed_stee sataeessseeee2e_e staage2gathherr___t__ gather Becther__bbitt... 1 wwwpttsteehhteeeeeeeeeeeeeeeeeet _on www_mmodee_________ save. No search over ZeRO optimiser knobs performed.
C.5 RANDOM SEED AND DETERMINISM SETTINGS
Seed pool. All tables and plots report the average over { (2,, 123, 2025 J. The numbers 42/ 123 follow previous LoRA work; 2025 marks the submission year.
Py PoTohh.
import torch, random, numpy as np, os
2 def seed_everything (s):
randon. seed (s); rander seed (s)
torch. madda..atmul..lll..
tf32 False
torch, cudnn. deterministic True
tarch.backonds. buudnn.encchart. Falsc secd_everything (SEED)
7
FFollowing
Tiwari al.
2025
found this proxy strongly correlated (r=0.87) with full-stream AA.
22

---

Published as conference paper ICLR 2026
Data order. HF datasets use shuffle files=falset we instead shuffle stateless LCG keyed by the global seed, ensuring identical batches across GPU replicas and re-runs.
Gradient noise. torch. oose.determin stic c.r.looriiiio..oo thms (True) is enabled to remove nondeterministic badäbran kernels; the resulting < 1% throughput hit is accounted for in Figure 6.
Checkpoint reproducibility. Hashes model optimiser states are logged on every save; we verified bit-wise reproducibility across two independent clusters.
The above specifications, combined with the code release in Appendix S8, allow any reader with access to comparable hardware (Z A100-40GB) reproduce META-UCF within +0.2 of the reported metrics.
D ADDITIONAL EXPERIMENTS AND RESULTS
D.1 HISTORICAL vs. CURRENT SUPPORT SETS
every meta-update we draw the 32 support examples from either (a) Historical replay memory only (HIST); (b) Current task only (CURR); or (c) 50/50 Mixed blend (MIx). sweep the buffer budget Mmax E {128, 256, 512] and report mean std over three seeds. Table [6] shows that relying only on current samples cuts accuracy by 1.8-2.3pp and increases forgetting by 2pp,, especially
the longer stream. Historical exemplars thus essential for stability, about 90 % of the benefit with half the buffer, halving extra GPU memory.
yet the MIX strategy
Std-CL
Long-CI 15
Strategy M max
Extra GPUMB
AA
FR
AA
FR
HIST
128
84.6 40.10
6.8 =0.11
69.6 s0.15
s0.15
210
HIST
256
85.6600.08
6.2 #0.10
70.7 #0.11
11.5 #0.13
420
MIX
128
84.5 20.12
7.0 =0.12
69.2
20.16
128 20.16
210
MIX
256
84.8 40,10
6.8 =0.11
69.8 20.13
12.0
420
MIX
512
85.1 40.10
=0.10
70.3 #0.12
11.4 20.13
820
CURR
N/A
83.4 10.12 8.3 #0.13 68.1 20.18
20.17
Table 6: Effect of support provenance and buffer size. AA = Average Accuracy (%, 1), FR = Forgetting
Ratio (%, 4).
D.2 FULL SEED-WISE SCORES
Tables 78
list seed-wise Average Accuracy (AA, %) and Forgetting Ratio (FR, %) for the two most competitive methods N-LoRA and META-UCE across all four task streams. The boldface row reproduces the micro-average reported in Tables and of the main paper.
Per-task AA/FR heterogeneous streams. Table 9 reports per-task average accuracy (AA) and forgetting rate (FR) N-LoRA and Meta-UCF on heterogeneous streams, together with the absolute differences AAA AFR (Meta-UCF N-LoRA). We also observe consistently lower or comparable FR across tasks, confirming that the stream-level improvements in Table are not concentrated on a single dataset or domain.
D.3 CONFIDENCE INTERVALS AND SIGNIFICANCE TESTS
95 % confidence intervals. For each metric we compute Clus 5 - 1.96 /nn with 3. Table intervals for the AA metric.
10 lists the
23

---

Published as conference paper ICLR 2026
Table 7: feed-wise Average Accuracy (higher = better).
Seed
Mean
2025
Sid-C
N-LoRA Mcta-UCF
83.3
83.7
83.5
83.5
85.1
85.3
85.6
85.6
N-LoRA
68.5
67.8
68.1
Long-CL 15
Meta UCF
70.2
70.7
70.3
70.7
SLq-7LUE7
N-LoRA
80.0
80.3
80.2
80.2
Meta-UCF
82.3
82.5
82.3
82.7
N-LORA
60.9
61.2
60.8
61.0
TRACE-8
63.4
Table 8: Seed-wise Forgetting Ratio (lower better).
Seed
Stream
Method
Mean
123
2025
N-LoRA
7.0
7.2
7.1
7.1
Std-CL
Meta-UCF
6.3
6.1
6.2
6.2
Long-CL 15
N-LoRA
12.1
12.4
Meta-UCF
11.6
11.4
11.5
11.5
7
N-LoRA
7.0
7.3
7.0
7.1
Mcta-UCF
6.4
6.2
6.3
6.3
N-LoRA
15.6
15.4
15.5
15.5
TRACE-8
Meta UCF
14.1
14.3
14.2
14.2
Table 10: confidence intervals (AA, %). Parenthesised numbers show + half-width.
Stream
N-LoRA
Meta-UCF
Std-CL
83.5 +0.16
85.6 +0.10
Long-CI. 15
68.1 +0.24
70.7 +0.14
Seq-GLUE
80,2 10.11
82.7 10.10
TRACE-8
61.0 +0.23
63.4 10.15
Wilcoxon signed-rank tests. Following Tiwari al. (2225) we compare per-task accuracies of Meta-UCS against N-LoRA using a two-sided Wilcoxon tes (a 0.05). Results in Table show that Meta-UCF significantly
outperforms N-LoRA on three streams and ties on SEQ-GLUE 7. All p-values are Holm-corrected over four compar- isons.
Table 11: Wilcoxon signed-rank p-values (Meta-UCF N-LoRA, AA per task).
Stream
p-value (L)
Std-CL
0.031
Long-Cl CL
0.008
Seq-GLUE 7
0.087
TRACE-8
0.012
Paared by task, aggregated across all three seeds.
24

---

Published as conference paper ICLR 2026
Table 9: Per-task AA and FR on heterogeneous streams.
AA (%)
FR (%)
Stream
Task
N-LoRA
Mcta-UCF
AAA
N-LoRA
Mcta-UCF
AFR
Long-Cl Sag-CC..1. 15
Task1
67.2
68.9
+1.7
13.6
12.2
1.4
Long-CL 15
Taska
68.7
71.3
+2.6
13.3
11.5
1.8
Long-CL 15
Taska
67.8
70.3
+2.5
13.0
11.3
117
Long-CI.
Task,
67.4
69.6
+2.2
13.4
11.8
-1.6
Long-CI.
Tasks
71.6
+2.7
-20
Long-CL
Taska
13.2
-1.5
Long-CI. 15
Task;
68.1
70.1
+2.0
12.9
11.3
-1.6
Long-CI. 15
Tasks
67.8
70.1
+2.3
13.1
11.8
- -133
Long-CL 15
Taska
68.3
70.7
+2.4
12.8
11.0
-1.8
Long-CI.
Taskjo
67.7
69.8
+2.1
13.3
11.7
-1.6
Long-CL
Taski
67.0
68.2
+1.2
13.5
Long-CL
Task12
72.2
13.4
-21
Long-CL. 15
Task1a
68.6
71.5
+2.9
13.0
11.3
-1.7
Long-CL 15
Task14
67.9
70.2
+2.33
13.1
11.5
1.6
Long-CL 15
Taskis
68.8
71.8
+3.0
13.2
11.4
-18
TRACE-8
Task,
78.1
80.3
+2.2
8.4
6,9
- -.5
TRACE-8
Task2
78.5
81.1
+2.66
8.5
6.8
-177
TRACE-8
Tasks
78.0
79.8
+1.8
8.1
6.8
-13
TRACE-8
Task,
78.4
80.8
8.3
6.7
-1.6
TRACE-8
Tasks
78.6
81.5
8.6
6.8
TRACE-8
Tasks
78.2
79.6
+1.4
8.2
6.8
-1.4
TRACE-8
Task,
78.3
80.6
+2.3
8.4
6,9
-155
TRACE-8
Tasks
78.1
80.1
+2.00
8.3
6.7
-16
D.4 PARTIAL-LAYER LORA INJECTION: ACCURACY-LATENT TRADE-OFF
Each configuration was run on LLAMA-3-8B with the STD-CL 5 stream; throughput is measured on single A 100-80G with sequence length 512 and batch 64, The bascline ("All") inserts rank-8 LoRA into every qkv and MLP projection, yielding 14.2M trainable parameters. We find that:
Top-Half adapters retain 99% of baseline accuracy while halving parameter count and gaining +8% throughput.
Last Layers achieve the fastest inference (+11%) with a modest 0.8pp accuracy drop useful for latency- critical deployments.
Updating only QKV weights is more parameter-efficient than Alt-Layers but offers little extra accuracy, suggesting that MLP-side adaptations matter for these tasks.
Table 12: Accuracy vs. throughput for selective LoRA injection.
Scheme
#Params (M)
AParams Throughput Speed-up AA( %)
All-Layers
14.2
285sps
85.6
Alt-Layers
7.1
-50%
301
+5.6%
84,9
Top-Half
7.1
-50%
309sps
+8.4%
84.7
QK QKV-Only
-35%
314sps
+10.2%
84.4
Last Layers
-75%
317sps
+11.2%
84.4
25

---

Published as conference paper ICLR 2026
D.5 DRDER-SENSITIVITY ANALYSIS
To assess robustness of Meta-UCF task ordering, we evaluate Meta-UCF and N-LoRA under multiple alternative permutations of the benchmark streams. For Std-CL. 5, we consider the canonical order (vl), permuted order that swaps the Amazon and Yahoo tasks, and fully reversed order. For Scq-GLUE 7, compare the canonical curriculum against permutation that front-loads MNLI and RTE, For Long-CL 15, follow the official vl and v2 orders released with the benchmark. For TRACE-8, we compare the canonical order with random permutation of tasks. Table reports the average accuracy and forgetting ratio for both methods.
Table 13: sensitivis analysis for Meta- UCFF and N-LoRA.
Stream
Order
Method
AA f
FR
AAA (Meta-N)
AFR (Meta-N)
5
canonical (v1)
N-LoRA
83.5
7.1
Meta-UCF
85.6
6.2
+1.7
-0.9
5
permuted (Amazone> Yahoo)
N-LoRA
83.3
7.6
Meta-UCF
84.9
6.0
+1.6
-1.6
Std-CL 5
reversed
N-LoRA
83.1
7.4
Meta-UCF
84.7
6.4
+1.6
-1.0
Scq-GLUE 7
canonical
N-LoRA
80.2
7.1
Mcta-UCF
82.7
6.3
+2.2
-0.8
Scq-GLUE7
permuted (MNLI/RTE front)
N-LoRA
80.0
7.4
Mcta-UCF
82.1
6.6
+2.1
-0.8
Long-Cla BlggCCL 15
canonical (vl)
N-LoRA
68.1
12.4
Meta-UCF
70.7
11.5
+2.3
-09
Long-CLi 15
official v2
N-LoRA
67.9
12.7
Mct
70.1
10.9
+2.2
-1.8
TRACE-8
canonical
N-LoRA
61.0
15.5
Meta-UCF
63.4
14.2
+2.2
-13
TRACE-8
random permutation
N-LoRA
15.9
Meta-UCF
63.0
14.5
+2.2
-1.4
Across all four streams and eight alternative task orders, Meta-UCF consistently outperforms N-LoRA: AA gains are stable in the range of approximately 1.6 to +2.3 pereentage points, while FR is reduced by about 0.8 to 1.8 percentage points. This suggests that the advantages of Meta-UCF not tied particular task curriculum, but persist under natural variations the order which tasks are presented.
D.6 JOINT GEOMETRY OF TASK EMBEDDINGS AND QUERY SUBSPACES
To make the roles of Lar and Lonh more concrete, analyse how task-code similarity and query-subspace overlap are related in practice. Recall that ex the layer-normalised, fz-normalised task embedding built from the support set (..1) and HA Rle.Ixd stacks the adapted CLS states the query set ($3.). For each task pair (i, on STD-CL
5, we compute:
the absolute task-code similarity cos 0ayl | (e,,, e,)l.
query-subspace overlap Puy = |el|, |H, 1|.
We report these statistics both Last-CLS ablation (where ex is single frozen CLS vector) and Meta-UCF.
Observations. On STD-CL 5, the Last-CLS ablation yiclds task-code similarities the range cos(e,, e,)l 10.14, 0.24] and overlaps 94, E 00.25.0331 with moderate correlation z 0.66 between the two. Under Meta- UCF, task codes substantially more dispersed the unit sphere: most pairs have cos(e,, e,) < 0.06, while query overlaps drop 2. E [0.11, 0.19). The correlation between cos((e;, E5) and Qy remains only moderately strong
26

---

Published as conference paper ICLR 2026
(query subspace overlap)
0.25
0.20
0.15
AG Arrwznn
Whweon Yahoo
Last: (p-0.66)
Amazon-Velo
Mota-UCF (p-0.78)
0.00
0.05
0,10
0.15
0.20
0.25
ccoste, e/ll (task-code similarity)
Figure 6: Joint geometry of task codes and query subspaces on STD-CL 5. Each point corresponds to a task pair (i,5), plotting task-code similarity cos(es; ej) on the æ-axis and query- query-subspace overlap S4 on the y-axis.
(p Re 0.78) and far from deterministic: several pairs exhihit very small task-cnde similarity ( (cos(es, e,]] 0.02) but still show noticeable overlap (M.y ze 0.15). This empirically supports the design choice that Ler and Zomn are not redundant: Lar shapes the imput geometry of task codes fed to the generator, while Loon directly regularises the output geometry of adapted query representations curb residual interference.
E
LLM USAGE
We used large language model for minor English editing (grammaa/worrting/laritt))) and small, localized code fixes (e.g., resolving syntax errors, adding missing imports). LLM not contribute to research ideation, experimental design, data processing, analysis, or figure generation. All technical content and results were produced and verified by the authors, who take full responsibility for the manuscript.
27


Reviews:
[{"review_id": "QKIVEE3e6V", "reviewer": "Reviewer_JLyJ", "summary": "The paper presents META-UCF, a new LoRA-based adaptation module intended to improve generalization across tasks while keeping parameter growth small, and reports strong results against diverse baselines.  I basically creates \"add-ons\" for each new task instead of storing a separate add-on bank per task. Training encourages tasks to stay distinct, which helps the model remember old skills while learning new ones. Across several benchmarks, this approach nudges accuracy up and reduces forgetting compared to strong baselines. The gains hold under different settings, suggesting the method is fairly robust rather than overly tuned.", "questions": "See Weakness.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "S1. The paper's idea is simple and well motivated. The figures, though simple, are intuitive and convey the point well. \\\nS2. Consistent gains in AA with lower forgetting across four continual streams versus strong LoRA and prompt baselines.  \\\nS3. The method has the potential to be the next \"SOTA\" with some fixes in writing.", "weaknesses": "W1. Section 3.3 clarity & missing intuition. The meta-objective is presented mostly as equations without enough narrative to make the design choices legible (e.g., how the positive view ​is constructed, why InfoNCE is preferred here, and how the orthogonality penalty interacts with the contrastive term during optimisation). The paper defines multiple notations (omega) and aggregates losses but gives little intuition for stability/plasticity trade-offs beyond symbol lists. Even a short paragraph walking through one training step with concrete shapes would help. As written, 3.3 is hard to follow for readers not already steeped in contrastive notation.\n\nW2. Experimental depth & statistical evidence. Main results are reported as averaged metrics per stream (AA/FR/BWT), but the tables omit confidence intervals or statistical tests against strong baselines; the text only notes averaging over three seeds, which is thin for claims of \"state-of-the-art\". Per-dataset/task breakdowns (especially for TRACE-8’s diverse domains) and order-sensitivity analyses are absent from the main paper, limiting interpretability of where gains come from. Without significance testing, improvements are hard to judge.\n\nW3. Compute/memory reporting is not actionable. While the method argues constant-size adapters and gives a big-O generator complexity plus one throughput as compared to last layers vs all, the paper lacks concrete parameter counts for the hypernetwork vs adapter banks, end-to-end training wall-clock vs baselines, and detailed memory footprints during training/inference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G5aI8b2StF", "reviewer": "Reviewer_KhxK", "summary": "Meta-UCF proposes a unified continual learning method for LLMs that uses a single hypernetwork to generate task-specific LoRA parameters from lightweight task embeddings derived from a small support set, without inner-loop updates or replay; task interference is mitigated via contrastive separation and orthogonality constraints in the embedding space, while keeping the backbone frozen and trainable parameters fixed.", "questions": "Although this paper addresses continual learning, I am curious, following the discussion in [1], about how the proposed method performs on standard LLM tasks beyond continual learning, such as mathematical reasoning, code generation, and instruction following.\n\n[1] HFT: Half Fine-Tuning for Large Language Models", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Meta-UCF maintains a fixed set of trainable parameters regardless of the number of tasks, eliminating the need for task-specific adapters or replay buffers.\n\n2. It effectively reduces task interference through contrastive learning and orthogonality constraints on task embeddings, without requiring inner-loop adaptation or data replay.", "weaknesses": "1. While parameter efficiency is emphasized, the paper omits analysis of the whole cost introduced by the hypernetwork and embedding generation pipeline, which is crucial for real-world deployment.\n\n2. The paper’s formatting is inconsistent and unclear, with poorly structured tables and ambiguous notation that reduce readability.\n\n3. Experiments assume relatively benign task sequences, with no analysis of performance under extreme domain shifts or adversarial task orderings that better stress-test continual learning robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ptzk8ZQ1VD", "reviewer": "Reviewer_TwVA", "summary": "This paper introduces Meta-Unified Contrastive Fine-Tuning, a novel approach to continual learning in LLMs. The method addresses the challenges of catastrophic forgetting and parameter growth by proposing a task-conditioned hypernetwork that generates low-rank LoRA updates dynamically for each task. Meta-UCF incorporates a meta-contrastive objective and orthogonality constraints to ensure task embeddings are near-orthogonal, preserving past knowledge without requiring additional adapters for new tasks. Experimental results demonstrate state-of-the-art performance across four benchmarks, with improvements in accuracy and reductions in forgetting compared to prior methods. The approach is computationally efficient, scalable, and adaptable to diverse backbone architectures.", "questions": "See weakness", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The task-conditioned hypernetwork and the integration of meta-contrastive and orthogonality objectives are innovative contributions that address key limitations of existing methods.\nThe experimental results are comprehensive, covering multiple benchmarks and baselines. The inclusion of theoretical analysis (e.g., expressivity bounds, PAC-Bayes generalization) adds depth to the work.\nThe paper is well-organized, with clear explanations of the methodology, experiments, and findings.", "weaknesses": "While the benchmarks are diverse, the paper does not evaluate Meta-UCF on real-world deployment scenarios (e.g., latency-critical applications or domain-specific tasks).\nAlthough the sensitivity analysis shows robustness, the reliance on specific hyperparameter configurations (e.g., orthogonality and contrastive weights) may pose challenges for practitioners.\nThe theoretical analysis assumes idealized conditions (e.g., compact task embeddings, independent task distributions), which may not fully capture the complexities of real-world task streams.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dHAYHFog9b", "reviewer": "Reviewer_ehuo", "summary": "This paper proposes a continual learning framework named Meta-UCF. Its core idea is to utilize a hypernetwork to dynamically generate LoRA parameters for new tasks, conditioned on a task embedding derived from a small support set. The method aims to solve the problem of linear parameter growth with the number of tasks found in existing methods, and it combines contrastive and orthogonality losses to maintain model performance while mitigating catastrophic forgetting.", "questions": "Please see \"weaknesses\".", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Meta-UCF generates task parameters via a single, shared hypernetwork, which keeps the total model parameter count constant as tasks increase. Compared to methods that add new modules for each task, this is an advantage in terms of scalability, especially for scenarios requiring continuous learning of new tasks.\n\n2. The paper conducts experiments under various task sequence settings (short, long, and heterogeneous sequences) and compares against multiple baselines. The ablation studies and sensitivity analyses are also quite thorough, validating the effectiveness of the framework's components.", "weaknesses": "1. The \"on-the-fly\" generation of LoRA parameters mentioned in the paper implies that a forward pass through the hypernetwork is required before processing a new task. This introduces additional inference overhead. The paper lacks a latency comparison against methods that load static LoRA modules. Without this data, it is difficult to assess the practical cost of the framework.\n\n2. The framework claims to solve the memory growth problem, but this primarily refers to model parameters. The algorithm itself relies on a replay buffer to construct embedding vectors for old tasks. The size of this buffer either grows with tasks or is constrained by a fixed budget, which somewhat undermines the claim of \"constant memory\"。 \n\n3. The entire adaptation process relies on the task embedding extracted from a small number of samples (32 by default). If these samples are not representative, the resulting task embedding will be biased, which in turn will affect the model's performance on that task. This is a potential point of failure, but the paper lacks a discussion of its robustness.\n\n4. The final loss function includes multiple weighting hyperparameters. Although the paper states that these parameters are kept fixed across all experiments, determining these values is a non-trivial challenge in itself. A discussion on the difficulty and sensitivity of tuning these hyperparameters is recommended.\n\n5. According to the ablation study (Table 2b), removing the dynamic bias calibration module results in only a 0.6% drop in accuracy on Std-CL 5. The improvement from this module is limited. The paper needs to provide a stronger motivation for its inclusion or demonstrate its value on a dataset known to require bias calibration.\n\n6. The orthogonality loss (L_orth) is calculated on the CLS hidden states from the query set. The paper should explain why the query set was chosen over the support set and how this loss, which acts on output representations, works in conjunction with the contrastive loss (L_ctr) that acts on task embeddings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)