Exploiting Task Relationships in Continual Learning via Transferability-Aware Task Embeddings
Authors: Yanru Wu
Jianning Wang
Xiangyu Chen
Enming Zhang
Yang Tan
Hanbing Liu
Yang Li
Venue: NeurIPS 2025 poster ()
Keywords: Continual learning
Transferability
Hypernetworks
Primary area: general_machine_learning
TL;DR: With an intention to better exploit the task relationships for continual learning, we propose a transferability-based task embedding named H-embedding and present a hypernet under its guidance.

Abstract: Continual learning (CL) has been a critical topic in contemporary deep neural network applications, where higher levels of both forward and backward transfer are desirable for an effective CL performance. Existing CL strategies primarily focus on task models — either by regularizing model updates or by separating task-specific and shared components — while often overlooking the potential of leveraging inter-task relationships to enhance transfer. To address this gap, we propose a transferability-aware task embedding, termed H-embedding, and construct a hypernet framework under its guidance to learn task-conditioned model weights for CL tasks. Specifically, H-embedding is derived from an information theoretic measure of transferability and is designed to be online and easy to compute. Our method is also characterized by notable practicality, requiring only the storage of a low-dimensional task embedding per task and supporting efficient end-to-end training. Extensive evaluations on benchmarks including CIFAR-100, ImageNet-R, and DomainNet show that our framework performs prominently compared to various baseline and SOTA approaches, demonstrating strong potential in capturing and utilizing intrinsic task relationships. Our code is publicly available at \url{https://github.com/viki760/Hembedding_Guided_Hypernet}.

Exploiting Task Relationships in Continual Learning via Transferability-Awan Task Embeddings
Yanru Wu!
Jianning Wang²
Xiangyu Chen Enming Zhang'
Hanbing Liu'
Yang Tan?
Yang Li
Shenzhen International Graduate School, Tsinghua University
2 Indeeendent Researcher
(wu-yr21, xy-c21, zem24, (achb24)@maille www.mmatttiinnuuaaa..aaa. edu. cn
jianning. wang@outlook. com tanyang12310163 com
yangli@ez.tsinghuu. tsinghua. edu.
Abstract
Continual learning (CL) has been a critical topic in contemporary deep neural network applications, where higher levels of both forward and backward transfer are desirable for an effective CL performance. Existing CL strategies primarily focus on task models either by regularizing model updates or by separating task-specific and shared components while often overlooking the potential leveraging inter-task relationships to enhance transfer. To address this gap, propose a transfcrability-ane construct a hypernet framework under its guidance to learn coss-conditioned model weights for CL tasks. Specifically, H-embedding is derived from an information theoretic measure of transferability and designed to be online and easy to compute. Our method is also characterized by notable practicality, requiring only the storage a low-dimensional task embedding per task and supporting efficient end-to-end training. Extensive evaluations R, and DomainNet show that our framework performs prominently compared to various baseline and SOTA approaches, demonstrating strong potential in capturing and utilizing intrinsic task relationships. Our code is publicly available at attt:s
task embedding, termed H H-embedding, and
benchmarks including CIFAR- 100, ImageNet--
//github. http://iiiii66///edeeene
Introduction
Continual learning (CL), also referred to incremental learning life-long learning, has become an essential topic in the modern application of deep neural networks. In CL settings, model is expected to sequentially learn a series of tasks (typically involving either category shifts or data
distribution changes QQu et al.] 202T) ) to progressively enhance its capabilities (Wang et al. [2024]. key practical challenge in this process catastrophic forgetting (CF) [Kirkpatrick et al. [017], phenomenon where learning new task undermines the knowledge acquired from previous ones.
The mitigation of CF has long been a primary focus in CL research (Wagg et al. 2024 due its severe hindrance to the overall accumulation of model capacity. Existing approaches can be broadly categorized into three classes: rehearsal-basee methods (e.g.. ER (Robins) 1995), DER [Buzzega et al. )020)) maintain a memory buffer of previous samples for later replay; engulariaation-aseee
"Corresponding Author
Yanru Wu, Xiangyu Chen, Enming Zhang, Hanbing Liu, and Yang Li are affiliated with the Shenzhen Key Laboratory Ubiquitous Data Enabling, SIGS, Tsinghua.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

methods (e.g., LwF [i and Hoiem 2017], SI [Znke et al., 2017D constrain parameter updates to protect prior knowledge; and architecture-baat based methods (c.g., PackNet [Mallya and Lazebnik, 2 WSN [Kang et al.) (022]) allocate task-specific and task-sharing components of the model from different architecture levels.
However, strategies that explicitly alleviate forgetting often compromise the acquisition of new knowledge, prioritizing retention over adaptability. In contrast, an effective CL system should facilitate the intrinsic knowledge transfer across all tasks, improving future task performance via forward transfer. and preserving (or even enhancing) prior task performance via backward transfer (von Oswald et al. 2020]. Moreover, most existing works adopt model-centrie perspective ! relying on posterior information obtained during training to manage cross cross-task interactions. Yet in CL settings, task sequences often exhibit certain degrees of noise and diversity, making such approaches sensitive to sample- and task-level variations. This can lead to potential negative transfer, reduced robustness, and limited scalability, particularly as the number or complexity of tasks increases.
These observations point to fundamental question in continual learning: Robust and positive knowledge transfers across tasks rely on capturing the underlying task relationships, but how do we efficiently learn and utilize such relations in CL settings? Despite the valuable contributions of previous explorations [e.g. Jin and Kim 2022 Riemer el al., 2018], they mostly remain constrained by a posterior model-centric paradigm with associated drawbacks, leaving the prior exploitation of task relations largely underexplored. Based on this insight, we argue that the incorporation of prior, lusk-relation-wwe -reeae guidance can effectively leverage task relation information and help mitigate instability and performance degradation in CL., especially over long or challenging task sequences.
Therefore, in this work we explicitly bring focus to the prior exploitation of task relationships by intro- ducing a statistically grounded, task-relation-aware embedding, and propose a CL framework under its guidance. Specifically, recognizing the shared goal between identifying inter-task relationships and the role of transferability metrics [Ding et al. 2024] in evaluating source-target task compatibility, we propose an online embedding scheme named H-embedding, which distills task-level transferability into a -ww-dimeniionl representation through optimization before task training. H H-embedding can be learned efficiently without revisiting previous samples by maximizing the consistency between the Euclidean distances among embeddings and the H-score transferability [Bao et al. 2019) among corresponding tasks. To ensure alignment between embedding distances and transferability scores, we apply analytic hierarchy process (AHP) normalization, which significantly improves learning stability over long task sequences while maintaining efficiency. Built upon this embedding, we present a hypernetwork-based CL framework, where task-conditioned hypernetwrkk generates task-specific model weights based on H-embedding modulated task embeddings.
In summary, with the aim of better understanding and utilization of the task relationships in CL, we propose in this work a novel H-embedding guided hypernet framework. Our framework is featured by: 1) efficient and reliable learning of task embedding based on the information theoretical foundation of H-score metric: 2) ease of practical use with end-to-end training and minimal additional storage beyond low-dimensional task embeddings; 3) the flexibility to serve as plug-in module for generating and substituting specific model parts, such as LoRA layers, therefore compatible with various existing CL strategies and pretrained models; 4) notable enhancement of CL in overall performance across different model backbones and benchmark datasets.
2 Related Works
2.1 Continual Learning Strategies
Up to now, there have been many studies dedicated to CL strategies, most of them involving the rehearsal of previous data to alleviate the knowledge degradation caused by CF [e.g. CF Buzzega et al. 2020). However, growing privacy and data safety concerns have made this solution not always feasible, bringing increased attention to the rehearsal-free CL setting Smith et al. (023b
Robinss 1995
Most rehearsal-free methods rely on regularization- either in parameter space [e.g. Kirkpatrick ct al. 2017 Zenke et al. 2017] or feature space [c.g. Li and Hoiem| 2017] Rebuffi et al. 2017 by constraining model updates to preserve past knowledge. While effective in CF mitigation, these methods are often based on assuming task similarity and pose limitations to model adaptability.
Moor discussion in Sec.

---

2024] offer an alternative by separating task-specific and shared components at various architectural levels [e.g. |Malyya and Lazebmik] 2018 Wortsman et al.) 2020 Jin and Kim 2022]. With the rise of pretrained models, parameter efficient finetuning (PEFT) components like prompts is le.g. Wang et al. 2022b Smith et al. 2023a Wang et al., [023] and LoRA modules [e.g. [Liag and Li 2024 Wu et al. 2025 are notably gainnng popularity in this paradigm. Nevertheless, the allocation of model parts to different tasks usually comes with scaling problems with the growth of task numbers, potentially resulting in insufficient model capacity or excessive model size growth. Task transferability has recently been explored to enhance CL performance (Ermis et al. 2022]. Yet, the current method relies storing both past models and selected samples, which is incompatible with the rehearsal-fre -free setting. Developing strategies that effectively incorporate task transferability while remaining compatible with the online CL paradigm therefore remains open question.
Architecture- based approaches [Wang et al.
2.2 Transferability Metrics
Task transferability [Zamir et al., 2018 investigates the relationships between tasks and provides an effective method evaluate and select source tasks in transfer learning. It also plays a crucial role in developing strategies for multi-task learning and meta-learning. For ease of use, previous
studies have proposed metrics based on task models and data distributions for a quick estimation of
transferability [Ding et al. 2024]. H-score [Bao et al. 2019 Ibrahim et al., 2022 Wu et al., [222]
uses an information-theorett framework to evaluate transferability by solving a maximum correlation problem. NCE (Tran et al. 2019] employs conditional entropy to assess transferability and task difficulty. LEEP score [Nguyen et al. 2020 AAgostinelli al., [022] offers a more generalized metric, defined by measuring the performance of classifier developed from source model predictions when applied to the target task. LogME (You et al. [022] assesses target task accuracy using a formulation integrating all possible linear classifiers derived from source model features. OTCE [Tan et al., 2021 2024] combines optimal transport with conditional entropy to both estimate the domain and task difference between source and target. These metries mostly designed with differed assumptions and source accessibility, with their use applicable to different problem settings.
3 Preliminary
3.1 Mathematical Formulation
the data of task j is denoted by Dj (X(), yu)), with input samples X (2)()))) N and output samples y (y( Here, N; = |X61 = = denotes the sample size of the j-th task, and the attributes of sample data x(1,)1, y(o depends on the particular CL setting as well as the form of tasks. In CL, the M tasks are learned sequentially during the training stage. To be specific, denoting neural network model as f(x Q) (where f represents the model function, x represents the input data, and @ represents the model weights) and the model weights acquired in task 1 as Q0--1, the goal of learning task j is to derive a new set of weights (()) that not only achieves the optimal performance on task j. but also performs better or not significantly worse than (Gi-1 on tasks T ... T2-1. For rehearsal -free CL setting, the previous data Di...., D5-1 are not accessible during the training of the j-th task.
Consider a problem setting consisting of M tasks {T JM
3.2 Hypernets
Hypernets (Ha at. 2017]. or hypernetworks, are specialized neural networks that generate weights of another neural network, i.c., the target network. Given target model or component f. hypernet- work fr produces its parameters O conditioned on an auxiliary input. This input can be naturally interpreted as a task embedding, as reflects the task-specific information needed generate ap- propriate parameters. Denoting the task embedding as and hypernet parameters as O,. we have θ = (e, O₂). By leveraging task embeddings, hypernets enable flexible parameter modulation across tasks, offering a mechanism for task adaptation without direct weight sharing or parameter interference. Recently, hypernets have gained recognition as a potent tool in deep learning, proving effective across diverse deep learning tasks [Chauhan et al. 2023). including continual learning vvo Oswald et al., 2200)), causal inference [Chauhan et al. 2024], domain adaptation [Volk et al. 2022
few-shot learning [Sendera et al. 2023 and reinforcement learning (Sarafian et al. 2021
3

---

3.3 H-score
H-score is firstly introduced by Huang al. in 2019 as a metric assessing the informativeness of features for task. Theoretically derived from the maximal correlation interpretation of deep neural networks, its mathematical foundation roots the information theory work known as maximal correlation analysis, which originates from the works of Hirschfeld, Gebelein and Renyi [Hirschfeld] 1935 Gebelein 1941, Rényi] 1959] and has been followed and further explored by a broad spectrum of successive work. The H-score of f with regard to the task casting X to Y is defined as:
(() = (f()) coo L [f(x)(x]))
(I)
with input data X, label Y and feature extractor f(X). Subsequent research has extended the application of H-score, establishing it as a metric for transferability and validating its efficiency through extensive experimental evaluations [Bao al. 2019. Ibrahim et al. 2022 This underscores H-score's 's potential for transfer learning and its relevance to related challenges. We have chosen incorporate H-score into our framework due to its robust theoretical reliability, alignment of assumptions with our problem setting, and its independence from source data, enabling effective online embedding estimation.
Previous Tasks (Inaccessible)
Current Task
Future Tasks
T1
T2
Tj -
Tj
Xj
@
T
M
Regularization
HyperNet Training
>
CL Direction
fn
H-embedding Guidance
e
e(1)
e(2)
...
eU-i)
Embedding Learning
e
Task Embeddings
Figure Illustration of the CL status on the step of learning task j under our framework. The hypernet is being trained to provide the optimal task model weight ( concurrently with the learning of current task embedding e(i), where regularization and guidance are applied using previous embeddings and H-embeddings.
Methodology
4.1
H-embedding
Unlike most existing approaches that focus on model elements, the core of this work we incorporate task relationship information derived from accessible data into prior embedding, and utilize it to guide hypernet- aapeeeet-based CL framework. To this end, we introduce H-embedding, a transferability- aware online task embedding built upon the H-score metric The acquisition of H-embedding is specially tailored fit with the training process and data accessibility constraints in CL. settings.
Specifically, during the training stage task j. we first measure the H-score transferability from each previous task [Tn b-=1 to T3 using Dj and previous task model parameters (0!n) p=1 by
(Tr, T3) (r(coo(f((2()), (())))=-. cov(Epx (f((z()) (11)).
(2)
Here, fi( denotes the output of the last hidden layer in the task model f. which can be viewed as the feature of task data (i) R We leverage the convenience that previous task models (Q(m))
*Our design of H H-embeeding and the adoption of H-score are primarily motivated by its computational efficiency. effectiveness in measuring transferability, and compatibility with the online CL setting; more detailed discussion task embedding alternatives is provided in Appendix
NNote. The first feature covariance is computed across samples x, and the second across class-conditional means over

---

can be reconstructed by the hypemnet fú. and corresponding task embeddings {el")== In=1 i.e. O(") =
fa(elm), ..). The H-embedding ε(1) of current task is then computed by minimizing the difference between the Euclidean distance of elm), and their reversed H-score transferability (Tn.TT)):
1-1
2هع arg min (l3)
Σ (11060) (1lat)-e(*))]
cl2) (1/
where (= is calculated and stored when learning previous tasks. Given that assessing transfer- ability requires a minimum of two tasks, H-embeddings and the guidance loss are computed only after completing the first tasks.
Nevertheless, due to its target-centered intrinsicality, the absolute value of H-score is largely dependent on the characteristics of target task, resulting in asymmetry and inconsistent scaling across task pairs. This discrepancy prevents direct alignment between the reversed H-score and the Euclidean distance among sequentially learned task embeddings. To alleviate this mismatch, we introduce Analytic Hierarchy Process (AHP) normalization (Zamir et al.] 2018 to process the H-score in Eqn. 3 Specifically, we construct a pairwise tournament matrix W E Rix for task j. with elements given by:
HTm:: Tj) (T,, T2)
U)
Vm, (1,2,...,5},
(4)
measuring how many times better task m. is compared to task when transferring to task j. Here, we define the self- transfcrabiiitty H (T),T) as the HGR maximal correlation between X, and Yj, with the consistency of transferability definition verified by the theoretical framework of H-score Conse- quently, the AHP normalized transferabilites are given by elements of the principal eigenvector v(i) of W ie. AHP((T, T,) = v!/), which could be reasonably converted to distance metric through a negatively correlated affinity-distance distance transformation, dist(T,, Tÿ) = 700 exp( AHHP( T2)).
The scaling constant 7(5)) would be optimized together with 2(t) modifying Eqn. 3 to:
e60) 20),,00)
arg min 803,00
E(1e0)) el") -0a---00eppppppppppl
exp(p (xP(_AAHP(TT,77)))
(5)
Given that H (T,,, n. T5) and (n)) are directly calculated, the above optimization problem is a benign bi-variate optimization problem. We could thus apply gradient descent algorithm effectively compute the H-embedding e(f) for the j-th task. As such, the H-embeddings for all tasks during the continual learning can be calculated in an inductive way.
4.2
H-Embedding Guided Hypernet
4.2.1
Hypernet-Based CL Framework
Based on the task-relation-awar H-embedding, we leverage the critical role of task embeddings in hypernet- based models, and design hypernet framework that incorporates H-embedding guidance into the task besk-specific parameter generation. particular, we present CL framework (illustrated in Fig. I where task-conan nuskcconditioned hypemetwork fn (e, Ox) with hypernet weights O,, is introduced to map task embeddings (c(t) M j== to the corresponding model weights (0()) J==1 of task model f
for all CL tasks M i.e., ( (eli), θ₁)) for task j. When training on each task T,. the task embedding eli) is updated together with the optimization of hypernet parameters O), while parameters other than θ, and e()) are fixed and can be viewed as constants. The learning of e()) and O₁ is regularized using previous task embeddings (e(x))i= and guided using H-embedding ((ا to ensure backward and forward transfer performance. The learning loss is composed of three parts:
1) Target loss, supervised loss to learn current task j
L = L (f(xl) )(0)) = c (f(et)) Sn(C)), Ow)),
(6)
Seee Appendix A.2 for brief proof and computation details.
5

---

2) Continual learning loss (following von Oswald et al. (0200). to prevent CF by ensuring that given
previous task embeddings {e!n) ) I==1 the training on task j are analogous
the network weights output by the hypernet before and after
i-1
Lc =
-
Σ R=1
- 0f)1.
(7)
3) H-embedding guidance loss, to provide the hypernet with additional prior knowledge about task relationships using transferability
Le = Le(e))), E(1).
(8)
Here, L denotes certain supervised task loss (cross-entropy loss in experiments), and 0 is the set of hypernet parameters before learning task j. The definition of the embedding regularization loss Le will be covered in later sections. To summarize, our final loss function is as follows with hyperparameters Be and Se:
L= L1 + BeLe + BeLe.
On the j-th task, approach for the training of Tj is depicted in Fig. 2 Notably, although it may appear that the task model weights are first generated and subsequently used for inference, the framework actually end-to-end, with the hypernet parameters O₁, and embeddings e( optimized directly by feeding the task data and minimizing the overall loss. Hence, there is no additional training procedure introduced our framework, and only information to save low-dimensional task embedding e(i)
MainNet
Prior H-embedding
Le =
Guidance Loss
1.1
L, = La (e 200)
Decoder
Main Model
Embedding
Weights
Encoder
Hidden Space
HyperNet Model
Y
HyperNet
prear y
Figure 2: Framework of our hypernet on the slice task j. A hypernet (left, blue) is utilized to learn the weights of the main model (right, orange), where the H-embedding guidance is introduced using an encoder-decoder module. The entire framework is trained end-to-end by inputting task data into the main model and propagating gradients backward to update both hypernet and embedding.
4.2.2 Embedding Guidance via Encoder and Decoder
In our framework, embedding regularization module is introduced to integrate the H-embedding guidance into the hypernet. Specifically, we view the output of the hypernet's intermediate layer hidden representation h. and the preceding layers transmission perspective, it can be presumed that should retain sufficient information to recover the H-embedding é. Therefore, reconstructs an embedding ? from h, such that the discrepancy between é and the H-embedding should be minimized. Formally, for task j.
encoder f From information
attach the layer lightweight trainable decoder fee that
a (D = fDee (Fenc(e(i))
(10)
The dimension of task embedding is set 32 in experiments.

---

CIFAR-100
(N 10)
ImageNet-R 10) FAA(1)
Backbone
Method
FAA(T)
DAA(1)
DAA(1)
Full Finetune
19.13 (1.35)
79.02
15.08 (0.62)
56.42
LwF
34.35 (1.06)
84.55 (0.27)
40.6610.271 (7.30(0.05)
EWC
(2.83
84.21 (0.04)
(0.29)
27.97 (0.34)
L2
(1.49((..4)
(4.96(0.18)
(5.95(0.a9)
63.77 (0.56)
ResNet-32
PredKD FeatKD
83.72 (0.45
(8.22
40.37 (0.75)
PackNet
70.68
70.68
34.63
34.63 (0.85)
HyperNet
81.57
(1.63
(1.03
38. 18 (0.04))
WSN
(0.44)
82.75 (0.44)
37.99 (0.27)
37.99 (0.27)
H-embed Hnet
83.08 (o. 12)
83.09 (0.09)
38.16
38.09 (0.09)
Full Finetune
69.49 (0.50)
80.35 (0.87)
60.57 (1.06)
72.31 (1.09)
L2P
83.18
87.69(1.05)
71.26(0.44)
76.13
DualPrompt
(0.86)
(0.66)
(0.22
73.81 (0.39)
CODA-Prompi
86.31 (0.12)
90.67 (0.22)
74.05
78.14
(0.39) (0.39)
ViT-B/16
HiDe-Prompt
93.48
(5.02
(4.65
78.46(0.18)
InfLoRA
91.72 (0.35)
74.75
80.67 (0.55)
SD-LoRA
87. (0.22)
92.05 (0.31)
77.18 (0.18
81.74 (0.24)
H-embed Hnet-LoRA*
97.07 (1.61)
(7.06)
81.38 (0.65)
81.67 (0.10)
Table 1: Accuracy (%) Comparison on CIFAR-100 (N 10) and ImageNet-R (N 10) on ResNet-32 and ViT-B/16 Backbones. Our method (marked by **') achieves the top FAA.
should approximate e(d) as close possible. Summing it up in mathematical form, we have the embedding guidance loss for task j:
L = La (f e00))
(f
(11)
L denotes a certain similarity criterion, set to the cosine similarity loss in our experiments. The H- embedding أاك is derived by 5 the decoder fDee is updated together with the hypernet during training. Notably, regularization module given the encoder and decoder are both shallow fully connected neural networks. We summarize the training process of task the algorithm in Appendix A.4
significant computing cost is posed with the introduction of the embedding
4.2.3 Plug-in Application in PEFT Settings
While our previous formulations are mostly based on assuming that the hypernet generates the full parameter set θ for the task model, this is not strict requirement. In practice, our framework can be easily adapted to generate only a subset modern parameter-efficient fine-tuning (PEFT) techniques such prompt tuning and LoRA which leverage frozen pretrained backbones and update only small, task-specific modules. Under this setting, method can be readily deployed as a plug-in component that exclusively generates LoRA parameters, with the rest of the model kept fixed. We validate this lightweight variant in experiments and demonstrate that it obtains strong performance, highlighting the flexibility, scalability, and practical utility of our approach.
model parameters, making naturally compatible with
5 Experiments
5.1 Experimental Settings
Benchmarks.
comprehensively verify the effectiveness of our framework and further analyze
its reliability, we select three representative benchmarks from previous work CL and perform extensive experiments on them: PermutedMNIST (10 tasks) [Goodfellow et al.) 2013). 1000
(10 tasks) [Krizhevsky et al., 2009], DomainNet (5 tasks) (Peng et al. 2019 and ImageNet-R (5

---

tasks. 10 tasks and 20 tasks) Heendrycks et al. 2021 8
A detailed description of these benchmarks is
listed in Appendix A.5
Evaluation Metries. Following previous works [Qu et al. 2021 Wang et al. 2024], we evaluate the different CL methods with two widely adopted metrics. Final Average Accuracy: F AA M Ej=1 aj.M, the average task accuracy measured after the completion learning all tasks;
During Average Accuracy:DAA
1, شز ajj- the average accuracy of each task measured immediately after it is learned. Here, as,j denotes the accuracy (%) measured on the test set of i-th task after learning the j-th task. In terms of CL analysis, FAA provides summary evaluation of overall CL performance, while DAA offers deeper insights into the forward/backward transfer behavior and model learning dynamics.
5.2 Performance Evaluation
Comparison Experiments with Baselines from Multiple Paradigms. Our primary evaluation study is conducted the CIFAR-100 and ImageNet-R benchmarks, both consists of 10 tasks. To ensure fairness and comprehensiveness in comparison, a ResNet-32 [He et al. 2016] and a pretrained ViT-B/16 (Dosovitskiy al. 2020 selected as the shared backbone models for the full-model and parameter-efficient methods respectively. All reported results are either obtained by our own implementation (averaged over three random seeds with standard deviation) or directly cited from previous works using the same benchmark and backbone.
The choice of baselines is based on the requirements that they should both conform to rehearsal- setting and be applicable to these benchmarks and backbones. For a thorough comparison with exist- ing methods to the greatest extent possible, we select representative baselines of varied methodology
categories. Full-moo model strategies include: Regularization Methods:
LwF [Li and Hoiem 2017].
EWC KKirpptricck et al. [217]] L2, PredKD+FeatKD (Smith et al.
20236]; Architecture Methods:
PackNet MMallya and Lazebnik 2018). HyperNet (von Oswald et al.
2020). WSN (Kang et al. 2022)
Parameter-efficient finetuning (PEFT) based strategies include: Prompt Methods: 1.2P Wang et al.
2022b DualPrompt (Wang et al.) 2022a), CODA-Prompt [Smith et al., [2023a] HiDe-Prompt Wang
al. 2023]; LoRA Methods: InfLoRA (Liang and Li, 2024), SD-LORA [Wu al., 2025 Full
finetuning is also included as a basic reference in both backbones. For our framework, we construct full model and LoRA generation versions on ResNet and ViT respectively. The experimental results are summarized in Tab. | where our method achieves the highest FAA across both backbones and benchmarks, demonstrating superior overall CL performance. In addition. attains competitive DAA that closely aligns with FAA, suggesting strong forward and backward transfer capabilities.
Extended Evaluation of PEFT Methods under Varying Task Numbers and Challenging Bench- marks. Given the stronger model capacity and CL performance typically exhibited by PEFT-based methods through leveraging pretrained models, we conduct additional evaluations to further assess their robustness under more challenging settings, including varying task sequence lengths and the more difficult DomainNet benchmark. As shown in Tab. 2 our method consistently outperforms all baselines. Notably, its advantage becomes more pronounced as the number of tasks increases, highlighting its robustness in scenarios with longer and more challenging task sequences.
Ablation Studies. To take a better concentration on validating our design of framework, we conduct ablation studies on ImageNet-R (N 5, 10, 20) benchmarks with ViT-LoRA backbone, and summarize the results in Fig. 3 The comparison baselines are: without H-embedding guidance (w/o Hemb), without CL regularizatinn (w/o CLreg), without AHP normalization (w/o AHP), and SD-LoRA, the SOTA baseline for reference. For a more comprehensive evaluation, we also conduct extra ablation studies on full model generation under different backbones and benchmarks, covering PermutedMNIST, CIFAR- 100 and ImageNet-R, with results list in Appendix A.7 A notable increase in CL. performance could be observed across all benchmarks and backbones.
'Training specifics of our experimental studies are listed in Appendix A.6 A.6 with codes available at hhtpps //github.com httpa//ooooo/iia///////eeeeeeede http://eeddeddeeeddddddeeeedeeee BeiideedHyppenes
8

---

ImageNet-R (N =)
ImageNet-R (N 20)
Domain (N
Method
FAAt
DAAT
FAAT
DAAT
FAAT
DAAt
Full Finetune
64.92
(0.50)
(5.32 (1.95(1.3))
51. (1,46
67.08
L2P
(3.04(0.71)
(6.94(0.41)
68.97(0.55)
(4.16(0.32)
(0.26
(0.88))
DualPrompt
72.24(0.41) (9.99)0.57)
(o.45)
(1.30
(8.26
(3.84(0.45)
CODA-Prompt
(6.63
(0.30(0.28)
(9.38
(3.95(0.63)
70.58(0.53)
76.68(0.44)
PriDe-Prompt
(0.27
78. (8.15
(3.59(a.19)
(0.9)
72.20(..08)
77.01 (0.04)
InfL.oR
(6.95(0.23))
81.81 (0.14)
69.89 (0.58)
(6.68
71.59(0.20)
(8.29
SD-LoRA
79.01 (0.26)
(2.50().38)
(4.05(0.61)
(0.65
72.58
78.79 (8.79
H-embed Hnet-LoRA*
79.27(0..42)
(0.22)
(9.90
84.79(0.37)
76.64
78.26(0.37)
Table 2: Further Evaluation on ImageNet-R (N 20) and DomainNet (N 5). Our method
(marked by **") consistently obtains the highest FAA.
Imagenet Tasks
Imagenet Tasks
Imagenet
Performance
1.0
1.0
1.0
64-
0.9
0.9
0.5
62-
0.6
Gavi (11))
0.5
0.5
0.5
0.4
0.4
0.4
70
was Arp
mit ctrep
wio Hemd
with Ato
Laok
Dore
with Hero
wis Alp
sou code
19 135455
20.2200 20-2545
ets
Figure 3: Illustration of Ablation Studies (a) and CIL Performance (b). Left (a): FAA and DAA results of ablation studies (averaged across seeds). Right (b): FAA and DAA (striped) results of CIL baselines.
5.3 Discussion and Further Performance Analysis
Optimal Overall Transfer Ability. As shown in previous results, our method demonstrates strong overall performance across various CL benchmarks. To gain deeper insight into its effectiveness, we analyze the framework from the perspective of forward transfer (FWT) and backward transfer (BWT) two essential indicators in CL., Formally, forward transfer is defined as FWT := DAA GTA, and backward transfer as BWT := FAA DAA, where GTA refers to the ground-truth accuracy obtained by individually training backbone model on each task. Higher values on both metrics indicate better CL capability. As illustrated Fig. 3 which provides a visually intuitive comparison of the discrepancies between FAA and DAA across baselines, our method demonstrates competitive abilities in both FWT and BWT, thereby displaying prominent CL performance. Due to the space limit, we include detailed numerical results the metrics in Appendix A.8
Parameter Size and Efficiency. our framework design, the overall number of hypernet pa- rameters (including task embeddings) is strictly constrained to be no larger than that of the main network. Both the decoder module and H-embedding are lightweight, implemented as two-layer MLP and a 32-dimmnsiooal vector respectively. This compact design ensures that the H-embedding guided hypernet does not introduce parameter overhead even with long task sequences. The storage efficiency is also preserved, as only the hypernet and task embedding need to be stored.
In terms of time efficiency, single prediction in the hypernet framework involves one forward pass through the hypernet and one through the main net, theoretically doubling the inference time consumption. However, in practice the hypernet inference is executed only once per task generate task-specift weights, and its time cost is amortized across all samples. As a result, the per-sample overhead becomes negligible on large datasets. The VÝT-LoRA setting even more efficient, where the hypernet only generates compact LoRA parameters rather than full model weights. We conduct time tests for both the ResNet backbone and the ViT-LoRA backbone, and the results support our analysis: on CIFAR100, vanilla ResNet32 takes 4.257s to process the full test set, compared to 2,260s with our framework; on ImageNet-R, vanilla ViT takes 4.313s, compared to 4.568s with our ViT-LoRA framework. Here bit more time cost is introduced by the addition of LoRA modules.

---

TIL versus CIL. Up to now, our main experiments are conducted under the musk-incremental learning (TIL) setting. where task identifiers are provided during both training and inference. However, in class-incremental learning (CIL.) the unavailable task ID) at test time can make the problem more challenging. Hopefully, due to the orthogonality of our framework with task ID inference, we could adapt it to the CIL setting by introducing an auxiliary module to infer the task ID. Specifically, during training, we cache the intermediate features of input samples extracted from a frozen pretrained model to train a task ID classifier, and use it to infer the task IDs during testing, and select the corresponding task embedding. We evaluate this extension on ImageNet-R (N 5, 10, 20). As shown in Fig. [ our CIL method remains competitive with strong baselines, demonstrating its applicability beyond the TIL setting. Nevertheless, this current approach is admittedly limited by its reliance on pretrained model features and the separately trained task classifier, which does not align tightly with the online, continual nature of our framework. We will continue to explore more integrated and dynamic task II) inference strategies that can be better coupled with our architecture in the future.
6 Conclusion
In this work, we propose a transferability-aware task embedding guided hypernet to exploit the task relationships for continual learning. By introducing the information theoretical transferability based task embedding named H-embedding and incorporating it in a hypernetwrrk, we establish an online framework capable of capturing the statistical relations among the CL tasks and leveraging this knowledge for guiding task-conditioned model weight generation. Through extensive experimental studies, we validate that the adoption of H-embedding guidance enhances continual learning by facilitating inter-tas, task transfer and improving the reliability of task embeddings, achieving the best final accuracy performance under various CL benchmarks.
Acknowledgments and Disclosure of Funding
This work is supported in part by the Natural Science Foundation of China (Grant 62371270, and Shenzhen Key Laboratory of Ubiquitous Data Enabling (No. (N0.2DSSS222005277110000
References
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6430-6439, 2019.
Andrea Agostinelli, Jasper Uijlings, Thomas Mensink, and Vittorio Ferrari. Transferability metrics for selecting source model ensembles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7936-7946 2022.
Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung- Woo Ha, and Jonghyun Choi. Rainbow memory: Continual learning with a memory of diverse samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2021.
Yajie Bao, Yang Li, Shao-Lun Huang, Lin Zhang, Lizhong Zheng, Amir Zamir, and Leonidas Guibas. approach to transferability in task transfer learning. In 2019 IEEE An conference on image processing (ICIP), pages 2309-2313 IEEE, 2019.
international
Eden Belouadah and Adrian Popescu. 112m: Class incremental learning with dual memory. In Proceedings of the IEEE/CVF international conference on computer vision, pages 583-592, 2019.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: strong, simple baseline. Advances in neural information processing systems, 33:1 15900-15990, 2020.
Rich Caruana. Multitask learning. Machine learning, 28:41-75, 1997.
Vinod Kumar Chauhan, Jiandong Zhou, Ping Lu, Soheila Molaei, and David A Clifton. A brief review of hypernetworks in deep learning. arXiv preprint arXiv:2306.0660 2023.
10

---

Vinod Kumar Chauhan, Jiandong Zhou, Ghadeer Ghosheh, Soheila Molaei, and David A Clifton. Dynamic inter- treatment information sharing for individualized treatment effects estimation. In International Conference on Artificial Intelligence and Statistics, pages
PMLR, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255 leee, 2009.
Yuhe Ding, Bo Jiang, Aijing Yu, Aihua Zheng, and Jian Liang. Which model to transfer? a survey on transferability estimation. arXiv preprint urXiv:240..52235 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprini arXiv:2010. / 1929, 2020.
Beyza Ermis, Giovanni Zappella, Martin Wistuba, Aditya Rawal, and Cedric Archambeau. Memory efficient continual learning with transformers, Advances in Neural Information Processing Systems, 35:10629-10642, 2022.
Hans Gebelein. Das statistische problem der korrelation als variations-und eigenwertproblem und sein zusammenhang mit der ausgleichsrechnung ZAMM-Journal of Applied Mathematics and Mechanics/ZZiischriff/ für Angewandle Mathematik und Mechanik, 21(():364433) 1941.
Ian Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investi- gation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv: 131.,221,, 2013.
David Ha, Andrew M Dai, and Quoe V Le. Hypernetworks. In International Conference on Learning Representations, 2017,
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3340-8349 2021.
Hermann O Hirschfeld. A connection between correlation and contingency. In Mathematical Proceedings of the Cambridge Philosophical Society, volume 31, pages 520-524. Cambridge University Press, 1935.
YenChang Hsu, YenCheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating continual learning scenarios: A categorization and case for strong baselines. continual learning workshop. In 32nd Conference on Neural Information Processing Systems, 2018.
Shao-Lun Huang, Xiangxiang Xu, Lizhong Zheng, and Gregory W Wornell. An information theoretic interpretation to deep neural networks. In 2019 IEEE International Symposium on Information Theory (ISFT), pages 1984-1999999 1988. IEEE, 2019.
Shibal Ibrahim, Natalia Ponomareva, and Rahul Mazumder. Newer not always better: Rethinking transferability metries, their peculiarities, stability and performance. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 693-709. Springer, 2022.
Hyundong Jin and Funwoo Kim. Helpful or harmful: Inter-task association in continual learning. In European Conference on Computer Vision, pages 519-535. Springer, 2022.
Haeyong Kang, Rusty John Lloyd Mina, Sultan Rizky Hikmawan Madjid, Jaehong Yoon, Mark Hossgawa-Johnson, Sung Ju Hwang, and Chang D Yoo. Forget-free continual learning with winning subnetworks. In International Conference on Machine Learning, pages 10734-10750. PMLR, 2022.
11

---

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho. Agnieszka Grabska- Barrkkkkaa,, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11)22278-2324 1998.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-299) 2017.
Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference-free low-rank adaptation for continual learning. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages (663--3367,, 2024.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7765-7773, 2018
Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: A new measure to evaluate transferability of learned representations. In International Conference on Machine Learning, pages 7294-7305. PMLR, 2020.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1406-1415, 2019.
Haoxuan Qu Hossein Rahmani, Li Xu, Bryan Williams, and Jun Liu. Recent advances of continual learning in computer vision: An overview. arXiv preprint arXiv:2109. 1169,, 2021.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010, 2017.
Alfréd Rényi. On measures of dependence. Acta mathematica hungarica, 10(3-4):441-4555 1959.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint 1rXiv:1810.11910 2018
Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7(2): 123-146, 1995.
Elad Sarafian, Shai Keynan, and Sarit Kraus. Recomposing the reinforcement learning building blocks with hyperntworks. In International Conference on Machine Learning, pages 9301-9311 PMLR, 2021.
Marein Sendera, Marein Przewię/likosski, Konrad Karanowski, Maciej Zięba, Jacek Tabor, and Przemyslaw Spurek. Hypershot: Few-shot learning by kernel hypernetworks. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2469-2478, 2023.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante- Bonilla, Donghyun Kim, Assaf Arbellc, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda Coda-prompt: Continual decomposed attention-based prompting for rehearsal free continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11909-1 19999,11919, 2023a.
James Seale Smith, Junjiao Tian, Shaunak Halbe, Yen-Chang Hsu, and Zsolt Kira. A closer look at rehearsal-free continual learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 2410-2420, 2023b.
12

---

Yang Tan, Yang Li, and Shao-Lun Huang. Otce: A transferability metric for cross domsinn cross-task representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15779-15788, 2021.
Yang Tan, Enming Zhang, Yang Li, Shao-Lun Huang, and Xiao-Ping Zhang. Transferability-gunee cross-domain cross-tase transfer learning. IEEE Transactions on Neural Networks and Learning Systems, 2024.
Anh T Tran, Cuong V Nguyen, and Tal Hassner. Transferability and hardness of supervised clas- sification tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1395-1405, 2019.
Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning, arXiv preprint arXiv:1904.07734, 2019.
Tomer Volk, Eyal Ben-David, Ohad Amosy, Gal Chechik, and Roi Reichart. Example-based hypernetworks for out-of-disrribution generalization. arXiv preprint arXiv:2203. /4276, 2022.
Johannes von Oswald, Christian Henning, Benjamin F Grewe, and João Sacramento. Continual learning with hypemetworks. In 8th International Conference Learning Representations (ICLR virtaa) International Conference on Learning Representations, 2020.
L Wang, X Zhang, H Su, and J Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Liyuan Wang. Jingyi Xie, Xingxing Zhang. Mingyi Huang, Hang Su, and Jun Zhu. Hierarchical decomposition of prompt-based continual learning: Rethinking obscured aub-optimality. Advances in Neural Information Processing Systems, 36:69054 -99066, 2023.
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European conference on computer vision, pages 631 631-648. Springer, 2022a.
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang. Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy. and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 139-149, 2022b.
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. Advances in Neural Information Processing Systems, 33:15173-15184, 2020.
Yanru Wu, Jianning Wang, Weida Wang, and Yang Li. H-ensemble: An information theoretic approach to reliable few-shot multi-source-fiee transfer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 15970-15978, 2024.
Yichen Wu, Hongming Piao, Long-Kai Kai Huang, Renzhen Wang, Wanhua Li, Hanspeter Pfister, Deyu Meng, Kede Ma, and Ying Wei. Sd-lora: Scalable decoupled low-rank adaptation for class incremental learning. In The Thirteenth International Conference on Learning Representations, 2025.
Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme: Practical assessment of pre-trained models for transfer learning. In International Conference on Machine Learning, pages 12133-12143. PMLR, 2021.
Amir R Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3712-3722, 2018
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference machine learning, pages 3987-3995. PMLR, 2017.
13

---

NeurIPS Paper Checklist
You should answer [Yes], [No], or [NA].
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We carefully checked the corresponding sections. See Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We include limitation discussion on CL settings in Sec. 5.3
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification: See Sec. 4
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Sec. 5 and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification:
Yes we provided the open-souree code base.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ "pipsscc/puullc//uuu /ttp://ggu..eeeoooouuuuumbibbi ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Sec. and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We did multiple runs and reported standard deviations, see Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
16

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics /tttss:/neuurps..co httpa//wwww......aaa./.ccccaaaaa
Answer: [Yes]
Justification:
We conform to the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The work is mainly focused on an ideal continual learning setting, and not closely related to social impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The work is totally based on open-source datasets and models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We added citations of all the assets we used, as well as ensured that all used assets under permissive licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We released our code base with necessary instructions.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
20

---

A Technical Appendices and Supplementary Material
A.1
Continual Learning Setting
A.1.1
Rehearsal-Free CL
The strictness of CL settings varies with the extent of allowed previous data accessibility. Multi- task learning [Caruana [997]. with full data availability of all tasks, actually be viewed CL [Smith et al.] 2023b], with no previous data involved
special case of CL, while rehearsal-Fi in the training of tasks, is the strictest CL setting under this criteria. Despite the success of rehearsal-base methods in various benchmarks [Bang et al. 2021 Shin et al. 2017, [Belouadah and Popescu 2019], rehearsal-free CL is catching the attention of researchers recently (Smith et al., 2023b] because of its low dependency on revisiting previous tasks and therefore broader application the era of growing data privacy concerns. Existing works on rehearsal-free CL mostly based on regularization strategies. EWC [Kirkpatrick et al. 2017] and SI (Zenke et al. [017]] introduce penalties to restrict the alteration of parameters vital for addressing prior tasks, thereby reducing the risk of CF. LwF [Li and Hoiem 2017 Rebufii et al. 2017 proposes a cross- cross-entropy loss between the predicted class distribution the (n- I-th task, as generated by the model before and after learning the n-th task. |mmith et al. (2023b) reviews these methods and proposes regularization combinations for better CL. performance. In this work, we follow these works and focus on the more challenging rehearsal- CL. setting.
A.1.2 TIL versus CIL
Based the discrepancy between Dj-1 and Dj. Hsu et al. (2018) and Van de Ven and Tolias [2019] categorize CL settings into three specific scenarios: task incremental, class incremental, and domain incremental. Table jummariesss the differences among these scenarios. For better concentration on the study of CL methodology, our work mainly focuses on the task incremental CL (TIL). In this scenario, the output spaces of tasks are partitioned by task IDs and mutually exclusive between D -1 and Dj. which is denoted as Yú- Y It can be then naturally indicated that P(Y(-1)) * p() and P(X()-1)) & P(X Notably, here task IDs are accessible during both training and testing. An adaptation of our method to other CL settings could be effectively conducted by introducing additional task inference modules similar to previous works. As an instance, we made a trial in this work that adapts our framework to class incremental CL (CIL) in Sec. 5.3
using a task ID classifier trained on image features.
Scenario
P(X # P(X()) P(Yú-"1) # P(YU)) Yύ-1) + Y
Task ID
Domain Incremental
x
Class Incremental
X
X
Task Incremental
Table 3: Categorization of CL settings based on the discrepancy between D,-1 and Dy.
denotes the scenario focused on in our work.
A.2 HGR maximal correlation for self transferability Tī)
As the theoretical basis of H-score transferability, the Hürschfeld-Geeelein--eee Gebelein (HGR) maximal correlation of random variables X and Y over alphabets X' and y is defined as:
HCR(X,Y)=
max ト - El E(X(X) XIXX (x)== [0))
ExxY [X))g(()))
(12)
Here, the correlation is derived by taking the maximum over all functions f. g with zero mean and unit variance, which hence extracts the most correlated aspects of X and Y. This definition is equivalent to the maximum of the two-sided H-score given by Huang et al. [2019] with normalized functions of and g:
1 - (r(coo((((X)coo(((())
H(f.g) = Exxx [f (x))()))
(13)
2
21

---

with the one-sided H-score extended from Eqn. 13 by assuming the function g as optimal:
H(f) = tr(cco(((())) coo [f(X)]x])
(14)
Notably, Eqn. 14 is exactly the definition of H-score transferability [Bao et al. 2019 which is actually proposed by applying the H-score theories transfer learning scenarios. Therefore, with normalized f. the HGR maximal correlation between Xj. Y is mathematically equivalent to the H-score metric of the theoretical optimal model fj on task j.
In our work, we utilize the H-score transferability metrics denoted TT) for tasks i + j. However, the H-score for task j cannot be computed while the model remains untrained for that specific task. To maintain consistency in definitions, we define (T, T2) the HGR maximal correlation between the input X, and the output Y,, implicitly leveraging the theoretical optimal model Jj for task j. Specifically, in our computations, the models f and g are implemented as fully connected neural networks equipped with normalization layers, and they undergo 100 epochs of training using sampled subset of the training data through gradient descent.
A.3 Discussion the Use of H-Score and Alternative Task Embeddings
While technically task embeddings can be estimated using various methods, the alternatives are generally less suitable for our target setting, as they either require access source data for each lask-assupttions that are infeasible under rehearsal-free CL-or do not explicitly model task-to-task relationships, thereby offering limited benefits in facilitating bidirectional transfer across CL tasks.
To address this, we propose a new formulation where task embeddings are computed online by explicitly optimizing them approximate transferability metric across tasks. In this design, nature, we adopt H-score as the target transferability metric primarily due to its source-data- which allows to maintain compliance with CL constraints. Additionally, H-score is theoretically grounded, computationally efficient, and empirically shown to correlate well with actual transfer performance. These characteristics make it particularly suitable and representative choice for embedding algorithm. Yet, our framework is not inherently tied to H-score, and there admittedly exist other source-free embeddings or transferability metries (e.g. Task2Vec [Achille et al., 2019), LEEP [guuen et al. 2020), LogME (You et al., 2021) )) that could also, in principle, be used within it.
As both ablation and further assessment of our framework's 's flexibility, conducted preliminary experiments on ImageNet-R 10 task benchmark by:
Replacing H-score with other source-free transferability metrics (LEEP and LogME) while keeping the embedding algorithm and overall pipeline unchanged.
Replacing the entire H-embedding formulation with Task2Vec.
The results are shown in Tab.
Method
H-Embedding*
H-Embedding (LEEP)
H-Embedding (LogME)
Task2Vec
FAA (%)
81.8
81.6
81.4
79.8
Table 4: Comparison of different task embedding variants on the ImageNet-R 10-task bench- mark.
As shown, replacing the transferability metric alone leads to only minor drop in performance. This supports our view that while H-score is a strong instantiation, the core contribution of our work lies in the transferability -ugiiey embedding formulation and its integration into hypernetwork-basee parameter generation pipeline-a general framework that remains valid even when alternative metrics are employed.
However, replacing the entire embedding formulation with task2vec leads to a more noticeable performance degradation. We believe this is because Vec computes task embeddings inde- pendently without explicitly modeling task-to-task relationships, which limits its capacity to guide transfer dynamics in CL. Additionally, Vec is highly sensitive to the choice and training of the probe network, which introduces extra variance effectiveness and incurs additional memory costs-challenge challenges not present in our H-embedding approach.
22

---

A.4 Algorithm for H-embedding Guided Hypernet Framework
For better understanding of our framework, we summarize the entire training process of task within our H-embedding guided hypernet as outlined in Algorithm[]
Algorithm 1: H-embedding guided Hypernet: Training of Task j
Input: Task data D,. previous task embeddings {e(*) J==1 hypernet weights θ₂ Parameter Learning rate À
Output: Current task embedding e(,), updated hypernet weights θ₁ Randomly initialize e(i), e(U); j then
for n f-1 1 to j do
// Compute transferability
( t fn (e(m), Ox):
HT... T2) tr (10) ( ((11))))11
Eq
end
Randomly initialize y00;;
20), اا0ر arg mingon, ,00 Ei=l (11ε6) c(n)112 - -000 -000ppppppppp0000000000100 (xp(-AXHPII,,T))) > Eq.
end
repeat
// Train hypernet
eb) t eli) - AV L;
O2 t O AV
Eq.
until converge;
Return c(,), (b)
A.5 Benchmarks of Experiments
[Goodfellow et al. 2013 (N=10) benchmark is variant of MNIST [LeCun et al. 1998). forming CL tasks from the original MNIST dataset by applying random permutations to the input image pixels. The permuting procedure can be repeated times in experiments to yield a task sequence of 10, with each task consisting of 70,000 images (60,000 for training and 10,000 for testing) of digits from
PermutedMNIST
CIFAR-100 KKrizhessky al., 2009 (N 10) is a benchmark composed of 10
ten-way classification tasks composed by splitting a CIFAR- 100 dataset into ten tasks. The model is sequentially trained on the tasks, each with 6,000 images (5,000 for training and 1,000 for testing).
ImageNet-R [Hendrycks et al. [02T], built upon the ImageNet dataset (Deng et al. 2009), features diverse range of renditions of ImageNet classes. This benchmark includes a total of 30,000 images across 200 classes from ImageNet. For continual learning evaluation, ImageNet-R (N = 10, 20) formed by organizing ImageNet-R into 5, 10, and 20 tasks respectively, each containing 40, 20, and 10 classes and around 6000, 3000, 1500 samples (roughly 5/6 for training and 1/6 for testing). DomainNet (Peng et al., 2019 (N 5) is a benchmark derived from the 345 classes classification task across six distinct domains, by fusing all domains and splitting into 5 * 69 classification tasks.
A.6 Experimental settings
A.6.1 Comparison Experiments (Table
Choice of Baselines. Our selection of baselines in this work aims to encompass a wide range of baseline categories, covering two of three primary categories in contemporary CL researches (i.e., replay-based, regularization-basee and architecture-based) with the replay-based methods not conforming to our rehearsal-free setting. The specific choice of baselines in each category is mainly based on performance comparison conclusions in recent works such as (Smth et al. [2023b] and Kang al. 2022 Therefore, we believe that our comparison study has included the most competitive and representative baselines.
General Settings. In CIFAR-100 and ImageNet-R(10) tasks) datasets with the ResNer-32 backbone network, we evaluate various baseline methods applicable to its model structure including Full Finetune, EWC, L2, PredKD+FeatKD, PackNet. HyperNet, WSN, All baselines are implemented
23

---

on our own using& ResNet-32 with "option A', i..,, leveraging the zero-padding shortcuts for increasing dimensions. CIFAR-100 and ImageNet-R datasets with the ViT-B/16 backbone network. we mainly evaluate PEFT baseline methods including I.2P. DualPrompt, CODA-PrompL, HiDe- Prompt, Infl.RRA, and SD>-LRRA. We cited the results of most baselines from previous works [Wang et al. 2023 Wu et al. 2025]. For ResNet, the experiments on CIFAR- 100 are conducted on NVIDIA GeForce RIX 3090 GPUs with 100 epochs of training (unless early-stop), and the ImageNet-R experiments are carried out on NVIDIA A800 or A100 GPUs with 200 epochs of training (unless early-stop). For ViT, all experiments are conducted on NVIDIA A800 or A100 GPUs until convergence.
To ensure a fair comparison, we adopt consistent training settings across all baseline methods we implement (unless listed separately). Specifically, the batch size is set to 32, and we use the Adam optimizer with an initial learning rate of 0.001. The learning rate is decayed by a factor of 10 after the 50th and 75th epochs. A weight decay of 10 1 is applied. For robustness, each experiment is run three times with different random seeds 22, 32, and 42, and the results are averaged.
Details of Specifie Methods. For the Finetune baseline, the model is sequentially trained on each task without any mechanisms to prevent catastrophic forgetting. The model is randomly initialized and trained from scratch on the first task. The training of subsequent tasks continues using the weights obtained from the previous tasks.
For the EWC baseline [Kirkpatrick et al. 2017]. we add a regularization term to the loss function to penalize significant changes to parameters important for previously learned tasks. The importance of each parameter is estimated using the Fisher Information Matrix. The regularization coefficient is set to 10, following standard practice.
In the L2 baseline, an L.2 regularization term is added to the loss function limit changes in the model parameters during training new tasks. The regularization coefficient À is set to 1.0, determined by tuning on a small validation set derived from the training data of the first task.
For the PredKD+ + FeatKD method (Smith et al., 2023b), we incorporate both prediction distillation and feature distillation transfer knowledge from previous tasks to new ones. The distillation loss combines the Kullback-Leibhe
divergence between the soft outputs of the teacher (model trained on previous tasks) and the student (current model), as well as the mean squared error between their intermediate feature representations. The loss weights are set to a 1.0 and B = 0.5 based on preliminary tuning.
In the PackNet method [Mallya and Lazebnik 2018), we employ iterative pruning to allocate dedicated network weights for each task. After training on each task, we prune certain percentage of the weights with the smallest magnitudes. Following the recommendations in the original paper, experiment with pruning rates of 0.5, 0.75, and 0.8. We select the pruning rate of 0.8, which yields the best performance in our setting. After pruning, we fine-tune the remaining weights for an additional 10 epochs with a reduced learning rate of 1 x 10
For Hyper method [von Oswald et al. 2020 we primarily follow the original work in train- ing settings, with the learning rate being 0.001, the CL loss beta being 0.05, and scheduling and transforming strategies being the same as those used by Oswald. The embedding dimension is set to 32.
For the WSN method (Kng et al., 2022], we follow its original paper and use the default values of parameters in its official code repository. We choose the sparsity parameter = 0.5 which performs best as listed in the WSN literature. Other parameters are set to the following values: optimization via Adam, a learning rate initialized at 1c-3 with minimum of Ic-6, and a patience of 6 epochs for reducing the learning rate by a factor of 2. The models are trained for 100 epochs, with a batch size of 64 for both training and testing.
For H-embedding guided hypernet, the learning rate is set to 0.0005, with the embedding loss beta and CL loss beta both set to 0.05. The scheduling and transforming strategies are set the same as those of'yon Oswald et al. 2020) and the embedding dimension also set to 32. For the learning of H H-embedding, we update Eqn. 5 using gradient descent for 2000 iterations and the f. in HGR maximal correlation for 100 epochs respectively, both using a subset of 1000 samples from the training set.
24

---

For the H-embedding guided hypernet with LoRA, the learning rate is set to 0.001, with both the embedding loss beta and CL loss beta set to 0.05. The LoRA rank and alpha are configured at 16, resulting in about 0.7 million parameters generated by the hypernetwork, which is only 0.86% of the full ViT model's 86 million parameters. The LoRA dropout rate is set to 0.1. For CIFAR-100) a batch size of 32 and 15 epochs are used, while for ImageNet-R, a batch size of 128 and 60 epochs are used.
In all methods, we adhere to the principles of continual learning by not tuning hyperparameters on the full task set. In hypernet generation, special care was taken in handling batch normalization layers, especially in methods involving parameter freezing or pruning. We store and update batch normalization statistics separately for each task to ensure proper normalization during both training and inference. Also, to ensure that the parameter size of the hypernet framework should not exceed that of a normal network, we follow von Oswald et al.) (2020) and chunk the weights into size of 7000, using additional chunking embedding to generate each chunked weight.
A.6.2 Additional Comparison Experiments (Table
Hyperparameters with Change of Task Length. In these experiments, we vary the number of tasks and adjusted the hyperparameters accordingly evaluate their impact on performance. For scenarios involving five tasks, we set the training to span 60 epochs with a beta value of 0.006. Conversely, when the number of tasks is increased 20, the training is conducted over 30 epochs with an augmented beta value of 0.05. These adjustments are designed to optimize learning by accommodating the varying complexities introduced by different task lengths.
DomainNet Experiments. In the DomainNet settings, the number of tasks also influences the hyperparameter configuration. For experiments involving five tasks, training is executed over 40 epochs with a beta of 0.0025 to ensure adequate learning across diverse domains. When expanded to ten tasks, the training epoch is reduced to 15, concurrently increasing beta to 0.05.
A.7 Ablation Studies
A.7.1 H-embed Hnet LoRA (VTT) Ablation
Due to space limit in main text, we list the results of ablation studies ViT-LoRA backbone here in Table 5, The experiments are conducted with the same hyperparameter setting as in comparison studies.
A.7.2 H-embed Hnet- Full Generation Ablation
To broaden the comprehensiveness of evaluation and take a better concentration on validating our introduction of H-embedding guidance, conduct extra ablation studies three differed settings with different benchmarks as well as model backbones. Namely, experimental settings include: PermutedMNIST (10 tasks) using MLP model, CIFAR-100 (10 tasks) using 4-layer CNN model, and IetgeNet-R (10 tasks) using ResNet-32 model. We compared across three methods where all hyperparameters are set exactly to the same: 1) Vanilla Hnet, the hypernet CL. framework without guidance module; 2) Rand-embed Hnet, the same framework as ours but replacing H-embedding with a random embedding; 3) H-embed Hnet, our framework. The performance is evaluated and summarized in Table 6 where broad increase in CL performance could be observed across all benchmarks and backbones. Detailed information listed below.
PermutedMNIST. Considering the smaller data dimension and model size in this setting, the embedding size is reduced to 24 and the training iteration number is set to 5000. The backbone model on PermutedMNIST is selected be an MLP with fully-connected layers of size 1000, 1000 as used by Van de Ven and Tolias (2019). We configure the learning rate as 0.0001 and the embedding loss beta as 0.05. The results are derived on NVIDIA GeForce RTX 3090 GPUs.
The tasks in this setting are derived the same as comparison studies. Yet, the backbone model is differently set to a 4-layer CNN as used by Zenke et al. [2017]. We also follow Oswald most of the hyperparameters, configuring the learning rate to 0.0001, embedding size to 32, as well using the same scheduling strategies. We train each method with 100 epochs and the
C1FAR-100.
25

---

embedding loss beta is set to 0.2 for H-embed and rand-embed hypernets. The results are derived on NVIDIA GeForce RTX 3090 GPUs.
ImageNet-R. For the ImageNet-R dataset, we split the original 200 classes into ten 20-way classification tasks. Because of the uneven class sample size the ImageNet dataset, each task has varied numbers of training and test samples: Task with 2,166 training samples and 543 test samples, Task 2 with 2,655 training and 716 test samples, until Task with 2,058 training samples and 471 test samples. In method, 0.05, training the models for 200 epochs. The backbone model is the same as used in comparison experiments. The results are derived on NVIDIA A100 GPUs.
learning rate of 0.0005 and embedding loss beta of
ImageNet-R (N = 5)
ImageNet-R (N = 10)
ImageNet-R (N = 20) DAA
Method
FAAt
DAAt
FAAt
DAAt
FAAt
w/o CLreg
39.08
79.93
39.89
83.21
34.57
85.11
w/o Hemb
77.37
78.56
42.85
82.05
74.82
83.55
w/o AHP
76.37
77.23
78.37
80.02
74.72
83.47
Ours
79.72
80.10
81.65
81.86
76.91
85.05
Table 5: Ablation Studies H-embed Hnet LoRA (ViT) on ImageNet-R =N= Our framework derives the best performance on all three task lengths.
5, 10, 20).
PermutedMNIST
CIFAR-100
ImageNet-R
Setting
MLP
CNN
ResNet-32
Method
FAAT
DAAT
FAAT
DAA
FAA
DAA1
Vanilla Hnet
97.495
97.488
69.620
79.490
38.202
38.307
Rand-embed Hnet
97.448
97.447
71.250
76.490
38.046
37.956
H-embed Hnet*
97.553
97.560
72.390
76.900
39.212
39.299
Table 6: Ablation Study of H-embed Hnet Full Generation on Different Benchmarks and Backbones. Our H-embedding guidance proves to be effective across all three settings, attaining the highest FAA and DAA.
A.8 In-depth Performance Analysis
For better analysis of the effectiveness of our strategy, further investigate the detailed training behavior displayed in CL strategies, showing that our H-embedding guided hypernet is characterized by the following superiority.
Quicker Convergence. With the intention of understanding how our guidance aids the training process, we visualize the test accuracy trends during the training stage tasks 3, 6, 10 of the 10 CL tasks under Lifar-ResNe setting in Fig. 4 It is shown in the figures that, compared to hypernet without H-embedding guidance, our method converges noticeably faster and achieves a higher final accuracy performance, especially with the growth of task numbers. Such phenomenon serves as further suggestion that our H-embedding guidance provides substantial enhancement to the task learning in CL through forward transfer.
Embedding Interpretability. To assess the task embeddings {cli) learned in our framework, we compute the task-wise Euclidean distances of the embeddings obtained with and without H- embedding guidance, and visualize the discrepancy between these two distance matrices in Fig. 5 Red signifies that the with-guidance embeddings result in closer distance between the two tasks compared to the without- without-guidane, guidance embeddings, while blue represents the opposite. Take task 8,
26

---

Task 3
Task
Task 10
02-
Vanilla Most
Vanila gudded Hoat
P-cmmbedding gulded Hiot
gudded Hhot
120
Figure 4: Plotting of test accuracy during training task 3, 6, 10 of CIFAR-100, with axis x and
y for the number of checkpoints and accuracy respectively. The blue curve represents the vanilla hypernet and the orange represents our H-embedding guided hypernet. As CL progresses, our method exhibits quicker convergence to higher accuracy in later tasks.
Embedding Distance Discrepancy
Tasks
10
2
5
7
10
Tasks
Figure 5: Visualization of discrepancy between the task embedding distances learned w/ and w/o H-embedding guidance. The grid of i-th row and j-th column represents the distance of task and j. Darker cells indicate larger discrepancy, with red for d(w/) d(w/o) and blue vice versa.
CIFAR-10 split task covering classes of people and reptiles, as an instance. The embedding derived in our H-embedding guided hypernet successfully marks tasks 3, 5, 6, 7, 9 as more related, which all contain coverage of terrestrial animal classes or human scenarios. Such correspondence with human intuition suggests better capture of task interrelationships. leading higher CL efficiency.
Optimal Overall Transfer Ability. Following Sec. of all methods in comparison study and list them in Table 7 Our framework derives the best overall ability, displaying competitive performance both forward and backward transfer.
we assess the FWT and BWT performance
27

---

Task
CIFAR-100
ImageNet-R
Method
FAA (t)
BWT (t)
FWT (1)
FAA (1)
BWT (1)
FWT (t)
Finetune (ResNet)
19.13
-59.89
0.20
15.08
-41.34
19.74
LwF
34.35
-50.20
5.73
17.30
-23.36
3.39
EWC
37.83
-46.38
5,40
15.77
-12.20
-9.47
L2
41.49
-43.47
6.14
15.95
-47.82
26.34
PredKD + FeatKD
34,43
-49.29
4.91
18.22
-22.15
3,39
PackNet
70.68
(N/A)
-8.14
34.63
(N/A)
1.99
HyperNet
81.57
-0.06
2.82
38.03
-0.15
4.88
WSN
82.75
(N/A)
3.94
37.99
(N/A)
5.67
H-embed Hner*
83.08
-0.01
4.27
38.16
0.07
4.80
L2P
83.18
44.11
7.34
71.26
-4.87
3.82
DualPrompt
81,48
-4.93
6.06
68.22
-5.59
1.50
CODA-Prompt
86.31
-4.36
10.32
74.05
-4.09
5.83
HiDe-Prompt
93.48
-1.54
14.67
74.65
-3.81
6.15
InfLoRA
86.75
-4.97
11.37
74.75
-5.92
8.36
SD-LoRA 87.26
-4799
11.70
77.18
-4.56
9.43
H-embed Hnet-LoRA*
97.07
0.01
16.71
81.38
-0.29
9.36
Table 7: Accuracy (%) and Transfer (%) Comparison on CIFAR-100 and ImageNet-R. All range of results are derived by three times running with different random seeds and calculating the average. Our method (marked by **") achieves the top average accuracy with high confidence.
28


Reviews:
[{"review_id": "s7ftBigyCO", "reviewer": "Reviewer_JmhA", "summary": "The paper proposes a continual learning framework that introduces a novel transferability-aware task embedding, H-embedding, derived from an information-theoretic H-score metric to capture inter-task relationships. This embedding guides a hypernetwork to generate task-conditioned weights, achieving improved forward and backward transfer on several CL benchmarks without rehearsal.", "questions": "- Please address the questions in Weaknesses.\n- In Fig. 3a, why is the FAA performance of \"w/o Hemb\" poor for ImageNet-R 10 Tasks, but competitive for ImageNet-R 5 Tasks and 20 Tasks?\n- In Fig. 3b, why does the proposed method achieve the same DAA for both TIL and CIL versions?\n- an the H-embedding be replaced by other task embeddings, such as those in Task2Vec (Achille et al., TASK2VEC: Task Embedding for Meta-Learning. ICCV’19) or by embeddings computed with different transferability metrics? How would this choice affect performance?\n- For ViT-LoRA, is the H-score computed using the CLS token features?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The proposed framework is storage-efficient, requiring storing only low-dimensional task embeddings.\n- The proposed framework is compatible with parameter-efficient finetuning (PEFT) settings like LoRA. \n- The paper presents extensive empirical comparisons against strong baselines across multiple datasets (CIFAR-100, ImageNet-R, DomainNet) and backbones (ResNet, ViT), showing improvements in both FAA and DAA.\n\nWeaknesses:\n- The contribution is not very significant compared to HyperNet \\[von Oswald et al., 2020]. The empirical gain over HyperNet appears notable only for FAA on ImageNet-R 10 Tasks with ViT-LoRA, suggesting that the benefit from the task relation prior is limited.\n- Methodological concerns:\n\t- In Eq. (11), both the encoder and decoder are trainable and can be updated to produce outputs close to \\$\\hat{e}^{(j)}\\$, but there is no constraint ensuring these outputs are close to the true \\$e^{(j)}\\$. How does this setup ensure that \\$e^{(j)}\\$ aligns well with \\$\\hat{e}^{(j)}\\$?\n\t- Lines 168–170: The AHP-normalized transferabilities are converted to distances using a “standard affinity–distance” method. However, the given formula for \\$dist(T\\_n, T\\_j)\\$ is usually used to convert distance to affinity, not the other way around. Also, the motivation for introducing the additional learnable scale \\$\\gamma^{(j)}\\$ is unclear.\n\t- It is not clear why transferability needs to align with the symmetry property of Euclidean distance.\n- The paper lacks discussion and comparison with related work that also uses transferability to facilitate continual learning: Ermis et al. Memory Efficient Continual Learning with Transformers. NeurIPS'22.\n- High training cost: When the number of tasks is large, the method needs to generate task-specific networks, use them to compute H-scores, and calculate continual learning losses. This can significantly increase computational cost and training time.\n- Limited extension to class-incremental learning (CIL): The paper only briefly mentions using an auxiliary task ID classifier on pre-trained image features. This solution is not well integrated with the main framework, which the paper also acknowledges as a limitation.", "was_revised": true, "final_justification": "The authors have addressed my main concerns. Therefore, I am increasing my score from 3 to 4.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and detailed feedback. Below, we respond point-by-point.\n\n## (Weakness 1 & Q1)\n\nFor the concerns about HyperNet [von Oswald et al., 2020] comparison and the performance fluctuations shown in Fig. 3a, we address both problems below.\n\n - **Performance Significance Compared to HyperNet:**\n    Despite its limitations, we acknowledge that the original HyperNet is a relatively mature CL strategy, and surpassing it by a large margin is naturally more challenging than typical ablation settings. However, we believe our improvements over HyperNet as a baseline are still notable and consistent. For example in ImageNet-R benchmark, recent SOTA method (SD-LoRA) improves upon hypernet by 1.95%, while our method achieves a 4.06% improvement over SD-LoRA and 6.09% over the original HyperNet (measured by averaging across 7 seeds and 3 task splits). Also, in our experimental studies, the performance gain of our framework is consistently observed across different backbones (e.g., CNN, ResNet, ViT) and benchmarks (e.g., CIFAR100, ImageNet-R, DomainNet), suggesting that our framework offers meaningful benefit beyond HyperNet.\n\n - **Explanation for Fig. 3a Performance:**\n    Currently, Fig. 3a presents single-seed results. The performance drop of “w/o Hemb” on the ImageNet-R 10-task benchmark suggests a degree of robustness issue with vanilla HyperNet when generating LoRA parameters across different random seeds, which also appears on 5- and 20-task benchmarks. To provide a more stable and comprehensive comparison, we additionally report multi-seed (7 in total) results of all three task settings in the Table, including mean and standard deviation. It can be observed that introducing our task-guided embeddings helps stabilize the performance and improves the overall average. We plan to further investigate the underlying cause of this phenomenon in future.\n\n| #Tasks    | Vanilla Hypernet (%)  |HyperNet w/ Hemb (%) |Increasing Ratio|\n|---|--|---|---|\n|5 | 74.09 (10.99)  | 78.29 (0.43)| 5.66% |\n|10 |   76.77 (10.19)  | 80.89 (1.12)  | 5.36%  |\n|20 |74.98 (5.78)   |  80.41 (2.24) | 7.24% |\n\n - **Novelty and Methodological Contributions:**\n    While our framework adopts a HyperNet-like architecture for parameter generation, it substantially differs from the previous work in both motivation and implementation. The central contribution of our work lies in the introduction of task relation priors, which guide the dynamic adaptation of model parameters across tasks. In contrast to the vanilla hypernet, where task embeddings are learned implicitly through task-specific data, we explicitly model and compute inter-task affinities to guide the learning of task embeddings and hypernet, resulting in better generalization and stability across continual tasks.\n\n    Moreover, as part of our contribution, we reformulated the implementation of HyperNet pipeline, improving its computational efficiency and enabling it to generate only LoRA parameters. These adjustments significantly reduce both training and inference overhead, and are especially beneficial in modern CL scenarios where lightweight adaptation is critical. It also distinguishes our work from most prior uses of HyperNet, which typically generate full model parameters or require heavier storage and compute budgets.\n\n  We will clarify these points in the revised paper with Fig. 3a including multi-seed results.\n\n## (Weakness 2)\n\n### 2.1\n\nEq.(11) is not intended to align the additional output with $\\hat{𝑒}^𝑗$,  but rather to enforce alignment between $\\hat{𝑒}^𝑗$ and the transformed version of $e^j$ through the encoder-decoder mapping. In essence, this alignment ensures that the guiding task embedding $\\hat{𝑒}^𝑗$ can be well approximated by a learned transformation of $e^j$, or intuitively from an information transmission perspective, $e^j$ should retain sufficient information to recover the H-embedding $\\hat{𝑒}^𝑗$.\n\nThe encoder-decoder structure can be equivalently viewed as an autoencoder, of which the hidden layer is coupled with the intermediate layer of the hyperNet, facilitating joint optimization and promoting a consistent representation space between embeddings and hyperNet outputs. This type of alignment using autoencoders has been widely adopted in previous literature(e.g., [Schonfeld et al., 2019]). It does not require reconstruction of the original $e^j$; rather, the core idea is to provide a bridge between the prior calculated H-embedding and the embedding being learned. We described it as an encoder-decoder module to maintain conceptual clarity and avoid confusion with the core HyperNet mechanism.\n\n### 2.2\n\nWe agree that the more common formulation is: $aff = exp(-\\gamma \\times dist)$, yet we adopts the reverse mapping: $dist = exp(-\\gamma \\times aff)$. Nevertheless, this reverse form is not incorrect, and has been used in prior works too — notably in Taskonomy [Zamir et al., 2018]. Intuitively, both affinity and distance are relational measures where the relative scale matters more than the absolute value. Such a transformation is to ensure a smooth inverse relationship, and our formulation also satisfies this requirement.\n\nThe inclusion of the scaling factor $\\gamma$ further enhances this effect: it adjusts the steepness of the exponential curve, ensuring the learned embedding distance emphasizes relative differences without being dominated by absolute magnitude. This is especially important in our framework where the scale sensitivity of H-score transferability across different target tasks can affect the optimization stability of H-embedding computation.\n\n### 2.3\n\nWe acknowledge that transferability is inherently asymmetric, and does not naturally match the symmetry property of Euclidean distance. However, in our framework, this alignment remains a reasonable and effective approximation, due to the following:\n\n - **Asymmetric guidance is sufficient:** In CL, we are primarily concerned with how previous tasks affect the current task. Thus, we only require one-way transferability and inter-task relation, and symmetric alignment is not strictly necessary.\n\n - **Constraint of CL:** Since our framework operates in a strict rehearsal-free CL setting, we cannot revisit or recompute full two-way transferability after a task is finished. The one-way transferability from known tasks is the only feasible choice.\n\nOverall, we believe this approximation still provides meaningful structure for guiding embedding space learning and has been validated by empirical improvements.\n\n## (Weakness3 & Q3)\n\nIn this work, we mainly focus on the increasingly popular rehearsal-free CL setting, where access to past data is prohibited. In contrast, [Ermis et al., 2022] relies on storing both past models and selected samples, which violates our problem setting. Therefore, this work is not currently included in our main comparisons. Nevertheless, we still thank the reviewer for bringing it up and the underlying ideas are indeed related to ours. We'll include a discussion in the revised version.\n\nDespite the setting mismatch, we conducted an additional experiment. As the code for the paper is not publicly available and the full reproduction would require considerable effort, we instead implemented our method on the benchmark (CIFAR-100 20 tasks + ViT) used in Fig 3 of the original paper, with their method reporting a FAA of approximately 0.78 and our method achieving 0.907.\n\nSince the original paper does not provide detailed numerical results, we estimated their performance by reading values from the figure. Also, due to differences in dataset configurations, the current comparison may not be entirely fair. We will conduct a more rigorous and fair comparison once we have sufficient time and resources in the future.\n\nThis also partially addresses Q3. Our choice of metric was driven by practical considerations. In rehearsal-free CL, it is infeasible to store large amounts of past data or model parameters between tasks. The H-score metric stands out because it requires no source data, and the only information it needs—source model parameters—can be derived using the HyperNet from the task embedding. Hence, our framework strictly adheres to the CL constraints, while still enabling fast and theoretically grounded estimation of task relationship. Additionally, H-score is one of the most currently recognized transferability metrics in the field, known for its solid theoretical backing and strong empirical correlation with true transfer performance. We believe these make it a well-justified choice.\n\nWe acknowledge that our current study does not cover the full spectrum of possible solutions. However, the presented method offers a feasible and effective approach to enhancing CL. Moving forward, we plan to explore alternative designs and conduct a more comprehensive analysis, in terms of both accuracy&efficiency.\n\n## (Weakness 4)\n..."}, {"review_id": "4FcRYdYhGD", "reviewer": "Reviewer_Wucx", "summary": "The H score based on model parameters from multiple tasks are used to learn a sequence of embedding vectors, and the similarity of these embedding vectors is used to guide task relatedness of a hypernet network for design of the parameters for the next task. Since the hypernet parameters are updated with each new task, the model is forward and backward information flow as a function of sequential tasks. There are multiple elements to the model (it is fairly complicated, with many networks interacting), including an encoder-decoder linking model features to the aforementioned embeddings. A comprehensive set of experiments are presented, including ablation studies.", "questions": "The H-score is a key element of this model. I think the definition in (2) should be clarified. You write \"cov\" in two places, and these mean different things. I think (2) is unnecessarily ambiguous. I had to go read Huang et al (2019) to understand it, and I think they gave a clearer definition. Please make clearer in a revision.\n\nHow well is this model going to scale for large \\theta^{(n)}? It seems hard for the Hypernet to work well for very large models. \n\nThere are a lot of \"moving parts\" to this model. In (9) there is \\beta_e and \\beta_c, and then from Sec 4.2.2 there is encoder-decoder. It would seem that this model could get into local modes, possibly poor ones, given the complexity of this model, particularly for large \\theta^{(n)}. Also, the model could be very dependent on the dimension of e^{(j)} and its characteristics. Please comment.", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nOverall, the paper is well written, but there are some elements that could be clarified (questions below).\nThe experiments are strong.\n\nWeaknesses:\n\nWhile the model makes sense, it is quite complicated. It is not clear how well this will scale to very large models, and large \\theta. Additionally, key to the whole thing is the dependence of the hypernet on e^{(j)} and the ability of the model to infer task interrelatedness, via the H-score. How sensitive are the results to the dimension of e^{(j)}? Since e^{(j)} plays such an important role, it seems its characteristics are important. This seems to be why so much regularization is done, for example via the encoder-decoder.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s positive evaluation and the insightful feedback provided on our work. Below, we address each of the points in detail.\n\n## (Weakness 1 & Question 2)\n\nWe agree that scaling full-parameter generation via hypernetworks to very large models can raise efficiency concerns. To address this, one of the key contributions of our work is to combine current hypernet frameworks with parameter-efficient fine-tuning (PEFT) techniques. Specifically, we employ the hypernet to generate LoRA parameters instead of full weights in our current design, with potential for future extensions to other module types. This design enables our method to be flexibly integrated into a wide range of large-scale models as a plug-in module, allowing us to leverage the capabilities of powerful pretrained backbones while keeping the hypernetwork lightweight and efficient.\nAs one of our main comparison studies, the HyperNet-LoRA framework has demonstrated strong and stable performance across our experiments, confirming the effectiveness of this design.\n\n## (Weakness 2)\n\nThe learned embedding indeed plays a critical role in hypernetwork-based frameworks, which is also one of the reason we propose to guide its learning through a transferability-aware objective. Yet, in our framework, the performance is relatively insensitive to the embedding dimensionality as a hyperparameter. To support this, we conducted experiments with embedding dimensions set to 8, 16, 24, and 32 on ImageNet-R 10 task ViT setting. As shown in the table below (`*' denotes the setting in experimental results), although FAA exhibits slight fluctuations across settings, the overall performance remains stable. This suggests that our method is relatively robust to changes in embedding size, and does not rely on fine-tuning this hyperparameter to achieve good results.\n\n| Embedding Size    | FAA |\n|----------------|------|\n|8 | 80.245 |\n|16 | 81.777  |\n|24 |   80.996  |\n|32*| 80.970    |\n\nWe appreciate the reviewer’s insightful comment regarding the characteristics of the embedding. We agree that this aspect is indeed worth further investigation, as it could provide valuable insights into the behavior of transferability-aware hypernetworks. We hope to conduct a more comprehensive study in this direction in future work.\n\n## (Question 1)\n\nWe thank the reviewer for the valuable feedback and agree that the definition in Eq. (2) can be clarified further. In future revisions, we will provide a more intuitive explanation of the formula to enhance its readability. Although the two instances of \"cov\" might suggest different interpretations, they are consistent from a computational standpoint — both denote the covariance computed over the first dimension (i.e., across the observations) of the input data. We will explicitly clarify this point in the revised manuscript.\n\n## (Question 3)\n\nWe appreciate the reviewer’s thoughtful observation. Firstly, we would like to clarify that compared to typical deep learning methods, our framework does not introduce significantly more “moving parts”. The two hyperparameters $\\beta_e$ and $\\beta_c$ serve as standard loss balancing factors, which are commonly adopted in many existing works; the encoder and decoder modules are designed as very shallow linear networks, jointly trained with the main HyperNet, making the overall architecture relatively straightforward and within the scope of typical deep learning designs.\n\nTo further illustrate this, we provide a parameter count comparison showing that the total parameter size of our HyperNet framework is comparable to that of conventional model architectures. The connections involved are also quite simple, and the training is end-to-end. So far, we have not observed any indications of local minima or poor local optima issues during experiments. Regarding the sensitivity to the hyperparameters $\\beta_e$ and $\\beta_c$, our empirical results suggest that the framework is relatively robust to their variations. Below we summarize the performance under different settings of these hyperparameters on ImageNet-R 10 task ViT setting (`*' denotes the setting in experimental results).\n\n|beta_c \\ beta_e | 0.01| 0.03|0.05* |0.07|\n|---|---|---|---|---|\n|0.01 |0.8188 |0.8169| 0.8117 |0.8132|\n|0.03| 0.7686| 0.7788 |0.7771 |0.7850|\n|0.05*| 0.7851| 0.7755 |0.7892 |0.7850|\n|0.07| 0.7763| 0.7882 |0.7852 |0.7878|"}, {"review_id": "mYq6nKUrm7", "reviewer": "Reviewer_ZN1R", "summary": "This paper explores how the H-score can be leveraged to model task relationships in continual learning (CL). It introduces an online task embedding mechanism called H-embedding, which feeds into a hypernetwork. The hypernetwork, trained jointly with the embeddings, predicts the weights of the inference model directly from these embeddings. The method is evaluated on CL benchmarks across various vision architectures, including ResNets, ViTs, and ViTs+LoRA adapters.", "questions": "The paper presents ideas that are theoretically grounded, though at times they come across as informed guesses about what might work best. That said, the results are strong, and the paper is both technically solid and well-written.\n\nI encourage the authors to answer the points presented in the Strengths/Weaknesses section of this review, particularly the concerns around understanding and controlling the behavior of the proposed framework. For now, I see no reason for a lower the score, and I’m open to revising my assessment after considering the authors' response and other reviewers’ comments.", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper is formally sound and presents promising results. However, the contribution appears to rely primarily on combining existing components (i.e. adding modules and optimizing loss functions) without a clearly articulated breakthrough. While the empirical improvements are appreciated, the core mechanisms driving these gains remain somewhat opaque.\n\nFor instance,\n\n- In the ablation study (Fig.3), it is unclear why removing the H-embedding is detrimental at $N=10$, but has negligible impact at $N=5$ and $N=20$.\n\n- Similarly, the AHP component seems to offer little performance benefits, yet it is retained in the final framework without sufficient justification.\n\nAlso, to better understand the contribution of your work, I would appreciate clarifications on the following points:\n\n- The paper seemingly is missing pointers clearly stating from which papers the benchmarks are inherited from (i.e., the baselines are all cited, but it could be useful to also understand if the specific dataset-architecture pairs were already studied in the literature, or these are novel test beds the authors are proposing)\n\n- The authors mention re-running some methods while transcribing results of others from the literature. Could the authors clarify which results were reproduced versus taken directly?\n\n- Have the authors considered comparing methods in terms of inference-time efficiency (both memory usage and runtime)? This would provide a full picture of the practical trade-offs to justify this added complexity.\n\n- What are the computational costs (in both memory and time) of training the Hypernetwork component?", "was_revised": true, "final_justification": "All of my primary concerns have now been resolved. While I agree that some reviewer remarks deserve further discussion, they are minor and can be handled during the camera-ready stage. Therefore, I increased my score.", "rebuttal": "We are grateful for the reviewer’s encouraging comments and thoughtful suggestions, and we address the raised points in detail below.\n## (Weakness 1)\n\nThank you for your positive comments on the soundness and empirical results of our paper. We would like to further clarify the novelty and motivation behind our method. Although our framework involves known components (e.g., the general architecture of a hypernetwork for parameter generation), it introduces key conceptual and technical innovations that set it apart from prior works. As stated in the Introduction, the key insight of our work is that task relationship can provide prior information of the task space structure in the continual learning context. Upon reviewing prior literature, we observed that most existing strategies are primarily performance-driven and heavily rely on training procedures, with limited consideration of prior inter-task information. Motivated by this gap, we proposed the use of task embeddings to guide the learning process, and designed a transferability-aware task embedding that is compatible with continual learning constraints—computed without accessing past data or models—thereby providing a principled way to inject task structure priors into the training process. Specifically, on the basis of the transferability metric H-score, we 1) proposed to use HGR divergence as a unified definition of self transferability conforming to the H-score theory framework; 2) introduced AHP normalization for transferability scale alignment across different tasks; 3) defined an optimization problem to distill the transferability information into a low-dimensional task embedding，enabling the incorporation of task-level information into continual learning strategies.\n\nMoreover, our use of existing components is not a simple reuse or combination; rather, we carefully adapt and integrate them into our framework to serve our specific goals. For instance, compared to conventional applications of HyperNetworks in continual learning, we reformulated the HyperNet pipeline to improve its computational efficiency and enabled it to generate only LoRA parameters. These modifications substantially reduce both training and inference overhead, which is particularly valuable in modern continual learning scenarios that demand lightweight adaptation, setting our approach apart from most previous uses of HyperNetworks, which typically generate full model weights and thus incur significantly higher storage and compute costs.\n\nAs for the listed concerns, we address them below:\n\n - **Explanation for Fig. 3a Performance Variation and Impact of Introducing H-Embedding:**\n\n   In the current version, Fig. 3a presents single-seed results. The performance drop of FAA “w/o Hemb” on the ImageNet-R 10-task benchmark suggests a degree of robustness issue with vanilla HyperNet when generating LoRA parameters across different random seeds, which also appears on 5-task and 20-task benchmarks. To provide a more stable and comprehensive comparison, we additionally report multi-seed results of all three task settings in the Table, including mean and standard deviation. It can be observed that introducing our task-guided embeddings helps stabilize the performance and improves the overall average. We plan to further investigate the underlying cause of this phenomenon in the future.\n\n| Benchmarks    | Vanilla Hypernet (%)  |HyperNet with H-Embedding Guidance (%) |Increasing Ratio|\n|----------------|------|-----------|---------|\n|ImageNet-R 5 task | 74.09 (10.99)  | 78.29 (0.43)| 5.66% |\n|ImageNet-R 10 task |   76.77 (10.19)  | 80.89 (1.12)  | 5.36%  |\n|ImageNet-R 20 task |74.98 (5.78)   |  80.41 (2.24) | 7.24% |\n   \n   Regarding the significance of performance improvement, in average our improvements compared to removing H-embedding are notable and consistent. For example, in ImageNet-R benchmark, recent state-of-the-art method (SD-LoRA) improves upon hypernet by around 1.95%, while our method achieves an additional 4.06% improvement over it and 6.09% over the original HyperNet (measured by averaged performance across 7 seeds and 3 task splits). Also, in our experimental studies, the performance gain of our framework is consistently observed across different backbones (e.g., CNN, ResNet, ViT) and benchmarks (e.g., CIFAR100, ImageNet-R, DomainNet), suggesting that our task-relation-guided prior offers meaningful benefit.\n\n   We will clarify these points in the revised paper and include modify Fig. 3a to include multi-seed results for to better illustrate robustness.\n\n - **AHP normalization:**\n\n   As mentioned in our response regarding the design of H-embedding, the introduction of AHP is primarily motivated by the need to perform transferability scale alignment across different tasks, which we believe is an intuitive and interpretable design. This design helps prevent scale mismatch with the embedding distances and ensure that the transferability signals remain comparable across tasks when used as guidance. \n\n   In essence, AHP serves as a normalization post-processing step for the transferability values, rather than introducing new information. This may partly explain its relatively modest quantitative impact. Nevertheless, incorporating AHP still leads to an empirical performance gain of approximately 4% (averaged across 3 task splits). While this improvement is not decisive, it demonstrates that AHP provides a non-trivial contribution to the final performance and justifies its inclusion in the framework.\n\n## (Weakness 2)\n\nRegarding the experimental setup, we would like to clarify that all our experimental benchmarks—both the datasets and their specific splits—are directly adopted from existing works that are well-established and repeatedly used in the recent literature (e.g. [Liang and Li, 2024], [Wu et al., 2025]). The architectures we employ, ViT-B/16 and ResNet-32, are also widely used and considered representative choices in prior studies. Therefore, we do not propose any new test beds in this work; this design choice was intentional to ensure that comparisons with previous methods are fair and objective.\n\n## (Weakness 3)\n\nThe baselines for full-parameter ResNet experiments (including CIFAR-100 and ImageNet-R benchmarks) were reproduced by us, while the baselines for ViT-based parameter-efficient fine-tuning (PEFT) were cited from existing literature. In particular, we mainly cited results from the SD-LoRA [Wu et al., 2025] paper (including L2P, DualPrompt, CODA-Prompt, HiDe-Prompt, InfLoRA, and SD-LoRA), and the results of HiDe-Prompt on CIFAR-100 benchmark was taken from its original paper.\n\nThis choice was made because the full-parameter methods span a relatively long time range, and many of them do not share a unified benchmark setting or backbone architecture. To ensure fairness and consistency, we reproduced all these baselines under our evaluation framework. In contrast, most of the ViT-PEFT methods are recent works with more consistent experimental setups, making it more reasonable to cite their results directly.\n\n## (Weakness 4)\n\nDespite its seemingly complexity, the hypernet framework does not incur significant time costs or memory usage during inference, as the number of parameters in the hypernetwork has been strictly controlled. In the original hypernet framework, the number of hypernet parameters is roughly comparable (or smaller) to that of the main network. Therefore, if processing one sample at a time, a single prediction involves one forward pass through the hypernet and one through the main net, theoretically leading to an inference time approximately twice that of a regular model. However, in practice, we only need to perform one forward pass through the hypernet per task, which means that the amortized time cost per sample becomes negligible when the dataset is large. The situation is even more favorable in the ViT-LoRA setting, where the hypernet is only responsible for generating LoRA parameters. Since these modules are much smaller in size than the full ViT model, the forward computation of the hypernet contributes only marginally to the overall inference time. We conducted runtime tests for both the ResNet and ViT backbones using full testing set, and the results support our analysis.\n\n| Backbone & Benchmark | Time Cost of Plain Backbone Model (s) | Time Cost of H-Embedding Guided Hypernet Framework (s) | Parameter Size of Plain Backbone Model |Parameter Size of H-Embedding Guided Hypernet Framework |\n|----------|---|---|---|---|\n|ResNet on Cifar100|4.257|4.260| 468540 | 457336 |\n|ViT on ImageNet-R |4.997| 5.041| 743624(LoRA) | 458424 |\n\nWe believe that the comparison of inference time and parameter size between our method and the plain backbone already demonstrates the efficiency of our framework at inference time. Due to time constraints in the rebuttal phase, we were unable to conduct a comprehensive comparison with all baseline methods, but we hope the above analysis and measurements sufficiently support the claim that our method introduces minimal inference overhead.\n\n## (Weakness 5)\n\nFor similar reasons as stated in analysing the inference efficiency of our framework, the training computational costs of training hypernet is also relatively minor. We list the results of 1-epoch runtime tests of ViT-LoRA backbone on ImageNet-R 10 task benchmark below:\n\n| Task ID    | 1  | 2 | 3| 4|5|6|7|8|9|10|\n|----------------|------|-----------|---------|---|---|---|---|---|---|---|\n| HyperNet Total (s) |  4.68 | 5.67 | 5.02 |6.24|5.67|6.18|5.36|6.26|5.63|5.46|\n| LoRA Finetune Total (s)  | 4.37| 5.16|4.41|5.40|4.90|5.17|4.41|4.94|4.41|4.15|\n\nAs shown, the 1-epoch training time of our method is slightly higher than that of plain LoRA, but the difference is minor and acceptable. Given their similar parameter sizes, the memory usage is also within a reasonable range."}]

Author rebuttal:

Meta-review: This paper extends the continual learning with hypernetworks (CLH) framework by introducing an additional objective that helps model the relationships between tasks, compared to simply learning task embeddings end-to-end as in the original CLH paper. The experiments are conducted on an array of well-chosen task families, and show some performance improvements over the original CLH method.

The reviewers generally found the paper to be well-written and appreciated its strong set of experiments. While the improvements over vanilla CLH may not appear to be very impressive, they are still significant, and CLH is a strong baseline method for the continual learning problems considered by the authors. I'm voting for acceptance.
Decision: Accept (poster)