Title: Exploiting Task Relationships in Continual Learning via Transferability-Aware Task Embeddings
Venue: NeurIPS 2025
Accepted: yes
Keywords: Continual learning

Transferability

Hypernetworks
Abstract: Continual learning (CL) has been a critical topic in contemporary deep neural network applications, where higher levels of both forward and backward transfer are desirable for an effective CL performance. Existing CL strategies primarily focus on task models — either by regularizing model updates or by separating task-specific and shared components — while often overlooking the potential of leveraging inter-task relationships to enhance transfer. To address this gap, we propose a transferability-aware task embedding, termed H-embedding, and construct a hypernet framework under its guidance to learn task-conditioned model weights for CL tasks. Specifically, H-embedding is derived from an information theoretic measure of transferability and is designed to be online and easy to compute. Our method is also characterized by notable practicality, requiring only the storage of a low-dimensional task embedding per task and supporting efficient end-to-end training. Extensive evaluations on benchmarks including CIFAR-100, ImageNet-R, and DomainNet show that our framework performs prominently compared to various baseline and SOTA approaches, demonstrating strong potential in capturing and utilizing intrinsic task relationships. Our code is publicly available at \url{https://github.com/viki760/Hembedding_Guided_Hypernet}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores how the H-score can be leveraged to model task relationships in continual learning (CL). It introduces an online task embedding mechanism called H-embedding, which feeds into a hypernetwork. The hypernetwork, trained jointly with the embeddings, predicts the weights of the inference model directly from these embeddings. The method is evaluated on CL benchmarks across various vision architectures, including ResNets, ViTs, and ViTs+LoRA adapters.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is formally sound and presents promising results. However, the contribution appears to rely primarily on combining existing components (i.e. adding modules and optimizing loss functions) without a clearly articulated breakthrough. While the empirical improvements are appreciated, the core mechanisms driving these gains remain somewhat opaque.\n\nFor instance,\n\n- In the ablation study (Fig.3), it is unclear why removing the H-embedding is detrimental at $N=10$, but has negligible impact at $N=5$ and $N=20$.\n\n- Similarly, the AHP component seems to offer little performance benefits, yet it is retained in the final framework without sufficient justification.\n\nAlso, to better understand the contribution of your work, I would appreciate clarifications on the following points:\n\n- The paper seemingly is missing pointers clearly stating from which papers the benchmarks are inherited from (i.e., the baselines are all cited, but it could be useful to also understand if the specific dataset-architecture pairs were already studied in the literature, or these are novel test beds the authors are proposing)\n\n- The authors mention re-running some methods while transcribing results of others from the literature. Could the authors clarify which results were reproduced versus taken directly?\n\n- Have the authors considered comparing methods in terms of inference-time efficiency (both memory usage and runtime)? This would provide a full picture of the practical trade-offs to justify this added complexity.\n\n- What are the computational costs (in both memory and time) of training the Hypernetwork component?\n\nQUESTIONS:\nThe paper presents ideas that are theoretically grounded, though at times they come across as informed guesses about what might work best. That said, the results are strong, and the paper is both technically solid and well-written.\n\nI encourage the authors to answer the points presented in the Strengths/Weaknesses section of this review, particularly the concerns around understanding and controlling the behavior of the proposed framework. For now, I see no reason for a lower the score, and I’m open to revising my assessment after considering the authors' response and other reviewers’ comments.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe H score based on model parameters from multiple tasks are used to learn a sequence of embedding vectors, and the similarity of these embedding vectors is used to guide task relatedness of a hypernet network for design of the parameters for the next task. Since the hypernet parameters are updated with each new task, the model is forward and backward information flow as a function of sequential tasks. There are multiple elements to the model (it is fairly complicated, with many networks interacting), including an encoder-decoder linking model features to the aforementioned embeddings. A comprehensive set of experiments are presented, including ablation studies.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nOverall, the paper is well written, but there are some elements that could be clarified (questions below).\nThe experiments are strong.\n\nWeaknesses:\n\nWhile the model makes sense, it is quite complicated. It is not clear how well this will scale to very large models, and large \\theta. Additionally, key to the whole thing is the dependence of the hypernet on e^{(j)} and the ability of the model to infer task interrelatedness, via the H-score. How sensitive are the results to the dimension of e^{(j)}? Since e^{(j)} plays such an important role, it seems its characteristics are important. This seems to be why so much regularization is done, for example via the encoder-decoder.\n\nQUESTIONS:\nThe H-score is a key element of this model. I think the definition in (2) should be clarified. You write \"cov\" in two places, and these mean different things. I think (2) is unnecessarily ambiguous. I had to go read Huang et al (2019) to understand it, and I think they gave a clearer definition. Please make clearer in a revision.\n\nHow well is this model going to scale for large \\theta^{(n)}? It seems hard for the Hypernet to work well for very large models. \n\nThere are a lot of \"moving parts\" to this model. In (9) there is \\beta_e and \\beta_c, and then from Sec 4.2.2 there is encoder-decoder. It would seem that this model could get into local modes, possibly poor ones, given the complexity of this model, particularly for large \\theta^{(n)}. Also, the model could be very dependent on the dimension of e^{(j)} and its characteristics. Please comment.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a continual learning framework that introduces a novel transferability-aware task embedding, H-embedding, derived from an information-theoretic H-score metric to capture inter-task relationships. This embedding guides a hypernetwork to generate task-conditioned weights, achieving improved forward and backward transfer on several CL benchmarks without rehearsal.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The proposed framework is storage-efficient, requiring storing only low-dimensional task embeddings.\n- The proposed framework is compatible with parameter-efficient finetuning (PEFT) settings like LoRA. \n- The paper presents extensive empirical comparisons against strong baselines across multiple datasets (CIFAR-100, ImageNet-R, DomainNet) and backbones (ResNet, ViT), showing improvements in both FAA and DAA.\n\nWeaknesses:\n- The contribution is not very significant compared to HyperNet \\[von Oswald et al., 2020]. The empirical gain over HyperNet appears notable only for FAA on ImageNet-R 10 Tasks with ViT-LoRA, suggesting that the benefit from the task relation prior is limited.\n- Methodological concerns:\n\t- In Eq. (11), both the encoder and decoder are trainable and can be updated to produce outputs close to \\$\\hat{e}^{(j)}\\$, but there is no constraint ensuring these outputs are close to the true \\$e^{(j)}\\$. How does this setup ensure that \\$e^{(j)}\\$ aligns well with \\$\\hat{e}^{(j)}\\$?\n\t- Lines 168–170: The AHP-normalized transferabilities are converted to distances using a “standard affinity–distance” method. However, the given formula for \\$dist(T\\_n, T\\_j)\\$ is usually used to convert distance to affinity, not the other way around. Also, the motivation for introducing the additional learnable scale \\$\\gamma^{(j)}\\$ is unclear.\n\t- It is not clear why transferability needs to align with the symmetry property of Euclidean distance.\n- The paper lacks discussion and comparison with related work that also uses transferability to facilitate continual learning: Ermis et al. Memory Efficient Continual Learning with Transformers. NeurIPS'22.\n- High training cost: When the number of tasks is large, the method needs to generate task-specific networks, use them to compute H-scores, and calculate continual learning losses. This can significantly increase computational cost and training time.\n- Limited extension to class-incremental learning (CIL): The paper only briefly mentions using an auxiliary task ID classifier on pre-trained image features. This solution is not well integrated with the main framework, which the paper also acknowledges as a limitation.\n\nQUESTIONS:\n- Please address the questions in Weaknesses.\n- In Fig. 3a, why is the FAA performance of \"w/o Hemb\" poor for ImageNet-R 10 Tasks, but competitive for ImageNet-R 5 Tasks and 20 Tasks?\n- In Fig. 3b, why does the proposed method achieve the same DAA for both TIL and CIL versions?\n- an the H-embedding be replaced by other task embeddings, such as those in Task2Vec (Achille et al., TASK2VEC: Task Embedding for Meta-Learning. ICCV’19) or by embeddings computed with different transferability metrics? How would this choice affect performance?\n- For ViT-LoRA, is the H-score computed using the CLS token features?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}