Title: Contrastive Consolidation of Top-Down Modulations Achieves Sparsely Supervised Continual Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: cortical learning

top-down modulations

continual learning

self-supervised learning

semi-supervised learning
Abstract: Biological brains learn continually from a stream of unlabeled data, while integrating specialized information from sparsely labeled examples without compromising their ability to generalize.
Meanwhile, machine learning methods are susceptible to catastrophic forgetting in this natural learning setting, as supervised specialist fine-tuning degrades performance on the original task.
We introduce task-modulated contrastive learning (TMCL), which takes inspiration from the biophysical machinery in the neocortex, using predictive coding principles to integrate top-down information continually and without supervision.
We follow the idea that these principles build a view-invariant representation space, and that this can be implemented using a contrastive loss.
Then, whenever labeled samples of a new class occur, new affine modulations are learned that improve separation of the new class from all others, without affecting feedforward weights. 
By co-opting the view-invariance learning mechanism, we then train feedforward weights to match the unmodulated representation of a data sample to its modulated counterparts. 
This introduces modulation invariance into the representation space, and, by also using past modulations, stabilizes it.
Our experiments show improvements in both class-incremental and transfer learning over state-of-the-art unsupervised approaches, as well as over comparable supervised approaches, using as few as 1% of available labels.
Taken together, our work suggests that top-down modulations play a crucial role in balancing stability and plasticity.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of catastrophic forgetting, especially in cases where models need to learn from both unsupervised and supervised data. To solve this problem, this paper introduces task-modulated contrastive-learning (TMCL). TMCL has a feedforward backbone that used contrastive loss for unsupervised learning. TMCL also has a network that is trained to learn modulations to implement supervised learning of object identities (while the feedfoward backbone is frozen). To learn modulation-invariant representations, the feedforward network is subsequently trained to with a contrastive loss to bring together representations of differently modulated views of the same image and push those representations away from the unmodulated representation of another image. They test TMCL against state-of-the-art alternative models, and their model performs in the same ballpark (often better than the other models, but generally within a few percentage points).\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- This paper tackles catastrophic forgetting, a long-standing and important issue in machine learning. Their approach to understanding catastrophic forgetting (noting that ecological learning is generally unsupervised with limited access to supervised labels) is also a refreshing approach.\n- I appreciate the biological grounding of the model.  The authors draw inspiration from complementary learning systems theory, mapping fast learning to task-driven modulation and slow learning to unsupervised consolidation. This biological framing is conceptually interesting and helps motivate the model architecture. Ultimately, I don't think this *needs* to be the best performing model to be interesting. It is simply interesting that this different approach can still compete with other state-of-the-art models.\n- I generally think that contrastive learning is a very promising approach. Contrastive learning has lost a little popularity as MAE approaches have gained popularity, but I'm still quite enthusiastic about the contrastive learning framework. This paper adapts contrastive learning in a novel way.\n- TMCL performs comparably (and sometimes slightly better) than existing state-of-the-art models across several tasks.\n\n## Weaknesses\n1. One of my main concerns is clarity in terms of the model description. The high-level description of the model is difficult to follow, particularly early in the paper. Key details about how the system works only become clear around page 5 (lines 156–166), which might be too late to engage the reader effectively. Figure 2, while clearly intended to clarify the architecture, is difficult to parse. The relationship between the top and bottom panels is unclear, and labels like “Learning Pathway: modulations” do not meaningfully aid interpretation. I also do not have a confident grasp on what the architecture of the modulated network looks like.\n2. The logic behind how contrastive training of modulated views leads to an unmodulated representation space with orthogonal class identities isn’t clearly justified. This seems to be an important theoretical claim, but it’s not adequately walked through. (The authors claim that this is true \"intuitively,\" line 158).\n3. Tables 2 & 3 suggest the model is performing similarly to others, with modest gains (~1–2%). This makes the paper feel like another iteration in the pattern of marginal improvement papers, rather than a conceptual leap forward.\n4. The ablation results raise some questions and concerns for me. While it’s great to see ablations included, some of the results are unintuitive or potentially undermining. For example, untrained/random modulations perform nearly as well as trained ones. Also, performance is higher without pred. AND stop grad than just without pred. These results are not well explained and leave open questions about what components are actually contributing to the model’s success.\n\nQUESTIONS:\nTo make this easier to follow and respond to during the rebuttal, I’ve ordered my questions and suggestions to correspond with the points raised in the Weaknesses section above.\n\n1. Model clarity\n\n     a) I'd like to see the model explained fully and more clearly earlier in the paper. This would help readers have context around Figures 1 & 2 and even Related Work. I realize that it is rare to put a model description before Related Work, but page 5 is quite late to fully explain the model, and readers might be better equipped to put the Related Work in context if they already understand the model.\n     b) I suggest revising Figure 2 to make it easier for readers to follow. \n     c) What is the architecture of the modulated network?\n\n2. Orthogonal representations of class identities\n\n     Why does contrastive training of modulated views lead to an unmodulated representation space with orthogonal class identities? Can the authors walk this logic out in the paper more clearly?\n\n3. Marginal performance gains\n\n     Do the authors believe that their model is promising because of superior performance or because TMCL has novel biological inspiration and still performs comparably to state-of-the-art models? If the latter, I suggest explicitly making this argument in-text.\n\n4. Confusing ablation results\n\n     a) Why do untrained/random modulations perform nearly as well as trained modulations? The authors note that \"This suggests that predicting representations based on a slightly perturbed model state already provides a regulatory effect.\" However, this does not sufficiently clarify why performance is so close to non-ablated VI+MI performance. \n     b) The authors claim \"Removing the stop-gradient while keeping the modulations frozen results in a moderate performance degradation,\" but that \"moderate\" degradation is only 0.7%. If the stop-gradient makes so little difference, does this call into question the necessity of a \"hypothesized cortical predictive coding pathway\"? \n     c) Why is performance higher without pred. **and** stop grad than just without pred.?\n\nOne last question unrelated to the weaknesses: The paper claims that \"information [arriving] at spatially segregated loci on sensory neurons [suggests] distinct roles in shaping the neuronal input-output relation\" and distinct plasticity processes (page 3). Can the authors clarify why information arriving at different loci should imply different plasticity processes?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMotivated by the continual learning ability of biological brains, this submissions considers the problem of learning good representations from mostly unlabelled data. The authors propose a representation learning method that uses a contrastive loss to learn modulations that orthogonalize representations. These modulations are then used to consolidate the data into the weights of the neural network. To evaluate the proposed method, the authors compare their method on different scenarios using CIFAR100, including on some downstream tasks.\n\n\nDecision: My current assessment is a weak rejection with room for improvement. While the submission provides some insight and develops a generally applicable method for improving representation quality, it is held back by some clarity issues and experiments which are not entirely convincing.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The idea of learning transformations to orthogonalize the representation is compelling and seems effective in the experiments presented.\n- Motivation: the authors make it clear that their motivated by understanding how learning algorithms can achieve the same continual learning ability as biological brains.\n### Weaknesses\n- While the experiments cover many different scenarios, the base dataset in all cases is CIFAR 100 can be limiting in its complexity and difficulty compared to ImageNet variants like tiny-ImageNet.\n- It is not detailed exactly what this algorithm does. The loss and architecture are clear, but it is never put together in a way that makes it clear how the modulations are being learned and used.\n- Minor but relevant given the motivation: Some choices in the submission add up to undercut the original biological motivation. For example, rather than a streaming setting as originally mentioned, the experiments are in the batch setting. Rather than alternating consolidation and task-learning, the model is given special phases in each task, one at the beginning for consolidation and then one for consolidation.\n\nQUESTIONS:\n- The technical meaning of the term \"modulation\" and \"top-down\" is never defined. In equation (1) and (2) it seems to be defined as additional elementwise multiplicative and additive parameters, which play a specific role in representing task information. But I would prefer to see the difference from the usual weights more explicit. \"BatchNorm [...] is equivalent to learning affine modulations\" This adds to my confusion because the weights for batchnorm are treated as elementwise trainable weights, carrying no semantic meaning about task information.\n- I see some discussion in reference to parameter-efficient fine tuning methods like refs 30 and 31. But what is the relationship between this method of adapting parameters to task and approaches that factorize matrices, such as low rank adapters?\n- \"modulations (‘How did we solve the task?’), and do not investigate methods with exemplar replay.\" -> it would still be interesting to compare results to replay methods for completeness.\n- Are the modulations and weights trained in completely independent phases?\n- If there are two phases, how is the data split between the phases? How can you train with the contrastive loss in the \"task-learning\" phase if it seems to require orthogonalization between classes, when the data is mostly unlabelled?\n- I like the idea of orthogonalizing the modulations of each class, but I wonder if this representation may impede effective generalization in cases where it is not needed, such as in a single task-setting? (It is fine if this is the case, but it is not clear to me one way or another)\n- Line 172: what is the rationale for using the average of the views here?\n- Table 1: it is comparatively less clear what the set of modulations are, how they are sampled, and why there are an equal number of modulations (+1 for unmodulated) and augmentations (since both are indexed by $v$. Does every single instantiation of an augmentation (e.g., rotate by x degrees), correspond to a unique modulation m_x?\n- Some theoretical justification for this approach would help clarify what the contribution of this method is beyond the presented motivation.\n\n### Minor Comments\n- Why is the term \"session\" used periodically when elsewhere the term \"task\" is used? They seem interchangeable, and should be standardized for clarity.\n- line 177 (typo): appling -> applying\n- Table 2 and 3 would be better served by emphasizing the proposed method, such as adding (Ours), to make the results stand out.\n- $\\mathcal{L}_{CL}$ is mentioned in the text but never defined, is it equivalent to $\\mathcal{L}_{BT}$?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a biologically inspired ML method for continual learning. The paper combines contrastive learning and orthogonal representation learning to mitigate forgetting and learn from partially labelled data in a largely unsupervised data stream. Results on the CIFAR-100 dataset show improvement over other baseline CL methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) The paper covers the biological concepts quite well and provides a strong motivation for biologically inspired learning methods for continual learning. \n2) The paper considers the problem of semi-supervised continual learning, where most of the data is unlabelled, which is closer to realistic settings.\n3) A large number of experimental results are reported on CIFAR-100 that show improvement compared to a few other models.\n\nWeaknesses:\n1) While the model does combine some well-known concepts into a new method, how does it compare to other biologically inspired methods? For example, SI [b], FearNet [a], CBCL [c], [d, e], to name a few. There have been a lot of CL methods proposed that have been biologically inspired. How does the current method compare to those methods? And more importantly, what is the novelty of the proposed approach compared to prior works?\n2) The proposed work has only been compared against a few baselines from the literature. How does this model compare against well-known CIL methods, such as replay-based, regularization-based, generative replay-based, etc., methods?\n3) From tables 2 and 3, I do not see significant changes in performance by adding different modules to VI. This might mean that many of these modules are not contributing much to the overall performance, and it could be because of using the ViT architecture compared to a CNN, such as ResNet-18. How does the model perform with ResNet-18? The paper states in L202-23 that ViT was chosen because of its ability to mitigate forgetting. However, ViT is not the main contribution of this work, so it would be interesting to see the contribution of the backbone compared to the proposed biologically inspired modules.\n\n[a] FearNet: Brain-Inspired Model for Incremental Learning, ICLR 2020.\n[b] Zenke, F., Poole, B., & Ganguli, S. (2017, July). Continual learning through synaptic intelligence. In International conference on machine learning (pp. 3987-3995). PMLR.\n[c] Ayub, A., & Wagner, A. R. (2023). Cbcl-pr: A cognitively inspired model for class-incremental learning in robotics. IEEE Transactions on Cognitive and Developmental Systems, 15(4), 2004-2013.\n[d] Madireddy, S., Yanguas-Gil, A., & Balaprakash, P. (2023, November). Improving performance in continual learning tasks using bio-inspired architectures. In Conference on Lifelong Learning Agents (pp. 992-1008). PMLR.\n[e] Wang, L., Zhang, X., Li, Q., Zhang, M., Su, H., Zhu, J., & Zhong, Y. (2023). Incorporating neuro-inspired adaptability for continual learning in artificial intelligence. Nature Machine Intelligence, 5(12), 1356-1368.\n\nQUESTIONS:\nPlease see the weaknesses section for my questions.\n\nLIMITATIONS:\nLimitations have been discussed adequately. However, positive and negative impacts have not been discussed. This should be added in the paper as there can be a variety of impacts that can be discussed for the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a biologically inspired approach for semi-supervised continual learning, called Task-Modulated Contrastive Learning (TMCL). Inspired by the segregated cortical architecture of top-down and feedforward pathways, the method uses modulations (learned affine transformations) to integrate sparse supervisory signals into a contrastive learning framework. The key insight is that top-down modulations learned from few labeled examples can be consolidated into a modulation-invariant representation space through predictive coding-like contrastive losses. This allows learning new concepts with minimal forgetting, even in class-incremental learning without task identity. The method is extensively evaluated on CIFAR-100 under varying supervision regimes and transfer learning tasks.\n\nThe paper is generally well-written.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:The orthogonal projection loss (OPL) to train class-specific modulations, and the consolidation of those modulations using modulation-invariant contrastive learning (MI), is a contribution.TMCL performs competitively or better than existing state-of-the-art methods in low-label or no-label continual learning settings.\n\nWeaknesses: -Since a unique modulation is stored per class, scaling to tasks like ImageNet-1K may pose challenges in both memory and management. This is briefly mentioned as a limitation, but a forward-looking strategy (e.g., shared modulation generators) would be helpful.\n\n-All experiments are performed on CIFAR-100. This dataset is limited in visual diversity and resolution. Results on higher-resolution benchmarks like miniImageNet or ImageNet-Subset would improve generalizability.\n\nQUESTIONS:\n1- can authors add a discussion on how to scale TMCL to larger datasets or class hierarchies?\n\n2- can authors provide a visualization of representation space dynamics across sessions (e.g., UMAP plots) to support the modulation-invariance hypothesis?\n\n3- can authors discuss robustness to label noise or incorrect modulations?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}