Title: Memory-Integrated Reconfigurable Adapters: A Unified Framework for Settings with Multiple Tasks
Venue: NeurIPS 2025
Accepted: yes
Keywords: Associative Memories

Domain Generalization

Continual Learning
Abstract: Organisms constantly pivot between tasks such as evading predators, foraging, traversing rugged terrain, and socializing, often within milliseconds. Remarkably, they preserve knowledge of once-learned environments sans catastrophic forgetting, a phenomenon neuroscientists hypothesize, is due to a singular neural circuitry dynamically overlayed by neuromodulatory agents such as dopamine and acetylcholine.
In parallel, deep learning research addresses analogous challenges via domain generalization ($\textbf{DG}$) and continual learning ($\textbf{CL}$), yet these methods remain siloed, despite the brain’s ability to perform them seamlessly. In particular, prior work has not explored architectures involving associative memories ($\textbf{AM}$s), which are an integral part of biological systems, to jointly address these tasks. We propose Memory-Integrated Reconfigurable Adapters ($\textbf{MIRA}$), a unified framework that integrates Hopfield-style associative memory modules atop a shared backbone. These memory modules store adapter-weight updates as values and retrieve them via learned keys. Associative memory keys are learned post-hoc to index and retrieve an affine combination of stored adapter updates for any given task or domain on a per-sample basis. By varying only the task-specific objectives, we demonstrate that $\textbf{MIRA}$ seamlessly accommodates domain shifts and sequential task exposures under one roof. Empirical evaluations on standard benchmarks confirm that our $\textbf{AM}$-augmented architecture significantly enhances adaptability and retention: in $\textbf{DG}$, $\textbf{MIRA}$ achieves SoTA out-of-distribution accuracy, and in incremental learning settings, it outperforms architectures explicitly designed to handle catastrophic forgetting using generic $\textbf{CL}$ algorithms. Extensive ablation studies validate the necessity of both associative memory storage and post-hoc key learning for robust interpolated retrieval of adapters. By unifying adapter-based modulation with biologically inspired associative memory, $\textbf{MIRA}$ delivers rapid task switching and enduring knowledge retention in a single extensible architecture, charting a path toward more versatile and memory-augmented AI systems.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a unified framework using Hopfield associative memory (AM) with reconfigurable adapters to address Domain Generalization (DG), Domain-Incremental Learning (DIL), and Class-Incremental Learning (CIL). A fixed ViT backbone is supplemented with task-specific adapters stored as \"values\" in AM. Dynamically learned \"keys\" enable sample-based retrieval of relevant adapters during inference. Training involves adaptation and consolidation stages. Evaluation is performed on both domain generalization and continual learning tasks, and demonstrate the superiority of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- First integration of AM with parameter-efficient fine-tuning (PEFT) for unified DG/CIL/DIL. Post-hoc key learning dynamically aligns adapter retrieval with task inputs.  \n- Achieves SOTA on multiple benchmarks, outperforming specialized methods in most tasks with significant margin.\n\n**Weaknesses**\n- Lacks ablation studies for Post-hoc key learning vs. other trivial keys such as random keys and more naive retrieval methods (e.g., image similarity)\n- It would be nice to discuss more about adapatation versus consolidation, especially in CL setting. E.g., what will happen if consolidation is applied only in the end like the case in DG\n- [Not confident, remind me if there is some misunderstanding] The thesis should discuss the relationship between the proposed method and existing techniques, such as, at least, **Neural Modulation** and **Mixture of Experts**. There appear to be significant similarities that warrant exploration and comparison.\n[1] Beaulieu, Shawn, et al. \"Learning to continually learn.\" ECAI 2020. IOS Press, 2020. 992-1001.\n[2] Zhou, Yanqi, et al. \"Mixture-of-experts with expert choice routing.\" Advances in Neural Information Processing Systems 35 (2022): 7103-7114.\n- Some minor typos\n\n     Undefined acronyms (e.g., \"MHN\" in Line 147).  \n     Incorrect table references (Table 1 vs. Table 3).  \n     Symbolic errors (e.g., \"[]\" in Line 201).\n\nQUESTIONS:\nThe two-stage training pipeline is a distinctive feature of the thesis. However, it would be valuable to elucidate whether this approach was essential or if it conferred particular benefits. In my view, it seems more like a limitation. Could the keys and memories be learned concurrently? For instance, Mixture of Experts (MoE) models do not necessitate two-stage learning.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes MIRA (Memory-Integrated Reconfigurable Adapters), an architecture inspired by biological associative memories. MIRA stores task-specific low-rank adapter updates in Hopfield-style associative memory modules atop a frozen backbone (ViT) and retrieves an affine combination of stored adapters per input using learned keys. The method unifies domain generalization (DG), class-incremental learning (CIL), and domain-incremental learning (DIL) under a single framework with only minor objective modifications. This frame work is conducted on DG, CIL and DIL task.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths :\n\nUnlike prior adapter-based PEFT methods (LoRA, DualPrompt) or associative memory works that store data or features, MIRA stores adapters and utilize the key to choose a proper adaptor by the input.\n\nMIRA adapts to diverse learning settings with the adaptor. The paper is supported by exceptionally strong empirical results. MIRA is shown to achieve SOTA performance across numerous datasets in all three settings.\n\n\n Weaknesses: \n\nWhile the method achieves high accuracy, there is no thorough evaluation of inference time, memory overhead, or scalability when storing and retrieving large numbers of adapters across tasks. (The framework's performance improves with more adapters per task, and the main experiments use 10)\n\nThe framework is designed for scenarios where task data is available during a training phase (either all at once for DG or sequentially for CL). As currently formulated, MIRA cannot perform zero-shot adaptation to a completely novel task encountered only at inference time, as it would lack the necessary adapters in its memory.\n\nQUESTIONS:\n(See Weaknesses)\n\nDoes retrieving affine combinations of adapters for inputs ever lead to negative transfer or performance degradation when tasks are highly dissimilar? If so, how can such interference be mitigated?\n\nWhat is the architecture and parameter overhead of the query modules (g_l) used to transform layer inputs to queries? How sensitive is performance to their capacity and design choices?\n\nLIMITATIONS:\nThe paper does not provide thorough analysis of inference latency, memory footprint, or retrieval time complexity when storing a large number of adapters, which is critical for deployment in continual learning at scale.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe proposed MIRA is a biologically inspired unified framework for adapting multiple tasks or domains without catastrophically forgetting. MIRA equips each layer with a Hopfield-style associative memory that stores task specific LoRA style low rank adapter updates indexed by learned keys. At inference time, the model generates query vectors from input activations, which retrieve the most appropriate adapter(s) via similarity-based matching. Depending on the separation function, this retrieval may result in a single adapter or a weighted combination. This dynamic, per-sample retrieval enables effective adaptation to domain generalization, class-incremental learning, and domain-incremental learning using a single architecture.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n•The paper is easy to read and algorithms clearly explain the flow of adaptation and consolidation. It also articulates a clear motivation for bridging continual learning and generalization by drawing parallels between neural computation.\n\n•The model addresses DG, DIL, and CIL within a single architecture by integrating LoRA style adapters with Hopfield-style associative memory per layer. It equips each layer of a frozen backbone with a Hopfield-style associative memory to dynamically retrieve task-adaptive parameters (adapters) on a per-sample basis without needing task labels or explicit domain identifiers to enable generalization and mitigate catastrophic forgetting.\n\n•Keys for memory retrieval are not trained jointly with adapters, but learned post-hoc, reducing the interference during adapter learning and separates adaptation from consolidation adding interpretability to the retrieval process\n\n•Adapters are retrieved on a per-input basis, allowing the model to dynamically tailor its behavior to each input, supporting fine-grained generalization.\n\n•It stores compressed adapter weights than the raw data making it efficient in terms of memory storage.\n\n•The architecture is modular making it compatible with existing transformer-based pretrained backbones.\n\nWeaknesses:\n\n•While MIRA introduces a combination of mechanisms, its core components Hopfield-style associative memory, LoRA adapters, and task specific fine tuning are well studied and not individually novel.\n\n•The framework performs key/query similarity operations at every layer and scales with the number of stored adapters. However, the paper lacks analysis of inference latency, runtime performance, or memory usage, making it unclear whether the approach is computationally feasible. The use of per-layer query modules adds to the model's complexity but is under specified and not ablated.\n\n•The experiments do not include statistical significance testing, such as confidence intervals or hypothesis tests, limiting the robustness of reported improvements. Furthermore, there is no strong comparison against simpler retrieval heuristics making it difficult to isolate the contribution of the associative memory mechanism.\n\n•Although the model claims improved interpretability through post-hoc key learning, the resulting adapter combinations, especially when involving multiple overlapping adapters may be difficult to interpret in practice. The lack of visualization or sparsity constraints weakens this claim. Additionally, the architectural modularity is not explored in depth, and their influence on retrieval quality is not clearly studied.\n\n•The framework assumes episodic task access in a sequential setting and is not evaluated in task free or online continual learning scenarios. Its performance also hinges on the assumption that low rank adapters are sufficient to capture all domain-specific variations, which may not hold in cases with more severe distribution shifts.\n\n•While MIRA demonstrates strong short-term retention, it is not evaluated on long sequences of tasks, where memory saturation or key/query overlap could degrade retrieval precision.\n\nQUESTIONS:\n•How critical is the associative memory mechanism to MIRA’s performance? Please provide ablations comparing MIRA with simpler alternatives such as Static adapter selection, attention based adaption mix, LoRA without any memory retrieval to understand the true contribution of the associative memory and validate over conventional baselines.\n\n•What is the computational and memory overhead of MIRA? Can the authors quantify inference-time latency, GPU memory usage, and per-layer retrieval cost? \n\n•Why is post-hoc key learning preferred over joint optimization with adapters? Please clarify whether joint training was attempted, what limitations or performance issues motivated decoupling,  whether hybrid approaches were considered as this directly impacts the interpretability and stability claims made by the paper\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes MIRA (Memory-Integrated Reconfigurable Adapters), a unified framework that tackles domain generalization (DG) and continual/incremental learning (CL) within a single architecture, inspired by biological memory mechanisms. Drawing from neuroscience, MIRA incorporates Hopfield-style associative memory (AM) modules into a shared model backbone, allowing per-sample dynamic reconfiguration via learned keys and stored adapter weight updates.\n\nThe key technical innovation lies in embedding Hopfield memory in each layer of a Vision Transformer (ViT), where memory keys are learned post-hoc to align with intermediate activations, enabling robust, context-aware retrieval of task-specific representations. This design allows MIRA to store, retrieve, and interpolate adapter updates across tasks and domains without explicit task identifiers.\n\nEmpirically, MIRA demonstrates strong state-of-the-art (SoTA) performance on standard DG and continual learning benchmarks, outperforming many task-specific baselines by large margins (up to 10%). Ablation studies further highlight the importance of both associative memory and key-learning mechanisms in achieving this performance.\n\nSTRENGTHS AND WEAKNESSES:\nQuality: The paper presents a well-engineered and carefully evaluated system, integrating associative memory into transformer-based architectures in a novel way. Ablation studies are thoughtfully designed, validating the core components (e.g., associative memory and post-hoc key learning).\n\nClarity: The paper is clearly written and well-motivated, with biological inspiration nicely woven into the design rationale.\n\nWeaknesses: Some insight into computational overhead (both in time and memory) compared to traditional adapters or CL methods would be helpful for practical deployment concerns.\n\nThe method is built on ViT backbones, but there is no exploration of how MIRA behaves across different pretrained models (e.g., different architectures, initialization scales, or training paradigms).\n\nAlthough effective on benchmark CL datasets, it is unclear how well MIRA would scale to hundreds or thousands of tasks/domains—a realistic scenario in continual learning.\n\nQUESTIONS:\n1) I noticed that your results outperform many adapter- or prompt-based CL methods (e.g., DualPrompt), but you did not include HIDE-PET [HIDE-PET: continual learning via hierarchical decomposition of parameter-efficient tuning], which recently reported stronger performance than many prior methods.\n\n2) Please discuss whether the associative memory has a limit in terms of how many adapter updates it can meaningfully store and retrieve. Can retrieval quality degrade due to interference or memory saturation?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}