Title: MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptation
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

MoE-ization

SVD

Orthogonality

PEFT

Multi-Task Adaptation
Abstract: Adapting large-scale foundation models in multi-task scenarios often suffers from task conflict and oblivion.
To mitigate such issues, we propose a novel "model MoE-ization" strategy that leads to a conflict- and oblivion-resistant multi-task adaptation method. 
Given a weight matrix of a pre-trained model, our method applies SVD to it and introduces a learnable router to adjust its singular values based on tasks and samples.
Accordingly, the weight matrix becomes a Mixture of Orthogonal Rank-one Experts (MoORE), in which each expert corresponds to the outer product of a left singular vector and the corresponding right one.
We can improve the model capacity by imposing a learnable orthogonal transform on the right singular vectors.
Unlike low-rank adaptation (LoRA) and its MoE-driven variants, MoORE guarantees the experts' orthogonality and maintains the column space of the original weight matrix.
These two properties make the adapted model resistant to the conflicts among the new tasks and the oblivion of its original tasks, respectively. 
Experiments on various datasets demonstrate that MoORE outperforms existing multi-task adaptation methods consistently, showing its superiority in terms of conflict- and oblivion-resistance.
The code is available at https://github.com/DaShenZi721/MoORE.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenges of task conflict and forgetting in multi-task adaptation of large-scale foundation models by proposing MoORE. MoORE applies SVD to the pre-trained weights and introduces a learnable router to dynamically combine orthogonal rank-one experts. To ensure fair comparison, all the compared methods are built upon the same base model, and extensive experiments across multiple datasets are conducted to verify the effectiveness of MoORE.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.The paper tackles an important problem in multi-task learning with large-scale foundation models.\n\n2.The paper conducts extensive comparisons and evaluations on diverse datasets to verify the effectiveness of the proposed method.\n\nWeaknesses：\n\n1.The proposed method in the paper does not achieve the best performance on all datasets.\n\n2.The paper claims superiority in efficiency in the abstract, but later notes that MoORE increases inference time compared to the original LLaMA-3.1 8B. This seems inconsistent.\n\nQUESTIONS:\n1. I'm a bit confused about the title — why is there a hyphen between “Conflict” and “and”, but no hyphen between “and” and “Oblivion”? That seems a bit odd.\n\n2. The manuscript claims to validate the superiority of the proposed method \"MoORE\" in mitigating task oblivion. However, Figure 2 presents results for \"SVDMoE\" rather than \"MoORE,\" leading to potential confusion about whether they are equivalent. The authors should clarify the relationship between MoORE and SVDMoE and use consistent terminology throughout the paper.\n\n3. The abstract claims MoORE demonstrates superiority in terms of efficiency. Could the authors clarify what aspects “efficiency” encompasses—is it a composite metric or specific factors? Also, please specify where in the paper this efficiency is demonstrated experimentally or analytically. Clearer definitions and supporting evidence are recommended.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes converting the dense pre-trained model weights into low-rank expert models and dynamically combining these expert models to learn new tasks as much as possible while reducing the forgetting of general knowledge (or old knowledge).\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- This paper proposes splitting a dense pre-trained model into a series of rank-one experts and dynamically combining these experts through a gating network.\n- The proposed method has been validated on a large number of commonsense reasoning (CSR) and natural language understanding (NLU) tasks.\n\n**Weaknesses:**\n- The meaning of adding orthogonal transformations to the input is unclear. Simply interpreting it as increasing representational capacity is insufficient, and the results in Table 4 also show that increasing the orthogonal projection of the input is not always effective.\n- In the last column of Figure 2, it should be MoORE instead of SVDMoE? According to the results in Figure 2, the proposed MoORE still shows a significant performance drop on general tasks (even though relative to other methods, the drop is smaller), such as a 6% drop on the BBH task and about a 3% drop on MMLU.\n- It is unclear how the proposed method compares with full fine-tuning (upper bound) on CSR and NLU datasets in terms of performance differences.\n- Existing works have also discussed applying orthogonal constraints to LoRA [1], which needs to be discussed.\n- Existing works have also explored splitting dense models into MoE models [2,3], and these works need to be discussed and compared.\n\n[1]Wang, Xiao, et al. \"Orthogonal subspace learning for language model continual learning.\" EMNLP  (2023).\n\n[2]Komatsuzaki, Aran, et al. \"Sparse upcycling: Training mixture-of-experts from dense checkpoints.\" ICLR (2023).\n\n[3]Wu, Haoyuan, et al. \"Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks.\" Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.\n\nQUESTIONS:\nIn SVD decomposition, directions with larger singular values are more important. Does this mean that most samples/tasks will choose the rank-one experts associated with the larger singular values, thus exacerbating the expert routing imbalance?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents MoORE, which repackages task-aware Singular Value Fine-tuning within a MoE story. \n\nThe authors decompose model weights via SVD, term each pair of left/right singular vectors as \"rank-1 experts\", and refer to the natural summation in SVD reconstruction as a \"mixture\" of experts. Projected singular values are rebranded as \"routing weights\" (router).\n\nMethodologically, model weights first undergo SVD decomposition. Subsequently, a task_id is input to train two projections: one on singular values and another between inputs and model weights. Household adaptors are utilized to ensure orthogonality.\n\nThe core contribution lies in applying Singular Value Fine-tuning to address key multi-task adaptation challenges: task conflicts and catastrophic forgetting.\n\nTheoretical underpinnings regarding orthogonality and column space preservation provide justification for the approach's efficacy.\n\nEmpirical results demonstrate modest improvements.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- Engineering-Friendly: The proposed method is straightforward to implement, requiring only SVD decomposition and lightweight projection layers, making it practically accessible for deployment.\n- The paper exhibits clear exposition with logical flow, making the story easy to follow despite the conceptual reframing.\n\n**Weaknesses:**\n- Fundamental Lack of Novelty and Conceptual Misrepresentation: The core contribution appears to be a rebranding of established SVD fine-tuning techniques rather than genuine innovation. The authors introduce \"Model MoE-ization\" as a novel concept, but the proposed method shows no essential difference from existing singular value fine-tuning approaches ( e.g. https://proceedings.neurips.cc/paper_files/paper/2022/file/f3bfbd65743e60c685a3845bd61ce15f-Paper-Conference.pdf ). The MoE terminology creates conceptual confusion without adding substantive technical value. This MoORE is fundamentally singular value adaptation disguised with MoE nomenclature.\n- Questionable Theoretical Motivation for \"Model MoE-ization\" and Misleading Framing: The rationale behind applying MoE-ization to already pre-trained language models remains unclear. Traditional MoE architectures primarily sparsify model parameters when processing vast training data to prevent overfitting, while the proposed SVD decomposition serves entirely different purposes. What meaningful purpose does \"MoE-izing\" an already-trained language model serve beyond SVD decomposition with alternative terminology? The authors fail to articulate why decomposing pre-trained model weights should be considered \"MoE-ization\" beyond superficial structural similarities.\n- Despite the MoE framing, the experimental evaluation completely omits comparisons with actual MoE methods, instead focusing on LoRA-based approaches. This fundamental disconnect between the conceptual narrative and empirical validation raises serious concerns about the paper's positioning and undermines the paper's central narrative.\n- Limited Generalizability due to Task_id Dependency: The method's reliance on explicit task_ids restricts its applicability to scenarios with tasks without a task_id. This fundamental limitation contradicts claims of general-purpose multi-task adaptation, as the approach cannot handle scenarios where task_id is unavailable or unseen.\n- Inconsistent Performance Claims: The authors claim MoORE \"consistently outperforms existing multi-task adaptation methods\", yet the experimental results reveal performance dependence on hyperparameter L selection. The paper lacks comprehensive hyperparameter sensitivity analysis.\n- Limited Model Scope: Despite that the MoORE theory seems to be general, all experiments are confined to one single language model, raising questions about cross-model transferability and robustness, limiting confidence in applicability on diverse model families and architectures.\n- Insufficient Mechanistic Validation: While the paper proposes that column space preservation promotes forgetting resistance, it lacks direct quantitative evidence supporting this causal relationship. The theoretical claim requires empirical validation through metrics such as activation space similarity or representation overlap analysis before and after fine-tuning.\n\n\n**Issues:**\n- Incomplete Analysis of SVD Components: The method consistently employs all singular vectors without analyzing performance variations across different rank selections, missing opportunities for efficiency optimization and theoretical insights.\n- Inadequate Table Documentation: Table captions are overly brief and fail to clearly explain notation and symbols (e.g., parameter L).\n- Missing Routing Analysis: The paper lacks analysis of routing strategy sparsity patterns or expert collapse phenomena, which would be essential for genuine MoE evaluation. Though given the fundamental differences from traditional MoE experts, this may be less critical.\n\nQUESTIONS:\n- The paper's central methodology appears to be functionally equivalent to established Singular Value Decomposition (SVD) based fine-tuning techniques. Could the authors precisely articulate the fundamental technical novelty of \"MoORE\" that distinguishes it from prior work on using SVD for model adaptation? Specifically, what aspects of the proposed method constitute a genuine innovation beyond a rebranding of existing concepts?\n- The paper frames its approach as \"Model MoE-ization,\" yet the theoretical motivation and empirical evaluation appear disconnected from the established principles of Mixture-of-Experts (MoE) architectures. Given that the method operates on pre-trained models via SVD and traditional MoEs focus on sparse activation during training, what is the theoretical justification for this MoE analogy? Why decomposing pre-trained model weights should be considered \"MoE-ization\" beyond superficial structural similarities? Consequently, why were direct comparisons with genuine MoE models omitted from the experiments, which instead focused on LoRA-based methods? \n- How does the proposed framework address scenarios where task identifiers are unavailable, such as in continual learning with unseen tasks or contexts where tasks are defined implicitly? This dependency seems to severely constrain the method's practical applicability in real-world settings.\n\nLIMITATIONS:\nthe authors adequately addressed the limitations and potential negative societal impact of their work", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMixture of Experts is now a popular paradigm for multi-task adaptation of transformers. In this approach an adapter is assigned/learned for each task and a learnable router selects the appropriate adapter for the given task. This achieves parameter efficient fine-tuning. An important drawback of this approach is destructive interference and catastrophic forgetting. A number of solutions has been suggested for to solve this problem. Many of them lack elegance and a theoretical foundation.\n\nThe present paper proposes an simple yet effective approach to solve this problem. It constructs a stack of \"weak\"experts using orthogonal factorization of the foundation weight matrix. A router is learned for choosing among these simple experts. The method is found to be effective on benchmark tasks.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper presents an excellent application of linear algebra tools for mixture of experts formulation of the parameter efficient fine-tuning process. The results demonstrate the effectiveness of this relatively simple approach.\n\nA number of very similar approaches to MoE-ization is existent in literature. Many of them are compared in the experimental results section. They are also qualitatively described. A more mathematical formulation of the \"weak\" MoE approach as opposed to exisitng ones using routers in the light of linear algebra concepts might be possible and will definitely add more value to the topic/ It will definitely affirm the novelty of the proposed technique.\n\nQUESTIONS:\n1. Is it possible to provide a set of formulations in terms of the underlying linear algebra operations of the proposed method vis-a-vis other MoE-zation techniques for PEFT?\n\n2. How does the proposed method compare with other state-of-art PEFT techniques in terms of efficiency on multi-task benchmarks?\n\nLIMITATIONS:\nThe proposed approach has a great deal of similarity with the eigenstructure assignment problem in linear algebra. It would be beneficial to study this relation in order to have a better theoretical understanding of the interference problem.Often \"regularization functionals\"are used in eigenstructure assignment. Such functionals may be useful for PEFT as well.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}