Title: Synergy over Discrepancy: A Partition-Based Approach to Multi-Domain LLM Fine-Tuning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Fine-Tuning

Multi-Domain
Abstract: Large language models (LLMs) demonstrate impressive generalization abilities, yet adapting them effectively across multiple heterogeneous domains remains challenging due to inter-domain interference. To overcome this challenge, we propose a partition-based multi-stage fine-tuning framework designed to exploit inter-domain synergies while minimizing negative transfer. Our approach strategically partitions domains into subsets (stages) by balancing domain discrepancy, synergy, and model capacity constraints. We theoretically analyze the proposed framework and derive novel generalization bounds that justify our partitioning strategy. Extensive empirical evaluations on various language understanding tasks show that our method consistently outperforms state-of-the-art baselines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of fine-tuning LLMs on multiple domains simultaneously. The authors argue that standard approaches suffer from negative interference, while training separate models is inefficient. They propose a partition-based, multi-stage fine-tuning framework. The core idea is to first partition the domains into clusters based on an objective function that balances inter-domain discrepancy, synergy, and model capacity. The LLM is then fine-tuned sequentially on these clusters using adapters. The authors provide a theoretical analysis with generalization bounds to motivate their partitioning strategy and conduct extensive experiments on three LLMs across four language tasks, showing consistent improvements over various baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Problem is relevant and well-motivated. The paper tackles a fundamental challenge in fine-tuning across multiple domains. This issue is highly relevant as real-world applications increasingly demand multi-domain adaptability. The authors clearly articulate the shortcomings of existing solutions (e.g., joint fine-tuning, domain adversarial training) and provide strong motivation for their proposed partition-based approach by grounding it in both practical scenarios and theoretical considerations.\n- Framework is coherent and mostly clear. The proposed framework is logically structured and well-integrated with the theoretical analysis. The use of a synergy-discrepancy-capacity trade-off objective offers a principled way to partition domains and fine-tune in stages. The theoretical part supports the methodology.\n- Extensive experiments. The experimental section is thorough. The authors evaluate across a diverse set of language understanding tasks using multiple backbone models (LLaMA, Falcon). They include comparisons with strong baselines across full fine-tuning, adapter-based methods, domain adaptation techniques, and data selection strategies. Detailed ablation studies and diagnostic analyses further validate the efficacy and robustness of the approach.\n\nWeaknesses\n- The theoretical analysis is presented as a rigorous justification for the proposed algorithm, but the connection is tenuous and potentially misleading. The synergy term is introduced heuristically into the objective function G (Equation 12) and unclear how the JS divergence relates to Definition 2. The theory seems to be more as a post-hoc rationalization for a heuristic objective rather than a foundational proof of its optimality. My suggestion is to tone down the claims about the theoretical justification and re-frame the theory as a motivation for their objective.\n- Novelty: The core idea of grouping related tasks/domains to improve multi-task learning is not new. This concept has been explored before in the multi-task learning and continual learning literature (e.g., task clustering, routing), e.g., [1,2]. The paper fails to properly position itself within this broader context, overstating its conceptual novelty.\n\n[1] Jacob, L., Vert, J. P., & Bach, F. (2008). Clustered multi-task learning: A convex formulation. Advances in neural information processing systems, 21.\n\n[2] Standley, T., Zamir, A., Chen, D., Guibas, L., Malik, J., & Savarese, S. (2020, November). Which tasks should be learned together in multi-task learning?. In International conference on machine learning (pp. 9120-9132). PMLR.\n\nQUESTIONS:\n- Can the authors more clearly explain how the carry out step 1 in their algorithm? and its complexity?\n- To what extent does the theoretical analysis genuinely inform the design of the objective function in Equation 12, given that the synergy term is introduced heuristically and its connection to the formal discrepancy measures is not clearly established?\n- How does the proposed partition-based multi-stage fine-tuning meaningfully differ from prior work on task/domain clustering in multi-task or continual learning, and why are these connections not discussed more thoroughly?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel partition-based multi-stage fine-tuning framework for adapting large language models (LLMs) to multiple domains (or tasks) simultaneously. Instead of fine-tuning on all domains jointly or one domain at a time, the authors cluster the domains into groups (stages) based on a measured synergy (similarity/complementarity) vs. discrepancy (distribution difference) between domains. The LLM is then fine-tuned sequentially: at each stage, only the group of domains assigned to that stage are used to update the model (with small, domain-specific adapter modules), while constraining how much the base model and adapters can change per stage. The paper provides a theoretical analysis with new generalization bounds (Theorems 3.1 and 3.2) showing that this multi-stage partitioned training can yield a tighter upper bound on risk than single-stage multi-domain training. In particular, the theory introduces terms for domain discrepancy (penalizing differences between domains) and synergy (credit for learning complementary domains together), and it formally justifies choosing the partition that maximizes a synergy-minus-discrepancy objective. Empirically, the authors evaluate on four diverse language understanding tasks (summarization, QA, sentiment classification, topic classification) treated as distinct domains, using three LLM backbones (LLaMA2-7B, LLaMA2-13B, Falcon-40B). The proposed method — termed Partition-Based Multi-Stage Fine-Tuning (PMS-FTP) — outperforms a wide range of baselines, including full fine-tuning, parameter-efficient adapters (LoRA, etc.), multi-domain adaptation methods, and data selection heuristics. PMS-FTP achieves the highest accuracy/score on all tasks for each model, with consistent gains of about 0.5–1.0 absolute points on metrics like ROUGE-L and exact match over the best prior methods. Notably, it also reduces GPU memory usage by roughly one-third compared to full fine-tuning by only updating a small subset of parameters per stage. In summary, the paper’s contributions include: (1) a new domain partitioning algorithm for multi-domain LLM fine-tuning that leverages inter-domain synergies while mitigating interference; (2) novel theoretical bounds that quantify how domain discrepancy, synergy, and adapter complexity affect generalization, supporting the multi-stage approach; and (3) comprehensive experiments demonstrating improved performance and efficiency over state-of-the-art baselines.\n\nSTRENGTHS AND WEAKNESSES:\nFrom perspective of the four dimensions, I can see strengths and weaknesses of this paper as following: \n\nQuality: \nStrengths: \nThe idea of partitioning training domains to maximize synergy is a fresh perspective in the context of LLM fine-tuning. Prior works in multi-domain or multi-task learning have often used techniques like adversarial training to make a single model invariant to domain differences, or data sampling strategies to balance domains, or at most simple sequential fine-tuning. By contrast, this paper explicitly clusters domains and fine-tunes in stages, which is a novel formulation. The authors combine concepts from different areas in an original way: they use adapter modules for domain-specific capacity (building on parameter-efficient fine-tuning literature) and introduce a clustering algorithm driven by a custom objective (balancing discrepancy and synergy) – this is not a standard approach in prior work. The theoretical derivation of a “synergy offset” term in generalization bounds is, to my knowledge, new. It extends domain adaptation theory to incorporate a notion that certain domains together can effectively reduce error (negative transfer offset) instead of only increasing it. The method also outperforms recent ideas like Tag-LLM (which used task-aware gating) and S2L (which did curriculum ordering), indicating that the authors’ approach isn’t just a minor tweak on those, but a qualitatively different strategy. In summary, the work’s novelty lies in reframing multi-domain fine-tuning as a partitioning problem with a provable optimal grouping, which is a creative and significant departure from straightforward multi-task training or domain alignment approaches.\n\nWeaknesses: The experiments focus on four domains, which is a reasonable starting point, though scaling to more would further strengthen the claims. The setting also blends multi-domain and multi-task learning, and the paper could clarify how differences in task formats are handled. Additionally, the synergy metric is a heuristic based on lexical and embedding similarity; while intuitive, alternative definitions are not explored. Theoretical guarantees rely on standard assumptions (e.g., Lipschitz continuity, bounded norms), which are enforced in practice but may constrain flexibility. Overall, these are minor concerns in an otherwise strong and well-executed paper.\n\nClarity: \n\nStrengths: The paper is well-written and structured in a logical manner. The introduction clearly motivates the problem of negative transfer when fine-tuning on multiple domains and introduces the key idea of partitioning domains to exploit “synergy over discrepancy.” The method is described with both conceptual reasoning and formal algorithmic steps (Algorithm 1 provides a clear step-by-step overview). The theoretical section is dense but does a good job laying out definitions and assumptions before presenting the theorems, which helps the reader follow the derivations. Important details such as the objective function for partitioning (including the role of lambda balancing synergy vs. discrepancy) are given, and the authors include an illustrative Corollary 3.1 to intuitively explain that high-synergy domains will optimally group together. Figures and tables are effective: Table 1 neatly summarizes the performance across models and tasks, and the improvements are easy to see (with best results bolded). The ablation tables and the memory usage table further clarify the trade-offs. In general, the writing is professional and clear, making the complex approach reasonably understandable.\n\nWeaknesses:\nWhile the paper is generally clear, a few points could be more precise. It’s unclear how domain-specific adapters are selected—whether via explicit domain IDs or automatic routing—during training and inference. The handling of different task formats (e.g., summarization vs. classification) is also not fully explained, which affects reproducibility. Additionally, the synergy metric is only briefly described, and a clearer exposition or example would aid understanding. Clarifying that the “domains” correspond to distinct tasks would also help set reader expectations. These are minor issues in an otherwise well-written paper.\n\nSignificance:\n\nStrengths: This work tackles an important practical challenge: efficiently fine-tuning one large model on multiple data sources without interference. As large models are increasingly adapted to specialized domains, the ability to handle multiple domains/tasks in one model (instead of training separate models or naively pooling data) is valuable – it can save deployment costs and potentially lead to models that generalize more broadly. The paper’s key insight is to formalize and exploit the idea that some domains can actually benefit each other (synergy) if trained together, while very disparate domains should be isolated to avoid damage. This is a nuanced departure from prior multi-domain adaptation work that mostly focused on mitigating negatives (interference) rather than actively leveraging positives (complementarity). By providing a principled algorithm and theoretical guarantees for this idea, the paper could inspire further research into adaptive multi-domain training strategies. In terms of empirical significance, the method delivers state-of-the-art results on the benchmarks considered. The gains over strong baselines like Tag-LLM (task-aware prompt tuning) and S2L (a curriculum clustering approach) – though modest in absolute numbers – demonstrate that partitioning strategy yields consistent improvements across varied tasks and model scales. Even a 1 point increase in ROUGE or accuracy can be meaningful in competitive settings, especially since it’s achieved alongside reduced memory usage. Moreover, the approach is model-agnostic and could be applied to different LLM architectures, which broadens its potential impact. Overall, the paper’s contributions feel significant for the subfield of multi-domain learning with LLMs, providing both theoretical insight and a practical technique that outperforms existing methods.\n\nWeaknesses:\nThe improvements, while consistent, are modest—typically around +0.5 to +1.0 in accuracy—which may feel incremental given the added complexity. The method focuses on diverse tasks rather than classic multi-domain scenarios (e.g., same task across domains), which slightly limits its demonstrated breadth. Also, it assumes known domain labels and pairwise metrics, which may not hold in more dynamic real-world settings. Still, within the scope of fine-tuning on a fixed set of domains, the contribution is meaningful and practical.\n\nOriginality:\n\nStrengths: The idea of partitioning training domains to maximize synergy is a fresh perspective in the context of LLM fine-tuning. Prior works in multi-domain or multi-task learning have often used techniques like adversarial training to make a single model invariant to domain differences, or data sampling strategies to balance domains, or at most simple sequential fine-tuning. By contrast, this paper explicitly clusters domains and fine-tunes in stages, which is a novel formulation. The authors combine concepts from different areas in an original way: they use adapter modules for domain-specific capacity (building on parameter-efficient fine-tuning literature) and introduce a clustering algorithm driven by a custom objective (balancing discrepancy and synergy) – this is not a standard approach in prior work. The theoretical derivation of a “synergy offset” term in generalization bounds is, to my knowledge, new. It extends domain adaptation theory to incorporate a notion that certain domains together can effectively reduce error (negative transfer offset) instead of only increasing it. The method also outperforms recent ideas like Tag-LLM (which used task-aware gating) and S2L (which did curriculum ordering), indicating that the authors’ approach isn’t just a minor tweak on those, but a qualitatively different strategy. In summary, the work’s novelty lies in reframing multi-domain fine-tuning as a partitioning problem with a provable optimal grouping, which is a creative and significant departure from straightforward multi-task training or domain alignment approaches.\n\nWeaknesses: The approach builds on established ideas—adapters, domain discrepancy, and clustering—assembled in a thoughtful and original way. While each component individually is not novel, their integration for multi-domain LLM fine-tuning is new and well-motivated. The synergy metric is heuristic and could benefit from comparison to alternatives, but it’s a reasonable starting point. Overall, the contribution feels fresh despite leveraging known tools.\n\nQUESTIONS:\n1. Could you clarify how the model selects the appropriate domain adapter during training and inference? Do you rely on known domain labels, or is there an automatic selection mechanism? Clarifying this would help assess the method’s applicability in real-world settings where domain labels may not be available.\n\n2. How did you handle the differing formats across tasks like summarization, QA, and classification? Did you use a unified text-to-text format or separate output heads/objectives? This detail is important for reproducibility and understanding the training setup.\n\n3. How sensitive is the method to the choice of synergy metric (lexical + embedding similarity)? Did you try alternatives or assess how different metrics affect the domain partitioning? More insight here would help evaluate the method’s stability and generalizability.\n\n4. Have you tested the method with more than four domains? Can it scale effectively to 10 or more domains, or does the performance/efficiency degrade? Discussing this would help clarify the method’s broader applicability.\n\n5. Could the framework support continual learning by adding new domains after training? For example, could you run another stage without harming previous domains? Insights here would highlight the method’s potential for long-term or evolving domain adaptation.\n\nLIMITATIONS:\nThe paper lacks a discussion of domain identification and real-world deployment assumptions. It also doesn’t explore settings where domains are unknown or where the synergy metric may be unreliable. The multi-task setup (rather than true multi-domain within-task) somewhat weakens the generality of the findings.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the critical problem of supervised finetuning (SFT) of LLM across multiple heterogeneous domains, specifically addressing the issue of inter-domain conflicts. The authors propose a novel multi-stage fine-tuning framework that strategically partitions domains into subsets (stages). This partitioning mechanism is theoretically motivated to minimize domain discrepancy while maximizing domain synergy within each group of tasks. The theoretical analysis demonstrates that this partition mechanism enjoys a controllable generalization bound. Extensive empirical evaluations across 4 real-world fine-tuning tasks show the effectiveness of the proposed method, which consistently outperforms baseline approaches across all SFT tasks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths: \n1. The paper is well-motivated, tackling the significant challenge of domain conflict and negative transfer in multi-domain LLM fine-tuning, a prevalent issue in real-world applications.\n2. A complete generalization bound analysis is derived for the multi-domain fine-tuning problem. This theoretical insight directly motivates and justifies the proposed partition mechanism, providing a robust scientific basis for the approach based on domain discrepancy and synergy.\n3. the empirical results on 4 various real-world finetuning tasks demonstrate the effectiveness of the proposed method, which consistently outperforms baseline approaches across all the tasks.\n4. The comprehensive ablation studies demonstrate the robustness of the proposed method on hyperparameter choices.\n\n### Weaknesses:\n1. **The algorithm, implementation and complexity of the partition mechanism require more explanation.** According to Equ. 12,  it is not clear how to estimate $\\Delta \\theta^t$ and $\\phi_j^t$ without retraining the model? or are they directly be replaced by the parameter shift bounds $\\rho_{\\theta}, \\rho_{\\phi}$? \n2. **The empirical evidence on the current 4 tasks are limited to relatively easy tasks.** Could you give the results on the reasoning tasks, e.g. Hellaswag, MMLU, ARC, SciQ, GSM8K? \n3. **Lack of comparison to some conventional, reweighting-based multi-task learning approaches**, e.g. iMTL [1], FAMO [2], ExcessMTL [3].  \n\n[1] Towards Impartial Multi-task Learning. Liu, et. al, 2021.\n[2] FAMO: Fast Adaptive Multitask Optimization. Liu, et. al, 2023.\n[3] Robust Multi-Task Learning with Excess Risks. He, et. al, 2024.\n\nQUESTIONS:\n### Questions from weakness\n1. **The algorithm, implementation and complexity of the partition mechanism are not clear.** According to Equ. 12,  how could the $\\Delta \\theta^t$ and $\\phi_j^t$  be estimated without retraining the model? do you apply the manually parameter shift bounds $\\rho_{\\theta}, \\rho_{\\phi}$? could you clarify more on how this can be implemented within the given complexity?\n2. **The empirical evidence on the current 4 tasks are limited.** Could you give the results on the reasoning tasks, e.g. Hellaswag, MMLU, ARC, SciQ, GSM8K, etc.?\n3. **Lack of comparison to some conventional multi-task learning approaches**, e.g. iMTL [1], FAMO [2], ExcessMTL [3].  Could you give the comparison results to these baselines?\n---\n4. Can you explain more on the \"Single-Domain LLM Fine-tuning\" baseline? Do you only tune the adaptor or both the backbone and the adaptor? A clarification on the number of tuneable parameters on each experiments could help.\n5. Can you explain the training loss curves on the QA task in Fig. 1? According to the given algorithm, the proposed PMS-FTP method adopts stage-wise training, the training loss on the QA task should only be available on one of the training stage, instead of the whole training process?\n\nLIMITATIONS:\n1. The scope of the tasks are limited to easier question-answering (e.g. SQuAD) and summarization tasks. More challenging reasoning tasks, e.g. Hellaswag, MMLU, ARC, SciQ, GSM8K, etc. have not been explored.\n2. Lack of comparison to some conventional multi-task learning approaches.\n3. The experiments are limited to small amount of tasks (\\#tasks=4), while the scalability and effectiveness to larger amount of tasks have not been explored.\n4. The effectiveness of stage-wise training using the partition mechanism is validated, while the impact of the order of training stages, $\\{S_1, ..., S_M\\}$ has not been explored or discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work presents \\textbf{PMS-FTP}, a partition-based, parameter-efficient scheme for fine-tuning a single LLM on multiple domains.  Domains are clustered into sequential stages by maximising an objective $G$ that trades off synergy, discrepancy, and adapter cost—supported by a worst-stage risk bound—and each stage fine-tunes lightweight LoRA adapters while largely freezing the backbone.  Across four tasks and three model sizes, PMS-FTP beats eleven competitive baselines by +0.8–2.0 points and cuts GPU memory by roughly 32\\,\\%, with ablations confirming the value of partitioning, stage count, and norm constraints.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n(1) Conceptual Novelty \\& Theory. The paper frames multi-domain fine-tuning as minimising a partition objective $G$ that balances \\emph{synergy}, \\emph{discrepancy}, and adapter cost, then proves new generalisation bounds showing this choice tightens worst-domain risk.  Coupling these guarantees with an $O(k^{2}\\log k)$ agglomerative clustering yields a rare, theory-backed PEFT recipe.\n\n(2) Consistent Empirical Gains. Across four tasks (news‐sum, sentiment, QA, topic) and three backbones (LLaMA-2 7B/13B, Falcon-40B), PMS-FTP beats eleven strong baselines—including full FT and the latest PEFT methods—by +0.8–2.0 points on standard metrics, demonstrating robustness to both task diversity and model scale.\n\n(3) Resource Efficiency. Stage-wise norm bounds cut GPU memory by $\\sim$32 \\% versus full fine-tuning (e.g., 18.4 GB vs.\\ 27.2 GB on an A100) while still improving accuracy; the only extra cost—computing synergy/discrepancy matrices—remains trivial for $k\\!\\le\\!10$ domains.\n\n\nWeaknesses:\n\n(1) Heuristic similarity metrics. The partition relies on fixed vocab-overlap, sentence-embedding cosine, and JS divergence, which often mis-estimate real transferability; domains can look similar yet teach different skills (or vice-versa), sending data into sub-optimal clusters. A task-aware or learned affinity signal—e.g., gradient similarity—would make the partition less brittle.\n\n(2) Scalability limits. Computing the full $k\\times k$ affinity matrix and running agglomerative clustering costs $O(k^{2}\\log k)$ time and $O(k^{2})$ memory, while adapter banks grow linearly with $k$. This becomes infeasible when the number of domains reaches dozens; approximate kernels or router-style sharing are needed.\n\n(3) Modest memory edge. While cheaper than full fine-tuning, two adapter banks still use more GPU memory than leaner single-adapter PEFT variants, which matters on edge devices; pruning, quantization, or cross-stage sharing could tighten this gap.\n\n(4) Theory untested empirically. The paper derives a neat worst-stage risk bound, but never shows how the objective $G$ (or its components) correlates with held-out error.  Without a plot or numbers, readers cannot tell whether the bound is predictive or merely a sanity check.  Reporting the correlation between $G$ and downstream accuracy across random partitions would clarify its practical value.\n\nQUESTIONS:\nplease refer to the weaknesses\n\nLIMITATIONS:\nplease refer to the weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}