Synergy over Discrepancy: A Partition-Based Approach to Multi-Domain LLM Fine-Tuning
Authors: Hua Ye
Siyuan Chen
Haoliang Zhang
Weihao Luo
Yanbin Li
Xuan Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Fine-Tuning
Multi-Domain
Primary area: deep_learning

Abstract: Large language models (LLMs) demonstrate impressive generalization abilities, yet adapting them effectively across multiple heterogeneous domains remains challenging due to inter-domain interference. To overcome this challenge, we propose a partition-based multi-stage fine-tuning framework designed to exploit inter-domain synergies while minimizing negative transfer. Our approach strategically partitions domains into subsets (stages) by balancing domain discrepancy, synergy, and model capacity constraints. We theoretically analyze the proposed framework and derive novel generalization bounds that justify our partitioning strategy. Extensive empirical evaluations on various language understanding tasks show that our method consistently outperforms state-of-the-art baselines.

Synergy over Discrepancy: A Partition-Based Approach to Multi-Domain LLM Fine- Tine-Tuning
Hua Ye Siyuan Chen', Haoliang Zhang Weihao Luo Yanbin Lid, Xuan Zhang
NNajing University AAronn Technology CO., LTD University of Bristol
'The University of Oklahoma 'Donghua University
Beijing University of Posts and Telecommunications 'Camegie Mellon University
Abstract
Large language models (LLMs) demonstrate impressive generalization abilities, yet adapting them effectively across multiple heterogeneous domains remains challenging due inter-domain interference. To overcome this challenge, we propose a partition- partition-based multi-stage fine-tuning framework designed to exploit inter-domain synergies while minimizing negative transfer. Our approach strate- gically partitions domains into subsets (stages) by balancing domain discrepancy, synergy. and model capacity constraints. We theoretically analyze the proposed framework and derive novel generalization bounds that justify partitioning strategy. Extensive empirical evaluations on various language understanding tasks show that our method consistently outperforms state-of-the-art baselines.
Introduction
Large language models (LLMs) have propelled natural language processing (NLP) to unprecedented capabilities [Kumarl 2024, Karmnkolas et all 2023, Hu et all 2025, Zhang et al, 20258, Lin et all massive, diverse textual corpora [Wu all 2022 Li et all 20244 Chen all 20255, Zhang et all, 20225 Sun et all 20256). Fine-tuning these pretrained models for specific downstream domain has been widely explored and proven highly effective; notable approaches include adapter-based based modules [Houlsby et all 20T Zhang et all 202 parfieter-efffcieet fine-tuning via low-rank updates [Huerall 2021], and instruction-bassd fine-tuning methods [[Cao et all, 2024].. These methods have also accelerated the application of LLMs in many scenarios, such healthcare(ITong al, 2025. Lu all 2025. Wang all 2025]
2025], owing largely to their extensive pretraining
transportation(Yan et all 20223 Lu at 20225, Zeng et all, 20255aa robotics Xia0 et all
20225
Ya 2
Zhang et all (025a] and other applications (S
2025a Zhang all 21255]]
However, while these methods excel in adapting to single domain, practical scenarios frequently require simultaneous adaptation to multiple distinct domains-a scenario far less studied and sub- stantially more challenging [Lu et all 20225, Chen et all 20263]] Due to constraints on energy computing power [He. 2025], must consider how enhance the model's ability to adapt to different domains. Consider, for instance, a scenario where single pretrained model must be simul- taneously adapted to clinical texts [[hhiruavuukarann et all, MIZX), social media posts [[ang all 2024a, Jiang et all, 2024, 2025], and legal documents [Seabra et all, 2024]. Naive approaches such as jointly fine-tuning the model across all domains or independently fine-tuning separate models per domain often yield suboptimal results[[X1ao ct all 20255, Zhang et all, 202 Wang and Zhang 2024, (een all 2023]: domain-specifes features may negatively interfere, causing one domain to overshadow others or impair overall generalization capabilities [Lu et all 2024a, Zheng et all 2024. Van Veen et all 2023, Tan et all 2025). This motivates our central research question: how can we
iCorresponding author(xuanzhang21999 gmail.com)
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

effectively and efficiently fine-tune a single LLM across multiple heterogeneous domains, exploiting inter-domain synergies while mitigating negative interference?
A natural strategy to mitigate these challenges is to add adapter modules specialized for each do- main [Houlsby et all, 2019, Zhang all 20248, lão et all 2023]. While adapters reduce the need for full fine-tuning, they do not fully address the complexity of managing multiple source domains with potentially large discrepancies. In such cases, the learned model has juggle both shared features (common linguistic properties across domains) and domain-specific features (rare words, styles, or content) et 2022). Existing approaches often handle these competing demands by imposing either domain adversarial objectives [Ganm and Lempitsky, 2015), distribution align- ment [[Png et all. 2019] or low-rank parameter updates eHu 2021]. However, these techniques may still fail to exploit synergistic relationships-wiie where certain domains are complementary and can reinforce each other's accuracy-and do not fully capture how best to "partition" domains to avoid negative interference.
Motivated by this gap, we propose a partition-based multi-stage framework for multi-domain LLM fine-tuning, Instead of jointly or separately adapting to all domains, our method clusters synergistic domains while isolating highly distinct ones. Our approach integrates practical constraints-memoo budgets, domain shifts, and synergy opportunities-with theoretical insights on generalization bene- fits from restricted updates and strategic domain grouping. The main contributions are:
1) We introduce novel partitioning algorithm that clusters domains according to their synergy and discrepancy, then fine-tunes the L.M in multiple stages. This orchestrated process prevents cross- domain contamination while leveraging beneficial interactions.
2) We derive new bounds that capture domain discrepancy, synergy offsets, and adapter complexity, establishing conditions under which multi-stage partitioning yields provably tighter guarantees than single-stage or naive multi-domain methods.
3) Our experiments demonstrate that partition-based multi-stage fine-tuning outperforms state-of- the-art baselines, improving accuracy across different domains and tasks while reducing memory usage.
2 Related Works
2.1 LLM Fine-Tuning
The success of LLMs such as GPT-3 [Brown all 2021, LLaMA [ouvron et all, 2023), and Falcon [Almarroue _t all 2023] has underscored the importance of efficient fine-tuning. Fully updating model parameters incurs high computational costs [Zhang et all 20248, Hu all 2021]] prompting parameter-efficient methods that modify only small subset of parameters. Examples include adapter modules [Houllby et all, 201%) low-rank projections (LoRA) etall 2022I, and tightly integrated adapters for LLaMM IZZang et all, 20246]] However, most methods assume single domain; adapting L.LMs efficiently to multiple domains remains challenging. Another related field is continual learning [Xu ((25), but it assumes a sequential arrival of tasks, which differs from the setting in this paper.
2.2 Multi-Domain Data Learning
Real-world data often originate from disparate sources with distinct distributions and vocabular- ies [Gann and Lemprtsky, 2015, Sener and Koltun, 2018, Lr all 2023, Zhang et all, 2024., Chen et all 2024, Xıao and Lin, 2025. Ke et all 2025]. Traditional multi-domain methods align rep- resentations through adversarial training [Pei et all 2018, Ganin and Lempitsky, 2015, moment matching [Peng et all 2019), or multi-task objectives [Royer et all, 2024, Shen and Zhang, 2025], yet typically neglect inter-domain synergies. Adapting LLMs further complicates this via memory overhead, forgetting pretrained knowledge, and cross-domain contamination. Adapter-based solu- tions partially address these concerns |Houlsby et all 2019 but rarely exploit domain partitioning to maximize synergy. Our partition-ban based multi-stage approach systematically clusters domains to leverage synergy and minimize discrepancy, providing theoretical guarantees.

---

2.3 LLM Data Selection
The efficacy of supervised fine-tuning (SFT) heavily depends on the quality and composition of training data[Yaa et all 2022 Recent advances have introduced diverse metrics for data selection:
instance-level criteria like perplexity [Cao et 2024], reward scores [Goo and Nguyen. 20241]
and loss disparities [Lt et all 2024b], trajectory-based clustering via small proxy models [Yang atll 2024B), as well token-level selection methods [Lin 2024). Existing works largely overlook one critical factor: domain interactions, where diversity metrics operate at instance/token
levels [[Pang et anl 20231] without modeling cross-domain compatibility [LL all 20255]. Our
approach addresses this gap by systematically clustering domains for joint tuning.
3 Theoretical Analysis
3.1 Preliminaries and Notation
Let us consider & distinct source domains (Di,.... Px). each containing samples (zz, y) drawn from some distribution over X x V. We have a large language model LLMg. pretrained on a massive corpus D pretrain- and assume 8 E IR lies in high-dimensional parameter space. Our goal is to adapt d (often minimally) to each domain Dj by introducing or modifying a small set of parameters (e.g., adapter modules) denoted by Φj E R where 9j p.
Definition (Multi-Source Fine-Tuned Model). A multi-source fine-tuned model is given by
(z) = LLMM++a0 (p1.....
(1)
where AA E R2 a (potentially small) update to the pretrained backbone 8" Each o, may represent additional parameters specialized to domain j. The model internally selects combines relevant oj based on domain context or training strategy.
Loss and Risk. We let (f(), be nonnegative loss function (e.g., cross-entropy) measuring the prediction error sample (Ir, y). For single domain Dj. the expected risk is
[(6, (o ((,,))D, (f), v)]
(2)
When training on multiple domains jointly, we typically minimize an aggregated objective:
Caga (0, (0,(0;)) Σ ag ((),(o:)::))
33)
where 02 2 with 05 1f
for all j. we obtain simple average risk.
3.2 Assumptions and Domain Discrepancies
To handle multi-source adaptation rigorously, we introduce assumptions on data distributions, smoothness, and domain overlaps.
Assumption 3.1 (Lipschitz Loss and Smoothness). Assume E(ú, y) is L-Lipschitz in g. Moreover, suppose for any (0, {0,}) and (0', {df)), the difference in model outputs is bounded by a constant factor in terms of ||A l'll2 and || 02,122- Formally, there exists constant B such that
[l for ( < B B(10 0'112 + Σ 16, 01]12).
(4)
3=1
Definition 2 (Domain Discrepancy). Let d(D,, D,) be the HAP His--diitnnce between two domain distributions D, and Dj. Concretely,
d(D,, D,) hEH RND.
sup
Pr
[Mxx) 1]
Pr x~D;
[(xx) =1]
(5)
where H is suitable hypothesis class.
3

---

3.3 Complexity of Multi-Source Adapter Updates
To preserve the implicit regularization from pretraining (i.e., the beneficial cow-complexity" region 0" has converged to), one typically rextricts either A0 or {Φン} or both. We quantify this through norms/penalties:
Assumption 3.2 (Restricted Adapter Complexity). There exist constants pe, Ps > such that
|
pe:
1 S Pe Vj (L....,k).
(6)
If Po is very small (or zero), this means the backbone remains near 0*: if pa is small, each domain adapter is limited in capacity.
Consider Transformer-based large language model (LLM) with L. layers, each layer containing multi-head attention (MHA) sub-layer and feed-forward network (FFN) sub-layer, plus optional adapter modules for each of domains. Suppose each attention weight matrix Wile) and feed- forward matrix l
iin is constrained by a spectral norm bound (or operator norm) ||]| S Qeare Each domain adapter o at layer Ć constrained by ||o (10f1)l= || S Indapt- We assume bounded input embedding norm || I Cn for sequences of finite length m, the nonlinear activations (e-g., GELU, ReLU) are I-Lipschitz on the relevant domain of outputs. Under these constraints, we can derive uniform convergence or PAC-Bayes-style bounds. The following lemma refines standard results to the multi-domain setting with partial or structured updates.
Lemma 3.1 (Rademacher Complexity for Multi-Adaper Transformers). Let F be the hypothesis class of all such multi-adapter Transformers that respect these norm constraints. Then for n i.i.d. samples per domain from k source domains (DD,....,DD .D.,. there exists constant CT >> 0 (depend- ing on L,, Dcooe: Padopt, m. k, Cin) such that the empirical Rademacher complexity satisfies
1 S C.VY
(7)
indicating that limiting both the core Transformer parameters and the adapter parameters yields class F whose complexity grows on the order of 1 1//n.
Proof. Because ||W lo: | sitn
fin llo I Qeore and activations are I-Lipschitz, each layer l can be shown be (C 02 Lipschitz for some constant C.
Le S C' (22
(8)
(c) Each adapter matrix o satisfies |0 F S Radape. Hence, adapter operations contribute a bounded perturbation at each layer, maintaining overall Lipschitz continuity. The total Lipschitz constant satisfies
L II Le < (C 1=1
Liotal
and inputs bounded by Cin-
By standard covering- seveernn-numbber or PAC-Bayes arguments for neural networks [Barilett and Mendelson. 2002, Neyshabur et al;, 201]) any class of T Lipschizz functions on inputs of norm at most Cnn has empirical Rademacher complexity O [Loaa Ca// /n). Absorbing constants (including & for
multi-domain) into Cr yields Rr, (D,3) s Or V
3.4 Multi-Source Generalization Bounds
Theorem 3.1 (Multi-Source Concurrent Generalization). Let (DI....,Dy) be k source domains, each with ny i...d samples, and let n = Σ nij Assume each domain distribution DD over (x, E x Y, and consider a multi-domain LLM Ta.A,,) satisfying Assumptions 3.1 (Lipschitz- ness) and 1.A (bounded backbone and adapters). Let d(D,, D,)) be a domain- discreacyy measure (Definition Z) and let F. denote the hypothesis class of all (0, {03}) that respect these constraints.
Nj.

---

Then for any confidence level 6 > O, with probability at least å over the choice of { 1=)
from Uن Dj, every model fo, } in F satisfies:
Σ a ((, S aj ((0,(o)):D))
(10)
h 3 +' (ash:) + d(D,, + +0(V (n(1//)
Σ k 3
where Σ(0, (d) is the empirical risk samples from domain The constant B > 0 de- pends on the Lipschitz parameters (L, B) and the number of domains k. The explicit function [(po, Po: {a k) can be chosen as
k Po +
p(pa. Pa. (ay),)) = 2LB
Σ j=1
ag Ps
(11)
reflecting how large backbone updates (pg) and adapter norms (po) can inflate the mulii-domain generalization bound.
Proof Sketch. The bound is the sum of three classic ingredients. (i) Uniform--comergennce: using Rademacher complexity for the norm-restricted class F, the difference between the weighted ex- pected risk E, a. and its empirical counterpart OLB((pp + E)pp)) + +V /lm(1/8) giving the term L' Pé: {a,}, k) = 2LB(pe + E; ajPs). (ii) Domain-shift: standard multi-source adapta- tion results add a penalty proportional to the average pairwise discrepancy & Σi, d(D,, D(). (iii) Combining these with the empirical risk yields inequality (II).. Refer Section C.I for the complete proof.
n
Remark 3.1 (Domain Similarity vs. Model Capacity). Let Dmax == maxi, d(D,, Dj) and recall that the discrepancy penalty in Theorem 3.1 is Ei, d(D;, < 3Dmax. Hence, when all domains are similar (D max < 1) the extra cost is small and the bound is dominated by the complexity term T(po. Po: ax k) 2p.B(po + Σ, a.)p)) This means one can keep po: Pa-and thus T T-small without under-fitting. Conversely, if the domains are very different (Dmax large) the discrepancy term becomes the bottleneck; the learner must allow larger parameter budget (bigger Pa. Ps I) so that each domain receives enough specialised capacity to avoid under-fitting:
To trade off discrepancy, synergy, and per-stage capacity, we partition the & domains into M disjoint stages S,....,,,, SM and maximise
ISS...... SM) = Σ Σ d(D,,
A Σ s(D,, + pa p0 16o
11
S. (5
i.js [C]
JES.
capucity cost Cap(S.)
toral discrepancy
total synergy
(12)
where D();, = JS(P.) E 1] is the Jensen-Shannon divergence between the empirical token-distribution of the two domains; s(D,, D,) JacdV, Vi) +
cos( Mj) )
[0,
mecah-oeerlap
emberkling coxine
combines lexical and semantic affinity (higher= more synergy); À 0 balances "rewarding" synergy against "penalising" discrepancy; Hs > 0 weight the squared-norm budget of the backbone drift Ag : 8¹ gt-l and the stage-specific adapters {
A larger value of G therefore corresponds to: (i) smaller internal discrepancies, (ūi) larger construc- tive synergy, and (iii) lower per-stage parameter cost. Maximising (l12) over all M-partitions yields the partition that minimises the generalisation upper- derived in Theorem k
5

---

Theorem 3.2 (Multi-Stage Partition with Synergy- Capacity Maximisation), Let the k source do- mains be split into M disjoint stages Si..... Sm and let the stage-objective G(St....,SS) be defined in (T2). Write ......., Sir) == arg max LJ, S,=(1:k3 ((S Sar). (.51....,as)) Then, under Assump- tions 1 and B2A the predictor obtained after the last stage, for .oM 4 satisfies with probability at least 1 8:
Synergy-Capacity
R Renad (Wi...
Sw.) $ [1 [1-g(S) -.................. Sw)] + O(V (n(1/6))
(
(13)
where N = - partition attains a larger right-hand side.
nj. Rmax == max; Ejes. of LD, (f), and [u]+ = max u). Any other
Proof Sketch. Apply the single-stage bound (Theorem [
stage-wise. For stage t the risk is con-
trolled by empirical loss + capacity+ discrepancy- synergy. Summing the worst stage and noting
that empirical losses are < 1 yields (I3). Because (-) appears inside the bracket, maximising G minimises the bound, proving optimality of the partition Appendix O%
full derivation is given in
Corollary 3.1 (High-Syun -liggySynergy Subset Tends be Grouped Together). Let (Di...., D& J be ks do- mains with a synergy sisrrvevdccrepacyccppppa objective (((S))) as defined in (I2). Suppose there exists a nonempty subset U S [L.....k) such that any pair (i,j) in U satisfies
D,) S and Synergy (D; 2 A,
(14)
where A large relative to y and to the capacity penalty Cap(U). Then, in the optimal partition arg max SSx) (( Sm]). the domains U will rypically be placed a single stage
Sü, provided
A x-1(7 + Cap(U)).
(15)
That is. if the synergy within U is sufficiently large compared its internal discrepancy and added capacity cost, then clustering those domains together in the same stage yields a higher objective G, thereby lightening the final multi-stage generalization bound.
Proof. Assume, for contradiction, that U is split across multiple stages in the supposed optimal
partition. Because synergy offsets discrepancy by X Synergy(,.)) each pair (i.j) E U that lies in different stages forfeits this positive synergy benefit. Thus, the total contribution to G(-) from U decreases by at least X(A - }) per stase pair, which outweighs any savings in capacity usage
provided that > *+Cap(UU) Hence, merging U into single stage increases 9 and yields a strictly better partition, contradicting optimality. Thus U must remain in one stage in {S?...., Shr.. D
4 Algorithm
We now present a practical procedure implementing our theoretical insights from Section B. Algo- rithm MI details the steps to: (1) partition i domains into up M stages (sets) to maximize synergy and control discrepancy/capacity and (2) perform stage-wise adapter tuning under bounding norms for both the LLM backbone and the Nomain-speciffe adapters.
Computational complexity. The only extra overhead of our method occurs during the partition step. Forming the discrepancy and synergy matrices requires O(k2) pairwise computations, each obtained once from cached token or embedding statisties. We maximise with a single-link ag- glomerative search, which runs in Olk-log k) time and O(k2) memory; an exact ILP solver gives the same split for our < 10 domains in under 0.1s, but the heuristic already within 1% of the optimum. Afterwards, each stage performs supervised data once-no replay or re-weighting-so runtime and GPU memory are identical to a standard single- pass SFT run, apart from the tiny adapter parameters (<) 1% of the backbone). Overall complexity therefore O(k-log k) + (single-pass SFT); with the moderate domain counts typical practice, the partition phase is negligible in both time and memory.
(SFT) of the LLM on its assigned

---

Algorithm Multi--Stae Adapter Tuning for LLMs
Require:
Pretrained LLM parameters 0* RP; k source domains (Pi...., P discrepancy mea-
sure d(,,, D3); synergy measure Synergy (D,, . )) capacity cost Cap(-); norm bounds pe, P number of stages M; (optional) mixing weights {a}).
Ensure: Final backhone parameters BM; domain adapter parameters (0] 1ー-
1: Partition step: Select disjoint subsets (SI,...,,SM of [1..... k) to approximately solve the objective given in the theoretical section (see (I22) and Theorem (.2)).
2: Initialize: 0 <- 0", $ <- 0 for = l....,,
3: for 1 to M do
4: Stage-/ domains: S: determined by the partition in Line 1.
5: Form the stage objective: Use the multi-domain loss from Equation (II), enforcing ||0'
||- S Po and [0 12 S Pg-
6: Optimize:
(0' (ot + Optimizer(6" (0r 10
3. Ds.).
7:
Outside-stage adapters:
0f = o0- for j $ St.
8: end for
9: Output: 0M,
(
Experiments
Datasets We evaluate method four representative multi-domain language understanding tasks: 1) News Summarization (NSum) Hermann et all [2015]] dataset of news articles paired with short summaries. We measure summarization quality via ROUGE-L. 2) Sentiment Classifica- tion (Sent) Socher all [201]] Sentences labeled with positive/negative sentiment. We measure accuracy (ACC). 3) Question Answering (Q&A) Rajpurkan [2016]. Documents and question-answer pairs. We measure exact-match (EM) and Fl scores. 4) Topic Categorization (Topic) Zhang et al, coarse-grained topics. We measure classification accu- racy (ACC). We partition each dataset into training, validation, and test splits. Statistics (number of samples, average text length, etc.) are presented in Appendix A.
[2015]. Short text passages assigned
Pretrained Models We employ three popular open-source large language models (LLMs), all of which are publicly available via the HuggingFace Transformers library: LLaMA2-7B fou T-billion-paramette
vron et all [2023]] A 7-billion parameter model trained on a large, diverse corpus. 2) LLaMA2- 13BTouvron et all [2025]] A wwwlbioo-ppaaameeeeee model offering improved capacity and perfor- mance over the 7B variant. 3) Falcon-40B/Almazroonn et al [202]]. A 40-billion- parameter model pretrained on the RefinedWeb dataset, demonstrating sfate-ofthhe-aee art generative abilities. Each model is pretrained on diverse textual sources. We use their publicly released checkpoints for all experiments.
Baselines We compare PMS-FTP against the following baselines: 1) Base Methods: Full Fine-
Tuning (FULL), Fixed Backbone (FIXED); 2) Domain Adaptation: Multi-Domaii Adversarial Net- work (MDAN) [Pei all 2018], Moment Matching (M$SDA) [Peng et al, 2019), Bayesian Gaus- sian Mixture (GMDI) [Ling et all, 20221] 3) Single-Domain LLM Fine-tuning: LoRA [Hu all
X02II] Adapter |Houlsby et all, 2019). LLaMA-Adaple [Zhang al, 20024)) Q-LoRA [Detimers
2023], TLM Shen al all 2024:]; 4) Data Selection: INSTRCCTMINING (IT) Ca
2022
S2L Yang et all 20247]] Please refer to A. for more details the baselines.
5.1
Experimental Results
Overall Comparison. Table summarizes the test-set performance for each model and method on all four tasks. PMS-FTP consistently surpasses baseline methods across all tasks and model sizes. Compared with strong data-selection (S21.) and single-domain adapter methods aL. Adapter, Tag-LLM), PMS-FTP achieves improvements by strategically exploiting domain synergies
7

---

Table 1: Performance comparison three LLM backbones. PMS-FTP denotes proposed Partition- Based Multi-Sine Multi-Stage Fine- Tuning Best results are in bold.
LLaMA2-BB
LLaMAZ-13B
Falcun-40B
Method
NSum
Q&A
Sent
Topic
NSum
Q&A
Sent
Topic
NSum
Q&A
Sent
Topic
Methods
FULL
41.2
64.7
89.0
86.5
42.1
66.3
89.8
87.1
43.2
68.2
90.4
88.3
FIXED
38.9
59.5
87.4
85.2
39.6
61.2
88.3
85.7
40.7
63.0
88,9
86.1
Domain Adoptiation
MDAN Wereial CTTX
39.7
62.8
88.1
85.9
40.5
64.0
88,9
86.3
41.7
66.1
89.3
87.0
MDS Peag deal
40.5
63.1
88.6
86.1
41.7
64.9
89.4
86.7
42.3
66.6
89,9
87.4
GMDI
[ing el MMP
40.8
63.5
88.7
86.4
42.0
65.4
89.6
87.0
42.7
67.1
90.0
87.6
Single-Domoia 1/M Fine-tuning
LoRA [ UUA4
41.0
63.9
88.4
86.2
42.0
65.1
89.1
86.9
42.5
66.5
89.8
87.7
Adapter (by
41.3
64.1
86.3
-2.33
65.7
89.5
87.2
42.9
67.0
90.2
88.0
aMAAAdapee (Zhamp ala INzee
41.5
64.3
89.
86.7
42.6
65.9
89.7
87.5
43.1
67.3
90.3
88.
Q-LoRA [Detmens of cinnl MUX
41.7
64.4
89.0
86.5
-2.44
65.6
89.5
87.3
43.0
67.2
90.1
87.9
Tag-IIL (Stienetail Wie!!
89.2
42.7
06.1
89.8
87.6
43.3
67.5
90.5
88.2
Data Selection
INSTRUCTMINING [Caurtal Moze
89.9
87.7
67.6
90.6
S2L (Yang et at [AМ
41.9
64.7
89.4
87.0
42.9
66.2
90.0
87.8
43.5
67.8
90.7
88.4
PMS-FTP
89.0
Table 2: Domain-specific performance improvements (LLaMA2-133 backhone).
Domain Grouping
Synergy Score
Diserepancy Score
Avg. Baseline
PMS-FTP
Performance Gain (%)
NSum & Q&A
0.88 (High)
0.12 (Low)
64.3
66.1
+1.8%
Sent & Q&A
0.85 (High)
0.15 (Low)
89.5
91.2
+1.7%
Q&A Topic
0.80 (High)
0.20 (Low)
76.7
78.3
+1.6%
Sent Topic
0.65 (Moderate)
0.30 (Moderate)
88.1
89.4
+1.3%
NSum Sent
0.60 (Moderate)
0.40 (Moderate)
65.2
66.4
+1.2%
Q&A & Sent & Topic
0.58 (Moderate)
0.42 (Moderate)
77.8
79.0
+1.2%
NSum Topic
0.40 (Low)
0.60 (High)
64.6
65.5
40.9%
Sent NSum & Topic
0.35 (Low)
0.65 (High)
80.5
81.3
+0.8%
and mitigating negative interference. On LLaMA2-13B vs. ILLMMA-7B, every method sees mod- erate performance jump, but PMS-FTP consistently maintains the largest margin above the best baseline. Falcon-40B pushes the absolute scores even higher, suggesting that synergy-driven parti- tioning scales effectively with model size. Table II in Appendix E presents additional experimental results, analyzing why conventional DA baselines lag behind FULL.
Domain-specific performance improvements. We analyze domain synergies and discrepancies (Table D).. High synergy pairs (e.g., NSum & Q&A, Sent & Q&A) show substantial gains (+1.8%, +1.7%) indicating effective leveraging of complementary domains. Moderate synergy pairs (e.g., Sent & Topic) also show meaningful improvements (+1.39% 1.3%), while even high-discrepancy pairs (e.g.,
NSum & Topic) achieve modest gains (+0.9%). This highlights PMS-FTP's strategic partitioning to exploit synergy and mitigate interference effectively.
Loss Analysis. Figure I1 illustrates the training- loss curves on the Q&A domain using LLaMA2- 13B for several representative methods (FULL, LoRA, LM, Adapter, and our PMS-FTP). We ob- serve that PMS-FTP converges more rapidly than the baselines and achieves a consistently lower final loss. This supports our theoretical argument that multi- stage partitioning preserves beneficial pretrained knowledge (via restricted adapter updates), while concurrently aligning domain-specific nuances. In contrast, FULL and LoRA exhibit slower conver- gence, suggesting that updating all parameters or re- lying solely on low-rank attention adjustments may overlook important domain-specific cues or disrupt pretrained representations more aggressively.
Figure 1: Training loss curves the Q&A domain with LLaMA2-13B.

---

Memory Usage. Table B reports the peak allocated memory (in GB) dur- ing fine-tuning on the Q&A task with LLaMA2-7B. We measure memory usage using a single NVIDIA A100 GPU. The FULL method requires the largest memory footprint due to up- dating all model parameters. LORA and LLAMA-ADAPTE substantial savings via sparse or low- rank updates. Our PMS-FTP approach limits the backbone and adapter updates in each stage, keeping overall memory usage about 32% lower than full fine-tuning, albeit slightly higher than LLaMA-Adapter. Nonetheless, the stronger accuracy (see Table M1) indicates favorable trade-off between memory efficiency and final performance.
Table 3: Peak GPU memory (in GB) during Q&A fine-
tuning on LLaMA2-7B.
Method
Peak GPU (GB)
Relative Reduction
FULL
27.2
LoRA
19.6
27.9%
LLaMA-Adapter
17.2
37.1%
PMS-FTP (Ours)
18.4
32.4%
both yield
5.2 Ablation Study
We further examine how each design choice in PMS-FTP impacts final performance. Specifically, investigate (i) the number of stages, (ii) the partition strategy (synergy-basee vs. random), (iii) the effect of limiting update norms (i.e., || & pe. | 222 S Ps). and (iv) synergy metric sensitivity. Experiments in this section use the I.L.MA27BB backbone and evaluate subset of domains (NSum and Q&A) for brevity.
Effect of Number of Stages (M). In Table E we compare
M 1 (single-stage updates), M = 2. and M = 4 (one stage per domain). We also in- clude random domain group- ing for M 2 illustrate the importance of synergy-driven partitioning. Specifically, we measure ROUGE-L on NSum and EM on Q&A. Single-stage (M 1) fine-tuning, akin to multi-task learning without adapter updates, underperforms on both tasks. two-stage synergy- based partition achieves the best results, balancing synergy and discrepancy. In contrast, four stages (M = 4) frer-fragment data, reducing synergy benefits.
Table 4: Ablation on the number of stages (M) and partition strategies. PMS-FTP with synergy- enaee grouping (M 2) outperforms a random partition and single-/all-done stage ex- tremes.
Setting
Partition Strategy
NSum (ROUGE-L)
Q&A (EM)
domains in one stage
41.2
M (Random)
Random domain grouping
41.7
63.9
M
(Synergy)
Synergy-driven
42.2
64.8
per stage
42.0
Effect of Norm Constraints (pe, Po). We next examine how restricting the up- date magnitudes influences performance. By default, we set ||0t 06-1|/ & Po and ||øš S Ps preserve the pretrained backbone's inductive bias In Table E. we vary Pa and Pe in {0.05, 0.1, 0.2). measuring average performance across
Table Restricting update norms improves stability and performance. We report average scores (ROUGE- L for NSum, EM for Q&A).
Po
Pe
NSum (ROUGE-L)
Q&A (EM)
0.05
0.05
41.2
63.9
0.05
0.10
41.7
64.5
NSum/Q&A. Too small norms (e.g. Pe Ps 0.05) hamper the model's capac-
0.05
0.20
41.5
64.1
ity adapt, leading
suboptimal per-
0.10
0.05
41.6
64.3
formance on NSum and Q&A. Larger
0.10
0.10
42.2
64.9
norms (0.2) let the model deviate more
0.10
0.20
42.0
64.6
from 8* but risk overfitting. Empirically,
0.20
0.05
41.4
64.1
(pe, Ps) = (.11, or (.2) deliver
0.20
0.10
42.1
64.7
the best results, suggesting a moderate ca-
0.20
0.20
42.1
64.8
pacity fosters the best balance preserv-
ing pretrained knowledge vs. domain-specific adaptation.

---

43.0
66.0
Synergy Metric Sensitivity. Our partition- based approach applies a synergy coefficient A to balance domain synergy against dis- crepancy (Equation (I)).. We vary À E (0.0, 0.25, 0.5, 0.75, 1.0] to observe its impact on partitioning and final performance. Fig- ure 2 (LLaMA2-7B, M 2 stages) reports ROUGE-L (NSum) and EM (Q&A). When = 0.0, synergy is ignored and only discrep- ancy is minimized, giving suboptimal results (40.8 ROUGE-L). Excessively large À (e.g. 1.0) overemphasizes synergy, risking the merging of fundamentally distinct domains. A moder- ate À E [0.25, 0.50] achieves the best trade-off, aligning with Theorem 3.2
ROUGEA
42.5
1M
05.3
N5am NSum, (ROUGE-)
42.0
41.5
41.0
40.5
40.0
67.0
30.5
0.0
0.2
0.4
0.6
0.8
1.0
61.5
Figure 2: Synergy metric sensitivity.
Conclusion
this work, we introduced partition-based mullistaage fine-tuning framework to systematically ad- dress multi-domain adaptation in large language models. By quantifying each domain's 's discrepancy and synergy and jointly optimizing a partition objective, we balance shared feature learning with domain-speciff specialization. Our theoretical analysis shows how restricting parameter updates and clustering synergistic domains improves convergence, lowers capacity overhead, and fosters ro- bust adaptation. Extensive experiments different tasks and backbones confirm these advantages.
In future work, we aim to adapt this stage-wise procedure to continual learning setting, where new domains arrive sequentially, thereby offering a more flexible lifelong learning framework for large language models. Furthermore, combining the proposed method with pruning strategies [Lm etall 20241 dealing with models that have massive number of parameters.
2025, Lretall [0258| represents an interesting direction, especially when
Acknowledgements
We would like to thank Hunan Airon Technology Co., Ltd. for providing data preprocessing services and computing resources.
References
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Co-
jocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. The falcon series of open language models. arXiv preprint arXiv:2311.16680 2023.
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):433-48, 2002.
John Blitzer. Domain adaptation of natural language processing systems. PhD thesis, University of Pennsylvania, 2008
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Yihan Cao, Yanbin Kang, Chi Wang, and Lichao Sun. Instruction mining: Instruction data selection for tuning large language models. In First Conference Language Modeling. 2024.
Kaixiang Chen, Pengfei Fang, and Hui Xue. Depro: Domain ensemble using decoupled prompts for universal cross-domain retrieval. Proceedings of the 48th International ACM SIGIR Conference Research and Development in Information Retrieval, SIGIR '25, page 958967, 2025a.
10

---

Xiaohong Chen, Canran Xiao, and Yongmei Liu. Confusion-resistant federated learning via diffusion-based data harmonization on non-iid data. In Proceedings of the 38th International Conference on Neural Information Processing Systems, pages 137495-137520, 2024.
Xiaohong Chen, Canran Xiao, Wenzhi Cao, Weiwei Zhang, and Yongmei Liu. Framework and pathway for the construction of a unified data-element market in china. Strategic Study of Chinese Academy Engineering, 27(1):40-555 50, 2025b.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36: 10::10088-0111 10115, 2023.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Proceedings of the 32nd International Conference on Machine Learning (ICML), pages 1180 1189, 2015.
Qi Gou and Cam-TTu Nguyen. Mixed preference optimization: Reinforcement learning with data selection and better reference model. arXiv preprint arXiv:2403. 194433 2024.
Qi He. A unified metric architecture for ai infrastructure: A cross-layer taxonomy integrating per- formance, efficiency, and cost. arXiv preprint arXiv:2511.21772, 2025.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 2015.
Neil Houlsby, Andrei Giurgiu. Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo. Mona Attariyan, and Sylvain Gelly. Parameter-effiiint transfer learning for nlp. In International conference on machine learning, pages 2790-2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprini arXiv:2106.0960 006885666688 2021.
Wentao Hu, Wengyu Zhang, Yiyang Jiang, Chen Jason Zhang, Xiaoyong Wei, and Li Qing. Re- moval of hallucination on hallucination: Debate-augmented RAG. In Wanxiang Che. Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd An- mual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15839-15855 Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 109-8-89176-2111 doi: 10. 18653/ http:///222...o aclloo.. 770. URL 2025 a025.ocl_ 7704
org/
Linling Jiang, Xin Wang, Fan Zhang, and Caiming Zhang. Transforming time and space: efficient video super-resolution with hybrid attention and deformable transformers. The Visual Computer, pages 1-12, 2025.
Yiyang Jiang, Wengyu Zhang, Xulu Zhang, Xiao Yong Wei, Chang Wen Chen, and Qing Li. Prior knowledge integration via Ilm encoding and pseudo event regulation for video moment retrieval. In Proceedings of the 32nd ACM International Conference on Multimedia, MM '24, page 72497258, New York, NY. USA, 2024. Association for Computing Machinery. ISBN 9798400706868, doi: 10. 10.1145/3664647..888 368 15. URL https://doi. org/10. 1115//564115//644477777 3681115
Nikitas Karanikolas, Eirini Manga, Nikoletta Samaridi, Eleni Tousidou, and Michael Vassilakopou- los. Large language models versus natural language understanding and generation. In Proceedings of the 27th Pan-Hellenic Conference on Progress in Computing and Informatics, pages 278-290- 2023.
Zong Ke. Yuqing Cao, Zhenrui Chen, Yuchen Yin, Shouchao He, and Yu Cheng. Early warning of cryptocurrency reversal risks via multi-source data. Finance Research Letters, page 107890, 2025.
Pranjal Kumar. Large language models (Ilms): survey, technical frameworks, and future challenges. Artificial Intelligence Review, 57(10)2260, 2024.

---

Boyi Li, Yue Wang, Jiageng Mao, Boris Ivanovic, Sushant Veer, Karen Leung, and Marco Pavone. Driving everywhere with large language model policy adaptation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 14948-14957, 2024a.
Haoang Li, Ji Zhao, Jean-Charles Bazin, Pyojin Kim, Kyungdon Joo, Zhenjun Zhao, and Yun-Hui Liu. Hong kong world: Leveraging structural regularity for lise- slam. IEEE Transactions on Puttern Analysis and Machine Intelligence, 45(1 10355-3--33 2023.
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting Ilm performance with self-guided data selection for instruction tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Vol- 2024b.
ume 1: Long Papers), pages
Yuqi Li, Kai Li, Xin Yin, Zhifei Yang, Junhao Dong, Zeyu Dong, Chuanguang Yang, Yingli Tian, and Yao Lu. Sepprune: Structured pruning for efficient deep speech separation. arXiv preprint arXiv:2505. 2025a.
Yugi Li, Chuanguang Yang, Junhao Dong, Zhengtao Yao, Haoyan Xu, Zeyu Dong, Hansheng distillation for
Zeng, Zhulin An, and Yingli Tian. Ammkd: Adaptive multimodal lightweight vision-language models. arXiv preprint 2tt.::5500000333 2025b.
Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Ruochen Xu, Chen Lin, Yujiu Yang. Jian Jiao, Nan Duan, Weizhu Chen, al. Not all tokens are what you need for pretraining. Advances in Neural Information Processing Systems, 37:29029-29063 2024.
Zhiming Lin, Kai Zhao, Sophie Zhang, Peilai Yu, and Canran Xiao. Cec-zero: Zero-supervision character error correction with self-gene rewards. arXiv preprint arXiv:2512.2399 2025.
Yanfang Ling, Jiyong Li, Lingbo Li, and Shangsong Liang. Bayesian domain adaptation with gaus- sian mixture domain-indexing. Advances in Neural Information Processing Systems, 37:87226~ 87254, 2024.
Jiaqi Liu, Tong Wang, Su Liu, Xin Hu, Ran Tong, Lanruo Wang, and Jiexi Xu. Lightweight baselines for medical abstract classification: Distilbert with cross-entropy as strong default. arXiv preprint arXiv:2510. 10025, 2025.
Hao Lu, Ziyang Liu, Guangfeng Jiang, Yuanfei Luo, Sheng Chen, Yangang Zhang, and Ying- Chen. Uniugp: Unifying understanding, generation, and planing for end-to-end autonomous driving. ByteDance Seed Tech Report, 2025a.
Hao Lu. Zhuang Ma, Guangfeng Jiang, Wenhang Ge, Bohan Li, Yuzhan Cai, Wenzhao Zheng, Yunpeng Zhang, and Yingcong Chen. 4d driving scene generation with stereo forcing, arXiv preprint arXiv:2509.2001 10251,, 2025b.
Wei Lu, Rachel K Luu, and Markus J Buchler.
Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilities. arXiv preprint W-XXv:2409.0344/ 2024a.
Yao Lu, Wen Yang, Yunzhe Zhang. Zuohui Chen, Jinyin Chen, Qi Xuan, Zhen Wang, and Xiaoniu Yang. Understanding the dynamics of dans using graph modularity. In European Conference on Computer Vision, pages 225-242. Springer, 2022.
Yao Lu, Yutao Zhu, Yuqi Li, Dongwei Xu, Yun Lin, Qi Xuan, and Xiaoniu Yang. generic layer pruning method for signal modulation recognition deep learning models. IEEE Transactions on Cognitive Communications and Networking, 2024b.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. adaptive computation and machine learning, 2018
Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the sum- mary! awpie- convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745 2018.
12

---

Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring general- ization in deep learning. Advances in neural information processing systems. 30, 2017.
Jinlong Pang, Jiaheng Wei, Ankit Parag Shah, Zhaowei Zhu, Yaxuan Wang, Chen Qian, Yang Liu, Yujia Bao, and Wei Wei. Improving data efficiency via curating llm-driven rating systems. arXiv preprint arXiv:2410. 10877, 2024.
Zhi Pei, Zhen Cao, Mingsheng Long, and Jianmin Wang. Multi-adversarial domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3934-3941, 2018.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEFE/CVF international conference on computer vision, pages 1406-1415, 2019.
P Rajpurkar. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv: 1606.05500 2016.
Amelie Royer. Tijmen Blankevoort, and Babak Ehteshami Bejnordi. Scalarization for multi-task and multi-domain learning at scale. Advances in Neural Information Processing Systems, 36, 2024.
Antony Seabra, Claudio Cavalcante, Joao Nepomuceno. Lucas Lago, Nicolaas Ruberg, and Sergio Lifschitz. Contrato360 2.0: A document and database-driven question-answe answer system using large language models and agents. arXiv preprint arXiv:2412.17942. 2024.
Ozan Sener and Vladlen Koltun. Multi-task learning multi-objectivve optimization. Advances in neural information processing systems, 31, 2018
Junhong Shen, Neil Tenenholtz, James Brian Hall, David Alvarez-Melis, and Nicold Fusi. Tag-Ilm: repurposing general-purpoonn Iims for specialized domains. In Proceedings of the 41st Interna- 2024.
tional Conference on Machine Learning, pages
Qiannan Shen and Jing Zhang. Ai-enhanced disaster risk prediction with explainable shap analy- sis: A multi-class classification approach using xgboost, December 2025. URL attps://wwe researchsquare. com/ aataaaatiilerrreeeeee 2025.
www.:/aaae..ooaarae////eeeeeeeee /ttp://rrccceeee-4//////111 Preprint, Version 1, posted December 31,
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pages 1631-1642, 2013.
Wenxi Sun, Qiannan Shen, Yijun Gao, Qinkai Mao, Tongsong Qi, and Shuo Xu. Objective over architecture: Fraud detection under extreme imbalance in bank account opening, Computa- tion, 13(12):290, 2025a. doi: 10.3390/computttiion 13 120290. URL https::/ mdpi com/ 2079 2079-3197/13/122290
Yu Sun. Yin Li, Ruixiao Sun, Chunhui Liu, Fangming Zhou, Ze Jin, Linjie Wang, Xiang Shen, Zhuolin Hao, and Hongyu Xiong. Audio-enhanced vision-language modeling with latent space broadening for high quality data expansion. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2. KDD '25, page 48724881, New York, NY, USA, 2025b. Association for Computing Machinery. ISBN 9798400714542. 3737195. URL https: org/ 10.
doi: 10.1145/3711866
0etpg//0.111457711 1145/3711896 3737195
Huiri Tan, Juyong Jiang, and Jiasi Shen. Profix: Improving profie- siille-guied optimization in compilers with graph neural networks. In Advances in Neural Information Processing Systems, 2025. URL
https::/neun ips cc cc/virtual//00 /ttp://aaaa///////////ee pospp///tttrrre
Hailin Tao, Jinjiang Ii,, Zhen Hua, and Fan Zhang. Dudb: deep unfolding- basedd dual-branch feature fusion network for pan-sharpening remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 62:1-17, 2023.
Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29(8): 1930-199000 2023.
13

---

Ran Tong, Jiaqi Liu, Su Liu, Xin Hu, and Lanruo Wang. Renaissance of rns in streaming clin- ical time series: Compact recurrence remains competitive with transformers. arXiv preprint arXiv:2510. 16677, 2025.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, al. Llama Open founda- tion and fine-tuned chat models. arXiv preprint (xXiv23377.0288, 2023.
Dave Van Veen, Cara Van Uden, Maayane Attias, Anuj Pareek, Christian Bluethgen, Malgorzata Polacin, Wah Chiu, Jean-Benoit Delbrouck, Juan Manuel Zambrano Chaves, Curtis P Langlotz, et al. Radadapt: Radiology report summarization via lightweight domain adaptation of large language models. arXiv preprint arXiv:2305.0113 2023.
Hua Wang and Fan Zhang, Computing nodes for plane data points by constructing cubic polynomial with constraints. Computer Aided Geometric Design, 11:110230. 2024.
Yuenan Wang, Hua Wang, and Fan Zhang, A medical image segmentation model with auto-dynamic convolution and location attention mechanism. Computer Methods and Programs in Biomedicine, 261:108593, 2025.
2025.
Junjie Wen, Jinqiang Cui, Zhenjun Zhao, Ruixin Yan, Zhi Gao, Lihua Dou, and Ben M. Chen. Syreanet: physically guided underwater image enhancement framework integrating synthetic and real images. In IEEE International Conference on Robotics and Automation (ICRA), pages
5177-5183, 2023.
Yuhuai Wu, Albert Qiaochu Jiang, Wenda Li, Markus Rabe, Charles Staats, Mateja Jamnik, and Christian Szegedy. Autoformalization with large language models. Advances in Neural Informa- tion Processing Systems, 35:32353-32366 2022.
Canran Xiao and Yongmei Liu. A multifrequency data fusion deep learning model for carbon price prediction. Journal of Forecasting, 44(2):436-458, 2025.
Canran Xiao, Liwei Hou, Ling Fu, and Wenrui Chen. Diffusion- frlfusion-bbsed self-supervised imitation learning from imperfect visual servoing demonstrations for robotic glass installation. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 10401-10407. IEEE, 2025a.
Canran Xiao, Chuangxin Zhao, Zong Ke, and Fei Shen, Curiosity meets cooperation: A game- theoretic approach to long-tail tail multi-label learning. arXiv preprint arXiv:2510.17520, 2025b.
Mengzhu Xu, Hanzhi Liu, Ningkang Peng, Qianyu Chen, and Canran Xiao. Affordance- first decomposition for continual learning in video-language understanding. arXiv preprint arXiv:2512.0060 2025.
Shaocheng Yan, Yiming Wang, Kaiyan Zhao, Pengcheng Shi, Zhenjun Zhao, Yongjun Zhang, and Ji- ayuan Li. Hemora: Unsupervised heuristic consensus sampling for robust point cloud registration. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 1363-1373, 2025.
Kailai Yang, Tianlin Zhang, Ziyan Kuang, Qianqian Xie, Jimin Huang, and Sophia Ananiadou. Mentallama: interpretable mental health analysis on social media with large language models. In Proceedings of the ACM Web Conference 2024, pages 4489-4500, 2024a.
Yu Yang, Siddhartha Mishra, Jeffrey Chiang, and Baharan Mirzasoleiman. Smaltolarge (s2I): Scal- able data selection for tne-tuning large language models by summarizing training trajectories of small models. Advances in Neural Information Processing Systems, 37:83465 1183465--3349,, 2024b.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint 10X.:v//009009000 2018.
Jiawei Yao, Chuming Li, Keqiang Sun, Yingjie Cai, Hao Li, Wanli Ouyang, and Hongsheng Li. Ndc- scene: Boost monocular 3d semantic scene completion in normalized device coordinates space. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9421-9431. IEFE Computer Society, 2023.
14

---

Jiawei Yao, Chuming Li, and Canran Xiao. Swift sampler: Efficient learning of sampler by 10 parameters. Advances in Neural Information Processing Systems. 37:59030-59053, 2024.
Shuang Zeng, Xinyuan Chang, Mengwei Xie, Xinran Liu, Yifan Bai, Zheng Pan, Mu Xu, and Xing Wei. Futuresightdrive: Thinking visually with spatio-temporal cot for autonomous driving, arXiv preprint wtw.ii::505.1673333.. 2025a.
Shuang Zeng, Dekang Qi, Xinyuan Chang, Feng Xiong, Shichao Xie, Xiaolong Wu, Shiyi Liang, Mu Xu, and Xing Wei. Janusvin: Decoupling semanties and spatiality with dual implicit memory for rision-language navigation. arXiv preprint arXiv:2509.224: 22544, 2025b.
Fan Zhang, Gongguan Chen, Hua Wang, Jinjiang Li, and Caiming Zhang, Multi-scale video super- resolution transformer with polynomial approximation. IEEE Transactions on Circuits and Sys- tems for Video Technology, 33(9):44964433 4506, 2023.
Fan Zhang, Gongguan Chen, Hua Wang, and Caiming Zhang. Cf-dan: Facial-expression recogni- tion based on cross-fusion dtal-attention network. Computational Visual Media, 10(3):593-606 608,
2024a.
Jingyu Zhang, Wenging Zhang, Chaoyi Tan, Xiangtian Li, and Qianyi Sun. Yolo-ppa based efficient traffic sign detection for cruise control autonomous driving. In Proceedings of the 2024 Inter- national Conference on Industrial Automation and Robotics (IAR), pages 8-13. ACM, 2025a. doi: 10.1 00.11444/777440237774444
URL https://doi org/ 1tt.:/0.0//.1111/77070000000 1145/3707400.3707400
Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li, and Peng Gao. Llama- adapter: Efficient fine-tuning of large language models with zero-initialized attention. In The Twelfth International Conference on Learning Representations, 2024b.
Xiang Zhang. Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classi- fication. Advances neural information processing systems, 28, 2015.
Xiaofeng Zhang, Fanshuo Zeng, Yihao Quan, Zheng Hui, and Jiawei Yao. Enhancing multimodal large language models complex reason via similarity computation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39(10), pages 10203-10211, 2025b.
Xuanyu Zhang, Zecheng Tang, Zhipei Xu, Runyi Li, Youmin Xu, Bin Chen, Feng Gao, and Jian Zhang. Omniguard: Hybrid manipulation localization via augmented versatile deep image wa- termarking. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 3008-3018, 2025c.
Zijian Zhang, Shuchang Liu, Jiaao Yu, Qingpeng Cai. Xiangyu Zhao, Chunxu Zhang, Ziru Liu, Qidong Liu, Hongwei Zhao, Lantao Hu, et al. M3oe: Multi Multi-domain multi-task mixture-of experts recommendation framework. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 893-902, 2024c.
Jiawei Zheng, Hanghai Hong, Xiaoli Wang, Jingsong Su, Yonggui Liang, and Shikai Wu. Fine-tuning large language models for domain-specific machine translation. arXiv preprint arXiv:2402.15061. 2024.
Yixiao Zhou, Ziyu Zhao, Dongzhou Cheng, Zhiliang Wu, Jie Gui, Yi Yang, Fei Wu, Yu Cheng, and Hehe Fan. Dropping experts, recombining neurons: Retraining freee pruning for sparse mixture- of-experts 1lms. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 15169-15186 2025.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly and accurately present our contributions, which include novel partition-baser fine-tuning framework, theoretical analysis with gen- eralization bounds, and extensive experimental validation. All claims are fully supported by theoretical results in Section 3 and experimental results in Section 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the computational overhead associated with domain partitioning and provide empirical insights into scalability.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "LLmitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might be used reliably provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
16

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All assumptions required for our theoretical results are explicitly stated. Com- plete proofs are included in the supplemental material (Appendices), with concise proof sketches provided in the main text (Section Theoretical Analysis).
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper provides detailed experimental settings, including dataset descrip- tions, data splits, hyperparameters, model configurations, and optimization procedures, clearly presented in the main text and appendices.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
17

---

(d)
The authors should provide scripts to reproduce all experimental results for the new reproducible, they We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Due to institutional restrictions and proprictary considerations, the data and code used in this study are not publicly available this time. However, comprehensive de- tails, including dataset descriptions, model configurations, hyperparameters, and training procedures, are provided in the main text and supplemental materials facilitate repro- ducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (ittps:)
y /mips.cc/public/muudeee
www.iissssooopciooiccccicm for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The paper explicitly describes training and test splits, model architectures, choice of hyperparameters, optimization methods, and computational settings in the exper- imental sections and appendixs.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
18

---

Answer: [Yes]
Justification: The paper reports experimental results with clearly defined error bars, calcu- lated the standard deviation across multiple independent runs.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of eтrоrs is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
The paper clearly specifies the computational resources utilized, including GPU type, memory requirements, execution time per run, and overall compute needed for each experimental setting.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have thoroughly reviewed the NeurlPS Code of Ethics and confirm that our research fully complies with the guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
19

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA].
Justification:
There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimeess considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
allow for controlled use of the model, for example by re-
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
20

---

Justification: All datasets and models used in our experiments are properly credited with citations to their original sources.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the pack- age should be provided. For popular datasets, eoperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the cense of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA].
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but the main contri- bution of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, cura- tion, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
21

---

Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: In this work, LLMs were employed solely for improving language clarity.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy /tteer//hhecrrpscc//ooorrreeoeee for what should or should not be described.
22

---

A Supplementary Description of Experimental Setup
A.1 Datasets
Table G presents the detailed statistics of the datasets used in this work. We also provide further
analysis of the selected datasets across the four major tasks.
Table 6: Summary of the multi-domain datasets used in our experiments.
Dataset
#Train
#Val
#Test
Metric
NSum (News Summ.)
20,000
2,000
2,000
ROUGE-L
Sent (Sentiment)
10,000
1,000
1,000
ACC
Q&A (Question Ans.)
15,000
1,500
1,500
EM/F1
Topic (Classification)
12,000
1,200
1,200
ACC
Discnesancy
Discrepancy
Sivllrity Discrepancy
64
6/5
6:0
6/6
6/6
6.0
6/1
6.99
644
6:3
6/5
615
3M
633
63
Oiteet
Dakast
Datasa
Figure 3: Domain discrepancy heatmaps across three dimensions: Token Distribution, Vocabulary Overlap, and Semantic Similarity.
As shown in Figure EL we illustrate the domain discrepancies across three distinct dimensions. In the Token Distribution dimension, NSum and Q&A exhibit relatively low discrepancy (0.15), indicat- ing similar token usage patterns, whereas NSum and Topic display larger discrepancy (0.55), In the Vocabulary Overlap dimension, NSum and Q&A share more vocabulary (discrepancy of 0.20), while NSum and Topic have significantly lower vocabulary overlap (discrepancy of 0.65). Regarding Semantic Similarity dimension, NSum and Q& show the highest semantic closeness (discrep- ancy of 0.10), whereas NSum and Topic present comparatively larger semantic gap (discrepancy of 0.40).
A.2 Baselines
To give fair and transparent point of comparison, we implement or re-run every base- line under exactly one of two backbone-updar update protocols:
Baselines.
1) Full FT - all LLM parameters are updated 1000 trainable). This is the strongest-but most memory-hungryy-sttingg
2) PEFT- the LLM backbone is frozen; only leight adapters (LoRA, Houlsby adapters, etc.) or newly added heads are trained 1% of parameters).
Our PMS-FTP always uses the PEFT protocol: the backbone drift per stage is bounded by Po. and only domain adapters Φ5 are newly trained.
The competing methods grouped follows:
*
Base Methods
FULL (Full FT): classical end-to-end fine-tuning.
Frozen-bar backbone PEFT has become standard practice in recent parameter-efficient lne-ttnning work and, in preliminary tests, performs on par with-or hetter than-full fine-tuning when data are limited.
23

---

FIXED (PEFT): backbone frozen, task head only.
Domain-Adappation (Full FT)
MDAN Peretall 201: multi- cullii-ddversariia domain classifiers.
MDD Peng et all 2019:: moment matching across domains.
-GMDI Ling all 2024|: Bayesian Gaussian Gaussian-mixture mixture domain indexing.
Implementation note: all DA methods update the entire LLM just as FULL, and their extra losses are added on top of the cross-entropy objective.
Single-Domain PEFT
LoRA HHuet all 2021] low-rank adapters in attention projections.
Adapter Houlsby et al. 2019: Houlsby bottleneck adapters.
-...MA-Adapter [Zhang al, 2024N]: zero-init residual adapters.
Q-LoRR [Detimers.em et all [023|:
4-bit quantised LoRA layers.
-Tag-LLM eh
2024 task-aware gating with soft prompts.
These methods freeze the backbone; only adapter prompt parameters are updated.
Data-Sclection PEFT
INSTRUCTMINING (IT) [Cao all 20241]: filters high-quality instructions before
LoRA fine-tuning.
S2L [Yang all [0246]]: curriculum ordering via proxy- clustering; uses LoRA layers.
A.3 Implementation Details
Hardware and Software. We conducted all experiments on an internal cluster with NVIDIA A100 GPUs GB memory per GPU) using Python 3.9, PyTorch 2.0.0, and HuggingFace Trans- formers 4.30.2. Each experiment was 1-2 GPUs under our parameter-efficient settings.
single node with 8 GPUs, though most tasks fit
Data Splits and Preprocessing. Each dataset is partitioned into train/validattonntess splits, as noted in Table We tokenize with the default HuggingFace tokenizer for each respective LLM (LLaMA2 or Falcon). For summarization (NSum), we truncate inputs at 512 tokens; for Q&A, we set maximum context length of 384 tokens plus question tokens. Other tasks are capped 256 tokens per sample, All special tokens remain as defined in each LLM's tokenizer.
Training Configuration. We use Adam with a linear decay scheduler, a warmup ratio of 10% of total steps, and gradient clipping at norm 1.0. Table gives key hyperparameters. We generally train for 3-5 epochs (depending on dataset size), selecting the best checkpoint via validation loss. Unless otherwise noted, we set the batch size 32 per GPU for all experiments, and accumulate gradients across fewer GPUs for smaller tasks if needed. We adopt the default mixed-precision (fp16) training Py Torch.
Table 7: Default hyperparameter values.
Hyperparameter
Value
Optimizer
Adam W
Learning rate (LLaMA2-7B)
3 10 5
Learning rate (LLaMA2-13B)
1
10
Learning rate (Falcon-40B)
5 10
Batch size (per GPU)
32
Max epochs
5
Warmup ratio
0.1
Gradient clipping
1.0
Precision
FP16

---

Partition-Based Multi-Stage Fine- Tuning We employ two consecutive stages (M=2) by de- fault: stage adapts the cluster with higher internal synergy, stage 2 covers the remainder. Both the domain discrepancy d(D;, Dj) and the synergy score Syn(D; Dj) are computed off-line from raw text:
Let P; and Pj be the empirical token distributions (unigram + bigram) of domains D; and D,. We define
dis(D,, D2) - Ku(P.I)) + M)), = = P3)+
(16)
where KL is the Kellback-Leibler divergence. We normalise dss E [0, 1] by dividing by log 2. For synergy we linearly blend lexical and semantic overlap:
Syn(D,, D2) (Ja V3) (os(11)))))
(17)
where Vú is the vocabulary of D.. Jaco Vj) IV; V/llIV Vjl. and μ4 is the mean Mattence-BERT embedding of domain i. Both terms are in [0, 1]: the average is therefore in [0, 1].
Table 8 contrasts four partition criteria on the 4-domain slice (SQuAD, HotpotQA, CNN/DM, XSum). Replacing our full metric with a single component (JS only or Embedding only) lowers performance, and random splitting is worst.
Table 8: Impact of different partition metrics (LLaMA2-7B).
Metric for
Q&A (FI)
NSum (ROUGE-L)
Random split
68.1
39.0
divergence only
69.3
40.0
Embedding cosine only
69.6
40.3
JS + Vocab/Embed (ours)
70.5
40.9
The joint metric gives further +1.2 FI +0.9 ROUGE-L over its best single-signal variant, con- firming that both lexical statistics and semantic proximity are needed to capture cross-domain rela- tionships effectively.
During each stage, we impose norm constraints | f' S po and | S Pe (Assumption B.2)). In practice, we simply project any update exceeding these norms after each gradient step. By default, we set (pe, P+) = (0.1, 0.1) unless specified otherwise.
B
Additional experimental results
B.1 Effect of Stage Ordering in Multi-Stage Fine-Tuning
After domains are optimally clustered into two stages by our G-objective, we can still choose which stage to run first. To verify that this ordering is an implementation detail, we tried three sequences on the same 4-domain slice (SQuAD [Kajpurkar, 2016], HotpotQA [Yang et all 2018], CNN/DM XSum [Narayan et all 2018)) using LLaMA2-78 7B: 1) High-Lov the default: high-synergy Q&A first, summarisation second: Low High reverse order; 3) Interleaved fine-tune one epoch on stage 1, then one epoch on stage 2, repeating until convergence.
Table Influence of stage ordering (M=2). Metrics: F1 (Q&A) ROUGE-L (NSum).
Ordering
Q&A (FI)
NSum (ROUGE-L)
High (default)
70.5
40.9
Log- Interleaved
70.4
40.8
70.3
40.7
All three runs land within 0.2 points of one another (Table D. well inside normal tuning noise, indicating that stage ordering has negligible impact. This robustness stems from the fact that each stage updates only its adapter blocks; subsequent stages cannot overwrite earlier domain-spechi parameters, knowledge learned in any order is preserved.
hhnps//github.ccm/eeepmnnnrr
25

---

B.2 Why conventional DA baselines lag behind FULL.
To investigate why conventional domain adaptation (DA) baselines (MDAN PP all 2018 S [[eng et al, 2019], GMDI [Ling et all (2024)) consistently underperform relative to FULL fine-tuning, we performed additional diagnostic analyses. Specifically, we measured (i) cross- domain gradient conflicts (via cosine similarity), (ii) parameter update magnitudes per domain, and (iii) the extent of catastrophic forgetring of pretrained knowledge, using the LLaMA2-13B model on NSum and Q&& domains. Table ITO summarizes the results of these additional experiments:
Table 10: Diagnostic analyses comparing conventional DA methods against FULL and PMS-FTP.
Method
Gradient Conflict (Cosine Similarityy
Avg. Update Norm
Perplexity Increase (%)
FULL
2.15
+5.6
MDAN
0.12
1.48
+11.5
M'SDA
0.15
1.32
+9.7
GMDI
0.18
1,27
+8.3
PMS-FTP (Ours)
0.57
1.86
+3.2
Our analyses reveal the following insights:
1) Gradient conflicts. DA methods exhibit substantially lower gradient alignment compared to FULL and our PMS-FTP, indicating significant gradient interference between domains. This conflict leads to suboptimal convergence, as competing updates negatively affect overall generalization.
2) Parameter update magnitudes. DA methods apply smaller updates due to regularization con- straints (adversarial or moment-matching objectives), limiting adaptation capacity for complex domain-specific tasks. In contrast, our PMS- FTP method achieves balanced updates via strategic domain partitioning and parameter-effiiien adapters.
3) Catastrophic forgetting. Conventional DA methods significantly increase perplexity relative FULL,, indicating stronger forgetting of pretrained representations. Our PMS-FTP maintains the lowest increase, demonstrating better preservation of pretrained knowledge due to controlled adaptation.
In summary, DA methods lag behind FULL due to severe gradient conflicts, overly conservative parameter updates caused by adversarial/matchingg regularization, and more pronounced forgetting of pretrained knowledge. Our PMS-F7 framework effectively addresses these challenges through synergy-awer awware partitioning, balanced updates, and controlled adaptation, resulting superior multi- domain performance.
B.3 Affinity metric alternatives
We replace the default affinity used in the partition objective G with several variants on the same 4-domain slice (LLaMA2-7B). Table I shows that our lightweight JS+vocab/embedding signal consistently outperforms single-source metrics or random split. The G-guided partition benefits from combining divergence (distribution gap) and lexical/semantic overlap (potential transfer). Us- ing either component alone underestinates complementary effects, yielding weaker partitions and lower task scores.
Table 11: Alternative affinity metries for G-guided partitioning (LLaMA2-7B, 4-domain slice).
Metric for
Q&A (F1)
NSum (ROUGE-L)
Random split
68.1
39.0
divergence only
69.3
40.0
Embedding cosine only
69.6
40.3
JS + Vocab/Embed (ours)
70.5
40.9
B.4 Robustness to gradient-based variants and stochastic perturbations
We (i) sweep À to stress-test synergy weighting, (ii) add a gradient-similarity component (cosine of per-domain gradients), and (iii inject Gaussian noise into the heuristic affinities. Table M shows all
26

---

variants remain within I 0.3 points of the default in I.D. he partition is flat around the optimum: muddient-augeenter scores add computational cost but negligible gains; moderate À values preserve the best trade-off between synergy and discrepancy.
Table 12: Partition robustness (LLaMA2-7B, 4-domain slice).
Variant
Q&A (F1)
NSum (ROUGE-I.)
À = 0 (no synergy)
70.3
40.8
À = 1.0 (synergy-only)
70.2
40.6
Gradient-min (0.7 heuristic 0.3 Vcos)
70.4
40.8
Heuristic Gaussian noise (a 0.05)
70.2
40.7
Default (Table IID)
70.5
40,9
B.5 Scalability to many domains
We synthetically vary the number of domains k and measure CPU partition overheads and peak GPU memory with 4-bit LoRA. Table I3 indicates sub-second CPU time and practical GPU usage up to k=50. Partitioning is CPU-side and negligible relative PEFT training; memory remains dominated by standard SFT/PEFT, confirming practicality at double-digit k.
Table 13: Large-k partition costs and peak GPU memory (synthetic up to k=50; A A100).
k
Affinity build (time RAM)
Clustering time
Peak GPU (4-bit LoRA)
20
0.47 s/ 180 MB
0.14
19 GB
35
2.10 s 620 MB
0.52
23 GB
50
3.20 950
0.90
26 GB
B.6
Scaling twelve domains (real mixture)
We combine Wiki-10 (topic classification) and Multi-News (summarization), deduplicated to 12 do- mains, and keep the same Syper-parameete as the main study. Table [4 shows gains over all-in-one SFT and over random 2-stage split while keeping memory low. The synergy-aware split general- izes beyond four domains to heterogeneous, double-digit regime with consistent improvements.
Table 14: Twelve-domain mixture (LLaMA2-7B LoRA).
Split strategy
Avg. ACC (Wiki-10)
ROUGE-L (Multi-News)
Peak GPU (GB)
Partition time (s)
All-in-one SFT Random 2-stage
83.1
37.2
27.3
n/a
83.7
37.5
18.6
PMS-FTP (ours)
84.0
38.1
18.7
0.7
B.7
Inference footprint after LoRA merging
After each stage we merge the finished LoRA into the frozen backbone, so only one 4-bit adapter is carried at inference. Table [T shows equal-or-lower memory than single-adapter baseline. Together with the accuracy gains in Table merging achieves a strictly better eecuracy-memong trade-off than training/keeping multiple adapters.
B.8 Empirical validity of the objective
We sample 20 random partitions, compute G. and measure worst-domain dev loss. Table IG reports Pearson -0.81 (p < 0.01), i.e., higher G predicts lower worst-domain error. This supports the practical usefulness of our bound-driven objective: G values correlate strongly with the metric we aim to improve.
27

---

Table 15: Measured inference memory (LLaMA2-7B, k=4, A100).
Precision
Tag-LLM (1 LoRA)
PMS-FTP (merged)
A
FP16
29.4
GB
-2.4%
INT8
19.1 GB
18.6 GB
-2.6%
4-bit
17.2 GB
16.8 GB
00.4 GB
Table 16: Correlation between G and worst-domain dev loss (LLaMA2-7B, 20 random partitions).
Statistic
Worst-Dev Loss
Mcan
0.432
1.72
Std
0.057
0.19
Min
0.318
1.38
Max
0.522
2.11
Pearson p -0.81 (p 0.01, n²² 00.65)
B.9 Reasoning benchmarks and reweighting baselines
We extend evaluation HellaSwag, MMLU-STEM, ARC-easy, SciQ, and GSM8K, and add reweighting-based MTL baselines (iMTL, FAMO, ExcessMTL) under identical PEFT budgets. Ta- bles I-H18 show PMS-FTP achieves the highest average per backbone. Synergy-aware partitioning is not limited to [NSum, Sent, Q&A, Topic): it transfers to reasoning-heavy suites and remains competitive against strong MTL optimizers.
Table 17: Reasoning tasks with 7B backbone (identical PEFT budgets).
Method
Hellaswag (Acc)
MMLU- STEM (Acc)
ARC-casy (Acc)
SciQ (Acc)
GSM8K (Pass@1 1)
Avg.
FULL
73.4
39.7
78.5
92.6
60.5
LoRA
73.1
39.3
77.9
92.4
17.5
60.0
Tag-LLM
74.8
41.1
79.6
93.3
19.3
61.6
iMTL
74.6
40.6
79.3
93.0
18.6
61.2
FAMO
73.7
41.3
78.6
92.6
18.7
60.8
ExcessMTL
74.2
40.5
79.7
92.1
17.5
60.8
PMS-FTP (ours)
75.6
42.1
80.6
93.8
19.9
62.4
B.10
Incremental addition of a new domain
After training on the original four domains, add Legal-QA as new stage and freeze prior adapters. Table shows negligible forgetting on domains and a gain over single-domain LoRA on the new domain. Disjoint, frozen adapters make PMS-FTP naturally amenable to one-shot do- main extension without replay.
B.11 Stability of domain centroids
(i) bootstrap mean SBERT embeddings to gauge centroid noise, (ii) replace each single centroid by K-3 weighted barycenter. 'Tbble 20 shows bootstrap deviations are < 3% of the smallest inter-domain distance; Table 21 shows K=3 centroids change final metrics by S 0.1 pp. single mean embedding is sufficiently stable domain signature for G-guided clustering.
28

---

Table 18: Reasoning tasks with 13B backbone (identical PEFT budgets).
Method
Hellaswag (Acc)
MMLU- STEM (Acc)
ARC-easy (Acc)
SeiQ (Ace)
GSM8K (Pass@) 1)
Avg.
FULL
77.5
46.3
83.7
94.8
24.5
65.4
LoRA
77.1
45.9
83.2
94.6
64.9
Tag-LLM
79.0
47.3
84.7
95.4
25.1
66.3
iMTL
77.4
47.2
83.3
95.1
24.8
65.6
FAMO
77.7
47.6
84.5
95.2
24.2
65.8
ExcessMTL
78.1
47.0
83.1
95.0
23.8
65.4
PMS-FTP (ours)
79.9
49.0
85.7
95.8
26.8
67.4
Table 19: One-shot incremental stage (add Legal-QA).
Model
Avg. score on original domains
Legal-QA (EM)
Before add-on
65.5
After add-on (PMS-FIT FTP)
65.4
68.2
Single-domain LoRA
n/a
67.6
C
Proofs
C.1
Complete Proof for Theorem
Before formally beginning the proof, we first revisit the setting and present key lemma:
There are k source domains (D,} each domain D with ny samples, total Σ}= nz. We let a; 21 or any other nonnegative weighting such that E Oj 1. Our LLM-based predictor fo,(o,) is constrained so that
|10 - 00 S po. 116l|12 I Pe:
(18)
for each j. By Assumption R.I. the model output is L-Lipschitz w.r.t. predictions, and the difference in outputs for different parameters is bounded by B(10 0'll +,, 11 dlll). Thus the entire class of such (8, {o;}) belongs low-capacity function family F.
Lemma C.1. Let F be the class of predictors with backbone/adapters bounded above. Then for any (0, 1), with probability at least å over all n samples,
I (p + Easpol
l2(2
VfEF
(19)
where -,,
Proof. Because the loss is L-Lipschitz and the model satisfies the output-difference bound from As- sumption [.Il each e F is LB(p= E3 P+) Pa)-sicschitz in its parameters relative l2. Let R,, (F) be the empirical Rademacher complexity over the pooled sample of size n. Standard contraction (e.g. Mohri et all (01)) yields
1
(F) < LB (pa kpp) Vn
(20)
Replacing kpp by E, aiPp (because losses are weighted by a;)) strengthens the constant. Applying the usual Rademacher tail bound with a union-bolt moond over 6/2 produces the stated incquality. A
Proof. Here is the proof of Theorem
29

---

Table 20: Bootstrap deviation of domain mean embeddings and cross-domain distances.
Domain
N
95% CT of |
Cross-domain min dist
NSum
12,000
[0.038, 0.065]
1.92
Q&A
15,000
[0.031, 0.060]
2.04
Sent
10,500
[0.042, 0.072]
1.87
Topic
11,200
[0.040, 0.068]
1.99
Table 21: Single centroid vs. K=3 weighted barycenters per domain.
Representation
ROUGE-L (NSum)
EM (Q&A)
Sent (ACC)
Topic (ACC)
! vs. 1-centroid
1 centroid (paper)
43,4
67.2
90.2
88.0
3 centroids (K=3)
43.3
67.1
90.1
87.9
-0.1
Uniform convergence (empirical - true risk)
Define [po, Po. az, k) = 2L2 + +E, aspo).
Lemma CT gives
/In(2/81 2n
( ()) + T(pe, Po. as, k)
(21)
Domain dinerp-d correction Blitzeret et al. [Bitzer 2008
show that for any hypothesis &i and Summing over all pairs and using
any two distributions P, 2, (hr(h) - Lo(h)) < dua (P, Q triangle inequality,
k
()) & aofto ( + Σ dD,
(22)
t,=11
where Dj denotes drawing as if every example came from a single mixture domain-henee its empir- ical risk is exactly (f). and the additive discrepancy penalty is weighted by 3 1 (absorbing the Lipschitz loss factor into the definition of d). Restoring the constant gives the B appearing in the theorem.
Combine bounds
Insert (22 into (22)
C-(S s (J) +
+
B k
Σ D,) + V
In(2/8) 2n
fJ=1
Replacing VIn /ln(2/8)/ (2n) by big-O( (gg-O(/n(118)//)) notation and recalling ())
Σ, agLr, (f) yields exactly inequality (II), proving Theorem 3.
C.2 Complete Proof for Theorem 3.
Here is the proof of Theorem
Proof. For any fixed stage t training domains S,, Theorem 3 with weights a} Eies gives
ny
Σ Lo (o,()) < Σa Ro)))) 2LB(po + po) + 8 Σ d(D., D,)+ Av JES.
In(1/8)
Eyes, ny
JES,
i.ESs. K3
(23)
Then we inject synergy and explicit capacity weight. Define Cap(St) = 1 -
Ms Ejes, || Because ||00||| S pa and [|o] S Ps. upper bound 2LB + p6) by Cap(SS)
30

---

after tuning Ma. Hs. Subtract and add > s(D,, Dj) to (255) to obtain
Σ
Lo S 1 Σ
(d As) Cap(S.) al (n(
(24)
JES.
i<JES,
Let Rmax (axxx(SS))) (Si,..,, SM) = maxt Ejes, aj LD, (f*). Taking the maximum of (24) over gives
(Smax(11.)) ........ SM) 1 1 G(S ................................ SMs) + Oly In(1/6)
(25)
because the bracketed term is exactly the t-th summand of G in (12). Define B(u) = [1 u|+ Then SMM) v (( SM)) + O (L/((L/)))
Because B strictly decreasing on (ooc, 1], maximising & minimises the bound. Hence the par- tition .S...... Sh l the maximiser of Gerealsses the smallest upper-bound, yielding (I3)) Any other split attains a weaker bound, completing the proof.
31


Reviews:
[{"review_id": "8GSoOnEjqy", "reviewer": "Reviewer_XGk5", "summary": "The work presents \\textbf{PMS-FTP}, a partition-based, parameter-efficient scheme for fine-tuning a single LLM on multiple domains.  Domains are clustered into sequential stages by maximising an objective $G$ that trades off synergy, discrepancy, and adapter cost—supported by a worst-stage risk bound—and each stage fine-tunes lightweight LoRA adapters while largely freezing the backbone.  Across four tasks and three model sizes, PMS-FTP beats eleven competitive baselines by +0.8–2.0 points and cuts GPU memory by roughly 32\\,\\%, with ablations confirming the value of partitioning, stage count, and norm constraints.", "questions": "please refer to the weaknesses", "limitations": "please refer to the weaknesses", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n(1) Conceptual Novelty \\& Theory. The paper frames multi-domain fine-tuning as minimising a partition objective $G$ that balances \\emph{synergy}, \\emph{discrepancy}, and adapter cost, then proves new generalisation bounds showing this choice tightens worst-domain risk.  Coupling these guarantees with an $O(k^{2}\\log k)$ agglomerative clustering yields a rare, theory-backed PEFT recipe.\n\n(2) Consistent Empirical Gains. Across four tasks (news‐sum, sentiment, QA, topic) and three backbones (LLaMA-2 7B/13B, Falcon-40B), PMS-FTP beats eleven strong baselines—including full FT and the latest PEFT methods—by +0.8–2.0 points on standard metrics, demonstrating robustness to both task diversity and model scale.\n\n(3) Resource Efficiency. Stage-wise norm bounds cut GPU memory by $\\sim$32 \\% versus full fine-tuning (e.g., 18.4 GB vs.\\ 27.2 GB on an A100) while still improving accuracy; the only extra cost—computing synergy/discrepancy matrices—remains trivial for $k\\!\\le\\!10$ domains.\n\n\nWeaknesses:\n\n(1) Heuristic similarity metrics. The partition relies on fixed vocab-overlap, sentence-embedding cosine, and JS divergence, which often mis-estimate real transferability; domains can look similar yet teach different skills (or vice-versa), sending data into sub-optimal clusters. A task-aware or learned affinity signal—e.g., gradient similarity—would make the partition less brittle.\n\n(2) Scalability limits. Computing the full $k\\times k$ affinity matrix and running agglomerative clustering costs $O(k^{2}\\log k)$ time and $O(k^{2})$ memory, while adapter banks grow linearly with $k$. This becomes infeasible when the number of domains reaches dozens; approximate kernels or router-style sharing are needed.\n\n(3) Modest memory edge. While cheaper than full fine-tuning, two adapter banks still use more GPU memory than leaner single-adapter PEFT variants, which matters on edge devices; pruning, quantization, or cross-stage sharing could tighten this gap.\n\n(4) Theory untested empirically. The paper derives a neat worst-stage risk bound, but never shows how the objective $G$ (or its components) correlates with held-out error.  Without a plot or numbers, readers cannot tell whether the bound is predictive or merely a sanity check.  Reporting the correlation between $G$ and downstream accuracy across random partitions would clarify its practical value.", "was_revised": true, "final_justification": "Thanks for the detailed responses from the authors. I think most of my concerns are well addressed. So I raised my score. Hope the authors could include the new content during rebuttal into the final version.", "rebuttal": "Dear Reviewer `XGk5`, thank you for your recognition of our work. Below, we will clarify and discuss some of your concerns.\n\n---\n\n> **Q1:  Discussion on task-aware or learned affinity signal**\n\nOur generalisation bound only requires **any bounded affinity measure**—it never assumes the metric is perfect. Hence the guarantee holds for both our current JS + vocab + embedding score and any learned/gradient metric. \n\nWe have added three targeted studies on the 4-domain slice (LLaMA-2 7B):\n\n| Variant | Q&A F1 | NSum R-L | Δ vs. default |\n|---------|-------|----------|---------------|\n| λ = 0 (no synergy) | 70.3 | 40.8 | −0.2 / −0.1 |\n| λ = 1.0 (synergy-only) | 70.2 | 40.6 | −0.3 / −0.3 |\n| Gradient-mix (0.7 heuristic + 0.3 ∇cos) | 70.4 | 40.8 | ≤−0.1 |\n| Heuristic + Gaussian noise (σ = 0.05) | 70.2 | 40.7 | ≤−0.3 |\n\nThe **largest drop is 0.3 pt**, confirming that (i) performance is flat across λ, and (ii) task-aware gradients or noise have negligible effect. Gradient similarity needs an extra forward+backward sweep per domain batch (≈ 1.7× training cost). Given the minimal gain (<0.1 pt) we favour the lightweight text-statistic metric.\n\n*Take-away——*  Based on experimental evidence, our partition metric is **fast, theory-compatible, and empirically robust**.\n\n---\n\n> **Q2： The O(k²) affinity matrix and per-domain adapters do not scale to many domains.**\n\nIndustry benchmarks (WILDS, Multi-OOD, Cross-Domain-NER, etc.) rarely exceed **k = 10**.  Within this range PMS-FTP’s partitioning costs are negligible (< 0.1 s, < 50 MB on CPU).\n\nTo measured worst-case, we synthetically generated up to **k = 50** domains (2 k samples each):\n\n| k | Affinity build | Clustering | Peak GPU (4-bit LoRA, A100) |\n|---|----------------|-----------|-----------------------------|\n| 20 | 0.47 s / 180 MB | 0.14 s | 19 GB |\n| 35 | 2.1 s / 620 MB  | 0.52 s | 23 GB |\n| 50 | 3.2 s / 950 MB  | 0.90 s | 26 GB |\n\nThe pipeline remains CPU-side; fine-tuning time and GPU usage are still dominated by normal SFT. Adapters scale linearly, but **stage-wise LoRA merging** frees earlier stages after convergence; with merging plus 4-bit weights the final footprint stays **< 27 GB** for k = 50 (same as full FT of a 7B model).  Edge deployment can optionally keep a *single* merged LoRA.\n\nIf k≫50, we may (i) subsample affinity via LSH-nearest-neighbour, or (ii) route low-traffic domains to a shared adapter bank.  Both drop-in replacements preserve our bound structure.\n\n*Take-away——*  PMS-FTP handles up to dozens of domains with trivial overhead, and simple merging keeps memory on par with existing PEFT baselines even in large-k regimes.\n\n---\n\n> **Q3 : Modest memory edge**\n\n\nPMS-FTP stacks adapters **only during training**.  After each stage we merge the finished LoRA into the frozen backbone (`torch.lora_merge()`); the next stage then trains one fresh LoRA.  At inference time **only the last 4-bit LoRA remains**, matching the footprint of any one-adapter baseline.\n\nFollowing your suggestion, we conducted quantitative tests and indeed achieved good results.\n\n*Measured footprint (LLaMA-2 7B, k = 4, A100):*\n| Precision | Tag-LLM (1 LoRA) | PMS-FTP **merged** | Δ |\n|-----------|-----------------|--------------------|---|\n| FP16      | 29.4 GB | 28.7 GB | −2.4 % |\n| INT8      | 19.1 GB | 18.6 GB | −2.6 % |\n| 4-bit     | 17.2 GB | **16.8 GB** | −0.4 GB |\n\nAccuracy with the merged model stays **+0.6 pt** over Tag-LLM on Q&A F1. Merging is loss-free for LoRA (rank ≪ d_model) and releases the first-stage weights, so edge devices carry exactly one adapter—no heavier than existing PEFT.\n\n*Take-away——*  PMS-FTP yields equal-or-lower inference memory than the lightest PEFT while retaining higher accuracy.\n\n---\n\n> **Q4:  Theory untested empirically**\n\nTo address your concern, we generated **20 random partitions** of the 4-domain slice (LLaMA-2 7B) and computed  \n(i) the objective value 𝓖 and (ii) the *worst-domain* dev loss.\n\n| Statistic | 𝓖 | Worst-Dev Loss |\n|-----------|----|---------------|\n| Mean      | 0.432 | 1.72 |\n| Std       | 0.057 | 0.19 |\n| Min       | 0.318 | 1.38 |\n| Max       | 0.522 | 2.11 |\n| **Pearson ρ** | **–0.81** | — |\n\n*ρ = –0.81 (p < 0.01) ⇒ R² ≈ 0.65* — It means that higher 𝓖 (tighter bound) strongly predicts lower worst-domain error.\n\n**Take-away:** The theoretical bound has substantial explanatory value in practice.\n\n---\n\nThank you again for your valuable and insightful comments!"}, {"review_id": "DpEC1XLMIJ", "reviewer": "Reviewer_j5Fq", "summary": "This paper tackles the critical problem of supervised finetuning (SFT) of LLM across multiple heterogeneous domains, specifically addressing the issue of inter-domain conflicts. The authors propose a novel multi-stage fine-tuning framework that strategically partitions domains into subsets (stages). This partitioning mechanism is theoretically motivated to minimize domain discrepancy while maximizing domain synergy within each group of tasks. The theoretical analysis demonstrates that this partition mechanism enjoys a controllable generalization bound. Extensive empirical evaluations across 4 real-world fine-tuning tasks show the effectiveness of the proposed method, which consistently outperforms baseline approaches across all SFT tasks.", "questions": "### Questions from weakness\n1. **The algorithm, implementation and complexity of the partition mechanism are not clear.** According to Equ. 12,  how could the $\\Delta \\theta^t$ and $\\phi_j^t$  be estimated without retraining the model? do you apply the manually parameter shift bounds $\\rho_{\\theta}, \\rho_{\\phi}$? could you clarify more on how this can be implemented within the given complexity?\n2. **The empirical evidence on the current 4 tasks are limited.** Could you give the results on the reasoning tasks, e.g. Hellaswag, MMLU, ARC, SciQ, GSM8K, etc.?\n3. **Lack of comparison to some conventional multi-task learning approaches**, e.g. iMTL [1], FAMO [2], ExcessMTL [3].  Could you give the comparison results to these baselines?\n---\n4. Can you explain more on the \"Single-Domain LLM Fine-tuning\" baseline? Do you only tune the adaptor or both the backbone and the adaptor? A clarification on the number of tuneable parameters on each experiments could help.\n5. Can you explain the training loss curves on the QA task in Fig. 1? According to the given algorithm, the proposed PMS-FTP method adopts stage-wise training, the training loss on the QA task should only be available on one of the training stage, instead of the whole training process?", "limitations": "1. The scope of the tasks are limited to easier question-answering (e.g. SQuAD) and summarization tasks. More challenging reasoning tasks, e.g. Hellaswag, MMLU, ARC, SciQ, GSM8K, etc. have not been explored.\n2. Lack of comparison to some conventional multi-task learning approaches.\n3. The experiments are limited to small amount of tasks (\\#tasks=4), while the scalability and effectiveness to larger amount of tasks have not been explored.\n4. The effectiveness of stage-wise training using the partition mechanism is validated, while the impact of the order of training stages, $\\{S_1, ..., S_M\\}$ has not been explored or discussed.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths: \n1. The paper is well-motivated, tackling the significant challenge of domain conflict and negative transfer in multi-domain LLM fine-tuning, a prevalent issue in real-world applications.\n2. A complete generalization bound analysis is derived for the multi-domain fine-tuning problem. This theoretical insight directly motivates and justifies the proposed partition mechanism, providing a robust scientific basis for the approach based on domain discrepancy and synergy.\n3. the empirical results on 4 various real-world finetuning tasks demonstrate the effectiveness of the proposed method, which consistently outperforms baseline approaches across all the tasks.\n4. The comprehensive ablation studies demonstrate the robustness of the proposed method on hyperparameter choices.\n\n### Weaknesses:\n1. **The algorithm, implementation and complexity of the partition mechanism require more explanation.** According to Equ. 12,  it is not clear how to estimate $\\Delta \\theta^t$ and $\\phi_j^t$ without retraining the model? or are they directly be replaced by the parameter shift bounds $\\rho_{\\theta}, \\rho_{\\phi}$? \n2. **The empirical evidence on the current 4 tasks are limited to relatively easy tasks.** Could you give the results on the reasoning tasks, e.g. Hellaswag, MMLU, ARC, SciQ, GSM8K? \n3. **Lack of comparison to some conventional, reweighting-based multi-task learning approaches**, e.g. iMTL [1], FAMO [2], ExcessMTL [3].  \n\n[1] Towards Impartial Multi-task Learning. Liu, et. al, 2021.\n[2] FAMO: Fast Adaptive Multitask Optimization. Liu, et. al, 2023.\n[3] Robust Multi-Task Learning with Excess Risks. He, et. al, 2024.", "was_revised": true, "final_justification": "This paper tackles the critical problem of supervised finetuning (SFT) of LLM across multiple heterogeneous domains, specifically addressing the issue of inter-domain conflicts. \n\nDuring rebuttal period, the authors provided additional comparison between multi-task learning baselines and evaluation on more challenging reasoning tasks, e.g. MMLU and GSM8k, which addressed most of my concerns. So I raised the score to 4. We hope the authors could present the results on more comprehensive task sets with various levels of domain conflicts.", "rebuttal": "Dear Reviewer `j5Fq`, thank you for your valuable comments, and some of the questions are very insightful. We will clarify these issues to address your concerns.\n\n---\n\n> **Q1: Partition algorithm & complexity**\n\n*How Eq. 12 is used without re-training*  \n- At **partition time** the real updates $\\(\\Delta\\theta^{t}\\) / \\(\\phi^{t}_{j}\\)$ are **unknown and unnecessary**.  \n- We substitute their **budget ceilings** $\\(\\rho_\\theta, \\rho_\\phi\\)$ (set once from single-domain PEFT practice: 0.05–0.20 of ℓ₂-norm, matching LoRA/Adapter magnitudes).  \n- Hence the capacity term collapses to the constant  $\\(\\mu_\\theta\\rho_\\theta^{2}+\\mu_\\phi\\rho_\\phi^{2}\\)$; clustering depends *only* on pre-computed domain distances (JS divergence + vocab/embedding similarity).\n\n*Algorithm & measured cost*  \n| k domains | wall-clock (CPU) | peak GPU mem | notes |\n|-----------|-----------------|--------------|-------|\n| 10        | **0.18 s**      | **0 GB**     | single-link agglom., distance matrix cached |\n\nTime scales as $\\(O(k^{2}\\log k)\\)$ and memory as $\\(O(k^{2})\\)$; the subsequent fine-tuning is a **single standard PEFT run**.\n\n*Take-away——*  The partition step is **offline, sub-second, GPU-free**, needs no parameter re-estimation, and the heuristic attains near-optimal partition quality with negligible overhead.\n\n---\n\n> **Q2：Evaluation on harder reasoning benchmarks & reweighting-based multi-task learning baselines**\n\nWe fine-tuned **LLaMA2-7B** and **LLaMA2-13B** under the *same PEFT budgets* (ρ_θ = 0.1, ρ_φ = 0.1) on five reasoning datasets.  We also compare ours with **three reweighting-based multi-task learning baselines**(iMTL, FAMO, ExcessMTL ).\n\nSettings: training split for SFT, dev set for early stopping; identical 8-GPU A100 setup.\n\n| Backbone | Method | Hellaswag(Acc) | MMLU-STEM(Acc) | ARC-easy(Acc) | SciQ(Acc) | GSM8K(Pass@1) | Avg. |\n|----------|--------|--------------------|--------------------|-------------------|---------------|-------------------|------|\n| 7B | FULL        | 73.4 | 39.7 | 78.5 | 92.6 | 18.1 | 60.5 |\n| 7B | LoRA        | 73.1 | 39.3 | 77.9 | 92.4 | 17.5 | 60.0 |\n| 7B | Tag-LLM | *74.8* | 41.1 | 79.6 | *93.3* | *19.3* | *61.6* |\n| 7B | iMTL        | 74.6 | 40.6 | 79.3 | 93.0 | 18.6 | 61.2 |\n| 7B | FAMO        | 73.7 | *41.3* | 78.6 | 92.6 | 18.7 | 60.8 |\n| 7B | ExcessMTL   | 74.2 | 40.5 | **79.7** | 92.1 | 17.5 | 60.8 |\n| **7B** | **PMS-FTP (ours)** | **75.6** | **42.1** | **80.6** | **93.8** | **19.9** | **62.4** |\n\n| Backbone | Method | Hellaswag(Acc) | MMLU-STEM(Acc) | ARC-easy(Acc) | SciQ(Acc) | GSM8K(Pass@1) | Avg. |\n|----------|--------|--------------------|--------------------|-------------------|---------------|-------------------|------|\n| 13B | FULL        | 77.5 | 46.3 | 83.7 | 94.8 | 24.5 | 65.4 |\n| 13B | LoRA        | 77.1 | 45.9 | 83.2 | 94.6 | 23.8 | 64.9 |\n| 13B | Tag-LLM | *79.0* | 47.3 | *84.7* | *95.4* | *25.1* | *66.3* |\n| 13B | iMTL        | 77.4 | 47.2 | 83.3 | 95.1 | 24.8 | 65.6 |\n| 13B | FAMO        | 77.7 | *47.6* | 84.5 | 95.2 | 24.2 | 65.8 |\n| 13B | ExcessMTL   | 78.1 | 47.0 | 83.1 | 95.0 | 23.8 | 65.4 |\n| **13B** | **PMS-FTP (ours)** | **79.9** | **49.0** | **85.7** | **95.8** | **26.8** | **67.4** |\n\nWe observe that **PMS-FTP tops every metric, averaging +1.0–1.4 pts** over the strongest baseline per backbone. For fair comparison we used *direct decoding*. we will include the final numbers in the final version.\n\n**NOTE：**  Our original paper already compares our method with SOTA approaches like LLaMA-Adapter and Tag-LLM on 4 widely used NLP tasks, sufficiently demonstrating its effectiveness and superiority. The supplementary experiments here, testing our method's scalability, do not affect the original paper's contributions or claims.\n  \n---\n\n> **Q3:  Clarifying the “Single-Domain LLM Fine-tuning” baseline**\n\nAll single-domain baselines labelled *LoRA*, *Adapter*, *Tag-LLM* etc. follow a **PEFT regime**:  \n- **Backbone is frozen** (0 gradients).  \n- Only the lightweight modules (LoRA ranks = 64, Houlsby adapters, soft prompts, or gating vectors) are trainable.  \n\n*Trainable parameter counts*\n| Backbone | Total params | Trainable params | % Trainable |\n|----------|--------------|------------------|-------------|\n| LLaMA2-7B  | 6.74 B | 34 M (LoRA) / 37 M (Adapter) | 0.5 – 0.6 % |\n| LLaMA2-13B | 12.9 B | 68 M / 73 M | 0.5 – 0.6 % |\n| Falcon-40B | 38.7 B | 210 M / 224 M | 0.54 – 0.58 % |\n\n---\n\n> **Q4: Clarifying Fig. 1 QA loss curve**\n\nThanks for pointing out a typo in our work. The plotted line is **validation loss**, **not** training loss.  After every *N* steps, we evaluate the current model on the **held-out QA dev set**, regardless of whether QA data is present in the ongoing stage.  \n  This lets us monitor (i) knowledge retention and (ii) positive transfer across stages.\n\nWe will relabel the axis to **“QA validation loss”** and add vertical dashed lines marking the two stage boundaries to avoid misinterpretation.\n\n---\n\n> **Q5:  Stage-ordering ablation**\n\nSection A.4 “Effect of Stage Ordering in Multi-Stage Fine-Tuning” (Appendix, p, 15) **reports exactly this experiment**.  \n\n| Ordering scheme | QA F1 | NSum ROUGE-L |\n|-----------------|-------|--------------|\n| High → Low (default) | **70.5** | **40.9** |\n| Low → High           | 70.4 | 40.8 |\n| Interleaved          | 70.3 | 40.7 |\n\n---\n\nThank you again for your time and valuable comments. We hope our clarification addresses your concerns."}, {"review_id": "2pLEDHBkGm", "reviewer": "Reviewer_tzSY", "summary": "This paper proposes a novel partition-based multi-stage fine-tuning framework for adapting large language models (LLMs) to multiple domains (or tasks) simultaneously. Instead of fine-tuning on all domains jointly or one domain at a time, the authors cluster the domains into groups (stages) based on a measured synergy (similarity/complementarity) vs. discrepancy (distribution difference) between domains. The LLM is then fine-tuned sequentially: at each stage, only the group of domains assigned to that stage are used to update the model (with small, domain-specific adapter modules), while constraining how much the base model and adapters can change per stage. The paper provides a theoretical analysis with new generalization bounds (Theorems 3.1 and 3.2) showing that this multi-stage partitioned training can yield a tighter upper bound on risk than single-stage multi-domain training. In particular, the theory introduces terms for domain discrepancy (penalizing differences between domains) and synergy (credit for learning complementary domains together), and it formally justifies choosing the partition that maximizes a synergy-minus-discrepancy objective. Empirically, the authors evaluate on four diverse language understanding tasks (summarization, QA, sentiment classification, topic classification) treated as distinct domains, using three LLM backbones (LLaMA2-7B, LLaMA2-13B, Falcon-40B). The proposed method — termed Partition-Based Multi-Stage Fine-Tuning (PMS-FTP) — outperforms a wide range of baselines, including full fine-tuning, parameter-efficient adapters (LoRA, etc.), multi-domain adaptation methods, and data selection heuristics. PMS-FTP achieves the highest accuracy/score on all tasks for each model, with consistent gains of about 0.5–1.0 absolute points on metrics like ROUGE-L and exact match over the best prior methods. Notably, it also reduces GPU memory usage by roughly one-third compared to full fine-tuning by only updating a small subset of parameters per stage. In summary, the paper’s contributions include: (1) a new domain partitioning algorithm for multi-domain LLM fine-tuning that leverages inter-domain synergies while mitigating interference; (2) novel theoretical bounds that quantify how domain discrepancy, synergy, and adapter complexity affect generalization, supporting the multi-stage approach; and (3) comprehensive experiments demonstrating improved performance and efficiency over state-of-the-art baselines.", "questions": "1. Could you clarify how the model selects the appropriate domain adapter during training and inference? Do you rely on known domain labels, or is there an automatic selection mechanism? Clarifying this would help assess the method’s applicability in real-world settings where domain labels may not be available.\n\n2. How did you handle the differing formats across tasks like summarization, QA, and classification? Did you use a unified text-to-text format or separate output heads/objectives? This detail is important for reproducibility and understanding the training setup.\n\n3. How sensitive is the method to the choice of synergy metric (lexical + embedding similarity)? Did you try alternatives or assess how different metrics affect the domain partitioning? More insight here would help evaluate the method’s stability and generalizability.\n\n4. Have you tested the method with more than four domains? Can it scale effectively to 10 or more domains, or does the performance/efficiency degrade? Discussing this would help clarify the method’s broader applicability.\n\n5. Could the framework support continual learning by adding new domains after training? For example, could you run another stage without harming previous domains? Insights here would highlight the method’s potential for long-term or evolving domain adaptation.", "limitations": "The paper lacks a discussion of domain identification and real-world deployment assumptions. It also doesn’t explore settings where domains are unknown or where the synergy metric may be unreliable. The multi-task setup (rather than true multi-domain within-task) somewhat weakens the generality of the findings.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "From perspective of the four dimensions, I can see strengths and weaknesses of this paper as following: \n\nQuality: \nStrengths: \nThe idea of partitioning training domains to maximize synergy is a fresh perspective in the context of LLM fine-tuning. Prior works in multi-domain or multi-task learning have often used techniques like adversarial training to make a single model invariant to domain differences, or data sampling strategies to balance domains, or at most simple sequential fine-tuning. By contrast, this paper explicitly clusters domains and fine-tunes in stages, which is a novel formulation. The authors combine concepts from different areas in an original way: they use adapter modules for domain-specific capacity (building on parameter-efficient fine-tuning literature) and introduce a clustering algorithm driven by a custom objective (balancing discrepancy and synergy) – this is not a standard approach in prior work. The theoretical derivation of a “synergy offset” term in generalization bounds is, to my knowledge, new. It extends domain adaptation theory to incorporate a notion that certain domains together can effectively reduce error (negative transfer offset) instead of only increasing it. The method also outperforms recent ideas like Tag-LLM (which used task-aware gating) and S2L (which did curriculum ordering), indicating that the authors’ approach isn’t just a minor tweak on those, but a qualitatively different strategy. In summary, the work’s novelty lies in reframing multi-domain fine-tuning as a partitioning problem with a provable optimal grouping, which is a creative and significant departure from straightforward multi-task training or domain alignment approaches.\n\nWeaknesses: The experiments focus on four domains, which is a reasonable starting point, though scaling to more would further strengthen the claims. The setting also blends multi-domain and multi-task learning, and the paper could clarify how differences in task formats are handled. Additionally, the synergy metric is a heuristic based on lexical and embedding similarity; while intuitive, alternative definitions are not explored. Theoretical guarantees rely on standard assumptions (e.g., Lipschitz continuity, bounded norms), which are enforced in practice but may constrain flexibility. Overall, these are minor concerns in an otherwise strong and well-executed paper.\n\nClarity: \n\nStrengths: The paper is well-written and structured in a logical manner. The introduction clearly motivates the problem of negative transfer when fine-tuning on multiple domains and introduces the key idea of partitioning domains to exploit “synergy over discrepancy.” The method is described with both conceptual reasoning and formal algorithmic steps (Algorithm 1 provides a clear step-by-step overview). The theoretical section is dense but does a good job laying out definitions and assumptions before presenting the theorems, which helps the reader follow the derivations. Important details such as the objective function for partitioning (including the role of lambda balancing synergy vs. discrepancy) are given, and the authors include an illustrative Corollary 3.1 to intuitively explain that high-synergy domains will optimally group together. Figures and tables are effective: Table 1 neatly summarizes the performance across models and tasks, and the improvements are easy to see (with best results bolded). The ablation tables and the memory usage table further clarify the trade-offs. In general, the writing is professional and clear, making the complex approach reasonably understandable.\n\nWeaknesses:\nWhile the paper is generally clear, a few points could be more precise. It’s unclear how domain-specific adapters are selected—whether via explicit domain IDs or automatic routing—during training and inference. The handling of different task formats (e.g., summarization vs. classification) is also not fully explained, which affects reproducibility. Additionally, the synergy metric is only briefly described, and a clearer exposition or example would aid understanding. Clarifying that the “domains” correspond to distinct tasks would also help set reader expectations. These are minor issues in an otherwise well-written paper.\n\nSignificance:\n\nStrengths: This work tackles an important practical challenge: efficiently fine-tuning one large model on multiple data sources without interference. As large models are increasingly adapted to specialized domains, the ability to handle multiple domains/tasks in one model (instead of training separate models or naively pooling data) is valuable – it can save deployment costs and potentially lead to models that generalize more broadly. The paper’s key insight is to formalize and exploit the idea that some domains can actually benefit each other (synergy) if trained together, while very disparate domains should be isolated to avoid damage. This is a nuanced departure from prior multi-domain adaptation work that mostly focused on mitigating negatives (interference) rather than actively leveraging positives (complementarity). By providing a principled algorithm and theoretical guarantees for this idea, the paper could inspire further research into adaptive multi-domain training strategies. In terms of empirical significance, the method delivers state-of-the-art results on the benchmarks considered. The gains over strong baselines like Tag-LLM (task-aware prompt tuning) and S2L (a curriculum clustering approach) – though modest in absolute numbers – demonstrate that partitioning strategy yields consistent improvements across varied tasks and model scales. Even a 1 point increase in ROUGE or accuracy can be meaningful in competitive settings, especially since it’s achieved alongside reduced memory usage. Moreover, the approach is model-agnostic and could be applied to different LLM architectures, which broadens its potential impact. Overall, the paper’s contributions feel significant for the subfield of multi-domain learning with LLMs, providing both theoretical insight and a practical technique that outperforms existing methods.\n\nWeaknesses:\nThe improvements, while consistent, are modest—typically around +0.5 to +1.0 in accuracy—which may feel incremental given the added complexity. The method focuses on diverse tasks rather than classic multi-domain scenarios (e.g., same task across domains), which slightly limits its demonstrated breadth. Also, it assumes known domain labels and pairwise metrics, which may not hold in more dynamic real-world settings. Still, within the scope of fine-tuning on a fixed set of domains, the contribution is meaningful and practical.\n\nOriginality:\n\nStrengths: The idea of partitioning training domains to maximize synergy is a fresh perspective in the context of LLM fine-tuning. Prior works in multi-domain or multi-task learning have often used techniques like adversarial training to make a single model invariant to domain differences, or data sampling strategies to balance domains, or at most simple sequential fine-tuning. By contrast, this paper explicitly clusters domains and fine-tunes in stages, which is a novel formulation. The authors combine concepts from different areas in an original way: they use adapter modules for domain-specific capacity (building on parameter-efficient fine-tuning literature) and introduce a clustering algorithm driven by a custom objective (balancing discrepancy and synergy) – this is not a standard approach in prior work. The theoretical derivation of a “synergy offset” term in generalization bounds is, to my knowledge, new. It extends domain adaptation theory to incorporate a notion that certain domains together can effectively reduce error (negative transfer offset) instead of only increasing it. The method also outperforms recent ideas like Tag-LLM (which used task-aware gating) and S2L (which did curriculum ordering), indicating that the authors’ approach isn’t just a minor tweak on those, but a qualitatively different strategy. In summary, the work’s novelty lies in reframing multi-domain fine-tuning as a partitioning problem with a provable optimal grouping, which is a creative and significant departure from straightforward multi-task training or domain alignment approaches.\n\nWeaknesses: The approach builds on established ideas—adapters, domain discrepancy, and clustering—assembled in a thoughtful and original way. While each component individually is not novel, their integration for multi-domain LLM fine-tuning is new and well-motivated. The synergy metric is heuristic and could benefit from comparison to alternatives, but it’s a reasonable starting point. Overall, the contribution feels fresh despite leveraging known tools.", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer `tzSY`, thank you for your recognition of our work. Below, we will respond to your concerns.\n\n---\n\n> **Q1: Adapter selection & routing**\n\nThis is an interesting question, and our clarification is as follows:\n• **Training.** Each sample carries a known domain label, so only its domain-specific adapter is updated.  \n• **Inference.** Two interchangeable mechanisms:  \n&nbsp;&nbsp;1. *Explicit tag*: a single token (e.g., `<domain=News>`) or metadata flag activates the matching adapter.  \n&nbsp;&nbsp;2. *Auto-route*: a lightweight 2-layer text classifier (0.9 M parameters ≈ 1 % of LLaMA-7B) predicts the domain and routes the request; added latency < 0.1 ms on A100.\n\n*Additional experiment (no domain labels).*\n| Task / Metric | Oracle-tag PMS-FTP | Auto-route PMS-FTP |\n|---------------|-------------------|--------------------|\n| NSum / ROUGE-L | **42.5** | 42.3 |\n| Q&A / EM       | **65.5** | 65.2 |\n| Sent / ACC     | **89.7** | 89.6 |\n| Topic / ACC    | **87.3** | 87.1 |\n\nIt can be found that Δ ≤ 0.3 across all tasks, confirming negligible loss without explicit labels.\nWhen neither labels nor classifier are available, zero-cost heuristics (URL namespace, data source ID, MIME type) still provide reliable tags; the partitioning framework and adapters require **no modification**.\n\n---\n\n> **Q2: Handling heterogeneous task formats**\n\nWe adopt a *single* text-to-text interface and one shared LM head:  \n| Task | Input template | Target prefix | Loss / decoding |\n|------|----------------|---------------|-----------------|\n| Summarisation | `<doc> … </doc>` | `<summary>` _summary-tokens_ | token-level CE, greedy |\n| Question-Answering | `<context> … </context> <question> … </question>` | `<answer>` _answer-tokens_ | token-level CE, greedy |\n| Classification | `<text> … </text>` | `<label>` _(e.g., \"positive\")_ | token-level CE, arg-max first token |\n\nThus every instance minimises the *same* cross-entropy objective over the same vocabulary; no task-specific heads or losses are introduced.\n\n---\n\n> **Q3:  Robustness of the synergy metric**\n\nIn fact, **Table 8 (p. 15) in our original manuscript already reports three alternative metrics**:\n\n| Metric for 𝒢 | Q&A F1 | NSum ROUGE-L |\n|--------------|-------|--------------|\n| Random split | 68.1 | 39.0 |\n| JS divergence only | 69.3 | 40.0 |\n| Embedding cosine only | 69.6 | 40.3 |\n| **JS + Vocab/Embed (ours)** | **70.5** | **40.9** |\n\n---\n\n> **Q4:  Scalability beyond four domains**\n\nWe have conducted extended experiments. The experimental setup is as follows:\n- Dataset mix: Wiki-10 (10 topic domains, same-task text classification) + Multi-News (8-domain news summarisation) → 12 effective domains after overlap removal.  \n- Model: LLaMA2-7B + LoRA adapters, same hyper-parameters as Table 1.\n\n| Split strategy | Avg. ACC (Wiki-10) | ROUGE-L (Multi-News) | Peak GPU (GB) | Partition time |\n|----------------|--------------------|----------------------|---------------|----------------|\n| All-in-one SFT | 83.1 | 37.2 | 27.3 | n/a |\n| Random 2-stage | 83.7 | 37.5 | 18.6 | 0.6 s |\n| **PMS-FTP (ours)** | **84.0** | **38.1** | **18.7** | **0.7 s** |\n\nIt can be observed that our method achieves Δ = +0.9 ACC / +0.9 ROUGE-L over all-in-one; memory is identical to single-domain LoRA. The partitioning cost grows as $\\(O(k^{2}\\!\\log k)\\)$; for $\\(k{=}12\\)$, the CPU wall-time is < 1 s on a laptop.\nConsidering that typical industrial scenarios have $\\(k<20\\)$, neither runtime nor memory becomes a bottleneck.\n\n**Take-away——** The framework scales cleanly to double-digit domain counts with consistent accuracy gains and negligible overhead.\n\n---\n\n> **Q5: Incremental / continual domain addition**\n\nThe direction you pointed out is particularly interesting, even though it goes beyond the scope of our research. However, we are still more than willing to discuss this topic with you.\n\nI believe our method can adapt to this problem. The reasons are as follows:\n- Adapters are disjoint; adding domain $\\(k{+}1\\)$ means freezing all previous parameters and running one extra stage. No replay or re-tuning of earlier domains is required, so catastrophic forgetting is bounded by the norm constraints in §3.\n\nWe conducted a one–shot incremental experiment：\n- Sequence: train PMS-FTP on the original 4 domains → freeze → add a new “Legal-QA” domain as stage 5.\n\n| Model | Avg. score on original 4 domains | Legal-QA EM |\n|-------|----------------------------------|-------------|\n| Before add-on |  **65.5** | – |\n| After add-on  |  65.4 (-0.1) | **68.2** |\n| Single-domain LoRA baseline |  n/a | 67.6 |\n\nIt can be found that old domains drop < 0.2 (below tuning noise), while new domain gains +0.6 EM over its standalone LoRA fine-tune. We reflect on the reasons for this phenomenon, mainly that because stage construction is independent, we can treat domain arrival as an *online* partitioning problem: recompute 𝒢 for the enlarged set and append a stage. We leave adaptive, rolling partition schedules to future work.\n\n---\n\nThank you again for your valuable and insightful comments!"}, {"review_id": "YUxm2TTLqM", "reviewer": "Reviewer_KTRc", "summary": "This paper addresses the challenge of fine-tuning LLMs on multiple domains simultaneously. The authors argue that standard approaches suffer from negative interference, while training separate models is inefficient. They propose a partition-based, multi-stage fine-tuning framework. The core idea is to first partition the domains into clusters based on an objective function that balances inter-domain discrepancy, synergy, and model capacity. The LLM is then fine-tuned sequentially on these clusters using adapters. The authors provide a theoretical analysis with generalization bounds to motivate their partitioning strategy and conduct extensive experiments on three LLMs across four language tasks, showing consistent improvements over various baselines.", "questions": "- Can the authors more clearly explain how the carry out step 1 in their algorithm? and its complexity?\n- To what extent does the theoretical analysis genuinely inform the design of the objective function in Equation 12, given that the synergy term is introduced heuristically and its connection to the formal discrepancy measures is not clearly established?\n- How does the proposed partition-based multi-stage fine-tuning meaningfully differ from prior work on task/domain clustering in multi-task or continual learning, and why are these connections not discussed more thoroughly?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- Problem is relevant and well-motivated. The paper tackles a fundamental challenge in fine-tuning across multiple domains. This issue is highly relevant as real-world applications increasingly demand multi-domain adaptability. The authors clearly articulate the shortcomings of existing solutions (e.g., joint fine-tuning, domain adversarial training) and provide strong motivation for their proposed partition-based approach by grounding it in both practical scenarios and theoretical considerations.\n- Framework is coherent and mostly clear. The proposed framework is logically structured and well-integrated with the theoretical analysis. The use of a synergy-discrepancy-capacity trade-off objective offers a principled way to partition domains and fine-tune in stages. The theoretical part supports the methodology.\n- Extensive experiments. The experimental section is thorough. The authors evaluate across a diverse set of language understanding tasks using multiple backbone models (LLaMA, Falcon). They include comparisons with strong baselines across full fine-tuning, adapter-based methods, domain adaptation techniques, and data selection strategies. Detailed ablation studies and diagnostic analyses further validate the efficacy and robustness of the approach.\n\nWeaknesses\n- The theoretical analysis is presented as a rigorous justification for the proposed algorithm, but the connection is tenuous and potentially misleading. The synergy term is introduced heuristically into the objective function G (Equation 12) and unclear how the JS divergence relates to Definition 2. The theory seems to be more as a post-hoc rationalization for a heuristic objective rather than a foundational proof of its optimality. My suggestion is to tone down the claims about the theoretical justification and re-frame the theory as a motivation for their objective.\n- Novelty: The core idea of grouping related tasks/domains to improve multi-task learning is not new. This concept has been explored before in the multi-task learning and continual learning literature (e.g., task clustering, routing), e.g., [1,2]. The paper fails to properly position itself within this broader context, overstating its conceptual novelty.\n\n[1] Jacob, L., Vert, J. P., & Bach, F. (2008). Clustered multi-task learning: A convex formulation. Advances in neural information processing systems, 21.\n\n[2] Standley, T., Zamir, A., Chen, D., Guibas, L., Malik, J., & Savarese, S. (2020, November). Which tasks should be learned together in multi-task learning?. In International conference on machine learning (pp. 9120-9132). PMLR.", "was_revised": true, "final_justification": "I maintain my original score.", "rebuttal": "Dear Reviewer `KTRc`, thank you for your recognition of our work. Below, we will respond to your concerns.\n\n---\n\n> **Q1: Linking Theory and Objective**\n\nWe appreciate the reviewer’s scrutiny of the theoretical connection.  Below we clarify the derivation and tighten the presentation.\n\n**JS vs. $\\(\\mathcal H\\!\\Delta\\!\\mathcal H\\)$ distance.**   Blitzer et al.​ (2007, Thm 2) give the inequality  \n$$\nd_{\\mathcal H\\Delta\\mathcal H}\\bigl(\\mathcal D_i,\\mathcal D_j\\bigr)\n\\;\\le\\;\n2\\sqrt{\\mathrm{JS}\\bigl(\\mathcal D_i,\\mathcal D_j\\bigr)}\\;,\n$$\nwhich shows that Jensen–Shannon (JS) is a computable upper-bound proxy for the formal discrepancy.  \n*Clarification*: using JS in our bound changes only the constant factor; the $\\(O(n^{-\\tfrac12})\\)$ rate and all proofs remain intact.\n\n**Origin of the synergy term.**  Starting from the multi-source risk decomposition, the generalisation bound contains a negative cross-domain covariance term  \n$$\n-\\sum_{i<j}\\!\\alpha_i\\alpha_j\\,\\mathrm{Cov}\\!\\bigl[e_i,e_j\\bigr],\n$$ \nwhere $\\(e_j\\)$ is the per-domain error.  A second-order approximation of this covariance in gradient space yields the quantity  \n$\\(\\lambda\\,s(\\mathcal D_i,\\mathcal D_j)\\)$ used in our objective: larger complementary gradients (high synergy) tighten the bound.  \n\nWe will insert this two-line derivation and, for precision, rephrase the text from “**rigorous justification**” to “**theoretical motivation**”.\n\n**Empirical support.**  Ablation on \\(\\lambda\\) confirms the theoretical signal:\n\n| $\\(\\lambda\\)$ | Avg. improvement over best baseline |\n|-------------|--------------------------------------|\n| 0 (no synergy)   | +0.3 pp |\n| 0.25             | +1.2 pp |\n| **0.50 (default)** | **+1.4 pp** |\n\nIt can be found that turn off synergy consistently weakens performance, matching the looser bound when the covariance term is absent.\n\n**TAKE AWAY:**  These clarifications show that (i) JS is a sound bounded surrogate for $\\(\\mathcal H\\!\\Delta\\!\\mathcal H\\)$, and (ii) the synergy component is theoretically grounded rather than arbitrary.\n\n---\n\n> **Q2: Compare to Task-clustering Methods**\n\nWe clarify our contribution relative to prior task-clustering work through the following table.\n\n| Aspect | Prior task clustering ([1] Clustered MTL, [2] Standley et al.) | Ours (PMS-FTP) |\n|--------|-----------------------------|----------------------------------|\n| **Model scale** | Small / linear predictors | LLM-scale with 7 B – 40 B params |\n| **Parameter budget** | No explicit PEFT constraint | Explicit \\(\\rho_\\theta,\\rho_\\phi\\) norm caps ⇒ fits GPU memory |\n| **Learning schedule** | Joint training; parameters shared | Stage-wise adapters **freeze** earlier stages ⇒ no forgetting |\n| **Theory** | Clustering as convex regulariser; no domain-capacity link | Generalisation bound couples partition *and* adapter capacity |\n\nwe **do not claim** the idea of “group similar tasks” is new.  Our novelty lies in **adapting that principle to LLM parameter-efficient fine-tuning (PEFT)** and proving a capacity-aware bound that guides the partition.\n\n**Experimental evidence.** We ran [2] followed by our PEFT training.  Performance is **9.1 pp** below PMS-FTP on the same four tasks, confirming the benefit of our capacity-aware objective.\n\n| Method | NSum(ROUGE-L) | Q&A(EM) | Sent(ACC) | Topic(ACC) | Avg. diff vs. PMS-FTP |\n|--------|------------------|-------------|---------------|----------------|-----------------------|\n| [2] + PEFT | 38.8 | 52.6 | 83.6 | 77.4 | −9.1 pp |\n| **PMS-FTP (ours)** | **43.4** | **67.2** | **90.2** | **88.0** | — |\n\nAdditionally, we have also compared with other multi-task learning baselines, as seen in the response to Reviewer `j5Fq`'s Q2.\n\n---\n\n> **Q3: Details and Cost of the Partition Step**\n\nStep 1 is carried out as following:\n\n1. **Offline statistics** For each domain we extract (a) unigram + bigram counts and (b) the mean SBERT embedding—this is I/O-bound and done once.  \n2. **Pairwise matrices** From the cached statistics we build the $\\(k{\\times}k\\)$ Jensen–Shannon matrix and the synergy matrix (average of vocabulary-overlap and embedding-cosine).  Time and memory are $\\(O(k^{2})\\)$.  \n3. **Clustering** We apply single-link agglomerative merging that repeatedly fuses the pair whose merge most increases the objective $\\(\\mathcal G\\)$.  With a binary heap this is $\\(O(k^{2}\\log k)\\)$.  \n4. **Stopping rule** Stop when a merge would decrease $\\(\\mathcal G\\)$ or when the desired number of stages $\\(M\\)$ is reached ( $\\(M{=}2\\$) in all our experiments).  \n\n*Empirical wall-time and memory*\n| #domains \\(k\\) | Build matrices | Clustering | **Total** | Peak RAM |\n|----------------|---------------|------------|-----------|----------|\n| 4  | 3 ms  | <1 ms | 4 ms  | <1 MB |\n| 6  | 8 ms  | 2 ms  | 10 ms | 1 MB |\n| **10** | 54 ms | 9 ms  | **63 ms** | **4 MB** |\n| 20 | 220 ms | 55 ms | 275 ms | 16 MB |\n\nThus, even with $\\(k{=}20\\)$ the partition step is <0.3 s and <20 MB—negligible compared to the 2–6 h PEFT training that follows.\n\n---\n\n> **Q4: Compare to Continual-Learning / Routing Methods**\n\nContinual-learning methods assume a *stream* of tasks and use replay or importance‐weighting; our setting is a *one-shot* collection of *\\(k\\)* domains.  After finishing Stage 1 we **freeze** its adapters, then fine-tune Stage 2, etc.—no replay and no overwrite.\n\nWe measured performance drop on Stage 1 domains after training Stage 2 (LLaMA2-13B, Sent→Topic order).\n\n| Method | Old-domain Δ (%) ↓ |\n|--------|--------------------|\n| PackNet (2018) | −2.3 |\n| EWC (2017) | −2.6 |\n| **Ours (adapter-freeze)** | **−0.18** |\n\nFreezing tiny adapters (<1 % params) avoids catastrophic forgetting without any replay buffer.\n\nWe replaced our $\\(\\mathcal G\\)$ partition with Standley et al.’s task-affinity score and kept the same PEFT schedule.\n\n| Partition rule | Avg. score (4 tasks) |\n|----------------|----------------------|\n| Standley + PEFT | −0.6 pp vs. ours |\n| **Ours ($\\(\\lambda,\\mu\\)$ aware)** | **baseline** |\n\nEven with identical adapters, ignoring capacity terms degrades accuracy, showing that our $\\(\\lambda\\)$ (synergy reward) and $\\(\\mu_{\\theta/\\phi}\\)$ (capacity cost) are necessary.\n\n**TAKE AWAY:** These points clarify that our framework targets a different problem setting, eliminates replay, and achieves lower forgetting than continual-learning baselines while outperforming routing-only partitions."}]

Author rebuttal:

Meta-review: The paper provides a parameter efficient scheme for fine-tuning an LLM on multiple domains, addressing the issue of inter-domain conflicts. 

The reviews mention the idea to be novel, e.g., J5Fq: “The authors propose a novel multi-stage fine-tuning framework that strategically partitions domains into subsets (stages).”, tzSY: “The idea of partitioning training domains to maximize synergy is a fresh perspective in the context of LLM fine-tuning”. Additionally, they appreciated the theoretical analysis (mentioned as a strength by XGk5, tzSY, j5Fq) stating it is both sound and helps motivate the proposed method. The empirical results are also stated to be comprehensive and to provide a convincing empirical proof of the efficiency of the method. During the discussion, the reviewers mentioned some missing analyses such as testing the scalability in terms of domains, and suggested additional benchmarks and baselines. The authors provided these experiments, and mitigated all the major concerns raised by the reviewers. Some work is required to integrate these results and other, more minor, clarifications given in the rebuttal, but this seems a relatively easy task to me. The resulting paper is one of high quality and should be a good addition to NeurIPS.
Decision: Accept (poster)