ADEPT: Continual Pretraining via Adaptive Expansion and Dynamic Decoupled Tuning
Authors: Jinyang Zhang
Yue Fang
Hongxin Ding
Weibin Liao
Muyang Ye
Junfeng Zhao
Yasha Wang
Xu Chu
Venue: ICLR 2026 Poster (2026)
Keywords: Continual Pretrain
Large Language Models
Parameter-Efficient Training
Primary area: foundation or frontier models, including LLMs
TL;DR: A novel continual pretraining framework that combines adaptive model expansion with dynamic parameter updating to efficiently adapt large language models to new domains while preserving existing knowledge.

Abstract: Conventional continual pretraining (CPT) for large language model (LLM) domain adaptation often suffers from catastrophic forgetting and limited domain capacity. Existing strategies adopt layer expansion, introducing additional trainable parameters to accommodate new knowledge. However, the uniform expansion and updates still entangle general and domain learning, undermining its effectiveness. Our pilot studies reveal that LLMs exhibit functional specialization, where layers and units differentially encode general-critical capabilities, suggesting that parameter expansion and optimization should be function-aware. We then propose ADEPT, Adaptive Expansion and Dynamic Decoupled Tuning for continual pretraining, a two-stage framework for domain-adaptive CPT. ADEPT first performs General-Competence Guided Selective Layer Expansion, duplicating layers least critical for the general domain to increase representational capacity while minimizing interference with general knowledge. It then applies Adaptive Unit-Wise Decoupled Tuning, disentangling parameter units within expanded layers according to their general-domain importance and assigning asymmetric learning rates to balance knowledge injection and retention. Experiments on mathematical and medical domains show that ADEPT outperforms full-parameter CPT by up to 5.76% on the general benchmarks and 5.58% on the target domain benchmarks with only 15% of parameters tuned and less than 50% training time. Ablation studies, theoretical analysis, and extended investigations further demonstrate the necessity of targeted expansion and decoupled optimization, providing new principles for efficient and robust domain-adaptive CPT. Our code is open-sourced at https://github.com/PuppyKnightUniversity/ADEPT

Published as a conference paper at ICLR 2026
ADEPT: CONTINUAL PRETRAINING VIA ADAPTIVE EXPANSION AND DYNAMIC DECOUPLED TUNING
Jinyang Zhang 1,,,, Yue Fang!.2. .2. Hongxin Ding 1,,,, Weibin Liao 1,2,,, MMuang Ye²,
Junfeng Zhao 11.2,44 Yasha Wangf 2.5.6 Xu Chuf.1,277 1 School of Computer Science, Peking University
Key Laboratory of High Confidence Software Technologies, Ministry of Education ZZeejiang University, Hangzhou, China
"Big Data Technology Research Center, Nanhu Laboratory
5 National Engincering Research Center For Software Engineering, Peking University
PPeking University Information Technology Institute (Tianjin Binhai)
7 CCenter on Frontiers of Computing Studies, Peking University
X j
wetpaann22estuuuuueiieee edu.cn
chuxu, zhaojf, wangyashaj@pku.eeuu.c
ABSTRACT
Conventional continual pretraining (CPT) for large language model (LLM) do- main adaptation often suffers from catastrophic forgetting and limited domain ca- pacity. Existing strategies adopt layer expansion, introducing additional trainable parameters accommodate new knowledge. However, the uniform expansion and updates still entangle general and domain learning, undermining its effective- ness. Our pilot studies reveal that LLMs exhibit functional specialization, where layers and units differentially encode general-critical capabilities, suggesting that parameter expansion and optimization should be function-aware. We then pro- pose ADEPT, Adaptive Expansion and Dynamic Decoupled Tuning for continual pretraining, a two-stage framework for domain-adaptive CPT. ADEPT first per- forms Cineral-Competent
Guided Selective Layer Expansion, duplicating lay- ers least critical for the general domain increase representational capacity while minimizing interference with general knowledge. It then applies Adaptive Unit- Wise Decoupled Tuning, disentangling parameter units within expanded layers according to their general-domain importance and assigning asymmetric learning rates to balance knowledge injection and retention. Experiments on mathematical and medical domains show that ADEPT outperforms full-parameter CPT by up to 5.76% on the general benchmarks and 5.58% on the target domain benchmarks with only 15% of parameters tuned and less than 50% training time. Ablation studies, theoretical analysis, and extended investigations further demonstrate the necessity of targeted expansion and decoupled optimization, providing new prin- ciples for efficient and robust nomain-adaptin CPT.
INTRODUCTION
Large language models (LLMs) have demonstrated remarkable performance across wide range of general-domate tasks (OpenAI, 2023; Dubey et al., 2024c). However, their deployment in special- ized domains, such as mathematics or healthcare, requires targeted adaptation (Ding et al., 2024; Chen et al., 2024; Ahn et al., 2024). Continual pretraining (CPT), which conducts post-pretraining on domain-specific corpora, has emerged capabilities into pretrained LLMs (Wu et al., 2024a; Ibrahim et al.. 2024; Yildız et al., 2024).
crucial paradigm for injecting domain knowledge and
Despite its promise, CPT faces persistent challenge: catastrophic forgetting. After pretraining, LLMs already encode substantial general knowledge, leaving limited parameter capacity for inte- grating new Nomain--spciffe information. While domain signals can be forcefully fitted through
"Equal contribution.
Corresponding author.

---

Published as a conference paper at ICLR 2026
gradient-based optimization, the aggressive updates on the existing parameters come at the cost of overfitting to the target corpora, which in turn disrupts general abilities and triggers catastrophic forgetting (Liu et al., 2024a; Luo et al., 2025). This tension between new knowledge injection and previous knowledge retention poses a central obstacle to reliable and stable domain adaptation.
To address catastrophic forgetting, some approaches attempt through data-centric strategies, such as data replay or rehearsal (Huang et al., 2024; Zhang et al., 2025). While replay partially preserves prior knowledge, it fails to expand model capacity, leaving the conflict between knowledge injection and retention unresolved. Others focus on increasing capacity via transformer-layer extension (Wu al., 2024b), yet typically insert new layers uniformly and update all parameters indiscriminately. This expansion strategy neglects the functional specialization within LLMs, where different layers and neurons serve distinct functional roles. Our pilot studies reveal that general-critical layers in LLMs are mainly located in early depths, and functional units within layers contribute unequally to general-domain performance, highlighting functional specialization similar to that found in the human brain (Xu et al., 2025; Zheng al., 2024; Dai et al., 2022c). Consequently, indiscriminate expansion and optimization may overwrite general-critical regions with new knowledge, compro- mising general competeney preservation and leaving forgetting unresolved.
Inspired by the functional specialization perspective, we propose our core insight: effective CPT should expand and update the model adaptively, preserving the regions responsible for the general domain and targeting more adaptable parameters. Specifically, we argue that capacity allocation must be amportance-guideen and optimization must be function- deccupled to minimize in- terference with general competencies, As illustrated in Figure 1, domain-specific extension should be allocated the regions less constrained by general-domain knowledge and skills, and param- eters within these regions should be decoupled and tuned accordingly, preserving general-critical parameters and allowing the rest to be more adaptable to absorb new domain- specific information.
Building on this insight, we propose Adaptive Expan- sion and Dynamic Decoupled Tuning for continual pre- training (ADEPT), framework for domain-adaptive continual pretraining. ADEPT comprises two stages: General-Competence Guided Selective Layer Expansion, which identifies and duplicates layers least critical for the general domain, allocating additional capacity pre- cisely where interference with general capabilities is min- imized, thereby preventing catastrophic forgetting. Adap- tive Unit-Wis Decoupled Tuning, which disentangles the parameters within the expanded layers based on their importance to the general domain. Asymmetric learn- ing rates are then applied on their subsets, ensuring that general-critical parameters are preserved while more adaptable parameters can fully absorb domain-specific knowledge. Extensive experiments mathematical and medicine domains demonstrate that ADEPT enables effi- cient and robust domain knowledge injection, while sub- stantially alleviating catastrophic forgetting. Specifically, compared to aall-parmetter CPT, ADEPT achieves up to 5.58% accuracy gain on target-domain benchmarks, and up 5.76% gain the general domain, confirming both effective knowledge acquisition and strong retention of general competencies. Furthermore, ADEPT attains these improvements with only 15% of parameters tuned, and reduces training time relative to other baselines greatly, highlighting its efficiency. Ablation studies and theoretical analysis further validate the designs of ADEPT.
General-nniccl ponmmeters
General Core Preserve General Knowledge Skills
densinendggeeble paremeters
Least Importont Region for General Domain
Target Domain Extension
Figure 1: Illustration of the core idea of ADEPT. Target domain extension applied on the least important region for general domain, minimizing catas- trophic forgetting. Asymmetric learn- ing rates are applied parameter sub- sets for targeted knowledge injection.
To summarize, our contributions are threefold:
1. Insightfully, we highlight the importance of considering functional specialization in LLMs for continual pretraining through empirical experiments and theoretical analysis, advocating for tar- geted layer expansion and decoupled training as a principled solution to domain adaptation.
2. Technically, we propose ADEPT, framework that consists of General-Competeenc Guided Selective Layer Expansion and Adaptive Unit- Wnse Decoupled Tuning, enabling adaptive and effective domain knowledge integration while minimizing catastrophic forgetting.
2

---

Published as a conference paper at ICLR 2026
Qtwen3-1.78 Baase
Quven3-4B-Bass
Qwen3-68-Bass
27
03
19
18
14
90
10
Figure 2: Layer- and unit-level importance distribution of the Qwen3 family. The vertical axis corresponds to different layers, while the horizontal axis denotes parameter units within each layer. Deeper blue indicates higher importance for preserving general-domain competencies.
Empirically, we conduct extensive experiments on both mathematical and medical domains, demonstrating that ADEPT consistently outperforms baselines in domain performance while pre- serving general competencies.
PILOT STUDY: PROBING PARAMETER IMPORTANCE
2.1 EXPERIMENTAL SETUP FOR IMPORTANCE PROBING
To investigate the functional specialization of LLMs and understand how different parameters con- tribute to preserving general-domain knowledge during CPT, we conduct importance probing on multiple backbone models, including Qwen3-Bas Base (1.7B, 4B, 8B) (Yang et al., 2025) and LLaMA3- (Dubey al., 2024b). Our analyses focus on probing general-knowe soooral-knowlede-eetttte parameters rather than domain-specifte ones. The rationale is that successful CPT must inject new, domain- specific knowledge without inducing catastrophic forgetting. This necessitates identifying and pre- serving the model's parameters that crucial for its general-domain competencies. By trast, domain knowledge can then be effectively allocated to less critical parameters, without risking the erosion of pre-existing knowledge and skills. To support this analysis, we construct General Competence Detection Corpus containing broad world knowledge and instruction-folloiing tasks in both English and Chinese, which serves the probing ground to reflect model's general com- petencies. Details of its construction are provided in Appendix B.3.
2.2 MAYER-LEVER IMPORTANCE PROBING
Our first research question is: How do different layers contribute to preserving general knowl- edge? To answer this, we measure the importance of each transformer layer by the model's degra- dation in general-domain performance when that layer is ablated. Formally, given the General Competence Detection Corpus D Dprobe, first compute the baseline next-token prediction loss of the pretrained LLM Mo:
Lruse
Σ (Mo())),),
(1)
ID proubel
WED
where E(-) denotes the standard next-token prediction loss in CPT. For each transformer layer E (1...., L). we mask its output via residual bypass and recompute the loss:
(
Σ
(x),
(2)
1D prube
BE DE pdbe

---

Published as a conference paper at ICLR 2026
where Ma-1 denotes the model with the 1-th layer masked. The importance of layer / defined as the loss increase relative to the baseline:
Layer (() Coase.
A larger layer indicates that layer plays more critical role in preserving general knowledge. Fig- ure 2 (left-hand bars) reports the layer-level importance distributions of the Qwen3 family (results for LLaMA3-BB provided in Appendix D). We find that seeerra--nowledge--eere centrated in the early layers, with importance gradually decreasing toward later layers. This uneven distribution suggests that uniformly expanding layers across the entire depth would be subopti- mal. Since some layers are tightly coupled with general knowledge while others more flexible, uniform expansion not only risks representational interference in critical layers but also allocates parametric budget where it is too constrained be leveraged for domain learning. In contrast, iden- tifying more adaptable layers with minimal impact on general knowledge and allocating expansion there for knowledge injection is a superior strategy. This leads our first key observation:
layers are con-
Observation I: Layers exhibit heterogeneous importance for preserving general competencies, which motivates a selective expansion strategy that targets layers less constrained by general abili- fies yet more adaptable for domain adaptation.
2.3 UNIT-LEVEL IMPORTANCE PROBING
Building on the layer-level exploration, our next research question is: How do parameter units within each layer contribute to preserving general knowledge? To answer this, partition each transformer layer into functional units (e.g., attention projections, MLP components, and normaliza- tion) and assess their relative contributions to preserving general competencies. The detailed parti- tioning scheme is provided in Appendix C. This granularity provides a more fine-grained perspective than layer-level probing, while avoiding the prohibitive cost of neuron-level analysis. Formally, for each parameter 0j in unit U, we estimate its importance using a first-order Taylor approximation:
= 0j Vo, f.
where L is the autoregressive training loss. The importance of unit is then defined as the average importance of its parameters:
Junit
Σ4 4
U
JEU
A higher Luiit indicates that the unit plays a more critical role in preserving general competencies. Figure (right-hand heatmaps) illustrates the unit-level importance distributions of the Qwen3 fam- ily (results for LLaMA3-8B provided in Appendix D). We observe that importance is unevenly distributed across modules within layer, with some units contributing more general competen- cies and others more flexible. This finding suggests that treating all parameter units equally would be suboptimal,
single update rule cannot simultaneously protect critical units and fully train adaptable ones, risking either damaging previous knowledge or failing to sufficiently learn new knowledge. This motivates us pursue unit-level decoupling, where training can selectively protect critical units while enabling less general-relevant units to absorb new knowledge without constraint. This leads to our second key observation:
Observation II: Parameter units within each layer exhibit heterogeneous importance, which mo- tivates unit-level decoupling that selectively protects critical units while enabling more adaptable ones to sufficiently absorb domain knowledge.
Summary. Building on the above observations, we propose ADEPT, a continual pretraining frame- work designed to enable effective domain knowledge injection while preserving general competen- cies. Inspired by the uneven importance distribution layers (Observation I), ADEPT adopts a se- lective expansion strategy that targets layers less constrained by general abilities yet more receptive to domain adaptation depaating from uniform expansion approaches like LLaMA-Pro (Wu et al., 2024b). Guided by the heterogeneous importance of parameter units within layers (Observation II), ADEPT further introduces unit-level decoupling on the expanded layers, enabling critical units to be protected while allowing adaptable units specialize in domain knowledge.
4

---

Published as a conference paper at ICLR 2026
Stage 1: Ceverol-Competence Guided Selective Layer Exponsion
Stage Adaptive Decoupled Tuning
Step Competence Aware Loyer Importonce Probing
Shep Unit-wise Neuron Decoupling
Probing Iteration
Detlctico Corpus
Colculate
Impartance
Frezen
(wi)))
Capy
LLM
General-rthical Units
AC
Units
Flom
Fareaed
Gewrdd Gompelone
Acthoted
Doman
Step Expension Identity Copy
Step 2: Dynomic Learning Rate Adaptation
Pretroin
Impartence
Genendl
Lowest-K
Original
Layers
Irdex
Copy
Copy
Copy
Decoupled
Learning
Masied
Tuning
Update
Learning
Expesded
Exponsion
Naxc Taken Leve
Figure 3: Illustration of ADEPT.
3 METHODOLOGY
As illustrated in Figure 3, ADEPT includes two stages:
# Stage 1: General-Competence Guided Selective Layer Expansion. adaptively selects and du- plicates layers that minimally affect general competencies while being more adaptable domain- specific knowledge, thereby introducing fresh representational capacity for domain adaptation. # Stage 2: Adaptive Unit-Wise Decoupled Tuning. further decouples units within the expanded layers and apply learning-rane
adaptive tuning according to their importance to the general domain, ensuring knowledge injection while preserving general competencies.
3.1 GENERAL-COMPETENNG GUIDED SELECTIVE LAYER EXPANSION
This stage aims to selectively expand model parameters in way that introduces fresh representa- tional capacity for domain adaptation while preserving general-domain competencies. To this end, we first estimate the contribution each transformer layer to preserving general knowledge through General- CoierallCCmeteeceeee Aware Layer Importance Probing, and then perform Selective Parameter Ex- pansion via Identity Copy to duplicate layers that are least critical for general abilities yet more adaptable to domain-specifie knowledge.
General- Comeral-Comeeeenee Aware Layer Importance Probing, To guide selective expansion, we lever- age the layer importance scores ( defined as Fq.3. Intuitively, I quantifies how much the I-th layer contributes to preserving general-domale knowledge. Layers with lower scores are deemed less critical for general competencies and are thus selected for expansion, as they can accommodate domain-specific adaptation with minimal risk of catastrophic forgetting.
Selective Parameter Expansion via Identity Copy. Based on the importance scores layer we sort layers by ascending importance and select the k least- coutt-mmpottat ones for general competence:
S₁ arg min (C.1....L Si=k
Σ
layer
6
LES
We denote the selected set S& as the Domain-Adaptable Layers. For each selected layer / Sk, create parallel copy by directly duplicating its parameters without re-initialization (6(1) (0).
5

---

Published as a conference paper at ICLR 2026
To preserve stability, we follow the Function Preserving Initialization (FPI) principle (Chen et al., 2015), ensuring that the expanded model M1 produces identical outputs as the original model Mo at initialization. Concretely, in the duplicated branch, set the output projections of both attention and feed-forward sublayers to zero [WMIJSA = 0, WPP = 0), so the forward computation remains unchanged (M (a)) (x);; Va)). The duplicated layers thus provide fresh sanneralkkoowldge-rrtiiic parameters in the original pathway. As formally established in Appendix F.I, expanding the layers with the lowest
fresh representational capac-
ity that can specialize for domain signals with minimal risk of eroding
importance provably minimizes the risk of forgetting. Intu- itively, this strategy ensures that new capacity is added where interference with general abilities is weakest, yielding the most favorable trade-off between domain adaptation and knowledge retention.
3.2 ADAPTIVE UNIT-WISE DECOUPLED TUNING
This stage aims further reduce catastrophic forgetting and enable fine-grained control over pa- rameters within the expanded layers. To achieve this, we first decouple each expanded layer into semantic units and evaluate their importance using gradient-basel estimation (Unit-wise Neuron Decoupling), and then dynamically adjust learning rates for different units according to their impor- tance scores during training (Dynamic Learning Rate Adaptation).
Unit-wise Neuron Decoupling, Guided by the heterogeneous importance of parameter units within layers, we performs unit-level decoupling on the expanded layers. Following the probing analysis in Section 2.3, we quantify unit importance Lunit using gradient sensitivity signals (cf. Eq.5), which aggregate the first-order contributions of parameters Og to the training loss L via Vo,f. higher Juni indicates greater contribution to general competencies and thus warrants more conservative updates, whereas less important units are encouraged to adapt more aggressively to nomain-specifi signals.
Dynamic Learning Rate Adaptation. Based on the unit importance Lunit in Eq.5, we assign adap- tive learning rates to different units within the expanded layers:
Iry 2 (1 Juni))
Iruse
where Iraas is the base learning rate, and the coefficient 2 normalizes the global scale to keep the effective average approximately unchanged. Units more important for general knowledge (higher Tunit) receive smaller learning rates to reduce overwriting, while less important units are encouraged to adapt more aggressively to domain-speciff data. Training proceeds with the standard autore- gressive objective: L = -' log P(z, Z<c; θ). Since the importance of units may change as training progresses,
periodically recompute Junit and update learning rates accordingly, ensuring dynamic adaptation throughout learning. The full training procedure is provided in Appendix I.. Appendix F.2 further shows that allocating learning rates inversely to unit importance minimizes an upper bound on general-domait forgetting. In essence, this design formalizes the intuition that highly general-critical units should be preserved via conservative updates, while less critical yet more adaptable ones can update more aggressively to absorb comain-specifer information.
EXPERIMENT
4.1 EXPERIMENTAL SETUP
Datasets. We evaluate ADEPT across two domains, Mathematics and Medicine. For the mathemati- cal domain, we use Open WebMath (Paster et al., 2023), together with AceReason-Math (Chen et al., 2025), concatenated into the continual pretraining corpora. For the medical domain, we adopt the multilingual MMedC corpus (Qiu et al., 2024), together with Industrylns and MMedBench, forming the medical pretraining corpora. Dataset statistics are provided in Appendix B. and Appendix B.2. In addition, for detecting
regions, we construct General Competence Detection Corpus, following the same setting as Section and described in Appendix B.3.
Baselines. We compare ADEPT with broad range of baselines from four perspectives:
Full- mallmparameter tuning. PT-Full directly updates all model parameters the target corpora. Replay-based tuning. Replay mitigates catastrophic forgetting by mixing general-domain data into the training process (Que al., 2024).

---

Published as a conference paper at ICLR 2026
Performance comparison across Mathematical and Medical domains. Bold numbers indi- cate the best performance, and underlined numbers denote the second best.
Table I:
Mathematics
General
Domain
General
Domain MMCU-Medical
MMLU
CMMLU
GSMSK
ARC-Easy
ARC-Challenge
MMLU CMMLU
MedQA
CMB
Qwen3 1.78 Base
Vanilla
57.62
PT-Fall
60.07
62.84
51.86
81.24
49.65
59.44
62.84
48.45
67.45
62.77
Replay
60.69
63.52
54.74
81.01
49.73
60.52
63.85
49.00
67.32
62.20
Llama-Pro
61.54
63.40
60.03
81.08
49.80
59.80
65.51
50.43
66.51
63.54
PT-LoRA
60.07
62.69
59.50
80.22
49.34
57.31
59.68
47.29
61.55
57.60
TaSL
60.34
62.95
59.07
79.76
48.89
62.48
66.14
47.06
67.62
61.15
ADEPT
52.62
50.75
Qwen3 Baxe
Vanilla
73.19
77.92
69.07
85.52
59.13
73.19
77.92
62.77
82.44
78.92
FT-Full
70.33
73.07
60.96
85.31
$7.59
69.48
72.77
62.84
81.34
76.88
Replay
Llama-Pro
72.42
77.39
73.16
85.14
57.76
72.28
77.28
62.53
81.20
78.12
PT-LoRA
70.20
72.90
71.34
84.18
$7.25
72.73
76.78
61.59
80.49
76.92
TaST.
73.03
60.99
79.20
ADEPT
73.21
78.30
76.19
88.44
60.98
72.95
78.77
64.49
84.58
79.87
Qwen3 8B Base
Vanilla
69.98
64.25
66.30
FTllll
74.90
78.49
80.21
85.90
61.77
74.06
78.82
67.24
87.69
85.27
Replay
75.19
78.92
81.12
85.98
62.37
74.51
78.86
68.89
86.66
84.73
Liama- Pro
76.16
81.42
80.97
86.62
63.91
76.58
81.69
66.77
87.19
83.76
PT-LoRA
82,87
86.36
62.46
76.60
67.01
86.70
TaS1.
76.63
80.37
80.54
84.81
$9.09
76.42
81.86
66.51
86.20
82.54
ADEPT
76.80
82.11
83.87
89.29
64.51
76.72
82.11
69.24
89.84
85.80
Llama3-88 8BB-ee
Vanilla
65.33
50.83
36.84
84.18
54.01
65.33
50.83
58.91
46.29
35.61
FT.Full
61.62
46.21
49.73
84.01
53,52
59.15
51.39
59.23
66.58
61.65
Replay
Llama-Pro
64.53
50.26
48.29
83.29
53,07
64.19
50.59
$9.94
$3.96
47.05
PT-L.RA
64.86
49.82
48.82
83.80
54.01
64.34
50.13
58.84
56.05
48.22
TaSL.
53.51
55.55
58.34
ADEPT
65.35
$1.90
50.57
84.96
55.52
65.17
$1.92
61.17
67.03
61.78
Architecture expansion. LLaMA-Pro (Wu et al., 2024b) expands the model by uniformly insert- ing new layers across the model, placing each new layer at fixed periodic intervals, while freezing the original weights. Only the newly introduced parameters are trained, enabling structural growth while preserving prior knowledge.
Parameter-efficient tuning. PT-LoRA performs CPT using Low-Rank Adaptation (Hu et al., 2022), updating only a small set of task-adaptive parameters. TaSL (Feng et al., 2024a) extends PT-LoRA multi-task regime by decoupling LoRA matrices across transformer layers, allowing different subsets of parameters to specialize for different tasks.
See Appendix B.6 for implementation details of all baselines.
Backbone Models. To assess the generality of our method, we instantiate ADEPT on multiple backbone models, including Qweni-Base (1.7B, 4B, 8B) (Yang et al., 2025) and ILaMA3.. /-8B- Base (Dubey et al., 2024b), covering wide range of parameter scales and architectural variants.
Evaluation Metrics and Strategy. We adopt multiple-choice question answering accuracy as the primary evaluation metric across all tasks (see Appendix B.9 for further details). For the Mathe- matics domain, we evaluate on GSM8K (Cobbe et al., 2021), ARC-Easy (Clark et al., 2018), and ARC-Challenge (Clark et al., 2018), which collectively span a wide range of reasoning difficul- ties. For the Medical domain, and CMB (Wang et al., 2023b), covering diverse medical subjects and varying levels of complexity. Among them, MedQA is an English benchmark, while MMCU-Medical and CMB in Chinese.
MedQA (Jin et al., 2021), MMCU-Medical (Zeng, 2023),

---

Published as a conference paper at ICLR 2026
Table 2: Ablation study on ADEPT in Medical domain. Bold numbers indicate the best perfor- mance, and underlined numbers denote the second best.
Method
Qwen3- B.ee33-L.BBBBa.. L.7B-Base
Llama3-SB-Base MedQA
MMLU
CMMLU
MedQA
MMCU Medical
CMB
MMLU
CMMLU
MMCU Medical
CMB
ADEPT
62.80
66.89
50.75
65.43
65.17
51.92
61.17
61.78
w/o Stage-1
$7.31
59.68
47.29
61.55
57.60
57.88
50.76
58.32
53.32
60.32
w/o Stage-2
61.56
64.33
49.23
64.34
59.60
Uniform Exponsion
59.80
65.51
50.43
66.51
63.54
64.19
50.59
59.94
47.05
53,96
To assess the model's ability to retain general-domain knowledge during continual pretraining, we additionally evaluate on MMLU (Hendrycks et al., 2020) and CMMLU (Li et al., 2023), two broad- coverage benchmarks for general knowledge and reasoning in English and Chinese, respectively.
4.2 EXPERIMENTAL. RESULTS
Performance Comparison. As shown in Table 1, ADEPT consistently outperforms all CPT base- lines across both mathematical and medical domains, confirming its effectiveness in domain-specific knowledge acquisition while substantially alleviating catastrophic forgetting. Concretely, ADEPT achieves substantial domain-specific improvements. Across all backbones and domain bench- marks, ADEPT consistently surpasses baselines, achieving the strongest performance. For instance, on Qwen3-1.78 1.7--Bas ADEPT boosts GSM8K accuracy from 57.62% to 70.51%1, bringing large gain that highlights its advantage enhancing LLMs' complex reasoning. Similarly, on LLaMA3- 8B-Buse, drastically improves CMB accuracy improves from 35.61% % to 61.78%f, underscor- ing the strong enhancement of medical-domanl capabilities. On average, ADEPT achieves up to 5.58% gains over full-pane parameter CPT on target-domain benchmarks, confirming its advantage in domain knowledge acquisition. Furthermore, ADEPT demonstrates clear ad advantages in miti- gating catastrophic forgetting. Whereas most baselines suffer noticeable degradation on general benchmarks such as MMLU and CMMLU, ADEPT preserves the pretrained LLMs' general-domain competencies, and in some cases even surpasses the vanilla backbone. Notably, with Qwen3-4B un- der medical CPT, ADEPT improves CMMLU accuracy from 77.92% to 78.77%.. It also results in an average performance increase of 5.76% % on general benchmarks over -all-paaeeeter CPT. We at- and general parameters, which prevents harm-
tribute this to the disentanglement of domain-spectieo ful representational interference during adaptation, ensuring that learning specialized knowledge does not corrupt the model's foundational abilities. Instead, this focused learning process appears to refine the model's overall competencies, leading to synergistic improvements on general-domain tasks. In summary, ADEPT offers a robust solution for CPT achieving superior domain adaptation while effectively preserving general knowledge.
Ablation Study. To investigate the effectiveness of each component in ADEPT, we conduct ablation experiments in the medical domain using two representative backbones, Qwen3-1.78 and Llama3- 8B. In wo Stage-l, we remove the directly apply Adaptive Unit-Wise Decoupled Tuning the l Domain-Adaptable Layers without introducing any new parameters. In w/o Stage-2, we discard the dynamic decoupled tuning stage and instead directly fine-tune the expanded layers from Stage- In Uniform Expansion, we replace importance-guided expansion with uniformly inserting layers at fixed periodic intervals followed by fine-tuning, which is equivalent removing either Stage-I or Stage-2 leads to clear degradation in both general and Nomain-sseecing performance, confirming that both adaptive expansion and decoupled tuning are indispensable. In particular, eliminating Stage-/ results in the largest performance drop, suggesting that adaptive capacity allocation is crucial for enabling effective domain adaptation without sacrificing general- domain competencies. Meanwhile, replacing importance-guiie expansion with uniform expansion yields inferior results, underscoring the advantage of expanding only the most addatneee layers.
Guided Selective Layer Expansion and
the strategy adopted in LLaMA-Pro. As shown in Table 2,
Decoupling Effectiveness on Expanded Parameters. We visualize cross- coss-domain activations using Kernel Density Estimation (KDE) (Silverman, 2018), sampling 500 instances from both Medical and General corpora. For the original @wen3-8B- Base (left in Figure 4), the most domain- adaptable layer (lowest /nyer) still shows heavy overlap between general and medical activations, evidencing
8

---

Published as a conference paper at ICLR 2026
Vanilla
W/o stage1
ADEPT
100
75-
Ganaral Teo
Nedica Wext
General Bed
Nedical
60
General Teat
40-
40-
25
20-
20
dmn
-25
-75-
-00
-100
50
dim
Figure 4: Activation distribution analysis of Qwen3-8B.
ccases
zed
med
cancen ennerr
earl
Lental
healthiog
research
cations
ca
MD
ICE
doctor
Omedical
care
thritis
LOS
reséargher medication
News
Hospi
best t
S e a
flealth
S
Drug
Paraas ways role
students
user pappppaaa
I ob a phenatiee space arrtance ma scatter
effects ex
et scripts
mathematic
sth
physics
new
quant
PhD
graups
bl
the
code
sall
question
ATEEF FT
he
Rig
error
profit
l cchner de
open
framer es
domen
t trea
aheer
rment
heal
Eny
pa
t
eeen
CM
uule
ymptoms call
lot
hospital
da a di diagoosed
diabetes
therapy
RNA
drigs
nang
ižation
heart
Protein
algorithm
upobbl em bit
talk
sense
ja fo
ed tor
reaction Honetty
Following to
pton
ormal
resium
$$
mental
Medi
ynec
tumor
Hea 1 t h
panients
seasses
presc cription roffect
vaccine
stay
brain
drug
results
study
MRI
udds us answer
Oct
connected
weswer
mode
Dr
eviouus
parameters
theoren
quantum
papers
issaiinn
results
AG
(a) Token distributions shift in Medical
(b) Token distributions shift in Mathematical
Figure 5: Token distribution shifts across domains. Word cloud visualizations of shifted tokens reveal that ADEPT achieves highly focused alignment, with most changes concentrated on domain- specific terminology.
strong parameter coupling. Direct decoupling without expansion (w/o Stage-1, middle) on the same layer fails to reduce this entanglement, confirming that pretrained parameters are inherently diffi- cult to separate. In contrast, after expansion (right), the duplicated layers serve as a "blank slate," yielding clearly separated activations across domains. Additional analyses on more backbones are provided in Appendix C.I, where we observe that this trend consistently holds across nearly all evaluated LLMs, further validating the generality of our approach.
Token Distribution Shift Analysis. To assess how ADEPT injects domain knowledge while pre- serving general competencies, we analyze token-level shifts between the base and continually pre- trained models. Following Lin et al. (2024), tokens are categorized unshifted, marginal, or shifted. Only small proportion of tokens shift, while most remain unchanged, indicating stable adaptation. In the medical domain, merely 2.18% shift (vs. 5.61% 1% under full pretraining), largely medical terms such as "prescription," "diagnosis." and "therapy" (Figure 5a). In the mathematical domain, only 1.24% shift, mainly scientific terms such as "theorem" and "equation" (Figure 5b). Further details and analyses are provided in Appendix I. These results demonstrate that ADEPT achieves precise and economical domain knowledge injection while minimizing perturbation to general competence.
Extended Investigations and Key Insights. We further investigate several design choices of ADEPT in appendix: Appendix E, we investigate alternative strategies for probing layer im- portance and observe the consistency of different measurement methods, offering insight into how importance estimation affects adaptation outcomes. Appendix G explores the effect of expanding different numbers of layers and reveals how the number of expansion layers should be selected under different cireumstances and the potential reasons behind this. Appendix H shows that even with relatively low-quality importance detection corpus from pretrain data, our approach maintains strong generalization across domains, suggesting the robustness ADEPT. Appendix demon- strates insights into the potential for merging expanded layers that independently trained on
9

---

Published as a conference paper at ICLR 2026
different domains, offering intriguing direction for achieving multi-domain adaptation with min- imal catastrophic forgetting. In addition, Appendix B.8 analyzes the training efficiency of ADEPT, showing that our selective updating design substantially accelerates convergence compared to base- lines. In addition to the core evaluation, we conduct a comprehensive set of extended analyses to further validate the robustness, generality, and adaptability of ADEPT. In Appendix M, present a sensitivity analysis of the importance-score update intervals, demonstrating that ADEPT is stable across a wide range of update frequencies, with only marginal performance variation. Appendix N investigates the applicability of ADEPT to supervised fine- fine-tuning settings, showing consistent gains over standard fine-tuning baselines without requiring architectural changes. To assess generalization beyond our primary benchmarks, Appendix O includes extended evaluations additional domains and datasets, where ADEPT continues to outperform strong baselines. In Appendix P, we eval- uate ADEPT specifically on code-domain tasks, confirming its effectiveness in structured, logic- intensive environments. Appendix Q further extends our evaluation to multilingual medical bench- marks, highlighting ADEPT's cross-lingual transfer capability in more challenging domain adaption setting. Finally, Appendix R addresses a key design question: whether to expand domain cortinncritical layers or general-noneritical ones. Appendix S investigated the impact of different inttiiiiiaaiznnionnn strategies. Our analysis reveals that selectively expanding domain-critical layers yields significantly higher domain performance but more forgetting, providing actionable guidance for layer selection in domain adaptation scenarios.
5 CONCLUSIONS AND FUTURE WORKS
We present ADEPT, framework for L.M continual pretraining for domain adaptation that effec- tively tackles catastrophic forgetting, leveraging functional specialization in LLMs. By selectively expanding layers less critical to the general domain and adaptively updating decoupled parameter units, ADEPT minimizes catastrophic forgetting while efficiently incorporating domain-specific ex- pertise. Our experiments show significant improvements both domain performance and general knowledge retention compared to baselines. Future work could focus on refining the decoupled tuning mechanism, designing more sophisticated learning rate strategies beyond linear mapping to allow for more precise adjustments. Another direction is explore better dynamic and real-time methods for measuring parameter importance during training.
6 ETHICS STATEMENT
All datasets used for training and evaluation in this study are publicly available versions obtained from the Hugging Face platform. The datasets have been eurated, cleaned, and de-identified by their respective data providers prior to release. No patient personal information or identifiable medical data present. Consequently, the research does not involve human subjects, and there are no related concerns regarding privacy, confidentiality, or legal liability. And for full transparency, we report all aspects of large language model (LLM) involvement in the Appendix K.
We strictly adhered to the usage and redistribution licenses provided by the original dataset authors and hosting platforms. Our research poses no risk of harm individuals or groups and does not contain any potentially harmful insights, models, or applications. Additionally, there are no conflicts of interest or sponsorship concerns associated with this work. We are committed to research integrity and ethical standards consistent with the ICLR Code of Ethics.
7 REPRODUCIBILITY STATEMENT
We actively support the spirit of openness and reproducibility advocated by ICLR. To ensure the reproducibility of our research, we have taken the following measures:
1. Disclosure of Base Models: All base models used in our experiments are explicitly identified and described in the main text. This allows readers directly reference and obtain these models.
2. Datasets and Experimental Details: All experiments are conducted on publicly available datasets from the Hugging Face platform. In Appendix B, we provide a comprehensive description of our experimental implementation, including dataset sources, browser links, and detailed data process- ing procedures. We also detail the experimental setup, such training duration, hardware envi-
10

---

Published as a conference paper at ICLR 2026
romment (e.g.. GPU type), and configuration of hyperparameters, including rank, number of extended layers, batch_size and mex_hhngth. These details facilitate transparent verification and replication of our results.
Open-Source Source Code Release:
To further support reproducibility, we release all training and eval- uation code in a public repository (https: /tt:://gtttt Som//uppyKnna Un ivrrsessl serrsstyyy y ADEP T). The repository contains clear instructions on installation, data downloading, prepro- cessing, and experimentation, allowing interested researchers to replicate our results with mini- mal effort.
We believe that these actions align with the open science principles championed by the ICLR com- munity, and we are committed supporting the reproducibility and transparency of our work.
ACKNOWLEDGMENTS
This research was funded by the Independent Research Project of the National Key Laboratory of Big Data and Decision.
REFERENCES
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.). Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 7319-7328. URL
Association for Computational Linguits Linguistics, 2021. doi: 10. 568.
https: /oii www.ddi.ooooggieee www.oorgg....ooo...o 18653/66331188 wwwp////.........
Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pp. 225-237, 2024.
Iftaah Arbel, Yehonathan Refael, and Ofir Lindenbaum. Transformllm: Adapting large language models via llnttransformed reading comprehension text. arXiv preprint arXiv:2410.2 2024.
Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski. David Dohan, Ellen Jiang. Carrie Cai, Michael Terry. Quoe Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2 07732, 2021.
Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. Huatuogpt-ol, towards medical complex reasoning with llms. arXiv preprint arXiv:2412.18925 2024.
Mark Chen. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Ner2net: Accelerating learning via knowledge transfer. arXiv preprint arXiv: 10:::5511.0000 1.05641, 2015.
Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catan- Advancing math and code reasoning through rein-
zaro, and Wei Ping. forcement learning. arXiv preprint arXiv:2505.16400, 2025.
Daixuan Cheng, Shaohan Huang, and Furu Wei. Adapting large language models via reading com- prehension. In The Twelfth International Conference on Learning Representations, 2023.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? analysis of bert's 's attention. In Tal Linzen, Grzegorz Chrupala, Yonatan Belinkov, and Dieuwke Hupkes (eds.), Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP,, BluckboxNI.@AACI.. 2019, Florence, Italy, August 1, 2019,

---

Published as a conference paper at ICLR 2026
pp. 276-286 Association for Computational Linguistics, 2019. doi: 1//65//////199443033
URL https: htop:s///dii wwg./o.ggoooooooooooooooo 101.00116633///11e/.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. 2018.
arXiv preprint
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10, 14168, 2021.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 1993-8502. Association for Computational Linguistics, 2022a. doi:
URL https:
//doi. org/ 10. 1ee.1186533118 ww20222/222222222/// acl long. 581.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume Long Papers), pp.
2022b.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8493 8502, 2022c.
Hongxin Ding, Yue Fang, Runchuan Zhu, Xinke Jiang, Jinyang Zhang, Yongxin Xu, Xu Chu, Jun- feng Zhao, and Yasha Wang. 3ds: Decomposed difficulty data selection's 's case study on llm medical domain adaptation. arXiv preprint arXiv:2410. 10900, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux. Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail, Grégoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Ko- revaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes. Jason Park, Jay Ma- hadeokar, Jeet Shah, Jeimer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jong- soo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The llama 3 herd of models. CoRR, 0772477777788. 2024a. doi: 1044550AARII.2407..11 https: /ttp:://doii http://ti..ogg...//// www.oogg.............oooo.... 48550/ 200.0/2......22..1.... 217833.
URL
Abhimanyu Dubey, Abhinav Jaubri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024b.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024c.
Yujie Feng, Xu Chu, Yongxin Xu, Guangyuan Shi, Bo Liu, and Xiao-Ming Wu. Tasl: Continual di- alog state tracking via task skill localization and consolidation. arXiv preprint 2xXiv:2408.098. 09857, 2024a.
12

---

Published as a conference paper at ICLR 2026
Yujie Feng, Xu Chu, Yongxin Xu, Guangyuan Shi, Bo Liu, and Xiao-Ming Wu. Tasl: Continual dialog state tracking via task skill localization and consolidation. In Lun- Wei Ku, Andre Mar- tins, and Vivek Srikumar (eds.), Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACI. 2024, Bangkok, Thailand, August I I- 16, 2024, pp. 2266-1279 Association for Computational I Linguisties, 2024b. doi: 10.18653/V1/ 2024.CCL-LONG.66 URL https: /ooii http://oi.ooggg//g///////o org/10.....oooooo..ooooo
acl1///...111111111......o long 69.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484-5495, 2021a.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward lay- ers are key-value memories. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 5484-5495, Association for Computational Linguistics, 2021b. doi: 11w.:/...111vvvvv 10. N0.18653/11/2221.EMMM WWW.EEENNNLL...EEE 446. URL htttps://doot http:::/oo...rrrg/...8855////// emr.pp main. 446.
Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009. 03300, 2020.
John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word repre- sentations. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, NAACL-IILT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4129 4134.118..4. Association for Computational Linguistics, 2019. doi: 14199
10.18653/V 1/N19-1419 URL https: / /ooi org/
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1,2):3, 2022.
Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, and Jinsong Su. Mitigating catastrophic forgetting in large language models with self senfhsyythesizedd rehearsal. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp.
2024.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi- level multi-discipline chinese evaluation suite for foundation models. In Advances in Neural Information Processing Systems, 2023.
Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats L Richter, Quentin Anthony, Timothée Lesort, Eugene Belilovsky, and Irina Rish. Simple and scalable strategies to continually pre-train large language models. arXiv preprint orXiv:2403.08733 2024.
Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 8580-8580 2018. URL https:/ /tttlp::/prroeeeee http:///ooceedeigee/ee 5a 4be fa34 6ec -aatt html.
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What dis- ease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, (114)):421, 2021.
13

---

Published as a conference paper at ICLR 2026
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: dataset for biomedical research question answering. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference natural language processing (EMNLP-IJCNIPP) pp. 2567-2577, 2019.
Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Contin- ual pre-training of language models. The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL net/ fow.mmiddn_GGDi GDIItaII..
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Bariskkkaaaiik al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13)335213335 2017.
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timo- thy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.092: 09212, 2023.
Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. Gsm-plus: A comprehensive benchmark for evaluating the robustness of Ilms as mathematical problem solvers. arXiv preprint 290.:2402.//255. 2024.
Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Raghavi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base umss: Rethink- ing alignment via in-context learning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: /tpp////eee/eeeeeeeeeeeeeeeeeeee net/ forum? wowrmmiddwwxuee wxdewddeewwdwwwee
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pp. 3214-3252, 2022.
Chengyuan Liu, Yangyang Kang, Shihang Wang, Lizhi Qing, Fubang Zhao, Chao Wu, Changlong Sun, Kun Kuang, and Fei Wu. More than catastrophic forgetting: Integrating general capabili- ties for domain-specific LLMs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 7531-7548, Miami, Florida, USA. November 2024a. Association for Computational Linguis- tics. doi: 10. emn.p-ma inn 429/.
www.ememlllmmm....mm main. 429. URL https: / wwwaannnahllooooooo org/ 2024.
Shih- Yang Liu, Chien-Yr Yi Wang. Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In Interna- tional Conference on Machine Learning, pp. 32100-32121. PMLR, 2024b.
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting large language models during continual fine-tuning. IEEE Transactions on Audio, Speech and Language Processing, 2025.
Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associ- ations in GPT. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural formation P Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 De- cember 9, 2022, 2022. URL http: httpp/aaaassseeeee pips. ce http:/pperrfeee hwt.ffllsspppeeeeeeeeeeeemeeee/e pppppp/pppr///////////// -bsttrattt-t----a.eee Conference htmi.
Eshaan Nichani, Jason D. Lee, and Alberto Bietti. Understanding factual recall in transform- ers via associative memories. In The Thirteenth International Conference on Learning Repre- sentations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https: //opezevieww net. fow.mmiddwwwwmmra..
OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
14

---

Published as a conference paper at ICLR 2026
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text. arXiv preprint arXiv:2310.06786, 2023.
Ru Peng, Kexin Yang, Yawen Zeng, Junyang Lin, Dayiheng Liu, and Junbo Zhao. Dataman: Data manager for pre-training large language models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025.
URL httph::t////ccm. htt.pp:://ppnneeiieeee oppp/eeeneeeeeeeeeeeeeeeeee net wtt/neeunninneeee id eNbA q.14.
Pengcheng Qiu, Chaoyi Wu, Xiaoman Zhang, Weixiong Lin, Haicheng Wang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards building multilingual language model for medicine. Nature Communications, 15(1):8384, 2024.
Haoran Que, Jiaheng Liu, Ge Zhang, Chenchen Zhang, Xingwei Qu, Yinghao Ma. Feiyu Duan, Zhiqi Bai, Jiakai Wang, Yuanxing Zhang, et al. D-cpt law: Domain-specific continual pre-training scaling law for large language models. Advances in Neural Information Processing Systems, 37: 90318-90354, 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&e bench- mark. First Conference on Language Modeling, 2024.
Bernard W Silverman. Density estimation for statistics and data analysis. Routledge, 2018.
Yifan Song, Peiyi Wang, Weimin Xiong, Dawei Zhu, Tianyu Liu, Zhifang Sui, and Sujian Li. Infocl: Alleviating catastrophic forgetting in continual text classification from information theoretic perspective. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp- 14557-14570, 2023.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, et al. Challenging big- big-bench tasks and whether chain-of- of-th-uhtth can solve them. In Findings of the Association for Computational Lin- guistics: ACL. 2023, pp. 13003-13051, 2023.
Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuan-Jing Huang. Orthogonal subspace learning for language model continual learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 10658-10611 2023a.
Xidong Wang, Guiming Hardy Chen, Dingjie Song, Zhiyi Zhang, Zhihong Chen, Qingying Xiao, Feng Jiang, Jianquan Li, Xiang Wan, Benyou Wang, et al. Cmb: A comprehensive medical benchmark in chinese. arXiv preprint 2308 000080888,,....8 2023b.
C Wu, W Lin, X Zhang, Y Zhang, W Xie, and Y Wang. Pmc-Ilama: toward building open-source language models for medicine. Journal of the American Medical Informatics Association: JAMIA, pp. ocae(45-ocae045 2024a.
Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ying Shan, and Ping Luo. Llama pro: Progressive Ilama with block expansion. arXiv preprint arXiv:2401.02415. 2024b.
Weimin Xiong, Yifan Song, Peiyi Wang, and Sujian Li. Rationale-enhanced language models are better continual relation learners. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 15489-15497, 2023.
Yongxin Xu, Ruizhe Zhang, Xinke Jiang, Yujie Feng. Yuzhen Xiao, Xinyu Ma, Runchuan Zhu, Xu Chu, Junfeng Zhao, and Yasha Wang. Parenting: Optimizing knowledge selection of retrieval- augmented language models with parameter decoupling and tailored tuning. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 11611-1111111 1662, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-44 doi: 00.18653/v1/2022 http///2222..21111mmaaaaaaa acl-long.571. URL https: / /aclanthology. org/2025, acl long.
15

---

Published as a conference paper at ICLR 2026
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 00Xiv:2500500388, 2025.
Zitong Yang, Neil Band, Shuangping Li, Emmanuel Candes, and Tatsunori Hashimoto. Synthetic continued pretraining. arXiv preprint arXiv:2409.07433 249900743,, 2024.
Yunzhi Yao, Ningyu Zhang, Zekun Xi, Mengru Wang. Ziwen Xu, Shumin Deng, and Huajun Chen. Knowledge circuits in pretrained transformers. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, UIrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Anmual Conference on Neural In- formation Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Camada, December 10 15, 2024, 2024. URL http: htwpaapppss.peaaa nips. cc/paper_fatteeceecceee files/ wwweee/pappr//0// 2024 /hash/ wew.eeiibbbeeeebeeee..... bb wtw.eebbbbbbbb----aattttte Abstract Conferenceeeeeeeeeeooooeeoooeene -aatr----ebtaaatt--oereeeeeeeee html.
Çağatay Yıldız, Nishaanth Kanna Ravichandran, Nitin Sharma, Matthias Bethge, and Beyza Ermis. Investigating continual pretraining in large language models: Insights and implications. arXiv
preprint arXiv:2402. 170.000000. 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint 1rXiv:1905.074 2019.
Hui Zeng. Measuring massive multitask chinese understanding. arXiv preprint arXiv:2004119966 2023.
Yunan Zhang, Shuoran Jiang, Mengchen Zhao, Yuefeng Li, Yang Fan, Xiangping Wu, and Qingcai Chen. Gere: Towards efficient anri-forggt in continual learning of Ilm via general samples replay, arXiv preprint arXiv:2508. 04667, 2025.
Zifan Zheng, Yezhaohui Wang, Yuxin Huang, Shichao Song, Mingchuan Yang, Bo Tang. Feiyu Xiong, and Zhiyu Li. Attention heads of large language models: A survey. arXiv preprint arXiv:2409.03752. 2024.
A RELATED WORK
A.1 CONTINUAL. PRETRAINING FOR LLMS
Continual pretraining updates pretrained LLMs with new corpora to equip them with new knowl- edge and capabilities. Data-centric approaches adopt data replay to mitigate catastrophic forget- ting (Huang et al., 2024; Zhang et al., 2025; Xiong et al., 2023; Song et al., 2023), or utilize data construction strategies synthesize training corpora (Yang et al., 2024; Arbel al., 2024). How- ever, these methods make no changes to the model or training procedure, failing to effective inject new knowledge due to capacity saturation and only partially alleviating forgetting. Another line of works focus adjusting model architecture and training strategy. LoRA (Hu et al., 2022) improve efficiency for fine-tuning by adapting low-rank updates on top of frozen backbones, but their lim- ited adjustments to LLMs can not effectively address continual pretraining for deep domain adap- tation. LLaMA-Pro (Wu et al., 2024b) expands model blocks and tunes the added parameters on new corpora, improving knowledge injection and mitigating forgetting compared to vanilla CPT. Yet existing expansion policies insert layers uniformly across depths and treat all expanded parame- ters indiscriminately during optimization, leaving open how to place capacity where domain signals concentrate and update it without disturbing general knowledge. Classical continual-learning regu- larizers (Kirkpatrick al., 2017) constrain updates on weights deemed important to previous tasks, but they do not guide where capacity allocation nor how to target LLM domain adaptation learning.
A.2 FUNCTIONAL SPECIALIZATION IN LLMS
Growing evidence indicates that, akin to human brains, LLMs exhibit functional specialization, where different regions such as layers, attention heads and neurons play distinct roles. A series of causal and studies show that factual knowledge are predominantly stored in FFN layers (Dai et al., 2022c), and attention heads usually play specialized roles for certain functions (Zheng al., 2024),
16

---

Published as a conference paper at ICLR 2026
suggesting that knowledge and skills are unevenly distributed in LLMs. Inspired by this special- ization, several methods have tried to decouple functional modules during training. For instance, Parenting (Xu et al., 2025) separates the subspaces responsible for evidence-followiee robustness in retrieval-augee retrieeal-uggmented generation, and optimizes them with tailored objectives to im- prove performance under noisy retrieval. Similarly, TaSI. (Feng al., 2024a) addresses multi-task adaptation by disentangling LoRA parameters from different tasks and merging them in a weighted manner, which helps reduce interference. Other works on orthogonal (Wang al., 2023a) decom- posed LoRA (Liu et al., 2024b) further reflects the idea that training different parameter subspaces separately improves robustness and transfer. Despite these advances, prior work does not address CPT, where the tension between knowledge injection and retention needs to be tackled. To our knowledge, our work the first to explicitly leverage functional specialization during CPT to simul- taneously improve domain performance and alleviate catastrophic forgetting.
and noise-
B DATA RECIPE AND EXPERIMENT SETTINGS
To demonstrate the applicability and generalizability of our approach, we conducted domain- adaptive continual pretraining experiments on two distinct and highly significant domains: Mathe- matics and Medicine, both of which play crucial roles in the advancement of artificial intelligence and the applications of LLM. The mathematical domain often poses challenges that emphasize a model's 's reasoning and computational abilities, while the medical domain predominantly requires deep understanding and memorization of medical concepts. From a cognitive perspective, we be- lieve that the capabilities that need to be infused into the model differ significantly between these two domains, which further demonstrates the generalisability of our approach.
The continual pretraining process leverages both pretraining datasets for foundational knowledge and supervised fine-tuning (SFT) datasets for task-speeific optimization (Cheng et al., 2023). Below, detail the data composition and processing details. All data used will be processed into the format of pre-training data.
B. MEDICAL PRETRAIN DATA SOURCE
Our medicine datasets are divided into pre-training data, designed to provide extensive general knowledge, and supervised fine-tuning (SFT) data, which refine the model's 's understanding for spe- cific instructions in the medicine domain (will be converted to pretrain data format when training).
Pre-training data: we utilize English and Chinese portions of MMedC dataset, multilingual medical dataset, furnishing total of 14.3 billion tokens.
Instrution tuning data: we incorporate two supervised datasets:
1. IndustryIns, contributing 1.6 billion tokens from instruction- based examples
2. MMedBench, with 18 million tokens focused on medical reasoning tasks.
Table Overview of medicine Datasets. This table summarizes medicine-specifte pre-training and SFT datasets, including their language coverage, dataset links, and used token counts. For MMedC, we only use the English and Chinese parts and we only use the Tealth-Medicne subset.
Dataset Name
Dataset Type
Language
Dataset Link
#Token Used
MMedC
Pre-training
Multilingual
Henrychur/MMedC
14.3B
IndustryIns
SFT
Chinese and English Chinese and English
BAA//IndsstyyInssmaiiin
1.6B
MMedBench
SFT
Honycchr/MMcdBenchh
18M
B.2 MATHEMATICS PRETRAIN DATA SOURCE
Mathematics pretrain datasets include both pre-training and fine-tuning data (will be converted to pretrain data format when training), structured similarly the medicine datasets.
17

---

Published as a conference paper at ICLR 2026
Pre-training data: use the Open-Wes Web-Math (Paster et al., 2023) dataset, containing diverse set of general mathematics knowledge amounting to 14.7 billion tokens.
For Instruction-tuning data: we use the Aceteason-Math (Chen et al., 2025), contributing 102 million tokens, with strong emphasis on shain-of-thought reasoning and problem-solving.
Overview of Mathematics Datasets. This table includes the pre-training and SFT datasets for mathematical reasoning, highlighting their contents, links, and used token counts.
Table 4:
Dataset Name
Dataset Type
Language
Dataset Link
Used Token
Open- Wwen-Web-Matth
Pre-training
English
14.7B
AceReason- Math
SFT
English
mvidia/AceReeson-Maill
102M
B.3 GENERAL COMPETENCE DETECTION CORPUS
To accurately probe which parameters critical for preserving general knowledge during contin- ual pretraining, we construct a General Importance Detection Corpus. This corpus designed to capture both broad world knowledge and instruction-following capability in English and Chinese. Specifically, we include the development splits of two widely recognized multi-task benchmarks, MMLU_dev and CMMLU_dev to capture general knowledge without data leakage.
MMLU and CMMLU are formatted as multiple-choice question answering tasks with explicit prompts and ground-truth answers. For these, we compute gradient-based importance only on the target answer tokens to avoid biases from prompt formatting, thereby capturing each parameter group's contribution accuracy.
To clarify how gradient signals are obtained, we illustrate two examples. In SFT-style corpora (e.g., MMLU, CMMLU), only the ground-truth answer token contributes to gradient computation, ensuring clean signals for decision-making importance. In P'T-style corpora (e.g., Fin Edu), all tokens contribute under the causal LM objective, providing dense gradients that reflect general modeling capacity, Examples are shown in Example and Example 2.
Table 5: General Competence Detection Corpus. #Examples means the number of examples used.
Dataset
Language
#Examples
Hugging Face Link
dev CMMLU.de
English
285
cais/mmlu haonan-licc -ineam-
Chinese
295
The statistics the selected datasets are summarized in Table 5.
Example
Gradient Flow in SFT Data for Importance Estimation
Input Prompt:
Question: Find all E Z3 such that Zg(z)/(22 + +)) is field.
A. 0 B. C. D.3
Answer:
B
Explanation:
In this SFT setup, only the target answer token (e.g., B) is used to compute gradients for pa- rameter importance. The input question and options are excluded from gradient computation to avoid encoding biases from instruction formatting. By focusing gradient signals solely on the correct answer token, we measure how each parameter contributes to decision-making accuracy under structured knowledge tasks, while preventing overfitting to input patterns and ensuring clean separation between training and probing data.
18

---

Published as a conference paper at ICLR 2026
Example
Gradient Flow in PT Data for Importance Estimation
Context (Compute Gradient):
The heart is a muscular organ responsible for pumping blood throughout the body. It con- sists of four chambers: the left and right atria, and the left and right ventricles. Oxygen-poor blood enters the right atrium, then flows to the right ventricle, which pumps to the lungs. After oxygenation, blood returns to the left atrium, moves to the left ventricle, and is finally pumped into the aorta for systemic circulation. This process regulated by electrical sig- nals originating in the sinoatrial node. These signals ensure synchronized contraction and efficient blood flow.
Explanation:
In PT-style training, parameter importance is computed using causal language modeling loss across the entire sequence. Every token - both context and continuation contributes to the gradient signal. This captures how parameters support general language modeling over natural text distributions. Unlike SFT, there is no explicit input/oupuut separation; stead, each token is predicted from its prefix, making the gradient flow dense and continuous. This allows us to assess parameter sensitivity in open-ended, nomain--elevant pre-training scenarios such those provided by Fineweb.Ecd Edu..
B.4 DATA PROCESSING
To generate training corpus in pretrain format, SFT data is structured by concatenating questions, Shain-of-thoughh (CoT) reasoning, and final answers for each instance. This ensures that the model is optimized for multi-step reasoning tasks common in medicine applications. We take Example as an example.
Example
On Liar Island, half the people lie only on Wednesday, Friday, and Saturday, while the other half lie only on Tuesday, Thursday, and Sunday. One day, everyone on the island "I will tell the truth tomorrow." What day it? (2021 Xin Xiwang Bei Competition, Grade 2, Preliminary Math Exam)
Problem:
says:
Analysis: We examine the truth-telling patterns over the week:
First group (lies on Wed, Fri, Sat): Truth pattern across days: True, True, False, True, False, False, True.
Second group (lies on Tue, Thu, Sun): Truth pattern: True, False, True, False, True, True, False.
Now evaluate each option:
Option A (Tuesday): If today Tuesday, the first group tells the truth today, their statement "I will tell the truth tomorrow" implies they should tell the truth on Wednesday. But they lie on Wednesday contradiction. The second group lies today, so their statement is false, meaning they will not tell the truth tomorrow (i.e., lie on Wednesday). But they actually tell the truth on Wednesday - also contradiction. So A is invalid.
Option B (Wednesday): First group lies today; their statement is false + they will nor tell the truth tomorrow (i.e., lie on Thursday). But they tell the truth on Thursday - contradic- tion. Second group tells the truth today - they should tell the truth on Thursday. But they lie on Thursday - contradiction. So invalid.
Option C (Friday): First group lies today - statement is false they will not tell the truth tomorrow (i.c., lie on Saturday). They do lie on Saturday - consistent. Second group tells the truth today - they will tell the truth Saturday. They do tell the truth Saturday consistent. So is correct.
Option D (Saturday): First group lies today should lie Sunday. But they tell the truth on Sunday - contradiction. Second group tells the truth today - should tell the truth on Sunday. But they lie on Sunday contradiction. So D is invalid.
19

---

Published as a conference paper at ICLR 2026
Option E (Sunday): First group tells the truth today - should tell the truth on Monday. They do - consistent. Second group lies today + their statement is false + they will not tell the truth on Monday (i.e., lie). But they tell the truth on Monday Monday-conteee contradiction. So is invalid.
Therefore, the correct answer is (Friday).
[This example demonstrutes how structured SFT data - consisting of a standalone problem (in blue), detailed step-by-step analysis fin green) and short unswer (in red) is concatenated into a single coherent narrative. In PT-style training, such concatenation enables models to learn implicit reasoning patterns from natural language flow, bridging supervised fine-tuning signals with pre-training objectives.]
To handle input sequences that exceed the maximum context length of 4096 tokens imposed by transformer-basee models, we apply a sliding window segmentation strategy with overlap. following the approach used in DATAMAN (Peng et al., 2025). For any sequence longer than 4096 tokens, we split it into multiple segments, each of length most 4096, using a sliding window with stride of 3072 tokens and an overlap of 1024 tokens (i.e., 1/4 of the window size). This ensures that consecutive segments share contextual information when training in the same or adjacent batches, preserving semantic continuity and high data utilization rate across boundaries.
Formally, given a token sequence D (2,, Lr.] of length L > 4096, we generate K L-1024 segments. The k-tth segment is defined as S D[e rk], where & = (k 1) -07230722 + 3072
and т min(6, + 4097, L). The overlapping region between Sş and S consists of the last 1024 tokens of Sg, which are identical to the first 1024 tokens of S
This method prevents information loss due truncation and allows the model to learn from contin- uous context during training. The 1024-token overlap helps maintain coherence at segment bound- aries, which is crucial for tasks requiring long-range understanding, while keeping computational overhead manageable.
B.5 FINAL DATA ORGANIZATION SCHEME
Our final training data is organized as follows:
1. English pre-training corpus
2. Chinese pre-training corpus (if have)
3. English supervised fine-tuning (SFT) corpus
4. Chinese SFT corpus (if have)
This organization is motivated by several key points in Qwen3 Technical Report (Yang et al., 2025) and Llama3 Technical Report (Dubey et al., 2024a). First, we follow the principle that high-quality data (SFT data in our work) should be used after extensive pre- nee-training large-scale general cor- pora, allowing the model to first acquire broad knowledge and language structure, and then specialize on more curated tasks and instructions.
What's more, according to the technical reports, it is further beneficial to place the same language's data together during training- this maximizes the coherence within each mini-batch and reduces unintended cross-lingual transfer until later stages. Most LLMs are dominated by English corpora in their pre-training phase, supporting the choice of placing English data first. Finally, during later training stages, continued training and decay are performed on SFT examples, which aligns with established recipes for improving supervised task performance.
B.6 COMPARED METHODS.
hull-parameter tuning. PT-Full directly updates all model parameters on the target corpus, serv- ing as the most straightforward yet commonly used baseline for continual pretraining. Replay-based tuning. Replay mitigates catastrophic forgetting by mixing general-domain data into the continual pretraining process (Que et al., 2024), thereby preserving part of the original knowledge distribution while adapting to the domain. Following (Zhang et al., 2025), based
20

---

Published as a conference paper at ICLR 2026
the data from Data Recipe, randomly sampled totally 1.91B data from FinewebEdu and Chinese at a ratio of 7:3, and randomly shuffled them into the domain- -seaiifc data, helping the model better recall general domain knowledge.
Architecture expansion. L.MoMPPro (Wu et al., 2024b) expands the model by uniformly insert- ing new layers into each transformer block while freezing the original weights. Only the newly introduced parameters are trained, enabling structural growth while preserving prior knowledge. Parameter-efficient tuning. PT-LoRA performs continual pretraining using Low-Rank Adapta- tion (Hu et al., 2022), updating only small set of task- aasktidaetvve parameters. TaSL (Feng et al., 2024a) extends PT-LoRA to multi-task regime by decoupling LoRA matrices across transformer layers, allowing different subsets of parameters to specialize for different tasks. This enables more fine-grained adaptation to domain-specific signals. We used the DEV sets MMLU and CMMLU to assess general capabilities, and their mathematics and medical subsets to specifically evaluate mathematical and medical competencies, respectively. Taking the medical domain as an example, we treat the original model as one equipped with LoRA module initialized to all zeros. The final LoRA module is then obtained by merging the domain-specifie LoRA with the original (empty) LoRA using TaSL.
B.7 EXPERIMENTAL IMPLEMENTATION.
We conduct our all pre-training experiments on the Qwen3-1. 1 ---BBee/QQee334B-- Baee4----aee/wen3-B81 Base/Llama3-88 8B model with the following hyperparameter configuration. Training is performed for
3 epochs using a batch size of 512 (8 NVIDIA H800 GPUs) and a maximum sequence length of 4096 tokens. We utilize cosine learning rate scheduler with an initial learning rate of 3.0e-5 and warmup ratio of 0.03. Optimization is performed bé16 precision.
For methods requiring block expansion, we expand 4 layers; for methods based on LoRA, set the LoRA rank to 256 to ensure the number of trainable parameters is roughly comparable between the two approaches. For the medicine injection into Llama models, which have poor Chinese support, we expand 8 layers for block expansion methods and set the LoRA rank to 512 for LoRA- -ase-based methods.
For our ADEPT, we calculate the importance score and update learning rate per 500 iterations. (It does not affect the impact of warmup, decay scheduler on the learning rate, but only performs reallocation.)
B.8 EFFICIENCY ANALYSIS OF ADEPT FOR MEDICAL APPLICATIONS
Table 6: Training Time Comparison in the Medical Domain. We select representative baselines including full- parlmearreeter (PT-Fulll training. PT-Lora, and Llama Pro validate the effectiveness of our method. The bold entries denote the optimal results.
Qwen3-1.7B
Qwen3-4B
Qwen3-8B days, 9h
Llama3-8B
PT-Full
2 days, 17h
days, 14h
7 days, 22h
ADEPT
1 day, 9h
2 days, 11h
days, 15h
days, 19h
PT-Lora
days, Oh
6 days, 4h
8 days, 23h
8 days, 2h
Llama Pro
days, 1h
days, 14h
days, 8h
4 days, 21h
As shown in the Table 6, our ADEPT approach achieves the fastest training time across all tested model sizes, with Llama Pro being the next most efficient competitor. The substantial efficiency gain of our method is mainly attributed to its design: ADEPT only updates a small subset of pa- rameters, primarily located in the deeper layers of the network. This structure allows the backward computation graph to terminate earlier, significantly reducing the overall training time.
We further analyze two aspects that explain and quantify the practical efficiency of ADEPT: (1) the runtime overhead of the iyporrtance-pobiing steps (layer masking and unit-level gradient probing) under single- and multi-GPU execution; and (2) the scaling behavior of training time when varying the number of expanded layers. These measurements complement Table and clarify why ADEPT achieves shorter end-to-end training times (including both probing and training) in practice.
21

---

Published as a conference paper at ICLR 2026
Probing overhead. The importaacc-probing in ADEPT comprises: one-time layer-importance pass (layer masking) and periodic unit-level gradient probing. Both operations are lightweight rel- ative to full training: layer masking is computed once before the main CPT loop and is fully paral- lelizable, and gradient probing requires only a single hackward pass per probe interval (every 500 steps). Table reports wall-clock times for these components single-GPU and 8-GPU setups, along with a representative total backpropagation time during training. All measurements use the same hardware configuration and identical probing data (CMMLU MMLU dev subsets, =580 ex- amples). The results demonstrate that (1) layer masking is trivially parallelizable across devices and thus benefits nearly linearly from multi-GPU execution; and (2) gradient probing a small fraction of the overall training backpropagation time.
Table 7: Wall-clock time for layer masking and unit-level gradient probing single-GPU and 8- GPU settings. 'Total Backprop (Train)' reports the total backward probing time during training.
Model
Layer Mask (1 GPU)
Layer Mask (8 GPUs)
Grad Prohe GPL)
Grad Prohe GPUs)
Total Backprop (Train)
Qwen3-1 3.7-B-B
36m30s
8m166
2m22s
1m50s
16m20s
Qwen3-4B-Basc
1h10m
17un39s
3m114
2m10s
25ar43s
Qwen B-BB BB--eB
1h32m
23m08s
5m24s
2m33s
29m033
Llama3-8B
1h24m
21m47s
6m266
3m19s
40mm22
Combined probing time is small relative the total training backpropagation time, probing does not meaningfully affect end-to-end efficiency, as the time saved due to ADEPT's reduced backprop- agation significantly outweighs the probing overhead.
Scaling analysis. We next quantify how training time scales we increase the number of expanded layers. Because each expanded Transformer layer contributes known parameter and compute footprint, wall-clock training time increases approximately linearly with the number of expanded layers our implementation. This near-linear behavior enables straightforward time-constrained auto-tuning: given time budget, one can estimate upper bound on the number of layers that may be expanded. Table 8 provides empirical training times for expanding different numbers layers in Qwen3 base models. These times measured under the same training configuration used for Table 6, isolating the effect of expansion count.
Table 8: Training time when expanding different numbers of layers in Qwen3 models.
Model
Layer
2 Layers
4 Layers
8 Layerrs
16 Layers
24h
1d 2h
1d 9h
2d lh
2d 20h
Qwen3-4B-Bas
Id 17h
2d 4h
2d 11h
3d 4h
4d 10h
Qwen3-8B-Ban
2d 18h
3d 2h
3d 15h
4d 6h
6d 22h
The empirical timings show near-linear increases in total training time with expanded-layer count for all evaluated model sizes. Notably, Qwwe3--8BBard shows a sharp time increase beyond 8 expanded layers, not due algorithmic nonlinearity but GPU memory limits that force smaller batch sizes and thus longer training. Time estimates should therefore account for compute resources. Still, our measured times offer practical guidance for layer scaling and selecting expansion size under fixed time budget.
Summary. The additional measurements above demonstrate that (I) the runtime overhead of our importance-probing is small and highly parallelizable, and (2) the training- time cost of expanding more layers grows predictably and nine-linearly. Together with the fact that ADEPT updates only fraction of parameters, these behaviors explain the consistent end-to-end time savings reported in Table 6.
B.9 EVALUATION SETTING
We evaluate the performance of large language models on multiple-choice question answering tasks using accuracy the primary metric. For given question with N candidate options (typically N = 4, labeled A, B, C. D), the model's prediction is determined by computing the sequence-level likelihood of each option when appended to the question stem.
22

---

Published as a conference paper at ICLR 2026
Specifically, let Q denote the input question and O, represent the i-th answer option (e.g., True, B. False). The model computes the conditional probability of the full sequence Q O (i.., the concatenation of the question and the i-th option) under the causal language modeling objective. We calculate the average negative log-likelihood (or perplexity, PPL.) of the tokens in O₂ given Q:
PPL(O, Q) = exp (T011
log P(o: 11 Q. Σ
(8)
The model selects the option with the lowest perplexity as its predicted answer:
y = arg
min (.E.A.B.C.)
PPL(O: Q)
(9)
This method, often referred to perplexity- decoding, does not require fine-tuning addi- tional parameters and is widely used for evaluation base models. It leverages the pre-training objective directly by predicting the next token, making it particularly suitable for evaluating general knowledge base LLMs.
Finally, accuracy is defined the percentage of questions for which the model's predicted answer matches the ground-truth label:
M I(ū; 9z)
1
Accuracy
(10)
M
where M is the total number of test questions, Uj is the model's prediction on the j-th question, the true label, and II(+) is the indicator function.
For our experiments, we evaluate all model checkpoints using the Im harness' framework. For the Mathematies domain, we adopt the default configurations of GSM8K cot, ARC-Easy, and ARC-Cha 1 lenge. For the Medical domain, we design custom configuration files for MedQA, MMCU-Med i cal, and CMB, following the official evaluation protocols of MMLU and CMMUU. For the General domain, we directly evaluate MMLU and CMMLU. In all cases, use 5-shot prompts and greedy decoding (temperature = 0) for inference. This standardized evaluation protocol ensures fair comparison across models and tasks.
C MODEL PARAMETER GROUP
To enable efficient and semantically meaningful parameter decoupling during fine-tuning, we parti- tion the model parameters into modular units based their functional roles within the transformer architecture. Given the substantial number model parameters, extremely fine-grained control at the neuron level-as used in methods like DAS (Ke al., 2023)-is computationally prohibitive and contradicts the goal of parameter-efficient adaptation. Moreover, such fine granularity often leads to training instability due noisy importance estimation.
On the other hand, treating an entire layer single unit (e.g., standard LoRA) is too coarse and lacks semantic discrimination. While TaSL (Feng et al., 2024b) proposes decomposing LoRA into LoRA A and LoRA B. this approach is specific to low-rank adapters and does not generalize well to full-layer decomposition.
To strike a balance between granularity and efficiency, we introduce a semantic-awanl module par- titioning strategy, which divides each transformer layer into multiple functional units according to their architectural semantics. This design allows us manipulate parameters at meaningful inter- mediate level-finer than whole layers, but coarser than individual neurons-achievinn practical trade-off between controllability and computational feasibility.
Table presents the detailed parameter grouping scheme used this work, exemplified on the LLaMA architecture.
https:// thub. com/ hww.///eeeteeaaaaaeee.aaa /mmmm/mmmmaaaa-e eneevalio uat iattion-areesss
23

---

Published as a conference paper at ICLR 2026
Table Model Parameter Grouping Scheme
Parameter Type
Parameter Name
Description
Attention
self attn. proj weight
Query projection weight; maps input to query space
self.attn.k.proj weight
Key projection weight; maps imput key space
self.attn.v.proo..oo
Value projection weight; maps input value space
self.attn.o.proj weight
Output projection weight; projects
attention output back to target dimension
MLP
mip. mlp.gate.proj weight
Gating projection weight; controls information flow in SwiGLU activation
m1p. up proj weight
Up-projection weight; maps features to higher-dimensional intermediate space
mlp. www.dddww.oo.w....o
Down-projed projection weight; projects features back original dimension
LayerNorm
input.layermom..ee..
Input layer normalization weight; normalizes input before attention
psstattent.oo.llyemmmm...
Normalization weight after attention; stabilizes post-attention outputs
As shown in Table 9, each transformer layer is decomposed into three primary functional modules: Attention, MLP, and LayerNorm. Within each module, parameters are grouped by their semantic role:
The Attention module includes all four linear projections (Q, K, V. O), which collectively handle context modeling through self attenatteniin.
The MLP module contains the up. gate, and down projection layers, responsible for non-linear feature transformation.
The LayerNorm components kept separate due to their distinct role in stabilizing activations and gradient flow.
This grouping enables targeted manipulation of specific sub-functions (e.g., disabling attention out- puts or freezing normalization statistics) while maintaining training stability and interpretability.
C.I COMPATIBILITY BETWEEN LAYER EXPANSION AND DECOUPLING
First, we would like to share our understanding of the Compatibility between Layer Expansion and Decoupling:
1. Although layer expansion can minimize changes to the original parameter space, this alone makes difficult to fully prevent model drift during long-term pre-training. Parameter decoupling offers more fine-gra grained means of controlling this phenomenon.
Since our models are pre-trained on large corpus, their parameter space is inherently uncon- trollable, making thorough decoupling of the original model parameters challenging- In contrast, the newly expanded parameters initially contribute nothing to the model's output. As we continue domain-specific training in the medical field, gradually decoupling these parameters is more conducive to achieving complete decoupling.
To examine the effectiveness of our layer extension strategy, conduct activation distribution anal- ysis across multiple backbones. For each model, we first identify the most domain- adaaaaabe layer (i.e.. the layer with the lowest Inyer). We then randomly sample 500 instances from both the Medical and General corpora, compute activations at the selected layer, and visualize their distributions using Kernel Density Estimation (KDE). The following three configurations are compared: (1) the origi-
24

---

Published as a conference paper at ICLR 2026
Genera
General
Ceneral
40-
20-
20-
-20
-20
30
dim
dim
dmm
(a) Vanilla
(b) W/o stagel
(c) ADEPT
Figure 6: Kernel Density Estimation of activations for Qwen3-1.7B-BB 7-BBcc under different configu- rations, Our layer extension strategy enables effective parameter decoupling, Expanded layers: 22, 23, 25, and 27.
nal base model, where we visualize the most domain-adaptable layer; (2) direct decoupling without expansion (w/o Stage-1), 1), where we visualize the same most domain-adaptable layer; (3) our method with expanded layers, where we visualize the newly created expanded layer (copied from the most domain-adaptall layer).
Figure presents the results from three different model configurations, providing compelling evi- dence for the advantages of our proposed approach.
Figure 6 a) shows the activation distribution in layer 27 of the original Qwen3-1.7B-Bal model. The substantial overlap between general and medical text distributions indicates strong parameter coupling, which is an expected consequence of mixed-domain pretraining. This coupling makes it challenging to achieve clean separation of domain-specinn fine-tuning approaches. However, the divergence between the peak values in the general domain and the medical domain also indicates the potential for decoupling.
functionalities through conventional
This coupling phenomenon persisted in our ablation studies with only the decoupling method in Figure 6 b). Despite our attempts to decouple the medical and general modules when training, the model's activation distributions remained largely entangled (the graph still shows substantial overlap), failing achieve distinct separation between domains. This observation further supports our argument that pre-existing parameter coupling from mixed-domain pretraining creates inherent challenges for direct decoupling approaches.
In contrast, Figure 6 c) demonstrates the activation distribution in layer 31 of our extended model, where we first expanded the model by copying parameters from layer 27 and then applied decoupling training. The clear separation between general and medical text distributions suggests successful pa- rameter decoupling. This superior decoupling effect can be attributed our "blank slate" approach: the extended layers, while initialized with copied parameters, provide fresh parameter space that hasn't been constrained by mixed-domain pretraining. During decoupling training, these extended layers can adapt more freely to domain -omain-specific patterns through gradient descent and importance- based learning rate adjustments.
To validate our hypothesis, we also examine the effect of applying in Qwen3-4B-BB (Figure 7), Qwen3-8B-PR (Figure 8), Llama3-8B (Figure 9). The results indicate limited separation between domains, which supports argument that the entangled parameters from mixed-domain pretrain- ing are challenging to decouple through training alone.
These findings demonstrate that our layer extension strategy provides more effective pathway for parameter decoupling compared to direct decoupling training. By creating a new parameter space through layer extension, avoid the constraints of pre-existing parameter coupling, allowing for cleaner separation of domain-specific functionalities during subsequent training. This approach offers promising direction for developing more modular and domain-adaptable language models.
25

---

Published as a conference paper at ICLR 2026
Genera Motial Rod.
General Modcal Red.
General Meskal Rec.
20
23
6m2
-25
-20
-50
-75
-75
100
30
30
100
30
50
-50
30
(a) Vanilla
(b) W/o stage1
(c) ADEPT
Figure 7: Visualization of activation distributions for Qwen3-4B model configurations show- ing the effectiveness of our layer extension strategy for parameter decoupling. We expand the layer 28, 30, 31, 35 of Qwen3-4B-- 4B---BB-......
Genera Hedtal Ind
General
General
25
Medcal Bed
Hedial Iec.
25-
20-
-20
-30
30
-100
-30
50
100
-100
-30
50
(a) Vanilla
(b) W/o stage1
(c) ADEPT
Figure 8: Kernel Density Estimation of activations for Baan3-8--Buse showing that our layer extension strategy enables clear parameter decoupling. We expand layers 26, 28, 29, and 30.
Canera
General
Tax:
30
5-
-20
-20
-40
-00
-10
-30
dim
di
dm
(a) Vanilla
(b) W/o stagel
(c) ADEPT
Figure 9: Kernel Density Estimation of activations for Llama3-8B, showing that our layer extension strategy enables clear parameter decoupling. We expand layers 22, 23, 24, and 28.
26

---

Published as a conference paper at ICLR 2026
D
DETAILED IMPORTANCE DISTRIBUTION
27
26
25
24
23
22
21
20
19
18
17
16
15
Layer
14
13
12
11
10
7
4
1
Layer Importance
jpo proj mip.up proj
self, attnt...................... proj
mlp. self, attn.v vtpoov_projj gagpaprrojjo .poj j_roj
self. attn.q_prro proj self attn.k_proj _rrj
post attention layernorm
input_ layernorm self. aattt.ttkkkk norm _ttmm
self attn q_ norm
mip.gate_po hip.gate_uu
mlp.down_
Parameter
Figure 10: Layer-wise and wara importance imporratn distribution of Owwen3-1.B--Baut model
To investigate which layers should be expanded, we conduct a comprehensive importance analysis at both the layer and parameter levels. Specifically, compute the importance scores for each layer and parameter across multiple models, and visualize their detailed distributions (see Figure 10, Figure 11, Figure 12, and Figure 13). Our analysis yields the following key observations:
27

---

Published as a conference paper at ICLR 2026
34
32
30
28
26
24
22
20
Layer
18
16
14
12
10
8
0
6
4
2
Layer Importance
mlp.down_nr proj
mip.up proj gat. ppoo self. atttn.o ppoj
self. attn.v_trar pproj
self attn.ttttpp...om prroj proj
self attn.k attn.k_proj proj
post attention, layernorm
input input_layenen layernorm
self aatnnk_nnatt norm
self attn q norm
Parameter
Figure 11: Layer-wise and parameter-wise importance distribution of the Qwen3-4B- Base model
28

---

Published as a conference paper at ICLR 2026
34
32
30
28
26
24
22
20
Layer
18
16
14
12
10
4
Layer Importance
hilp.don_ppro proj mip.up proj mlp. gate mlp.gate_ppi _proj
self. attn.t..o....................a.. o .poj
self attn.q prroj self attn.v ppro self attn.k_proj
post attention layernorm
input input_layene layerorrm
self aatttttttna... sttm. norm
self attn q norm
Parameter
Figure 12: Layer-wise and parameter saree importance distribution of the Qwen3-8BB 8B-Base model
29

---

Published as a conference paper at ICLR 2026
30
28
26
24
22
20
18
Layer
16
14
12
10
8
6
4
2
Layer Importance
mlp.down_pray proj
mip.up proj
mip. mip.gate -ap-gate_uu .poj
self aatttttt_ppp.. proj
self aattttttvvaaa. prooj
self, atttn.q_proo proj
post crrtt_ateetto_aayyee layernorm
self attn.k_prop proj
input input_layemonn Jayerorr
Parameter
Figure 13: Layer-wise and parameter-win wise importance distribution of the Llama3-8B model
30

---

Published as a conference paper at ICLR 2026
1. Layer and parameter importance alignment. Overall, the distributions of layer-wise impor- tance and parameter wise importance are highly aligned across all four models. This alignment is expected, both metrics are fundamentally computed under the same crinciple-estmnnne pact of masking out (setting to zero) a given layer or parameter on model performance. Since param- eter importance essentially decomposes the contribution the layer level, this consistency reflects the intrinsic, nested relationship between the two. It also indicates that layer-level and parameter- level interventions affect the model's predictive capability in a coherent manner.
the im-
High importance in lower layers and the penultimate layer exception. notable pattern across all models is that the most important layers tend to be concentrated in the lower (early to middle) layers the network, with importance values generally decreasing towards higher layers. This pattern suggests that the early layers play critical role in the overall function of the model.
One plausible explanation, is that lower layers are responsible for capturing general syntactic prop- erties and foundational compositionality (Clark al., 2019; Hewitt & Manning, 2019), such as basic grammar and phrase structure. In contrast, deeper layers are typically responsible for integrat- ing more task- or context-specifie semantic information. This division of labor (earlier layers for generic linguistic structure, deeper layers for task semantics) naturally results in higher sensitivity interventions at the bottom layers. This also provides theoretical basis for layer expansion in deep layers.
An interesting exception observed in all models is that the penultimate layer does not follow this general trend: its importance appears elevated relative to immediately adjacent layers. This may stem from the model's need consolidate high-level semantic features just before producing the output prediction. The penultimate layer may act as "bottleneck" for aggregating information necessary for the final decision or token generation poooonnillll as final representation refinement step. Similar phenomena have been observed in works such as Intrinsic Dimensionality Explains the Ffffectiveness of Language Model Pruning (Aghajanyan et al., 2021), which highlight the special role of upper- and penultimate layers in output formation.
3. Intra- and inter-family patterns: Qwen vs. Llama models.
Qwen family: Across the Qwen models (Qwen3-1.7B, 4B, 8B), the overall trends are similar:
Importance is strongly concentrated in the lower and middle layers, particularty within the first 10 layers, regardless of total model depth.
Among parameters, mlp. down-proj and mlp. up-proj typically dominate in the most im- portant layers, suggesting that feed-forward (MLP) components contribute substantially to the information processing in the Qwen series.
With increasing model size (from 1.7B to 8B), the importance distribution appears to spread out slightly, showing less sharpness at the very bottom possibly reflecting increased capacity and redundancy in larger networks.
Cross-family: Comparing Qwen models to Llama3-8B, we observe both notable similarities and differences:
Both model families consistently exhibit high importance in MLP-related parameters (mlp. dww..proo.ooooooooooooooooooooo. mlp.up up-proj. and mlp. groe.,roj), especially in the most important layers. This underscores the universal role of the feed-forward network in transforming and inte- grating information beyond the capabilities of self-attention alone.
Llama3-8B shows a broader distribution of importance across layers, with non-negligibe val- ues extending further into the middle and upper layers, suggesting a more distributed processing pipeline. In contrast, Qwen models tend to concentrate importance more the lower layers.
The dominance of MLP components in Llama3-8B is somewhat less pronounced than in Qwen, with parameter importance appearing more diffuse overall. These inter-family differences may be attributable to variations in architecture (such as normalization, attention mechanisms, or feed- forward design), pre- pre-training data, or other modeling choices, leading to distinct strategies of information flow and representation across the network depth.
31

---

Published as a conference paper at ICLR 2026
E LAYER-WISE IMPORTANCE ESTIMATION METHODS COMPARISON
To investigate which layers contribute most to model performance, we employed four different strategies compute layer-wise importance:
Cumulate importance of parameters: For each parameter p in a layer, we compute the product Ppp and sum across all parameters in the layer:
ac
Jayer
Σ
P
(11)
Op
pElayer
2. Module-wise rank aggregation: For each module (c.g., attention, MLP, normalization), calculate the importance score, rank layers by their score within each module, and aggregate rankings to obtain a total rank for each layer.
3. Masking out: For each layer, we mask out its parameters (i.c., set to zero) and evaluate the change in loss:
Iuyer = (model with layer masked) - L(original model)
(12)
4. Fisher information: imation
For each parameter p in a layer, using the Fisher information approx-
(0logp(el)) (
F(P) E
(13)
Öp
Layer-level Fisher importance is obtained by summing over all parameters in the layer.
To further understand the significance and robustness of these metrics, we conducted preliminary experiment on the Qwen3-1.7H8 Siven--1.B--Bas in the medical domain with dev subset of MMLU, CMMLU to detect the importance of layers, focusing on how different gradient computation strategies affect downstream performance.
Table 11: Performance of different expansion methods on medical-domain tasks (best result in each column is bolded). The numbers in parentheses after each method in the table indicate which layers were expanded. The Qwen3-1. 1..nn-1117B-B-- 1.BB-ase model has a total of 28 layers, indexed from 0 to 27.
Methods Name
mmiu
cmmlu
medqa
cmb
Qwen3-1.77 Brom3-1.7B-BBe 7.BB-Bacc
62.57
66.86
48.39
63.67
69.17
Uniformly Expansion (6,13,20,27)
59.06
64.98
48.78
64.25
70.10
Uniformly Expansion for first 16 layers (3,7,11,15)
59.60
64.91
48.78
64.07
69.80
Uniformly Expansion for last 16 layers (15,19,23,27)
61.60
66. 15
49.32
65.55
71.09
Importance Cumulation (23,24,25,27)
62.63
66.81
50.19
63.85
69.48
Rank Aggregation (22,24,25,27)
62.72
66.86
50.57
63.97
69.78
Masking Out (22.23.25,27)
62.80
66.89
50.75
65.43
71.98
Fisher (23,24,25,26)
61.84
66.43
49.15
64.13
68.82
Table compares the effect of different layer selection methods for expansion on variety of medical-domain tasks using Qwen3-1.7B-Bass Several key observations can be made:
Similarity selected layers across methods. All importance calculation methods lead to the selection of similar layers for expansion. For instance, the layers chosen by methods such Im- portance Cumulation (23,24.25.27), Rank Aggregation (22,24.25,27), Masking Out (22,23,25,27), and Fisher (23,24.25,26) 26) significantly overlap, especially in the last layers of the model (layers 22 and above). This convergence strongly validates our previous observations that general capability- critical layers tend to be concentrated in the latter half of the model in Appendix D.
In addition, the results show that uniform expansion into the last 16 layers (Uniformly Expansion for last 16 layers (15,19.23,27)) consistently outperforms expansion into the first 16 layers (Uniformly
32

---

Published as a conference paper at ICLR 2026
Expansion for first 16 layers (3,7,11,15)) uniformly across all layers, further supporting the result in Appendix D.
2. Robustness of expansion results across methods. Despite minor variabilityy in the specific lay- ers chosen by each method, the final performance of all importance-based expansion approaches consistently better than both the vanilla baseline and uniform expansion. For example, the MedQA dataset, all methods using calculated importance exceed the baseline score (e.g., Masking Out achieves 50.75 vs. baseline 48.39), and on MMLU-med, Rank Aggregation achieves 67.95 versus the baseline 66.49. Crucially, the differences in scores among Masking Out, Rank Aggrega- tion, Importance Cumulatation, and Fisher are relatively small for most tasks (typically less than points), indicating that the overall framework is robust to the choice of importance calculation tech- nique. Since our principal contribution is the training paradigm rather than the specific importance metric, for subsequent experiments, we employ the masking out approach, which demonstrated the strongest effect in preliminary experiment.
F THEORETICAL ANALYSIS
Our theoretical analysis relies on several simplifying assumptions as outlined below. We discuss the rationality and limitations of each assumption:
(A1) Linearized Model Structure:
We model the transformer as stack of independent residual blocks, effectively ignoring cross-layer coupling effects such as those arising from pre-norm and residual connections.
Justification: In our layer expansion scheme, the newly added layers are always sepa- rated by least one original frozen layer and never arranged in a eascading manner. This design substantially weakens direct coupling between newly expanded layers, which, in turn, reduces the degree of inter-layer interaction and nonlinearity affecting our analysis. And this abstraction is commonly used in theoretical studies (e.g., NTK analysis or pruning literature) to make layerwise analysis tractable.
(A2) Loss Function Smoothness: We assume the loss function ((.,.) is B-smooth and To Lipschitz with respect to predictions.
Justification: Standard loss functions such as cross-entropy (with stability improvement) and mean squared error are widely established to satisfy these properties. These condi- tions allow us to relate small output perturbations to controlled changes in loss, facilitating theoretical bounds.
(A3) Training Dynamics: Our analysis assumes training is performed with a first-order SGD- like optimizer, disregarding effects from Adam or other adaptive methods.
Justification: First-order SGD provides well-understood theoretical properties and is com- monly used in theoretical deep learning research. While Adam introduces adaptive scaling that can affect convergence, many results (e.g., generalization gap bounds) transfer quali- tatively between SGD and Adam practice.
(A4) NTK Regime and Sensitivity: Our analysis of layer sensitivity relies on the NTK (Neural Tangent Kernel) approximation (Jacot et al., 2018), which essentially assumes the model behaves locally linearly around its current parameters. Moreover, we should consider the model training process be relatively stable, with no anomalous occurrences such as gra- dient explosion.
Justification: This assumption is particularly well-motivated in our setting for two reasons. First, our adaptation protocol only updates a small number of newly introduced parameters while keeping the vast majority of the pre-trained weights frozen and decouples parameters to maximize the retention of general capabilities. This ensures that the parameter changes remain minimal, keeping the network within the local linear (NTK) regime throughout adaptation. Second, unlike random initialization, our starting point is well-trained model on a large general-domain corpus, which already provides robust and meaningful repre- sentations. Perturbations induced by finetuning thus intrinsically local in the function space and less likely to induce sudden or nonlinear model behavior, further enhancing the validity of the NTK approximation.
33

---

Published as a conference paper at ICLR 2026
Overall, these assumptions enable to derive interpretable upper bounds and provide actionable layer selection criteria, but should be considered as idealizations. The correspondence between these thcoretical insights and practical behavior is also validated in empirical experiments.
F.1 OPTIMALITY LEAST-IMPORTIN BLOCK EXPANSION FOR PRESERVING GENERAL CAPABILITIES
Notation: Let My denote the original base model, and M denote the model after T steps of adaptation, wherein only the set S of layers unfrozen and updated, and ((:,)) is the loss func- tion (e.g., cross-entropy) which is L-Lipschitz and 3-smooth in its first argument. A represents the importance score of layer as defined below.
Layer Importance Score:
A BawD (((M6 (((((())
where MD-1) Mo with the 1-th layer masked out.
Theorem F.1 (Upper Bound on Generalization Gap by Layer Importance). Let S [L] be the set of layers selected for expansion/adlpttaion,, and G(S) denote the source-domain generalization gap after adaptation, i.e,,
G(S) EE ( (T) (())))))))))))))) (
initialization, limited adaptation steps, and L-Lipschitz and B-smooth loss, the following upper bound holds:
Under
C(S)SC AA =
LES
where C is constant depending the learning rate, steps, loss smoothness, and initialization, and AW the maximal per-layer parameter change over adaptation.
Proof. Step 1: Output Deviation Linearization. By dunction-preserviin initialization, M (x)
(0)
Ma(x). After adaptation, since only layers in S are modified and changes are small (Assumption A4). the output difference admits first-order Taylor expansion:
M
(2) (x) Mo(x) Σ IES
J((z) AW
where [1(z)
8M aW;
and AW, W(TT) - w/
W-
Step 2: Lipschitz Property Application. By L-Lipschitzness of E(:, w) in its first argument,
((M (z), e(Mo(z),)) | =
Mo(x)
Taking the expectation over x gen:
G(S) < EE (x) (x))
Step 3: Breaking by Layer via Triangle Inequality. According to Assumption Al and using the triangle inequality,
| (M) -Mo(x))22 S Σ [W?ll))W)),
LES
thus,
G(S) < Σ E, [1) AW]
IES
34

---

Published as a conference paper at ICLR 2026
Step 4: Relating to Layer Importance Score. Recall the definition:
( = Eu [((M) v) - ((Mo(z))
By Taylor expansion and Lipschitz continuity,
1e(M (x),))(x))))) - ((M6(z)))) [ 1|,
so for small modifications,
E. (1)) AWilla] | A O(IlAWI)). | 12
Assume |AWilla & AW for all I E S and
|| (0) Il2 are similar or lower-bounded by wp > 0, so
AW
G(S) <
A + +(KKAW))).
LES
Step 5: Optimization Control. In standard SGD (Assumption A3), AW is controlled by learning rate 77, steps T, batch size N, and bounded gradients:
S 1/I N
AW
max (w; 9:)l2.
Thus, all learning and initialization constants be absorbed into scalar constant C (Assumption A3 and A4). Step 6: Conclusion. Thus,
G(S) S C Σ A +0 (k(A))?).
LES
which completes the proof.
Due to the use residual connections, the original block and the expanded block can be viewed as a single aggregated unit. Importantly, before training, the addition of the new block does not alter the model's s output, and thus the overall importance of the aggregated block remains exactly the same as that of the original block (i.e., (1)) As a result, when we train the parameters of the new block, it is effectively equivalent to adapting the aggregated block as a whole, whose im- portance is still characterized by the original importance score A (1) This justifies why the potential impact of training the expanded layer is governed by the original layer's importance.
The tightness of the derived upper bound hinges on both the local linearity of the expansion regime and the control over parameter updates during adaptation. In cases where the expansion layers are initialized to be function preservinn and the adaptation performed with sufficiently small learning rates and moderate step sizes, the Taylor and Lipschitz approximations used in the proof become increasingly sharp. Thus, the upper bound is not only theoretically attainable, but also approaches the realistic generalization gap observed in practice under these conditions. This means that mini- mizing the sum Eies A (4) when selecting layers for expansion is not merely mathematical conve- nience-it is a principled, actionable strategy for controlling catastrophic forgetting and generaliza- tion degradation. As a consequence, those layers with the lowest importance scores, practitioners can reliably minimize negative trans- fer from domain adaptation, especially when adapting large pre-trained models with limited new capacity.
criterion provides practical guidance: by limiting updates to
F.2 OPTIMALITY OF IMPORTANCE- BASED LEARNING RATE ADJUSTMENT FOR MODULES
We provide a rigorous analysis of learning rate reallocation in Stage 2. Specifically, let the impor- tance of each parameter 8, in the general domain be defined as
Lg
To =
00
35

---

Published as a conference paper at ICLR 2026
where Lgen denotes the general-domain loss and Le; quantifies the sensitivity of the overall perfor- mance with respect to 8z. Under the constraint of fixed average learning rate, our strategy assigns lower learning rates parameters with high general-domain importance, and higher learning rates to those deemed less important. This iuportance-weighter reallocation is provably optimal for min- imizing the upper bound of catastrophic forgetting in the general domain, subject the constant average learning rate constraint. Furthermore, we formulate and analytically solve the underlying constrained optimization problem ensure that our reallocation approach achieves relative optimal- ity in practice.
Let D Dge be the general domain distribution with loss Lgen (0). With 0* 0 as the original pre-trained parameters, we define parameter importance 1 _ 0 00, le- and unit importance:
Setup and Notation
lu
1 [0
(14)
U1
JEU,
under learning rate budget constraint:
Ul |
Iru.
Irbase
(15)
F.2.1
UPPER BOUND ON FORGETTING
Define forgetting as:
F A Lgen (8(T)) Lgen
(16)
Assuming Legnn is B-smooth, the first-order Taylor expansion provides:
F < VoLgen (0") A(T) + 8 ||(T))
(17)
Due to parameter freezing, the gradient V VeLgen (0") is only non-zero for expanded parameters:
VoLgen (0*) = ->> 4e,
(18)
JEUL
where I alyen 86, ej are basis vectors.
Assuming gradient descent with per-group step size Iju and T steps, for each parameter j E Ug (As-
sumption A4):
aLmed
A,(T) -Tnu; 00,
(19)
Substitute into the smoothness bound:
8 ΣΣE
F F F<ΣD (T)
Σ Σ (A,
(20)
2
jEU
jell
3
Σ Jull Te,l (Tnu, G) + T2
U.li 02
(21)
where
max,
@mmed 30,
upper-bounds the adaptation gradients.
The derived upper bound encompasses all possible learning rate allocations and ensures conservative control catastrophic forgetting. Note that if group gradients G' or importance scores Iy, heterogeneous, more refined bound can be obtained by analyzing variance rather than worst-case values.
36

---

Published as a conference paper at ICLR 2026
F.3 OPTIMAL IMPORTANCE DRIVEN LEARNING RATE REALLOCATION
Problem Statement:
We aim to allocate learning rates !!; for each parameter group Us to minimize the upper bound on forgetting:
F a Σ wiIiu + Σ W
where wy |UA the number of parameters in group U, ITu indicates the average im- portance of parameters U2 a, > O are constants determined by training steps, gradient norms, and the smoothness constant (B (Assumption A2)). The constraint is that the average learning rate remains fixed:
Σ winu Wavvg
where W E; we is the total number of trainable parameters.
Lagrangian Formulation:
Introduce a Lagrange multiplier À and write the Lagrangian:
S(mmo..A)) a Σ wrimo. + 6Σ w; win +
Σ
Whaz
Optimality Condition:
Taking derivatives and setting to zero, we obtain for each j:
dc
aw,1y 15 + 2bru,nu; + Aw,
Onu,
ni
2b Ij
2b
Including the constraint:
Σ w Wmavg
Plugging in the expression for nú, gives:
a
A
W
W Wylaug
2h
2h
Solving for A:
22bang
a W I
Σ
3
So the optimal learning rate for group U, is:
TÜ, Bang
a
Σ
wz Ij'
(22)
2h
W
j'
Interpretation and Guidance:
When the theoretical upper bound is tight- tight-which is often the in well-controlled, locally linear training regimes-this this result has direct practical utility. Notably, the optimal learning rate allo- cation ni; is affine (linear) function of the group importance I5. Our method, which assigns Inv = 2 (1 Amiit) Iraaee can be viewed as simplified implementation of the derived optimal form. By decreasing the learning rate for groups with high general-domain importance and increas- ing it for those with low importance, this strategy effectively minimizes the risk of catastrophic forgetting while respecting the global learning rate constraint. Thus, our approach provides action- able guidance for tailoring learning rates based on parameter importance in continual learning and domain adaptation.
37

---

Published as a conference paper at ICLR 2026
G EXPERIMENT ABOUT THE NUMBER OF EXPANDED LAYERS
In Stage I, determining the optimal number of expanded layers emerges as a crucial hyperparameter. To investigate this, conducted systematic experiments across various model scales in the medical domain by expanding different numbers of layers. These comprehensive experiments aim to pro- vide empirical insights into selecting the most effective layer expansion strategy, offering valuable guidance for future research in this direction.
Table 12: Comparative Performance of Different Layer Expansion Strategies across Model Scales and Medical Tasks. Bold indicates the best-performing setup for each task; underline shows the second-best. This highlights optimal and near-optimal choices for each scenario.
Model
MMLU
CMMLU
MedQA
MMCU-Medical
CMB
Qwen3-1.7B
Vanilla
62.57
66.86
48.39
69.17
63.67
1-layer
62.31
66.23
48.08
69.95
61.40
2-layer
62.48
66.91
48.63
70.78
62.89
4-yayer
62.80
66.89
50.75
71.98
65.43
8-layer
61,84
66.02
49.57
72.41
65.00
16-layer
60.96
64.65
48.86
70.13
64.88
Qwen3-4B
Vanilla
73.19
77.92
62.77
82.44
78.92
l-layer
72.98
77.69
63.39
82.83
78.21
2-layer
73.10
77.84
63.08
82.80
78.48
4-layer
72.95
78.77
64.49
84.58
79.87
8-layer
73.06
77.65
65.02
84.22
78.81
16-layer
72.06
77.11
62.61
82.09
78.61
Qwen3-8B
Vanilla
76.94
82.09
66.30
86.45
81.67
l-layer
76.84
82.06
67.87
86.95
81.50
2-layer
76.70
82.10
67.93
87.99
82.90
4-layer
76.77
82.11
69.24
89.84
85.80
8-layer
76.77
82.15
68.34
88.02
84.85
16-layer
77.12
82.28
68,56
87.76
84.32
LLaMA3-8B
Vanilla
65.33
50.83
58.91
46.29
35.61
l-layer
65.29
51.12
58.97
50.83
40.45
2-layer 65,61
50.98
59.56
55.92
47.83
4-leerr 65.25
51.73
60.82
63.17
54.65
8-layer 65.17
51.92
61.17
67.03
61.78
16-layer
65.12
52.45
61.92
70.86
65.31
For general language tasks such as MMLU and CMMLU, all models largely preserve their baseline performance regardless of the number of expanded layers. This indicates that layer expansion does not compromise the models' general language capabilities and robustness.
However, for domain-specinn medical tasks (MedQA, MMCU-Medicas and CMB), the impact of layer expansion is more pronounced. Across all Qwen model variants (1.7B, 4B, and 8B), expanding 4 layers consistently yields optimal performance, as shown by the bolded results in Table 12. Specif-
38

---

Published as a conference paper at ICLR 2026
ically, the Qwen3-1.7B, 4B, and 8B models improve MMCU-Medical by up to 2.8%, 2.1%, and 3.4%, respectively, when increasing from baseline to 4-layer expansion. Notably, expanding be- yond layers (e.g., to 8 or 16 layers) does not systematically improve performance- and in several cases, results in diminishing or even degraded accuracy. This suggests that moderate layer expan- sion (4 layers) achieves balance between performance gain and model stability, while excessive expansion may introduce optimization difficulties, overfitting, or disrupt the pre-trained knowledge representations, leading to suboptimal outcomes.
In contrast, the LLaMA3-8B model displays a unique trend: performance improvements are con- tinuous as more layers are expanded, with the best results observed at expanding 16 layers. The gains are considerable for tasks like MMCU-Medical and CMB, where scores rise dramatically from 46.29% and 35.61% % in the vanilla model to 70.86% and 65.31% % with 16 expanded layers. This behavior contrasts with the Qwen models and is likely due to LLaMA's more limited Chinese capability in its original configuration. The need for extensive architectural expansion reflects the necessity to build new, specialized representations to compensate for baseline deficiencies when ad- dressing Chinese-centric tasks. Therefore, while moderate layer expansion is optimal for models pre-trained on Chinese data (Qwen), more substantial expansion may be required for models less adapted to the target language or domain (LLaMA).
Overall, these results indicate that expanding more layers does not guarantee better performance. For well-aligned models, excessive expansion may lead to interference with the original knowledge or cause optimization instability. In contrast, for models lacking target domain competence, increased expansion helps establish the missing representations, albeit at the cost of greater computational complexity.
H TAKE PRETRAIN DATA AS IMPORTANCE SOURCE
Our previous experiments employed the dev sets of MMLU and CMMLU benchmark datasets for gradient- based importance estimation. However, such high-quality and carefully curated bench- marks are often scarce, especially in practical industrial scenarios. To investigate the robustness of our ADEPT method under more realistic conditions where benchmark data may not be available, we explore the use of noisier pretraining data for importance estimation.
Table 13: General Competence Detection Pretrain Corpus. #Examples means the number of exam- ples we used.
Dataset
#Examples
Hugging Face Link
FineWeb Edu
500
HuggiingFaceeW//inewweeee
FineWeb Edu Chinese V2.1
500
Hluggingaace/WWiineeeeeee
Specifically, we utilize the FibeWe and FineWebEdduCCin in Table 13), extracting the top 500 samples with the highest educational scores from the first 10,000 entries in each corpus to serve as our importance estimation set. Compared to curated benchmarks, these datasets are much more accessible in real-worla applications. Furthermore, the computational cost for filtering out such high-quality samples is negligible relative to the overall cost of large-scale pretraining-
datasets (Data overview and links
This experimental setting allows us to rigorously evaluate the robustness of ADEPT when real- world, easily accessible pretraining data replaces ideal benchmark datasets for importance-based layer expansion decisions.
Table 14 summarizes the performance of our ADEPT method when the importance estimation is conducted with either high-quality benchmark data or more easily accessible pretraining data across different model scales. Overall, the results demonstrate that ADEPT not only consistently out- performs the vanilla baseline but also shows remarkable robustness across most scenarios when using pretraining data for importance calculation. In Qwen3 series models, the difference between cenchmark-basee and pretraining-daat data- based importance estimation is minimal. In several cases, the latter even slightly surpasses the benchmark version (c.g., Qwen3-L.77 1.7B on MMLU and Qwen3-8B on MMLU and CMMLU), validating the practical applicability and flexibility of our approach.
39

---

Published as a conference paper at ICLR 2026
Table 14: Performance comparison of ADEPT with benchmark-based and pretraining-dan setrraininn-dat-baaaae importance estimation across model scales. Bold indicates the best performance per column; underline marks the second-best.
Model
MMLU CMMLU
MedQA
MMCU-Medical
CMB
Qwen3-1.78
Vanilla
62.57
66.86
48.39
69.17
63.67
ADEPT (Benchmark)
62.80
66.89
50.75
71.98
65.43
ADEPT (PT Data)
62.85
66.87
49.39
70.84
63.07
Qwen3-4B
Vanilla
73.19
77.92
62.77
82.44
78.92
ADEPT (Benchmark)
72.95
78.77
64.49
84.58
79.87
ADEPT (PT Data)
73.14
77.96
63.94
83.34
79.62
Qwen3-8B
Vanilla
76.94
82.09
66.30
86.45
81.67
ADEPT (Benchmark)
76.77
82.11
69.24
89.84
85.80
ADEPT (PT Data)
76.83
82.20
67.56
87.20
83.92
LLaMA3-8B
Vanilla
65.33
50.83
58.91
46.29
35.61
ADEPT (Benchmark)
65.25
51.73
60.82
63.17
54.65
ADEPT (PT Data)
65.21
50.27
59.13
60.29
51.32
For LLaMA3-8B, ADEPT with pretraining data still yields clear improvements over the vanilla baseline on all tasks, particularly in domain-specific metrics such as MedQA and MMCU-Medical. However, compared to the benchmark- lower performance, with gap of approximately 1-5% across tasks. This modest drop can be at- tributed to two main factors: first, the inherent discrepancy between noisier pretraining data and expertly curated benchmarks introduces less precise gradient signals for importance estimation. Sec- ond, LLaMA3-8B's weaker baseline in Chinese tasks means its optimization is more sensitive quality of importance source, and benefits more from highly targeted benchmark data. Nonethe- less, even with this gap, the pretraining-de data approach remains highly valid, especially in practical scenarios where access dedicated benchmarks is limited.
ADEPT, the pretraining-date variant shows slightly
In summary, ADEPT demonstrates strong effectiveness and robustness when layer expansion is guided by pretraining data, making it highly suitable for mance drop observed in LLaMA3-SB highlights the additional value of benchmark data for models tasks with substantial baseline limitations, but does diminish the overall utility of our method in aasource-constraiig settings.
deployment. The slight perfor-
I TOKEN DISTRIBUTION SHIFT
Following the methodology proposed by Lin et al. (2024), conducted comprehensive analysis of token distribution shifts between the base and aligned models using the MMLU (Massive Multitask Language Understanding) dataset. The analysis focuses identifying and quantifying the changes in token prediction patterns that occur during the alignment process.
Our analysis procedure consists of the following steps:
1) For each position in the input text, we use the aligned model with greedy decoding to generate the output token 0r-
40

---

Published as a conference paper at ICLR 2026
2) We then examine how this token is ranked in the base model's probability distribution Pbaae. This ranking, denoted as serves as our primary metric for categorizing token shifts.
3) Based on the base ranking 7). classify each token position into three categories:
Unshifted positions (17 1): The token is top-ranked in both base and aligned models
Marginal positions (1 < 77 S 3): The token has relatively high probability in the base
model
Shifted positions (17 3): The token is unlikely to be sampled by the base model
4) For shifted tokens, calculate Rank Improvement Ratio:
buse base_rank
aligned.rank
Our analysis of the MMLU dataset revealed significant distribution shifts between the base and continual pretrained models by ADEPT. Figure 14 visualizes the most significantly shifted tokens, where the size of each token is proportional its rank improvement ratio.
FDA
GDPR
r research Care hea th
Ca
arthritis
concentrations iespital
role
people
care
Hospital
ital Health
cases
illness
medi
redications
blood
th
News
disease
besi
healt
Lomized
zed
protein
pental
doctor
serious
researchers
medication
LOS
best
patients
i
Drug
r
Cance
ementia
MDICE
ICE
most
D
medical
i
ildenafil
ways
Hear
cancer
heal
thy
investigator
patient
t
reettment
cDC
gency
warning
Protein
emergence
hospital
diabetes
heart
ccnnitionn
kidney Baggosst
emer
data
pain
lot
diagnosed
RNA
drugs
n
s
symptom
oms
therapy
ization
1
of
redicinHHen
you
cell
1
th
stay
drug
g
study
ynec
po
brain
leading
MRI
ice
patients
results
significant
ipsiiontion
effects
tumor
UE
pres
mentall
MS
addiction
diseases
time
vaccine hospitals photographeffect
Figure 14: Word cloud visualization of shifted tokens. The size of each token represents its rank improvement ratio Larger tokens indicate more significant shifts in the model's prediction patterns.
aligned rank indicating the magnitude distributional shift during alignment.
basc_raank_
Our analysis of the MMLU dataset revealed significant and efficient distribution shifts between the base and aligned models. Figure 14 visualizes the most significantly shifted tokens, where the size of each token is proportional to its rank improvement ratio.
The analysis revealed notably efficient token distribution shift pattern. Specifically, only 2.18% of tokens underwent significant shifts (compared to 5.61% in full pretraining), with 18.78% remaining unshifted and 9.04% showing marginal changes (Totally 645496 tokens analyzed). This represents more focused and efficient alignment compared full pretraining scenarios, which typically show higher shift percentages (unshifted: 75.59%, marginal: 18.80%, shifted: 5.61%).
Most remarkably, the shifted tokens demonstrate clear concentration in medical terminology and medicine-related concepts. Key examples include: "prescription", "diagnosis", "symptoms", "dia- betes". "arthritis", "tumor", "MRI". "therapy", "treatment", "hospital", "care", "patients".
This specialized distribution stands stark contrast to the more general token shifts observed in full pretraining, where top shifted tokens (such as </im end|>, "CIF". "Registered",, "progression",
41

---

Published as a conference paper at ICLR 2026
"median") show particular domain focus and more noise. This comparison suggests that ADEPT achieved a more targeted and efficient knowledge injection, specifically enhancing the model's ' med- ical domain expertise while maintaining stability in other areas. The lower percentage of shifted tokens (2.18% 18% vs 5.61 %) combined with their high domain relevance indicates more precise and economical alignment process that effectively injects medical knowledge without unnecessary per- turbation of the model's general language capabilities.
These findings suggest that domain-specific alignment can be achieved with minimal token distribu- tion disruption while maintaining high effectiveness in knowledge injection. This efficiency in token shifting demonstrates the potential for targeted domain adaptation without the broader distributional changes typically seen in full pretraining scenarios.
Similarly, in mathematical domain alignment (Figure 15), we observed an even more efficient ken distribution shift. The analysis shows only 1.24% of tokens underwent significant shifts, with 91.51 % remaining unshifted and 7. 7.25% showing marginal changes. This represents an even more concentrated alignment compared to full pretraining (unshifted: 85.45%, marginal: 10. 10.18%, shifted: 4.37%)
The shifted tokens clearly reflect mathematical and scientific terminology. evidenced by terms such "theorem", "quantum". "parameters", "physics", and "equation". This highly focused shift pattern, utilizing merely one-third of the token shifts compared to full pretraining (1.24% vs 4.37%), demonstrates the effectiveness of our approach in precisely targeting mathematical knowledge in- jection while maintaining model stability in other domains.
effects $
bl
FTEFF
error
ex new
quant
PE
phenotype
code
he
profit open
ee
ob
space
paper
PhD
students
ariance
Sch
question
papp
ma
thematic
groups
scatter
physics
"
the
value
al
2
features
et scripts
Rig
{ chner 00
rieric
CM
ule
rii
algori thm
example
talk
Tony
S
Sized
Sizededitor
ormal
esium
ja
following
to
A
user
sense reaction
identity
p problem bit
pton
$$
Ag
papers
model
quantum
Oct
p connected
previous
results (sation
sh
case
de
(
udos
answer theorem
parameters
Figure 15: Word cloud visualization of shifted tokens in mathematical domain alignment. The pre- dominance mathematical and scientific terminology demonstrates the precise targeting of domain- specific knowledge.
J LINEAR MERGE OF DOMAIN-SPECIHIC EXTENSIONS: RESULTS AND INSIGHTS
In Table 15, compare the performance of the Vanilla model and the Merged Model, which was constructed by linearly merging the fomaiin-peccific extension layers (with equal weights of 0.5 for medical and mathematical domains) after independent training. Our results show that the merged
42

---

Published as a conference paper at ICLR 2026
Table 15:
Performance comparison Vanilla and Merged Models on multiple benchmarks (Qwen3-
1.7B and Qwen3-4B).
MMLU
CMMLU
GSM8K
ARC-E
ARC-C
MedQA
MMCU
CMB
Qwen3-1.7B
Vanilla
62.57
66.86
57.62
81.44
51.19
48.39
69.17
63.67
Merged Model
62.70
65.83
60.80
81.06
51.94
48.39
68.61
64.83
Qwen3-4B
Vanilla
73.19
77.92
69.07
85.52
59.13
62.77
82.44
78.92
Merged Model
72.96
77.99
73.16
85.27
58.96
62.83
82.83
78.42
model does not exhibit any significant collapse, and in some indicators even surpasses the original base model. For example, on the GSM8K benchmark for Qwen3-1.7B, the merged model achieves 60.80%, compared to 57.62% for the vanilla model. This demonstrates the generalization and ex- tensibility our method, enabling fusion across multiple vertical domains.
Our extension approach ensures that each newly added layer is separated by least one original frozen layer, rather than being directly adjacent. This design leads to greater stability during model merging. On one hand, if the merged models were purely cascaded, the non-linear transforma- tions introduced could lead to more unpredictable interactions between layers. On the other hand, because each layer operates within consistent contextual environment provided by surrounding frozen layers during continual pre-training, we believe that this fixed hierarchical structure imposes constraints that make the semantic representations learned by the new layers more aligned certain dimensions. As a result, the merging process becomes more reliable and beneficial to overall model performance.
It is worth noting that our merging strategy adopts the simplest possible weighted average. The specific merging algorithm the focus of this work; believe that with more scientific weight- ing schemes, even better results can be obtained. Here, we hope to stimulate further research and provide preliminary insights based on our observations.
K USE OF LLM
In the preparation of this article, we utilized large language models (LLM) solely for writing assis- tance purposes. Specifically, we employed the GPT-4.1 1-044.1-0414 model to polish language expressions, condense sentences, and improve the overall clarity and readability of the text. The model was used exclusively for editing and refining manuscript language and did not participate any conceptual or technical aspects of this work.
research ideas, theoretical proof methods, experimental designs, and visualizations were con- ceived. executed, and finalized by the authors without the involvement of any LLM tools. The development of new concepts. formulation and validation of proofs, experimental setups, analysis of results, and the creation of figures were performed independently by the research team. At no point was the L.M model used to generate, modify, or validate the scientific content, methodology, results presented in this article.
We emphasize that the role of GPT-4.1-0414 in this research was strictly limited to linguistic en- hancement the writing stage, and that all substantive intellectual and scientific contributions orig- inate solely from the authors.
L ALGORITHM
Please Algorithm 1.
43

---

Published as a conference paper at ICLR 2026
Algorithm ADEPT
Require:
Pretrained LLM Mo with layers {O(), ........ @(0), domain probing corpus D Probe con- tinual pretraining corpus Drriin number of layers to expand k, base learning rate Irasse update interval Tupdate
Stage Ceneral-Competence Guided Selective Layer Expansion
2: Compute base loss Lrase
[ Pproel Σx (2))
3: for + L do
4: Temporarily mask layer I to get Mb
5: Compute masked loss ((I) Dintel E. e(Mo (x),))))
6: Compute importance score (1) + (()) Cbasc
7: end for
8: Select least-important layers S < Lowestk((A
9: for each 1 Sx do
10: Duplicate parameters ( <- O
b. Identity copy
Initialize W/out MISA 0, W W/out
Function Preserving Init
Freeze original (1),, mark (9) as trainable
13: end for
Stage Adaptive Unit- Wise Decoupled Tuning
15: for each training step t do
16:
if mod updatte = then
17:
for each expanded layer ( do Partition into semantic units {UI...., Un for each unit Ug do
19:
20:
Compute gradient- based importance Tú -- [8:1 Ejeu, Vo,L
21:
Assign adaptive learning rate Iry, t 2- IU.) Irass end for end for
23:
end
25:
Sample training sequence (I1, (2.,, Drrain
26:
Compute autoregressive loss:
27:
C
- log P(xx Teci +)
28:
Update parameters (6(1) using adaptive learning rates (Irg,) }
29:
end for
M SENSITIVITY ANALYSIS IMPORTANCE-SCOMT UPDATE INTERVALS
To assess the effect of the update frequency in Stage 2, conduct systematic sensitivity anal- ysis on the interval at which unit importance scores are recomputed. While the main experiments adopt update interval of 500 steps, it remains unclear how sensitive ADEPT more or less fre- quent updates. To this end, we perform additional experiments on Qwen3-4BBBBB domain, evaluating intervals ranging from 10 5000 steps. This setup allows us quantify how dif- ferent recomputation frequencies influence model performance and overall training time. As shown in Table 16 and Table 17, we report both the medical-domain and general-domain performance of ADEPT under different update intervals, together with the corresponding training time (Descriptions of these newly added benchmarks and their evaluation protocols are provided in Appendix O). This allows us to clearly quantify how the recomputation frequency impacts domain-specific adaptation, general knowledge retention, and overall efficiency.
the medical
Interestingly, the most frequent update setting (every 10 steps) does not yield the best results. We hypothesize that this due to training stability: excessively frequent adjustments of unit-wise learning rates may introduce noise, as the importance scores are estimated via a first-order approx- imation and thus inherently sensitive to stochastic fluctuations. The performance of the 500-step interval is largely comparable to that of the 100-step interval across benchmarks, while the latter already offers a substantially reduced training time comparing to 10-step interval. This further indi- cates that the computational overhead of backpropagation doninates the overall training cost,
44

---

Published as a conference paper at ICLR 2026
Table 16: Sensitivity of ADEPT to the update interval for recomputing unit importance scores on in the medical domain.
Interval
PubMedQA (%)
MedQA (9)
MMCU (%)
CMR (%)
CMB-Clin (%)
Time
Oween-4B-Baas
Step-10
77.60
63.26
79.04
54.12
10h
Step-100
77.40
64.45
84.34
80.46
$3.28
12h
Step-500
77.20
64.49
84.58
79.87
54.40
11h
Step-1000
77.40
62.97
82.62
79.44
53.64
2d
11h
Step-5000
77.20
62.05
81.48
77.58
52.26
24 11h
Table 17: Sensitivity of ADEPT to the update interval for recomputing unit importance scores on Base on general-purpose benchmarks.
3wen -Bseaase
Inserval
TruhhfaQA-MCIII (%)
TrubbfulQA-MC2 (%)
CEval (%)
CMMLU (%)
BBH (%)
HellaSwag (%)
MMLU (%)
Time
Quen3-4B-Buse
36.84
53.38
79.49
77.92
70.73
55.41
73.19
Step-10
72.98
5d
Step-100
37.68
54.33
79.68
78. 06.60
71.51
55.21
73.01
24 12h
Step-500
38.31
54.63
79.00
78.77
71.08
55.35
72.95
24 11h
Step-1000 1000
37.21
53.39
78.90
78.65
70.70
54.26
72.43
2d 11h
Stsp-5000
37.09
53.31
79.05
77.68
70.66
$4.00
72.1
24 11h
and that the recomputation of importance scores is relatively lightweight. Notably, even an extremely infrequent update interval such as 5000 steps (i.e., only six updates throughout training) maintains competitive performance, highlighting the robustness of ADEPT.
We also oberserve the parameter importance during training and find that the overall distribution of importance across modules remains relatively stable after the initial update. Subsequent updates produce smaller changes, which explains ADEPT's stable effectiveness.
N APPLICABILITY TO SUPERVISED FINE-TUNING
To demonstrate ADEPT's transferability beyond continual pretraining, we further conduct Super- vised Fine-Tuning (SFT) experiments the medical domain on two widely used instruction-tuner and Qwen3-8B.
backbones,
SFT Experimental Setup. For the SFT experiments, we fine-tune each model the MMedBench training split (the SFT portion of MMedC dataset). We adopt mixed training regime that interleaves CoT-style samples (including detailed rationale) and non-CoT sam- ples (without rationales), with each item appearing once in both formats. We train with global batch size of 128, learning rate of 1 10-5 and all other optimization and regularization hy- perparameters identical to those used during CPT. LoRA applied with rank 128. The full SFT dataset contains 75,156 samples, and all models are trained for 3 epochs. For ADEPT, we employ a single-layer expansion and recompute uni--mmpottance scores every 100 steps, consistent with our CPT-stage procedures. To illustrate how each SFT item appears once in both CoT and non-CoT formats, we provide an example pair in Examples 4 and 5. Both correspond to the same underlying clinical question, with the former including detailed rationale and the latter containing only the concise answer.
Overall, the SFT results reveal several noteworthy patterns. First, lightweight adaptation methods such as LoRA and TaSL perform relatively well in the SFT setting, substantially better than in CPT. This reflects the suitability of parameter-efficient adapters for supervised updates performed on relatively small training corpora. Second, fully updating all model parameters (SFT-Full) yields consistently weaker performance, likely due to the large degree of parameter perturbation introduced by full-model fine-tuning the same magnitude gains as in the CPT experiments, it remains competitive and stable across both medical and general benchmarks. Given the limited size of the supervised dataset used here,
relatively small SFT dataset. Finally, while ADEPT does not exhibit
45

---

Published as a conference paper at ICLR 2026
with different adaptation strategies. "Vanilla" denotes the original pretrained model without any further training. All metrics are reported in %. Best and second-bes best scores for each metric are highlighted in bold and underlined, respectively.
Table 18: SFT results on -
Medical Domain
General Beachmarks
Method
PubMcolQA CMB-Clin MedQA
CMB
CEval
TruliQA-MCI
TruliQA-MC2
HellaSwag
Vanilla
78.90
62.68
57.36
52.40
51.04
35.12
51.03
51.91
67.90
56.32
65.30
55.74
SFT-Lora
Llama-Po Pro
78.20
51.44
61.74
57.46
51.66
51.26
35,98
51.26
51.78
66.76
57.06
65.24
TaSt.
76.40
53.14
61.32
59.85
52,30
51.76
36.32
51,86
51.76
67.62
57.73
64.64
ADEPT
78,90
53.76
63.92
60.85
53.81
51.70
37,10
52.10
$1.90
68.00
57.20
65.40
Table 19: SFT results on @wen3 8B with different adaptation strategies. "Vanilla" denotes the original pretrained model without any further training. All metries are reported in %. Best and second- best scores for each metric are highlighted in bold and underlined, respectively.
Benehmarks
Method
PubMadQQ
CMB-Clin
MedQA
MMCU
CMB
CEvaa
TrullQA-MC1
TrullQA-MC2
CMMLU
BBH
HellaSwag
MMLU
Vanilla
78.40
63.55
79.88
73.17
78.75
35.86
53,49
77.97
79.18
55.85
74.59
SPF:Pall
76.30
52.48
65.04
79.28
75.53
74.59
37.57
55.91
75.39
56.90
56.90
73.03
SFT-Lora
77.40
51.84
66.84
82.29
77.50
78.82
37,82
55.94
78.26
80.47
56. 12
74.87
Llama-Pro Pro
28.00
54.16
63.70
80.59
75.38
78.75
35.98
53.93
77.75
80.63
55.
74.59
75.65
56.92
ADEPT
we anticipate that ADEPT's benefits, particularly its targeted capacity expansion, would become more pronounced when scaling SFT to larger and more diverse training corpora.
Example 4
Problem: A 23-year-old pregnant woman at 22 weeks gestation presents with burning upon urination for one day, worsening despite increased fluid intake and cranberry extract. She otherwise well and is followed regularly for prenatal care. Vitals are normal (T 97.7%F, BP 122/77 mmHg, HR 80/min, RR 19/min, SpO2 98%). Physical examination shows no costovertebral angle tenderness and gravid uterus. Which of the following is the best treatment for this patient?
Analysis:
This patient is pregnant, afebrile, and presents with symptoms of acute cystitis, with no signs of pyelonephritis. Safe antibiotic selection during pregnancy is essential:
Nitrofurantoin is first-line therapy for uncomplicated cystitis in pregnancy (especially after the first trimester), with an excellent safety profile.
Ampicillin and ceftriaxone may be used for certain infections but not first-line for simple cystitis in pregmant patients.
Doxycycline contraindicated during pregnancy due to risks of teratogenicity and adverse effects on fetal bone and tooth development.
Given her presentation and gestational age, nitrofurantoin the safest and most appropriate therapy.
Therefore, the correct answer is Nitrofurantoin.
Example
23- 23-year-old pregnant woman 22 weeks gestation presents with dysuria and worsening urinary discomfort for one day. Vitals and examination are normal. Which of the following is the best treatment for this patient? Answer: D. Nitrofurantoin.
Problem:
46

---

Published as a conference paper at ICLR 2026
O EXTENDED EVALUATION ON ADDITIONAL BENCHMARKS AND DOMAINS
In this section, we incorporate a substantially more diverse collection of domain-specific and general-purpose benchmarks more rigorously evaluate the robustness and effectiveness of our method.
Extended Medical-Domain Benchmarks. To enrich the evaluation of medical-domain capabili- ties, we additionally include two complementary benchmarks. CMB-Clin (Wang et al., 2023b) is an open-ended clinical QA benchmark containing 74 complex consultation questions (each ques- tion will be followed by several sub-questions and will score each sub-question independently) across all major medical specialties. Each question is paired with a reference answer, and evaluation follows an LLM-as-a-judge protocol, where GPT-5 compares model outputs against the reference to compute a pairwise win rate to Vanilla model. (We omit the percent sign (%) in the presentation.) PubMedQA (Jin et al., 2019) a biomedical question answering dataset constructed from PubMed abstracts. Each question asks whether specific biomedical claim is supported by the evidence in the abstract, and the answer is one of yes, no, or maybe. The benehmark therefore evaluates a model's 's ability to read short biomedical abstract and make an evidence- baaeeee--addjeeee judgment. We report accuracy following standard practice.
Extended Mathmatii-Domainn Benchmarks. To more comprchensively assess improvements in mathematical reasoning, we alsp include two challenging benchmarks that target advanced problem- solving skills. GPQA (Rein et al., 2024) is multiple-choice benchmark of 448 expert- weprtewritten ques- tions designed to demand deep scientific reasoning and deliberate problem-solving. The questions are intentionally difficult, providing a rigorous testbed for evaluating whether Math-domain training improves a model's capacity for complex, expert-level reasoning. GSM-Plus (Li et al., 2024) is an adversarial extension of GSM8K that introduces controlled variations such as added statements and altered question targets to evaluate robustness in mathematical reasoning. These perturbations reduce reliance on pattern matching and require models to generalize their reasoning beyond sur- face cues, making GSM-Plus a stringent benchmark for assessing the stability of Mathmatic-domain training. We report accuracy for both benchmarks.
Extended General-Capabillty Benchmarks. To more fully assess the general-capability perfor- mance of our trained models, we introduce a broader set of diverse and challenging benchmarks that are largely orthogonal to our existing evaluation suite. These additional datasets provide more comprehensive evaluation of the model's overall general ability beyond the settings covered in our primary experiments. BBH (Suzgun et al., 2023) is a curated subset of 23 challenging tasks from BIG-Bench that remain difficult for LLMs. These tasks emphasize multi-step reasoning, abstrac- tion, and compositional generalization, making BBH a stringent measure of cross-domain general capability. HellaSwag (Zellers et al., 2019) is challenging commonsense inference benchmark created through adversarial filtering. Models must choose the most plausible continuation among highly confounding distractors. making it strong test of robustness in everyday reasoning. CE- val (Huang et al., 2023) is a comprehensive Chinese exam-style benchmark covering a wide range of subjects and professional knowledge areas. It assesses broad general-domain understanding and factual reasoning through multiple-choice questions. For BBH, HellaSwag, and CEval, we report accuracy. TruthfulQA (Lin et al., 2022) evaluates model's factuality by testing whether it can avoid reproducing widely held misconceptions. The benchmark comprises 817 questions across 38 domains, each constructed so that factually incorrect but popular answers are tempting. We report performance under both official TruthfulQA metrics. MC1 evaluates single-answer multiple choice: given a question and candidate options, the model must select the uniquely correct answer, and ac- curacy is computed over all questions. MC2 evaluates multi-answer probability assignment: given a question and sets of true and false reference answers, the score is the normalized total probability that the model assigns to the true answer set. MC1 thus measures strict answer accuracy, whereas MC2 assesses how much probability mass the model places on factually correct responses.
Table 20 reports the results under the Medical domain, where directly reuse the same medical- domain checkpoint from Table and evaluate it on additional medical benchmarks as well as broader capabiliy tasks. The results show that ADEPT maintains consistently strong performance on the newly introduced medical benchmarks, demonstrating robust domain adaptation. Notably, ADEPT also exhibits a clear advantage in mitigating catastrophic forgetting general-capability
47

---

Published as a conference paper at ICLR 2026
Table 20: Comparison ADEPT and baseline methods on broad suite of medical and general benchmarks. All results are obtained using the same medical-domainntrainn checkpoint reported in Table here evaluated on additional medical and general tasks to further assess cross-domain generalization. For ease of comparison, all metrics are uniformly mapped to the [0, 100] range.
Medical
General
Method
PubMedQA
TrubQA-MCI
TrutQQA-M
HellaSwng
LLaMA-3-8B Base
Vanilla
76.60
26.81
43.95
50.45
62.33
60.46
Full
77.00
56.84
23.75
37.74
49.70
47.47
58.03
Replay
78.20
59.20
29.50
44.15
52.75
51.48
58.71
Llama-Pro
76.40
54.92
24.85
38.03
49.70
62.31
60.00
PT-LoRA
73.20
53.48
24.24
38.16
49.93
61.77
$9.79
TASL
26.35
ADEPT
78.20
59.70
28.40
45.03
52.11
62.52
60.39
@went: 1.78 Base
PT-Full
70.20
51.92
29.87
45.24
62.78
46.38
48.73
Replay
69.40
52.76
31.82
46.28
63.19
49.17
48.30
Llama-Pro
70.40
$2.88
28.27
42.42
60.03
37.37
48.74
PT-LoRA
68.40
48.08
26.81
42.53
64.04
37,87
48.76
TASL
68.00
45.67
25.95
40.83
59.88
36.15
47.15
ADEPT
69.60
53.84
34.39
66.64
$2.96
Qwven3-4B-Bac
Vanilla
73.60
36.84
53.38
79.49
70.73
55.41
PT-Full
76.60
52.64
33.05
47.10
73.11
63.22
53.15
Replay
76.40
51.64
33.05
48.83
73.18
65.32
50.79
Llama-Pro
76.60
52.16
34,27
77.71
69.37
52.82
PT-LORA
74.00
51.36
33.90
78.01
58.02
53.54
TASL.
ADEPT
77.20
54.40
38.31
54.63
78.60
71.08
55.35
Qwen3 -B--3---B-e-ee
Vanilla
77.40
35.13
82.91
Full
78.80
50.36
32,93
47.41
80.09
68.81
56.46
Replay
79.00
$2.46
32.93
48.50
79.72
69.73
55.75
Llama-Pro
78.60
51.64
35.74
52.04
78.16
71.80
54.87
PT-LORA
78.00
$1.16
32.56
48.58
81.20
71.11
$7.33
TASL
78.20
52.40
31.64
49.92
80.79
70.14
58.39
ADEPT
79.00
52.88
37,58
53.91
82.54
76.33
59.05
benchmarks, outperforming baseline methods by a large margin. Table 21 summarizes the results under the Mathematical domain. We observe similar trends: ADEPT retains strong mathematical reasoning performance on extended math benchmarks while preserving general abilities more effee- tively than competing approaches. These findings further validate the robustness and effectiveness of ADEPT across heterogeneous domains.
CODE-DOMAIN EVALUATION
To further examine the applicability of ADEPT beyond natural-language and scientific domains, we additionally conduct experiments the Code domain. Due computational constraints, we select Qwen3-44 -4B as representative model and perform continual pretraining on a Python-only corpus constructed from two public datasets: Meallow-Codee v2 and
-oonnn-odeepee The
'https: http://r..../aaaaaaaaatooooeeeee http:/asssttt/tooootceehceceeeee 11m swal low- code-v2 "https://hf-min rror www.rroo....oooooooooooooooo wtttsssssssschtccteeeecc httphhhhhhhhhooooooo www.pyyttooooo----ooeeeooooeeeee codes- 25k
48

---

Published as a conference paper at ICLR 2026
Table 21: Comparison ADEPT and bascline methods on broad suite of medical and general benchmarks. All results are obtained using the same mathmatical-dmmain-rre reported in Table 1, here evaluated on additional medical and general tasks to further assess cross- domain generalization. For ease of comparison, all metrics are uniformly mapped to the [0, 100] range. GSM+ refers to GSM-Plus benchmark.
checkpoint
Math
General
Method
GPQA
GSM+
TQA-MCI
TQA-MC2
CFval
BRH
HelloSwag
11MMA-3-8
60.46
PT-Full
23.99
33.19
29.13
45.09
62.93
55.60
Roplay
27.23
33.62
29.25
44.14
51.20
62.51
56.10
Llama Pro
29.54
26.68
43.22
50.96
63.51
58.65
PT-LoRA
25.22
29.90
27.78
43.46
50.14
62.75
59.11
TASL
24.86
30,26
27.71
41.68
49.68
62.37
58.91
27.78
28.78
44.31
60.77
Owen3--177--Bal
Vanilla
26.12
50.16
32.19
48.80
65.53
53.05
49.15
PT-Full
27.23
52.11
29.87
49.73
62.85
48.20
48.99
Replay
27.34
52.32
33.23
51.23
62.43
48.93
49.04
Llama-Pro
27.01
54.37
32.
49.13
62.26
51.45
48.9
PI-LORA
46.97
48.4
ADEPT
31,02
54.82
33.65
50,00
66.27
54.40
49.17
Qwen3-44-BBr Baze
Vanilla
27.68
61.62
36.84
53.38
79.49
70.73
55.41
PT-Full
26.44
57.37
31.82
47.83
74.07
68.80
50.61
Replay
26.10
57.87
32.06
49.14
75.48
69.73
51.94
Llama-Pro
27.23
61.74
33.05
49.44
77.27
71.11
52.34
LT.L
26.89
60.26
31.33
47.31
77.71
71.80
51.64
TASL.
26.86
62.63
32.31
48.18
77.43
70.83
53.23
ADEPT
29.90
63.76
38.31
54.25
28.26
72.21
55.29
Qwenn3-88- 88-Baase
Vanilla
35.26
63.42
35.13
52.29
82.91
76.69
$9.25
PT-Full
35.66
64.54
32.93
49.83
79.42
73.15
55.83
Replay
34,55
65.14
34,27
50.18
78.08
74.62
55.64
PT-LORRA
33.78
79.81
54.86
TASL
35.19
65.58
34.16
49.38
77.98
75.55
$4.42
ADEPT
38.91
66.72
38.18
53.39
82.39
79.83
59.16
former provides large, professionally curated Python corpus, while the latter contributes high- quality SFT-style Python tasks covering code generation, code-oriented natural language under- standing, behavior analysis, and educational coding variations. Combined, the corpus contains ap- proximately 13.7B tokens (~ 4.2 x 10s samples of length ~ 512 tokens), enabling controlled examination ADEPT's behavior when adapting models to specialized programming domain.
For downstream assessment, we adopt three widely used code domain benchmarks, all evaluated using the standard pass@k metric: HumanEval (Chen, 2021), functional correctness benchmark for code synthesis; MBPP (Austin al., 2021), curated set of introductory-level programming tasks; and CRUXEval (Gu et al., 2024), which tests execution- correctness across diverse constraint-solving problems. Together, these benchmarks provide comprehensive view of coding capability under continual pretraining in the code domain.
Table 22 summarizes the results of Code-domain continual pretraining. Across the three code domain benchmarks, ADEPT demonstrates the most balanced and robust performance among lightweight adaptation methods. While PT-Ful 1 and Repl ay benefit from updating the entire
49

---

Published as a conference paper at ICLR 2026
Table 22: Code-domain and general-capability results of Qwen3-4B-Base under different CPT strategies. All metrics are mapped to the [0, 100] range. Best and second-best results for each column are highlighted in bold and underlined, respectively. TQA means TrukfulQA.
Code
General
Method
HumanEvall
MBPP
CRUXEval
TQA-MCI
TQA MC2
HollaSwag
BBH
CEval
passin
passirt
passér 10
passie1
855
pass 10
passe8
passils
passit
Vanilla
50.37
78.61
86.
8.71
12.42
49.80
48.95
76.69
34.69
36.84
53.38
55.41
70.73
79.49
Pr-Pull
50.79
83.85
89.02
26.50
58.62
66.77
46.64
72.62
80.18
32.07
48.29
$2.30
63.94
71.77
Roplay
53.23
85.37
Lama-Pro
49.70
79.69
67.10
20,86
45.86
59.81
42.81
70.65
78.42
54.27
51.97
$3.99
66.29
77.27
PT-LoRA
36.77
74.43
82.93
28.17
61.64
70.04
39.65
70.97
79.80
31.82
49.28
51.94
34.48
75.85
TSSL
37.42
48.70
52.67
ADEPT
51.13
84.81
87
20
31.17
55.07
67.98
74.57
83.69
35.13
53.83
$4.98
70.77
79.63
Table 23: Multilingual medical and general-capability evaluation of Qwen3-4B 41-03---B-ass- Base after con- tinual pretraining multilingual medical corpus. All metrics are mapped to the [0, 100) range, and best/second-bes. results are highlighted in bold and underlined. Language abbreviations: ES (Spanish), FR (French), JA (Japanese), RU (Russian), EN (English), CN (Chinese). TQA means TruthfulQA.
Multilingual Medical
Multilingual General
CNVEN General
MMeeBench
MMMLU
TQA-MCI
TQA-MCZ
MMLU
CMMLU
CEval
Method
ES
FR
JA
RU
ES
FR
JA
RU
EN
EN
EN
CN
CN
Vanilla
75.08
77.92
PT-Full
73.34
75.40
66.33
59.68
61.77
60.86
56.91
59.68
30.60
43.97
66.91
68.88
69.61
Replay
74.47
72.83
62.31
59.54
63:47
62.33
58.19
59.54
30.72
45.21
68.39
72.19
71.62
Llama-Pro
73.74
74.43
53.27
60.70
65.32
64.43
59.79
60.70
35.86
52.54
72.03
77.08
77.79
PT-LoRA
72.36
73.95
57.29
61.33
64.22
63.56
58.26
61.33
30.82
46.75
70.27
72.52
74.89
TaSL
71.22
73.88
56.45
61.58
65.49
62.59
58.97
60.49
30.70
45.67
70.40
71.65
76.98
ADEPT
74.56
74.92
62.26
74.61
66.64
66.15
61.01
63.19
36.60
54.01
73.69
78.67
28.63
model. ADEPT achieves competitive results despite modifying only four layers, and consistently outperforms L.1 sama-Pro under the same expansion budget. LoRA shows clear instability across tasks, whereas ADEPT maintains stable performance across HumanEval, MBPP, and CRUXEval. In addition, ADEPT preserves general capabilities substantially better than full-parameter CPT and Re- play, matching exceeding Llama Pro on most general benchmarks. Overall, ADEPT provides a favorable trade-off, combining strong performance, stable generalization, and parameter efficiency.
Q MULTILINGUAL MEDICAL EVALUATION
In Table 1, the continual pretraining data are predominantly English and Chinese. To further strengthen the domain-transfer setting and evaluate cross-lingual robustness, we extend our study to multilingual medical tasks. This setting introduces substantially larger linguistic and domain distribution shift, providing stringent test of whether ADEPT can retain its effectiveness under multilingual medical evaluation.
For CPT, we construct a multilingual medical corpus by combining the multilingual portion of MMedC with the training split of the multilingual portion of MMedBench, explicitly removing all English and Chinese samples and retaining only the remaining languages. The resulting cor- pus comprises approximately 16B tokens spanning diverse medical subdomains and typologically varied languages. For evaluation, we use the multilingual section of the MMedBench test set to measure multilingual medical capability, and MMMLU (Hendrycks et al., 2020) to assess multilingual general capability. Finally, we report English and Chinese general capabilities using TruthfulOA, MMLU, CMMLU, and CEval.
50

---

Published as a conference paper at ICLR 2026
Table 24: Layer-importance rankings for General, Math, Medicine, and calibrated variants (lower index = higher importance).
Setting
Layer Importance Ranking
General
1.
10, 13, 26,
17.
24
25,
Math
0,2,9, 4, 3, 6. 16, 15.
13,
20, 24,
18,
19,
22,
27,
23,
Medicine
2, 6. 1. 9, 17, 11 15, 21, 10, 8, 16, 14, 13, 12, 18, 20,
24,
19,
26
22,
23,
27, 25
Math (Calibrated)
16, 15, 6, 8, 27, 24, 7, 9, 5, 14, 4, 11, 10, 18, 19, O, 20, 21,
23,
12,
22,
13,
26
17.
3.
25,
Medicine (Calibrated)
16, 6, 7. 15, S. 23, 9. 10, 8, 11,4, 14, 18, 19, 20, 21, O, 27.
12, 2.
22,
25,
17.
13, 2,
Table 23 presents the multilingual medical and ceneral-capabilin evaluation results. A clear pattern emerges in the Japanese subset of MMedBench, where both PT-Full and Replay obtain the strongest scores among all methods. This aligns with the fact that the Japanese portion exhibits a relatively large distributional shift, and full- hull-parameter as well as replay-based strategies tend to absorb such shifts more directly, yielding higher task-specific performance. However, this adaptation is accompanied by noticeable decline in multilingual general capability and in CN/EN evaluations.
In contrast, ADEPT maintain consistently stronger CN/EN general performance, showing minimal degradation on TruthfulQA, MMLU, CMMLU, and CEval. Among all evaluated methods, ADEPT achieves the most balanced multilingual behavior. Its importance-aware expansion and decoupled tuning effectively limit overfitting to individual languages while preserving broad generaliza- tion, resulting in stable improvements across multilingual medical benchmarks and multilingual as well as CN/EN general tasks.
R DOMAIN-CRITICAL GENERAL-NONCRITICAN WHICH LAYERS SHOULD BE EXPANDED?
Motivation for Expanding General-Nonerrtical Layers. The design choice in ADEPT to ex- pand layers that are least important for the general domain is grounded in several considerations. First, injecting new domain knowledge is relatively easy for gradient-based optimization, whereas catastrophic forgetting is far more harmful and substantially degrade model abilities; prior work (Dai et al., 2022b; Geva et al., 202 has shown that domain-specific fine-tuning may improve in-domain performance but often causes large drops in general skills. Thus, ADEPT intentionally minimizes interference with general-criticll parameters. Second, new domain-specinn information may overwrite existing domain knowledge (e.g., medical factual updates), meaning that even ac- curately identifying domain -imorrat layers does not guarantee preservation of previously learned domain-specific representations.
contrast, general-critical layers correspond to knowledge that must remain stable, making them more reliable target for protection. Third, domain-importann lay- ers frequently overlap with general-critical layers due to shared semantic structures across domains, expanding them still risks inducing forgetting. Finally, in practical settings, high-quality domain data may be insufficient to reliably identify domain-critical units, whereas provide safe and universal expansion region that works consistently across different domains.
units
Probing and Expanding Domain-Critical Layers. To analyze whether expanding domain- critical layers could further improve adaptation, we first probe the importance distribution for the mathematics and medical domains and examine their overlap with the general-critical set. The prob- ing procedure follows exactly the same strategy used for identifying general-critical layers. For the Medical domain, the probe corpus consists of 500 Chinese and 500 English samples drawn from the training split of MMedBench; for the Mathematics domain, we use the math-500 dataset.
Table 24 reports the probed inyorrimpoetance rankings for the General, Math, and Medical do- mains. We observe substantial overlap across these three distributions, with early layers consis- tently ranked highly important in all domains. This pattern indicates that early transformer layers encode fundamental semantic representations shared across tasks and domains. As a result, directly expanding comminn-mmortaann layers would risk interfering with these shared representations, making them unsuitable as expansion targets during domain-specific continual pretraining.
htttps:/hugginng
htt.//taasssttteeeeeeeee /taeseeesttttgggggee Huggi a /MATH-500
51

---

Published as a conference paper at ICLR 2026
Table 25: Comparison of Math-domain and General performance between domain-critical expan- sion (calibrated) and least-ge least-general-ee geeegee-eerriicceee expansion (default ADEPT) on Qwen3-1. 7B-Base. All metrics are mapped [0, 100). Best and second-best results per column are marked in bold and underline, respectively.
Math-Domsin Benchmarks
General Benchmarks
Method
GPQA
GSM
GSMBK
ARC
ARC-2
BBH
TQA.MCI
TQA. (QA.MC2
CEval
HellaSwag
MMLU
CMMLU
Owes3 1.78 Base
26.12
50.16
57.02
91.44
51.19
53.05
32.19
48.80
65.53
49.15
62.57
66.36
ADEFT (least peneral-cttical critcal expension)
31.03
$4.81
70.51
82.48
52.60
56.40
3366
50,00
66.27
49.17
62.62
67.06
APRPT (demsineriical exguncion)
31.79
52.47
71.02
81.78
8384
55.36
32.82
65.16
48,33
66.26
Table 26: Comparison of Medical-domaile and General performance between domain-critica expan- sion (calibrated) and least-gen general-criticee expansion (default ADEPT) on Qwen3-1, 7B-Base. All metrics are mapped to [0, 100). Best and second-best results per column are marked in bold and underline, respectively.
Mrdical-Dhomain Benchmarks
General Benchmarks
Method
PahMalQ4
CMB-Clis
MalQA
MMCU
CMB
TQ4-MC1
TQ4-MC2
HellsSway
MMLU
CMMLU
Owea3-1.78-Bos
60:20
48.39
69.17
63.67
32.19
48.50
63.53
62.57
ADEPT Geast general crttical expansion)
69.60
53.84
90.75
71.98
65.43
34.39
51.05
56.64
49.28
62.50
66.89
ADEFT (demiin critical expansion)
67.00
69.70
72:55
66.82
32.31
48.30
66.90
46.30
61.69
66.15
Therefore, we introduce an importance calibration technique: for each layer we subtract its general- domain importance rank from its domain-specific importance rank, and use this calibrated score to identify layers that are important for the target domain but not important for the general do- main(see the calibrated results in Table 24). We then perform layer expansion on this calibrated domain-importan subset, while keeping the same decoupling strategy as in ADEPT. In subsequent experiments, we refer this variant as domain- critiicll expansion.
Table 25 and Table 26 summarize the empirical results comparing calibrated domain-critical ex- pansion with the original ADEPT strategy of expanding the least important layers for the general domain, across both Mathematics and Medical CPT settings. We observe that domain-critical expan- sion yields slightly stronger specific importance can indeed enhance in-domain adaptation. However, this improvement consis- tently comes at the cost of greater degradation on general benchmarks. This behavior reflects an inherent trade-off between aggressive domain adaptation and preservation of broad general abilities.
injection, demonstrating that focusing on domain-
These findings reaffirm the motivation behind ADEPT's original design: expanding the least im- portant layers for the general domain minimizes interference with general-critical parameters and reduces the risk of catastrophic forgetting. While domain-critical expansion is a useful alterna- tive when stronger domain specialization is desired, the reneral-noneriticn strategy offers a more balanced and stable solution. We have therefore included the domain-critical expansion variant as an optional configuration in the released ADEPT codebase to support both adaptation preferences.
In the figures below, we present the domain importance profiles across all layers for both the medical and mathematical domains, including both the original (uncalibrated) and calibrated attention vari- ants. Here, parameter importance may be more sensitive to fine-grained calibration, so we primarily focus on layer importance. We can observe that without calibration, the similarity in importance dis- tribution across general, math, and medicine domains is remarkably high. We can observe that the early layers related to semantic understanding are of paramount importance across all tasks. Notably, from layer-wise perspective, the medical domain exhibits important layers distributed throughout all layers, likely because medical tasks demand a complex combination of reasoning, factual recall, and contextual understanding. In contrast, the mathematical domain shows concentration of im- portance in the middle or slightly carlier-middie layers, suggesting that the core computational and deductive processes for mathematical reasoning are primarily localized in these regions.
52

---

Published as a conference paper at ICLR 2026
84
80
20
22
Layer
Layer
16
10
14
12
10
10
2
Layer Importance
self, attmat proj
self atteec srorj
catt Latt horm
arto mog
mij drren... pny
dell artin rorm
Inport, lapemam
pood at intentioon lopertom
ripo getto prr
snff sitss poo
Layer Importanco
mig dovnnntuuu pvay
mipp pvaj
maj gast,, prof
snit attr o sway
cuif aetm Juvoj
salf sttn suaroj
post attentorn Jeyerner
calf ottn K, pooj
Ixport Jayerrom
Parsmeter
Perameter
Llama3-8B- on Math (calibrated impor-
(a) Llama3-8B- 3B-Basc on Math (raw importance)
tance)
30
30
20
28
25
28
24
24
20
20
Layer
Layer Impertance
mip guteeateeaa proy
colf attt mmmm
inpist, Aapermmm
aust atto (rarr)
nelt istm s_rra
prstt attertiont Strermom
depsa... .oro
self, attuk mrrr
Layer inpertaroe
meppdtownt.o seral
mijp srroj
onlo gista proj
ewti ottoo sorof
sent attin sorfl
unti wrtttnttin proj
bost attertionn Aeyemorm
aelt artudt .Pof
Ingust. Aayemamm
roia dereettt.cc
Parameter
Parameter
(d) Llama3-8B- BB--BBs on Medical (calibrated im- portance)
(c) Llama3-8P 8B-Base on Medical (raw importance)
Figure 16: Importance visualization for Llama3-8B 8B-Base across Math and Medical domains, with and without calibration.
53

---

Published as a conference paper at ICLR 2026
27
25
22
23
21
18
10
Layer
Layer
10
12
11
Layer importanoe
mijup, piaj
calp diodonn,, soay
only artin mayy
mhip gate Jaaj
east articc (Poj
owtf atto anoo
proat attertior. layernam
Inpoit Eyprnam
att. nomm sell atinn norm
Layer Insprtaance
rigo sewent proj oel alte, prrj
sett Lustis horm
dett ont a.hom
selit attin.t.. cL/m
soro)
atirn proj
powe attentoo Layoroom
reip crro
isport lnyerrm
gots proj
sott artnnk
(b) Qwen3-1.7B-Ban on Math (calibrated impor- lance)
(a) Qwen3-1.7BB 1.7-3---.BBBBeeee on Math (raw importance)
27
28
28
27
23
23
22
22
19
19
18
Layer
Leyer
12
Layer Importeronn
migus poof imp down piog
csio gats, pirol
owt metooo .oro seit atin srof
oulf atto orry
suit artrck pooj
postt altontonn, byenomi
topuut Layornom
anii auittnng.ee nomm norm
sett ettnk nerm
Layer Insortanca
aeti Luttn raom
catt ettn a Laarn
mijedonm sPof
enlt attrutt soro
knit astes proy
attes .rr//
fnyuut Joyarorm
sest. stststts oo/o
post attentionn Jayemon
inio proy
mis gants, poof
Parameter
Parameter
(c) Qwen3-1,7B-B.an on Medical (raw impor- lance)
(d) Biven3-1.7--BBB 18-Basc on Medical (calibrated im-
portance)
Figure 17: Importance visualization for Qwen3-1.77 1.7n3--1.7B-BBB B-BB across Math and Medical domains, with and without calibration.
54

---

Published as a conference paper at ICLR 2026
82
30
22
Layer
18
Layer
14
14
10
10
Layer Importesce
owit attre ettrs...rr)
selt attoo s.rr
aetf auttting rorm
atti pooe
mip down orej
arth room
mig pro
Irpast, Japerromm
poott attenttion loyensem
gnts proy
nelf attns poov
Layer Importanc annc
mip ripddowr.um srroj
mipuup. cooo
mijp araj
aslt, aetin sory
calk attnuz srrj
aeli.atii aatiinn att.ing..o aveyy
post, affention Eapernory
atttu, praj
tnpuat Javerom
anit attan rorm
only attrrnk foom
Parameter
Parameter
Qwen3-4B-BB Math (calibrated impor-
(a) Qwen3-4B-BBl on Math (raw importance)
tance)
34
34
30
30
26
24
22
20
18
14
10
Layer Importancca
koll actruu proj
snei artunt prof
selt attmk..rmm eoor
muit arter Ereer
atto proj
irguit, romm
ont wtt.kkkkkk prof
mig un prof
mnpo dlow ofro
pouit artenstun Joyerron
gate. proj
Layor importance
mip dowet, mip up srmj
intp gate proj
kmif atte proj
etts proo
Gelz artn .uroj
lepust teyerem
intonntton, Jeveram
out orth norm
artn norm
Peremeter
Paramater
(d) Qwen3-4B-BB on Medical (calibrated im- portance)
(c) Qwen3-4B-Banl on Medical (raw importance)
Figure 18: Importance visualization for Qwen3-4B- Base across Math and Medical domains, with and without calibration.
55

---

Published as a conference paper at ICLR 2026
9
18
52
28
12
%
18
8
18
Layer
88
8
10
Layor Importanne
salff any
self atte sLarg
inpust Impur.layyrne Ltyornons
nett aite Lavs)
entt Latt- norm
bett ettri morm
proj
mig SapddnLLor
ripuup, pro
ottention Jayemonn
Layer Ingortaacce
mip peor
mipus army
mip gate, proj
malt asto oney
self attris proj
post eittention lay peroomm Expul, lopertort
self attiitt, pkay
colf with sorm
eatl alttuut mtom
saft adlio
Perameter
Parameter
(b) Qwen3-8B-Bil on Math (calibrated impor- tance)
(a) Bwee3--B-Basse Math (raw importance)
34
92
30
28
28
86
84
24
20
Layer
26
12
Importasee
dulf aattto srri
seit Lutee LLave
inpor mommr
colt stom
atts svaa
anet atimn sivo)
and atteul rarm
mit mitutoon aore
Jost attreninn Jayemom
mip aroy
mip gate stol
Layer importaroe
máp droen proj
mip proj
mia pano. crry
euif attnoo. sorol
eett att.attmm may
nett ott sarol
aitantion Syprroo
inout, Inyeroom
and atting Form
awlf ottn norm
east with
Parameter
Parameter
(d) Qwen3-8B-Base on Medical (calibrated im- portance)
(c) Qwen3-8B BB-nn---BBBseeee on Medical (raw importance)
Figure 19: Importance visualization for Qwen3-8B- 8B--aBs across Math and Medical domains, with and without calibration.
56

---

Published as a conference paper at ICLR 2026
Table 27:
Comparison of Math-domain and General performance between different zero- Initialization strategies on Qwen3-1. 7B-Base. All metrics are mapped to [0, 100]. Best and second- best results per column are marked in bold and underline, respectively.
Math-Domuin
Benchmarks (%)
General Benchmarks (
Method
GPQA
GSM+
GSMSK
ARC-e
ARC-c
BRH
TQA-MCI
TQA-MC2
C.vval
HellaSwag
MMLU
CMMLU
Qwen3-1.7B. Bwen
26.12
50.16
57.62
81.44
51.19
53.05
32.19
48.80
65.53
49.15
62:57
ADEPT
31,03
54.83
20.51
82.48
52.62
54,40
33,66
50.00
66.27
49.17
62.62
67.06
ADEPT_apppe pppppprojcctoonn
29.01
53.14
71.11
82,10
51.62
51.82
32,68
47.92
66.34
48.54
62.32
66.72
ADEPT gate projecttit
26.89
54.19
69.58
81.81
51.79
52.53
33.29
47.88
66.20
48.61
62.28
66.33
S ANALYSIS OF ZERO-INIIALIZATIIN STRATEGIES IN MLP EXPANSION: UP-PROJECTION, DOWN- WWWN-PROJETTOO, AND GATE PROJECTION
S.1 PRELIMINARY OBSERVATION: DOMINANT IMPORTANCE MLPs AND DIFFERENT ROLES OF PROJECTIONS.
The dominance of MLP layers observed in Figure is not merely consequence of their param- eter scale, but also reflects their intrinsic representational role in storing and conveying factual knowledge, as supported by growing body of interpretability research. Prior work has shown that feed-forward networks in transformers behave as key-value memories (Geva et al., 2021b), that individual "knowledge neurons" in FFNs encode relational and entity-level facts (Dai et al., 2022a), and that modifying mid-layer FFNs can directly alter factual outputs during generation (Meng al., 2022). Additional studies further demonstrate that specific MLP pathways are repeatedly reused when reasoning with particular facts (Yao et al., 2024), reinforeing the view that MLPs serve as central repositories of semantic and factual information. Complementing this, large-scale analy- indicate that the factual capacity of MLPs grows linearly with parameter count (Nichani al., 2025), underscoring that their quantitative dominance aligns with their qualitative representational role. Together, these findings explain why ADEPT's importance probe naturally highlights MLPs as high-value components for preserving and modifying general-domaii competence.
Building on this understanding, zero-initializing the MLP down nown-projection (the output projection) emerges as principled and effective design choice. First, this choice is consistent with established model-expansion techniques such as LLaMA-Pro, which similarly employ zero-initialized output projections to guarantee strict function preservation. Second, zeroing the down-projection ensures that the replicated MLP branch remains a lossless adapter, preserving the original key-value map- pings without interfering with existing knowledge; formal justification is provided in Appendix F. Third, zero initialization actively facilitates knowledge acquisition: because ADEPT's decoupled update mechanism assigns learning rates inversely proportional to parameter importance, zeroing the new projection reduces its initial importance and thereby increases its learning rate. This allows the expanded region to rapidly absorb information, matching the established view that MLPs act as the primary storage units for factual knowledge. Finally, zero-initializing the down-projection is particularly suitable for domain-specific knowledge updates. Since the down-projection corresponds to the "value" component of the MLP memory, resetting it permits efficient learning of new values while preserving the semantic keys encoded by the up-projection. This structure desirable for controlled knowledge rewriting (e.g., updating medical facts), where modifying values while main- taining stable keys essential. In contrast, zeroing the gate projection provides minimal semantic capacity for learning, and zeroing the up-projection disrupts key retrieval, making both alternatives less suitable for reliable and interpretable knowledge injection.
In summary, the empirical importance of MLPs, their theoretically grounded role key-value mem- ories, and the optimization dynamics introduced by ADEPT jointly motivate zero-initializing the down-pron pwojecroeettion as the most principled and effective strategy for stable, function-preservinn preserving model expansion.
S.2 EXPERIMENT ON DIFFERENT LAYER EXPANSION STRATEGIES
To validate the correctness of our theoretical conjecture above, we employed different initialisation methods for layer expansion.
57

---

Published as a conference paper at ICLR 2026
Table 28: Comparison of Medicine-domain and General performance between different zero- Initialization strategies on Qwen3-1. 7B-Base All metrics are mapped to [0, 100]. Best and second- best results per column are marked in bold and underline, respectively.
Medical- Domain Benchmarks (%)
General Benchmarks (%)
Method
PahMocIA
Media
MMCU
CMB
TQA-MC2
CEval
HellaSwag
MMLU
CMMLU
Qwen3-1.7B 1.r3--.7--Bss-
69.20
48.39
69.17
63.67
53.05
32.19
48.90
65.53
19.15
62.57
66.85
ADEPT
60
53.84
50.75
71.98
65.43
52,96
34.39
51,05
66.64
49.25
62.80
66
72.60
71.62
65.04
52.56
66.57
67.84
ADEPT gate projection
68.00
52.76
49.18
71.16
64.39
52.57
32.93
47.83
65.97
48.51
62.60
67.02
In our experiments, we observe that zero-initializing either the MLP up-projection or down- projection yields consistently strong performance across both mathematical and medical bench- marks, whereas zeroing the gate projection leads to a clear degradation in accuracy (Tables 27 and 28). This discrepancy arises from the distinet functional roles these projections play within the MLP and their interaction with the key-value structure of knowledge storage.
The up- and pewn-pprjections jointly form the semantic key-value mapping of the MLP: the up- projection builds high-dimensional semantic keys, while the down-projection retrieves values asso- ciated with those keys. Zero-initializing either side preserves the original key-value mapping of the pretrained MLP.
When the down-projection is zeroed, the expanded branch produces no output at initializa- tion, thus strictly maintaining functional equivalence with the original model. Meanwhile, the up-projection remains intact and continues to generate meaningful keys, enabling the expanded subspace learn new value information. This behavior aligns with prior prelim- inary research on controlled knowledge injection and with theoretical views of MLPs as fact-storage modules.
When the pp-projection is zeroed, the semantic keys of the expanded branch reinitial- ized, while the original MLP's key-value mapping remains untouched. Although this limits the ability of the expanded dimensions encode richer key structures, the overall perfor- mance remains comparable to zero-down initialization, consistent with the small empirical gap observed in Tables 27 and 28.
In contrast, the gate projection plays fundamentally different role from the up- and down- projections, which primarily modulate the dynamic importance of activation channels and govern amplitude-level feature selection, rather than providing semantic space suitable for storing mod- ifying knowledge. Zeroing the gate projection, therefore, does not offer the expanded dimensions any meaningful representational capacity for knowledge injection rewriting. This misalignment is reflected in the empirical results: zero-gate consistently underperforms zero-up and zero-down across mathematical reasoning tasks and medical tasks in Tables 27 and 28. In contrast, zero-up and zero-down preserve the core semantic pathways of the MLP, enabling the expanded dimensions learn new information effectively. Consequently, these results reinforce the distinct functional decomposition of MLPs into value mapping and gating components in our preliminaries, and they offer practical guidance for selecting initialization strategies in ADEPT's layer expansion.
58


Reviews:
[{"review_id": "iKeuhnWws4", "reviewer": "Reviewer_E9mT", "summary": "The paper proposes Adaptive Expansion and Dynamic Decoupled Tuning for domain-adaptive CPT (ADEPT). The authors utilize a two-stage training process:  1) General-competence-guided selective layer expansion, and 2) Adaptive unit-wise decoupled tuning. \n\nIn the first stage, the authors identify which layers are important for general knowledge and which are for domain knowledge by using curated probing data. In the second stage, they modularize the computation units and identify which units contribute to domain knowledge, updating these units with adaptive learning rates determined by gradient values. The authors use Llama3.1-8B and Qwen3 as base models and perform CPT for the math and medical domains. They experiment with model sizes ranging from 1.7B to 8B parameters and show that ADEPT outperforms the baselines.", "questions": "- Does Table 6 in Section B.8 represent the end-to-end time from probing to training?  \n- Is the proposed method applicable to SFT and RL?  \n- How did you prepare $D_{\\text{probe}} $?  \n- What is \"Vanilla\"? Is it the base model without CPT?  \n- Why is Vanilla better than PT-Full and Replay in the domain benchmarks for the 1.7B and 4B models? If training is properly executed, shouldn't they at least be better than the base model on the domain benchmarks?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written and easy to understand.  \n- The proposed method is generally well-motivated.  \n- The experiments are performed on various model sizes and show that the proposed method outperforms the baselines.", "weaknesses": "- The authors should conduct more experiments to justify their conclusions. Only MMLU and CMMLU are included to evaluate general knowledge, and only three domain-specific benchmarks are used. There are numerous benchmarks for both general and domain knowledge. To demonstrate that the proposed method is truly effective, the authors should include a more diverse and orthogonal set of benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LLfM1mM5dH", "reviewer": "Reviewer_gCgy", "summary": "This paper proposes ADEPT which is a 2-stage framework for continual pretraining (CPT) of LLMs that aims to inject domain knowledge while avoiding catastrophic forgetting of general abilities. Stage1 selectively expands only those transformer layers that are least important for general competence. These layers are identified via a probing procedure that masks each layer and measures loss increase by duplicating them with function-preserving initialization. Stage2 decouples units such as attention and MLP, inside the expanded layers and assigns asymmetric learning rates inversely proportional to each unit's importance to the general domain, updating only the duplicates. Across Qwen3 models with sizes 1.7B, 4B, 8B and Llama3-8B, ADEPT reports higher target-domain accuracy and better retention on MMLU and CMMLU compare to the full CPT, LoRA, replay, and LLaMA-Pro, while tuning 15% of parameters and cutting training time more than 50%.", "questions": "- Can you quantify the total compute overhead of importance probing and periodic unit re-weighting, and show how it scales from 2B to 8B or beyond?\n\n- Can you autotune the number of expanded layers under a fixed parameter/time budget, instead of fixing by hand?\n\n- Paper says: \"Experiments on mathematical and medical benchmarks show that ADEPT outperforms full-parameter\nCPT by up to 5.76% on the general domain\". I am confused by this, aren't general-domain benchmarks MMLU and CMMLU? But it says experiments on mathematical and medical benchmarks. What is that improvement?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea exploits functional specialization in LLMs where some layers and units are general-critical, while others are easier places to stuff domain knowledge. ADEPT explicitly measures that and routes updates accordingly.\n\n- The method is simple with a mask-and-probe score to pick layers, a function-preserving copy to stay stable at step zero and a first-order importance signal to scale learning rates.\n\n- On math and medical adaptation across several models, ADEPT generally boosts target-domain scores while avoiding the general-domain retention that is often seen after continual pretraining.\n\n- The selective duplication step uses a straightforward probing signal and a function-preserving initialization to keep behavior unchanged at the start. The decoupled tuning step uses a first-order importance estimate to steer learning rates. It is kind of practical guardrail that you can integrate into a production CPT pipeline without rewriting everything.", "weaknesses": "- Layer masking for general ability and periodic unit-importance recomputation are not free. The paper shows strong results, but there is no crisp computation of how much extra compute the probing adds as models scale, or how sensitive it is to the quality or size of the probing corpus. \n\n- The authors fix the number of expanded layers, which works but looks hardcoded. An auto-selection strategy would make it easier and boost the performance.\n\n- The evaluation is largely multiple-choice accuracy. That is fine for comparability, but I would love to see at least some open-ended reasoning or robustness/safety checks.\n\n- Medical gains for Llama3 rely on 8-layer expansion given Chinese weakness. That may suggest that the performance may hinge on relatively heavy expansion in challenging cross-lingual settings. How does ADEPT fare on code, law, finance, or multilingual transfer beyond Chinese and English?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bZ0wQHaL1f", "reviewer": "Reviewer_eXJC", "summary": "The paper proposes ADEPT, a two-stage framework for domain-adaptive continual pretraining (CPT) of LLMs: (1) General-competence guided selective layer expansion—duplicate the least general-critical layers (identified via layer-ablation loss deltas) with function-preserving init; (2) Adaptive unit-wise decoupled tuning—partition each expanded layer into functional units and assign inverse-importance learning rates to protect general-critical units while adapting flexible ones. On math and medical corpora, ADEPT reports gains over full-parameter CPT and other baselines on both domain metrics and general benchmarks.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Clear, principled motivation from functional specialization observations (early layers more general-critical; heterogeneous unit importance), which directly informs where to expand and how to tune.\n2. Sound expansion design via function-preserving identity copy (zeroing output projections) to avoid initial interference, consistent with Net2Net/FPI ideas.\n3. Simple, implementable LR rule, periodically refreshed, giving a practical recipe for decoupled protection vs. adaptation.\n4. Strong empirical results across Qwen3 (1.7B/4B/8B) and Llama3-8B on GSM8K/ARC and MedQA/MMCU/CMB; ADEPT often improves both domain and general benchmarks vs. full-CPT, replay, LLaMA-Pro, and LoRA/TaSL\n5. Ablations and analyses show both stages matter; uniform expansion underperforms selective expansion; token-shift analysis suggests focused domain injection.", "weaknesses": "1. Importance identification and Fig. 2: \n\n(a) From the training-domain perspective, it seems more appropriate to identify domain-specific important layers, not just those important to the general domain. Would it further boost performance if expansion happens on important layers for the target training domain? Could you report per-domain importance profiles and their overlap with the “general-critical” set?\n\n(b) This raises a potential domain conflict at expanded layers: a layer deemed “not important” for the general domain might be crucial for math, yet gets expanded and trained on medical data. How often does this conflict occur, and how does ADEPT mitigate it?\n\n(c) MLP dominance. Fig. 2 suggests MLPs dominate general-knowledge importance. Is this driven by parameter scale or by an intrinsic representational role of MLPs? \n\n2. It makes sense that zero-initializing the expanded layer’s output projection, and it promotes stability, but if many MLP output projections are marked important, does zeroing them hinder adaptation? To ensure that forward computation remains\nunchanged, making other modules (e.g., mlp gate/up projection) should also work.\n - Clarification: #264, does the ffn out projection mean down projection?\n\n3. In Table 2, why does uniform expansion produce inferior results in the medical domain? considering it enables more adaptation capacity", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4zqP5sDIRk", "reviewer": "Reviewer_nRAZ", "summary": "This paper introduces **ADEPT** (Adaptive Expansion and Dynamic Decoupled Tuning), a two-stage framework designed to improve continual pretraining (CPT) for large language models (LLMs) when adapting them to specialized domains like mathematics and medicine. The primary challenges it addresses are catastrophic forgetting (losing general knowledge) and the limited capacity of models to absorb new information. The method is based on the key insight that LLMs exhibit functional specialization, where different layers and parameters have varied importance for preserving general knowledge. In its first stage, ADEPT selectively expands the model by duplicating layers least critical to general competence, adding capacity with minimal interference. The second stage performs decoupled tuning on these new layers, applying asymmetric learning rates to protect general-critical parameter units while allowing adaptable ones to absorb domain knowledge. Experiments demonstrate that ADEPT outperforms the baselines in both target and general domain performance, while being more efficient by tuning only small number of parameters in less than half the training time.", "questions": "Thank you for this well-written and insightful paper. The proposed ADEPT framework is an elegant and effective solution to a significant problem in continual learning for LLMs. The empirical results are strong and the analysis is thorough. To further strengthen the work and clarify some points for the reader, I have the following questions and suggestions:\n\n1. **On the Generality of the Framework**: The choice of Mathematics and Medicine as test domains is excellent, as they represent distinct challenges (reasoning vs. factual knowledge). However, to fully substantiate the claim of the framework's general applicability, further evidence would be beneficial.\n    - Question: Have the authors considered evaluating ADEPT on a domain with fundamentally different characteristics, such as code generation? Success in a domain governed by strict syntax and algorithmic logic would provide powerful evidence for the method's versatility.\n\n2. **On the Dynamics of Decoupled Tuning**: The methodology for Stage 2 mentions that unit importance scores are periodically recomputed to keep the learning rates adaptive throughout training.\n    - Question: Could the authors please specify the update interval for these importance scores used in the experiments (e.g., every 500 steps, as noted in the appendix )? Further, was any sensitivity analysis performed on this interval? It would be interesting to know if performance is robust to less frequent updates, which could further improve efficiency.\n\n3. **Minor Proofreading and Formatting Suggestions**: I noticed a few minor typos that could be easily addressed. Could the authors perform a final proofread to catch minor errors? A few examples I noted include:\n\n    - Page 2, Figure 1: The label \"Target Domian Extension\" appears to have a typo. \n\n    - Page 3, Section 2.2: In the first paragraph, the corpus is described as something that \"servers as the probing ground\". The correct grammar would be \"serves as the probing ground\". \n\n    - Page 29, Appendix E: In the table 9, the method name \"Importance Cumulatation\" is misspelled. It should likely be \"Importance Cumulation\" or \"Importance Accumulation\".", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper demonstrates a fair degree of originality by effectively combining existing ideas in a novel manner. The significance of the work is clear, as it addresses an catastrophic forgetting problem in LLM area. The clarity of the writing is commendable, with proper use of English grammar and well-designed visualizations that aid in understanding the proposed methods.\n\n- **Novelty Through Methodological Rigor and Transparency**: While the individual components—such as parameter importance analysis, layer expansion, and adaptive tuning—are not entirely new, the paper’s novelty stems from the rigorous process used to select and combine them. The authors build a compelling case for their approach by meticulously justifying their unique combination of existing techniques. This is best illustrated in the appendix, where a systematic comparison of four different methods for probing layer importance is presented. By transparently demonstrating that their chosen \"masking out\" strategy delivers superior performance, the authors provide a clear, evidence-based rationale for their design choice, which greatly enhances the reader's understanding and trust in the proposed method.\n\n- **Significance and Strong Motivation**: The work addresses the critical and highly relevant problem of catastrophic forgetting during domain-adaptive continual pretraining, a key challenge for deploying specialized LLMs. The authors establish a compelling motivation not just by stating the problem, but by conducting a pilot study (Section 2) to provide an empirical foundation for their core hypothesis of \"functional specialization\". This initial analysis, which demonstrates that different layers and parameter units contribute unequally to preserving general knowledge, serves as a powerful justification for the entire ADEPT framework.\n\n- **Exceptional Clarity and Presentation**: The paper is written with outstanding clarity, making the complex methodology accessible. The logical flow from the problem statement to the empirical validation is seamless. This is further enhanced by well-designed visualizations; for instance, the main methodology diagram (Figure 3) provides an intuitive step-by-step overview of the ADEPT process , while the analytical figures on activation distributions (Figure 4) and token shifts (Figure 5) offer insightful, visual proof of the method's effectiveness.", "weaknesses": "While the paper is methodologically sound and the results are compelling, its primary weakness lies in the limited scope of its experimental validation, which could be expanded to further solidify the claims of generality.\n\n* **Scope of Domain Adaptation**: The authors make a strong case for ADEPT's effectiveness by testing on two distinct and challenging domains: Mathematics (emphasizing reasoning) and Medicine (emphasizing factual knowledge). This is a commendable choice. However, the claim of the framework's general applicability would be significantly bolstered by including experiments in other complex domains. For instance, code generation represents a crucial application area for LLMs that combines logical structure, strict syntax, and algorithmic reasoning. Demonstrating ADEPT's success in adapting a model to a specific programming language or codebase would provide more robust evidence of its versatility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)