Leveraging Pretrained Knowledge at Inference Time: LoRA-Gated Contrastive Decoding for Multilingual Factual Language Generation in Adapted LLMs
Authors: Gwangseon Jang
Hongseok Choi
Chanuk lim
Kyong-Ha Lee
Mun Yong Yi
Venue: ICLR 2026 Poster (2026)
Keywords: Contrastive Decoding
Multilingual Language Models
Inference-Time Knowledge Integration
Token-Level Confidence Gating
LLM
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose LoRA-Gated Contrastive Decoding (LGCD), a training-free decoding method that mitigates catastrophic forgetting in language-adapted LLMs by dynamically incorporating knowledge from the original pretrained model during inference.

Abstract: Large language models (LLMs) adapted to specific languages through continual pretraining or instruction tuning often suffer from catastrophic forgetting, which can lead to factual inaccuracies. This issue is particularly pronounced in multilingual settings, where adaptation may override general world knowledge with language-specific patterns. We propose LoRA-Gated Contrastive Decoding (LGCD), a training-free inference-time decoding framework that improves factuality in language-adapted LLMs by leveraging knowledge from the original pretrained model. LGCD operates by (1) extracting factual representations from Feed-Forward Network (FFN) layers via LoRA-based decomposition, approximating pretrained knowledge, (2) dynamically gating decoding based on token-level confidence, and (3) applying contrastive decoding with Top-K masking to revise uncertain predictions by referencing the approximated representation of pretrained knowledge. LGCD requires no additional training or access to the original pretraining data. Extensive experiments with LGCD on multilingual multiple-choice and long-form QA tasks across nine languages demonstrate its strong effectiveness in mitigating hallucinations and enhancing factual accuracy in language-adapted models. These results further indicate that pretrained knowledge can be strategically reintroduced during decoding to promote factual multilingual generation.

Published as a conference paper at ICLR 2026
LEVERAGING PRETRAINED KNOWLEDGE AT INFER- ENCE TIME: LORA-GATEL CONTRASTIVE DECODING FOR MULTILINGUAL FACTUAL LANGUAGE GENERA-
TION IN ADAPTED LLMS
Gwangseon Jang!. 1.3 Hongseok Choi², Chanuk Lim Kyong-Ha Lee Mun Yong Yi
GGadduate School Data Science, KAIST, Daejeon, Republic of Korea
2 "Language Intelligence Research Section, ETRI, Daejeon, Republic of Korea
31 Large-scale AI Research Center, KISTI, Daejeon, Republic of Korea 4 University Science and Technology (UST), Daejeon, Republic of Korea (ray jang27, munyi Wwaist/kaist...... ac kr, hongking 9etrrirrr.... re. kr (gsjang, chanuklim, kyongha pronghaaakinnn wwwpstiiiitti......... re.kr
ABSTRACT
Large language models (LLMs) adapted to specific languages through contin- ual pretraining or instruction tuning often suffer from catastrophic forgetting. which can lead to factual inaccuracies. This issue is particularly pronounced in multilingual settings, where adaptation may override general world knowledge with language-specific patterns. We propose LoRA-Gated Contrastive Decod- ing (LGCD), a training-free inference-time decoding framework that improves factuality in language-adapted LLMs by leveraging knowledge from the origi- nal pretrained model. LGCD operates by (1) extracting factual representations from Feed-Foowward Network (FFN) layers via LoRA-based decomposition, ap- proximating pretrained knowledge, (2) dynamically gating decoding based on token-level confidence, and (3) applying contrastive decoding with Top-K mask- ing to revise uncertain predictions by referencing the approximated representa- tion of pretrained knowledge. L.GDD requires additional training or access to the original pretraining data. Extensive experiments with LGCD on multilin- gual multiple-choice and long-form QA tasks across nine languages demonstrate its strong effectiveness in mitigating hallucinations and enhancing factual accu- racy language-adapted models. Moreover, these results indicate that pretrained knowledge can be strategically reintroduced during decoding to promote factual multilingual generation.
INTRODUCTION
Large Language Models (LLMs) have demonstrated remarkable capabilities across wide range of natural language tasks. common practice to enhance their performance for specific languages or domains involves continual pretraining (CPT) instruction fine-tuning (Gururangan et al. 2020 Zhang et al. 2024] [Huang et al. 2023). While these adaptation techniques often inject new knowl- edge and improve task-specific abilities the target language or domain, they frequently suffer from a critical drawback: catastrophic forgetting (Luo et al. 2023 OLMo et al. 2024 Li & Lee 2024
Li al. 2024 Kalajdzievski 2024). This phenomenon leads the degradation of general knowl- edge acquired during the initial pretraining phase, often resulting in increased factual inaccuracies
or hallucinations (Ji et al.] 2023: Luo et al. 20233 Li & Lee 2024). Empirical studies confirm that LLMSs undergoing CPT or instruction tuning can lose previously learned knowledge, sometimes pri- oritizing stylistic alignment or fluency in the target language over the factual consistency inherent in
the original model (Luo et al. 2023
"Corresponding author.

---

Published as a conference paper at ICLR 2026
Mitigating catastrophic forgetting during adaptation is challenging. Ideally, would retrain the model using a mixture of the original pretraining data and the new adaptation data. However, the original pretraining datasets for many state-of-the-a art LLMs (e.g.. LLaMA, Qwen) are generally undisclosed and inaccessible, though efforts towards fully open models like OLMo exist (OLMo) et al. 2024). Furthermore, retraining from scratch even extensive CPT demands prohibitive computational resources and time. Although various techniques aim to reduce forgetting during the training process (Gu al., 2024, Huang et al., 2024, He al., 2024; Wang al.) 2023b Vol et al., (024), they remain limited in preserving general knowledge, especially when adapting to new domains or languages. This limitation motivates the exploration of alternative approaches that can enhance the factuality of adapted LLMs without requiring further training or access to original
pretraining data.
Recent research has highlighted the role of Feed-Forward Network (FFN) layers within the Trans- former architecture as key-value memories, crucial for storing factual knowledge acquired during pretraining (Geva et al. 2020: Qiu et al. 2024 Dai et al., 2023). Inspired by this understanding, we hypothesize that the knowledge implicitly stored within the FFN weights of the original pretrained model can be explicitly leveraged to support the generation process of an adapted (e.g.. continually pretrained or instruction-tuma tuned) model at inference time, thereby improving its factual accuracy.
In this work, we propose LoRA-Gated Contrastive Decoding (LGCD). novel training- ffree de- coding method designed to enhance the factuality of LLMs, particularly those adapted for specific languages domains. LGCD addresses the inherent trade-off between domain-specifie fluency and general factual knowledge by dynamically switching between decoding strategies based on token- level confidence and applying contrastive decoding when necessary.
The framework of LGCD is characterized by three key components: First, it performs LoRA LoRA-based factual knowledge extraction from FFN layers and obtains a lightweight approximation of the pre- trained model (PTM), by computing parameter differences between pretrained and adapted models and decomposing them using Singular Value Decomposition (SVD) to recover factual knowledge in FFN layers without modifying the language- adapted model (LAM).. Second, it employs confidence- based dynamic gating that measures token-level confidence from the LAM and determines when to trigger factual knowledge injection, ensuring that domain fluency is preserved when the model is confident while leveraging pretrained knowledge when uncertainty arises. Third, it implements con- trastive decoding with Top-K K masking, which computes contrastive logits by subtracting the LAM's logits from the logits of the approximated PTM (aPTM), and applics this correction only to the top-K candidates predicted by the LAM. This selective adjustment injects factual knowledge while minimizing disruption to fluent generation.
We conduct comprehensive evaluation of LGCD across nine diverse languages, highlighting its broad applicability in multilingual settings. Our experiments demonstrate LGCD's effective- ness across multiple evaluation settings, including multilingual multiple-choice benchmarks such as Global MMLU (Singh et al.) 2024a) and multilingual TruthfulQA (Dac Lai et al., 2023) for comain-specific and general factual knowledge, long-form generation benchmarks such as Multi- FAct (Shafayat et al. (2024) for factual consistency, and long-form medical QA tasks for precise knowledge grounding high-stakes domains.
Our contributions are threefold:
1. We propose LGCD, novel training-free, decoding-time framework to mitigate hallucina- tion and enhance factuality in aiguage-adapteer LLMs by leveraging knowledge from the original pretrained model through dynamic model switching and contrastive decoding.
2. We introduce specific techniques within LGCD, including LoRA-based knowledge extrac- tion from FFN layers, cedidenee-basee dynamic gating for token-level decision making, and contrastive decoding with Top-K masking.
3. We provide extensive empirical evidence demonstrating LGCD's effectiveness across mul- tilingual multiple-choice QA and long-form generation tasks, using nine languages and twelve models. Our approach consistently outperforms adapted models without requiring additional training or external resources.
2

---

Published as a conference paper at ICLR 2026
< Confidence Bosed Dynamic Gating
Response
probability
pod cet
with Tap-k Masking
GouEdraae Suededdd
(CL-
Aileeedr (lnee y01
Token
Query
LAM
Lلااا (60
prababiltt Threshold
Legits logits
Who dicieeee (nu ESS 983 (11 REE WMB
1 loB4-based Knodledge (Ne-ccomputation Onic))
Figure 1: Overview of the LGCD framework.
RELATED WORK
2.1 HALLUCINATION MITIGATION IN LLM
Addressing hallucinations in LLMs involves various strategies, including improvements in training data and model architecture, fact-checking mechanisms, and integrating external knowledge sources like retrieval systems knowledge graphs (Izacard & Grave 2020 Wang al., 2023a). While effective, the aforementioned external methods often introduce complexity or dependencies. Our proposed method, LGCD, focuses on an internal, decoding time approach to mitigate hallucination without requiring external models or significant architectural changes.
2.2 DECODING STRATEGIES FOR FACTUALITY
Factual consistency in LLM decoding is often improved by tweaking output probabilities. For in- stance, Contrastive search (Su et al.) 2022) combats repetition by picking tokens that both likely and semantically distinct from prior context. Similarly, DoLa (Chuang et al. 2024) leverages deeper layers by contrasting logits across different internal layers of the shape probability distributions based on disagreements or differences within the model or its vari- ants. Our LGCD takes a different approach. Instead merely contrasting probabilities, LGCD explicitly extracts and integrates factual knowledge from FFN layers of the original PTM-a pro- cess that is performed only once. This knowledge then directly influences the LAM's logits via confidence- selidence-ggted mechanism, offering training-free solution to inject specific factual signals and enhance accuracy in adapted LLMs without further access to PTM during inference.
model. These methods
2.3 KNOWLEDGE IN FREEDFORWARD NETWORKS
Previous research has shown that FFN layers within transformer models serve key repositories for factual and world knowledge, often interpreted as key-value memories (Geva et al. 2020: Qiu
al., 2024 Dai et al. 2023). This perspective suggests that structured knowledge is encoded within their weights. Our LGCD leverages this by explicitly recovering factual knowledge from FFN lay- ers of the original PTM. This is vital because catastrophic forgetting can degrade such knowledge in LAMs. LGGD achieves this through LoRA-based factual knowledge extraction: it calculates parameter differences between the PTM and LAM, then uses SVD to pinpoint and recover PTM's factual knowledge without altering the LAM. This precisely extracted knowledge is then dynami- cally injected during decoding, central to LGCD's factuality enhancement.
METHODOLOGG
This section introduces LGCD, a training-free free decoding framework that combines the strengths of LAM and PTM. LGCD addresses the inherent trade-off between domain-specific fluency and gen- eral factual knowledge by dynamically switching between models based on token-level confidence and applying contrastive decoding when necessary. F illluttraess Filusteisl framework.
3

---

Published as a conference paper at ICLR 2026
The core motivation behind LGCD is grounded in the observation that FFN layers in Transformer architectures act as key-value neural memories for factual knowledge (Geva et al. 2020 Dai et al, 2023 Qiu et al. 2024). During CPT instruction fine-tuning, this knowledge can be degraded due to catastrophic forgetting. LGCD mitigates this by explicitly recovering factual knowledge from the pretrained model's FFN layers and dynamically injecting it during decoding-
The LGCD framework consists of three components: (1) LoRA-based factual knowledge extraction from FFN layers, (2) confidence-based dynamic gating, and (3) contrastive decoding with Top-K masking.
3.1 LORA-BASED KNOWLEDGE EXTRACTION FROM FFN LAYERS
To capture factual knowledge preserved in the pretrained model, LGCD performs LoRA extrac- tion from all FFN layers. Specifically, for each FFN layer E, we compute the parameter difference between the pretrained model MyTM and the language-adapted model MLAM:
AW WPTM W!AAM
We then apply SVD:
AW; U
(2)
LoRA matrices are constructed by retaining the top-r singular components:
A Uz|:: r] v [
(3)
Bj = v Ee[: r [r,
(4)
The pretrained FFN weight is approximated as: W.PIM = W.AAM + Ae
(5)
Notably, this process is performed only once in the entire framework and allows LGCD to retrieve factual knowledge from the PTM without modifying the LAM directly or incurring additional mem- ory overhead from deploying separate models. It is applied only to the FFN layers, leaving all other components of the LAM unchanged. To empirically validate this design decision, we compare dif- ferent layer-wise LoRA approximation strategies in Appendix A.3 and find that targeting only FFN layers yields the best performance.
3.2
CONFIDENCE- BATIDENCE-BBSED DYNAMIC GATING
At each decoding step I, LGCD first queries the LAM to compute logits 1LAM The token-level confidence is measured the maximum probability over the vocabulary:
= max (softmax(11-MM)
(6)
A fixed confidence threshold 5 determines the decision. T in our setting is determined based on language-specific data availability, as detailed in Appendix A.8
If 2 Decode with LAM.
. If c, Contrastive decode with the aPTM.
This dynamic gating balances the fluency of the LAM with the factual reliability of the aPTM.
3.3 CONTRASTIVE DECODING WITH TOP-K MASKING AND LAYER-WISE CONTRAST
When contrastive decoding is triggered, LGCD first computes logits of aPTM I.PTM approximated with LoRA:
PPPM
aL.M + LORA(AWe, h(-AM)
To prevent contrastive decoding from selecting tokens with low probabilities across both models, LGCD applies Top-K masking, considering only the K most probable tokens from A.A...
Tk TopK K)
(8)
The contrastive logits computed as:
4

---

Published as a conference paper at ICLR 2026
i E Tk:
[i] + 8- [IPPM[[ a [aM
conntrstt
otherwise:
-00
where a hyperparameter controlling the overall contrastive weighting and a E [0, 1] controls the degree of down-weighting applied to the LAM logits within the correction term.
This formulation goes beyond standard contrastive decoding by introducing correction [:] [i]) that prioritizes the aPTM's factual knowledge while gently penalizing po- tentially overconfident LAM predictions. By tuning a, we modulate the LAM's influence without abruptly overriding it. Hyperparameter details are provided in Appendix A.4 and for completeness
we include the full decoding algorithm and pseudocode in Appendix A.2
EXPERIMENTAL SETUP
We evaluate our model on two task types-mulipple-hoiice QA and long-form generation-across nine target languages: Chinese (zh), German (de). Portuguese (pt), Arabic (ar). Persian (fa), Japanese (ja), Korean (ko). Indonesian (id), and Swahili (sw). Unless otherwise noted, all exper- iments are conducted using 12 models.
4.1 MULTIPLE-CHOICE QA
Global MMLU. To assess multilingual factual understanding, Global MMLU (Singh al. 2024a), a culturall--warr extension MMLU with 14K curated questions spanning 57 subjects in 42 languages. We report accuracy per language in zero- and five-shot settings for nine target languages.
Multilingual TruthfulQA. We the MC1 version of TruthfulQA (Dac Lai al. 2023 Lin al. 2021a) across 31 languages. The number of questions per language varies, with most languages containing at least 700 items. We evaluate six models in zero-shot and five-shot settings.
4.2 LONG-FORM GENERATION
Medical QA. For high-stakes generative evaluation, we use multilingual medical QA datasets with expert-validated answers (Appendix A.5) Evaluation includes:
L..M-as-a-Judge: GPT-4o performs pairwise comparisons with baselines Li al, 2023a).
Human Evaluation: For 12 LAMs, we sample 20 questions each (240 total). Outputs translated English and rated by three experts fluency, coherence, specificity, and factuality. Final labels (Win/Tie/l.ose) are based on majority vote.
Multi-FAct. We evaluate factuality with Multi-FAct (Shafayat al. 2024), which uses FActScore Min et al. (023) to decompose generations into atomic facts and verify them against trusted sources multilingual settings.
4.3 MODELS BASELINES
Models. We evaluate 12 publicly available LAMs from Hugging Face, each specialized for one of the 9 target languages (zh, de, pt, ar. fa, ja, ko, id, sw). All models are based on multilingual LLM backbones (mainly LLaMA-3 variants) and fur- ther adapted via CPT, instruction tun- ing, both. Model selection was guided by language specificity, pub- li availability, and community gagement (c.g., download count, ac- tive maintenance). Table [ summa- rizes model specifications.
Model
CPT Instr. tuning
Beenzhiiwang/LLaee Laaaaangg/aaa--------- Ch-nma---BB--hies--hhaall
zh hil/llama- 3 chinese Sh instract
DiscoResearchll lama Discol Ins.inntttSB..... -0.
X
MobamsdRatla/AArahii Orp-le-------L-aa------ --ama--88--nnsiin
X
ja PartA/Doomm- Laana3 8B Instruct
elyza/Llama-33 --LAA
ko
KISTI KONİ/KONI Llama3 8B Instruct 20240729
Instmuet-22240729
ML.P-KTL www.lliii/laaaaaaa Koorean-Bllossone
GoToCompany/lllaa--eo hat-e-gg--aaaattii---iieiee
sw
Jacaranda/ Laaaandd//Uizzal lama3
Table 1: LAMs used our experiments. "/" indicates
application of CPT and/or instruction tuning.
5

---

Published as a conference paper at ICLR 2026
Baselines.
We compare LGCD against Nucleus Sampling (NS), DoLa (Chuang et al. 2024), TIES Yadav et al. 2023a), and SLERP (Shoemake) 1985 This set covers widely used decoding meth- ods and model-merging approaches, enabling comprehensive assessment of LGCD.
@-shot
5-shot
Tang.
Model
PTM
I.AM
Dhlla
TIES
SI.RRP
LGGC
PTM
1.M
Dolla
TIFS
SLERP
LGCD
2h milllam---chieee Ih-inrmm
0.496
0.465
0.457
0.502
0.434
0.519
03.533
0.515
0.514
0.536
0.552
0,545
<h
shenohi-sang1 -mm---3-----ii ShB-CChnnn---huu
0.494
0.500
0.500
0.408
0-494
0.502
0.533
R.543
0.542
0.533
0.593
0.545
d:
Diss:Rescanchll Jema MenadeDo insdasinniistoooo
0.514
0.466
0.492
0.540
0.514
0.566
0.538
0.548
0.550
1.566
0.584
0.574
pt
Thyymiso/gemmm peruguese luzna 26
0.355
0.315
0.278
0.357
0.353
0.325
1.316
0.298
0.339
0.325
0.324
or
AattmmrrdkattddAraiiiiiii Orpo Laama Instrut
0.45
0.430
8.42)
0.625
0.441
0.967
0
0:425
0.467
0,883
0.481
9
PanAl Jama Intruct
0.424
0:425
0.423
0.444
0.865
0.806
0.468
0.466
0.465
0.465
31
Inbyotoch-liml Jama-1 Syoillo--4B--nttteee
0.811
0.478
04462
0.46
0.827
0.527
0.525
0.881
31
chooel lana- --..----..-.-11
0.466
045
0.50
0.50%
0.11
0.81
0.528
HISTIKONDKOONA Iamu 2------0mmm---222733
0.445
0.445
0457
0.495
6,81
0.511
MLP I Bllusem---
0.576
0.45
0457
0.445
6.411
0.530
0.466
0.76
0.566
0.362
0.335
0.359
0.367
0.372
0,432
Average
Table 2: Evaluation accuracy of 12 models on Global MMLU benchmark under O-shot and 5-shot settings using various decoding and merging strategies. PTM refers to the performance of the pretrained Model, while LAM denotes the language-adapted model, both evaluated using Nucleus Sampling (NS). DoLa represents the results when applying the DoLa decoding strategy to the LAM.
5 EXPERIMENTAL RESULTS
We evaluate LGCD on two multilingual multiple-choice QA benchmarks-Global MMLU and Mul- tilingual TruthfulQA-unde zero-shot and five-shot settings. Comparisons include decoding-ting baselines (DoLa), model merging methods (TIES, SLERP), and standard nucleus sampling (NS) for both pretrained (PTM) and eagguage-adapter models (LAM).
5.1 GLOBAL MMLU
Table 2 shows accuracy across 12 LAMS covering 9 languages. LGCD achieves the best average performance in both zero-shot and five-shot settings, outperforming all decoding and merging base- lines.
In the zero-shot setting, relative to the LAM, LGCD improves accuracy in 10 of 12 cases. Gains largest where the LAM lags the original PTM (Korean: 10.3 pp: German: +6.0 pp: Japanese: +2.9-3.6 pp: Portuguese: +4.1 pp), indicating that LGCD effectively recovers knowledge lost during adaptation.
In contrast, in the five-shot setting, LGCD continues to provide consistent improvements, especially in high-resource languages such as Chinese and German. This indicates that LGCD not only recov- ers forgotten knowledge but also scales robustly with richer context, effectively balancing domain adaptation and pretrained factuality across diverse conditions.
Overall, LGCD consistently improves multilingual QA performance, showing robustness across cul- turally diverse and knowledge-heavy questions where language adaptation typically disrupts general knowledge.
5.2 MULTILINGUAL TRUTHFULQA
Results on Multilingual TruthfulQA (Table further validate LGCD's ability to enhance factuality. In the zero-shot setting, LGCD achieves the highest average accuracy, outperforming both decoding and merging baselines across evaluated languages.
LGCD demonstrates consistent improvements across all tested languages, with particularly notable gains in Portuguese (+10.8 pp), Arabic (+9.2 pp), and German (+3.9 pp). These improvements hold in the five-shot setting as well, where LGCD shows even larger gains, reinforcing the method's robustness across different prompting regimes.
SLARP Merge

---

Published as a conference paper at ICLR 2026
Mode
LAM
TIES
SLERP
L4GCD
LAM
TIES
SLERP
LGCD
Infilaane 3-cinsse-
0.352
0.355
[1.37
01381
0.363
0.471
shenohi wampl ais Chinese-Chrt
0.54
0.565
0.326
0.357
0.363
0.357
0.579
0.391
0.360
0.367
0.400
0.484
DicckResrschhl Limal-Diccl Discal co-Iintmcct-i8--0)))
0.317
0.545
0.320
0.35
0.350
0.382
0.967
0.367
0.354
0.373
0.393
0.391
0.272
0.501
0.268
0.283
0.302
0.409
0.319
0.325
0.399
0.322
0.321
0.431
0.525
0.508
0.331
0.526
0.318
0.400
0.376
0.360
0.323
0.363
0.365
0.426
Go GurooCoupuary Iiburs SeinnnSSStseaai Sistttoesaatttiiii biislluct
0.329
0.334
0.334
0.541
0.354
0.353
0.377
0.370
0.369
0.373
0.365
0,406
Avengs
0.523
0.334
0.315
0.331
0.350
0.376
[0.366
0.367
0.340
0.370
0.368
0,435
Table 3: Evaluation accuracy of 6 models on multilingual TruthfulQA benchmark under O-shot and
5-shot settings using different merging and decoding strategies.
Win
Myfiomm S-blarond of
Chat 1nn
ai anl
Haaye
DMA
13.94
740%
38
2EP4
322%
3,00
SUIP
20
40
60
93
100
27
80
100
20
40
60
60
100
22
40
60
80
100
20/2
3:45
183
11a
HET NET--O/EN0ALLLmDD00 ver 2004C7
jeo www.)
7M
PMA
MMA
DMA
71.65
2,00
21.2
20
40
60
89
150
50
100
20
40
60
100
23
40
60
90
100
Figure 2: Pairwise comparison results of our model vs. baselines (evaluated by GPT-4o)
These results confirm that LGCD effectively resists plausible but incorrect generations, excelling in settings in TruthfulQA that probe a model's ability to distinguish truth from commonly held misconceptions.
5.3 LONG-FORM MEDICAL QA
We further evaluate LGCD in high-stakes, long-form medical question answering tasks across 12 LAMs. Each model is tested in its target language using domain-specifet medical QA datasets. Evaluation is conducted via two complementary methods: (1) GPT-4o as an automatic judge per- forming pairwise comparisons, and (2) expert human preference evaluations fluency, coherence, and factual correctness.
LLM-as-a-Judge Evaluation. Figure shows GPT-4o-based preference comparisons. LGCD achieves the highest win rates in most languages, outperforming PTM, LAM, and all decoding or merging baselines. On average, LGCD is preferred over PTM in 63. of cases, over LAM in 53.5%. and over DoLa (53.8%) and SLERP (51.9%), while performing competitively with TIES (65.3%).
Human Preference Evaluation. We further as- sess LGCD through human preference evaluation. As shown in 3 LGCD is consistently favored across baselines, achieving higher win rates than PTM (52.0%). DoLA (45.8%), and SLERP (45.8%). These results reinforce LGCD's ability to generate fluent and factually grounded answers in complex medical domains. Despite cross-lingual pooling, the consistent preference trend across models suggests that LGCD offers robust and training-fre free alter- native to improve factuality in long-form, domain- specific generation.
Lowe
PTM
52.0%
LAM
41.7%
27.1%
31.2%
DoLA
45.0%
16.77 16.7%
37.5%
TIES
62.5%
16.7%
20 0%
SLERP
16.7%
37.5%
20
40
60
80
100
Figure 3: Human preference comparison be- tween our model and baseline models
7

---

Published as a conference paper at ICLR 2026
244
04
04
04
02
04
06
C44
02
04
5.
04
06
Tranod
Cridess Treied
3 mm
222
Figure 4: Contrastive decoding behavior in LGCD across 12 LAMs. For each model, we plot the token-level usage ratio of the base LAM (blue) and contrastive candidate (red) as a function of confidence threshold т. The yellow vertical line marks the threshold selected for that model. LGCD dynamically adjusts model usage based on token uncertainty, striking a balance between domain alignment and factual recall.
LAM LGCD
14.0
-17.5
... jaccard Similaar ky
30-
12.5
10.0
Auarage
-500
0.0
Liumaz Ch-C------C
Luma EE 4B
Fuma2 3b- sha--laasaabbllaaaa +1: rotruct
Models
Figure 5: Average entity count (left) and Jaccard similarity (right) between LGCD and LAM outputs for Chinese, Japanese, and Indonesian models
Contrastive Usage Analysis Isoww the proportion of tokens decoded by the LAM (blue)
and via contrastive decoding (red) as the confidence threshold T varies. The yellow band indicates the threshold selected for each model.
In higher-resource languages (e.g., zh, de, pt, ar, fa), models adopt higher thresholds (typically T = 0.7-0.8), leading to a sharp increase in contrastive usage. This refeect the stronger performance of the pretrained model in these languages, allowing LGCD to revise uncertain predictions more effectively.
In contrast, for lower-resource languages (e.g., ja, ko, id, sw), LAM usage remains high even at high thresholds. These models tend to produce overconfident outputs, possibly due to limited token coverage during adaptation. Since the PTM is also less reliable in these settings, LGCD uses lower thresholds to favor the LAM, which yields better results despite overconfidence.
To probe where LGCD's gains come from in long-form generation, we analyze named-entity behav- ior under general Named Entity Recognition (NER) schema and compare L.GDD to the underlying LAM. Further details are provided in the Appendix A.7 Figure 5 summarizes two signals: (i) av- erage number of entities extracted per output and (ii) Jaccard overlap between the entity sets from LGCD and the LAM. Across three representative LAMS (Chinese, Japanese, Indonesian), LGCD consistently produces more entities than the LAM while the set overlap remains low-moderate (*= 1-16%). This pattern suggests LGCD is not merely echoing the LAM's choices but is adding complementary, likely factual, mentions that the LAM omits. Linking back to the Contrastive Usage
8

---

Published as a conference paper at ICLR 2026
Locoo B1
18
L. BR
Figure 6: Token-level view a Japanese medical QA example cLLyal.L.mma--EELYAA..--111
LGCD output in Japanese (top) with English alignment (bottom). Tokens highlighted in green mark time-steps where LGCD switched on contrastive decoding; their aligned tokens in the transla-
tion are highlighted in yellow Tokens outlined in red indicate decisive positions that steered the continuation toward the factual answer.
Lang.
Model
LAM
LGCD
zh
Dolllama-3-chinese--ee -iweeesee------tteeee 8b-innstruct
0.260
0.246
-0.014
zh
DiscoResearch/l BB-hhhhhshhhhhaatt
shonzhi-wang/ hatphh/wwww////mma--- Jll.al--a-----a
0.229
0.313
+0.084
DiscooLe-Insttrut-BBe
0.387
0.520
rhaymison/gemma portuguese luana 2b
0.288
0.267
-0,021
ja
tokyotech-Tlm/1 1oma-3-Swallow-333 -0n.-----nsttcttt0....
0.267
0.197
0.070
elyza/Llama 3 ELYZA JP-8B
0.298
0.302
+0.004
ko
KISTI KONI/KONI- -ama3-88--- Inn-aa--------tttt--000
0.196
0.218
+0.022
MuLP-KTLim/llma----oeee Bllossom--
0.189
0.196
+0.007
id
0.334
0.547
+0.212
Average
0.272
0.312
+0.040
Table 4: Comparison of Multi- FFct score between LAM and LGCD. Models with a LAM score below 0.05 were excluded, resulting in 9 evaluated models.
Analysis, these results explain why factuality can improve even when overall contrastive usage is low-e.g., in Japanese (elyza/LLamaa3-ELLLAA..PP and Indonesian (GoToCComanyy/llaaaa3 iitiaabbbtaiil--itruucy)
Figure 6 further illustrates that LGCD intervenes only sparsely, yet the activated gates coincide with tokens that are not only entity mentions but also those carrying decisive factual content. Despite their small number, these targeted interventions are sufficient to steer the generation toward factual answers. A direct comparison of LAM and LGCD outputs for the Japanese medical QA example is provided in the Appendix A.10
5.4 MULTI-FACT
On the Multi-FAct benchmark, LGCD improves factual consistency in 6 of 9 models, with +0.04 average gain over LAMs (Table 4 Notable gains in Indonesian, German and Korean confirm LGCD's effectiveness in enhancing factuality for long-form generation.
5.5 THROUGHPUT
Table 5 presents the decoding speed of dif- ferent generation strategies measured a single A100 GPU. Results are averaged over responses to 100 questions using the htt.lllaaa--ccineeee
Decoding Strategy
Throughput (Token's)
Greedl search
19.21
Nucleus sampling
17.47
Contrastive scarch
11.87
-an---b-iinttt ruct model. Among baselines, greedy search achieves the highest throughput (19.21 token- s/sec), followed by nucleus sampling (17.47). Contrastive Search is slower (11.87) due to the similarity penalty over top-k candidates, though it runs in single forward pass without reranking. LGCD slows decoding by querying the aPTM when the LAM lacks confidence.
DoLa
16.81
LGCD-0.
LGCD-0.4
10.32
LGCD-0.6
10.32
LGCD-0.8
10.22
Table 5: Average decoding throughput for each strategy. For LGCD-r, the numeric suffix denotes the confidence threshold used in decoding. However, the overhead varies with the confidence

---

Published as a conference paper at ICLR 2026
threshold lower thresholds (e.g., LGCD-0.2) lead to fewer contrastive decisions and thus higher throughput (14.37), while higher T (e.g.. LGCD-0.8) leads to more aggressive factual intervention and slower speed (10.22). Notably, LGCD-0.2 approaches the speed of DoLa (16.81), demonstrat- ing that factual calibration can be achieved without substantial efficiency loss when appropriately tuned. This tunability enables L,CDD balance quality and speed for practical deployment.
LIMITATION
While LGCD effectively enhances factuality in language-adapted models, several limitations re- main. First, our framework relies on the availability of the original PTM weights for a one-time offline knowledge extraction process. Although the PTM is not required during inference this ini- tial requirement may limit the applicability of LGCD scenarios where PTM weights are entirely inaccessible undisclosed. Second, the effectiveness of our dynamic gating tied to the reliability of logit-based confidence signals. the model is extremely overconfident or logit-derived un- certainty is miscalibrated. Future work could explore enhance the reliability of the gating process across diverse model architectures.
robust uncertainty measures further
7 CONCLUSION
We present LGCD, novel training-free method that enhances the factuality of language-adapted LLMs by dynamically injecting pretrained knowledge during inference. By extracting factual sig- nals from FFN layers via LoRA-based decomposition and applying contrastive decoding condi- tioned token-level confidence, LGCD effectively mitigates catastrophic forgetting without training or access to pretraining data. Extensive multilingual experiments across multiple-choice QA and long-form generation tasks demonstrate that LGCD consistently improves factual accuracy over both decoding and model-merging baselines, offering practical, scalable solution for factual- ity preservation in domain-specialized LLMss.
ACKNOWLEDGMENT?
This research was supported by the Korea Institute of Science and Technology Information (KISTI) in 2026 (No. (KISTI)K2LL3MICI), aimed at developing KONI (KISTI Open Neural Intelligence), large language model specialized in science and technology.
Additionally, this work was also supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) RRS-2022-NR068777
REFERENCES
Julien Abadji, Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. Ungoliant: An op- timized pipeline for the generation a very large-scale multilingual web corpus. Proceedings of the Workshop
Challenges in the Management of Large Corpora (CMLC-9) 2021. Lim- erick, 12 July 2021 (Online-Event), pp. 1 9, Mannheim, 2021. Leibniz-Institu Deutsche Sprache. doi: 10.14618/ids pub- 10468. URL patttps://nbn-e. htw.///bbbrreeovineeoeeeo org/urn. nno.: de: mh39 104688
Julien Abadji. Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. Towards a Cleaner Drcement-Orientedd Multilingual Crawled Corpus. arXiv e-prints, art. trXiv:2201.06641 Jan- uary 2022.
Isaac Caswell, Julia Kreutzer, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadim- itriou, Salomey Osei, Pedro Javier Ortiz Suárez, Iroro Orife. Kelechi Ogueji, Rubungo Andre Niyongabo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaven- ture F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Balh, Stella Biderman,
10

---

Published as a conference paper at ICLR 2026
Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofe- toluwa Adeyemi. Quality Glance: An Audit of e-prints, art, arXiv:2103. 1022,, March 2021.
Multilingual Datasets. arXiv
Yung- Chuang. Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations, 2024.
https::::: Utt.httss:: //operee/eeee
net/ wtt.oormmmmd-hhhiieee
Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Demoncourt, Ryan A Rossi, and Thien Huu Nguyen. Okapi: Instruction-tunce large language models in multiple lan- guages with reinforcement learning from human feedback. arXiv e-prints, pp. arXiv 2207, 2023.
Damai Dai, Wenbin Jiang, Qingxiu Dong, Yajuan Lyu, and Zhifang Sui. Neural knowledge bank for pretrained transformers. In CCF International Conference on Natural Language Processing and Chinese Computing, pp. 772-783. Springer, 2023.
MohammadReza Davari and Eugene Belilovsky. Model breadcrumbs: Scaling multi-task model merging with sparse masks. European Conference on Computer Vision, pp. 270-287. Springer, 2024.
Pala Tej Deep, Rishabh Bhardwaj, and Soujanya Poria. Della-merging: Reducing interference in model merging through magnitude-based sampling. arXiv preprint arXiv:2406. 11617, 2024.
Mor Geva, Roci Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint 2tt.::2//2./99.... 2020.
Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee's MergeKit: A toolkit for merg-
ing large language models. In Franck Dernnncourt, Daniel Preotiuc-Pietro, and Anastasia Shimorina (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing: Industry Track, pp. 477-485, Miami, Florida, US, November 2024. Associ- ation for Computational Linguistics. doi: 18653/v1/20244 122002/4..mmll-nnnddstt...... URL StLhttp:: emn 1p- industrry.33.
Jiawei Gu, Zacc Yang. Chuanghao Ding, Rui Zhao, and Fei Tan. Cmr scaling law: Predicting critical mixture ratios for continual pre-training of language models. arXiv preprint urXiv:2407.17444 2024.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't stop pretraining: Adapt language models domains and tasks. arXiv preprint arXiv:2004. /0964, 2020.
Jinghan He, Haiyun Guo, Kuan Zhu, Zihan Zhao, Ming Tang, and Jinqiao Wang. Seekr: Selective attention guided knowledge retention for continual learning of large language models. arXiv preprint arXiv:2411.000 1.0671,, 2024.
Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, and Jinsong Su. Mitigating catastrophic forgetting in large language models with self-syythesized rehearsal. arXiv preprint arXiv:2403.0124. 2024.
Quzhe Huang, Mingxu Tao, Chen Zhang, Zhenwei An, Cong Jiang, Zhibin Chen, Zirui Wu, and Yansong Feng. Lawyer Ilama technical report. arXiv preprint arXiv:2305.1506 2023.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.040085 2022.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint XXiv:2007.0/282. 2020.

---

Published as a conference paper at ICLR 2026
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey hallucination in natural language generation. ACM computing surveys, 55(12):1-38, 2023.
Damjan Kalajdzievski. Scaling laws for forgetting when fine-tuning large language models. arXiv preprint arXiv:2401.05603 2024.
Chen-An Li and Hung-Yi Lee. Examining forgetting in continual pre-training of aligned large language models. arXiv preprint arXiv:2401.03129, 2024.
Hongyu Li, Liang Ding, Meng Fang, and Dacheng Tao. Revisiting catastrophic forgetting in large language model tuning. arXiv preprint urXiv:2406.04884 2024.
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting Ilm performance with self-guided data selection for instruction tuning. arXiv preprint arXiv:2308. /2032, 2023a.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Perey Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In Proceedings the 61 annual meeting of the association for computational linguistics (volume 2023b.
1: Long papers), pp.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2021a.
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary. Brian O'Horo, Jeff Wang. Luke Zettlemoyer, Zornitsa Kozareva, Mona T. Diab, Veselin Stoyanov, and Xian Li. Few-shot learning with multilingual language models. CoRR, abs/2112. 2021b. URL http::tt:://aae www.riii.orrrraeeeeeoeoeeooeeeee abs/////////// /2t...1111.22 10668
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint 1rXiv:2308.087477 2023.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannanch Hajishirzi. Factscore: Fine tinenerained atomic evaluation of factual precision long form text generation. arXiv preprint arXiv:2305. 14251, 2023.
Hellina Hailu Nigatu, Atnafu Lambebo Tonja, Benjamin Rosman, Thamar Solorio, and Monojit Choudhury. The zeno's paradox oflow-resource dloo-resourceellanuagee. arXiv preprint arXiv:2410.20817, 2024.
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.0080 00656, 2024.
Zihan Qiu, Zeyu Huang, Youcheng Huang, and Jie Fu. Empirical study updating key-value memories in transformer foed-forward layers. arXiv preprint arXiv:2402.12233, 2024.
Sheikh Shafayat, Eunsu Kim, Juhyun Oh, and Alice Oh. Multi-fact: Assessing multilingual llms' multi-regional knowledge using factscore, 2024.
Ken Shoemake. Animating rotation with quaternion curves. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques, pp. 245-254, 1985.
Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, et al. Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evalu- ation. arXiv preprint 2rXiv:241..03322 2024a.
12

---

Published as a conference paper at ICLR 2026
Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F. Karlsson, Abinaya Mahendiran, Wei- Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzeminski, Hakimeh Fadaei, Irem Ergün, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Minh Chien, Sebastian Ruder, Surya Guthikonda, Emad A. Alghamdi, Sehastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzich Fadacc, and Sara Hooker. Aya dataset: An open-access collection for multilingual instruction tuning, 2024b.
Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. contrastive framework for neural text generation. Advances in Neural Information Processing Systems, 35: 21548-21561, 2022.
Anh-Dung Vo, Minseong Jung, Wonbeen Lee, and Daewoo Choi. Redwhale: An adapted korean IIm through efficient continual pretraining. arXiv preprint arXiv:2408. 1294, 2024.
Fanqi Wan, Longguang Zhong, Ziyi Yang, Ruijun Chen, and Xiaojun Quan. Fusechat: Knowledge fusion of chat models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 21629-21655 2025.
Cunxiang Wang, Haofei Yu, and Yue Zhang. Rfid: Towards rational fusion-in-decode for open- domain question answering. arXiv preprint arXiv:2305.170411 2023a.
Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning. arXiv preprint arXiv:2310. 14152, 2023b.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes. Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference machine learning, pp.
PMLR, 2022.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Resolving interfer- ence when merging models. arXiv preprint arXiv:2306. 000006.17700 01708, 1, 2023a.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Re- solving interference when merging models. Advances in Neural Information Processing Systems, 36:7093-7115, 2023b.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models super mario: Ab- sorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning, 2024.
Di Zhang, Wei Liu, Qian Tan, Jingdan Chen, Hang Yan, Yuliang Yan, Jiatong Li, Weiran Huang, Xiangyu Yue, Dongzhan Zhou, et al. Chemllm: A chemical large language model. arXiv preprint arXiv:2402.06852 2024.
A APPENDIX
A.I LLM USAGE DISCLOSURE
We used LLMs in the following limited ways to aid our research process:
Writing Assistance: ChatGPT-5 was used to polish and refine writing, including grammar correction, translation for clarity, and ETEX table formatting support. All generated or suggested content was carefully reviewed and edited by the authors before inclusion.
Evaluation (LLM-as-Judge): GPT-4o was employed as an automatic evaluator for long- form medical QA performance, and internally within the Multi-FAct benchmark, following established "LLM-as-judge" protocols.
Code Debugging: ChatGPT-5 was used in a supporting role to debug code. The authors independently verified the correctness of all outputs.
13

---

Published as a conference paper at ICLR 2026
The LLMs did not contribute research ideation the design of experiments. All final content and claims in the paper remain the responsibility of the authors.
A.2 DECODING PROCESS
Algorithm summarizes the complete LGCD procedure. At each timestep t, the framework eval- uates token-level confidence c, from the LAM. When
decoding proceeds with the LAM alone using Top-K sampling. When C < т, indicating potential factual uncertainty, LGCD activates the contrastive mechanism: (I) computing LoRA- MoBRA--aprxximater pretrained logits, (2) dynamically gating based on token-level confidence, and (3) applying contrastive weighting within the Top-K space. This confidence-driven approach enables LGGDD dynamically balance domain fluency and factual accuracy without requiring model retraining, making it particularly suitable for scenarios where language adaptation may compromise general knowledge.
Algorithm LoRA- Gated Contrastive Decoding (LGCD)
Require: Lunguage-Adapter Model MLAM. Approximated Pretrained Model Mаrои (for AWr), confidence
threshold Top-K size K, contrastive weighting hyperparameter
Ensure: Generated output sequence y
1: Initialize y
2: while )'(t] # <eos>, 1, 1, do
3: Compute LAM logits 1/AM
4: Compute token-level confidence es:
5: Ct = (anx(ssftaax(1)AAM)))
6:
if 2 then
7:
Apply Top-K masking: Tk = Topk (I,AM, K)
8:
Select next token y by sampling from the masked ILA
9:
else
Compute LoRA aeiRA---ppxximtted pretrained logits:
I.PTM 1IAM LORA(AW,, h!,4M)
12:
Apply Top-K masking: Tx Topk (1{ (ILAM K)
13:
Initialize 1s punirast = [-20, - oo] E De woab size
14:
for each token in vocabulary do if E Tx then Compute contrastive logit: 1. predtratt [1]
15:
17:
A.AM [i] + 3 (1:TM(() a -. (1)))
18:
else
19:
20:
end if
end
22:
Select next token yi by sampling from 1. poaiirast
23:
end if
24:
Append yt output sequence y
25:
end while
26:
return y
A.3 LORA-BASED PTT APPROXIMATION FEN VS QV VS ALL--AAER COMPARISON
LGCD relies approximating the PTM by applying rank matrices derived from the difference between PTM and LAM parameters. In our primary design, this approximation targets only the FFN layers, based prior research showing that FFNs
updates to the LAM, using low-
encode core factual knowledge in LLMs (Geva et al. 2020 Quu et al. 2024 Dai et al. 2023).
To assess the importance of this design choice, we compare three strategies for selecting which layers to approximate via LoRA during the offline distillation step:
QV-only: Apply LoRA decomposition only to the attention projection matrices (Q, V).
FFN-only: Apply LoRA only to FFN layers (our default).
All Layers: Apply LoRA both FFN and attention projection layers.
14

---

Published as a conference paper at ICLR 2026
Shows the performance of LGCD using each variant the Global MMLU benchmark under the zero-shot setting. Results show that FFN-only approximation consistently achieves the best or comparable performance, while QV-only performs worse nearly all models. Approximating all layers introduces marginal gains a few cases but often results in unstable or degraded performance, suggesting that unnecessary modification of attention layers may introduce noise. These findings empirically validate our FFN-only design as both effective and efficient for factuality-oriented ap- proximation of PTMs.
Lang. Model
QV-only
FFN-only
All Layers
zh htl//lamm--.cchieee-----ttttt
0,485
0.519
0.520
0,500
0.502
0.503
de phiccttteerrrt/llaaaaiaiiiiooeee
0.520
0.546
0.544
pt
0.330
0.357
0.340
ar Mohamme/aahha///trriiropp-----md
0,460
0.465
0.460
fa PartA//Domma-llmaa3--------sss
0,423
0.423
0,423
ja
0,499
0.507
0.503
ja Llyza//lamaa---LLLZZA----PP
0.499
0.509
0.509
LISTT-KONN/KOON--Lam3----
0.484
0.490
0.484
ko
0.471
0.479
0,461
id
Go haapppppmyyllmaa-------ssaaaaaaa innsstt
0,533
0.527
0.528
SczaradaaUUlizaLlanaa
0.375
0.399
0.395
Average
0.465
0.477
0.472
Table 6: Ablation study LoRA-based PTM approximation: comparing factual accuracy
Global MMLU (0-shot) when contrastive decoding uses knowledge reconstructed from different layer subsets.
A.4 CONFIGURATIONS FOR LGCD
LGCD is configured with task-specific decoding strategies address the differing demands of multiple-choice QA and long-form QA. While both settings use the same LoRA-based decom- position applied to all FFN layers with fixed rank of 32, the confidence threshold and usage of contrastive decoding vary by task.
For multiple-choice QA, high threshold leads to frequent activation contrastive decoding, ensuring that the model does not rely solely on the LAM but actively considers both the LAM and the PTM when scoring candidate answers. In this setup, LGCD functions not merely as fallback for low-confidence predictions, but as a mechanism integrate knowledge from both models during answer selection, enhancing factual reliability without requiring additional training. Decoding uses top-k sampling (k- = 100) with temperature 0.7 and top-p 0.9, and all contrastive updates are scaled with / 1.0.
fixed confidence threshold of r 0.9 across all languages. This
For long-form QA, the decoding configuration remains identical in terms of sampling and con- trastive scaling, but the confidence threshold language-spedde calibrated according resource availability (see Appendix A.8). This allows LGCD to adaptively determine when to apply con- trastive decoding based on the reliability of the LAM for each language- more frequently for low- resource languages and conservatively for high-resource ones.
Component
Multiple-Choice QA
Long-Form QA
Temperature
0.7
0.7
Top-k
100
100
Top-p
0.9
0.9
Contrastive 9
1.0
1.0
Contrastive a
0.1
0.1
LoRA (Rank/ Scope)
32/ All FFN layers
32/ All PFN layers Language-specific
Confidence
Fixed (0.9)
Contrastive Use
Joint scoring
Factual conrectinn
Table 7: LGCD decoding configurations for multiple-choice long-form QA.
15

---

Published as a conference paper at ICLR 2026
A.5 EVALUATION DATASETS FOR MULTILINGUAL MEDICAL QUESTION ANSWERING
This appendix provides detailed account of the datasets employed evaluate the multilingual medical question answering capabilities of the models discussed in this study. We have curated a collection open-ended medical QA datasets across various languages, drawing from both estab- lished benchmarks and language-specific resources.
Our evaluation utilizes open-ended medical question answering data. For several languages, the Healthcare QA task from the AIR- AIR-Bench 24.05 benchmark (https://githu) com/ MER-BenchhAIR-BR
serves the primary data source. This task comprises questions within the medical domain designed to assess model's ability to provide informative responses. Specifically, the original Healthcare QA data from AIR-Bench was directly used for evaluation in Arabic and German.
For broader linguistic evaluation where native high-quality long-form medical QA datasets were not readily available, we constructed evaluation sets by translating the English version of the Health- care QA task from AlR-Bench These translations were systematically performed using Google Translate. This methodology was applied generate the evaluation datasets for Persian, Indone- sian, Japanese, Portuguese, and Swahili. This approach enables a comparative analysis based on a consistent source structure across these languages, although with potential translation artifacts.
In addition to the AlR-Bench based datasets, we incorporated independent, language-specific medi- cal QA datasets for Korean and Chinese:
Korean: We used the Chucyyol//cenkkeGG7--5---- dataset (https:: Suggingfface co htt///aattett//huyyuuuee (ttrreyoooo///eeeeee----------
for long-form medical QA in Korean. This dataset, containing approximately 5,000 question-answer pairs, is a Korean translation, performed using DeepL, of medical QA data sourced from https://github. wwtp://uw..ooooo.ettteeee Liahan--ihhhaaaaaaaaaaaoo
Chinese: For Chinese medical QA, we utilize the cMedQA2 dataset (https:// // This dataset is a dedicated resource for medical QA in Chinese, comprising a collection of questions and corresponding expert answers within the medical domain.
github. com/ 3ttgg/shhnnn//MMMddeeee
A.6 BASELINE CONFIGURATION DETAILS
We compare LGCD against both decoding basedd and model-merging baselines. All decoding strate- gies use the same chat template with max_new_tonn nokenss 2048.
DECODING-BASE BASELINES
We evaluate five decoding strategies: greedy decoding (GS), contrastive search (CS), nucleus sam- pling (NS), and DoL.a All hyperparameters follow either the original paper Hugging Face imple- mentation standards.
In particular, DoLa adopts task- contrastive scheme: it uses higher transformer layers for multiple-choice QA and lower layers for long-form generation, following prior findings on factual calibration via depth selection.
Method
Hyperparameters
GS CS NS
Hugging Face default (greedy decoding)
penalty_alpha 0. 6, kop_k =
do_sample True, temperature - 0. top_p 0.9
DoLaa
do_sample False
dola_layers "high" (Multiple-Choice QA)
lolarrayers = "low" (Long-Form QA)
Table 8: Decoding hyperparameter settings for baseline methods.
16

---

Published as a conference paper at ICLR 2026
MODEL-MERGING BASELINES
We further compare LGCD with two weight-space integration buselines-SLERP and TIES insplemeemee using the me mergekit framework. Both methods perform symmetrie merg- ing between the PTM and the LAM with the following common settings:
Merge ratio: 0.5 (LAM)) 0.5 (PTM)
Merge range: all transformer layers
Data type: 6
Base model: LAM
Other parameters: mergekit defaults
A.7 NER-BASE BASED FACTUALITY PROBE
NER extraction. We measure entity-level differences between LGCD and the baseline LAM by applying general-purpose NER extractor to every generated output. Specifically, we use GPT-4o as the extractor. The schema is based coarse-grainet categories commonly used in NER (c.g.. LOCATION, DATE, NJMBER), and we extended it with DISEASE to
PERSON, ORGANI capture domain-releven mentions. In total, use 13 categories. Each entity is returned with its surface form, category, and character offset.
Comparison metrics. For each output pair, form entity sets ELGCD and ELAM- We compute [Еιсол[BLv)) We also track absolute entity counts per output. This setup allows Jaccard similarity [EaecDUELAwI us to test whether factuality gains from LGCD are reflected in increased coverage or correction of entity mentions beyond those produced by the base LAM.
A.8 BOGGUAGE-SPECIREE CONFIDENCE THRESHOLDS BASED ON DATA AVAILABILITY
Our decoding framework dynamically balances contributions between language-adapted model and a pretrained multilingual backbone based on token-level confidence threshold. When the model's token-level confidence falls below the threshold, incorporate contrastive logits from the pretrained model to supplement or correct uncertain predictions.
Given that the quality and extent of training for language-adapted models are often opaque, espe- cially in multilingual settings, we estimate data availability using the OSCAR 22.01 corpus Caswell al. (202T): Abadji et al.] (2022); Abadji et al. (202T) as a public proxy for the quantity of avail- able web-scale text per language. Languages are categorized into three tiers High, Medium, and Low base on word count.
We assign lower thresholds (e.g., 0.1) to low-resource languages such as Korean (ko), Indonesian (id), and Swahili (sw), allowing the pretrained model's generalized knowledge to exert greater in- fluence. Conversely, for high-resource languages like German (de) and Chinese (zh), which have abundant data, we set higher threshold (e.g., 0.8), favoring the language-speciff model. Medium- resource languages such as Japanese (ja) and Persian (fa) are assigned intermediate values (e.g., 0.4). This approach ensures informed integration of pretrained knowledge proportional to resource availability.
17

---

Published as a conference paper at ICLR 2026
Category
Languages (ISO code)
Word Count
High (2 2B)
German (de)
46.8B
Chinese (zh)
23.1B
Portuguese (pt)
18.4B
Medium (0.5-2B)
Persian (fa)
6.4B
Arabic (ar)
6.1B
Japanese (ja)
5.6B
Low (< 0.5B)
Korean (ko)
3.9B
Indonesian (id)
2.0B
Swahili (sw)
77MB
Table Language data categorization based on the public OSCAR 22.01 corpus.
A.9 EVALUATION ON EXTREMELY
LANGUAGES
To address concerns regarding the definition of "low-resource" languages and test the robustness of LGCD, we conduct additional experiments on extremely low-resource languages as categorized by Nigatu et al. 2024). Specifically, we evaluate three South African languages: Zulu, Tswana, and Northern Sotho.
Experimental Setup Following our established utilize meta-1lama /wwaaaaaaaaaammmmmmmmmm 1- 1-8B-Instruce as the PTM and africa-ise ieeecaainneelgggnnnnne ammaa 8b south-africa as the LAM. We conduct evalu- ations two benchmarks:
evaluation protocol, we
Since standard Global MMLU benchmarks are not available for these languages, we translated the English portion of Global MMLU into Zulu, Tswana, and Northern Sotho using GPT-5. We report accuracy under both 0-shot and 5-shot settings, comparing LGCD against the PTM, LAM, DoLaa, TIES, and SL.RR baselines.
Global MMLU:
Healthcare QA: We adapt the Healthcare QA task from the AIR-Bench 24.05 benchmark, translated into the target languages using GPT-5 following the methodology described in perform pairwise compar- isons, scoring the outputs on factual accuracy and fluency. We compare LGGDD against the LAM, DoLa, TIES, and SLERP.
Appendix A.5. GPT-5
employed
I.LM-as-a-judee
Consistent with strategy for low-resource tiers, applied confidence threshold of 0.2.
Global MMLU Results Table 10 and Table II present the accuracy on the translated Global MMLU benchmark under 0-shot and 5-shot settings, respectively. In the 0-shot setting. LGCD achieves the highest average accuracy (0.459), outperforming all baselines including DoLa (0.456). In the 5-shot setting, LGGDD matches Dol la with the highest average accuracy of 0.432, while consis- tently outperforming TIES and SLERP. Notably, LGCD achieves the best performance on Tswana in both settings, demonstrating robustness across different evaluation conditions.
I Language
PTM
LAM
DoL.a
TIES
SLERP
LGCD
Zulu
0.374
0.470
0.474
0.424
0.446
0.468
Tswana
0.358
0.448
0.452
0.386
0.446
0.456
Northern Sotho
0.366
0.438
0.442
0.438
0.460
0.454
Avg
0.366
0.452
0.456
0.416
0.451
0.459
Table 10: 0-shot accuracy on the translated Global MMLU benchmark for extremely low-resource languages.
Healthcare QA Results Tables 12 15 present the pairwise comparison results LGCD against
each baseline the Healthcare QA task. Against the LAM (Table 12. LGCDD achieves clear
18

---

Published as a conference paper at ICLR 2026
Language
PTM
LAM
DoLa
TIES
SLERP
LGCD
Zulu
0.392
0.430
0.434
0.414
0.422
0,431
Tswana
0.390
0.440
0.440
0.420
0.430
0.442
Northern Sotho
0.378
0.408
0,421
0.418
0.414
0.422
Avg
0.387
0.426
0.432
0.417
0.422
0.432
Table 11: 5-shot accuracy on the translated Global MMLU benchmark for extremely low-resource languages.
advantage across all three languages, with win rates of 39%-46% versus loss rates of 25%-31 %. When compared to DoLa (Table (3). LGCD maintains a favorable win-loss margin in Zulu and Tswana, while showing comparable performance in Northern Sotho. Against model merging base- lines TIES and SLERP (Tables 14 and (5). LGCD demonstrates a decisive advantage, particularly Zulu where win rates exceed 85%.
Language
Win
Tie
Lose
Language
Win
Tie
Lose
Zulu
41%
28%
31%
Zulu
43%
31%
26%
Tswana
46%
29%
25%
Tswana
40%
33%
27%
Northern Sotho
39%
31%
30%
Northern Sotho
37%
28%
35%
Table 12: LGCD vs. LAM
Table 13: LGCD vs. DoLa
Language
Win
Tic
Lose
Language
Win
Lose
Zulu
85%
12%
3%
Zulu
86%
12%
2%
Tswana
52%
27%
21%
Tswana
54%
23%
23%
Northern Sotho
53%
24%
23%
Northern Sotho
54%
27%
19%
Table 14: LGCD TIES
Table 15: LGCD vs. SLERP
A.10
DIRECT COMPARISON OF LAM AND LGCD OUTPUTS
Eveluation
Laber B. B:B
Score:
1.5 2002 Wil
Pivédins techetier
Incl iine complicaionn rlaynwets
Whakesoaza
Ircomeet chtin tun
12 CL. BD. F.B
trcludio zocotamabilea in
& Tar TrA M E O E. MRR. 6.011 CU1
prostemppat
LOCD. it. titt 1- 11 2510000
tttti R
1.3 Ridull: V B
Correct and concias cofitition
2. (i-tet P.P" Sightl C.P. F.Bodd 9-11
Covera main ak factors
& BP 11 F.R R. &. 11 1 4 11-118 E
WWaatoncas
(i-i. 18 9 SECICC
Sligita koo con proffaaatteee
Incuuko oevor"
vnaltal Fucterr
LOCO
Figure 7: Direct comparison of LAM and LGCD outputs for Japanese medical QA example (elyawLLama3-ELYZZAJ--88 with evaluation
A.11 ADAPTIVE CONFIDENCE THRESHOLD
We further study the robustness of LGCD to the choice confidence threshold 7 by introducing an adaptive thresholding mechanism that adjusts the current decoding dynamics. At each decoding
19

---

Published as a conference paper at ICLR 2026
Lang.
Model
0-shot
5-shot
Adapt.
Heur.
Adapt.
Heur.
zb
hil/llama--- 3-chincse----inne
0.506
0.519
0.529
0.543
zb
shenthi hamm//lmmm>--- SB Chinese Chat
0.504
0.502
0.544
0.543
DiscoReseurclvl Lama3- Disii-ic so-InntracttS8---00
0.531
0.546
0.569
0:574
pt
thaymison/gemmm partuguese lumna-2h
0.347
0.357
0.324
0.324
Aaweee/KKnsaaaniiii--eeee Crpp--..laman 1ac.l..mm-----8- -------8----iie Instrrettt
0.464
D.465
0.479
0.481
Ja
tokyetech 3 S-aallooo--- Instruct-vO.11
0.507
0,507
0.527
0.525
elyza/l waw.lllama--HHl B.ma-3-ELYYA.JPP....
0.507
0.509
0.528
0.528
KISTI- KONUKONN--aams -8--usstuuct2224477222
ko
MLP KTLii//llall--- Korean-BB Bilessom-
0.477
0.479
0.498
0.501
kd
GoTDCompanyllamma
linsrract
0.537
0.527
0.574
0.566
Cicaaad//UiizLllang
0.388
Average
0.473
0.477
0.495
0.498
Table 16: Comparison of adaptive and heuristic confidence thresholds r for LGCD across languages and instruction-turs LMs. Numbers denote task performance (higher is better).
step t, LGCD queries the LAM and obtains confidence
max (LAM M<t),
i.e. the maximum softmax probability over candidate tokens. LGCD maintains rolling history
H2 of recent confidence values with |H,] < 100. Once at least 10 values have been accumulated
(HH 2 10), we compute the adaptive threshold as
Tadaptive = clip(uw 0.50 0.1, 0.9).
(10)
where ja and H denote the mean and standard deviation of the values in H,, and clip(r, a, b) min (m. a), constrains the threshold to [0.1, 0.9]. Intuitively, T decreases when the model becomes less certain (lower mean or higher variance in recent confidences), and increases when the model is consistently confident. The heuristic threshold used in our main experiments can be viewed as a resource-awar but static instantiation of this idea, whereas Eq. equation 10 adapts online to the LAM's confidence trajectory.
compares the adaptive threshold Tadaptive with our heuristic threshold across multiple lan- guages and inttructionttunnd LMs, under both O-shot and 5-shot settings. The adaptive achieves performance that is slightly lower but nearly identical to the heuristic т (typically within 0.002- 0.01). Crucially, both thresholds consistently outperform strong baselines (PTM, LAM, DoLa, TIES, SLERP) across almost all languages (see Section 4). indicating that LGCD is robust to the precise choice of 7 and that our simple heuristic threshold already provides an effective and eegguageresource-aee aware criterion.
Table 16
A.12 EVALUATION ON NON- SNNOWLEDG--NNENNEEE TASKS
While LGCD is designed primarily to mitigate factual degradation language-adapted models, is crucial verify that it maintains the model's robustness across broader range of general NLP tasks, encompassing creative, narrative, and fundamental reasoning capabilities. To this end, we conduct comprehensive evaluations on four diverse benchmarks encompassing creative, narrative, and fundamental reasoning capabilities: XStoryCloze (commonsense narrative reasoning). the Aya Evaluation Suite (open-ended conversational quality). PAWS-X (paraphrase identification), and XNLI (natural language inference).
EStoryCloze (commonsense narrative reasoning) Lin al. (2021b). XStoryCloze is a multi- lingual extension of the StoryCloze Test, in which model must select the most plausible ending to four-sentence story. Because this task depends primarily on narrative coherence and commonsense reasoning rather than factual world knowledge, it serves direct test of whether LGCD harms reasoning or storytelling capabilities.
Each dataset supports only specific subset of languages, so evaluate in the languages where the benchmark's 's coverage overlaps with those of our adapted models. For XStoryCloze, this corre- sponds Chinese, Arabic, Indonesian, and Swahili, and we follow the standard zero-shot evaluation setting used in prior work. We compare the pre-trate pre-trained model (PTM), the ainguage-adapted model (LAMM), and LGCD.
20

---

Published as a conference paper at ICLR 2026
Lang.
Model
PTM
LAM
LGCD
zh
htV/llama- 3-chinese wwwineee----inseee 88-insttrcct
0.689
0.682
0.695
shenzhi-wange watth/.aaaaa-/--33-- Baaaa-----hhiie-Chha
0.689
0.681
0.670
ar
MohamedRashad/A IatiaaddAraae-Omrre -rahi--Oro--.aama-3 -lama-3-8B-Instuue
0.609
0.608
0.587
GoToCompany/llaaal hatpaaanyyllaaa-----pap- -a-aa--ttttt--aaaataa ceiitti----iinttuu vi-instruct
0.669
0.721
0.708
SW
Jacaranda/Ulizaa Ulizadda//izzallaaaaa
0.567
0.704
0.697
Avg.
0.645
0.679
0.671
Table 17: Performance on XStory (commonsense narrative reasoning). Metrics are accuracy. LGCD matches or slightly trails LAM, and overall performance remains comparable, indicating that LGCD does not meaningfully harm reasoning or narrative capabilities.
Across models, LGCD closely tracks the LAM performance, with only small differences in accuracy (Table [7p. Overall, LGCD preserves narrative understanding and commonsense reasoning while focusing on mitigating factual degradation.
Aya Evaluation Suite (open-ended creative dialogue) Singh et al. (2024b). The Aya Evaluation Suite contains 26,750 open-ended conversation-style prompts across seven languages, and is de- signed to evaluate creativity, coherence, and conversational quality rather than factual accuracy. As above, we evaluate only in languages that overlap with our adapted LAMs, namely Chinese. Arabic, and Portuguese. We perform GPT-4o-based pairwise evaluation comparing LGCD against LAM, and report the percentages of prompts where LGCD is preferred (Win), tied (Tie), or worse (Lose).
Across all tested languages, LGCD preferred in roughly 55-58% of cases open-ended dialogue generation (Table I8). indicating that LGCD does not harm diversity or expressiveness in genera- tion and often yields more coherent and controlled outputs by avoiding over-confident hallucinated continuations.
Lang
Model
Lose
hilllama---chineee
58.0%
8.4%
zh
3B-Chinese-Chat
56.4%
2.0%
41.6
ar
MohamedRashad// Arabic Orpo Jaama 8B-Instruct
54.4%
0.4%
45.2%
GoTTCComparylllmm--8----- -ob-umuuut-------uuuaaaaa----- -innstruct
55.2%
3.2%
41.6%
Table 18: GPT-4o pairwise preference of LGCD vs. LAM the Aya Evaluation Suite. "Win" indicates that LGCD is preferred over LAM for a given prompt, and "Lose" the opposite.
PAWS-X (Paraphrase Identification). PAWS-X is cross-lingual adversarial dataset used to de- termine if two sentences are paraphrases. As shown T9 LGCD achieves an average accuracy of 0.555 across Chinese, Japanese, and Korean models, slightly outperforming the LAM average of 0.548. Notably, LGCD shows consistent improvements in Korean models and one of the Chinese models, suggesting that our method's intervention robust paraphrase identification tasks.
Lang.
Model
LAM
LGCD
zh S-china-----hicc--eeeee- 8b-instruct
0.528
0.553
Chenzhi-wang/LL.e Llaman//Laaa3-888 SB-CCinese-CCC
0.579
0.553
ja tokyotech-IIm/L.n haah--lllmmmmmma--------- -awaaaa-wwllllwSS---------- Swallow-SB-IInstret--..
0.569
0.559
0.554
0.559
KISTI-KON/KONN--Llnns
S---Instruct222240777
0.531
0.552
MLP-KJLim/llame- BrrcannBlloosom--e ko
0.529
0,552
Avg.
0.548
0,555
Table 19: Performance PAWS-X X. LGCD maintains improves performance compared to
LAM, demonstrating robustness in detecting paraphrases.
XNLI (Natural Language Inference). We also evaluated performance on XNLI, a task predicting textual entailment (entailment, contradiction, neutral). Table 20 shows that LGCD performs on par with LAM, with slightly higher average score (0.331 vs. 0.329). In Arabic and Swahili, the performance remains identical, while Chinese, results vary slightly depending on the base model but remain comparable.
21

---

Published as a conference paper at ICLR 2026
Lang.
Model
LAM
LGCD
0.340
0.336
zh
Baazzhi-waag/Llaaee (ama3-8B--- -B-C-BBBee-
0.336
0.347
RohaamedRashad/Araaiie Orpo--lopoaa Llama-3-8B 3-1---3-8--B--B- Instruct
0.320
0.320
Jacaranda/Uiizallamaa
0.320
0.320
Avg.
0.329
0.331
Table 20: Performance on XNLI (Natural Language Inference). LGCD preserves the reasoning capabilities of the adapted models, showing no significant degradation in NLI tasks.
A.13 GENERAL-PURPOST KOREAN MMLU EXPERIMENTS
We first examine general-purpons SFT scenario in Korean to study how different fine-tuning strate- gies affect factual knowledge. We fine-tune purpose SFT dataset used for training the KONI model, and evaluate zero-shot performance on the Korean portion of the Global MMLU benchmark.
an 1KK Korean general-
Table 21 shows that all standard fine-tuning methods (full- parameter,,,, attention-only and FFN-only) suffer from catastrophic forgetting, performing worse than the original base model (0.437). Notably, attention-only fine-tuning yields the largest degradation (0.412), despite updating the parameters most directly associated with token-to-tokes interactions. In contrast, LGCD, which dynamically composes the language-adapter model (LAM) with the pre-traines model (PTM) at inference time, achieves the best performance (0.462), surpassing both the base model and all fine-tuning variants.
Model
O-shot (Global MMLU, ko)
base model (no finetuning) train (all layers)
0.437
0.432
train (attention only)
0.412
train (FFN only)
0.431
LGCD (ours)
0.462
Table 21: Zero-shot performance the Korean portion of Global MMLU after general-purposs SFT in Korean. All standard fine-tuning strategies underperform the base model, whereas LGCDD, which composes the PTM and LAM at inference time, yields the highest accuracy.
These results indicate that freezing FFNs and updating only attention layers not a reliable way avoid catastrophic forgetting, and that dynamic inference-time composition via LGCI) provides a more robust approach to preserving and enhancing factual knowledge. In the next subsection (Section A.14). we show that LGCD also improves accuracy in domain-matched medical QA setting where fine-tuning itself is beneficial.
A. DOMCIN-MATCHED MEDICAL QA EXPERIMENTS
We further evaluate LGCD in task- aligned, domain-matehi matched setting, where the adapted model is fine-tuned on the same downstream task which it is evaluated. This setup tests whether LGCD remains beneficial even when the fine-tuned model already possesses strong domain knowledge.
Experimental setup. We consider three languages (Korean, Chinese, and Japanese) and evalu- on GlobalMedQAA] multilingual medical multiple-choice dataset designed to assess medical knowledge in LLMs. The base model is met a -aaalllle wwwlaaaaaaaiimmm / Llama-3 3. 1.1- 8B ruct. We com- pare the following adaptation variants: (i) full fine-tuning of all layers, (ii) fine-tuning attention layers only, (iii) fine-tuning feed-forward (FFN) layers only, and (iv) LGCD (ours), which com- poses the base and adapted models inference time. All models are trained and evaluated on the same GlobalMedQA task for each language.
MedQA
22

---

Published as a conference paper at ICLR 2026
Model
Chinese
Japanese
Korean
Average
base (no finetuning)
55.21
41.70
48.02
48.31
train (all layers)
59.20
43.70
49.15
50.68
train (attention only)
57.40
42.90
47.59
49.30
train (FFN only)
59.40
43.00
49.09
50.50
LGCD (ours)
60.60
43.70
49.42
51.24
Table 22: Performance on GlobalMedQA in a domain-matched setting, where all adaptation
variants are trained and evaluated on the same medical multiple-choice task. Metrics are accuracy (%). LGCD achieves the best average performance across the three languages.
Across Chinese, Japanese, and Korean, LGCD attains the highest average performance, outperform- ing full fine-tuning and layer-specific fine-tuning variants (Table 22). These results indicate that LGCD is effective not only when the adapted model forgets some truths, but also when the adapted model retains domain knowledge very well, and that composing the base and adapted models at inference time can provide additional gains even in fully domain-matchen matched scenarios.
A.15 TOP-K K MASKING VS. ADAPTIVE PLAUSIBILITY CONSTRAINT
A natural alternative to LGCD's Top-K gating mechanism is the Adaptive Plausibility Constraint (APC) originally proposed in Contrastive Decoding (Li et al. 2023b), which retains only tokens whose probability exceeds a fraction of the maximum probability:
VaPC Úz)) {y V PPTM(y z, ū.)<)) 2 mmax x, 0ct))
While APC adaptively narrows the candidate set when single token dominates, Top-K masking provides a stable, language-agnostic candidate pool that preserves a fixed number high-probability alternatives at every step. This stability is particularly beneficial in multilingual contrastive decod- ing, where the entropy landscape varies significantly across languages and tanguage-speccffie models.
To validate this design choice, replace Top-K masking in LGCD with APC-based masking (using a = 0.1 following Li et al. (2023b)) and compare both variants on Global MMLU and long-form medical QA.
Multiple-choice QA. Table 23 reports Global MMLU accuracy across 12 language-model pairs under 0-shot and 5-shot settings. LGCD with T masking consistently outperforms the APC variant, achieving +2.8% higher accuracy on average the O-shot setting and +2.3% in the 5-shot setting.
Lung
Mudel
0ashot
Soshut
APC
Top-K
APC
Top-K
zh
hilllama-3 chiness Sb- instruct
0.460
0.519
0.500
0.544
zh
sheuzhi- wang/Llama B---B-Chincse-CCC Chinese-Chat
0.494
0.502
0.500
0,543
DiscoReseurch// haw.serrrrr//raaaaa DiscoL Instructtt-tttttteo Instruct-SDvvol
dhaymison/gemmss- Sertuguse-llana-2
0.286
0.357
0.307
0.324
CatuumdRuuhad/Arrii Orpo Llooo-llmaaa 3-8B
I-strussuuutttt Instrct
0.443
0.465
0.466
(148)
PanAA/Dema- Jama I-8tt-nnstract
01.43
tokyotech- Jama. 3.5 Swalllow88B 8BB 38-IInsract-vO...
0.494
0.507
0.522
0.525
elyza/.Jama- Jama- 3-EJY2A-JP-88
0.509
0.528
ko
KISTI- KONI/KONI-LLA lama-----aaa---- Lann3-8B--nstmacc 20244779
0.458
0.490
0.481
0.511
ko
MLP- K KTLim/llaaa- Korean- Biloan----
0.464
0.479
0.487
0,501
GoToCompanylllms -a--3-----p----- bab-spt-shhbbbtai-VV L-imstrrd
Sw
Jacaranaa Uanzalamaa?
0.352
0.399
0.353
0.409
Average
0.477
0.475
Table 23: Global MMLU accuracy comparing LGCD with Top-K masking against LGCD with APC-based masking across languages and instruction-tuned LMs. Bold indicates the better variant for each row.
presents pairwise GPT-4o evaluation results long-form med- ical QA, comparing outputs generated by LGCD (Top-K) against those from LGCD (APC). On av- erage, LGGDD with Top-K masking wins 53.3% of comparisons versus 44. 7% for the APC variant,
Long-form generation. Table 24
23

---

Published as a conference paper at ICLR 2026
with only 2.0% ties. The T variant is preferred in 8 out of 12 language- modee pairs, confirm- ing that the fixed-size candidate pool leads to more fluent and informative multilingual generations. These results suggest that, in the multilingual contrastive decoding setting, the consistent candi- date coverage provided by Top-K masking is more effective than APC's adaptive but potentially over-aggressive pruning strategy.
Lang.
Model
Tie
Lose
zb
Dillamu-3-chinese Sh- instrtctctt
61.0
0.7
38.3
shenzhi-wangl hhtp://ww//ll.33-58PP21 Laam3--8--- Ca--Chheeee-hht
5.7
de
DiscoReseurch/L.lame lamal Discol Instracc-----vvd
58.9
0.0
41.2
Daymissovgeemaa-poonni
1.0
38.0
ar Mocamme@Raaaad/Arahiccc Orw.-iii-r-----aaaaa-- lama-3-8BB Ifs-nnstnet
47.7
1.0
51.3
fa
PartAl/Demma Lama3-8B-Instraes
51.9
1.9
46.3
ja
tokyotech-IIm/1 waamn--SwallloS555 10-
ja chyes/Liana-3444 Maamaa--ELLZA-J----3
47.0
0.0
53.0
DISTI-KONIKONN 1amk0n----- 1at-mm--------see--200000 202247992
53.6
0.4
46.0
ko MLP-KTLing Bacollannnan---------l-a-e-
46.0
3.4
50.6
GoToConmanyyllaaa --t-llaaaaa----pp-p---- -------p----haaaattiee -iraabataii---intuuuue
57.3
5.4
37.3
Jacarands/ Uizaalaaa
Average
53.3
2.0
44.7
Table 24: Pairwise GPT-4o evaluation on long-form medical QA: LGCD (Top-K) vs. LGCD
(APC). Win Tin/Loe (%) are reported from the perspective of the Top-K variant. Bold indicates the higher value between Win and Lose for each row.
A.16 ADDITIONAL MODEL-MERGING RESULTS FOR GLOBAL MMLU
In Table report two representative merge methods, TIES and SLERP. This appendix extends the analysis to fourteen widely used model-merging methods, including arcee fusion Goddard et al.
(2024). breadcrumbs Davari & Belilovsky (2024), breadcrumbs.tt Davari & Belilovsky (2024).
dare linear Yu et al. (002)). dare_tics Yu et al. (2024). della Deep et al. (2024). della-linea, Deep
al. (2024), karcher, linear Wortsman et al. (2022), multislerp, sce Wan et (2025), slerp Shoe-
make (195), task arittmeete Ilharco et al. (2022), and ties Yadav et al. (002b)). All evaluations
use the same setup as described in the main text. For method definitions and references, please re-
fer to the official MergeKit documentation: httpps:/// thub. com/amcmaaoooooooo aar/a-rceeeee a wetcceeeaimmrreeeee //mr i t/
blob/ ma n/ docs/ me rge mww.mots..... md
Lang.
Model
Method
0-shot
5-shot
zh
arcee fusion
0.471
0,510
zh
B-hilam---- -b-instructt
breadcrumbs
0.490
0.532
zh
htl/llama-3-chinsee -b-issrructt
breadcrumbs' ties
0.490
0.532
zh ht//lama-3-chinese-888 instruct
dare linear
0.460
0.475
zh ha/Mlmma---hiinee-----ittttu
dare 'iees
0.469
0.497
zh
ifMllmma---chinsse-------mnn
della
0.312
0.309
zh
ht/Mlama---chhineee----mmmamaa
della linear
0.346
0.368
zh ht/Mlama--chiinee----iiiii ibsiuuttttt
karcher
0,488
0.529
zh
ht/llama---chnese-Bhh -ht-
instruct
linear
0.490
0.531
zh
at/hamma---chinee----iinmi
multislerp
0.441
0.496
zh
if/llmma--chiinss-----iit-t
sce
0.441
0.498
zh
ht/llmma---hhineee-----mm-a
slerp
0.492
0.531
zh
ht/llamm---chiinse-------mmae
task arithmetic
0.490
0.532
2h
allllamm----hhinns----niinntt
ties
0.475
0.524
zh
shenzhiivaang/LLaaa3388 ChB-----hhhhChCCB
arcee fusion
0.480
0.527
zh
shenzhi-wanng/Llama3--- Se--CChhneeehhhr
breadcrumbs
0.482
0.530
zh
shenzhi-waang/Llamas-- Chinese-nhhhhiie
breadcrumbs 'ties
0.482
0.530
zh shenzhi-wang/LLlmaa-- C-s--Chhneee--ee Chat
dare' linear
0.483
0,529
zh shenzhi-wang/LLlaas--88 www.aaaggLLaaa-----hhheeeee
Chat
dare' ties
0.483
0.529
zh shenmni-wang/lamma----Chhines
della
0.473
0.525
zh stenzhhiwwang/LLLmm3-----CCCihC
della linear
0.472
0.524
zh shenzhi-waang/llaama8---------
karcher
0.482
0,529
zh
shenzhi-wann/LLlmaa--e -B---hnBe---------- Chat
linear
0.482
0.529
zh
shenzhi-wang/Llamaa Law.llmmaa------hneeee
multislerp
0.476
0,526
zh
shenzhi-wange
Benzziiwang/LLamaa3- B---a-------h---CBCC
sce
0.475
0.526
Continued page
24

---

Published as a conference paper at ICLR 2026
Table 25 continued from previous page
Lang.
Model
Method
O-shot
5-shot
zh Sannziiwwan//Llmmaa3-3 B---Chhnsee-CLCe
slerp
0.482
0.529
zh macnzti-wang/LLamas--- -aaeel--haa--e Chast
task arithmetic
0.482
0.530
zh wonzzhi-waggLLlaaa-88---Co Chass
ties
0.505
0,550
de
arcee fusion
0.505
0.556
de AesooResaacch/LLaaaaaiiiooooo
breadcrumbs
0.515
0.567
de srccReesechh/ammm--DDioooooooooo
breadcrumbs' ties
0.515
0.567
de
dare linear
0.505
0,556
de
dare' ties
0,504
0.558
de de BakoeeeaactLLaaaaaa-ioooooooooos de
della
0.404
0.503
della llinear
0.415
0.512
karcher
0.514
0.563
de de Dissoferacc/LLmmmmDDsseeeeonnnne de
linear
0,516
0.567
multislerp
0.463
0.532
0.463
0.532
de de Dise@ssecc///ammmmDDDsoeeeenents DiscoResearch/
slerp
0.516
0.565
task arithmeetic
0.515
0.567
de
ties
0,502
0.565
pt poay portuuguese-luana-26
arcee arcce'fusion fusion
0.324
0.301
pt hww.mmmooongnmmmmpprtuususs--ees
breadcrumbs
0.332
0.322
pt
breadcrumbs' ties
0.332
0.322
pt
dare linear
0.327
0,314
pt
dare ties
0.323
0.318
pt
della
0.273
0,277
pt
thaymison/gen
della llinear
0,258
0.273
pt
rhaymison/ wwtrmmiio//gmmaaaaae
karcher
0.335
0,323
pt
rhaymison/
lincar
0.332
0.322
pt
multislerp
0.294
0.301
pt wws.so//gemiiinn-ooruuueeeeeeee
pt
sce
0.293
0.301
hayymisoongeamm--ppruuuuussuuuuu
slerp
0.334
0.323
pt haaymissonggenmm--poouuuuuuuuuuu
task arithmetic
0.332
0.322
pt hhyymiisnn/emmm--pppruuuuuuuuuus
ties
0.334
0.327
MobamceRRhha////rraicroooo--oand Instruct
arcee fusion
0.413
0.466
ar
Nohnmmeillcchh//ttraaccooooooo.m Instrucct
breadcrumbs breadcrumbs 'ties
0.418
0.466
ar Mohnmmdlssthh//rrrbi--po-------- Instruct
0.418
0,466
ar Mohanceleehh////rrrccr-pm--aaaaa
dare linear
0.420
0,466
dare's dare'ties
0.418
0.466
della
0.407
0.453
ar Nonneeelsssha/tttaiiccoooooaaa--
della llieaa
0.406
0.453
ar Moharcelssshh//ttalicc-ppLaaaaaa
karcher
0.417
0,465
ar OohaureUisshh///rrrri------aaaaa Instruct
linear
0.418
0.466
multislerp
0.408
0.454
ar MohammdRRshha//ttrii---ooo---omo
see slerp
0.406
0.454
ar Nohamne-isshh///Arnttt---p------
0,418
0,466
ar Noinndllsshh///rrbiro-ooolaa----
task arithmetic
0.418
0.466
ar Mohaneeeiistaw/rrrrrriooooo-----
ties
0.433
0.476
fa
TrrrA//Domma-Lmmaa------nnnnu
arcee fusion
fusion
0.400
0.449
fa AarAooomaaLLamaa
breadcrumbs
0.401
0.449
fa MritAl/Dona-LLamatt
-nssstuuuctttt
breadcrumbs 'ties
0,401
0.449
fa
dare' linear
0.400
0.448
fa
Part AlDDooma--Jama33888-mmme
dare' ties
0.399
0.449
fa
Part MonnA//Domaa-Lama-----n-nmm
della
0.400
0.450
fa
della linear
0.399
0.449
fa
Part A
karcher
0.398
0.448
fa BarAAl/Daamaa-Lamaa-8----ii
linear
0.400
0.449
fa
multislerp
0.400
0.450
fa PartAl/Derma-Lamm-------m
sce
0.402
0.449
fa PartAl/Dorna- Llama3- Instruct
slerp
0,400
0,449
fa HartAl/Dona-Llamme Baamaa--BBInnce 8B--Inttruct
task arithmetic
0.401
0.449
fa
ties
0.429
0.471
ja Mkayotech-ImmLLlamm---- S---S-aloo-----nssneeeee
arcee fusion
0.458
0.502
ja tokyotech-IImi Penotech-Ilm/L.lamaaa - 1
breadcrumbs
0.473
0.515
Continued page
25

---

Published as a conference paper at ICLR 2026
Table 25 continued from previous page
Lang.
Model
Method
O-shot
5-shot
ja
Seyyotch-IIm/Llam--333 -alaammm--wwllo-------- -B--nstrutt--O.....
breadcrumbs ties
0.473
0.515
ja
Seayottch-IIm/Llamge
dare' lincar
0.456
0.495
ja
tokyotech-IIm/1
dare' 'iess
0,461
0,508
ja
della
0.429
0.478
ja SarpllcchhhnntlmmmmmSaSSalollll-
della 'linear
0.427
0.481
ja
tokyotech-IIm/L.ln
karcher
0.472
0.519
ja
linear
0.473
0,516
ja
multislerp
0.456
0,510
ja
Srpotech-IIm/Llanneee Ilno:Lamaa33SwalllowwS3 Instructtv-.
0.455
0,510
ja
lokyotech-IIm/Lllanss
slerp
0.474
0.517
ja
task arithmetic
0.473
0.515
ja clyzalLLamaa3--LLYZAA-----
ties
0.482
0,528
ja
arcee arsconfusion
0.440
0.472
ja alyaadLLamaa---ELYZZA-----1
breadcrumbs
0.471
0.505
ja clyzallaamaa3-ELLYZZAA-PP11.
breadcrumbs ties
0.471
0.505
ja clyzalLlamaa3--LLYYZA-P--11
dare linear
0.449
0,470
ja dlyzalllama-3--LLYZAAA-P.
dare ties
0.453
0.482
ja alyaa/Llamaa--ELLYZAA-----
della
0.411
0.431
ja clyzvLLama-33ELLYAAJJA..
della liiear
0.387
0.424
ja clyzalllamaa33HLL7AAA--PP-1
karcher
0.474
0.513
ja clyralllama---BLLYAA----
linear
0.469
0.504
ja elyzafllamaa3-LLLZZAA-----1
multislerp
0.452
0.488
ja 2lyaa/Llamaa--ELYZZA-P---
0.451
0.488
ja Byzza/Llama-3-ELLZAA-II
slerp
0.470
0.505
ja alyaaLlamma---ELYYAA---.
task arithmetic
0.471
0.505
ja LlazallLama33BBLYYZA-P---
tics
0.470
0.506
ko JISTT-KON/KKON-LLamaa- 118-Insttutt-22240722
arcee fusion
0.439
0.486
20240729
breadcrumbs
0.452
0.503
ko
KISTI-KON//KON/Lamm33898 fnstruct-202400799
breadcrumbs' ties
0.452
0.503
ko
dare linear
0.438
0,482
ko
dare'ties
0.442
0.485
ko
KISTI- WIATT-KOO/OON-Lllniinnne Instruct-2024001
della
0.409
0.443
ko
KISTI-KONIJIN KONI/KONI-LLama Instruct-20240722
della linear
0.396
0.444
ko -rt.-O/////KOON///aaaaa----nnntt
-la---aaaaa--------tt---------- Ies--nssrrutt-t0 20240729
karcher
0.450
0,500
ko
linear
0.452
0.502
ko BASTT-KONI/KKON11_aaa2---------1 0-na--8880--mmmmc--22220772222
multislerp
0.436
0.476
ko JISTI-KONJKKONILLamaa--- Instruutt-22200229
0.436
0.477
ko JIST-KKON/KKONL-Jaaa3BBBB 0t...---nnntruut22222779999
slerp
0.451
0.503
ko
task arithmetic
0,452
0.503
ko LLL-KTTLim/lmmaa-KKrcnBBBBBlB
ties
0.469
0.507
ko
areec'fusion
0.360
0.431
ko MLP-KKTLii/lllaaa-3-onnnnn
LLL-KKTLim/llama---KKrannnn
breadcrumbs
0.407
0.470
ko
breadcrumbs 'iiss
0.407
0.470
ko TLL--TTLii//lmmm33KKnaaaannllll
dare linear
0.374
0.449
ko
dare 'iess
0.373
0.450
ko
della
0.310
0.392
ko LLL-KTTLim/llam----Korean
8B
della liiear
0.307
0.369
ko
MLLP-KTLLim/llaa----Krraa
karcher
0.215
0,242
ko
MLP-KKLLim/llaaa---Koraan
linear
0.408
0.469
ko
MI.-K
multislerp
0.352
0.433
ko Lu.--TTLii/lllmm-----meennn----
sce slerp
0.353
0.432
ko ML.-KKTTimm/llmm----koaannn
0.392
0.466
ko
task arithmetic
0.407
0.470
ko LLLPATTLii/llmma3--Koeeeallllll
ties
0.380
0.469
id TofCCommaayyllnnnnkssspphhhhhhha
arcee fusion
0.466
0.500
id ToToCommaaayllllmm-----aaaaaaaaa
breadcrumbs
0.511
0.562
id GottCCommpnn/llmmm------sssshaaa
breadcrumbs 'iess
0,511
0.562
id GoToCompant
SattCCooaaallllaa---------aaaaaa -instrutct
dare linear
0.490
0.539
id ToTTCCmmaanylllams------shhhaaaa
dare' ties
0.486
0.517
id GGTTDCCCmmaa////mmmm--///psshhsa
della della linear
0.456
0.509
id Goooocompayllllaa-------aaaaaaaa
0.463
0.461
Continued page
26

---

Published as a conference paper at ICLR 2026
Table 25 continued from previous page
Lang.
Model
Method
O-shot
5-shot
id
GoTioCompanylllma388h Saaaammmma---------saaaataaa-i -iintrruct
karcher
0.511
0.558
id
haammpany/llma----------h--aaaae -innstruct
linear
0,511
0.560
id
wtt.r/maaalllllmmm----ts-aaaaaaa -iinstruuc
multislerp
0.505
0.560
id
0.507
0.559
id
slerp
0.512
0.562
id
task arithmetic
0.511
0.562
id
ties
0.528
0,571
sw
Jacaranda/ UlizaL
arcee fusion
0.321
0.427
Facaranda/Ulizaa Lamz
breadcrumbs
0.395
0,427
sw
Jacaranda/ Cliaandd/////aaaaaaaaaa lamza
breadcrumbs ties
0.395
0.275
sw
Jacaranda/ Ulizal laam3
dare' linear
0.282
0.336
SW
Jacaranda/ U Ulizal llama3
dare ties
0.298
0.255
J Ulizallama3
della
0.260
0.227
Jacaranda/U UlizaLlama
della linear
0.245
0.422
SW
Jacaranda/ UlizaLlama3
karcher
0.392
0.426
SW
Jacaranda/ UlizaLlamaa
linear
0.396
0,350
SW
Jacaranda/ U lamaa3
multislerp
0.340
0,352
Jacaranda/U Saaida///liaaaaaaa
0.340
0.426
SW
Jacaranda/U UlizaLlamaa
slerp
0.397
0.427
Sw
Jacaranda/ Uliiaa/ Jama3
task arithmetic
0.395
0.381
SW
Jacaranda/ Ulizal Jamm3
ties
0.380
0,381
Table 26: Results of fourteen model-merging methods on Global MMLU under O-shot and 5-shot settings.
27


Reviews:
[{"review_id": "1cgjZgPOvk", "reviewer": "Reviewer_DKFz", "summary": "I'm not confident enough to review this paper since I'm not familiar with multi-lingual tasks / LoRA technology.", "questions": "n/a", "limitations": "", "rating": 6, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "n/a", "weaknesses": "n/a", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5icrVCsZ7B", "reviewer": "Reviewer_Cujt", "summary": "This paper proposed a method to resolve catastophic forgetting issue for LLM. The motivation is that LLMs often \"forget\" general knowledge after the model was fine-tuned for specific domains. This can lead to factual errors when user query the model. The authors propose a method named LoRA-Gated Contrastive Decoding (LGCD). The method is a training-free method that aims to help LLMs recover lost knowledge during text generation. The method functions by extracting forgotten knowledge through computing the difference between original and adapted model weights, then creating a lightweight LoRA approximation, monitoring the model's confidence and only applying intervention when some empirical confidence threshold is reached, and using contrastive correction to adjust predictions with the original model's knowledge while preserving fluency of generation. Experimental results show restoring factual accuracy on some datasets.", "questions": "see weakness", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The method delivers consistent performance across many settings, with particularly good gains where adapted models underperform their originals. Both automated and human evaluations are conducted.\n\n2. The paper shows which model layers matter most and demonstrates that LGCD adds new factual content rather than just reordering existing predictions.", "weaknesses": "1. The method needs full access to both original and adapted model weights, which is not practical in some cases and can cause extra GPU/CPU memory, and inference overhead. The method is impractical\n\n2. The confidence threshold is crucial but currently set using indirect heuristics. How can we trust the logit based confidence? Are they reliable? More reliable and adaptive approaches for automatic threshold selection should be considers. \n\n3. The method works for moderate adaptations but can be struggling with models that have been extensive domain-specific trained where there is high-rank differences between model weights. Another question: what if the adapted model did not forget the truth? DO we still need to extract from original model?\n\n4. The method introduces extra decoding overhead, which is problematic for high-throughput applications. Meanwhile, If we have weight of original model, we can just inference it to get answer, why do we need the extra lora component? Besides, how can we guranttee that the output from original model is correct? The pretrained model can hallucinate as well.\n\n5. The figures/table is hard to read with very small font.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xM2VQeESRO", "reviewer": "Reviewer_Pezx", "summary": "The paper presents LGCD, and its core idea is that fine-tuning sometimes hurts a model’s original factual knowledge due to catastrophic forgetting. However, we still want the model to be fine-tuned (for example to better follow instructions in a specific language). In this situation, we can adaptively apply the base model when a decoding step requires factual knowledge and use the fine-tuned model otherwise.\n\nThe use of the base model is replaced with a LoRA-based approximation of the difference between the base and fine-tuned models. The decision of whether to apply contrastive decoding or not at each step is determined dynamically based on the model’s confidence.\nThe experiments are across benchmarks such as Global MMLU, Multilingual TruthfulQA, Multi-FAct, and medical QA. The results show that LGCD consistently improves factuality and reduces hallucinations compared to other baseline methods like decoding or model-merging, without any additional training.", "questions": "The authors propose that applying LoRA only on FFN layers can recover the base model’s knowledge. However, it might be better if during the alignment stage only the attention layers are fine-tuned while freezing the FFN layers to prevent forgetting. Compared to this contrastive decoding method that is more like a remedy for the forgetting issue, directly fine-tuning non-FFN layers only could be a more straightforward and potentially effective option. Have the authors tried or considered this method?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-motivated by the catastrophic forgetting problem and proposes a concrete solution by contrastive decoding.", "weaknesses": "1. The authors proposed Top-K masking, which is actually very similar to the Adaptive Plausibility Constraint (APC) proposed originally in Contrastive Decoding (Li et al., 2022). Why not just use APC? Top-K tokens may include low-probability tokens when only one token has a large probability but K > 1, so more low-probability tokens are still included. In contrast, APC can exclude these low-probability tokens when there is only a single token with high probability.\n2. The method assumes that both the pretrained model and the adapted model weights are available, which may not always be true. Other baselines such as DoLa only require access to the adapted model, so the comparison may not be entirely fair.\n3. The method introduces four hyperparameters (τ, α, β, K)—confidence threshold, contrastive weights, and Top-K size—which require language- or model-specific tuning with some “training data”. This could limit its generalization ability to other models or settings.\n4. The evaluation focuses mainly on factual QA and medical-domain QA, while the effects on creative, open-ended, or reasoning tasks remain unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Oj3KeNaDKn", "reviewer": "Reviewer_GJXc", "summary": "This paper proposes LoRA-Gated Contrastive Decoding (LGCD), an inference-time framework designed to improve factual accuracy in language-adapted LLMs that have undergone continual pretraining or instruction tuning. The method approximates the pretrained model’s factual knowledge by extracting low-rank updates from FFN layers, and dynamically switches between the adapted model and the approximated pretrained model based on token-level confidence. Experiments across multiple languages show improvements in factuality without additional training or access to pretraining data.", "questions": "1. How do you think your method will compare to replacing equations 3 and 4 with actual LoRA CPT on the FFN layers instead of SVD-derived $A_\\ell\\$ and $B_\\ell\\$​? Can you show that?\n\n2. How exactly did you empirically show that your method mitigates hallucination?\n\n3. Can you provide results for benchmarks beyond QA tasks? The Multi-Fact eval (Table 4) only shows results for your method and PTM, which is limiting, as Figure 2 shows that other methods are also competitive.\n\n4. With LGCD, do the pretrained and adapted models have to use the same architecture? Table 1 indicates that you used models with a mix of Gemma and Llama architecture.\n\n5. At inference time, do you deploy both PTM and LAM while alternating the decoding with LGCD? A short description of that part will help.\n\n6. Do you have an idea of what happened to Japanese in Figure 2?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method does not require access to the original pretraining data, nor retraining or additional model parameters.\n\n- Only extracting weights from the FFN layers makes implementing the method lightweight.\n\n- The confidence-based gating is intuitive and allows the method to selectively intervene only when needed, balancing fluency and factuality.\n\n- Experiments across multiple languages demonstrate general applicability.", "weaknesses": "- Incomplete evaluation: The paper motivates the problem by arguing that continual pretraining and naive LoRA fine-tuning lead to catastrophic forgetting; however, their evaluation does not include LoRA fine-tuned models as a baseline, even though this is a directly relevant and stronger baseline than the model-merging baselines used. This omission weakens the empirical evidence for the claimed benefit of the proposed method.\n\n- The evaluation is limited to QA-style benchmarks, which constrains the generalizability of the findings. Additional tasks would help demonstrate broader applicability, while the Multi-FAct score eval should be expanded.\n\n- The work focuses primarily on mid- and high-resource languages. Nigatu et al. (2024) classify Swahili as a low-resource language; Swahili has recently become closer to a mid-resource language due to expanding corpora and tooling. Similarly, the other languages evaluated are not particularly low-resource, which limits the claim that the method addresses low-resource multilingual adaptation. See Nigatu et al. (2024) for some low-resource languages.\n\n$~$\n\n### **References**\n\nNigatu, Hellina Hailu, et al. *\"The Zeno's Paradox of Low-Resource 'Languages'.\"* arXiv preprint arXiv:2410.20817 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)