Walking the Tightrope: Autonomous Disentangling Beneficial and Detrimental Drifts in Non-Stationary Custom-Tuning
Authors: Xiaoyu Yang
Jie Lu
En Yu
Venue: NeurIPS 2025 poster ()
Keywords: Concept Drift
Reinforced Fine-tuning
MLLMs
Primary area: theory

Abstract: This paper uncovers a critical yet overlooked phenomenon in multi-modal large language models (MLLMs), especially for chest diagnosis: detrimental concept drift within chain-of-thought (CoT) reasoning during non-stationary reinforcement fine-tuning (RFT), where reasoning token distributions evolve unpredictably, thereby introducing significant biases in final predictions. To address this, we are pioneers in establishing the theoretical bridge between concept drift theory and RFT processes by formalizing CoT's autoregressive token streams as non-stationary distributions undergoing arbitrary temporal shifts. Leveraging this framework, we propose a novel autonomous counterfact-aware RFT that systematically decouples beneficial distribution adaptation from harmful concept drift through concept graph-empowered LLM experts generating counterfactual reasoning trajectories. Our solution, Counterfactual Preference Optimization (CPO), enables autonomous and stable RFT in non-stationary environments, particularly within the medical domain, through custom-tuning of counterfactual-aware preference alignment. Extensive experiments demonstrate our superior performance of robustness, generalization and coordination within RFT. Besides, we also contribute a large-scale dataset CXR-CounterFact (CCF), comprising 320,416 meticulously curated counterfactual reasoning trajectories derived from MIMIC-CXR. Our code and data are public at: https://github.com/XiaoyuYoung/CPO.

Walking the Tightrope: Autonomous Disentangling Beneficial and Detrimental Drifts in Non-Stationary Custom-Tuning
Xiaoyu Yang, Jie Lu, En Yu Australian Artificial Intelligence Institute (AAII),
Faulty of Engineering and Information Technology, University of Technology Sydney, Australia. xiaoyu. yang-yanngg-aaaa (egg-30studdet. uts edu. au; {jie. lu, en. yu- @utus...........u...uu......t... edu. au
Abstract
This paper uncovers a critical yet overlooked phenomenon in multi-modal large language models (MLLMs), especially for chest diagnosis: detrimental concept drift within chain-of-thought (CoT) reasoning during non-stationary reinforce- ment fine-tuning (RFT), where reasoning token distributions evolve unpredictably, thereby introducing significant biases in final predictions. To address this, we are pioneers establishing the theoretical bridge between concept drift theory and RFT processes by formalizing CoT's autoregressive token streams non-stationary distributions undergoing arbitrary temporal shifts. Leveraging this framework, we propose novel autonomous counterfact-awen RFT that systematically decouples beneficial distribution adaptation from harmful concept drift through concept graph- empowered LLM experts generating counterfactual reasoning trajectories. Our solution, Counterfactual Preference Optimization (CPO), enables autonomous and stable RFT in non-stationary environments, particularly within the medical domain, through custom-tuning of counterfactual-ave -awar preference alignment. Extensive experiments demonstrate our superior performance of robustness, generalization and coordination within RFT. Besides, we also contribute a large-scale dataset EKR-CounterFact (CCF), comprising 320,416 meticulously curated counterfactual reasoning trajectories derived from MIMIC-CRR Our code and data are public at: htttps://githei thu.. com http//iaayyYoooo/PPPPP
Introduction
Reinforcement Fine-Tuning (RFT) 2 has emerged as promising paradigm for domain-speciff -pecific customization of multi-modal large language models (MLLMs) K15). demonstrating remarkable capability in facilitating efficient domain shift with minimal data requirements, particularly for medical downstream tasks. However, the reinforcement-driven custom-tuning is fundamentally challenged by non-stationary environmental dynamics, especially for inherent domain-specinf data characteristics such as long-tailed distributions in medical diagnosis, and systemic data imperfections including noise and sparsity. This complex synergy induces latent concept drift that progressively disaligns the model's representation space from domain reality, culminating in catastrophic error propagation that particularly jeopardizes the reliability of MLLMs in safety- critical applications like radiology report generation.
Concept drift theory 6 7 provides perspective for analyzing the domain shift of RFT in non- stationary custom-tuning which focuses on the unpredictable distribution changes in data streams.
Correspondence Jie Lu and En
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

We posit that the autoregressive decoding paradigm inherent to MLLMs can be characterized as a sequential token stream generation process. Within this framework, each token generation step propagates through the model's internal reasoning pathways. which remain opaque to external observation, while manifesting inherent stochasticity in the evolving token probability distributions across successive decoding iterations.
Within the concept drift framework, our analysis reveals critical limitations in reinforcement fine- tuning approaches that depend on verifiable rewards in thain-of-thouge (CoT) 2
Observation 1.1. Specifically, while RFT operate through optimal reasoning pathway selection to maximize outcome certainty, we empirically observe that MLLM-gend -eeL---geerated CoT processes in specialized domains frequently demonstrate susceptibility to concept drift. This progressive devia- tion in intermediate reasoning ultimately induces substantial output divergence in non-stationary environments.
Intuitively, we provide representative case study that demonstrates this phenomenon in clinical reasoning con-
texts as presented in [ When diagnosing chest DR images, the model generates a reasoning trajectory con- taining the statement: "Asymmetric lung opacity in the right middle lobe is concerning for pneumonia." " We found that in the token-level probability, "lung opacity" shows negligible diffcrentiation from its ambiguous counterpart "opacity". Despite this minimal probabilistic disparity in the thinking process, our diagnostic outcome distribution analysis presents a radical divergence in final predictions as exhibited in Fig In particular, the diagnosis is com- pletely opposite in terms of atelectasis, cardiomegaly and pneumonia.
<thinko .... Asymmetri: [lung apantty opacity] right lobe is conceming for preumonia </thnio Diseases suffered by include:
0r
Therefore, summarizing the above challenges of reinforced fine-tuning in MLLMs, it raises the important question:
Molecasa
Cartionegiyy
Lung Qpacity
Prosavoni
Pheaa
How to adapt thinking process to concept drift under non-stationary reinforced custom-tuning?
Figure 1: Concept Drift in RFT's rea- soning for chest diagnosis. Despite analogous occurrence probabilities of "lung opacity" " (in red) and "opacity" (in blue) tokens during the CoT, non- stationarity induces significant bad dis- tributional drift in clinical conclusions, especially the opposite diagnosis of at- electasis, cardiomegaly and pneumonia.
Inspired by causal inference [8]10), we develop Counter- factual Preference Optimization (CPO), principled ap- proach that systematically perturbs reasoning trajectories to discriminate between beneficial distribution adaptation and detrimental concept drift.
Firstly, we construct a hicrarchical concept graph that codifies domain-specinn knowledge structures through triadic relation embeddings, including positive correlation, irrelevance, and opposition. Subsequently, we structurally embed the hierarchically structured concept graph into the LLM's reasoning architecture as an expert- -uiddd module, automatically generating semantically-constrai during reinforced custom-tuning, we formulate dual-preference optimization objective that jointly maximizes likelihood alignment with human preferences while minimizing similarity to adversarially generated counterfactual paths, thus achieving decoupling of beneficial domain adaptation from detrimental concept drift. Finally, we contribute CXR- CoR-CoouterFact (CCF), the chest diagnosis preference dataset comprising 320,416 fine-curated counterfactual reasoning trajectories derived from MIMIC-XXR [ radiologic findings, aiming to validate our method and catalyze research advancements in counterfactual-awer aware reinforcement fine-tuning paradigms
counterfactual inference paths. Consequently,
In summary, our paper mainly makes the following contributions:
1. First, we establish novel theoretical framework that formalizes autoregressive token gener- ation in MILMSs through the lens of concept drift theory, enabling systematic identification and causal analysis of detrimental reasoning divergence during non-stationary reinforced custom-tuning

---

2. Second, we propose Counterfactual Preference Optimization (CPO), which synergises structured -omaa--ppeeffic knowledge with systematic counterfactual intervention, driving reinforcement learning. By embedding learnable
the MILMs with preference- concept graphs as the expert and generating adversarially-constaanne reasoning trajectories, our approach achieves substantial decoupling between beneficial distribution adaptation and detrimental concept drift.
3. Third, we conduct comprehensive empirical validation across various clinical benchmarks for chest radiograph, including disease classification, diagnostic report generation and zero- shot generalization. The superior results demonstrate statistically significant improvements in robustness, generalization, and accuracy of our method under non-stationary custom- tningg Besides, we also provide ablation experiments to validate the effectiveness of various modules.
4. As a pioneer contribution to the community, we introduce scale dataset comprising 320,416 meticulously curated counterfactual reasoning trajectories derived from MIMIC-CXR.
(CCF), a large-
2 Methodology
suggesta preumanis and atslcianls
+
(Alributt))
(Diggnek)
Addtizeall the clear right normal inllams wideepread infection, ar meselels hamers.
dtructuree
Aralonical
Lowr Lowllungieeamn yelomes
Freumatianus
Prompts
Model
Thinking
Buridig contsphrenic
Marpholagy
Alreciais
patient may
Sependent Dpacit
have:
Desaity
Lung Spacity
opacity
Chest X-ray Image
Concept Drift
Diagnose atelactesis
--neumnnis
Pantiiona 7
Dynenes
Air brenchograms
(a) Concept Drift Behind the Thinking of MLLMs
(b) Concept Graph for Counterfactual Cause
specity eaapasta praumesia steactusls, Addibonaly candlac prrcctrres canditions tallars wideepreat lntectionn
Preference Thinking
Left lower lobe opacity and consolidatiion noted with asoociated effucion. Findings be due to pneumonia.
Prompts
Thinking
LLM Expert with Chest Knowledge Embedding
Model
may
Counterfact Reasoning
is noted in the left unerr tobe with disilacamen fissure towardl
like linear
lung
Chest X-ray Image
Diagnose
eftusion present. Findings may be due to stotactanis.
abolactasis
Counterfactual Preference Optimization
(c) Counterfactual Preference Optimization
Figure 2: The main contributions of our methods. (a) By formalizing autoregressive CoT generation stream of next-token prediction actions under the theoretical lens of concept drift, reveal that even minor perturbations in reinforced fine-tuning can induce unpredictable distributional changes of final predicted results. (b) To disentangle detrimental drift, introduce the concept graph that generates radiologically plausible counterfactual CoT's through controlled attribute perturbations. Green lines represent attributes that are positively correlated with the disease, while red denote they are exclusive. (c) We propose counterfactual preference optimization to drive the reinforced custom- tuning of MLLMs, enabling generalized CoT reasoning in non-stationary environments through disentanglement of beneficial domain adaptation from spurious concept drift, thereby achieving robust human-aligna decision-making via preference distillation.
2.1 Underlying Concept Drift Behind Thinking
In this section, we first extend the concept drift theory to reinforced custom-tuning, highlighting the phenomenon wherein the distributional characteristies of targets undergo arbitrary changes over the course of the thinking process. Operating through recursive on-policy sampling, the MLLM
3

---

autoregressively generates the token at position j in the reasoning chain, conditioned on both the visual input image v, the textual prompt and the partial token sequence tz of the CoT trajectory:
1 T T-1,,,,,
Therefore, we formally define concept drift behind the thinking as follows:
Definition 2.1. The MLLM's autoregressive reasoning trajectory manifests a thinking stream So = {s0,.., Si). where each cognitive state sj = (Lzj.2)) encapsulates all tokens generated so far ررا and its latent predicted distribution 2, of the results by tez. Therefore, in position i, S follows a certain distribution Fo,5(2,2), thus the concept drift behind the thinking can be formalized
3i : P.(t, 2) + Pt)
where the joint probability Pt 2) can be decomposed Pt(t,z) P((t) (((2)t).
Consequently, this concept drift framework behind the thinking of MLLMs enables simultaneous characterization of temporal dynamics Chain-of- Thoughtt reasoning, formalized as the concept drift process P. (t),, and its induced probabilistic divergence (z(zt,, capturing the evolving discrepancy between intended and actual outcome distributions throughout cogmitive progression.
To adapt the reinforced custom-tuning to concept drift behind the CO',, it is essential to adapt the model align with the evolving thinking distribution under non-stationary environment, which can be formally defined as:
I+T
L T 0
min
E
(3)
(t ((+1) (++T)
j=1
where 17 nt (1 10, (1) denotes the probability of CoT token sequence, ( represents the preferred CoT, symbolics the max length of tokens within the CoT, and ( signifies the MLLM at the cognitive status i. And the model is driven by the target metric L continuously to adapt the drift in a given time period [i, ( r]. Thus, we get the optimization object within the concept drift framework.
2.2 Disentangling Concept Drift with Counterfactual Causes
The optimization objective formalized Eq necessitates
disentanglement of two competing goals: advantageous policy-induced domain adaptation and versus pathological concept drift arising from suboptimal policy execution, which are both sampled from policy æ within the time pe- riod [i, i + +). However, it is challenging to determine the optimal preferred CoT and explicitly judge which strate- gies will cause unpredictable changes in tokens.
T
D
<
D
X
Fortunately, counterfactual causes provide an explicit man- ner to decouple these two competing goals. We construct structural causal graph g to formula the causal relation- ship among elements as discussed in Section 2. including inputs (X) consisting of image and prompt 7, prediction result (2), Shain-of-Though (T) and the concept drift in the reinforcement custom-tuning (D) as illustrated in Fig 3 where A B denotes that A is the causer of B. The causal graph of (X, Z,T, D} presents the tollowing causal connections:
Figure Structural Causal Graph. X: Inputs, Z: Prediction Results, T: Chain- of-Thought, and D: Latent Concept Drift within CoT under Non-stationary Rein- forced Custom -uniin.
(X, D) + T: This link denotes the chain-of-thouge T derived from the inputs X through policy π under the impact of latent concept drift D.
(X,T, D) + Z: This link presents that, apart from the regular reasoning pathway of T) + Z, the prediction is also impacted by the concept drift D through the pathway of D T Z. In the constructed structural causal model, nodes D and T are formally characterized as the con- founder and mediator [12], respectively. The confounding variable D induces bias through the

---

backdoor path T D - Z [8. simultaneously influencing both the mediator T and the outcome variable Z. This interference systematically distorts the estimation of CoT reasoning's 's causal effect on model predictions, particularly under non-stationary adaptation scenarios. The resulting spuri- ous correlations enter the mediation pathway ultimately compromising the stability of customized model tuning.
+
+ through the confounded mediator T,
Building upon the above analysis grounded in cause, we formally decouple the concept drift dynamics in Chain-of-Thoughe reasoning of Eq grounded in the cause. By constructing interventional distributions through do operations. P(Z do t) d). we quantify the latent causal effect:
V BI T
where the potential outcome T + t represents the counterfactual scenario when forcibly maintaining the mediator I' at value t, while preserving the confounder state d. This formulation explicitly isolates the causal effect of Chain-of-Thoughe T on prediction Z, from backdoor concept drift propagation T < D Z.
2.3 Embedding Counterfactual Causes with LLM expert
Having operationalized concept drift decoupling through controlled counterfactual interventions in Section 2.2 we identify the generation of autonomous counterfactual causes as the subsequent bottleneck that requires maintaining causal consistency within the chain-of-thon tfthh-of- while avoiding semantic entanglement.
we constructed hierarchical concept graph for custom-tuning through autonomous knowledge extraction from chest radiograph reports. It systematically organizes medical concepts into four semantic dimensions: disease entities, radiographic features, clinical relationships, and taxonomies. Specifically, the knowledge extraction pipeline leverages medical comain-adapter large language model with radiological prior knowledge, to process 160,208 chest X-ray reports from the MIMIC-CXR dataset [11]. Through iterative semantic parsing, the model autonomously identifies 12 distinct pulmonary disease entities accompanied by 53 clinically relevant attributes. These attributes are meticulously annotated across four diagnostic categories, namely mor- phological alterations, density anomalies, anatomical deviations, and functional/dynamie indicators. To capture clinical interdependencies, we formalize three types of ontological relationships, including association, irrelevance, and exclusion. For instance, the framework automatically detects pathophys- iological contradictions between emphysema emphyssema-ssociated hyperinflation and gellectssis-ellttee lung volume reduction, leading to the exclusion relationship between emphysema and atelectasis. As a toy example illustrated in Fig 2b the resulting concept graph provides nulti-relational representations where each disease entity is instantiated with its associated attributes and constraint relationships, enabling structured reasoning about pulmonary pathology.
Accordingly, inspired by
Through systematic integration of the concept graph, cestoall- LLM evolves as an expert with causal prior knowledge of chest radiology, which effectively simulates radiologists' differential diagnosis protocols, and synthesises counterfactual diagnostic narratives through controlled feature perturbation while maintaining radiological plassicclpppllulll plausibility, exhibited below:
2.4 Reinforced Custom- Custom-Tuning with Counterfactual Thinking
Obtaining counterfactual diagnosis in Section 2.3 we propose Counterfactual Preference Optimzation (CPO) to drive the reinforced custom-tuning of the multi-modal large language models.
Formally, have decomposed the CoT generation process into a stream of next token prediction actions So,; = {So.... s;). where each cognitive state sj = (t<r, encapsulates all tokens generated far Lej and its latent predicted distribution zj of the results by lej,, as exhibited in Definition 2.1 1. At timestep j, the action lú is sampled from the policy π (-||r, I, (ez))) where lj can be any token the vocabulary. After each action, the resulting state 8,+1 is the concatenation of the current state s, and the action to with its latent predicted results:
$ $J+1 (tz, o tj, Py(1t о tt)), SJSL
(5)
where denotes the concatenation between tokens stream Lej and action token 4,. L represents the maximum length of CoT, and P is the latent predicted distribution of results derived by lej as presented in Eq As the start of the chain-of of-nn-tf-thought, ag is usually the token <think>. While it
5

---

Ground Truth
Findings:
Moderate cardiomegaly is increased. No focal consolidation or pncumothorax, There is a slight blunting of the costophrenic angles, which may indicate small pleural effusion or scarring. There is increased density at the perihilar regions which may indicate pulmonary vascular congestion.
Diagnosis:
The disease of this patient is Cardiomegaly.
Generated Counterfactual Reasoning of Pneumonia.
Findings:
PA and lateral views of the chest show moderate cardiomegaly. Focal consolidation is noted in the right lower lobe with accompanying bronchial airspace opacification. No pneumothorax is observed. Slight blunting the costophrenic angles suggests the presence of small pleural effusion. Increased density at the perihilar regions indicates pulmonary vascular congestion, but also suggests possible pncumonia.
Diagnosis:
The discase of this patient Pneumonia.
Table I: Example to illustrate the generated counterfactual diagnosis. Ground Truth denotes the original diagnosis report from MIMIC-CXR, and the bottom is the counterfactual report designed for pneumonia. Underline indicates generated counterfactual diagnostic attributes through controlled perturbation while maintaining radiological plausibility, leading to the counterfactual diagnosis.
produces the </think> token, the resulting state 81+1 the terminal state, thereby concluding one thain-of-thoughe generation process. We regard the report issued by the professional doctors as the saain-of-thougee preferred by humans, which are positive samples. As for negative samples, generate counterfactual CoT for diagnostic report instances our negative samples, that is, perturbing specific radiological features to interfere with the diagnosis results according to our proposed concept graph. Thereby, the chain-of-thouge preferred by humans is considered to be ++. namely the diagnosis report stemming from the radiologist, while the generated counterfactual CoT is represented by t
Consequently, following the DPO 15 we derive the optimal policy that maximizes the reward function through:
(t I) x (ter(tlu, () exp r(u,4,t)
(6)
B
where 3 a parameter controlling the deviation from the base reference policy Tet namely the initial supervised fine-tuned (SFT) model, and To denotes the fine-tuning model. With counterfactual effect in Eq the reward difference between human-preferred positive samples and counterfactual samples can be defined as:
To [o((+lu,1) log To To v. (tar(t [,
= log
(7)
( ((++),11)))))
Thus, based on the Bradley- Terry model, the counterfactual preference optimization (CPO) is driving the reinforced custom-tunie of the MLLMs through the maximum likelihood objective:
Lero(To; mer) -- -E,,,,, loo
BBog
(1,,)) The (t+,t)
Blog To (o(((1,))) Tee (w, }
In this context, it culminates in counterfactual reinforced custom-tunine an adaptive framework that effectively differentiates between ad advantageous domain adaptation and harmful concept drift in non-stationary environments, achieving equilibrium preservation through causal intervention and dynamic policy reinforcement, walking the tightrope.
2.5 Building CXR-CounterFact Dataset for Clinical Reasoning Chains
Since we are pioneers in introducing counterfactual cause into reinforced custom-tuning of MLLMs, we are deeply aware of the scarcity counterfactual CoT downstream tasks, especially in the highly professional medical field. Thus, aspiration is for the model to adeptly acclimate to the

---

concept drift by itself, acquiring abundant knowledge with more and more data, but not exhibiting bias.
In this context, a more realistic training dataset for multi-modal large language models is required to validate their potential to be trained under the non-stationary reinforced custom-tuning. Recognizing the demand for higher-quality multi-modal data with COT, we develop datasets called CXR- CounterFact Dataset (CCF), extending the MIMIC-CXR(II with counterfactual thain-of-thoughi This novel dataset introduces 320,416 meticulously curated counterfactual pairs spanning 14 thoracic pathologies, establishing a pioneering large-scat scale benchmark for causal interpretation in clinical chest X-ray analysis. More details are given in Appendix B
3 Experiments
In this section, we verify the robustness, generalization and coordination of our proposed counterfactual preference optimization in reinforced custom-tuning in non-stationary environments.
Poalivv Uncertain
MIMIC-CXR[I] is utilized to train the MLLMs via re- inforced destom-tuning for domain adaptation, which presents 371,920 chest X-rays associated with 227,943 imaging studies from 65,079 patients. And images are
20
10
provided with 14 labels with corresponding free-text ra- diology reports, namely Atelectasis (Atc.), Cardiomegaly (Car.), Consolidation (Con.), Edema (Ede.), Enlarged Car- diomediastinum (ECM), Fracture (Fra.), Lung Lesion (LL), Lung Opacity (LO), Pleural Effusion (PE), Pneu- monia (Pna.), Pneumothorax (Pnx.), Pleural Other (PO), Support Devices (SD) and No Finding (NF).
50
PE
10
AM
Car
Edc
MP
Con
Pha
LL BOM Foo
Figure 4: Non-stationarity of MIMIC-
CXR with its percentage of diseases. Blue signifies patients with clinically confirmed diagnoses showing the long- tailed characteristic, while red demar- cates suspected cases emphasizing the inherent uncertainty within medicine.
We selected the MIMIC-CXR I1] dataset not only for its well-established benchmark enabling rigorous perfor- mance evaluation but also due to its authentic clinical representation that ex- hibits inherent non-stationnriit, particularly long-tail and diagnostic ambiguity. As illustrated in Fig the statistical profiling of 14 thoracic pathologies in MIMIC-CXI MXRI reveals dual clinical characteristics: the number of confirmed diseases showed a clear long-tail distribution, and each disease had a large number of uncertain patients. Beyond that, we found that 40.87% of the patients suffered from two or more diseases, with nearly 19.97% experiencing three or more. They all reflect the non-stationary environments of our experimental setup within MIMIC-CRR as the training dataset for reinforced custom-tuning.
real-world downstream medical tasks,
In terms of the model. we employ Qwen2.5-Ve VL (7B) [I6] to perform supervised fine-tuning (SFT) and reinforced fine-tuning (RFT), cascadedly. And they only train one epoch with batch size of 2.
More detailed experimental implementations are given in Appendix C
3.1 Taming the Non-stationary Custom-Tuning
First, to explicitly demonstrate the superior performance of our proposed method in non-stationary environments, especially in robustness, we compare it with other models on MS-CXR-T [17), where instances are chosen from the public MIMIC-XXR. As exhibited in Table our counterfactual preference optimization approach achieve the superior overall performance of 81.8% surpassing the second CoCa- CXR [18] nearly 12.6%. It demonstrates the robustness of our approach to reinforced fine-tuning in non-stationary environments. While our approach trails TempA- -LP [19 by 1.8% on edema (Ede.) detection, we argue that this performance gap emerges from the utilization of additional annotations from Chest ImaGenome 220 in addition to standard MIMIC-CXR. In terms of the pneumothorax (Pnx.), SFT has achieved a high result of 95.99 05.9%, so the slight decrease in CPO does not affect the overall performance of the model.
7

---

Notably, supervised fine-tune tine-tuning (SFT) exhibits suboptimal performance on the clinically correlated diseases, namely consolidation (Con.) and pneumonia (Pna.). as presented in Table Section It empirically validates our Observation 1.I that the inherent concept drift in disease attribute representation within chain-of-thought reasoning introduces systematic prediction biases.
Beyond that, the substantial performance gains of 22.8% and 17.4% in CPO for consolidation (Con.) and pneumonia (Pna.), respectively, underscore our core contribution, which disentangles the concept drift of CoT in reinforcement learning under non-stationary custom-tuning, achieving robust reasoning.
In terms of D50[15]. while proposed CPO and DPO share preference-style op- timization framework, CPO fundamentally distinct. DPO contrasts human-preferred with dispreferred responses, while CPO contrasts fac- tuals with counterfactuals generated under explicit causal interventions, specifically designed to isolate the causal effect. Crucially, as shown Table our proposed CPO is significantly superior to DPO with the same number training data, proving they not functionally equivalent. Besides, We also tried to
Venue
Con.
PE
Pna.
Pnx.
Ede.
Avg.
CTrans 21
CVPR'23
44.0
61.3
45.1
31.5
65.5
49.5
CheXRe
22
MICCAI'22
47.0
47.0
47.0
36.0
49.0
45.2
BioViL. 23
ECCV*22
56.0
63.0
60.2
42.5
67.5
57.8
BioViL- [21
CVPR'23
61.1
67.0
61.9
42.6
68.5
60.2
Med-ST 24
ICML'24
60.6
67.4
58.5
65.0
54.2
61.1
TempA-VLP 19
WACV'25
65.2
59.4
73.4
43.1
77.1
63.6
CoCa-CXR [8]
Arxiv'25
70.4
69.6
61.4
72.8
71.8
69.2
SFT DPO CPO
54.9
71.7
70.0
95.9
76.5
73.8
This paper
63.2
72.4
76.7
93.5
76.3
76.4
77.7
72.7
87.4
95.8
75.3
81.8
Table 2: Evaluation results of multi-label chest diseases classification on MS-CXR-T. Top-1 accuracy is applied evaluate the performance of different methods. The best-performing models are highlighted in red, with the second-best blue. SFT denotes the results of supervised fine-tuning, and DPO indicates the direct preference optimization with random negative samples, while the CPO represents our counterfactual preference optimization method.
use GRPO (25) to train Qwen2.5-vV and DeepSeek- VL.2 on MIMIC-CRR, but both encountered reward collapse that led to failed training. Inspired by DAPO paper (66), we attribute this to GRPO's reliance on sparse final-answer rewards, where suboptimal reward assignment obscures high-quality samples. In contrast, CPO provides denser causal trajectories, significantly improving generalization and robustness by operating controlled counterfactual interventions.
3.2 Concept Drift-Aware CoT for Accurate Reasoning
Venue
BLEU-1 BLEU-2
BLEU-3 BLEU-4 ROUGE-L METEOR CIDEr
R2Gen 27
EMNLP'20
0.353
0.218
0.145
0.103
0.277
0.142
PPKED 28
CVPR'22 21
0.360
0.224
0.149
0.106
0.284
0.149
0.237
AlignTrane
29
MICCAI'2
0.378
0.235
0.156
0.112
0.283
0.158
CMCL 330
ACL'21
0.344
0.217
0.14
0.097
0.281
0.133
Clinical-BERT 31
AAAI'22
0.383
0.230
0.151
0.106
0.275
0.144
0.151
METransformer
32
CVPR '23
0.386
0.250
0.169
0.124
0.291
0.152
0.362
DCL 33
CVPR '3
0.109
0.284
0.150
0.281
R2GenGPT 34
MetaRad'23
0.408
0.256
0.174
0.125
0.285
0.167
0.244
PromptMRG
35
AAAI'24
0.398
0.112
0.268
0.157
BtspLLM 36
AAAI24
0.402
0.262
0.18
0.128
0.291
0.175
MambaXray 33
Arxiv'24
0,422
0.268
0.184
0.133
0.289
0.167
0.241
CPO
This paper
0.426
0.288
0.186
0.155
0.321
0.236
0.375
Table 3: Evaluation results of diagnostic report generation on MIMIC-XXI with various metrics including BL.EU-1/-//3/-4 ROUGE-I, METHOR and CIDEr. The best-performing models highlighted in red, with the second-best blue.

---

Beyond the classification, we verify our main contribution of accurate reasoning, preserving the beneficial CoT within domain adaptation, while eliminating harmful concept drift. exhibited in the Table the experiments of diagnostic report generation on MIMIC-CXR are conducted to assess the performance of the thinking our proposed model with shain-of-thoughle Our evaluation combines multiple metrics: BLEU evaluates terminology accuracy with higher-order scores indicating logical coherence in clinical reasoning, ROUGE-L assesses completeness through narrative alignment, METEOR enables synonym-aware lexical matching, and CIDEr prioritizes clinical details via corpus- informed weighting.
The experimental findings demonstrate that our reasoning framework achieves prominent perfor- mance across all evaluation metrics, with particularly notable improvements in BLEU-4 (16.5% improvement), ROUGE-L (10.3% increase) and METEOR (34.8% enhancement) scores, indicating the coherence, completeness, and professionalism of our model's thinking. We attribute it to the enhanced fidelity of our reasoning chains in combating concept drift during non-stationary reinforce- ment learning processes. These enhancements reveal that our method's superior accuracy stems from its capacity to maintain coherent reasoning pathways even when faced with dynamically shifting environmental parameters of complex RL scenarios.
3.3
Generalized Reinforced Custom-tuning
Method
Open-I
PadChest
PadChest20
ChestXray14 14
ChestXpert
ChestXDet10
MedCLIP
38
55.1
50.8
50.1
56.4
74.4
57.1
BiomedCI
39
57.7
51.3
51.0
63.9
67.7
63.0
GLoRIA
40
58.9
56.5
55.8
61.0
75.0
64.5
BioViL 21
70.2
65.5
60.8
72.9
78.9
70.8
CheXzero 41
75.9
62.9
68.8
72.6
87.9
71.3
MedKLIP 42
75.9
62.9
68.8
72.6
87.9
71.3
KAD 43
80.7
75.0
73.5
78.9
90.5
73.5
CARZero
83.8
81.0
83.7
81.1
92.3
79.6
CPO
84.4
82.0
85.1
81.7
92.5
80.1
Table 4: Evaluation results of zero-shot diseases classification Open- [45),
PhdChest|46),
PadChest20 [46), ChestXray14 14 [47], ChestXpert [48] and ChestXDet10 [4).. AUC is applied to
evaluate the performance of different methods. The best-performing models are highlighted in red, with the second-best in blue.
Furthermore, we validated the generalization of our model on downstream tasks with zero-shot multi-label classification across six different benchmarks, as presented in Table 4 Experimental results demonstrate that our CPO-driven MLLMs achieve zero-shot superiority over the second-best baseline CARZero (44) across all benchmark datasets, underscoring our remarkable robustness and generalization capabilities even when trained under non-stationary environmental regimes.
3.4 Ablation Study on Inherent Compatibility of CPO and CoT: Two Peas in a Pod
Moreover, conduct ablation exper- iments on MIMIC-XXR to validate the feasibility and coordination of the
RFT CoT
SFT
Con.
PE
Pna.
Pnx.
Ede.
Avg.
CPO
chain-of-thought (CPO) and counter-
factual preference optimization (CPO) within reinforced fine-tuning (RFT) in non-stationary environments, as presented in Table 5 Among them, only CoT without CPO in RFT rep- resents the utilization of direct prefer- ence optimization (DPO) [15] to drive MI.LMs for reinforcement learning. While, the only CPO RFT denotes the reinforced fine-tuning only applies
54.9
71.7
70.0
95.9
76.5
73.8
58.4
71.2
75.0
94.4
75.5
74.9
/
70.5
72.7
77.3
95.2
75.8
78.3
/
77.7
72.7
87.4
95.8
75.3
81.8
Table 5: Ablation evaluation results on chain-of-thought (CoT) and counterfactual preference (CPO) within re- inforced fine-tuning (RFT) on MIMIC-CXR, where all RFT stages follow the supervised fine-tuning (SFT). The / denotes that the results are trained with the corresponding module. The results based the test split of the MS- CXR-T, with Top-1 accuracy (Acc) as the metric.

---

the diagnosis results without the think- ing process during the training.
The experimental analysis reveals CPO's superior performance gain in reinforcement learning (4.5% CoT's 1.1%). We argue that it is mainly attributable to its mechanism of introducing causally attributed negative samples that enable decision boundary refinement feature space, whereas CoT primarily operates through stepwise cognitive scaffolding via enhanced positive samples.Therefore the inherent synergistic compatibility between CoT and CPO emerges through their complementary roles in reinforcement learning frameworks, with CoT generating reinforcement-aligee aligned positive exemplars and CPO pro- viding causality- negative specimens, jointly orchestrating MLLM training optimization as empirically validated through comprehensive benchmarking results achieving state-of state-of-the-art performance.
Conclusion and Outlooks
In this paper, we present counterfactual preference optimization (CPO), novel, robust and gen- eralized reinforced custom-tunie tunning paradigm tailored for non-stationary environments. We employ concept drift theory methodically formalize the bias within the autoregressive token generation of MLLMs and put forward causal counterfactual thinking to mitigate these detrimental drifts and keep good domain adaptation. By virtue of this framework, CPO is devised to counteract the unpredictable distribution changes occurring within non-stationary environments.
We hope that work will inspire future advancements in counterfactual cause of reinforced learning paradigm, specifically addressing biases originating from real-world data challenges. In future research, we will focus on the efficiency of counterfactual causes in reinforced fine-tuning, and broader applications.
Acknowledgment
The work was supported by the Australian Research Council (ARC) under Laureate project FL190100149.
References
[1] Trung, L., X. Zhang, Z. Jie, al. ReFT: Reasoning with Reinforced Fine-Tuning In L.-W. Ku, A. Martins, V. Srikumar, eds., Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 7601-7614. Association for Computational Linguistics, Bangkok, Thailand, 2024.
[2] Liu, Z., Z. Sun, Zang, et al. Visual-RFT: Visual Reinforcement Fine-Tuning, 2025.
[3] Alayrac, J.-B., J. Donahue, P. Luc, et al. Flamingo: A Visual Language Model for Few-Shot Learning. Advances Neural Information Processing Systems, 35:23716-22 -23766, 2022.
[4] Bai, J., S. Bai, S. Yang, et al. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023.
[5] Dai, W., J. Li, D. Li, et al. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning, 2023.
[6] Lu, J., A. Liu, E. Dong, et al. Learning under Concept Drift: Review. 31(12):2346-366; 2019.
[7] Yang, X., J. Lu, E. Yu. Adapting multi-modal large language model to concept drift from pre-training onwards. arXiv preprint 13XWv24405.3345, 2025.
[8] Pearl, Causal diagrams for empirical research. Biometrika, 92(4):669-688, 1995.
[9] Causal inference in statistics: primer. John Wiley & Sons, 2016.
10

---

[10] Yang, X., J. Lu, E. Yu. Causal-informed contrastive learning: Towards bias-resilient pre- pre-training under concept drift. arXiv preprint arXiv:2502.07620, 2025.
[11] Johnson, A. E., J. Pollard, S. J. Berkowitz, et al. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317, 2019.
[12] Pearl, J. Direct and indirect effects. In Probabilistic and causal inference: the works of Judea Pearl, pages 373-392 2022.
[13] Zhang, X., C. Wu, Y. Zhang, et al. Knowledge-enhanced visual-language pre-training on chest radiology images. 14(1):4542, 2023.
[14] Zhou, X., X. Zhang. C. Wu, et al. Knowledge-enhanne Visual-Language Pretraining for Computational Pathology, 2024.
[15] Rafailov, R., A. Sharma, E. Mitchell, et al. Direct Preference Optimization: Your Language Model is Secretly Reward Model. 36:53728-53741, 2023.
[16] Team, Q. Qwen2.5-vl, 2025.
[17] Bannur, S., S. Hyland, F. Liu, et al. Learning to exploit temporal structure for biomedical lision-language processing. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2023.
[18] Chen, Y. S. Xu, A. Sellergren, et al. Coca-cxr: Contrastive captioners learn strong temporal structures for chest x-ray vision-language understanding, 2025.
[19] Yang, Z., L. Shen. Tempa-vip: Temporal-awarr vision- jision-language pretraining for longitudinal exploration in chest x-ray image. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 4625 4634. 2025.
[20] Wu, J. T., N. N. Agu, I. Lourentzou, et al. Chest imagenome dataset for clinical reasoning. arXiv preprint arXiv:2108 10000:100000016666,. 2021.
[21] Bannur, S., S. Hyland, Q. Liu, al. Learning exploit temporal structure for biomedical vision-language processing. In Proceedings of the IREE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15016-15027. 2023.
[22] Karwande, G., A. B. Mbakwe, J. T. Wu, et al. Chexrelnet: An seatomy-awur model for tracking longitudinal relationships between chest x-rays. In International Conference Medical Image Computing and Computer- cassuter-Assisted Intervention, pages 581-591. Springer, 2022.
[23] Boecking, B., N. Usuyama, S. Bannur, et al. Making the most of text semantics to improve biomedical vision-language processing. In European conference on computer vision, pages 1-21. Springer, 2022.
[24] Yang. J., B. Su. Zhao, al. Unlocking the power of spatial and temporal information in medical multimodal pre-training. In Forty-first International Conference on Machine I Learning. 2024.
[25] Shao, Z., P. Wang, Q. Zhu, et al. Deepseekmath: Pushing the limits of mathematical reasoning open language models. arXiv preprint arXiv:2402.03300, 2024.
[26] Yu, Q., 7. Zhang, R. Zhu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
[27] Chen, Z., Y. Song, T.-H. Chang, et al. Generating radiology reports via memory-driven trans- former. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1439-1444 1449. 2020.
[28] Liu, F., X. Wu, S. Ge, et al. Exploring and distilling posterior and prior knowledge for radiology report generation. In Proceedings of the CerFeeeeccofeeeencce conference on computer vision and pattern recognition, pages 13753-1323 13762. 2021.

---

[29] You, D., F. Liu, S. Ge, et al. Aligntransformer: Hierarchical alignment of visual regions and dis- ease tags for medical report generation. In Medical Image Computing and Computer Assisted IMCeVVentionMMICCAI 2021: 24th International Conference, Strasbourg, France, September 27-October 1, 2021, Proceedings, Part 24, pages 72-82. Springer, 2021.
[30] Liu, F,, S. Ge, X. Wu. Competence-based multimodal curriculum learning for medical report generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3001-3012. 2021.
[31] Yan, B., M. Pei. Clinical-bert: Vision- laiinnla pre-training for radiograph diagnosis and reports generation. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pages 2982-2990. 2022.
[32] Wang, Z., L. Liu, L. Wang, et al. Metransformer: Radiology report generation by trans- former with multiple learnable expert tokens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11558-11567. 2023.
[33] Li, M., B. Lin, Z. Chen, et al. Dynamic graph enhanced contrastive learning for chest x- ray report generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 333433343. 2023.
[34] Wang, Z., L. Liu, L. Wang, et al. R2gengpt: Radiology report generation with frozen Ilms. Meta- MadioRadio 1(3):100033, 2023.
[35] Jin, H., H. Che, Y. Lin, et al. Promptmrg: Diagnosis-driven prompts for medical report generation. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pages 2607-2615. 2024.
[36] Liu, C., Y. Tian, W. Chen, et al. Bootstrapping large language models for radiology report generation. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pages 18635-18643, 2024.
[37] Wang, X., F. Wang, Y. Li, et al. CXPMRG-Bench. Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Dataset, 2024.
[38] Wang, Z., Z. Wu, D. Agarwal, et al. Medelip: Contrastive learning from unpaired medical images and text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, vol. 2022, page 3876. 2022.
[39] Zhang, S., Y. Xu, N. Usuyama, al. Biomedclip: multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint #rXiv:2303.00915 2023.
[40] Huang, S.-C., L. Shen, M. Lungren, et al. Gloria: multimodal global-local representa- tion learning framework for label-efficient medical image recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3942-3951. 2021.
[41] Tiu, E., E. Talius, P. Patel, et al. Expert-level detection of pathologies from unannotated chest x-ray images via selevsupe learning. Nature biomedical engineering, 6(12):1399-1406, 2022.
[42] Wu, C., X. Zhang, Y. Zhang, et al. Medklip: Medical knowledge enhanced language-image pre-training for x-ray diagnosis. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21372-21383, 2023.
[43] Zhang, X., C. Wu, Y. Zhang, et al. Knowledge-enhanced visual-language pre-training chest. radiology images. Nature Communications, 14(1):4542, 2023.
[44] Lai, H., Q. Yao, Z. Jiang, et al. Carzero: Cross-attention alignment for radiology zero-shot classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11137-11146. 2024.
12

---

[45] Funnerr-uushmn, D., S. Antani, M. Simpson, et al. Design and development of a multimodal biomedical information retrieval system. Journal of Computing Science and Engineering. (12):168-177, 2012.
[46] Bustos, A., A. Pertusa, J.-M. Salinas, et al. Padchest: A large chest x-ray image dataset with multi-labe annotated reports. Medical image analysis. 66:101797, 2020.
[47] Wang, X., Y. Peng, L. Lu, et al. Chestx-ray8: Hospital-scale chest x-ray database and bench- marks on Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017.
classification and localization of common thorax diseases. In
[48] Irvin, J., P. Rajpurkar, M. Ko, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, vol. 33, pages 590-597, 2019.
[49] Liu, J., J. Lian, Y. Yu. Chestx-det1 O: Chest x-ray dataset on detection of thoracic abnormalities, 2020.
[50] Lu, J., A. Liu, Song, et al. Data-driven decision support under concept drift in streamed big data. Complex & intelligent systems, 6(1):157-163, 2020.
[51] Wang. K., L. Xiong, A. Liu, et al. A self-adaptive ensemble for user interest drift learning. 577:127308, 2024.
[52] Jiao, B., Guo, D. Gong, et Dynamic Ensemble Selection for Imbalanced Data Streams With Concept Drift,
2024.
[53] Cerqueira, V. H. M. Gomes, A. Bifet, et al. STUDD: A suudent-teacher method for unsupervised concept drift detection. (12(11):4351-44)
2023.
[54] Yang. X., Chen, X. Yue, al. T-distributed Spherical Feature Representation for Imbalanced Classification. Proceedings of the AAAI Conference on Artificial Intelligence, 37(9):10825- 10833, 2023.
[55] Yu, E., J. Lu, B. Zhang, et al. Online boosting adaptive learning under concept drift for multistream classification. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pages 16522-16530. 2024.
[56] Yu, E., Song, G. Zhang, et al. Learn-to-adap:: Concept drift adaptation for hybrid multiple streams. 496: 121-130, 2022.
[57] Yu, H., W. Liu, J. Lu, et al. Detecting group concept drift from multiple data streams. 334:109113, 2023.
[58] Liu, W., X. Yue, Y. Chen, et al. Trusted multi-view deep learning with opinion aggregation. In Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pages 7585-7593. 2022.
[59] Liu, W. Y. Chen, X. Yue. Enhancing testing-tima robustness for trusted multi-view classification in the wild. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 15508-15517. 2025.
[60] - Building trust decision with conformalized multi-view deep classification. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 7278-7287 2024.
[61] Li, W., X. Yang, W. Liu, et al. DDG-DA: Data Distribution Generation for Predictable Concept Drift Adaptation. 36(4):4092-410) 2022-06-28.
[62] Yang, X., H. Zhang, J. Cai. Deconfounded Image Captioning: A Causal Retrospect. 45(11):12966-13010 2023.
[63] Liu, B., D. Wang, X. Yang, et al. Show, Deconfound and Tell: Image Captioning With Causal Inference. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 18041-18050. 2022.
13

---

[64] Zhang, C., L. Zhang, D. Zhou. Causal Walk: Debiasing Multi-Hop Fact Verification with Front-Door Adjustment, 2024.
[65] Choi, S., M. Jeong, H. Han, et al. C2L: Causally Contrastive Learning for Robust Text Classification. 36(10):10526-1054 2022.
[66] Rohekar, R. Y., Y. Gurwicz, S. Nisimov. Causal Interpretation of Self-Attention in Pre- Pre-Trained Transformers. 36:31450 1450-31466, 2023.
[67] Yang, X., H. Zhang, G. Qi, et al. Causal Attention for Vision-Languuage Tasks. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9847-9857. 2021.
[68] Yang. X., L. Xu, H. Li, et al. One leaf reveals the season: Occlusion-based contrastive learning with semantic- aware views for efficient visual representation. In Forty-second International Conference on Machine Learning, 2025.
[69] Christiano, P. F., Leike, T. Brown, et al. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[70] Ouyang, L., J. Wu, X. Jiang, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[71] Jaech, A., A. Kalai, A. Lerer, al. Openai ol system card. arXiv preprint arXiv:2412.16720. 2024.
[72] Bai, Y., S. Kadavath, S. Kundu, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022.
[73] Guo, D., D. Yang, H. Zhang. et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint urXiv:2501.12944 2025.
[74] Gulcehre, C., T. L. Paine, S. Srinivasan, et al. Reinforced self-training (rest) for language modeling, 2023.
[75] Yuan, Z., H. Yuan, C. Li, et al. Scaling relationship on learning mathematical reasoning with large language models, 2024.
[76] Zeng. W., Y. Huang, L. Zhao, et al. B-STar: Monitoring and balancing exploration and exploitation in self-taught reasoners. In The Thirteenth International Conference on Learning Representations, 2025.
[77] Zhang, Z. C. Zheng, Y. Wu, al. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301. 2025.
[78] Yang, X., L. Xu, H. Sun, et al. Enhancing visual grounding and generalization: A multi-task cycle training approach for vision-language models. arXiv preprint arXiv:2311.1232) 2024.
[79] Yang, X., J. Lu, Yu. Learning from all: Concept alignment for autonomous distillation from multiple drifting milms. arXiv preprint 2xXx:2510000444: 2025.
[80] Li, Z.-Z., D. Zhang, M.-L. Zhang, et al. From system I to system 2: A survey of reasoning large language models. arXiv preprint arXiv:2502.17419, 2025.
[81] Chen, Q., L. Qin, J. Liu, et al. Towards reasoning era: A survey of long chain-of-thought for reasoning large language models. arXiv preprint arXiv:2503.09567, 2025.
[82] Zhang, Z., A. Zhang, M. Li. al. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.009923 2023.
[83] Chen, Q. L. Qin, J. Zhang, et al. m3 cot: A novel benchmark for multi-domain multi-step multi-modal chain-of- thhthught.. arXiv preprint arXiv:2405. 16473, 2024.
[84] Wang, Y., S. Wu, Y. Zhang, et al. Multimodal chain-of-thought reasoning: comprehensive survey, arXiv preprint 122. 2025.
14

---

[85] Zheng, G., B. Yang, J. Tang, et al. Ddcot: Duty-distine chain-of-thougie thouugt prompting for mul- timodal reasoning in language models. Advances in Neural Information Processing Systems. 36:5168 1685119, 2023.
[86] Liu, w. Y. Chen, X. Yue, et al. Enhancing reliability in medical image classification of imperfect views. IEEE Transactions on Circuits and Systems for Video Technology, 2025.
15

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
16

---

Justification:
We discuss limitations of our approach and the outlook of further work in the section of conclusions.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-ten system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We provide the full of assumptions and a complete proof in the main manuscript and the supplemental.
Guidelines:
The answer NA means that the paper does not include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide full information about the replication experiments in the main manuscript and make code and data publicly available.
17

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
cases, in which case
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We made our code and data publicly available on github as anonymous. The anonymous link is htttt::///tnnnnee 4oppeh.ssieenccc///PPP-11111
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ hhiiss.cc//ublic//uuu
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
18

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We present the details of the experiments in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report results of experiments with statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the information about computer resources in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
19

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We made code and data publicly anonymous in the review stage.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts of our work in the section of conclusions.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We have such risks.
20

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, thcode thcode. com datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
We have talked about the details of the dataset and code as part of our submissions.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
21

---

Answer:
[NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non stanndaddrrd components.
Please refer to our LLM policy for what should or should not be described.
wwtpuurrisccc///oaoaa cc/Conffreennae cc//oonferencce//222///1111
22

---

Appendix
A
Related Works
A.1 Concept Drift
In their survey spanning multiple studies, Lu et al. 6 50 establish a comprehensive taxonomy of concept drift mitigation strategies, categorizing prevailing approaches into three principal paradigms:
error rate-driven adaptations [ST [2]. data distribution-awe -ware methodologies 7 33 [4]] and multi-
hypothesis frameworks [55 56 Our work aligns with the distribution-centrie paradigm, which distinguishes itself through its dual capacity for both precise drift detection via rigorous statistical analysis and holistic drift characterization across temporal, spatial, and quantitative dimensions. Specifically, these distribution-drive occurrence but also facilitate granular diagnostics through temporal localization of drift emergence, feature subspace attribution, and severity quantification capabilities that render them particularly advantageous for developing interpretable adaptive systems requiring both drift awareness and targeted model recalibration.
techniques enable not merely the identification of concept drift
Recent vannee in concept drift adaptation have yielded sophisticated methodologies across diverse learning scenarios. The Online Boosting Adaptive Learning (OBAL) framework [55] has emerged as a dual-phase solution for multistream classification challenges, initially employing Adaptive Covariate Shift Adaptation (AdaCOSA) to model dynamic inter-stream correlations before transitioning to Gaussian Mixture Model-based weighting for asynchronous drift mitigation. Complementing this, CDMLLM 7 reveals critical vulnerabilities in vision-language models through systematic analysis of concept drift-induces biases across pre-training and fine-tuning stages, proposing a unified framework that synergizes T-distribution adaptation for long-tailed calibration with explicit -iittiiiiitiitooonn detection to enhance multimodal alignment robustness. Expanding the scope beyond individual data streams, GDDM (57) introduces a distribution free statistical framework for detecting subtle group- level concept drifts in multi-stream environments through adaptive hypothesis testing mechanisms. Additionally, the studies by (58(60) introduce a multi-perspecne
uncertainty framework designed to handle concept drift in diverse data streams. This approach utilizes set-based prediction methods to unify probabilistic results into clear categorical formats. In parallel, DDG-DA [61 pioneers anticipatory concept drift adaptation by modeling environmental evolution through predictive factor analysis and synthetic data generation, effectively bridging current observations with projected distribution shifts. Advancing unsupervised detection paradigms, STUDD (53) establishes teacher- student discrepancy framework that leverages predictive consistency analysis to enable label-agnostic drift identification while maintaining detection sensitivity, thereby addressing practical deployment constraints in evolving data environments.
A.2 Causal Inference
Recently, increasing researchers have incorporated causal inference into deep-learning models, espe- cially in large models. Deconfounded Image Captioning (DIC) [2] is proposed to address dataset bias in vision-language models through causal lens, that integrates backdoor and front-door adjust- ments for systematic bias mitigation. The framework provides principled causal analysis of spurious correlations in multimodal alignment, offering theoretical grounding for decomposing bias sources through structured interventions. Likewise, aiming for spurious correlations induced by visual and linguistic biases during training, CIIC (63) is proposed causal intervention framework combining an Interventional Object Detector (IOD) and Interventional Transformer Decoder (ITD) guided by structural causal models. By applying backdoor adjustment through IOD's feature disentanglement and ITD's 's dual de-confounding mechanism, their approach systematically mitigates confounding effects across encoding and decoding stages, demonstrating enhanced generalization through causal correlation modeling. Similarly, targeting multi-hop fact verification bias in the large language model, Causal Walk [64] is proposed, a front-door adjustment framework that disentangles complex spurious correlations in evidence chains. The method models reasoning paths as mediators in structural causal models, decomposing causal effects via random walk-based treatment-mediator estimation and geometric mean-based mediator-outcom approximation. By integrating adversarial and symmetric datasets synthesized with large language models, the approach demonstrates superior debiasing performance.
23

---

Recent advances in causal representation learning have produced innovative methodologies to ad- dress confounding biases in large models. The C2L framework [65] tackles model fragility through contrastive counterfactual synthesis, introducing a collective decision mechanism that aggregates predictions across probabilistically generated counterfactual sets while enforcing causal invariance via distributional consensus supervision, thereby overcoming dataset-inherent bias limitations of conventional augmentation approaches. Building on causal interpretability. ABCD [66] establishes formal theoretical grounding for Transformer architectures by reinterpreting self-attention mecha- nisms as structural equation estimators that capture conditional independence relations through partial correlation analysis in deep attention layers, enabling zero-shot causal discovery over input sequences while accounting for latent confounders through repurposed pre-trained models. Expanding the causal intervention paradigm, Causal Attention (CATT) [67] implements front-door adjustment via dual-path processing of In-Sample and Cross-Sample Attention, strategically integrating external contextual information through CS-ATT while preserving standard attention mechanisms to dynam- ically mitigate spurious correlations without explicit confounder specification, thereby achieving bias-resistant vision-language alignment through implicit causal disentanglement. Moreover, Re- silientCL [10 68) proposes the causal interventional contrastive objective to mitigate the concept drift within the momentum network of contrastive pre-training paradigm.
A.3 Reinforced Fine-tuning
The integration of reinforcement learning (RL) into post-training alignment of large language models (LLMs) has undergone remarkable evolution since OpenAl's seminal work on Reinforcement Learn- ing from Human Feedback (RLHF) [69), which established foundational paradigm for aligning model outputs with human values (70] While early implementations like OpenAl-ol 771 demon- strated the efficacy of human preference modeling, the prohibitive costs of manual annotation have catalyzed paradigm shift toward automated reward generation through pre-trained systems. This transition has yielded innovative methodologies ranging from Bai et al.'s [72) constitutional approach utilizing sparse natural language feedback as proxy signals, to DeepSeek's progressive framework that first established baseline performance through pure RL (RO) before introducing their RI variant 773. The latter achieved enhanced generalization through cyclic alternation between supervised fine-tuning and their novel GRPO optimization protocol [25], exemplifying the field's progression toward self-contained alignment systems.
Besides, the landscape of alignment methodologies continues to diversify through innovative paradigms: ReST [74] employs iterative self- seff-generation of policy-derived samples to refine LLMs via offline reinforcement learning, while DPO [15] fundamentally reformulates alignment as direct preference optimization through implicit reward modeling. Concurrent developments span Rejection Sampling Fine-Tuning's 's (75) curation of validated reasoning trajectories for supervised augmentation, and ReFT's (1 phased optimization combining SFT initialization with PPO-driven exploration of automated reasoning path generation. Building upon these foundations, Visual-RFT [2] extends GRPO-based strategies to multimodal contexts, enhancing visual- language alignment under data scarcity, whereas B-STaR (76) introduces dynamic configuration adaptation for self-teaching systems through principled exploration-exxlootaiie Qwen-Math-PRe 77 synergizes Monte Carlo estimation with LLM-aa-judge consensus filtering while pioneering hierarchical assessment framework integrating stepwise and holistic performance metrics. Moreover, ViLaM 778 performs visual grounding unsupervised via reinforced learning under the open-world environment, Besides, APO [79 leverages reinforcement learning to align the knowledge in multiple teacher models for distillation.
balancing. Pushing the boundaries of evaluation rigor,
A.4 Multimodal Reasoning
Recent advances in Long Chain-of- Thhuught (Long CоT) reasoning [80 81 have significantly enhanced the capacity of Large Language Models (LLMs) perform multi-step reasoning and self- correction. By incorporating self-reflection strategies, these models can dynamically diagnose and revise their intermediate reasoning traces, thereby mitigating certain types of reasoning inconsistencies during inference. In contrast, our approach introduces a proactive intervention at the training stage through counterfactual sample generation, which explicitly shapes the model's 's causal representations and decision boundaries. This enables the model to internalize more consistent causal reasoning patterns from the outset, rather than relying on post-hoe correction during inference. Hence, we view our method and Long CoT-based self-reflection strategies complementary: while Long CoT
24

---

enhances reasoning reliability at runtime, our approach strengthens causal robustness during learning We believe future work could explore integrating both directions directions-leeveraiing Long CoT's reflective inference mechanisms together with counterfactual training to further advance consistent causal reasoning in reasoning-oriented L.MSs.
In parallel. Multimodal Chain-of Chain-of-Thought (MM-CoT) reasoning 82 82-85 has emerged as a crucial
paradigm for enabling LLMs align and reason across heterogencous modalities such as vision and
language. For instance, M [3] and Ddcot [85] introduce multi-domain, multi-step reasoning frameworks that emphasize structured comprehensive taxonomy of MM-CoT paradigms and benchmarks. Compared with these inference- time reasoning approaches, our method introduces proactive causal intervention during training through counterfactual sample generation, aiming to enhance causal robustness within multimodal reasoning processes. We view our approach as complementary to both Long CoT and MM-CoT waraligms-while they improve the reasoning trajectory at inference time, ours focuses on stabilizing causal representations during model optimization. This synergy could be a promising direction for future research.
modall inference, while the survey in [84] provides
Ground
Findings:
demanstaate low lung volumes. Tiny pleural effisions are new No signs of preumomis or pulmonary vascular congestion. Heart is top nomall in size though this is stable. Aorta markedly tortoous, unchanged. Aortic arch calcifications There preumuthoras. focal consolidation. Partially inaged upper abdomen unvemarkable.
views
Diagnosis: discase this patient Pleural Infuusion.
Counterfuctual Thinking:
Findings:
Findings:
PA and lateral views of the chest demonstrate low lung volumes Tiny
PA and lateral views of the chest demonntrrae low lung valumes. Thny bilateral plesral effusions No signs pncumunia pulmonary vascular congastoo Heart is top normal in size though this is stable, Aorta markedly wochanged. Aorti arch cokeifications no pncumathurax, Focal comsliddatinn ion noted in right lower Jobe. Partially imaged upper abdomen is unremarkable.
with signs consolidation both bilateral pleural effusioms are noted, No signs of peseumenia were previously present. There evidence of pulmonary congestion. The hoart sligltly enlarged, thought not born stable. sorts markedly tumacus, unchangod Aortic arch calcifications are seen. There is poeumnohoras. partially imaged upper ahdomen unremaskable.
since last esaminstion.
There
Dingnosis:
Diagnosis:
The discuse of this pasient Consolidation.
The discase of patient Pneumonia.
Ground Truth:
Findings
The patient's overying arm the lateral view partially obscures the view and makes evaluztion of the lnteral subogtimal. Left greater than rigltt biopical sconing noted. No pleural effusion pazumothorax seen. The cardine and mediastinal silhoucttes are unmmarkable. Multiple singical clips are noted overlying the left hemi thorax and the left axilla, Difficult to assess for medial left clavicular injury, nendisplaced fracture excluded.
Diagnosis: Te disease this Lung Opacity.
Counterfactual Thinking:
Findings
Findings:
partially obscures the view and makes evalustion the lnteral view subopcimal. Left greater than right biopical acoming is noted. A small left-sided pneumuthurax suspected difficult to the arm overlap. No plzural effusion seea. The cardiac and medias- tinal silhonetts are untemerkable. Multiple surgieal clips are noted overlying lefl hemiihoraa the axills. Nondisphaed fracture the medial left clavicle cannot escluded.
patient's overlying arm on
putients overlying arm the view and makes evaluation of the lateral view suhoptimal. Left greater than right hispical searring noted with evidence of edema the
partially obscures
definitively
lobe, No pleural rux seen. The carciac and mediastinal silhouestes are unremark- ablc. Multiple surgical clips noted overlying the left hemitho- axills. Difficult assess for medial clavicular injury, nondisplaced fracoure not escluded.
or pneumotho-
Diagnasis:
Diagnosis:
The discase this patient Pneumotherax.
The discase this patient Edemma
Figure 5: Samples of CXR-CounterFace (CCF) Dataset.
25

---

B CXR-CounterFact (CCF) Dataset
The motivation for choosing chest diagnostics as the application is due to the abundance of public professional medical diagnosis reports [86). which serve as rich CoT reasoning process. This domain provides an ideal platform for studying MLLM reasoning processes and validating the robustness of our proposed CPO in real-world settings.
Figure 5 showcases the samples utilized for training and validation in our study. We use a medical- specific L.M to generate the related caption of the image, with the prompt of:
"This is a radiology chest DR examination report of a patient: <Report>.
This is diagram of the relationship between lung diseases and their radiographic manifestations: < Conccept Graph>
Please generate a counterfactual radiology text showing <disease> based on the relationship and above context, with the same formatting."
As depicted in F3. comprehensive descriptions of the image are provided through long-form text, encompassing details such as size, position, relationships, and other relevant information about the disease present in the image. This ensures a detailed and information-rice depiction of the visual content. We have publicly released the datasets used for training and validation.
C Implementation Details
In this section, implementation details are provided.
In terms of the supervised fine-tuning progress, the hyperparameters are presented in Table 6 Qwen2.5-VL (7B) [16] is applied as our pre-trained model. During the SFT, we utilize the Adam W optimizer, which is configured with cosine annealing schedule the learning policy. The initial learning rate set to x 10-1, and the AdamW optimizer is employed with hyperparameters 3 = (0.9, 0.98). Additionally, we set the weight decay to 0.05 and the dropout rate to 0.1. During the first 20 warm-up steps, the learning rate increases to 1 x 10-4, and subsequently decays to 10 Unless otherwise specified, the supervised fine-tuning of our multi-modal large language model consists of 660 steps, executed on x 2 NVIDIA A100 GPUs.
Table 6: The training hyperparameters of our MLLM.
Supervised Fine-tuning
Counterfactual Preference Optimzation
Training Steps
660
Training Steps
7,750
Warmup Steps
20
Warmup Steps
0
Warmup Ratio
0.05
Optimizer
AdamW
Optimizer
Adam W
Learning Rate
2e-5
Learning Rate
le-4
Learning Rate Decay
Cosine (0.9, 0.98)
Learning Rate Decay
Cosine (0.9, 0.98)
Adam ,3
Adam B
Weight Decay
0.05
Weight Decay
0.05
Batch Size
4
Batch Size
15
While in the counterfactual preference optimization (CPO), the initial learning rate is reduced to 2 x 10-5 without the warmup. The visual encoder and text decoder are frozen out of the training. Thus, the batch size can be decreased to 4. The reinforced custom-tuning consists of 7,750 steps, executed on x 2 NVIDIA A100 GPUs. Other training parameters are the same the fine-tuning.
is worth noting that, both the SFT and CPO models were trained for exactly one epoch, as suggested by Qwen2.5-vl 4 The difference in the number of training steps arises solely because the CPO dataset is approximately three times larger than SFT dataset due to the addition of counterfactual trajectories. Therefore, both models completed their training after seeing their respective datasets once, indicating comparable convergence points terms of epoch count, rather than SFT being stopped prematurely.
26

---

C.1
Computational Cost
The main computational overhead of our framework stems from counterfactual sample generation using LLM experts. To ensure feasibility, we employ targeted generation strategy based on the hierarchical concept graph, which encodes disease entities, radiographic features, and their statistical dependencies. This allows to selectively perturb drift-prone features and generate only the two most relevant counterfactual samples per instance, rather than performing random perturbations.
Moreover, all counterfactuals are generated once offinn rather than dynamically during training. The construction of the CCF dataset took approximately five days on four A100 GPUs, including both inference and validation. Although this process is computationally intensive, it represents one-time cost that is affordable for real-world deployment. The subsequent CPO fine-tuning and evaluation were performed on 2xA100 GPUs with negligible extra overhead compared to standard fine-tume the-ttning.
C.2 Concept Graph Construction
The prompt used for automated extraction from the MIMIC-CXR dataset is follows:
TASK ROLE:
You are a senior chest radiologist. I will provide large number of chest DR case report texts. Please automatically extract key imaging feature words related various chest diseases from these reports, and use these features to build structured imaging knowledge graph to reveal the association between different diseases based common imaging features.
CORE REQUIREMENTS:
1. Standardized feature extraction:
Extract all abnormal imaging descriptors from reports. NNrmaliie terminology using Radiology Lexicon (RadLex) and Fleischner Society guidelines.
2. Disease-feature mapping:
Link standardized features to diagnoses diseases report. Identical features MUST use identical normalized terms across diseases.
3. Knowledge graph construction:
Nodes:
Diseases, such as Pneumothoraz, Atelectasis and Pneumonia; Features, such as lung opacity and air bronchograms. Relationship:
(Disease) [HAS] [EAASUEEATURR]--- -> (>eetuure);
(Feature) - [ASSOCIATED WITH] ->
Semantics:
Diseases sharing more identical feature node are interconnected.
OUTPUT REQUIREMENTS:
Please output the final constructed radiological knowledge graph and association data extracted from
the standardized the report structured JSON format. format as follows:
{" (" "features": []},
//
1,
fessesse {
"Pneumonia": ["Cor:::"""""iiii ("Consolidation", (Ground-glas" opacity", "Air bronchogran", "Pleural effusion"), |/ relationships
27


Reviews:
[{"review_id": "GrNTJeFqa5", "reviewer": "Reviewer_3kr3", "summary": "This paper addresses a critical issue in MLLMs—concept drift during CoT reasoning in non-stationary RFT. Concept drift refers to the unpredictable changes in reasoning token distributions during RFT, which can introduce significant biases in final predictions. The authors establish a theoretical link between concept drift and RFT, viewing CoT’s autoregressive token streams as non-stationary distributions subject to temporal shifts. To counteract the negative effects of concept drift, the paper introduces a novel method, CPO, which separates beneficial distribution adaptation from harmful drift.", "questions": "Refer to Weakness", "limitations": "Refer to Weakness", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n- The paper establishes a novel theoretical connection between concept drift and reinforcement fine-tuning, addressing a significant gap in the literature regarding the stability of reasoning processes in MLLMs.\n- The creation of the CXR-CounterFact dataset, with over 320,000 counterfactual reasoning trajectories, is a valuable contribution to the research community, facilitating future studies on counterfactual reasoning in medical applications.\n- The proposed Counterfactual Preference Optimization (CPO) offers a methodologically robust way to manage and counteract the harmful effects of concept drift during non-stationary RFT, especially in the context of chain-of-thought reasoning. This is a significant step forward in ensuring stability and accuracy in dynamic environments.\n\nWeakness\n- This paper focuses on the issue of concept drift in CoT reasoning in language models, but the drift problem in the visual modality has not been addressed.\n- The title and abstract of the paper should narrow the scope of the work, limiting it to medical imaging or chest diagnostics. The current title overclaims the contributions of the paper, as it only discusses chest diagnostics.\n- It appears that the CPO algorithm is essentially the DPO algorithm, with the negative samples replaced by counterfactual samples. Therefore, referring to it as a new CPO algorithm may not be entirely appropriate.\n- Your baseline should likely compare with the DPO using random negative samples, as this would demonstrate the effectiveness of your counterfactual approach. It would not be sufficient to simply use reasoning errors as negative samples and claim effectiveness.\n- In fact, most of your benchmark baselines use LLM models that are not pre-trained or fine-tuned with instructions, which makes the comparison unfair. Even if your method outperforms them, it would be difficult to attribute this to the CPO rather than the optimization of the backbone model.\n- Moreover, I noticed that the number of training steps for SFT is much smaller than that for CPO. The two should be compared at similar convergence steps, as CPO has about three times the amount of training data as SFT, which may mean that SFT was stopped before fully converging.\n- In the era of Long CoT [1-2], can various Reasoning LLMs with self-reflection strategies partially solve the problem of inconsistent causal inference?\n- Recently, methods like GRPO, which match based only on the final answer, are gaining popularity in reinforcement learning (RL) and other techniques, as they can achieve good reasoning abilities without requiring additional preference annotations. One concern I have is whether, if GRPO works, we still need CPO, which relies on the difficult process of constructing counterfactual data.\n- There is a lack of case studies and statistical analysis, making it difficult to demonstrate that CPO truly addresses the model's conceptual causal relationships, rather than simply allowing the model to better learn how to reason or acquire more knowledge.\n- Missing Multimodal Reasoning Reference: [1-6] \n\n[1] From System 1 to System 2: A Survey of Reasoning Large Language Models\n\n[2] Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models\n\n[3] Multimodal Chain-of-Thought Reasoning in Language Models\n\n[4] M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought\n\n[5] Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey\n\n[6] Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful, thoughtful and detailed review. Your positive feedback serves as a great source of encouragement for our team, especially recognizing the strengths: 1) **novel theoretical connection** addressing a significant gap in the stability of reasoning in MLLMs, 2) **valuable contribution** of the CXR-CounterFact **dataset**, and 3) **methodologically robust way** of our proposed method. Furthermore, we have diligently responded to each of your questions and incorporated your feedback into our manuscript revisions:\n\n1. **(Weakness 1) The drift in visual modality.** Thanks so much for your sincere and constructive suggestions. We acknowledge that this work does not address concept drift within the visual modality, as rightly pointed out by the reviewer. **The motivation for focusing on the concept drift in language models is based on our Observation 1.1 in the manuscript.** We find that when MLLMs are performing CoT reasoning within the language model, even if a slight perturbation does not change semantics, it will cause concept drift. Consequently, our current analysis and proposed methods focus primarily on this textual reasoning aspect. **We sincerely thank the reviewer for this valuable insight. In further work, we will focus on the drift in visual modality.**\n\n\n2. **(Weakness 2) The scope of the abstract.** Thank you for your thoughtful and helpful feedback. **We have revised the abstract to narrow the scope:** \"This paper uncovers a critical yet overlooked phenomenon in multi-modal large language models (MLLMs), especially for chest diagnosis: detrimental concept drift within chain-of-thought (CoT) reasoning during non-stationary reinforcement fine-tuning (RFT), where reasoning token distributions evolve unpredictably, thereby introducing significant biases in final predictions\". **The motivation for choosing chest diagnostics as an application is that, there are a large number of public medical diagnosis reports as professional CoT.** It is particularly helpful in studying the reasoning process of MLLMs and verifying the robustness of our proposed CPO in real environments. **We will continue to expand more multimodal applications in future works.** Thanks for your constructive suggestions.\n\n3. **(Weakness 3 & 4) CPO algorithm.** Many thanks for your kind and astute suggestions. While CPO and DPO share a preference-style optimization framework, CPO is fundamentally distinct. DPO contrasts human-preferred vs. dispreferred responses, which are random negative samples. **CPO instead contrasts factuals with counterfactuals generated under explicit causal interventions**, specifically designed to isolate the causal effect. CPO has a tighter decision boundary than DPO. Crucially, **we added additional experiments** that replace CPO's counterfactuals with DPO-style random negatives on multi-label chest diseases classification on MS-CXR-T as shown in Table 2. Our proposed CPO is significantly superior to DPO with the same number of training data, proving they are not functionally equivalent. **We have added discussions about DPO in the section of Methodology and Experiments.** Thanks for your sincere review.\n\n    ||Con.|PE|Pna.|Pnx.|Ede.|Avg.|\n    |-|-|-|-|-|-|-|\n    |SFT|54.9|71.7|70.0|95.9|76.5|73.8|\n    |DPO|63.2|72.4|76.7|93.5|76.3|76.4|\n    |CPO|77.7|72.7|87.4|95.8|75.3|81.8|\n\n4. **(Weakness 5) Pre-trained or fine-tuned Baseline.** Many thanks for your comments. We are sincerely sorry for your misunderstanding. **We follow the benchmark settings from the BioViL-T[7], and most of the compared methods have been pre-trained and fine-tuned.**  For example:\n\n    |Methods|Pre-trained Datasets|Fine-tuned Datasets|\n    |-|-|-|\n    |BioViL|PubMed，MIMIC-III, MIMIC-CXR|RSNA，MIMIC-CXR|\n    |BioViL-T|MIMIC-CXR v2|Chest ImaGenome, MS-CXR, RSNA, MS-CXR-T|\n    |Med-ST|MIMIC-CXR|MS-CXR-T，RSNA，COVIDx|\n    |TempA-VLP|MIMIC-CXR, Chest ImaGenome|MS-CXR-T，Chest Imagenome|\n    |CoCa-CXR|MIMIC-CXR,Chest ImaGenome|MIMIC-CXR, Chest ImaGenome|\n    |PromptMRG|MIMIC|MIMIC, IU X-ray|\n    |MambaXray|CXPMRG|MIMIC-CXR，CheXpert Plus, IU X-ray|\n    |**Ours**||**MIMIC-CXR, MS-CXR-T**|\n\n    Significantly, **our method outperforms the baselines despite utilizing less data,** indicating that the performance improvement is largely attributable to CPO, not simply backbone model optimization. **We have added more discussion of the comparison method in the section of Implementation Details.** Thanks again!\n\n5. **(Weakness 6) Training steps of SFT and CPO.** Thank you for raising this point. We appreciate your attention to the training details. **To clarify, both the SFT and CPO models were trained for exactly one epoch, as suggested by Qwen2.5-vl [9].** The difference in the number of training steps arises solely because the CPO dataset is approximately three times larger than the SFT dataset due to the addition of counterfactual trajectories. Therefore, both models completed their training after seeing their respective datasets once, indicating comparable convergence points in terms of epoch count, rather than SFT being stopped prematurely. **We have also added a discussion about the training steps in Implementation Details.** Thanks for your detailed review.\n\n6. **(Weakness 7) Inspiration of Long CoT.** Thank you very much for your constructive and thoughtful inspiration. We agree that LLMs employing long CoT reasoning with self-reflection strategies represent significant advancements and can indeed help mitigate certain forms of reasoning errors. However, self-reflection strategies primarily act as a reactive correction mechanism during the model's own inference process. **In contrast, our counterfactual samples are a proactive intervention during the training phase.** This actively shapes the model's internal representations and decision boundaries to be more robust against causal fallacies from the outset, reducing the need for later correction. Thus, **we view these approaches as highly complementary rather than mutually exclusive.** Moreover, **we have cited these papers [1-2], and added more discussion in the Outlooks.**\n\n7. **(Weakness 8) Discussion about GRPO.** Thank you for your insightful comment. We noticed GRPO's excellent efficiency and reasoning ability, especially on DeepSeekMath and DeepSeek-R1. **We also tried to use GRPO to train Qwen2.5-vl and DeepSeek-VL2 on MIMIC-CXR, but both encountered reward collapse that led to failed training.** Inspired by DAPO paper[8], we attribute this to GRPO's reliance on sparse final-answer rewards, where suboptimal reward assignment obscures high-quality samples. **In contrast, CPO provides denser causal trajectories**, significantly improving generalization and robustness by operating controlled counterfactual interventions. **We have also added a discussion about the GRPO in the section on Experiments.** Thanks for your detailed review.\n\n\n8. **(Weakness 9) Case studies.** We appreciate the reviewer's insightful feedback. **We have added the case studies based on Observation 1.1 in our manuscripts.** The CPO-finetuned logit results for the case under Observation 1.1 are presented below.\n\n\n    ||Atelectasis|Cardiomegaly|Lung Opacity|Pneumonia|Pneumothorax|\n    |-|-|-|-|-|-|\n    |\"lung opacity\"|0.19|0.22|0.83|0.75|0.08|\n    |\"opacity\"     |0.19|0.17|0.88|0.80|0.04|\n\n    Compared to the results shown in Fig. 1 of our manuscripts, CPO shows consistent performance even when slight perturbations are applied during CoT inference. In addition, our model assigns higher logits to the predictions of “lung opacity” and “pneumonia” inferred from the “opacity” attribute, and lower logits to all other diseases. This indicates that our model successfully performs causal inference.\n\n    **Besides, we also provided comparative experimental results of DPO in our answer to (Weakness 3 & 4) CPO algorithm.** Our superior experimental statistical results illustrate that the performance is attributed to the learn conceptual causal relationships, rather than acquiring more knowledge. Thanks for your thoughtful review. \n\n9. **(Weakness 10) References.** Many thanks for your detailed review. **We have cited these papers [1-6], and added more discussion about the multimodal reasoning in the section on Introduction and Related Works.** \n\nThanks again for your valuable time and thorough review. Your feedback is immensely valuable to us, and we welcome further communication from you. If you find our response satisfactory, could you please consider helping us improve our rating score?\n\n[7] S. Bannur et al., “Learning To Exploit Temporal Structure for Biomedical Vision-Language Processing,” CVPR 2023.\n\n[8] Yu, Qiying, et al. \"Dapo: An open-source llm reinforcement learning system at scale.\" 2025.\n\n[9] Bai, Jinze, et al. \"Qwen technical report.\" arXiv preprint arXiv:2309.16609 (2023)."}, {"review_id": "jCD0BIRHIj", "reviewer": "Reviewer_PZmM", "summary": "This paper addresses the critical issue of detrimental concept drift in CoT rea\u0002soning during non-stationary RFT of MLLMs. This paper reveals that unpre\u0002dictable shifts in token distributions during reasoning lead to significant biases\nin final predictions. To tackle this, the authors establish a theoretical bridge\nbetween concept drift theory and RFT, formalizing CoT’s autoregressive token\nstreams as non-stationary distributions undergoing temporal shifts. Based on\nthe theoretical framework, they propose Counterfactual Preference Optimiza\u0002tion(CPO), which leverages a concept graph-empowered LLM expert to gener\u0002ate counterfactual reasoning trajectories, systematically disentangling beneficial\ndistribution adaptation from harmful concept drift. CPO optimizes a dual ob\u0002jective to maximize alignment with human preferences while minimizing similar\u0002ity to adversarial counterfactual paths, enabling stable RFT in non-stationary\nenvironments, particularly in medical domains. Additionally, they introduce\nCXR-CounterFact(CCF), a large-scale dataset with 320,416 curated counter\u0002factual reasoning trajectories derived from MIMIC-CXR. Extensive experiments\ndemonstrate CPO’s superior performance in robustness, generalization, and ac\u0002curacy, validating its effectiveness in mitigating concept drift for non-stationary\ncustom-tuning", "questions": "See weakness.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strength**\n1. Theory, method, and experiments in a closed loop\n\nThis work establishes a solid theoretical foundation to formalize concept drift,\nwhich directly motivates the CPO method. The approach is validated through\nwell-designed experiments and ablation studies.\n\n2. Practical significance\n\nThis paper proposes a method to address the detrimental concept drift within\nCoT reasoning, which is of practical significance. The release of the CCF dataset\nalso provides a valuable resource for research in causal inference and robust AI.\n\n\n**Weakness**\n1. Computational overhead: The entire pipeline (concept graph construction→counterfactual data genera\u0002tion via LLM experts→CPO fine-tuning) appears to incur substantial computa\u0002tional costs. Notably, generating counterfactual samples would require extensive\nLLM inference operations, representing a significant computational burden. In\nAppendix C, the authors provide model training details (hyperparameters, 2x2\nA100 GPUs) but offer no analysis of the thorough analysis of these computa\u0002tional demands, which is critical for evaluating the method’s practical feasibility in real-world deployment scenarios.\n2. Generalizability of concept graph construction: The method’s core relies on a high-quality domain-specific concept graph. While\nthe paper mentions using Med-PaLM for automated extraction from MIMIC\u0002CXR dataset, the following aspects remain underexplored: the technical details\nof this extraction process, the required human verification costs(if need), and\nthe feasibility of extending this approach to other domains. And the current\nvalidation focused solely on chest X-ray may represent a potential bottleneck\nfor broader applications.\n\n3. Some suggestions for word choice: \n* Line 16: contributed→contribute. I think the tense here should be present tense\nrather than past tense?\n\n* Line 263: under→in. It seems a bit strange to use ”under” with ”environments”\nhere, should it be changed to ”in”?", "was_revised": true, "final_justification": "The authors have addressed my concerns, and thus I have decided to increase my ratings to accept.", "rebuttal": "Sincerely thanks for your meaningful and detailed review. Your positive feedback is a huge encouragement to our team, including: 1) **solid theoretical foundation** and **well-designed experiments** and 2) **practical significance** and **valuable dataset**.  And, we have provided detailed responses below to address your questions one by one, and made revisions to our manuscript according to your feedback:\n\n\n1. **(Weakness 1) Computational overhead.** Many thanks for your helpful and thoughtful suggestions. We acknowledge the reviewer's valid concern regarding computational costs and appreciate the opportunity to clarify the practical feasibility of our approach. The additional computational overhead is mainly concentrated in the generation of counterfactual samples by LLM, as pointed out by the reviewers. We mainly alleviated it from the following two points:\n\n    - **Targeted Counterfactual Generation instead of Random Perturbation.** The constructed hierarchical concept graph establishes disease entities, radiographic features, clinical relationships and taxonomies. By analyzing the statistical relationship between radiographic features and disease entities in the concept graph, **we can identify the features that are prone to drift and make targeted perturbations instead of random perturbations**. Therefore, for each sample, we only generated the two most relevant counterfactual samples, which greatly reduces the number of counterfactuals generated and reduces computing resources.\n\n    - **Static Generation.** Because we perform targeted counterfactual generation, we do not need to generate them dynamically during training. **Therefore, we only need to build the entire dataset once through the inference of LLMs, which is the CCF dataset we published.** The entire dataset construction process took about 5 days on 4 A100 GPUs. Although this is still very time-consuming and computationally intensive, **it is affordable and feasible in real-world deployment scenarios.**\n\n    **We have added more detailed discussions about the computation and generation of counterfactuals in the section on Implementation Details.** Thanks so much for your elaborate and thoughtful review. \n\n\n2. **(Weakness 2) Generalizability of concept graph construction.** We sincerely thank the reviewer for raising the points about the generalizability of our concept graph construction. \n\n    - First, **we added more technical details of this extraction process in the Implementation Details.** The prompt we used for automated extraction from the MIMIC-CXR dataset is as follows:\n        ```markdowns\n        TASK ROLE: You are a senior chest radiologist. I will provide a large number of chest DR case report texts. Please automatically extract key imaging feature words related to various chest diseases from these reports, and use these features to build a structured imaging knowledge graph to reveal the association between different diseases based on common imaging features.\n\n        CORE REQUIREMENTS:\n        1. Standardized feature extraction:\n            - Extract all abnormal imaging descriptors from reports.\n            - Normalize terminology using Radiology Lexicon (RadLex) and Fleischner Society guidelines.\n        2. Disease-feature mapping:\n            - Link standardized features to diagnoses diseases per report.\n            - Identical features MUST use identical normalized terms across diseases.\n        3. Knowledge graph construction:\n            - Nodes: \n                Diseases, such as Pneumothorax, Atelectasis and Pneumonia;\n                Features, such as lung opacity and air bronchograms.\n            - Relationship:\n                (Disease)-[HAS_FEATURE]->(Feature);\n                (Feature)-[ASSOCIATED_WITH]->(Disease)\n            - Semantics:\n                Diseases sharing one more identical feature node are interconnected.\n\n        OUTPUT REQUIREMENTS:\n        Please output the final constructed radiological knowledge graph and the standardized feature-disease association data extracted from the report in a structured JSON format. The format is as follows:\n        {\n            \"data\":[\n                {\"diseases\":[], \"features\":[]},\n                // ... cases\n            ],\n            \"disease_feature_map\":{\n                \"Pneumonia\": [\"Consolidation\", \"Ground-glass opacity\", \"Air bronchogram\", \"Pleural effusion\"],\n                // ... relationships\n            }\n        }\n\n        ```    \n        As can be seen,**we have achieved highly automated concept graph extraction and construction through LLM.** Then, one chest radiology expert assisted us in verifying the constructed concept graph.\n\n    - Second, in terms of **Broader Applications**, we have achieved highly automated concept map extraction and construction through LLM, with minimal manual verification. **We argue that our method can be transferred to other multimodal fields for broader application.** We will continue to expand more multimodal applications in future works. Thanks for your sincere review.\n\n    - Third, **The motivation for choosing chest diagnostics as the application is due to the abundance of public professional medical diagnosis reports, which serve as rich CoT reasoning process.** This domain provides an ideal platform for studying MLLM reasoning processes and validating the robustness of our proposed CPO in real-world settings. **Future work will explore additional multimodal applications. And we have added related discussions in the section of Outlooks.** Many thanks for your kind comments.\n\n\n3. **(Weakness 3) Typos.** Thanks very much for your corrections. We have corrected these typos and double-checked the entire manuscript to make sure there are no further typos.\n\nThank you for your continued time and thorough review. Your feedback is immensely valuable to us, and we are always eager to hear from you. If you find our response satisfactory, could you please consider helping us improve our rating score?"}, {"review_id": "GsHsYbIf3A", "reviewer": "Reviewer_T1Us", "summary": "This paper proposes and theoretically analyzes the problem of harmful concept drift in chain-of-thought (CoT) of multimodal large language models (MLLMs) in the field of medical diagnosis during unstable reinforcement learning fine-tuning. The authors introduced the concept of counterfactual and proposed counterfactual preference optimization (CPO) to solve the concept shift problem. In addition, the authors built a new dataset CXR-CounterFact based on the existing MIMIC-CXR and conducted experiments to prove the effectiveness of CPO.", "questions": "1. How does the authors get the positive/negative samples? (in Sec. 2.4, line 202-203) Is there any additional human involvement here?\n2. Line 151 and 155 might have a incorrect typo. The X might be T.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The author raised the problem of concept shifting and proposed a solution.\n2. The author's work is done with both theoretical analysis and experimental support.\n\nWeaknesses:\n1. It may not be clear enough about the introduction of the background and problem in Sec. 1 (such as what concept shifting is and how it manifests, how non-stationary environmental dynamics are manifested here, etc.). I hope the author could give a more understandable explanation or description of this concept you proposed.\n2. The author mentions that this phenomenon occurs in scenarios with inherent domain-specific data characteristics (in Section 1, lines 24-25). But this paper only mentions medical diagnosis. There are certainly many scenarios or fields with such features. Can this framework be transferred to other fields?\n3. The trade-off between time/cost and the efficiency of CPO is not discussed. CPO introduces an additional LLM expert. And from the loss function, each step involves at least four inferences by MLLMs, which may introduce additional training costs.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your kind and helpful review, especially for summarizing the strengths of our work, including 1) **raising the problem** of concept drift in CoT of MLLMs and solution, and 2) **theoretical analysis and experimental support**.  In response to your concerns, we have provided a detailed explanation below and revised our manuscript accordingly.\n\n1. **(Weakness 1) Background and problem.** Thank you very much for your sincere review. Our background and problem are based on our Observation 1.1 in our manuscripts, where we impose a small perturbation on the CoT reasoning process that does not affect the semantics, but the distribution of the results will be greatly deviated, as shown in our Fig.1 of our manuscripts. Specifically,\n\n    - Concept drift denotes unpredictable distribution changes in data streams [1], where concept drift manifests itself as a large deviation in the distribution due to a small perturbation in CoT reasoning, which is viewed as a sequential token stream generation process.\n\n    - Non-stationary environment refers to the CoT reasoning process, which is generated by the autoregressive decoding paradigm inherent to MLLMs, vulnerable to the interference from the data characteristics. And we posit that it is characterized as a sequential token stream generation process.\n\n    **We have added more details and explanations of our background and proposed problem in the section of Introduction.** Thanks for your kind review.\n\n2. **(Weakness 2) Transferred to other fields.** Thanks very much for your detailed and thoughtful review. **Our framework can be transferred to other fields.** \n\n    - First, **chest diagnostics was chosen as the application domain in our manuscript due to the wealth of publicly accessible professional diagnostic reports, which constitute valuable CoT reasoning data.** This makes it particularly suitable for studying MLLM reasoning and assessing the real-world robustness of our CPO framework.\n    \n    - Second, we mainly focus on the methodology for solving the concept drift problem in the CoT reasoning process. **Our implementation is also highly automated and can be easily transferred to other similar fields.**\n    \n    **We have added more discussion about our future work, exploring additional multimodal fields in the section on Outlooks.** Many thanks for your kind comments.\n\n\n3. **(Weakness 3) Training costs.** We appreciate the reviewer's valuable feedback regarding computational costs. As noted, the primary overhead arises from the additional LLM expert that generates counterfactual samples. We mitigate this through two key strategies:\n\n    - **Targeted Counterfactual Generation**: Leveraging our hierarchical concept graph, we statistically identify features susceptible to concept drift. This allows targeted perturbation of high-drift-risk features instead of random perturbation. Consequently, **we generate only two highly relevant counterfactual samples per original instance, significantly reducing computational demands.**\n\n    - **Static Dataset Construction**: The targeted approach enables pre-generation of all counterfactuals offline. **Thus, the entire counterfactual dataset, our published CCF dataset, is built once via LLM inference prior to training, eliminating dynamic generation costs during model optimization.** This process required approximately 5 days using 4 A100 GPUs. While still time-consuming and computationally intensive, this one-time cost is feasible and manageable for real-world deployment. We are sincerely sorry for your confusion of each step involving four inferences by MLLMs. **We have added more details in the section of Methodology**.\n\n    Many thanks for your kind and thoughtful suggestions.\n\n4. **(Question 1) Positive and negative samples.** Thanks for your kind question. We are sincerely sorry for your misunderstanding. **In terms of positive samples**, MIMIC-CXR not only provides the diagnosis results of the disease, but also provides a detailed diagnosis report issued by the doctor. In the diagnosis report, the doctor explains the reasons for the diagnosis results. **Therefore, we regard the report issued by the professional doctors as the chain-of-thought preferred by humans**, which are positive samples. **As for negative samples**, we generate counterfactual CoT for diagnostic report instances as our negative samples, that is, perturbing specific radiological features to interfere with the diagnosis results according to our proposed concept graph. **We have added more details of our positive and negative samples in Line 202-203.** Many thanks for your detailed review. \n\n\n5. **(Question 2) Typos.** Thanks very much for your corrections. We have corrected these typos and double-checked the entire manuscript to make sure there are no further typos.\n\nWe appreciate your time and thorough review. Your feedback is highly valuable to us, and we welcome further communication from you. If you are satisfied with our response, we would be grateful for your support in enhancing our rating score.\n\n[1] Lu, Jie, et al. \"Learning under concept drift: A review.\" IEEE transactions on knowledge and data engineering 31.12 (2018): 2346-2363."}, {"review_id": "aX60DEtvpN", "reviewer": "Reviewer_4zYr", "summary": "This paper investigates the issue of concept drift in multi-modal large language models (MLLMs) during non-stationary reinforcement fine-tuning (RFT), particularly within chain-of-thought (CoT) reasoning. The authors propose a novel method called Counterfactual Preference Optimization (CPO), which combines causal inference and counterfactual reasoning to disentangle beneficial distribution adaptation from harmful concept drift. CPO utilizes a hierarchical concept graph to guide the generation of counterfactual reasoning trajectories, enabling more stable and robust model tuning in dynamic environments, especially in the medical domain. The study makes four key contributions: (1) introducing a theoretical framework that formalizes autoregressive token generation in MLLMs using concept drift theory; (2) proposing the CPO approach that integrates structured domain knowledge with counterfactual interventions; (3) validating the method through experiments across multiple clinical benchmarks for chest radiograph tasks; and (4) contributing the CXR-CounterFact (CCF) dataset, comprising 320,416 curated counterfactual reasoning trajectories derived from MIMIC-CXR. Experimental results demonstrate that CPO outperforms existing methods in robustness, generalization, and accuracy under non-stationary conditions.", "questions": "Why does Table 2 compare only SFT and not directly compare with DPO?", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n1. This paper covers a wide range of content and has an extensive scope of work, such as: (1) introducing a theoretical framework that formalizes autoregressive token generation in MLLMs using concept drift theory; (2) proposing the CPO approach that integrates structured domain knowledge with counterfactual interventions; (3) validating the method through experiments across multiple clinical benchmarks for chest radiograph tasks; and (4) contributing the CXR-CounterFact (CCF) dataset.\n\n2. The authors apply relatively novel techniques from the AI field to medical scenarios.\n\n\n# Weakness\n\n1. The motivation section of the paper does not clearly describe the specific issues and consequences of concept drift in real-world medical problems, making it difficult for readers to grasp its practical impact.\n\n2. The authors introduced too many new concepts, making the paper overly incremental and lacking a focused research innovation. For example, the authors mentioned concept drift, counterfactual inference, reinforcement custom-tuning, and also contributed a dataset, but each of these contributions appears to be only a minor improvement. Meanwhile, the proposed CPO method seems to be nothing more than a simple variant of DPO.\n\n3. The authors' main contribution involves making minor modifications to existing methods, such as DPO, and then applying them in the medical domain. From the perspective of AI technology, I believe the paper's technical contribution is relatively limited.", "was_revised": false, "final_justification": "", "rebuttal": "Sincerely thanks for your meaningful and detailed review. Your positive feedback is a huge encouragement to our team, including 1) **theoretical framework**, 2) **CPO method**, 3) **experiments across multiple clinical benchmarks**, 4) contributing the CCF **dataset** and 5) **novel techniques**. In response to your concerns, we have provided a detailed explanation below and revised our manuscript accordingly.\n\n\n1. **(Weakness 1) Motivation of concept drift.** Thanks for your kind and helpful suggestions. **Our motivation is based on our Observation 1.1** in our manuscripts, where we impose a small perturbation on the CoT reasoning process that does not affect the semantics, but the distribution of the results will be greatly deviated, which is the concept drift, as shown in our Fig.1 of our manuscripts. \n\n    **In terms of real-world medical environments**, diagnosis requires strong robustness to resist noise, bias, and other issues from the data. However, the current MLLM will bring biased results by adding only small perturbations that do not affect the semantics to CoT. It poses a huge challenge to the application of MLLM in the medical field. **We have added more details to clarify our motivation for concept drift in real-world medical environments.** Thanks for your helpful suggestions.\n\n\n2. **(Weakness 2 & 3) Contributions.** Many thanks for your comments. We are sincerely sorry for the misunderstanding. **As for contributions:**\n\n    - First, we **formalized the issues** we found in Observation 1.1 using the concept drift framework, regarding the autoregressive CoT generation as a stream of next-token prediction actions, **providing a solid theoretical foundation to study the non-stationary reinforced custom-tuning**. \n    \n    - Second, within the concept drift framework, **counterfactual causal inference is used to further tighten the model's decision boundary under the non-stationary reinforced custom-tuning**, leading to our proposed Counterfactual Preference Optimization (CPO). By embedding learnable concept graphs as the expert,**our approach automates the generation of adversarially-constrained reasoning trajectories,** substantial decoupling between beneficial distribution adaptation and detrimental concept drift.\n\n    - Third, **we conduct comprehensive empirical validation to verify our proposed method across various clinical benchmarks**, including disease classification, report generation and zero-shot generalization. The superior results demonstrate statistically significant improvements in robustness, generalization, and accuracy of our method under non-stationary custom-tuning.\n\n    - Fourth, we contribute the **CXR-CounterFact dataset**, with over 320,000 counterfactual reasoning trajectories, facilitating future studies on counterfactual reasoning in medical applications.\n\n    **In terms of differences between our proposed CPO and DPO**, although CPO and DPO both utilize a preference-based optimization framework, CPO is fundamentally distinct from DPO. DPO compares human-preferred responses against dispreferred ones. CPO instead contrasts factuals with counterfactuals generated under explicit causal interventions, specifically designed to isolate the causal effect. CPO has a tighter decision boundary than DPO, as shown in the results of our additional experiments in the answer of 4. Experiment of DPO.\n\n    **As for the motivation of choosing chest diagnostics as the application**, it is due to the abundance of public professional medical diagnosis reports, which serve as rich CoT reasoning process. This domain provides an ideal platform for studying MLLM reasoning processes and validating the robustness of our proposed CPO in real-world settings. **We will explore additional multimodal applications in the future. And we have added related discussions in the section of Outlooks.**\n\n    **We have added more details and descriptions about our contributions in the section of Introduction.** Many thanks for your helpful suggestions.\n\n3. **(Weakness 2) Relationships among concepts.** In terms of relationships of different concepts** such as concept drift, counterfactual inference, reinforcement custom-tuning, **they are strongly related to each other in our work.** We initially discovered the problem of concept drift in reinforcement custom-tuning of chest diagnosis, so counterfactual causal inference is used to solve it. **We have added more details about their relationships to clarify them in the section on Methodology.** Thanks for your kind review.\n\n4. **(Question) Experiment of DPO.** we apprecaite your kind and helpful suggestions. **We have added additional experiments about DPO** on multi-label chest diseases classification on MS-CXR-T as shown in Table 2. Our proposed CPO is significantly superior to DPO with the same number of training data, proving the performance of our propsoed counterfactual preference optimization. **We have added discussions about DPO in the section of Methodology and Experiments.** Thanks for your sincere review.\n\n    ||Con.|PE|Pna.|Pnx.|Ede.|Avg.|\n    |-|-|-|-|-|-|-|\n    |SFT|54.9|71.7|70.0|95.9|76.5|73.8|\n    |DPO|63.2|72.4|76.7|93.5|76.3|76.4|\n    |CPO|77.7|72.7|87.4|95.8|75.3|81.8|\n\nThank you once more for generously dedicating your time to provide a thorough review. Your feedback is tremendously valuable, and we are open to hearing from you at any time. If you find our response satisfactory, we would greatly appreciate your assistance in improving our rating score."}]

Author rebuttal:

Meta-review: This paper addresses detrimental concept drift in chain-of-thought reasoning during non-stationary reinforcement fine-tuning of multi-modal LLMs. The author(s) propose Counterfactual Preference Optimization, which uses concept graphs to generate counterfactual reasoning trajectories that decouple beneficial adaptation from harmful drift in medical chest X-ray diagnosis. The work establishes an interesting theoretical connection between concept drift theory and reinforcement fine-tuning. The CPO method demonstrates statistically significant improvements over baselines across multiple benchmarks. Nevertheless, concerns such as computational overhead remain. The author(s) should address these concerns in the final version and explore applications beyond medical imaging to demonstrate broader generalizability of their approach.
Decision: Accept (poster)