Reinforcement Learning for Out-of-Distribution Reasoning in LLMs: An Empirical Study on Diagnosis-Related Group Coding
Authors: Hanyin Wang
Zhenbang Wu
Gururaj J. Kolar
Hariprasad Reddy Korsapati
Brian Bartlett
Bryan Hull
Jimeng Sun
Venue: NeurIPS 2025 spotlight ()
Keywords: RL
DRG
Medical Coding
Primary area: machine_learning_for_sciences
TL;DR: DRG-Sapphire, an LLM trained with GRPO, achieves SOTA accuracy in DRG coding and demonstrates a logarithmic scaling relationship between SFT examples and RL performance.

Abstract: Diagnosis-Related Group (DRG) codes are essential for hospital reimbursement and operations but require labor-intensive assignment. Large Language Models (LLMs) struggle with DRG coding due to the out-of-distribution (OOD) nature of the task: pretraining corpora rarely contain private clinical or billing data. We introduce DRG-Sapphire, which uses large-scale reinforcement learning (RL) for automated DRG coding from clinical notes. Built on Qwen2.5-7B and trained with Group Relative Policy Optimization (GRPO) using rule-based rewards, DRG-Sapphire introduces a series of RL enhancements to address domain-specific challenges not seen in previous mathematical tasks. Our model achieves state-of-the-art accuracy on the MIMIC-IV benchmark and generates physician-validated reasoning for DRG assignments, significantly enhancing explainability. Our study further sheds light on broader challenges of applying RL to knowledge-intensive, OOD tasks. We observe that RL performance scales approximately linearly with the logarithm of the number of supervised fine-tuning (SFT) examples, suggesting that RL effectiveness is fundamentally constrained by the domain knowledge encoded in the base model. For OOD tasks like DRG coding, strong RL performance requires sufficient knowledge infusion prior to RL. Consequently, scaling SFT may be more effective and computationally efficient than scaling RL alone for such tasks.

Reinforcement Learning for Out-of-Distribution Reasoning in LLMs: An Empirical Study on Diagnosis-Relatee Group Coding
Hanyin Wang 1:2
Zhenbang Wu? Gururaj Kolar
Hariprasad Korsapati'
Brian Bartlett*
Bryan Hull
Jimeng Sun 2.5
Mayo Clinic Health System
School of Computing and Data Science, UIUC
Mayo Clinic Rochester
*Mayo Clinic Phoenix
CCalle Illinois College of Medicine, UIUC
wang. hong.hanyieemay.o edu, jimeng@illinois edu
Abstract
Diagnosis-Relane Related Group (DRG) codes essential for hospital reimbursement and operations but require labor-intensive assignment. Large Language Models (LLMs) struggle with DRG coding due to the out-of-distribution (OOD) nature of the task: pretraining corpora rarely contain private clinical or billing data. We introduce ORG-SAPPHINE
which uses large-scale reinforcement learning (RL) for automated DRG coding from clinical notes. Built on Qwen2.5-7B and trained with Group Relative Policy Optimization (GRPO) using rule-based rewards. DRG- SAPPHIRE introduces series of RL enhancements to address domain-specific challenges seen in previous mathematical tasks. Our model achieves state-of- the-art accuracy on the MIMIC-IV benchmark and generates physician-validated reasoning for DRG assignments, significantly enhancing explainability. Our study further sheds light on broader challenges of applying RL to knowledge-intensing OOD tasks. We observe that RL performance scales approximately linearly with the logarithm of the number of supervised fine-tuning (SFT) examples, suggesting that RL effectiveness knowledge encoded in the base model. For OOD tasks like DRG coding, strong RL performance requires sufficient knowledge infusion prior to RL. Consequently, scaling SFT may be more effective and computationally efficient than scaling RL alone for such tasks.
fundamentally constrained by the domain
DRG-Sapphire
Accuracy Sample
55
DRG-Sasphir (oura)
53.9
DRG-LLMMA-2
50.2
50
DisicaBERR
447
Data, 50%
CAVI
accursyy
50
(ccuracdy
40
Model Type
K.2)
Genorative Model
20
Class fication Modal
8.8
30
3.3
Sample Size befors
Figure 1: Main Results. (A) Accuracy of DRG coding on the MIMIC-IV test set (N=26, 244)). DRG- SAPPHIRE outperforms proprietary reasoning models and the previous SOTA model, DRG-LLaMA. Notably, classification models could not generate reasoning for DRG code assignments. (B) Best RL performance increases linearly with the logarithm of the SFT sample sizes. Dashed line marks where 50% of training data was used for SFT. Best results from vanilla GRPO runs are shown.
Our code is available at htltttss//gtthu/ http://aaaannn////---/aaaaa
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Medical codes such DRG play pivotal roles in modern healthcare. DRG codes fundamental to the inpatient prospective payment system, directly inftuencing hospital reimbursement and key quality metrics [30]. Currently, assigning DRG codes from clinical notes remains a costly and iabor-intensive task, performed manually by highly trained coding specialists.
With the emergence of LLMs, there has been growing interest in leveraging these models for auto- mated medical coding 8 34 38 21 44]. However, DRG coding remains a particularly challenging task for LLMs (Figure J with prior attempts yielding limited success [3 34 A primary difficulty arises because DRG coding represents an
(OOD) task for off off-the-shelf LLMs. Due to the private nature of medical records, most LLMs likely have minimal exposure patient notes or billing data during pretraining. Additionally, DRG coding is inherently challenging due to: (1) high-dimensional search space with over 700 DRG codes; (2) advanced clinical reasoning required to link diagnoses with hospital resource use and disease severity; and (3) strict hierarchical rules governing DRG assignment.
Recent advances in reasoning models, such as OpenAI-ol [5] and DeepSeek-RI [12.. have intro- duced paradigm shift in LLM post-training. By leveraging large-scale RL with verifiable rewards, these models exhibit test-time scaling through extended chain-of-thought (CoT) reasoning, achieving tfae-ooftthe-ar (SOTA) performance complex tasks like competitive mathematics. Despite this progress, the design of optimal RL algorithms for scalable training remain an open challenge [45] (24). In the healthcare domain, RL applications using verifiable rewards are still in their early stages, with prior work primarily focused on medical knowledge benchmarks
19 20
In this paper, we present comprehensive exploration of large-scale, reasoning-orieen oriennted RL training for automated DRG coding from unstructured clinical notes. In theory, training towards a reasoning model is well-suited for this task: (1) it promotes the development of complex reasoning skills required for accurate code assignment; and (2) more importantly, it generates transparent rationales through CoT reasoning- key requirement for trust and explainablity in real-world clinical applications.
Through this work, we aim further derive insights into applying RL to challenging OOD tasks with off-the-shelf L.LMs. Using Qwen2.5-7B model and GRPO with DRG-rule-baeed rewards, we systematically investigate the prerequisites for successful RI., the allocation of data between SFT and GRPO under fixed data budget, and the impact of scaling SFT data. We also explore series of RL algorithmic enhancements and adaptive learning strategics. Our core contributions are as follows:
We introduce SOTA performance in automated DRG coding. Unlike prior methods, DRG-SAPPHIRE generates clinically helpful, physician-valideee validated reasoning. significantly improving explainability.
a novel model developed through large-scale RL,, achieving
2. We demonstrate that the performance ceiling of RL in this OOD task is bounded by the model's s capabilities before RL training. Specifically, observe that RL performance increases linearly with the logarithm of the number of SFT examples, suggesting that scaling SFT may be more effective and computationally efficient than scaling RL alone for such tasks.
3. We propose series of algorithmic enhancements and identify unique challenges in applying RL to DRG coding that distinguish it from mathematical domains- such as preference for an Answer-First cognitive pattern, and sensitivity to KL divergence for stable training.
2 Related Work
Automated DRG Coding Given their critical role in hospital operations and reimbursement, there is significant interest in automating DRG coding and enabling early DRG prediction 23
13 38 [0]
The prior SOTA method, DRG-LLaMA, fine-tunes a LLaMA model as a sequence classifier by replacing its generation head with a classification head (38.. Most existing approaches similarly
frame DRG coding as a multi-class classification task, offering limited insight into the rationale behind code assignments. While methods like DRGCoder provide input-level weight visualizations [13] their interpretability remains insufficient for real-world clinical deployment, where transparency and explainability are critical.

---

Replication Efforts of Deepseek-RI Recent studies have actively explored replicating the RL recipes of DeepSeek-RI. particularly mathematical and coding domains, with varying degrees One line of work has proposed approaches to address biases and improve sample efficiency the original GRPO algorithm 45 24 22]. Another active research area focuses
of success 477 14 40
on curriculum and staged learning strategies during reasoning-oriented RI. 48 36
41
16 3
central debate concerns whether RL truly fosters new capabilities beyond those already encoded in the base model. In DeepSeekMath, RL improved Majority @ K but not Pass @ K performance on mathematical tasks [33]. A comprehensive analysis across mathematical, coding, and visual reasoning tasks found that RL with verifiable rewards primarily reinforces existing reasoning capabilities rather than fostering novel ones [46]. Recently, Ma et al. (26) analyzed training dynamics complex reasoning tasks, showing that RL strengthens performance within model's
New Capabilities from RL?
existing capabilities, whereas SFT more effectively extends them beyond its current scope.
3 Large-scale RL for Automated DRG Coding
3.1 Problem Formulation
We aim to automate the hierarchical assignment of Medicare Severity Diagnosis-Related Group (MS-DRG) codes using LLMs. The MS-DRG system classifies each hospitalization into a single DRG code based on clinical complexity and resource utilization (see Appendix [A.] for details). Given a hospitalization represented by a set of clinical documents D, the DRG coding process applies extraction function l to identify the principal diagnosis wa procedure upp. and the presence of Complications or Comorbidities (CC) or Major Complications or Comorbidities (MCC). A hierarchical mapping function f then determines the final DRG code. Formally, the MS-DRG assignment is defined as:
(wa, Upp CC, MCC) = h(D), g = f(wa, Wp, CC, MCC),
where g the assigned DRG code. In this paper, we use an LLM automate this complex process.
3.2 Preliminary: GRPO
Compared to Proximal Policy Optimization 32 GRPO eliminates the value function and estimates the advantage using relative rewards within group
For each question q, GRPO samples group of outputs {01, 02. oG from the old policy Th.. and then optimizes the target policy те. In this paper, we enforce To.. = To to ensure strict on-policy learning. Under this setting, we maximizing the following objective:
33
(onro(!) Blq P(Q), fo Tomi (Ol2)]
0. A.. - B( (res(onsla 0r)
Trer(onala,0ree) To(on.1g, ol loo mo(oralg. or,ct)
(I)
G
where 3 is the coefficient for the KL divergence penalty, ae, is the reference policy, and Ai, is the advantage, computed based the relative rewards within each group {r:)a as:
A T. mean ((r))))) std (td)(r))g))
(2)
Here, denotes the reward assigned to output o, for prompt q. The gradient of JGRPO (0) is:
(o Rlgg P(Q), (or) To (O1z)
1)] V log Te(os,lu,..
(3)
Aes +
G
3.3 Improving GRPO Beyond the Baseline
We propose set of strategies to address key limitations of GRPO.

---

Dynamic Resampling for Advantage Preservation Existing RL algorithms suffer from the crudient-diminisiin problem. In GRPO, all completions {o, for a prompt q receive the same reward value, the resulting advantage for this group becomes zero. As training progresses, this issue becomes more pronounced due to policy optimization and accompanying entropy collapse [45), as more prompts yield completions with reward variance- either because all completions are perfectly correct or uniformly incorrect. This leads to a progressive decrease in the learning signal from the reward-based advantage.
To address this, we propose a dynamic resampling strategy (Equation 4 For each prompt q, sampled completions yield zero reward variance, we resample up to Nmax times until nonzero variance observed. Optionally, we enforce that at least one completion receives positive reward, guiding gradient updates toward high-reward trajectories.
(O) = Ely ~ P(Q), (0
locl
Ans - S0 (one(on))
log Foellorlyy.one -1)
(4)
To To( o1,
To To(oi, 06,4a)
s.t. Var((r:) >
within Nmax, optionally: Ho: r. > 03} >
which discards prompts that yield uniformly correct or incorrect completions. Given the data scarcity in clinical domains, we instead maximize the utility of each training example by resampling rather than discarding.
Our approach differs from the dynamic sampling strategy in DAPO 45
Intervening on Cognitive Behaviors
Cognitive behaviors, such as verification and backtracking, critical for effective reasoning-orientedd RI [T]] We explored additional reward functions and
) to incentivize three cognitive patterns in CoT reasoning, as shown in Figure 2 These are: (1) Answer-First, where the model outputs the DRG code before CoT; (2) CoT-First, where the model generates CoT reasoning before the DRG code; and (3) Differential Thinking, where the model evaluates three potential DRG codes before selecting the most appropriate.
a specialized SFT dataset (detailed in Section A.3)
Answwe-FFrrt
CaT-Fint
Differental Thinking
Jale Po viaal The yoecor
prokled
Bhe Melpy Eaest palteet brouk
G CITUUAAT SHSTIM bo-GNoees-Wii withico
Pate Onerr Fer evrnt Factore..
"Whrcco Deggnis
"Soud Codes
and Conoraclies CC MCC/H...
COANCCT 1100 cra der systemt aphowe with MCC
HV Shear Compllation or Comonleity
halloopp
eiy Colcias
omaz DIRCULATORR andiliu DANGNOSES WiTH
bncluse
spnoiiics desple
inoreiis voult be-1 -oTT DINCALAFORY SYSTTN
poolleed
Lasp DR0 Coos Johw deculakoy qutac daggeras Cif
enaayy concer rond de audaaaaae
OTHER CIICUL ATORY DYSTOM 340
Figure 2: Examples of Cognitive Behaviors.
KL Divergence Decay The KL divergence term in the GRPO objective (Equation I regularizes the divergence between the target policy Fe and the reference policy Te... However, this term exacerbates the zero-variance responses, the gradient, per Equation 3 becomes dominated by the KL term, pulling me toward Taw This drives over-regularization toward the reference policy and risks policy degradation. Recent work suggests that removing the KL penalty enhances reasoning capabilities in mathematical
problem in RL; as training progresses and more prompts yield
domains 45 24, [4]. Motivated by this, we explored two setups: (1) completely removing the
KL divergence term from the objective, and (2) applying a cosine decay schedule to the KL term's coefficient 3, smoothly reducing it to zero during training (see Section A.4 for details).
GRPO Variants In Equation [ dividing by lo,| during group-level advantage normalization introduces a length bias, diminishing the influence of longer completions on the policy gradient. To address this, DAPO 45] uses Σe |os| as the denominator, while Dr. GRPO 24 adopts constant normalization factor. Additionally, Dr. GRPO removes the division by std (rr) ) in Equation

---

mitigate question-leve' difficulty bias. We systematically evaluated these three strategies. Due to the strict on-policy nature of our setting (Te.. TT), we did not explore other modifications, such as clip-higher
45
Reward Shaping We implemented two straightforward yet robust rule-based reward components: Format Reward and Aceuracy Reward (detailed in the Section For the Accuracy Reward, investigated three distinct strategies: Dense Reward, Balanced Reward, and Striet Reward. These reward functions were designed to provide varying levels of reward signal sparsity, contingent on the correctness of the DRG code, its associated principal diagnosis, and the CC/MCC status.
3.4 Adaptive Learning Strategy
Curriculum Learning We investigate whether curriculum learning strategy, which organizes training cases by difficulty, improves performance compared to a mixed-difficulty baseline. We evaluated four setups, detailed Appendix AA6 (1) excluding easy cases, (2) excluding hard cases, (3) excluding both easy and hard cases (i.e.. using only medium-diiee ciddium-dfffiullty cases), and (4) training on easy cases first, then progressing hard cases.
Staged Learning Lastly, we explored staged learning strategy with three training phases of roughly equal length. After each phase, we identified easy and hard cases and evaluated two approaches: (1) additional SFT on hard cases, and (2) additional DPO on hard cases, before advancing to the next stage. As detailed in Appendix|A. |A.7 these approaches aim to improve the model's ' handling of challenging cases through targeted learning.
4 Implementation Details
Dataset We utilized the DRG-LLaMA training and test sets (38) derived from the publicly available MIMIC-IV dataset of real-world medical records [17]. The full training and test sets include 236, 192 and 26,244 cases, respectively. Each case uses the "brief hospital course" section from the discharge summary as input, with MS-DRG codes consolidated to version 34.0.
Training Pipeline and Scaling Strategy An overview of the training pipeline shown in Figure We first sampled a reduced dataset termed DRG-Small, comprising 20% of the full data (N=46,758), This subset served the foundation for extensive experiments
methodological variants and SFT-RL data mixtures, as detailed in Sections 5.2 through [53] After identifying the optimal configuration, we scaled training the full dataset produce the final DRG-SAPPLIRE model.
Step1
e Cold Start Eatnent
0+0 hoveee MANC-W
Boorstrpp OCT Res Beeenniingg for DRG Anckymmeri lainn
SFT on the Ooent Liiing
Large-Sorie RI. GaPO-aad and
Cold-Btan Modil Checkpaint
DIOG Sipphrre
Figure 3: Overview of Pipeline. We construct CoT cold-start dataset using Qwen2.5--7BIInstruce followed by SET with this dataset and large-scale GRPO.
Construction of SFT Dataset We prompted the Qwen2.5-7B. --enn.s--B-nsttructt model with medical records and groth DRG codes, tasking it to generate reasoning for DRG assignments (prompt provided in Section D. After extensive prompt engineering, manual inspection by domain expert revealed that the dataset exhibits strong reasoning logic (e.g., analyzing principal diagnosis first) but frequently contains factual errors (e.g., misclassifying a condition's 's CC/MCC status). We also included the complete list of original V34.0 MS- codes in a question answer format within the SFT dataset.
Model and RL Training We selected Qwen2.5-77 ---nttnttssssss 43 for the main experiments after evaluating various model size. GRPO training was conducted using the TRL package (37 for one epoch across all experiments.
We report model performance on the full test set using Pass 1, Pass@8,, and Majority@e @8 (Maj@8), following prior work in reasoning reasoning-orientted RL [33 46). Pass @ @1, reported as
Evaluation Metrics
5

---

the model's accuracy, is the mean aecuracy across eight runs. Pass assesses whether the correct DRG code appears among eight generated outputs, while Maj@8 determines the most frequent output matches the correct DRG code.
5
Experiments
5.1 Results of DRG-SAPPHIRE
Our best DRG-SAPPHIRE SPPHIRE model was trained using 90% SFT and 10% RL ratio on the full dataset (see Section 5.2 for SFT vs. RL ratio experiments), incorporating optimal GRPO enhancements and adaptive learning strategies (see Section 5.3tionn53331 for ablation studies).
significantly outperforms proprietary reasoning models, non-reasoning models, and the DeepSeek-distille Qwen 32B. It achieves new SOTA performance on DRG coding, surpassing the previous best, DRG-LLaMA-7B (54.8% vs. 53.9%). In addition to improved accuracy, DRG-SAPPHIRE provides interpretable reasoning-a compelling advantage over prior models trained purely as classifiers.
Comparison with Baselines As shown in Figure TA
Expert Reader Study Results Four physicians in hospital leadership roles, actively engaged DRG-related initiatives (e.g., reducing geometric mean length of stay), evaluated DRG- SAPPHRRE 's reasoning across 30 cases. On the dimensions of Helpfulness and Accuracy. DRG-SAPPHIRE received median rating of 4 out of 5, suggesting good potential for real-world applications (Figure 4). Qualitative assessments highlighted the explainability of DRG coding as highly valuable for DRG-related initiatives (see Section D.I for details), despite occasional factual inaccuracies in the reasoning.
S= Good
Review Score
4=Good
A=Accoptable
2-Poor
1-Very
Helpfulness
Accuracy
Figure Expert Reader Study.
5.2
Optimizing Data Allocation Between SFT and GRPO
GAPO
70
70
70
-8.2%
60
+30.66
+26.0%
80
4D
G8.0
70.9
30
$4.0
30
$0.9
30
25..8
31.4
32.9
45.0
10
36.7
47.2
Passay1
Passg98
SFT, 29% RL
100% SFT,
Training Time Accuracy by
80
DaPO
Accuraty
GARO
70
Voall baliin Lime Uhis
60
Accunoy
50
+6.4%
Acturary (hg
50
Accurcc comor
72.1
20
59.5
46.7
20
30.5
10-
Pasugl
Passggd
Hagg
Pass03
Fusuge
Magge
Figure 5: Impact of SFT-GRPO Data Ratios on DRG-Small Subset. (A-E) GRPO consistently improves Pass and Maj Luj@8 8 across all SFT ratios but reduces Pass (F) Total training time decreases with higher SFT ratios, GRPO is more time-consuming.
Effect of SFT-GRPO Ratios on DRG-Small First, we investigated the impact of varying the allocation of fixed data budget between SFT and GRPO on the DRG-Small subset (N=46,758) This contrasts with Deepseck-Rl-se R1-stll training, where only minimal SFT precedes RL. Across all

---

data splits, GRPO consistently improved Pass@1 I over the SFT baseline by an absolute margin of upproximately 10 percentage points (see Figure 5) We observed that this gain driven by improvements in Maj @8, not @ in fact, @ declines with GRPO. This pattern suggests that RL sharpens the model's output distribution toward higher-reward pathways, rather than introducing new reasoning capabilities in our experiments. Notably, the decline in Pass @8 during training indicates that RL may constrain diverse reasoning exploration. These findings align with recent studies [46] 33,, which question whether RL improves reasoning beyond the base model's capabilities. Moreover, the ultimate performance ceiling achievable with GRPO appears to be largely determined by the capacity of the initial SFT model; stronger SFT foundation generally leads to better post-GRPO results. From a computational perspective, scaling SFT prior to RL is more efficient, as GRPO entails substantial inference-tine time cost (see Figure 5F).
A. Accuracy vs. Global Stops
B. Pass@8 SFT Sample Size
C. Maj08 vs SFT Sample Size
57.5
tull 52% SFT Pull SP!
GIUPO
50
Ful Dats, 50%
Accuracy (%6
52.5
65
Maige
-35
50.0
40
67.5
35
CAPO
35
45.0
Fall Dato, 50% 5FT
250
300
750
1000
1300
212
213
224
215 222
Global Steps
SFT Sample Size before
SFT Sample before
Figure 6: Results on Full Dataset. (A) Accuracy from the two longest GRPO runs. (B-C) P and Maj @8 vs. SFT size. Dashed line marks where 50% of training data was used for SFT. Best results from vanilla GRPO runs are shown.
Log-Linear Scaling of GRPO with Increasing SFT Next, we scaled our training pipeline to the full dataset (N=236,192) 192). Based on the results above, we started with an SFT-GRPO data ratio of 50%-50% and progressively increased the SFT ratio under a fixed total data budget. Plotting these results alongside the DRG-Small subset revealed that both GRPO and SFT performance scale approximately linearly with the logarithm of the number of SFT examples (Figure B3). Although the number of GRPO steps varies across experiments in Figure D3, the benefit of scaling RI. appears limited in our study. Figure illustrates results from our longest GRPO runs, demonstrating modest benefits beyond 500 global steps. Consistent with earlier findings, GRPO reliably improves Pass @ I and Maj @8 while reducing @ass@8 (Figure 6 and C). As the number of SFT samples increased, the slope of the GRPO curves converged toward that of SFT across all metries. Additional results from scaling to the full dataset are detailed in Section
5.3 Ablation Studies on GRPO Enhancements and Adapative Learning
We present the results of ablation studies in Table D and Figure 12 All ablation studies were conducted on the DRG-Small dataset using Deepseek-BR RI -style training, with cold-start SFT on I 1% of the training data (N=2,362) before RL.
Reward Std
Training
C. Training Rosampling
Resamsling Resempling
2.95
601
0.95
Training Time (hours)
Reeamping
Vanila
Reward Std
Reward Score
0.70
272
0.45
Resampiing ing
Von
0.20
Global Steps
Global Stops
Figure 7: Dynamic Resampling, (A) Reward variance remains high under dynamic resampling. (B) During training, reward scores from both resampling strategies generally underperform vanilla GRPO. (C) Dynamic resampling substantially increases training time.
Dynamic Resampling Surprisingly, dynamic resampling-vve or without a positive reward con- straint-yiclder marginally better even worse performance than vanilla GRPO (Table I.. despite
7

---

Principal Diagnosis Pass@8
CC/CC
Model
Pass @1
P
Maj@8
Pass@1
Maj@s
Pass@1
P
Maj@s
Baseline
Vanilla GRPO Dense Reward
38.5
48.2
39.3
52.5
58.5
33.4
47.8
60.0
49.0
Dynamic Resampling
Neutral Resampling
20.3
41.9
38.1
27.0
52.5
50.5
25.6
52.6
48.0
Positive Reward Resampling
39.2
44.8
39.6
52.9
56.4
53.3
48.3
55.6
49.0
Coginitive Behvaiors Intervention
COT-First
35.5
52.2
37.4
50.9
59.6
52.4
46.3
66.7
48.4
Differental Thinking
63.0
GRPO Variants
DAPO L.oss
40.1
48.0
40.6
53.8
58.5
54.3
49.4
59.1
50.3
GRPO Loss
37.5
47.6
38.1
50.9
57.2
51.4
48.8
60.7
49.8
Dr. GRPO Advantage
38.5
$1.9
39.6
53.4
60.5
54.3
47.6
63.6
49.1
Divergence
No KI.
39.8
42.4
39.9
53.6
55.2
53.7
49.1
52.3
49.3
Docay
38.2
42.0
38.3
52.2
54.7
52.4
53.7
49.0
Reward Shaping
Striet Reward
40.1
49.1
40.9
52.8
58.1
53.7
47.6
59.0
48.8
Balanced Reward
38.1
51.3
40.0
52.1
60.4
53.8
48.2
64.0
50.7
Curriculum Learning
Remove Easy Cases
35.8
51.9
37.6
59.2
51.7
65.8
48.7
Remove Hard Cases
40.4
46.6
40.7
53.2
56,5
53.7
49.5
57.2
50.1
Remove Easy and Cases
38.7
39.4
53.4
59.9
49,3
From Easy Hard
29.4
51.7
32.7
43.4
58.6
46.5
40.8
68.5
44.3
Staged Learning
Staged SET
40.0
53.8
58.6
Staged DPO
29.3
46.1
31.2
43.8
54.3
45.5
43.1
64.2
45.7
Table 1: Ablation Study Results. Rows with a blue background indicate superior Pass @ perfor- mance compared to Vanilla GRPO + Dense Reward. Bold values denote the highest score for each metric.
preserving high reward variance (Figure 7A). More importantly, dynamic resampling proved compu- tationally inefficient due the frequent need to regenerate responses (Figure (C). We hypothesize that dynamic resampling may introduces training instability by oversampling vero-variance prompts, thereby skewing batches toward OOD responses rarely generated by the current policy. Additionally, this approach may inadvertently over-penalize low-reward outputs newly introduced into the batch, further distorting the learning signal.
Intervening on Cognitive Behaviors Our SFT dataset includes diverse reasoning styles, notably both Answer-First and CoT-First patterns. Interestingly, during training, the policy frequently converged toward the Answer-Fine strategy. To encourage CoT-First behavior, we experimented with additional rule-based reward and adjusted the SFT dataset to explicitly promote Differential- Thinking. Although both interventions successfully elicited the intended cognitive behaviors. their performance lagged behind the naturally emerging Answer-First pattern (Table TT. This finding is surprising, as CoTI-First strategies are often effective in complex reasoning tasks (39. We hypothesize that DRG coding benefits from direct prediction strategy, where outputting the DRG code first leverages implicit knowledge in the model's latent space, outperforming explicit CoT-grounded reasoning. These findings also align with recent studies 22 51, which suggest that CoT and extended reasoning may not always be necessary for reasoning models, and a "no-thinking" pattern can sometimes yield better performance.
KL Divergence In our experiments, remov- ing the KL penalty frequently led model collapse (Figure [A). This contrasts sharply with findings in mathematical reasoning tasks, where the KL term is less critical, underscoring its importance for cross-domain generalization. However, in cases where training successfully completed without the KL penalty, performance surpassed that of vanilla GRPO [ Addi- tionally, a cosine KI. decay schedule appeared beneficial. While it yielded no significant gains
T..Taninngg Gollaace wio
Training Curen for Hlaal Stepp
Det
Mears
120
333
imspore
54
150
Global
Global Stope
$10
Figure 8: KL divergence. (A) Examples of train- ing collapse when removing the KL divergence. (B) KL decay appears beneficial late in training.

---

in small-scale runs, it improved the training curve toward the end, suggesting that a lower KL penalty in later stages may help prevent ver-regularriztion the reference policy (Figure (B). Indeed, KL decay proved beneficial when scaling training on the full dataset, as shown in Table 2
GRPO Variants Among three GRPO vari- ants, the DAPO loss achieved the highest per- formance, while the Dr. GRPO loss performed the lowest (Table []. This finding aligns with re- cent work reporting that Dr. GRPO does not out- perform vanilla GRPO (6. Across all settings, we observed completion length contraction: as accuracy improved, output lengths sharply decreased before stabilizing (Figure (9B). This contrasts with trends observed in mathematical tasks, where longer completions are often asso- ciated with better performance.
Training Cores
R. Completion Length
cand Gal usu Vas daro
1:50
er xX
Global te
41
Global
Figure 9: GRPO Variants. (A) Dr. GRPO loss underperforms other GRPO variants from training curve. (B) All GRPO variants exhibit similar com- pletion length contraction.
Reward Shaping The strict accuracy reward, despite providing the sparsest reward signals, outper- formed both dense and balanced reward variants [ Notably, we observed no improvement in pincipal diagnosis or CC/MCC accuracy under the denser reward schemes. We hypothesize that denser rewards may lead the policy to converge prematurely to local optima, trading off global performance for easier-to-optimize intermediate signals.
Adaptive Learning We observed benefits from removing easy and hard cases during training (Table D.
Similarly, recent studies in the math domain suggest that maintaining medium-level difficulty cases may be most effective for RL training (36 resulted in only modest performance gains despite additional compute.
16 42). Staged learning with SFT
5.4 Prerequisites for Effective GRPO Training
We found that vanilla Qwen2.5 models (base and instruct) without SFT failed to generate correct DRG codes using GRPO alone, despite rapidly adopting the target reasoning format (Figure TOA). Post-SFT, all models showed improved RI. performance that generally scaled with model size, though gains from 7B to 14B were modest (Figure 10B). Higher SFT learning rates (up to 10-5) and extended training epochs both improved GRPO performance, though gains from additional epochs diminished
higher learning rates (Figure TOC). These results align with recent findings 29 emphasizing the importance of aggressive SET for reasoning- ientoning-intensive tasks.
from Out- Models
B. from SFT Models (by Size)
from SFT Models (by LR/Epochs)
Owen2.5 Base
0.5
0.8
0.5
Verila I-Brraut
Reward Score
0.0
Reward Score
-0.5
Reward
-0.5
-1.0
-1.0
-0.2
SFT 36-6 for spocts
-1.5
1.5
Owen2 5 148 inst act Oweno 7-44mmutttt
-0.7
SFT 35-6 for spocts
5IT Red5 for spochs
Owen2.5 1.5 Instruct
SFT 4o for spocts
-2.0
-2.0
-1.2
100
200
400 Global Steps
200
500
600
700
100
200
300
400
500
600 700
0
100
200
500 700 Global Steps
300
400
500
Global Steps
Figure 10: Prerequisites for GRPO Training. (A) Vanilla models without SFT fail to explore effectively, rarely generating correct DRG codes to receive positive reward signals. (B) GRPO performance increases with model size post-SFT. (C) Higher SFT learning rates and extended training epochs boost GRPO performance.
6 Conclusion
In this work, we used DRG coding as an empirical study to explore RL for OOD reasoning in LLMs. Our approach, applying GRPO with verifiable rewards, achieved a new SOTA performance while offering a key advantage over prior methods: the generation of physician-validated explanations through CoT reasoning. Critically, our findings reveal that RL performance this OOD task is

---

fundamentally constrained by the base model's capacity prior to RL. We observed logarithmic scaling relationship between the number of SFT examples and subsequent RL performance.
Following the successes of reasoning models like DeepSeek-R1, a prevailing narrative has been to "scale RI."" leaving critical question unanswered: what, precisely, should be scaled? Our work addresses this for complex, OOD tasks where knowledge infusion emerges as critical component. We find that scaling SFT can be more effective and computationally efficient than scaling RL alone. Moreover, despite extensive experimentation with RL algorithmic enhancements and adaptive learning strategies, these refinements yield only modest gains compared to simply initializing RL from stronger SFT baselines highlighting a "bitter lesson" in applying RL to tasks that fall outside the pretraining distribution of LLMs.
7 Acknowledgments
This research was supported by NSF awards SCH-2205289. The funder played no role the study design, data collection, analysis, and interpretation of data, or the writing of this manuscript. This research was supported (in part) by the Mayo Clinic Clinical Practice Innovation Program. This work was also supported (in part) by the Mayo Clinic Robert D, and Patricia E. Kern Center for the Science of Health Care Delivery.
References
[I] MIMIC-IV on physionet. http::///phisssnntt..oooo/ooooot
[2] Responsible use of mimic data with online services like gpt. post/ post/ggttrresposibleee
org/n/ws/
[3] S. Bae, J. Hong, M. Y. Lee, H. Kim, J. Nam, and D. Kwak. Online difficulty filtering for reasoning oriented reinforcement learning. arXiv preprint arXiv:2504.00 033380, 2025.
[4] J. Chen, Z. Cai, K. Ji, X. Wang, W. Liu, R. Wang, J. Hou, and B. Wang. Huatuogpt-ol, towards medical complex reasoning with llms. arXiv preprint 2x 2024.
[5] Y. Chen, J. Benton, A. Radhakrishnan, J. Uesato, C. Denison, J. Schulman, A. Somani, P. Hase,
M. Wagner, F. Roger, et al. Reasoning models don't 't always say what they think. arXiv preprint 45X5v:2505.054400 2025.
[6] X. Chu, H. Huang, X. Zhang, F. Wei, and Y. Wang. Gpg: A simple and strong reinforcement learning baseline for model reasoning. arXiv preprint arXiv:2504.022 02546, 2025.
[7] CMS. lcd-10-cm/ ms-drg v34. 0 definitions manual. htttp::://wwwcss.eeco cms/p0001.....
ott.aoooeeeeo/uooooooooeeeeoooo. html, 2016
[8] H. Dong, M. Falis, W. Whiteley, B. Alex, J. Matterson, S. Ji, Chen, and H. Wu. Automated clinical coding: what, why, and where we are? NPJ digital medicine, 5(1):159, 2022.
[9] H. Face. Open rl: A fully open reproduction of deepseek-rl. https://githu/ com/ huggingface/ taagginnace//pp--r11 open-r1, 2025.
[10] Y. Feng. Can large language models replace coding specialists? evaluating gpt performance in medical coding tasks. http:://wwww....aaaccooaaooeaeep 2025.
[11] K. Gandhi, A. Chakravarthy, A. Singh, N. Lile, and N. D. Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arX.v:2503.01307. 2025.
[12] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint 1511:22501..2948, 2025.
10

---

[13] D. Hajialigol, D. Kaknes, T. Barbour, D. Yao, C. North, J. Sun, D. Liem, and X. Wang. Drgcoder: Explainable clinical coding for the early prediction of diagnostic- related groups. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 373-380, 2023.
[14] J. Hu, Y. Zhang, Q. Han, D. Jiang. X. Zhang. and H.-Y. Shum. Open-reasoner-zero. An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290, 2025.
[15] A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, et al. Openai ol system card. arXiv preprini arXiv:2412.16720, 2024.
[16] Y. Ji, S. Zhao, Tian, H. Wang. S. Chen, Y. Peng, H. Zhao, and X. Li. How difficulty- aware staged reinforcement learning enhances 1lms' reasoning capabilities: A preliminary experimental study. arXiv preprint ArXiv:2504 X504:2504.00029, 2025.
[17] A. E. Johnson, L. Bulgarelli,, L. Shen, A. Gayles, A. Shammout, S. Horng, T. Pollard, Hao, B. Moody, B. Gow, al. Mimic-iv, a frrely accessible electronic health record dataset. Scientific data, 10(1):1, 2023.
[18] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, I.. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 61 1-626, 2023.
[19] Y. Lai, J. Zhong, M. Li, S. Zhao, and X. Yang. Med-rl: Reinforcement learning for generalizable medical reasoning in vision-language models. arXiv preprint arXiv:2503. 2025.
[20] W. Lan, W. Wang. C. Ji, G. Yang. Y. Zhang, X. Liu, S. Wu, and Wang. Clinicalgpt-rl: Pushing reasoning capability of generalist disease diagnosis with large language model. arXiv preprint arXiv:2504.0944 09421, 2025.
[21] R. Li, X. Wang, and H. Yu. Exploring Ilm multi- multi-agents for ied coding. arXiv preprint arXiv:2406. 15363, 2024.
[22] Z. Lin, M. Lin, Y. Xie, and Ji. Cppo: Accelerating the training of group relative policy optimization-based reasoning models. arXiv preprint 2rXiv:2503.22342, 2025.
[23] J. Liu, D. Capurro. A. Nguyen, and K. Verspoor. Early prediction of diagnostic-related groups and estimation of hospital cost by processing clinical notes. NPJ digital medicine, 4(1):103, 2021.
[24] Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. Lee, and M. Lin. Understanding -l-zeoolikke training: A critical perspective. arXiv preprint arXiv:2503.20783 2025.
[25] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.05101, 2017.
[26] L. Ma, Liang. M. Qiang. L. Tang, X. Ma, Z. H. Wong, J. Niu, C. Shen, R. He, B. Cui, et al. Learning what reinforcement learning can't 't: Interleaved online fine-tuning for hardest questions. arXiv preprint arXiv:2506. 005007,0........ 2025.
[27] W. Ma, He, C. Snell, Т. Griggs, S. Min, and M. Zaharia. Reasoning models can be effective without thinking, arXiv preprint arXiv:2504.0988 09858, 2025.
[28] T. Mu, A. Helyar, J. Heidecke, J. Achiam, A. Vallone, I. Kivlichan, M. Lin, A. Beutel, J. Schul- man, and L. Weng. Rule based rewards for language model safety. Advances in Neural Information Processing Systems, 37: 37:108877-108901 2024.
[29] G. Penedo, L. Tunstall, A. Lozhkov, H. Kydlicek, E. Beeching, L. B. Allal, Q. Gallouédee. L. von Werra, A. P. Lajarín, and N. Habib. Open rl update 3: Steady progress and new technical report, 2024. Hugging Face Blog.
[30] K. Quinn. After the revolution: Drgs at age 30. Annals of internal medicine, 160(6))426-4299 2014.
11

---

[31] J. Rasley, S. Rajbhandari. O. Ruwase, and Y. He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 3505-3506, 2020.
[32] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1177.066: 170700637777 2017.
[33] Z. Shao, P. Wang. Q. Zhu, R. Xu, Song, X. Bi. H. Zhang, M. Zhang, Y. Li, Y. Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.00 03000, 2024.
[34] A. Soroush, B. S. Glicksberg, E. Zimlichman, Y. Barash, R. Freeman, A. W. Chamney G. N. Nadkarni, and E. Klang. Large language models are poor medical coders benchmarking of medical code querying. NEJM AI, A():Aldbp22000000 2024.
[35] S. Sun, A. Schubert, G. M. Goldgof, Z. Sun, T.. Hartvigsen, A. J. Butte, and A. Alaa. Dr-llava: Visual instruction tuning with symbolic clinical grounding. arXiv preprint arXiv:2405.1955) 19567, 2024.
[36] K. Team, A. Du, B. Gao, B. Xing, C. Jiang, C. Chen, C. Li, C. Xiao, C. Du, C. Liao, et al. Kimi k1. 5: Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501.1259, 2025.
[37] L. von Werra, Y. Belkada, L. Tunstall, E. Beeching, T. Thrush, N. Lambert, S. Huang, K. Ra- sul, and Q. Gallouédec. Trl: Transformer reinforcement learning. https://github. com/
huggingface/
2020.
[38] H. Wang. C. Gao, C. Dantona, B. Hull, and J. Sun. Drg-Ilama: tuning llama model to predict diagnosis-related group for hospitalized patients. npj Digital Medicine, 7(1):16, 2024.
[39] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. Le, D. Zhou, et al. Chain-of- thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
[40] T. Xie, Z. Gao, Q. Ren, H. Luo, Y. Hong, B. Dai, J. Zhou, K. Qiu, Z. Wu, and C. Luo. Logic-rl: Unleashing I1m reasoning with rule-based reinforcement learning, arXiv preprint 1.6.: 2025.
[41] W. Xiong, J. Yao, Y. Xu, B. Pang. Wang, D. Sahoo. J. Li, N. Jiang, T. Zhang, C. Xiong, et al. A minimalist approach to Ilm reasoning: from rejection sampling to reinforce. arXiv preprint arXiv:2504.1134 2025.
[42] J. Yan, Li, Z. Hu, Z. Wang, G. Cui, X. Qu, Y. Cheng, and Y. Zhang. Learning to reason under off-policy guidance. arXiv preprint arXiv:2504. 14945, 2025.
[43] A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al. Qwen2. 5 technical report. arXiv preprint 15115:2412.15115, 2024.
[44] Z. Yang, S. S. Batra, J. Stremmel, and E. Halperin. Surpassing gpt-4 medical coding with a two-stage approach. arXiv preprint arXiv:2311.13753 2023.
[45] Q. Yu, 7. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, T. Fan, G. Liu, L. Liu, X. Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint arXiv: 14476. 2025.
[46] Y. Yue, Z. Chen, R. Lu, A. Zhao, Z. Wang, S. Song, and G. Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint a504 13837, 2025.
[47] W. Zeng, Y. Huang, Q. Liu, W. Liu, K. He, Z. Ma, and J. He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503.18892 2025.
[48] X. Zhang. J. Wang, Z. Cheng, W. Zhuang, Z. Lin, M. Zhang, S. Wang. Y. Cui, C. Wang. J. Peng. et al. Srpo: cross-domain inplementation of large-scale reinforcement learning on 1mm. arXiv preprint arXiv:2504.14286, 2025.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have written the abstract and introduction accurately reflect the paper's contributions and scope, including highlights from the experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
discussion of the limitations can be found in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:[NA)
13

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided an anonymous GitHub link to the code, and data access for reproducing the results is described in Section G
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification: We used publicly available data and have open sourced the code. Data access is described in Section G
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
httpoolliigggddde/oooooee http://uddde/ooeSuubnrb ssicnlolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide all training and testing details in Sections and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We discuss the statistical approach used to calculate accuracy in Section B.4
and display error bars representing standard deviation in Figure T2
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
15

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Computational resources are described in Section B and training time is discussed Sections an3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///eerriis..ccoo http:///rrrr..ccppaaaaccccccaaaa
/thhics/ttss::/hrrpp....o
Answer: [Yes]
Justification:
The research in this paper fully conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: We discuss the potential impact our methodology in real-world use cases in Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: Data and model release from this work adheres to the same data use agreement as MIMIC-IV, as outlined in Section G
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have properly cited the original owners of all assets used in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have open-sourced code via GitHub link.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/cood/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
The complete instructions given to reviewers are included in Section H
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification:
Our research was approved by IRB at our institution.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
18

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
19

---

Appendix
Contents
Addtional Methods
21
Problem Definition MS-DRG Coding
21
A.2
Rule-Based Reward Modeling
................................
21
A.3
Enforcing Cognitive Behaviors
22
A.4
KL Divergence Decay
22
A.5
GRPO Variants
22
A.6
Curriculum Learning
22
A.7
Staged Learning
23
B
Additional Implementation Details
23
B.1
SFT Training Details
23
B.2
GRPO Training Details
................................
23
B.3
Experimental Hyperparameiers
23
B4
Evaluation Details
23
B.5
Dynamic Resampling Detais
24
Additional Results
24
Experiments with GRPO Hyperparameters)
24
Accuracy with RL Training in Ablation Studies
................................
24
Additional Results from Scaling the Full Dataset
24
C,4 Error Analysis
25
Additional Results on a World Datasel
26
Additional Discussion
26
D.1 Clinical Applications of Automated DRG Coding with Reasoning
26
D.2 Practical Implication of Improved Pass but Not @a
26
D.3 DRG vs ICD Coding
26
E Additional Related Work
27
F Limitations and Future Work
27
Data Access
27
H Instruction to Reviewers
28
Prompts to LLM
29
Example Outputs from DRG-Sapphire
30
Example Outputs Demonstrating Different Cognitive Behaviors
32
20

---

A Addtional Methods
A.1 Problem Definition of MS-DRG Coding
Under the Medicare Severity DRG (MS-DRG) system, each hospitalization assigned a single DRG code based on clinical complexity and resource utilization, following rules established by the Centers for Medicare & Medicaid Services (CMS) 17]. Given hospital stay D = {d1, d2,..., dn where each d; represents clinical document generated during the hospitalization, the DRG assignment process performed by human coders can be mathematically represented as follows:
Extraction of Diagmoses and Procedures. From D, extract set W {W1, W2..... Wm J. where each wi E W corresponds to distinct medical diagnosis managed or a procedure performed during the hospital stay.
2. Identification of Principal Diagnosis Procedure. Select principal diagnosis wa W (for medical DRGs) or principal procedure up E W (for surgical DRGs), representing the main reason for admission or the primary surgical intervention. Only one diagnosis procedure is designated as principal depending on the case type.
Detection of Complications and Comorbidities. Identify the presence of Complications or Comorbidities (CC) and Major Complications or Comorbidities (MCC) within W, forming subsets:
CC C W, MCC S W, ccn 0,
which reflect distinet levels of clinical severity and resource impact.
4. Hierarchical Mapping to DRG. The final MS-DRG code is determined via:
(wa, Wp: MCC) h(D), g fwa, Wp, MCC),
where h extracts the principal diagnosis or procedure and CC/MCC from D, and f represents the
CMS-defined DRG mapping logic.
A.2 Rule-Based Reward Modeling
We adopted the following simple yet rigorous rule-based reward components.
Format Reward. The Format Reward enforces a structured response, requiring reasoning con- tent to be enclosed within <think></thinkk> tags and the final answer (DRG code) within (nnewe>></anwwe>> tags. The reward is defined as:
0,
if the response format is correct
Stormat
otherwise
Accuracy Reward. The Accuracy Reward evaluates the correctness of the DRG code, and applied only if the Format Reward condition is satisfied. We explored three reward shaping strategies:
(a) Dense Reward
2,
if full match
1.5,
if principal diagnosis match only
Sdense
0.5,
if CC/MCC match only
-0.,
if valid DRG but no partial match
1.5,
if invalid DRG
(b) Balanced Reward
2,
full match
1,
principal diagnosis match only
Spalanced
1,
if CC/MCC match only
-0.5,
if valid DRG but partial match
1.5,
invalid DRG
21

---

(c) Striet Reward
2
if full match
Sariet
0,
if partial or no match but valid DRG
-1.5,
if invalid DRG
A.3 Enforcing Cognitive Behaviors
To incentivize CoT-first cognitive behaviors, we introduced additional format penalty, If the model outputs DRG code within the first 50 tokens of the reasoning, penalty score of -0.5 is assigned.
To encourage differential thinking, designed new prompt for the Qwen2.5-7B -B-strnttt model to generate three potential DRG codes per case (prompt provided in Section [] each accompanied by reasoning, before selecting the most appropriate DRG code.
reconstructed the SFT dataset using the same data. We
A.4 KL Divergence Decay
To gradually relax the regularization imposed by the KL penalty, we apply cosine decay to the KL coefficient 3, reducing from its initial value to zero over the course of training. For global step and total training steps T, the decay factor is defined as:
0.5 (1+ cos
Nt T
decay_factone
The decayed coefficient at step is then:
3: = B
This decay schedule promotes stability in the early stages of training while encouraging exploration later updates.
A.5 GRPO Variants
We implemented the loss functions for different GRPO variants as follows:
#Vanilla GRPO loss. This is the original implementation in TRL v0.15.1. Bacpll__RPO___o_ = ((pe__token_ loss * completion_mast) sum (dim=1) completion_mask sum (dim=1)) .maa
#DAPO loss. This is the default implementation in TRL since v0.16.0. LAPO_GRPO_loss (per_token_ile loss completion_masti sum() completion_maske sum()
#Dr #DP. loss. We max tokens 1024.
max_ token 1024
Dr PP_GRPO_loss = ((per_token, token_ loss * completion (ostiionnmssk) mask). sum (dim=11)) max_ max_tokens) mean
For the Dr. GRPO advantage, we modified the advantage computation by removing the denominator
in Equation 2
A.6 Curriculum Learning
Unlike mathematical problems, the difficulty of DRG coding is not easily defined. High prediction accuracy does not necessarily indicate that a DRG code is inherently easy. For instance, the fre- quently occurring code "Septicemia or Severe Sepsis without MV >96 Hours with MCC" may be straightforward in most cases but can become challenging when the clinical narrative emphasizes a different primary condition, such as urinary tract infection. Moreover, standardized benchmark exists to quantify DRG coding difficulty.
To address this, we employed a static online filtering strategy. For each experiment, we first ran the model without filtering to establish a baseline. Easy cases were defined as those with zero reward variance and perfect accuracy scores. Hard cases were defined those with zero reward variance, an accuracy score of -0.5 under the dense or balanced reward, and under the strict reward. We then reran the experiment from the SFT model after excluding these filtered cases.
22

---

A.7 Staged Learning
For staged learning, we divided the training process into three stages, each with approximately the same number of global steps. After each stage, we identified hard and easy cases using the methodology described in Section For hard cases, we prompted the most recent GRPO model checkpoint generate reasoning given the case and the correct DRG code, We then performed SFT or DPO on this new dataset. For SFT, we used learning rate of 4 10 5 for epochs. For DPO, we designated the original model output as the rejected response, and trained with learning rate of 3 x 10 for epochs.
B
Additional Implementation Details
B.1 SFT Training Details
We used the SFT trainer from the TRL library for all SFT runs [37 with DeepSpeed ZeRO Stage 3 B1 and the AdamW optimizer [25 Training was conducted on H100 or A100 GPUs, depending on availability, using 6 precision. We set Packing-False and max_seq_length to 12846. A cosine learning rate schedule with a minimum of 10% of the initial rate was applied, along with warm-up ratio of 0.05. The global batch size was adjusted based on VRAM constraints to roughly match the number of unique cases per step used in GRPO training.
B.2 GRPO Training Details
Our implementation of GRPO was based the Open RI framework [9] which leverages vLLM [[]] for inference and GRPO Trainer from TRL library (v0.15.1) 37 for training. All training was conducted on 3 to H100 or A100 GPUs, depending on availability, using 6 precision. For dewvicettrainnbateeeeeeeeeeeeeee
_sie to 2 or 4. and gradient_ predient_accumllet accumulationnooooooccccccccccc.t attpptttttssspsssseeeeee to 32 64, ensuring consistent global batch size of 512 across experiments. Each global step consisted of 64 unique prompts, each with 8 generated completions. We set the max_ _exx__prmpp___ prompt_eengta t_lenthh to 4096 and the max_ faxx__copletion length 10240. The temperature the policy model is set to 1. All other training parameters were kept at their default values, including KL. regularization coefficient of B= 0.04.
all GRPO experiments, we set num gew.eeaeeationn to 8,
All GRPO experiments were run for a single epoch. As we enforced Te = aa to ensure strict on-policy learning, this is equivalent to setting num num_ cum_iterations to in later versions of the TRL library. We adopted the default system prompt from Open RI.
B.3
Experimental Hyperparameters
SFT For SFT, we experimented with different learning rates and training epochs, as detailed in Section 5.4 For all experiments in Section 5.3, we initialized GRPO training with an SFT model trained using learning rate of 4 x 10-5 for 9 epochs. The only exception is the result shown in Figure SA, which illustrates training collapse from earlier runs using a learning rate of 3 x 10 6 For experiments in Sections 5.2 used SFT models trained with learning rate of 4 x 10-5 but for epochs, as the SFT data was scaled.
GRPO For GRPO, we experimented with different learning rates and scheduling strategies, as detailed in Section C.T For all experiments in Sections 5.2 to 5.3] we used a GRPO learning rate of 3 x 10-6 with constant learning rate scheduler and a warmup ratio of 0.1.
B.4 Evaluation Details
We used vLLM [88 for inference during evaluation. All evaluations were conducted on the full test set (N 26,244). We the temperature to 0.6, cop-p to 0.95, and max_ max_tokenn tokens to 4096. To compute @8, we set to 8 in Sampl SamplingPare ingParams. generating eight completions per case. Pass reported as the mean accuracy across these eight generations. For evaluation, we extracted the DRG code enclosed within <answer><< a/anweer>>/ tags and computed exact match against the reference code after text normalization. All training curves figures in Section 5.3 are smoothed using a moving average with window of 50 steps.
23

---

B.5 Dynamic Resampling Details
For both neutral and positive dynamic resampling, we set the maximum number of regenera- tion attempts to 12. During regeneration, the model randomly selects a temperature from the set (0.7,0.8, 0.9, 1.0).
C Additional Results
C.1 Experiments with GRPO Hyperparameters
We conducted limited hyperparameter search to tune the learn- ing rate and scheduler for GRPO. As shown in Figure learning rate of x 10-6 consistently outperformed x 10 yielding faster convergence and higher final reward scores. We further compared constant and decaying schedules and observed similar overall performance. Notably, the constant schedule more effective at lower learning rates, though this advan- tage diminished as the rate increased. It is possible that constant learning rate helps mitigate gradient vanishing during RL training, as discussed in Section 3.3
Reward Score
0.50
GRPO 36-6 Decay
GRPO 30-6 Constant Decay GNO 1e-6 Constant
0.25
0
100
200
400 500 600 700 Global Staps
300
Figure 11: Effect of Learning Rate and Scheduler on GRPO.
C.2 Accuracy with RL Training in Ablation Studies
We present accuracy results from various ablation studies in Section 5.3 as shown in Figure 12
Dynamic Resampliin
Cognitive Thinking
Divergence
45
45
Resampling
Sat-trst
Resampling
Decay
40
Accuracy
Accuracy
60
20
200
300
400
500
600
700
200
300
400
500
600
700
200
300
400
500
600
700
Global Steps
Global Steps
Global Steps
D. GRPO Varlants
E. Reward Shape
F: Curriculum Learning
DAPO Less
Strict Roward
Dr GRPO Loss
Balanced Roward
GRPD Advamage
Accuracy 150
Accuracy (9)
Accuracy
-5
Remove Cases Hard Cases
200
400
600
200
400
300
400
500
Global Steps
Global
Global
Figure 12: Accuracy with RL Training in Ablation Studies. The dashed line indicates the baseline performance vanilla GRPO with dense rewards. Error bars indicate the standard deviation across runs.
C.3 Additional Results from Scaling the Full Dataset
We present experimental results on the full dataset with varying SFT-to-RI. data splits (Table 2 Our best configuration, combining DAPO loss, striet accuracy reward, and KI. decay (Section 5.3) consistently outperformed vanilla GRPO across all experiments. Curriculum learning, implemented
24

---

Principal Diagnosis
OCMCC
Model
Pass
Pasa@
Maj@8
Pus
Pass
Maj@8
Pass
Pass G/8
Majos
50% SFT
SET
44.6
75.3
50.9
58.1
77.1
63.4
51.8
80.7
58.4
Vanilla GRPD
52.8
64.2
63.9
64.9
69.6
Best Conlig
53.7
59.1
53.9
63.5
66.9
63.9
58.8
64.1
59.4
75% SPT
46.5
Vanilla GRPO
53.5
64.99
54.5
64.0
71
65.1
59.2
69.7
60.5
Best Coufig
54.6
60.5
54.9
64.4
68.2
64.8
59.6
65.3
60.2
Conlig Decay
Best Config Remove Hard Case
54.4
58.1
54.5
63.8
66.1
64.0
59.1
62.7
59.4
Best Config Remove Easy and Hard Casc
54.7
58.8
54.8
64.5
67.1
64.8
59.5
63.5
59.9
SFT*
10.5
41.4
35.7
12.4
47.1
43.6
11:7
47.0
41.6
Vanilla GRPO
54.1
65.2
55.0
64.3
71.2
65.2
59.8
70.0
60.7
Conlig
Best Config KI. Decay
54.2
66.9
55.4
63.9
72.1
65.2
59.1
70.8
60.8
Best Confii Remooe Hard Case
54.8
60.3
54.9
64.4
68.
64.7
59.9
64.9
60.4
Coulig Remove Easy Hard
95% SET
SIT
47.0
76.9
53.3
59.7
78.5
65.0
53.9
81.0
59.9
GRPO
68.6
72.3
Best Config
54.4
64.9
55.1
64.2
70.5
65.0
59.4
69.4
60.5
Best Config KL. Decay
53.0
69.4
55.1
63.2
73.5
65.1
58.4
73.4
60.8
Best Config Remove Hard
54.7
64.9
Best Coufig Remove Easy and Hard Casc
52.9
69.3
34.9
63.1
73.0
65.0
58.1
73.3
60.5
Table 2: Scaling of GRPO the Full Dataset. All experiments
conducted the full training set (N=236,192) 192) with varying OFF-to-RL ratios, and the best result from each experiment reported in the table. The best configuration of GRPO consists of DAPO loss, strict accuracy reward, and KI. decay. The row highlighted in blue indicates the best Pass@ performance. Bold values denote the highest exhibited format-following instability, resulting in lower-than-expected scores. Despite this unstable SFT baseline, RL training remained robust.
for each metric. The SFT checkpoint from the 90% SFT runs
by excluding hard or easy cases, further improved performance. The best overall performance of achieved with 90% SFT and 10% RL split using the best GRPO configuration and hard-case exclusion.
Notably, we excluded the SFT result from the 90% SFT runs and the vanilla GRPO result from the 95% SFT runs as outliers in Figure 3 and Figures 6 and C. The SFT checkpoint from the 90% SFT runs exhibited format-following instability, leading lower-than-cxpected scores. Despite this unstable SFT baseline, RL training remained robust, effectively leveraging the knowledge infused through SFT and ultimately producing our best overall results. The RL results from the 95% SFT runs are likely not representative of true RL potential due to insufficient RL training (< 250 global steps). Additionally, we did not conduct experiments without KL decay with curriculum learning for the 50% SFT runs, given the limited performance observed with vanilla GRPO that setting. Lastly, applied early stopping for the GRPO experiments from the 50% SFT runs due lack of improvement (Figure (A).
C.4 Error Analysis
We identified 9,537 cases consistently misclassified by all three checkpoints from different training stages: the cold-start SFT (SFT model in Figure 5 the small-scale GRPO (GRPO model in Figure 3), and the final ORG-SAPPHIRE model. As shown in Table even in these challenging cases, the model improved progressively during training, becoming notably "less wrong."
Model
Wrong Principal Diagnosis
Wrong CC/MCC
% Both Wrong
Cold-start SFT Cold-start SFT + GRPO
85.1% %
64.2%
47.4%
80.4%
1
41.1 1
DRG-SAR SAPPHIRE
74.9%
62.7%
33.6%
Table 3: Progressive error reduction across training stages on consistently misclassified cases.
25

---

Additionally, physician conducted manual error analysis on 100 randomly selected cases misclas- sifted by the final DRG-SAPPHIRE model, using the same error taxonomy DRG-LLaMA. The results are presented in Table 4
Error Category
Percentage (%)
Difficulty selecting correct base DRG
32
Erroneous CC/MCC
25
Potential incorrect DRG label
13
Inadequate clinical concept extraction
Information needed for DRG prediction unavailable
Other (e.g., procedure instead of medical DRG)
8
Table 4: Physician-annotated error taxonomy for 100 misclassified cases by DRG-SAPPHIRE.
C.5 Additional Results Real-Worid Dataset
We evaluated ORG-SAPPHIRE on an internal dataset from a healthcare institution. This dataset was constructed via stratified random sampling of 2,500 real-world cases, with DRG codes sampled in proportion to their frequencies in the MIMIC test set to ensure comparability, DRG-SAPPHIRE achieved an accuracy of 53.6% on this internal dataset, compared 54.8% MIMIC.
D Additional Discussion
D.1 Clinical Applications of Automated DRG Coding with Reasoning
DRG-SAPPHIRE shows good potential for real-world clinical applications. Two illustrative use cases are presented below:
Currently, DRGs assigned by professional coders and are typically available only after hospital discharge, DRG-SAPPHIRE can provide early DRG predictions to inform hospital operations and financial forecasting.
2. DRG-SAPPHIRE support DRG-related operational and quality improvement initiatives, such as those aimed at reducing the geometric length of stay, a metric directly determined by DRG. It provides transparent, interpretable explanations of DRG assignments, enabling clinicians to improve their clinical documentation to better reflect patient severity.
In discussions with domain experts, also found that the impact of erroneous DRG code assignments highly dependent on the intended use case. For high-stakes applications such as automated billing, the acceptable eтror must be extremely low due potential financial implications. In contrast, for operational or educational soch as the one mentioned above that helps physicians understand DRG assignments higher degree of fault tolerance may be acceptable.
D.2 Practical Implication of Improved Pass @1 1 but Not Pass @k
Our experiments demonstrate that RL improves Pass@ (i.e., accuracy) but not Pass@k for higher values, indicating that RL enhances the model's ability to produce the correct DRG code in a single attempt without necessarily improving its broader reasoning capacity. However, this outcome aligns well with the requirements of high-stakss clinical applications like DRG coding, where only the first prediction truly matters, as users typically do not sample multiple outputs. Morcover, selecting the correct answer from multiple candidate responses is challenging, methods beyond the best-of-N approach, which RL already optimizes by improving Pass through better majority voting (Maj @ k), not well-established.
D.3 DRG ICD Coding
Although both DRG and International Classification of Diseases (ICD) codes serve clinical and administrative purposes, they differ significantly in classification approach and real- worl-wolld applications.
26

---

DRG assignment is typically formulated as multi-class classification task, in which exactly one DRG code is assigned summarize resource utilization and clinical complexity for entire hospitalization. In contrast, ICD coding is multi-label classification problem, multiple ICD covernnovering both diagnoses and procedures- may be assigned to document single encounter. Furthermore, the two coding systems exhibit distinct hierarchical structures: DRG assignment explicitly emphasizes identifying principal diagnosis or procedure that primarily drives the hospitalization, along with secondary conditions that influence clinical complexity and reimbursement [7]. Finally, the utilization contexts of these codes differ substantially: DRGs are directly linked to inpatient reimbursement and hospital resource management, whereas ICD codes serve broader purposes spanning both inpatient and outpatient documentation.
E Additional Related Work
Several studies have explored constructing reward signals in RL beyond rule-based approaches. Mu et al. [28] proposed an effective method that combines fine-grained, composable rules with an LLM-based grader form hybrid reward signal. balancing model safety and usefulness. Sun et al. (35] introduced framework that grounds vision-language models (VLMs) in medical knowledge through symbolic representations of clinical reasoning and employs a symbolic reward function to assess VLM outputs for correctness and clinical validity.
F Limitations and Future Work
Our study encountered several limitations. First, we employed only rule-based rewards for final DRG assignments, without utilizing process supervision during the reasoning steps. While it is unclear how best to implement such supervision, theoretically, more granular and dense reward signals throughout the reasoning process could help guide the policy toward more effective exploration. Future work exploring this direction potentially combining explicit DRG rules with techniques such as process reward modeling represents intriguing avenue.
Second, we applied relatively static curriculum learning and case-filtering strategies, which were conducted only once following the completion of a base run, A dynamic, online, difficulty- filtering applied at the per-batch level-may be more effective and warrants further investigation.
Lastly, our work focused exclusively on the challenging task of DRG coding. Extending our approach to additional medical-domain tasks and to OOD tasks across domains would be a valuable direction for future work. In particular, it would be compelling investigate whether scaling RL across multiple tasks and domains fosters exploration of diverse reasoning trajectories beyond the base model, rather than merely refining its output distribution toward higher-reward outcomes.
G Data Access
Access to MIMIC-IV can be requested via ①. which requires signing data use agreement. The training and test datasets used in this study can be obtained by following the code repository provided in [38]. For experiments involving MIMIC-IV data and proprietary models, we adhered to the guidelines in [ and utilized the Azure OpenAl service.
27

---

H
Instruction to Reviewers
Physician reviewer instructions for scoring DRG-SAPPHIRE'S reasoning traces are provided below.
Instruction to Reviewers
1. You will be provided with discharge summary from the public MIMIC-IV dataset, along with a corresponding DRG code assignment and its rationale generated by large language model (LLM)..
2. Please note that, similar existing DRG prediction tools currently in use, the LLM-generatee DRG code assignment may be either correct or incorrect.
3. Your task is to rate the LLM output along two dimensions: Helpfulness and Accuracy. using scale from 1 to (very poor, poor, acceptable, good, or very good). where higher scores indicate better quality.
4. Helpfulness: For this dimension, please answer the question: "Is the LLM's reasoning and explanation helpful to frontline healthcare providers?" Reflect on real-world initiatives you engaged in that center around DRG optimization (e.g., efforts to reduce geo- metric mean length of stay). Assess whether the information provided by the LLM would meaningfully assist physicians in such settings, addressing questions commonly raised in practice.
Rubric:
Score of I (very poor):
The content is not helpful - for example, may be too generic, lack necessary detail, or be overly vague.
Score of (acceptable): The content is sufficiently helpful and acceptable for use in real-world clinical settings.
Score (very helpful): The content highly helpful and could positively impact real-world DRG-related initiatives.
5. Accuracy: For this dimension, please answer the question: "Does the information provided by the LLM accurately reflect MS-DRG assignment rules?" Base your evaluation on your best knowledge and understanding of the MS-DRG system.
Rubric:
Score of 1 (very poor): The information is substantially inaccurate.
Score of 3 (acceptable): The information is accurate enough to support Decision-making by frontline healthcare providers.
Score of 5 (very accurate): The information is highly accurate and consistent with MS-DRG assignment rules.
28

---

I
Prompts to LLM
The prompt used with Qwen2.5-7B-Instruct for generating the SFT cold-start dataset is provided below.
Generate Reasoning for DRG Assignment
MS-DRG (Medicare Severity Diagnosis-Related Groups) is a system used by the U.S. Centers for Medicare Medicaid Services to classify hospital inpatient stays for payment purposes. It groups patients with similar clinical conditions and resource needs into categories to determine reimbursement amounts. Each MS-DRG is assigned based on the patient's s principal diagnosis, secondary diagnoses, procedures performed, age, discharge status, and other factors. The goal ensure fair and consistent hospital reimbursement based on the severity of the illness and the complexity of care required.
CC and MCC in MS-DRG:
CC (Complication or Comorbidity): A secondary diagnosis that increases the com- plexity of care and resource utilization.
MCC (Major Complication Comorbidity): A more severe secondary condition that has sigmificant impact on resource use and hospital reimbursement.
MCCs have a greater effect on DRG weight than CCs due to increased patient care complexity.
What is the most likely Medicare Severity Diagnosis Related Group (MS-DRG) based on the following discharge summary of hospitalization? Provide the complete description of the DRG code without including the numerical code.
[Discharge Summary]
The answer is [DRG Code]
Explain the reason why the DRG code is assigned. In your reasoning step, assume you don't know the right DRG code yet.
For SFT dataset generation with differential thinking, we added the following content to the prompt above.
Additional Prompt Generate Differential Thinking
First, analyze the case details, including the principal diagnosis, secondary diagnoses, procedures performed, and any other relevant factors that may influence DRG assignment.
Next, list three possible DRG codes that you believe are the most likely. Enclose each DRG code in square brackets, like this: [Heart failure without MCC). One of the three DRG codes should be [DRG Code].
Then, provide a detailed analysis for each of the three DRG codes. For each one, explain:
Why this DRG code might be appropriate?
Why it might not be appropriate?
Finally, based on your analysis, identify the most likely DRG code and it should be [DRG Code].
29

---

Example Outputs from DRG-Sapphire
Example One
<think>
Based on the provided discharge summary, the most likely Medicare Severity Diagnosis-Relatee Group (MS-DRG) is "ESOPHAGITIS GASTROENTERITIS AND MISCELLANEOUS DIGESTIVE DISORDERS WITHOUT MCC." Let's break down the reasoning step-by-step:
1. Principal Diagnosis and Conditions:
The principal diagnosis is colitis, which is digestive disorder.
There complications comorbidities (CCs MCCs) that significantly im- pact the complexity of care or resource utilization more than the primary diagnosis
of colitis.
2. Secondary Conditions and CCs/MCCs:
Atypical Carcinoid Tumor of the Lung: A significant condition, but not the primary reason for the hospitalization and does not elevate complexity of care related to colitis.
Metastatic Pancreatic Neuroendocrine Tumor: Also significant, but not directly impacting the care for colitis.
Perforated Diverticulitis: Required prior hospitalization but is not contributing to the current hospital stay.
Transaminitis: Mild and resolved quickly; not related the colitis.
Hypothyroidism: Chronic and well-managed; no impact on the complexity of care.
possible cause of colitis, but not confirmed and does significantly impact complexity.
Ischemia:
3. Complications or Comorbidities (CCs):
The atypical carcinoid tumor, metastatic panereatic neuroendocrine tumor, and ischemia are significant but not classified as CCs that elevate complexity for colitis.
The perforated diverticulitis is historical and not relevant to the current hospital stay.
4. Major Complication or Comorbidity (MCC):
No significant conditions qualify as MCC that would elevate the DRG weight beyond the primary diagnosis of colitis.
Given these conditions, the DRG code is assigned based on the primary diagnosis of digestive disorders, specifically colitis, without any significant complications or comorbidities that would elevate it to higher complexity category. Therefore, the most appropriate DRG is "ESOPHAGITIS GASTROENTERITY AND MISCELLANEOUS DIGESTIVE DISORDERS WITHOUT MCC." </think>
<answer>
ESOPHAGITIS GASTROENTERITIS AND MISCELLANEOUS DIGESTIVE DISOR- DERS WITHOUT MCC
</answer>
30

---

Example Two
<think>
Based on the discharge summary provided, the most likely Medicare Severity Diagnosis-Relan Related Group (MS-DRG) PANCREAS LIVER AND SHUNT PROCEDURES WITH MCC. To determine this, let's break down the key elements of the case:
1. Principal Diagnosis and Procedures:
The patient underwent left hepatic lobectomy and placement of biliary stents.
The lobectomy is major liver surgery-thie this alone is significant factor for DRG assignment.
2. Complications and Comorbidities:
Post-operative bilious drainage from the JP (jejunostomy) tube.
Inereased wound drainage requiring VAC (wound vacuum) placement.
ERCP performed manage bile duct stones and issues.
Biliary stents placed during ERCP
Elevated WBC count and suspected infection requiring IV antibiotics and infectious disease consultation.
3. MCC Analysis:
The ERCP, biliary stents, and post-op complications significantly impacted recov- ery.
These events qualify as a Major Complication or Comorbidity (MCC) under DRG criteria.
4. Severity and Complexity:
Multiple interventions (ERCP, stent placement, IV antibiotics).
Extended inpatient stay due to complications.
High resource utilization consistent with MCC-level DRG classification.
5. Consistency with MS-DRG Criteria:
MS-DRG assignment accounts for diagnosis, procedures, and complications.
This case aligns with the category for major liver/pancreas procedures involving MCCs.
Therefore, the DRG code PANCREAS LIVER AND SHUNT PROCEDURES WITH MCC is assigned because the case involves significant liver procedure (left hepatic lobectomy) and multiple major complications and interventions. The care complexity and extended course fulfill criteria for an MCC classification.
</think>
<answer>
PANCREAS LIVER AND SHUNT PROCEDURES WITH MCC
<lanswer>
31

---

K
Example Outputs Demonstrating Different Cognitive Behaviors
Example from Answer-First
<think>
Based on the provided discharge summary, the most likely Medicare Severity Diagnosis-Red Rellate Group (MS-DRG) is "OTHER CIRCULATORY SYSTEM DIAGNOSES WITH CC." Reasoning:
1. Principal Diagnosis:
The patient presents with chest pain and ST elevations, initially suggestive of STEMI (ST-Elevation Myocardial Infarction).
Cardiac catheterization revealed normal coronary arteries, ruling out coronary artery disease.
ST elevations without coronary lesions point to myopericarditis or related circu- latory system condition.
2. CC (Complication Comorbidity):
Chest pain, ST elevations, and treatment with ibuprofen and colchicine increased care complexity.
Mild echocardiographic findings (dilated aortic root and ascending aorta) required follow-up and additional monitoring, qualifying CC.
3. MCC (Major Complication Comorbidity):
Despite significant diagnostic workup, no MCC-level condition was present.
No major complication or comorbidity substantially elevated resource use reimbursement criteria.
4. Discharged with Specific Medications and Follow-Up Plans:
The patient was discharged ibuprofen and colchicine.
A follow-up echocardiogram was planned, indicating ongoing resource use.
Conclusion:
The DRG code "OTHER CIRCULATORY SYSTEM DIAGNOSES WITH CC" is assigned because the patient's 's primary condition, although initially concerning, did not meet the criteria for MCC. The presence of chest pain, ST elevations, treatment complexity, and follow-up needs justify a CC classification to increased resource utilization and care complexity. </think>
<answer>
OTHER CIRCULATORY SYSTEM DIAGNOSES WITH CC
</answer>
32

---

Example from CoT-Fiirt
<think>
To determine the most likely MS-DRG for this patient, let's break down the key elements from the discharge summary:
1. Principal Diagnosis:
The patient presents with ST elevations in the EKG, initially consistent with a STEMI (ST-Elevation Myocardial Infarction).
Cardiac catheterization revealed normal coronaries (RRA Right Rectus Artery), indicating a false positive STEMI.
The patient was diagnosed with myopericarditis, which can cause chest pain and ST segment elevations.
2. Secondary Diagnoses and Procedures:
Myopericarditis (secondary to chest pain and ST elevations)
Normal cardiac catheterization
No significant past medical history
Mildly dilated aortic root and ascending aorta on TTE
GERD, insomnia, and OSA active treatment or significant impact on hospital- ization
3. Complications and Comorbidities (CC and MCC);
Myopericarditts increases the complexity care and qualifies as a CC.
There is no indication that it qualifies as major complication or comorbidity (MCC).
Reasoning for the DRG Assignment:
Myopericarditis is the principal diagnosis and considered CC, not an MCC.
Mildly dilated aortic root and other secondary findings do not affect DRG assignment.
GERD, insomnia, and OSA are managed conditions that do not impact hospital resource use.
DRG Code Assignment:
The most likely MS-DRC for this patient. based on the provided information, is "OTHER CIRCULATORY SYSTEM DIAGNOSES WITH CC." This assignment reflects the presence of a complication (myopericarditis) that increases care complexity, without rising to the level of MCC.
</think>
<answer>
OTHER CIRCULATORY SYSTEM DIAGNOSES WITH CC <aanswe>>
33

---

Example from Differential Thinking
<think>
Analysis of the Case Details
Principal Diagnosis:
ST-Elevations and Myopericarditis
Secondary Diagnoses:
No significant past medical history
GERD (Gastroesophageal Reflux Disease)
Insomnia
OSA (Obstructive Sleep Apnea)
HIV and Syphilis (screening recommended)
Procedures Performed:
Cardiac Catheterization
Other Relevant Factors:
Normal cardiac catheterization result
Mildly dilated aortic root and ascending aorta
Started on ibuprofen and colchicine
Possible DRG Codes
1. Heart failure and shock without CC/MCC
2. Other circulatory system diagnoses with CC
3. Other circulatory system diagnoses with MCC
Analysis for Each DRG Code
1. Heart failure and shock without CC/MCC
Why this DRG code might be appropriate:
The patient was admitted with ST-elevations, which can be indicative of heart issues.
Mild dilation of the aortic root and ascending aorta could suggest early signs of heart failure.
Started on medications for pain and inflammation, which are common in heart failure management.
Why it might not be appropriate:
Cardiac catheterization was normal, which doesn't support a diagnosis of heart failure.
No history of heart failure shock-only myopericarditis and mild aortic dilation.
GERD, insomnia, and OSA are unrelated to heart failure.
2. Other circulatory system diagnoses with CC
Why this DRG code might be appropriate:
ST-elevations and myopericarditis are circulatory system issues.
GERD CC) increases complexity of care and resource utilization.
Why it might not be appropriate:
Condition managed with ibuprofen and colchicine does not indicate severe complica-
tions.
Absence of heart failure severe circulatory system condition warrant more specific DRG.
3. Other circulatory system diagnoses with MCC
Why this DRG code might be appropriate:
34

---

ST-elevations and myopericarditis significant cireulatory issues.
GERD (a CC) could be misclassified as MCC if chronic impact is overestimated.
Why it might not be appropriate:
Condition managed conservatively- not indicative of MCC-level complexity.
No evidence of heart failure or other major circulatory complications.
Conclusion
Most Likely DRG Code: Other circulatory system diagnoses with CC
Reasoning:
Myopericarditis with ST-elevations is significant circulatory condition.
GERD, as CC, increases care complexity but not to the level of MCC.
No evidence of heart failure or other severe conditions to justify different DRG code.
Based on the provided information, the most appropriate DRG code is Other circulatory system diagnoses with CC, as it best reflects the patient's 's primary condition and the presence of a CC (GERD) that increases complexity care. </think>
<answer>
OTHER CIRCULATORY SYSTEM DIAGNOSES WITH CC </answer>
35


Reviews:
[{"review_id": "xOMEzBZKS4", "reviewer": "Reviewer_DCiq", "summary": "This paper explores the use of reinforcement learning methods represented by GRPO for the DRG coding task and introduces the DRG-Sapphire model, which achieves SOTA performance. \nThe authros design and conduct extensive ablation experiments, providing a comprehensive analysis of the effectiveness and interactions of RL and SFT in this task.", "questions": "see above", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1. The problem addressed is highly useful in real world. The authors are (probably) the first to apply advanced reasoning models and GRPO methods to this task.\n2. The ablation studies are astonishingly rich, providing valuable insights for the community to understand the dynamics of RL trained LLMs.  \n3. There are several novel and interesting settings, such as the exploration of cognitive behaviors and the investigation of how to allocate data between SFT and RL under a fixed budget.  \n\n\nWeaknesses\n\nMajor\n\n1. My major concern lies in the exact source of the performance improvement: the baseline models are highly likely to lack exact knowledge of the coding system and thus fail to generate correct codes. The performance improvement may then stem from the adaption to the coding system and the text distribution, rather than enhanced capacities.\n2. Only one train and test set is used, which come from the same distribution, which further indicates that the performance improvement comes from domain adaption.\n3. There lacks error analysis for baseline models or the abalted models, which may reveal deeper understanding of the model dynamics.\n4. Also, it is not clearly explained how rewards are determined. If rewards are based on naive string match, then the model may be simply memorizing the vocabulary.   Combining the above weaknesses, I’m quite confident that your model is simply memorizing the coding vocabulary and adapting to the MIMIC corpus distribution, which undermines the claim of enhanced medical reasoning capacities.\n\nMinor\n\n1. Dynamic resampling has been proposed in numerous existing studies.  \n2. The construction of SFT data using Qwen2.5-7B is insufficient. The human evaluation is vague and lacks quantitative metrics, failing to validate the data quality.  \n3. The improvement over the previous SOTA (DRG-Llama) is marginal.", "was_revised": true, "final_justification": "I acknowledge the authors' responses, and hence raise my score towards acceptance.", "rebuttal": "We thank the reviewers for their thoughtful and positive feedback. We are encouraged that they found our motivation and idea clear (R2), novel (R1, R2), and important (R3, R4). We are particularly grateful that our experiments were unanimously found to be comprehensive and thorough (R1, R2, R3, R4). The paper was also praised for being well-written and structured (R2, R3), and the code release was noted as a major plus (R3). We are pleased that reviewers recognized that our work provides valuable guidance for resource allocation in training reasoning models for out-of-distribution (OOD) tasks (R1, R3, R4). Below, we address the specific concerns and suggestions raised by the reviewer.\n\n## 1. Question on the exact source of the performance improvement\nWe address this important question as follows:\n* **TL;DR**: Initial SFT (or “adaptation,” as the reviewer puts it) is necessary but NOT sufficient for improving LLM performance on OOD tasks. Reinforcement learning with verifiable rewards (RLVR) further enhances reasoning and generalization beyond what SFT alone provides.\n* We agree with the reviewer that the baseline models indeed lack relevant knowledge about medical coding. This is precisely the main focus of our paper: investigating how RLVR can be effectively applied to OOD tasks. We demonstrate that RLVR alone is insufficient for injecting domain-specific knowledge (Fig 10A); thus, initial SFT (\"domain adaptation,\" as noted by the reviewer) is essential. However, after this warming-up phase, RLVR further enhances performance, indicating that RLVR offers benefits beyond simple domain adaptation.\n* We also agree with the reviewer that the \"domain adaptation,\" such as memorizing DRG vocabulary and adapting to the MIMIC corpus, is important but represents only a partial view of knowledge infusion in our complex task. Medical coding tasks vary in complexity; for instance, ICD coding may straightforwardly match terms like \"bladder infection\" to \"cystitis\" when MEAT criteria (Monitoring, Evaluating, Assessing, and Treating) are satisfied. However, DRG coding is significantly more intricate. A clinical note mentioning \"cystitis\" may or may not correspond to the DRG code \"KIDNEY AND URINARY TRACT INFECTIONS,\" depending on principal diagnosis designation and competing conditions or procedures. Moreover, the severity and complications (e.g., simple infection versus septic shock) influence whether \"cystitis\" is categorized as a CC or MCC. As detailed in lines 42-45, this complexity highlights the importance of reasoning capabilities in our model.\n* Throughout the RLVR process, we clearly observed advancements in the critical reasoning capabilities required for accurate DRG assignment. Specifically, the model learned to identify the correct principal diagnosis and assess whether secondary conditions meet CC/MCC criteria based on clinical complexity (see Sec I for examples). Additional error analysis presented below (Q3) further support this observation. **Crucially, these reasoning capabilities emerged during the RLVR stage, after domain adaptation via SFT was complete, thus reflecting learning beyond mere supervised memorization.** \n* When discussing memorization and overfitting to a data distribution, it is important to distinguish between RL and SFT. Recent work has demonstrated that RL in LLMs can achieve much better generalization than SFT and moves beyond memorization, with findings suggesting that “SFT Memorizes, RL Generalizes” [1]. Additionally, RL-tuned math reasoning models generalize well across diverse domains and tasks [2].\n* Additionally, below we provide robust DRG-Sapphire performance on our internal dataset, extending beyond the MIMIC corpus. Collectively, these findings strongly suggest our model has achieved deeper learning than simply memorizing coding vocabulary and adapting to the MIMIC corpus distribution.\n\nReference:\n1. SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training. \n2. Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning. \n\n## 2. Only one train and test set is used\n* We appreciate the reviewer’s valuable suggestion and, following IRB approval, conducted an additional evaluation using an internal dataset from a healthcare institution. We constructed this dataset by stratified random sampling of 2,500 real-world cases, drawing each DRG code in proportion to its frequency in the MIMIC test set to ensure comparability. DRG-Sapphire achieved an accuracy of 53.6% on this internal dataset, compared to 54.8% on MIMIC, demonstrating its robustness beyond the MIMIC corpus. We also submitted an IRB application for validating and implementing our model in real-world clinical settings at large scale.\n* To the best of our knowledge, MIMIC is the only publicly available dataset containing both clinical notes and DRG labels, and was the sole dataset used in prior work such as DRG-LLaMA. While we acknowledge the limitation of a single source, MIMIC’s training and test sets are large (230k and 28k cases, respectively) and reflect real-world patient distributions in North America, as they span over a decade of ICU and ED admissions at Beth Israel Deaconess Medical Center in Boston, Massachusetts.\n\n## 3. A lack of error analysis\nWe conducted additional qualitative and quantitative error analyses as kindly suggested by the reviewer.\n* We identified 9,537 cases consistently misclassified by all three checkpoints from different training stages: the cold-start SFT (SFT model in Fig. 5A), the small-scale GRPO (GRPO model in Fig. 5A), and the final GRPO-Sapphire model. As shown in the table below, even in these challenging cases, the model improved progressively during training, becoming notably \"less wrong.\"\n\n| Model                  | % Wrong Principal Diagnosis | % Wrong CC/MCC | % Both Wrong |\n|-|-|-|-|\n| Coldstart SFT          | 85.1%                       | 64.2%                  | 47.4%         |\n| Coldstart SFT + GRPO   | 80.4%                       | 63.1%                  | 41.1%         |\n| DRG-Sapphire           | 74.9%                       | 62.7%                  | 33.6%         |\n\n\n* Additionally, a physician expert conducted a detailed manual error analysis on 100 randomly selected cases misclassified by the final GRPO-Sapphire model, applying the same error taxonomy used in DRG-LLaMA.\n\n| Error Category                                        | %     |\n|------------------------------------------------------|-------|\n| Difficulty selecting correct base DRG                | 32.0% |\n| Erroneous CC/MCC                                     | 25.0% |\n| Potential incorrect DRG label                        | 13.0% |\n| Inadequate clinical concept extraction               | 12.0% |\n| Info needed for DRG prediction unavailable           | 10.0% |\n| Other (e.g., procedure instead of medical DRG)       |  8.0% |\n\n* The physician expert also marked 15% of these cases as ambiguous, noting that the correct DRG was unclear based on DRG guidelines, for example due to multiple potential principal diagnoses. **Collectively, these results underscore the inherent complexity of DRG assignment.** Interestingly, the identification of potentially incorrect DRG labels highlights a promising potential use case for our model in detecting medical coding errors.\n\n## 4. Question about reward determination\nWe detailed our rule-based reward modeling in Sec A.2 (lines 460–468). For the accuracy reward, we used exact string matching. This stringent rule-based approach follows the standard practice in recent RLVR work, beginning with DeepSeek-R1. Such minimalism is key to mitigating the well-known issue of reward hacking in RL and forms a cornerstone of RLVR methodology. In early experiments, we explored more relaxed reward regimens (e.g., partial string matching). However, consistent with prior work, we observed frequent reward hacking—for example, the model would generate incorrect but verbose answers to game the reward function through partial matches. Finally, we wish to clarify that while we used string match, we assigned different scores for full versus partial DRG matches (e.g., correct principal diagnosis only or CC/MCC status only), thereby providing denser and more informative reward signals.\n\n## 5. Novelty about dynamic resampling\nWe emphasize that this work is not about introducing yet another SFT/RLVR trick. Rather, our goal is to understand how SFT and RLVR influence LLM learning in OOD tasks. To our knowledge, we are among the first to highlight several practical insights, particularly the training cost analysis in Fig. 7C, which we hope will be valuable to the community.\n\n## 6. Question about quality of SFT dataset constructed by Qwen2.5-7B\nWe are happy to clarify this important detail. The SFT dataset contains two parts: (1) the true DRG label from MIMIC, which is always correct, and (2) the CoT reasoning trace. For the CoT trace, we differentiate between its **factuality** (e.g., whether a condition is CC vs. MCC) and its **cognitive pattern** (the reasoning logic). We do not expect the initial CoT traces to be factually perfect, as the Qwen base model lacks DRG-specific knowledge. Instead, it is crucial that the cognitive pattern is correct. Through prompt engineering (Sec. H), we introduce diverse and valid cognitive patterns for the model to learn during SFT (lines 162–165). This provides a sound structure for the RLVR phase to build upon and learn factuality.\n\n## 7. Performance improvement\nCompared to prior classification-based methods, DRG-Sapphire offers the key advantage of generating high-quality explanations for DRG assignments (Fig 4), while also achieving SOTA performance. In the medical domain, such interpretability is essential for trust and transparency. As emphasized by physician leaders (Sec D.1), these interpretable rationales are foundational for real-world deployment, enabling actionable clinical insights."}, {"review_id": "a697JWvG3x", "reviewer": "Reviewer_A6Eg", "summary": "The authors propose a GRPO based algorithm for training LLMs to predict and reason in domains with out of distribution data compared to their training data. Specifically, they focus on Diagnosis-Related Group (DRG) codes prediction from clinical notes which involves identification of the principal diagnosis, principal procedure, complications or comorbidities, and major complications or comorbidities. They present their trained model called DRG-Sapphire, which is a fine-tuned Qwen2.5-7B-Instruct model. This model achieves SoTA results in DRG code prediction as well as provides a human-interpretable explanation for the DRG code assignment. The authors perform extensive ablations and provide detailed analysis of various algorithmic variants which they considered in developing DRG-Sapphire. The authors also provide insights such as the importance of SFT prior to RL training for new data domains and also a scaling law for RL performance with respect to SF data size.", "questions": "1. Some analysis of evidence on LLMs struggling with DRG coding would be useful. Specifically, what are the kinds of examples where other LLMs fail where DRG-Sapphire does well and vice-versa.\n2. According to the authors, what is the reason for completion length contraction?\n3. The authors mention a “Bitter-lesson” in their conclusion. Is this realization from a single domain example? Can this be attributed just to novelty of data or also to the complexity of the task?\n4. What was the accuracy of the final SFT dataset used? Does SFT dataset produced by a larger (better) model such as Qwn2.5-14B-Instruct improve all downstream RL performance?\n5. Can this process be done iteratively? Generate new SFT data using DRG-Sapphire and continue the process?\n6. Was LoRA tested? Are all observations consistent even when LoRA is used?\n7. Why did the authors choose a pure-online variant of GRPO?", "limitations": "Have been discussed except for impact due to misclassification by DRG-Sapphire. Some discussion regarding that would be useful.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is very well written and the concepts, problem and the solution approach are clearly explained.\n2. The authors provide algorithmic improvements such as a novel dynamic resampling strategy\n3. The authors conduct a thorough exploration and experimentation with various algorithmic approaches presented in literature and present  extensive ablation studies.\n\nWeaknesses:\n1. I did not see experiments using in-context learning examples with large models, which have not been specifically trained with DRG data.\n2. I also could not find any discussion on impact of erroneous DRG code assignment.\n3. Please also see my questions for additional points.", "was_revised": true, "final_justification": "Based on my interaction with the authors and the comments from the other reviewers, I would like to retain my score. All my major concerns have been addressed.", "rebuttal": "We thank the reviewers for their thoughtful and positive feedback. We are encouraged that they found our motivation and idea clear (R2), novel (R1, R2), and important (R3, R4). We are particularly grateful that our experiments were unanimously found to be comprehensive and thorough (R1, R2, R3, R4). The paper was also praised for being well-written and structured (R2, R3), and the code release was noted as a major plus (R3). We are pleased that reviewers recognized that our work provides valuable guidance for resource allocation in training reasoning models for out-of-distribution (OOD) tasks (R1, R3, R4). Below, we address the specific concerns and suggestions raised by the reviewer.\n\n## 1. Question about in-context learning examples\n* Due to the large DRG search space (over 700 distinct codes) and the base model's lack of DRG-specific knowledge (Fig. 1A), we found in-context learning (ICL) examples insufficient for effective knowledge injection. It is not feasible to cover the full code space and their associated clinical notes within a few ICL examples. While increasing the number of ICL examples could improve coverage, it would also lead to a quadratic increase in inference cost.\n* A RAG-based approach for selecting examples would also be problematic for two reasons: (1) Clinical notes with similar semantic features can map to very different DRG codes, making retrieval unreliable. (2) The retrieved examples would constrain the model's output to a small subset of DRG codes, limiting its ability to reason over the entire code space. For these reasons, we chose to apply SFT to infuse domain-specific knowledge before RL.\n\n## 2. Suggestion to add error analysis\nWe conducted additional error analyses as kindly suggested by the reviewer.\n* We identified 9,537 cases consistently misclassified by all three checkpoints from different training stages: the cold-start SFT (SFT model in Fig. 5A), the small-scale GRPO (GRPO model in Fig. 5A), and the final GRPO-Sapphire model. As shown in the table below, even in these challenging cases, the model improved progressively during training, becoming notably \"less wrong.\"\n\n| Model                  | % Wrong Principal Diagnosis | % Wrong CC/MCC | % Both Wrong |\n|-|-|-|-|\n| Coldstart SFT          | 85.1%                       | 64.2%                  | 47.4%         |\n| Coldstart SFT + GRPO   | 80.4%                       | 63.1%                  | 41.1%         |\n| DRG-Sapphire           | 74.9%                       | 62.7%                  | 33.6%         |\n\n* Additionally, a physician expert conducted a detailed manual error analysis on 100 randomly selected cases misclassified by the final GRPO-Sapphire model, applying the same error taxonomy used in DRG-LLaMA.\n\n| Error Category                                        | %     |\n|------------------------------------------------------|-------|\n| Difficulty selecting correct base DRG                | 32.0% |\n| Erroneous CC/MCC                                     | 25.0% |\n| Potential incorrect DRG label                        | 13.0% |\n| Inadequate clinical concept extraction               | 12.0% |\n| Info needed for DRG prediction unavailable           | 10.0% |\n| Other (e.g., procedure instead of medical DRG)       |  8.0% |\n\n* The physician expert also marked 15% of these cases as ambiguous, noting that the correct DRG was unclear based on DRG guidelines, for example due to multiple potential principal diagnoses. Collectively, these results underscore the inherent complexity of DRG assignment. Interestingly, the identification of potentially incorrect DRG labels highlights a promising potential use case for our model in detecting medical coding errors.\n\n* When reviewing cases where other proprietary LLMs fail, we observed that most failures stem from a lack of DRG-specific knowledge, highlighting the OOD nature of the task. For example, these models may output invalid DRG codes or struggle to determine whether a secondary condition qualifies as a CC or MCC.\n\n## 3. Question about impact of erroneous DRG code assignment\nWe thank the reviewer for raising this important point. After consulting with physician leaders, we agree that the impact of errors is highly use case-dependent. For high-stakes applications like automated billing, the error tolerance must be extremely low due to potential financial consequences. In contrast, for operational or educational tools—such as the one in Appendix D.1 that helps physicians understand DRG assignments—a higher degree of fault tolerance may be acceptable. To rigorously assess these factors in practice, we have submitted an IRB application to prospectively validate the model's performance and impact within a healthcare system. This will enable a formal evaluation from a healthcare delivery perspective.\n\n## 4. Question about the reason of completion length contraction\nWe hypothesize that the completion length contracts because DRG assignment is a highly structured reasoning task, allowing the model to learn a more concise and efficient solution path. Due to word limits, we refer the reviewer to our response to RJgS (Q1) for a full analysis with supporting references.\n\n## 5. Question about cross-domain generalization of the “Bitter-lesson”\nOur conclusion is grounded in comprehensive experiments on the OOD task of DRG assignment, and we suggest these findings are likely to generalize to other OOD tasks where the model has minimal pre-training exposure. We provide a detailed discussion with supporting literature in our response to Reviewer RJgS (Q3).\n\n## 6. Performance when training on SFT dataset produced by 14B model\nThis is a great thought, and we have conducted additional experiments as the reviewer suggested.\n\n| Setting             | Experiment | 7B    | 14B   |\n|-|-|-|-|\n| 5% SFT, 95% RL      | SFT        | 25.0% | 21.8% |\n|                     | +GRPO      | 38.7% | 34.6% |\n| 75% SFT, 25% RL     | SFT        | 38.5% | 35.7% |\n|                     | +GRPO      | 47.4% | 43.8% |\n\nInterestingly, RL training based on SFT data produced by the 14B model did not improve performance. We suspect this may be due to a teacher–student capacity gap: the smaller 7B student model struggled to imitate the complex style and output distribution of the larger model, resulting in suboptimal policy initialization that hindered RL optimization.\n\n## 7. Question about SFT dataset\nWe are happy to clarify this important detail. The SFT dataset contains two parts: (1) the true DRG label from MIMIC, which is always correct, and (2) the CoT reasoning trace. For the CoT trace, we differentiate between its **factuality** (e.g., whether a condition is CC vs. MCC) and its **cognitive pattern** (the reasoning logic). We do not expect the initial CoT traces to be factually perfect, as the Qwen base model lacks DRG-specific knowledge. Instead, it is crucial that the cognitive pattern is correct. Through prompt engineering (Sec. H), we introduce diverse and valid cognitive patterns for the model to learn during SFT (lines 162–165). This provides a sound structure for the RL phase to build upon and learn factuality.\n\n## 8. Question about iterative training\nWe agree that training can be done iteratively, either via repeated SFT/RL cycles as the reviewer suggests or through multiple policy updates per generation. However, there are two key considerations for such a process:(1) RL training risks entropy collapse as training progresses [1], necessitating careful entropy management to ensure training success [2, 3]. (2) If the SFT dataset from the latest checkpoint does not introduce new knowledge, iterative learning may be ineffective. A recent study proposes a compelling approach, introducing partial solutions during iterative training to simplify problem difficulty and reshape the learnability landscape [4]. This increases the likelihood of discovering correct trajectories, leading to improved RL training outcomes.\n\nReference:\n1. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models.\n2. Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning. \n2. Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs. \n3. QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation. \n\n## 9. Question about LoRA\nWe did not test LoRA because the primary bottleneck in our RL work, as in many RL with verifiable rewards studies, was the computational cost (inference time) of generating rollouts, not GPU memory. Since memory was not the constraint, we opted for full-parameter training to obtain a cleaner evaluation of our method, thereby avoiding the complexity and potential confounding variables of LoRA-specific hyperparameters (e.g., rank selection).\n\n## 10. Question about pure-online GRPO\nWe chose the pure-online variant of GRPO for several reasons.\n* Our approach aligns with the original GRPO implementation from DeepSeek-Math [1] which is pure-online.\n* This decision is supported by growing evidence that on-policy RL can yield better performance than off-policy settings [2, 3]. Specifically, off-policy learning has been shown to accelerate entropy collapse [2]. Furthermore, recent work by Hao et al. (2025) identifies potential overfitting issues associated with off-policy learning [3].\n* From an experimental design perspective, this choice reduces the number of hyperparameters to tune, simplifying the setup and leading to a cleaner analysis.\n\nReference:\n1. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. \n2. Skywork Open Reasoner 1 Technical Report. \n3. On-Policy RL with Optimal Reward Baseline."}, {"review_id": "HLVaC6PMkM", "reviewer": "Reviewer_4M3r", "summary": "This paper explores the application of large-scale reinforcement learning (RL) for out-of-distribution (OOD) reasoning in large language models (LLMs), focusing on automated Diagnosis-Related Group (DRG) coding—a critical task for hospital operations and reimbursement. The authors introduce DRG-SAPPHIRE, built on Qwen2.5-7B, which is fine-tuned using supervised fine-tuning (SFT) followed by RL with Group Relative Policy Optimization (GRPO) and rule-based rewards. The study provides an extensive empirical evaluation of GRPO variants, adaptive learning strategies (e.g., curriculum and staged learning), and reward shaping techniques. Results demonstrate state-of-the-art (SOTA) accuracy on the MIMIC-IV dataset and show that DRG-SAPPHIRE generates physician-validated reasoning, improving transparency and explainability. A key finding is that RL performance scales approximately linearly with the logarithm of SFT sample size, reinforcing that strong base model knowledge is essential for success in OOD domains.", "questions": "See my comments above.", "limitations": "The manuscript would benefit from a dedicated limitations discussion (e.g., on generalizability beyond MIMIC-IV, reliance on high-quality SFT data, scalability of training cost).", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "> Strengths\n\n- Well written and structured: The paper is clear, professional, and accessible despite the technical complexity of the topic.\n- Important and timely problem: The focus on DRG coding — a high-impact, real-world OOD task in healthcare — is highly relevant and valuable.\n- Comprehensive evaluation: The authors perform rigorous analyses of GRPO, including ablation studies, enhancements (e.g., dynamic resampling, KL decay), and data allocation strategies between SFT and RL.\n- Informative insights: The investigation into RL’s limitations and failure modes provides a useful reference for the community.\n- Did a great job on sharing their codes. \n\n> Weakness\n- While the empirical exploration is thorough, the main insight — that strong RL performance depends on prior knowledge infusion — is expected and aligns with existing understanding (i.e., the “bitter lesson” that scaling data or pretraining matters more than algorithmic tweaks).\n- Lack of coverage of related recent work: I personally love the idea of rule-based reward as it will be a huge cost saving but I also like to see you have more discussion on other people's work on rule-based rewards. For instance: \n   Mu, Tong, et al. \"Rule based rewards for language model safety.\" arXiv preprint arXiv:2411.01111 (2024).\n   Sun, Shenghuan, et al. \"Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Grounding.\" arXiv preprint arXiv:2405.19567 (2024).\n- Missing limitations section: The manuscript would benefit from a dedicated limitations discussion (e.g., on generalizability beyond MIMIC-IV, reliance on high-quality SFT data, scalability of training cost).", "was_revised": true, "final_justification": "Again, I think this is a solid paper. I appreciate that the authors made significant efforts to support their claims. I recommend accepting this manuscript.", "rebuttal": "We thank the reviewers for their thoughtful and positive feedback. We are encouraged that they found our motivation and idea clear (R2), novel (R1, R2), and important (R3, R4). We are particularly grateful that our experiments were unanimously found to be comprehensive and thorough (R1, R2, R3, R4). The paper was also praised for being well-written and structured (R2, R3), and the code release was noted as a major plus (R3). We are pleased that reviewers recognized that our work provides valuable guidance for resource allocation in training reasoning models for out-of-distribution tasks (R1, R3, R4). Below, we address the specific concerns and suggestions raised by the reviewer.\n## 1. Question about the insight relates to the “Bitter lesson”\n* Thank you for your insightful feedback. While our conclusion aligns with Sutton’s \"bitter lesson,\" we hope our work offers a timely and empirically grounded clarification for the domain of complex, out-of-distribution (OOD) tasks.\n* Following successes of reasoning models like DeepSeek-R1, a prevailing narrative has been to \"scale RL,\" **leaving a critical question unanswered: what, precisely, should be scaled?** Our work addresses this for complex, OOD tasks where the answer is not straightforward. We demonstrate that naively scaling RL (in terms of data and compute) on a base model fails entirely for DRG coding, as vanilla models were unable to produce correct outputs with RL alone.\n* Through rigorous experimentation, we conclude that for OOD challenges, knowledge infusion is the critical component to scale. Our findings suggest that scaling SFT is often more effective and computationally efficient than scaling RL alone. In this light, our work encourages the community to more carefully consider what is being scaled when applying RL. We hope this clarification serves as a meaningful and timely contribution, offering an evidence-based explanation and a clear path forward for the many practitioners working to apply RL to domain-specific OOD tasks.\n\n## 2. Additional literatures\nThank you for suggesting these insightful references. We would be happy to include both papers as related work in the camera-ready version. The first paper from OpenAI presents an effective approach that uses fine-grained, composable rules and an LLM grader to create a hybrid reward signal, balancing model safety and usefulness [1]. We will also highlight the novel framework in the second paper, which employs symbolic representations of clinical reasoning to ground vision-language models (VLMs) in medical knowledge and uses a symbolic reward function to evaluate VLM responses for individual correctness and clinical validity [2].\n\nReference:\n1. Rule based rewards for language model safety. \n2. Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Grounding.\n\n## 3. Missing limitations section\n* We thank the reviewer for the helpful suggestions and wish to clarify that a limitations section is provided in Appendix Sec E. In the camera-ready version, we will expand this section to incorporate your valuable points on scalability and SFT data.\n* Of note, following another reviewer’s kind suggestion and with IRB approval, we conducted an additional evaluation using an internal dataset of 2,500 real-world cases from a healthcare institution. DRG-Sapphire achieved an accuracy of 53.6% on this internal dataset, compared to 54.8% on MIMIC, demonstrating its robustness beyond the MIMIC corpus. We also submitted an IRB application for validating and implementing our model in real-world clinical settings at large scale."}, {"review_id": "PrXSRPQwCM", "reviewer": "Reviewer_RJgS", "summary": "This paper introduces DRG-SAPPHIRE, a model for automated DRG coding from clinical notes using large-scale reinforcement learning. Built on Qwen2.5-7B and trained with GRPO, the model addresses the out-of-distribution nature of DRG coding, where LLMs have limited exposure to private clinical data. The authors systematically investigate the relationship between SFT and RL performance, finding that RL effectiveness scales logarithmically with SFT examples. DRG-SAPPHIRE achieves SOTA on MIMIC-IV and generates physician-validated reasoning for code assignments. The study reveals domain-specific challenges in applying RL to medical tasks, including preference for Answer-First cognitive patterns and sensitivity to KL divergence.", "questions": "Please see the questions mentioned in the weaknesses.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "# Strenghs\n1. The proposed method achieves new state-of-the-art performance on the MIMIC-IV benchmark, significantly outperforming proprietary reasoning models including GPT-4o.\n2. The finding that RL performance scales logarithmically with SFT sample size, and that stronger SFT foundations lead to better post-GRPO results, provides valuable guidance for resource allocation in training reasoning models for OOD tasks.\n3. The ablation study systematically examines multiple GRPO variants, reward shaping strategies, cognitive behavior interventions, and adaptive learning approaches, providing thorough insights into RL training for medical coding tasks.\n4. The paper identifies unique challenges in medical domains that contrast with mathematical reasoning tasks, such as completion length contraction during training and the superiority of Answer-First over CoT-First patterns, contributing to research in this community.\n# Weaknesses\n\n1. Several interesting findings lack deeper investigation or further experiments to validate them. For instance, the reason of completion length contraction phenomenon, the superiority of Answer-First reasoning despite conventional wisdom favoring CoT-First approaches, and the hypothesis that denser rewards lead to premature convergence to local optima are presented without sufficient mechanistic explanation or validation. \n\n2. The paper contains formatting inconsistencies, such as \"Kl Decay\" (lowercase 'l') in Table 1.\n\n3. The central claim that RL effectiveness is \"fundamentally constrained by the domain knowledge encoded in the base model\" and that \"scaling SFT may be more effective and computationally efficient than scaling RL alone\" is based solely on DRG coding experiments. This conclusion requires validation across multiple medical tasks and datasets to establish its broader applicability to OOD reasoning tasks.", "was_revised": true, "final_justification": "The authors have successfully addressed all three concerns raised in my initial review. First, they provided a compelling justification for prioritizing a large-scale empirical study over deeper mechanistic investigations of individual phenomena. Their explanation clarified that the observed findings about completion length contraction and answer-first patterns align with recent work in other domains. Second, the formatting inconsistency has been corrected. Third, while the study focuses on a single task, the authors effectively contextualized their findings within broader research that supports their conclusions about SFT-RL trade-offs.\n\nSince there are no major problems that need to be addressed, I continue to suggest weak acceptance. Because of the limited task and scope that this paper's methodology may be applicable to, as well as the absence of thorough mechanistic investigations, I do not raise my score.", "rebuttal": "We thank the reviewers for their thoughtful and positive feedback. We are encouraged that they found our motivation and idea clear (R2), novel (R1, R2), and important (R3, R4). We are particularly grateful that our experiments were unanimously found to be comprehensive and thorough (R1, R2, R3, R4). The paper was also praised for being well-written and structured (R2, R3), and the code release was noted as a major plus (R3). We are pleased that reviewers recognized that our work provides valuable guidance for resource allocation in training reasoning models for out-of-distribution (OOD) tasks (R1, R3, R4). Below, we address the specific concerns and suggestions raised by the reviewer.\n\n## 1. Mechanistic explanation or validation on various findings from our study\nWe thank the reviewer for highlighting these insightful observations and agree that a deeper mechanistic investigation into each would be highly valuable. In this work, we aimed to strike a balance between the depth and breadth of our exploration. Our primary objective was to conduct a large-scale empirical study on the optimal resource allocation between SFT and RL for a challenging OOD task, a focus which required comprehensive ablations and experiments that consumed the majority of our computational resources. This necessarily limited the scope for dedicated studies into all of the secondary phenomena we observed. Consequently, we present findings—such as the completion length contraction, the surprising efficacy of the 'Answer-First' pattern, and the hypothesis that denser rewards lead to local optima—as important, empirically-grounded early observations. While a full mechanistic validation of each was beyond the scope of this paper and would deserve a dedicated study, we believe they are valuable contributions in their own right, revealing unique dynamics of RL in this domain and providing a strong foundation for future exploration.\n\n\nNevertheless, we provide a brief discussion on some of the findings as below, in combination with some very recent work from general domains that share similar observations.\n* **Length Contraction**: For DRG assignment, the reasoning process is structurally straightforward: identify the principal diagnosis → extract relevant secondary diagnoses → assess CC/MCC status → assign the DRG (Sec. 3.1). While domain knowledge is nontrivial, the sequence of steps is generally fixed, unlike mathematical problems where solution strategies vary considerably. Hence, it is intuitive that a well-trained LLM may converge toward shorter, more structured outputs. Indeed, Cheng et al. (2025) found that response length increases correlate with performance gains only in certain domains; notably, length contraction is observed in Code, Logic, and Tabular tasks [1]. Moreover, another study shows that RL-trained reasoning models can learn to bypass verbose CoT by internalizing the reasoning process while still producing correct answers [2].\n* **Preference for Answer-First**: Our observation that answer-first prompting yields superior performance likely reflects how LLMs encode and retrieve latent knowledge. These findings align with recent studies, which suggest that CoT and extended reasoning may not always be necessary for reasoning models, and a “no-thinking” pattern can sometimes yield better performance [3, 4]. In addition, presenting the answer early may help the model condition its explanation generation on a confident anchor, avoiding hallucinated or self-justifying CoT paths. This hypothesis aligns with the work showing that answer-first prompting can expose inconsistencies in reasoning and even outperform conventional CoT-first strategies in certain settings [5].\n\nReference:\n1. Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective. \n2. Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models. \n3. Reasoning models can be effective without thinking. \n4. Reasoning models don’t always say what they think. \n5. Order Matters in Hallucination: Reasoning Order as Benchmark and Reflexive Prompting for Large-Language-Models.\n\n## 2. Formatting inconsistency\nThank you for your sharp eye and for pointing this out. We will fix the \"Kl Decay\" typo and ensure all formatting is consistent throughout the final camera-ready version.\n\n## 3. Necessity of cross-domain validation\nWe thank the reviewer for the valuable feedback on generalizability, and we agree that broader validation is an important next step. Our study was intentionally designed as a deep, empirical investigation into a single, highly complex OOD task—DRG coding—to enable a rigorous analysis that would be difficult to replicate with the same depth across multiple domains. This focused approach allowed for the extensive experiments on SFT-RL data trade-offs (Sec 5.2), detailed ablations (Sec 5.3), and investigation of RL prerequisites (Sec 5.4) that led to our central finding: a clear, log-linear scaling relationship between the volume of SFT data and subsequent RL performance. While based on one task, this result provides strong, empirical evidence for our hypothesis that for complex OOD tasks, RL effectiveness is fundamentally constrained by the domain knowledge infused via SFT. We therefore position our claim as an important finding that provides strong motivation for future work to test this hypothesis across other domains, as noted in our limitations (Sec E).\n\nBelow, we provide a brief discussion on some very recent work from other tasks and domains that resonates with our conclusion:\n* **No New \"Knowledge\" from RL?**: An ongoing and pertinent debate is whether RL truly incentivizes new reasoning capabilities beyond those already encoded in the base model. Yue et al. (2025) conducted a rigorous analysis across **mathematical, coding, and visual reasoning benchmarks**, concluding that RL does not induce novel reasoning patterns beyond those already present in the pretrained model [1].\n* **Hybrid SFT and RL Training is Needed**: Lu et al. (2025) analyzed training dynamics on **complex reasoning tasks**, observing that RL excels at maintaining and improving performance within a model’s existing capabilities, whereas SFT is more effective for making progress on questions beyond the model’s current scope [2].\n* **Medical Domain Evidence**: Zuo et al. (2025), analyzing Qwen-7B models across **medical question-answering** and mathematical tasks, noted that medical tasks demand much richer factual/domain knowledge compared to mathematical tasks, which emphasize logical reasoning [3]. They concluded that in medical domains, SFT remains important due to the necessity of detailed domain knowledge, while RL enhances reasoning accuracy by pruning inaccurate or irrelevant reasoning paths.\n\nIn summary, our core conclusion is reinforced by these very recent cross-domain studies. **While these papers provide corroborating evidence, our work is distinguished by its systematic and comprehensive investigation of the SFT-RL trade-off, offering a granular, quantitative analysis of this critical relationship.**\n\nReference:\n1. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? \n2. Learning What Reinforcement Learning Can’t: Interleaved Online Fine-Tuning for Hardest Questions.\n3. Knowledge or Reasoning? A Close Look at How LLMs Think Across Domains."}]

Author rebuttal:

Meta-review: The paper had mixed results with two suggesting a borderline accept and two suggesting accept. The reviewers agree that the paper is well motivated and that the experiments are convincing. I am happy to see a paper that successfully uses RL in the medical domain and recommend acceptance.
Decision: Accept (spotlight)