Why Chain of Thought Fails in Clinical Text Understanding

## Abstract

Abstract Large language models (LLMs) are increasingly being applied to clinical care, a domain where both accuracy and transparent reasoning are critical for safe and trustworthy deployment. Chain-of-thought (CoT) prompting, which elicits step-by-step reasoning, has demonstrated improvements in performance and interpretability across a wide range of tasks. However, its effectiveness in clinical contexts remains largely unexplored, particularly in the context of electronic health records (EHRs), the primary source of clinical documentation, which are often lengthy, fragmented, and noisy. In this work, we present the first large-scale systematic study of CoT for clinical text understanding. We assess 95 advanced LLMs on 87 real-world clinical text tasks, covering 9 languages and 8 task types. Contrary to prior findings in other domains, we observe that 86.3% of models suffer consistent performance degradation in the CoT setting. More capable models remain relatively robust, while weaker ones suffer substantial declines. To better characterize these effects, we perform fine-grained analyses of reasoning length, medical concept alignment, and error profiles, leveraging both LLM-as-a-judge evaluation and clinical expert evaluation. Our results uncover systematic patterns in when and why CoT fails in clinical contexts, which highlight a critical paradox: CoT enhances interpretability but may undermine reliability in clinical text tasks. This work provides an empirical basis for clinical reasoning strategies of LLMs, highlighting the need for transparent and trustworthy approaches.

## 1 Introduction

Large language models (LLMs) are increasingly being explored for clinical workflows, including clinical decision support, documentation drafting, information extraction, and telemedicine communication (Park et al., 2024; Vrdoljak et al., 2025). However, deploying LLMs in healthcare, an inherently safety-critical setting, requires both high accuracy and clear interpretability (Habli et al., 2020; Ennab & Mcheick, 2024). Providing explicit and transparent reasoning allows clinicians to critically evaluate LLM decision-making processes, identify gaps or misapplications of evidence, and make more informed, trustworthy decisions in patient care (Amann et al., 2020). Therefore, reliability and transparency are crucial for developing trustworthy clinical AI tools and their integration into clinical practice (Kim et al., 2025).

Chain-of-thought (CoT) prompting (Wei et al., 2022), which encourages language models to generate intermediate reasoning steps prior to producing a final answer, has become a widely adopted technique for exposing intermediate rationales.
Using explicit instruction, e.g., “Let’s think step by step”, CoT prompting can unlock the latent reasoning capabilities of LLMs, substantially improving performance and perceived interpretability across various domains. This effect is especially pronounced for tasks requiring advanced reasoning and logical coherence, such as mathematical problem-solving and coding (Kojima et al., 2022).

However, the effectiveness of CoT in clinical contexts remains underexplored and poorly understood. Existing evidence largely stems from exam-style or open-domain settings (e.g., USMLE-style QA) (Wu et al., 2024a; Liévin et al., 2024) rather than real-world, multi-source clinical narratives that demand heavy factual grounding and strict differential reasoning. The clinical text in electronic health records (EHRs) is noisy, ungrammatical, and highly variable, creating additional challenges for understanding. Moreover, previous studies have revealed that LLM outputs can contain hallucinations and incorrect information, potentially introducing patient-safety risks (Asgari et al., 2025; Roustan et al., 2025).
Recent analyses further caution that CoT explanations may be unfaithful, representing superficially plausible narratives that do not reflect the model’s actual decision basis, inviting over-trust if used as a reliability signal (Lyu et al., 2023). These open questions motivate a dedicated, large-scale assessment of CoT clinical text understanding.

In this study, we present the first large-scale investigation on the effectiveness of CoT prompting in LLMs in the understanding of clinical text.
We evaluate 95 contemporary LLMs, including proprietary, open-source, and medical-domain variants, across 87 real-world clinical tasks spanning nine languages and eight task types, using standardized zero-shot and CoT inference protocols.
Contrary to general-domain findings, we reveal that CoT consistently reduces performance despite increasing the apparent transparency of model outputs.
To further investigate when and why CoT fails, we conduct four complementary analyses:
(i) relating accuracy to reasoning trace length;
(ii) quantifying clinical concept alignment between inputs and CoT traces;
(iii) comparing lexical characteristics of CoT traces for correct vs. incorrect predictions; and
(iv) auditing failure modes via an LLM-as-a-Judge with expert validation.

Overall, the main contributions of this work are:

- •
A comprehensive head-to-head comparison of CoT vs. zero-shot across 95 LLMs and 87 multilingual clinical tasks, covering various clinical NLP applications under a unified evaluation protocol.
- •
Empirical evidence demonstrating that CoT systematically undermines accuracy in clinical text understanding, with the larger drops for models of lower capability.
- •
Mechanistic analyses linking CoT failures to longer reasoning chains and weaker clinical-concept grounding, along with characterization of lexical signatures associated with incorrect answers (e.g., clinical abbreviations and numeric mentions).
- •
Development of an error taxonomy (hallucination, omission, incompleteness) evaluated via an LLM-as-a-Judge framework with clinician review, offering actionable guidance for safer use of CoT in the clinical context.

## 2 Related Work

### 2.1 Large Language Models and Chain-of-Thought

The large-scale pre-training and extensive fine-tuning equip LLM with rich world knowledge and logical reasoning capabilities, enabling them to generalize to broad tasks (Raiaan et al., 2024). Wei et al. (2022) demonstrated that CoT prompting can guide models to generate intermediate reasoning steps and significantly improve performance on tasks spanning arithmetic, commonsense, and symbolic reasoning.
Then, a simple “Let’s think step by step” instruction can elicit zero-shot reasoning Kojima et al. (2022), and self-consistency Wang et al. (2023) further boosts performance by aggregating diverse reasoning paths. These techniques have become a common practice for providing detailed explanations and improving performance.
Subsequently, more advanced frameworks are built on CoT: ReAct interleaves reasoning with tool use (Yao et al., 2023b); Tree-of-Thoughts (Yao et al., 2023a) and Graph-of-Thoughts Besta et al. (2024) integrate structured planning and search with CoT; Least-to-Most (Zhou et al., 2023) decomposes complex problems into solvable subproblems.
Furthermore, reinforcement-learning-based reasoning models (e.g., OpenAI o1 (Jaech et al., 2024) and DeepSeek-R1 (Guo et al.,)) report additional gains by training models to “think before answering,” though these advances are demonstrated mainly on math, coding, or synthetic reasoning benchmarks rather than safety-critical clinical text.

### 2.2 Medical Reasoning

The clinical applications of LLMs have rapidly emerged. Notably, Med-PaLM-2 (Singhal et al., 2025), integrating base-model improvements with prompting/ensemble strategies inspired by CoT, reports expert-level performance across several medical examinations.
Meanwhile, In-Context Padding (Wu et al., 2024b) infers important medical entities from the input and uses them to steer the model’s reasoning trajectory, improving alignment with clinical decision pathways. CLINICR (Nachane et al., 2024) mirrors the prospective process of incremental reasoning to support differential diagnosis. These methods aim to make intermediate steps clinically grounded rather than purely free-form rationales.
However, Wu et al. (2024a) and Liévin et al. (2024) also revealed that CoT does not consistently improve the performance of general LLMs on medical questions.
But there is still not a systematic, large-scale evaluation of CoT on real-world clinical text tasks beyond exam-style QA.
Furthermore, safety-oriented assessments in clinical documentation and summarization report hallucinations (Shah, 2024), such as fabricated facts, incorrect dosages, misattributed conditions, and instruction-following failures with potential patient-safety implications, underscoring the need for domain-specific evaluation protocols that go beyond exam-style QA and measure factual grounding in real-world clinical text (Asgari et al., 2025).

## 3 Methodology

### 3.1 Task Formulation

Let $\mathcal{T}$ denote a collection of clinical text tasks.
For each task $t\in\mathcal{T}$, we consider a dataset
$\mathcal{D}_{t}=\{(x_{i},y_{i})\}_{i=1}^{N_{t}}$,
where $x_{i}$ is the clinical text that will serve as the model input, such as a clinical note or patient query, and $y_{i}$ is the task-specific reference output.
Each task is paired with a detailed task instruction $p_{t}$ that specifies the clinical application scenario, the task requirement, and the expected output.

Given the concatenation of task instruction and input, an LLM $M$ is instructed to produce an output $\hat{y}_{i}$ according to the prompting protocol.

Specially, we investigate the model’s capabilities under two prompting strategies:

- •
Zero-shot:
The zero-shot prompt $p^{\text{zs}}_{t}$ is a concise, task-specific instruction that asks the model to return only the final answer in the required format, without any explanation or auxiliary text. Formally, in the *zero-shot* setting, the model is instructed to output only the final answer for a given input:
$\hat{y}^{\text{zs}}_{i}\;=\;M\!\big([\,p^{\text{zs}}_{t}\,\|\,x_{i}\,]\big).$
No intermediate reasoning is requested or scored in this mode.
- •
Chain-of-thought prompting:
Similar to zero-shot, the CoT prompt $p^{\text{cot}}_{t}$ includes the same task instruction and input, but adds an explicit request for step-by-step reasoning before the final answer. Formally, in the *chain-of-thought* (CoT) setting, the model is instructed to produce a stepwise reasoning trace before providing the final answer.
$(r_{i},a_{i})\;=\;M\!\big([\,p^{\text{cot}}_{t}\,\|\,x_{i}\,]\big),\qquad\hat{y}^{\text{cot}}_{i}\;=\;a_{i}.$
where $r_{i}$ is the free-form reasoning analysis generated by the model, and $a_{i}$ is the final answer; only $a_{i}$ is used for answer extraction and scoring.

Figure: Figure 2: Evaluating CoT in clinical text understanding and probing why it fails.
Refer to caption: x1.png

### 3.2 Clinical Text Datasets

We include $87$ clinical text datasets from the BRIDGE benchmark (Wu et al., 2025).
All tasks originate from real-world clinician–patient interactions drawn from electronic health records, clinical case reports, and online consultation records, with appropriate de-identification and license compliance.

The collection spans nine languages: English, Chinese, Spanish, Japanese, German, Russian, French, Norwegian, and Portuguese.
They cover 8 clinical NLP tasks to support a broad scope of clinical applications, including text classification (i.e., disease diagnosis), Named Entity Recognition (NER) and event extraction (i.e., information extraction), and Question-Answering (QA).
This breadth of languages, task types, and authentic clinical settings captures the diverse reasoning demands of practice and provides a rigorous substrate for analyzing the capabilities and limitations of CoT reasoning in clinical contexts.

### 3.3 Models and Evaluation Protocol

We investigate $95$ widely used LLMs covering proprietary systems (e.g., Google-Gemini, Openai-GPT-4o), open-source families (e.g., Deepseek, Qwen, LLaMA, Mistral), and domain-specialized medical models (e.g., Me-LLaMA, Meditron), with parameter scales from $\sim$1B to 671B. The complete model list is provided in the Appendix Table LABEL:tab:all_model_results.

We compare performance under the two prompting strategies introduced in Sec.[3.1](https://arxiv.org/html/2509.21933v1#S3.SS1).
For each task, we leverage the task-specific scripts for label extraction and scoring provided by datasets, ensuring consistency and comparability across tasks and models.

- •
For text classification, semantic similarity, natural language inference, and document-level normalization and coding, the primary metric is accuracy.
- •
For NER, event extraction, and entity-level normalization and coding, we use event-level $F_{1}$ score, which requires the model to recover the target entity or event and all its associated attributes, measuring precision and recall jointly over structures.
- •
For QA and summarization, we report ROUGE-average, the micro average of ROUGE-1, ROUGE-2, and ROUGE-L, to capture overlap at token and sequence levels between model response and reference.

Formally, each task $t$ specifies a deterministic scoring function $S_{t}(\hat{y},y)\in[0,1]$.
Given predictions under zero-shot and CoT for the same instances, we quantify the CoT effect for a task–model pair by:

$$ $\Delta Score\;=\;S_{t}^{\text{cot}}-S_{t}^{\text{zs}},\qquad\Delta Score(\%)\;=\;\frac{S_{t}^{\text{cot}}-S_{t}^{\text{zs}}}{S_{t}^{\text{zs}}},$ $$

For each model $M$, the overall score is the macro-average of task-level scores across $\mathcal{T}$, yielding $\overline{S}^{\text{zs}}_{M}$ and $\overline{S}^{\text{cot}}_{M}$, and the corresponding aggregate effects $\overline{\Delta Score}_{M}$ and $\overline{\Delta Score(\%)}_{M}$.

### 3.4 Implementation Details

#### Prompt design.

To examine LLMs under broadly adopted settings, the CoT prompt follows the classical CoT instruction (Wei et al., 2022; Wang et al., 2023; Singhal et al., 2025), instantiated as brief directives like “Solve it in a step by step fashion” and then state the final answer explicitly. We do not employ any prompt optimization to avoid introducing confounding factors that could obscure the core CoT effect.

#### Model access and inference setting.

Details can be found in Appendix [A.1](https://arxiv.org/html/2509.21933v1#A1.SS1).

## 4 Experiment and Results

To assess the practical impact of CoT in clinical contexts, we first investigate *what* the overall impact of CoT is on performance.
Then, we analyze *how* CoT behaves by examining the relationship between reasoning length and performance as well as the alignment of generated content with medical concepts.
Finally, we investigate *why* CoT fails via an error taxonomy supported by LLM-as-a-Judge assessments and clinical expert validation.

### 4.1 CoT Undermines LLM Performance in Clinical Text Tasks

#### Setup.

For each model $M$, we compute $\overline{\Delta Score}_{M}$ and $\overline{\Delta Score(\%)}_{M}$ to summarize the overall CoT effect across tasks (See [3.3](https://arxiv.org/html/2509.21933v1#S3.SS3)).
We report macro-averages at the model level and further stratify by model capability (quartiles based on $\overline{S}^{\text{zs}}_{m}$).

Figure: Figure 3: CoT prompting shows diminishing negative impact as model capability increases.
Refer to caption: x2.png

#### Main result and findings.

As shown in Figure [1](https://arxiv.org/html/2509.21933v1#S0.F1), most LLMs ($82$ of $95$, $86.3\%$) exhibit a performance reduction under CoT prompting.
Full model-wise scores are provided in Appendix Table LABEL:tab:all_model_results.
Specially, even the best-performing LLMs decline under CoT: Gemini-2.5-flash ($-1.6,-3.5\%$), Deepseek-R1 ($-2.1,-4.9\%$), and GPT-4o ($-3.5,-8.0\%$).
Among the $13$ models with improvements, gains are small ($0.1$–$2.2$ points) and $10/13$ are below $1$ point, indicating marginal improvements.

To verify the robustness of this trend, we hand-crafted four semantically equivalent prompt pairs for each task ($p^{\text{zs}}$ and $p^{\text{cot}}$) and re-ran evaluations on three widely used open-source model families, Llama-3.3-70B-Instruct, Mistral-Large-Instruct-2411, Qwen2.5-72B-Instruct and one high-performing variant (Athene-V2-Chat).
Across all variants, the direction of the effect remained consistent, and the magnitudes are were comparable (See Appendix Table [S3](https://arxiv.org/html/2509.21933v1#A1.T3)), supporting the robustness of this global finding.

Grouping models by zero-shot capability reveals a graded pattern: higher-capability models are relatively more robust under CoT, whereas lower-capability models suffer larger drops. The linear relationship ($R^{2}=0.973$) implies a systematic interaction between model capability and the effectiveness of reasoning strategies. This stratification suggests that generating reliable reasoning imposes higher competence demands on the model.

### 4.2 Longer Reasoning Is Associated with Higher Performance Decline

Figure: Figure 4: Performance under CoT prompting decreases as reasoning length increases.
Refer to caption: x3.png

#### Setup.

We extract the *reasoning trace* $r_{i}$ (the text before the final answer) using the output format required by $p^{\text{cot}}_{t}$.
The reasoning length $L(r_{i})$ is measured as the word count of $r_{i}$; word-level counts avoid tokenizer-specific artifacts.
Within each model–task pair, instances are sorted by $L(r_{i})$ and partitioned into ten equal-sized bins (deciles).
For each decile $d$, we compute the task score under CoT, $S_{t}^{\text{cot}}(d)$, and—on the *same items*—the zero-shot score, $S_{t}^{\text{zs}}(d)$.
We then summarize the length-conditioned relative change

$$ $\Delta Score(d)(\%)\;=\;\frac{S_{t}^{\text{cot}}(d)-S_{t}^{\text{zs}}(d)}{S_{t}^{\text{zs}}(d)}.$ $$

Pairing by instance controls for difficulty: each decile compares CoT and zero-shot on identical inputs.
We report results for four models—GPT-4o, Qwen2.5-72B-Instruct, Llama-3.3-70B-Instruct, and MeLLaMA-70B-chat, which are representative proprietary and open-source model families, and one medical variant.

#### Main results and findings.

Figure [4](https://arxiv.org/html/2509.21933v1#S4.F4) shows a clear negative association between reasoning length and performance for all four models.
Both CoT and zero-shot scores decrease as $L(r)$ increases, reflecting that longer traces often correspond to harder instances.
However, the *relative* CoT drop grows with length: $\Delta\%(d)$ becomes increasingly negative from short to long deciles.
For example, GPT-4o exhibits a widening decline of roughly $5$–$15$ percentage points across deciles, while MeLLaMA-70B-chat ranges from about $5$ to $22$ points.
Zero-shot scores computed on the identical items are comparatively flat or decline much less, indicating that the amplified degradation is not merely a consequence of instance difficulty.

Figure: Figure 5: Impact of medical concept alignment on performance. Both Overlap and Coverage alignment metrics show systematic performance improvements across percentiles.
Refer to caption: x4.png

### 4.3 Medical Concept Alignment Mitigates CoT Degradation

#### Setup.

To examine how clinically grounded content relates to CoT reliability, we extract medical concepts from both the input $x_{i}$ and the CoT reasoning trace $r_{i}$ using cTAKES (Savova et al., 2010) (version 4.0.0.1), a clinical text analysis and knowledge extraction system that extracts clinical entities in free text and maps them into Concept Unique Identifiers (CUIs).
For each instance, we define the set of input concepts $C_{x}$ and the set of concepts mentioned in the reasoning trace $C_{r}$ after normalization and de-duplication.
We quantify *alignment* using two metrics to reflect the concept alignment:

$$ $\text{Alignment-Overlap}=Jaccard(C_{r},C_{x})=\frac{|\,C_{r}\cap C_{x}\,|}{|\,C_{r}\,|\cup|C_{x}\,|},\quad\text{Alignment-Coverage}=\frac{|\,C_{r}\cap C_{x}\,|}{|\,C_{x}\,|}W$ $$

Intuitively, Alignment-Overlap captures the extent to which the concepts *introduced by the model* overlap with those already present in the input, while Alignment-Coverage measures how much of the *input’s clinical content* is covered by the reasoning trace.

Following the above length analysis protocol, we rank instances within each model–task pair by the alignment metric and partition them into ten equal-sized bins (deciles).
For each decile $d$, we compute the task score under CoT, $S_{t}^{\text{cot}}(d)$, and—on the identical instances—the zero-shot score, $S_{t}^{\text{zs}}(d)$, and report the relative change (Sec. §[4.2](https://arxiv.org/html/2509.21933v1#S4.SS2)).
To capture representative model behavior while ensuring sufficient CoT data, we include models whose valid-response rate under CoT exceeds $90\%$.

#### Main results and findings.

Figure [5](https://arxiv.org/html/2509.21933v1#S4.F5) summarizes the results.
Both task performance and the CoT performance drop improve with stronger concept alignment.
When Alignment-Overlap (left) increases from the lowest to the highest deciles, average scores rise steadily for both protocols, and the relative CoT deficit shrinks.
A similar pattern holds for Alignment-Coverage (right): covering more of the input’s clinical concepts is associated with higher accuracy and a smaller CoT penalty.
These trends indicate that CoT degradation is attenuated when the reasoning trace remains tightly grounded in the clinical content of the input, suggesting that concept-faithful rationales are a key precondition for reliable CoT.

Figure: Figure 6: Lexical contrast between CoT-correct and CoT-incorrect traces. Each point is a word; axes show normalized frequency in two groups. Color encodes density from infrequent (yellow) to frequent (blue/red). The right panels list the most characteristic words for each group.
Refer to caption: figures/word_difference.png

### 4.4 Lexical Signatures of CoT Reasoning Trace for Correct and Incorrect Instances

#### Setup.

We partition the evaluation instances into two disjoint groups based on the correctness of the *final answer* under CoT:
*CoT-correct* and *CoT-incorrect*.
For each instance, we take the CoT reasoning trace $r_{i}$ and compute corpus-level word statistics after lowercasing, punctuation stripping, and lemmatization.
To reduce confounding from input copying, we remove tokens that also appear in the corresponding input $x_{i}$.
Let $V$ be the vocabulary after stopword removal.
For a term $w\in V$, let $f_{w}^{\mathsf{cor}}$ and $f_{w}^{\mathsf{inc}}$ denote its token counts in CoT-correct and CoT-incorrect traces, with $N^{\mathsf{cor}}=\sum_{w}f_{w}^{\mathsf{cor}}$ and $N^{\mathsf{inc}}=\sum_{w}f_{w}^{\mathsf{inc}}$.
We report (i) normalized frequencies $p_{w}^{\mathsf{cor}}=f_{w}^{\mathsf{cor}}/N^{\mathsf{cor}}$ and $p_{w}^{\mathsf{inc}}=f_{w}^{\mathsf{inc}}/N^{\mathsf{inc}}$ and (ii) differential usage via informative log-odds with a Dirichlet prior (Monroe et al., 2008; Kessler, 2017):

$$ $\delta_{w}=\log\frac{f_{w}^{\mathsf{cor}}+\alpha_{w}}{N^{\mathsf{cor}}+\alpha_{0}-(f_{w}^{\mathsf{cor}}+\alpha_{w})}\;-\;\log\frac{f_{w}^{\mathsf{inc}}+\alpha_{w}}{N^{\mathsf{inc}}+\alpha_{0}-(f_{w}^{\mathsf{inc}}+\alpha_{w})},\quad z_{w}=\frac{\delta_{w}}{\sqrt{\sigma^{2}_{w}}},$ $$

where $\alpha_{w}$ is the background frequency scaled by $\alpha_{0}$ and $\sigma^{2}_{w}$ is the approximate variance from Monroe et al. (2008).
This prior-regularized contrast mitigates small-sample artifacts and yields a ranked list of words characteristic of each group.

#### Main results and findings.

Figure [6](https://arxiv.org/html/2509.21933v1#S4.F6) contrasts term frequencies between CoT-correct and CoT-incorrect traces.
Terms in the right-lower region (high in CoT-incorrect, low in CoT-correct) are disproportionately associated with errors.
We observe that *numbers, units, and measurement-related tokens* are overrepresented among CoT-incorrect traces (e.g., pt, ct, numeric values), as are abbreviations for laboratory and physiologic assessments such as plt (platelet count) and pft (pulmonary function test).
Conversely, CoT-correct traces contain a higher proportion of disease and therapy terms that are semantically aligned with the input clinical background.
These lexical signatures align with our earlier analyses: tasks requiring precise handling of quantitative data and shorthand notations are more vulnerable under CoT, consistent with reports that LLMs are brittle with numeric reasoning (Akhtar et al., 2023; Khandekar et al., 2024). and domain-specific abbreviations (Rajkomar et al., 2022; Kugic et al., 2024).

Figure: (a) Distribution of error combinations
Refer to caption: x6.png

### 4.5 Error Analysis: Hallucination, Omission, and Incompleteness

#### Setup.

To characterize failure modes in the CoT reasoning trace $r$, we adopt an LLM-as-a-Judge protocol.
Following prior studies about LLM-as-a-Judge on clinical task (Liu et al., 2024; Gu et al., 2024), we evaluate three primary dimensions:

- •
Hallucination.
Claims are unsupported by the input or medical knowledge.
- •
Omission.
Omission of task-relevant or clinically important facts from the input.
- •
Incompleteness.
Truncated, underspecified, or insufficient to justify the answer.

We adopt OpenAI-o3 as the judge model. The judge model was manually assessed by a clinical expert in a random subset ($n=200$).
The full prompts can be found in Appendix [A.5](https://arxiv.org/html/2509.21933v1#A1.SS5).
To minimize bias, model identities are anonymized and responses are randomly ordered during judging.
We analyze three challenging EHR-based tasks—mortality prediction on MIMIC-III-Outcome (van Aken et al., 2021), disease diagnosis on MIMIC-IV-DiReCT dataset (Wang et al., 2024) and MIMIC-IV-CDM dataset (Hager et al., 2024).

#### Main results and findings.

Figure [7](https://arxiv.org/html/2509.21933v1#S4.F7) reports the distribution of detected error types across models.
Generally, the best-performing models, GPT-4o, show the fewest errors; the following Qwen2.5-72B-Instruct and Llama-3.3-70B-Instruct are comparable. However, MeLLaMA-70B-Chat, trained primarily for medical Q&A rather than EHR cases, exhibits substantially higher omission and incompleteness rates, indicating difficulty in extracting and organizing patient-specific evidence from longitudinal notes.
Overall, hallucination and omission are the predominant failure modes (Figure [7](https://arxiv.org/html/2509.21933v1#S4.F7)). The expert review highlights cases where historical conditions mentioned in the input are incorrectly treated as current.
Compared to expert annotation, the positive predictive value (PPV) of the judge was $97.22\%$ for hallucination, $97.83\%$ for omission, and $100\%$ for incompleteness, demonstrating the reliability of the LLM-as-a-Judge.

## 5 Limitations

This study has several limitations.
First, the scale of expert validation is limited due to time and budget constraints; although the audited sample supports the reliability of LLM-as-a-Judge, broader expert annotation can further refine the judging rubric and provide more key insights.
Second, our goal is to investigate the effectiveness of CoT in general settings; we do not evaluate engineering interventions such as self-consistency, retrieval-augmented generation (RAG), or few-shot learning, which may improve model performance.

## 6 Conclusion

We conducted a large-scale, systematic investigation of LLM reasoning in clinical contexts, spanning 95 LLMs and real-world clinical text tasks, and show that reliable reasoning in clinical settings sets a higher demand than general domains, with current models insufficiently adapted to clinical understanding. Analyses on CoT reasoning trace implicate error accumulation in longer chains and insufficient grounding to clinical concepts as key mechanisms, exposing weaknesses in clinical abbreviations and numerical reasoning.
An LLM-as-a-Judge framework with expert review also highlights the hallucination and information omission as dominant failure modes of CoT.
Overall, these results establish a foundational evidence for when and why CoT underperforms in clinical settings. More efforts are needed to enhance LLMs’ understanding of medical knowledge and clinical contexts, and then develop clinically aligned reasoning strategies that reconcile transparency with trustworthy performance.

## References

- Akhtar et al. (2023)
Mubashara Akhtar, Abhilash Shankarampeta, Vivek Gupta, Arpit Patil, Oana Cocarascu, and Elena Simperl.
Exploring the numerical reasoning capabilities of language models: A comprehensive analysis on tabular data.
In *Findings of the Association for Computational Linguistics: EMNLP 2023*, pp. 15391–15405, Singapore, December 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.findings-emnlp.1028.
URL [https://aclanthology.org/2023.findings-emnlp.1028/](https://aclanthology.org/2023.findings-emnlp.1028/).
- Amann et al. (2020)
Julia Amann, Alessandro Blasimme, Effy Vayena, Dietmar Frey, Vince I Madai, and Precise4Q Consortium.
Explainability for artificial intelligence in healthcare: a multidisciplinary perspective.
*BMC medical informatics and decision making*, 20(1):310, 2020.
- Asgari et al. (2025)
Elham Asgari, Nina Montaña-Brown, Magda Dubois, Saleh Khalil, Jasmine Balloch, Joshua Au Yeung, and Dominic Pimenta.
A framework to assess clinical safety and hallucination rates of llms for medical text summarisation.
*npj Digital Medicine*, 8(1):274, 2025.
- Besta et al. (2024)
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michał Podstawski, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler.
Graph of Thoughts: Solving elaborate problems with large language models.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 38, pp. 17682–17690. AAAI Press, Mar 2024.
doi: 10.1609/aaai.v38i16.29720.
URL [https://ojs.aaai.org/index.php/AAAI/article/view/29720](https://ojs.aaai.org/index.php/AAAI/article/view/29720).
- Ennab & Mcheick (2024)
Mohammad Ennab and Hamid Mcheick.
Enhancing interpretability and accuracy of ai models in healthcare: a comprehensive review on challenges and future directions.
*Frontiers in Robotics and AI*, 11:1444763, 2024.
- Gu et al. (2024)
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al.
A survey on llm-as-a-judge.
*arXiv preprint arXiv:2411.15594*, 2024.
- (7)
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.
URL [https://arxiv.org/abs/2501.12948](https://arxiv.org/abs/2501.12948).
- Habli et al. (2020)
Ibrahim Habli, Tom Lawton, and Zoe Porter.
Artificial intelligence in health care: accountability and safety.
*Bulletin of the World Health Organization*, 98(4):251, 2020.
- Hager et al. (2024)
Paul Hager, Friederike Jungmann, Robbie Holland, Kunal Bhagat, Inga Hubrecht, Manuel Knauer, Jakob Vielhauer, Marcus Makowski, Rickmer Braren, Georgios Kaissis, et al.
Evaluation and mitigation of the limitations of large language models in clinical decision-making.
*Nature medicine*, 30(9):2613–2622, 2024.
- Jaech et al. (2024)
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al.
Openai o1 system card.
*arXiv preprint arXiv:2412.16720*, 2024.
- Kessler (2017)
Jason S Kessler.
Scattertext: a browser-based tool for visualizing how corpora differ.
*arXiv preprint arXiv:1703.00565*, 2017.
- Khandekar et al. (2024)
Nikhil Khandekar, Qiao Jin, Guangzhi Xiong, Soren Dunn, Serina Applebaum, Zain Anwar, Maame Sarfo-Gyamfi, Conrad Safranek, Abid Anwar, Andrew Zhang, et al.
Medcalc-bench: Evaluating large language models for medical calculations.
*Advances in Neural Information Processing Systems*, 37:84730–84745, 2024.
- Kim et al. (2025)
Chanwoo Kim, Soham U Gadgil, and Su-In Lee.
Transparency of medical artificial intelligence systems.
*Nature Reviews Bioengineering*, pp. 1–19, 2025.
- Kojima et al. (2022)
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.
Large language models are zero-shot reasoners.
In *Advances in Neural Information Processing Systems*, volume 35, pp. 22199–22213. Curran Associates, Inc., 2022.
URL [https://proceedings.neurips.cc/paper_files/paper/2022/file/8bb0d291acd4acf06ef112099c16f326-Paper-Conference.pdf](https://proceedings.neurips.cc/paper_files/paper/2022/file/8bb0d291acd4acf06ef112099c16f326-Paper-Conference.pdf).
- Kugic et al. (2024)
Amila Kugic, Stefan Schulz, and Markus Kreuzthaler.
Disambiguation of acronyms in clinical narratives with large language models.
*Journal of the American Medical Informatics Association*, 31(9):2040–2046, 2024.
- Liévin et al. (2024)
Valentin Liévin, Christoffer Egeberg Hother, Andreas Geert Motzfeldt, and Ole Winther.
Can large language models reason about medical questions?
*Patterns*, 5(3), 2024.
- Liu et al. (2024)
Xiaocong Liu, Jiageng Wu, An Shao, Wenyue Shen, Panpan Ye, Yao Wang, Juan Ye, Kai Jin, and Jie Yang.
Uncovering language disparity of ChatGPT on retinal vascular disease classification: Cross-sectional study.
*Journal of Medical Internet Research*, 26:e51926, 2024.
- Lyu et al. (2023)
Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch.
Faithful chain-of-thought reasoning.
In *The 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL 2023)*, 2023.
- Monroe et al. (2008)
Burt L Monroe, Michael P Colaresi, and Kevin M Quinn.
Fightin’words: Lexical feature selection and evaluation for identifying the content of political conflict.
*Political Analysis*, 16(4):372–403, 2008.
- Nachane et al. (2024)
Saeel Sandeep Nachane, Ojas Gramopadhye, Prateek Chanda, Ganesh Ramakrishnan, Kshitij Sharad Jadhav, Yatin Nandwani, Dinesh Raghu, and Sachindra Joshi.
Few shot chain-of-thought driven reasoning to prompt LLMs for open-ended medical question answering.
In *Findings of the Association for Computational Linguistics: EMNLP 2024*, pp. 542–573, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.findings-emnlp.31.
URL [https://aclanthology.org/2024.findings-emnlp.31/](https://aclanthology.org/2024.findings-emnlp.31/).
- Park et al. (2024)
Ye-Jean Park, Abhinav Pillai, Jiawen Deng, Eddie Guo, Mehul Gupta, Mike Paget, and Christopher Naugler.
Assessing the research landscape and clinical utility of large language models: a scoping review.
*BMC Medical Informatics and Decision Making*, 24(1):72, 2024.
- Raiaan et al. (2024)
Mohaimenul Azam Khan Raiaan, Md Saddam Hossain Mukta, Kaniz Fatema, Nur Mohammad Fahad, Sadman Sakib, Most Marufatul Jannat Mim, Jubaer Ahmad, Mohammed Eunus Ali, and Sami Azam.
A review on large language models: Architectures, applications, taxonomies, open issues and challenges.
*IEEE Access*, 12:26839–26874, 2024.
- Rajkomar et al. (2022)
Alvin Rajkomar, Eric Loreaux, Yuchen Liu, Jonas Kemp, Benny Li, Ming-Jun Chen, Yi Zhang, Afroz Mohiuddin, and Juraj Gottweis.
Deciphering clinical abbreviations with a privacy protecting machine learning system.
*Nature communications*, 13(1):7456, 2022.
- Roustan et al. (2025)
Dimitri Roustan, François Bastardot, et al.
The clinicians’ guide to large language models: A general perspective with a focus on hallucinations.
*Interactive journal of medical research*, 14(1):e59823, 2025.
- Savova et al. (2010)
Guergana K Savova, James J Masanz, Philip V Ogren, Jiaping Zheng, Sunghwan Sohn, Karin C Kipper-Schuler, and Christopher G Chute.
Mayo clinical text analysis and knowledge extraction system (ctakes): architecture, component evaluation and applications.
*Journal of the American Medical Informatics Association*, 17(5):507–513, 2010.
- Shah (2024)
Savyasachi V Shah.
Accuracy, consistency, and hallucination of large language models when analyzing unstructured clinical notes in electronic medical records.
*JAMA Network Open*, 7(8):e2425953–e2425953, 2024.
- Singhal et al. (2025)
Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al.
Toward expert-level medical question answering with large language models.
*Nature Medicine*, pp. 1–8, 2025.
- van Aken et al. (2021)
Betty van Aken, Jens-Michalis Papaioannou, Manuel Mayrdorfer, Klemens Budde, Felix Gers, and Alexander Loeser.
Clinical outcome prediction from admission notes using self-supervised knowledge integration.
In *Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume*, pp. 881–893, Online, April 2021. Association for Computational Linguistics.
doi: 10.18653/v1/2021.eacl-main.75.
URL [https://aclanthology.org/2021.eacl-main.75/](https://aclanthology.org/2021.eacl-main.75/).
- Vrdoljak et al. (2025)
Josip Vrdoljak, Zvonimir Boban, Marino Vilović, Marko Kumrić, and Joško Božić.
A review of large language models in medical education, clinical decision support, and healthcare administration.
In *Healthcare*, volume 13, pp. 603. MDPI, 2025.
- Wang et al. (2024)
Bowen Wang, Jiuyang Chang, Yiming Qian, Guoxin Chen, Junhao Chen, Zhouqiang Jiang, Jiahao Zhang, Yuta Nakashima, and Hajime Nagahara.
Direct: Diagnostic reasoning for clinical notes via large language models.
In *Advances in Neural Information Processing Systems*, volume 37, pp. 74999–75011, 2024.
- Wang et al. (2023)
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.
Self-consistency improves chain of thought reasoning in language models.
2023.
URL [https://openreview.net/forum?id=1PL1NIMMrw](https://openreview.net/forum?id=1PL1NIMMrw).
- Wei et al. (2022)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou.
Chain-of-Thought prompting elicits reasoning in large language models.
In *Advances in Neural Information Processing Systems*, volume 35, pp. 24824–24837. Curran Associates, Inc., 2022.
URL [https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf](https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf).
- Wu et al. (2024a)
Jiageng Wu, Xian Wu, Zhaopeng Qiu, Minghui Li, Shixu Lin, Yingying Zhang, Yefeng Zheng, Changzheng Yuan, and Jie Yang.
Large language models leverage external knowledge to extend clinical insight beyond language boundaries.
*Journal of the American Medical Informatics Association*, 31(9):2054–2064, 2024a.
- Wu et al. (2024b)
Jiageng Wu, Xian Wu, and Jie Yang.
Guiding clinical reasoning with large language models via knowledge seeds.
In *Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence*, IJCAI ’24, 2024b.
ISBN 978-1-956792-04-1.
doi: 10.24963/ijcai.2024/829.
URL [https://doi.org/10.24963/ijcai.2024/829](https://doi.org/10.24963/ijcai.2024/829).
- Wu et al. (2025)
Jiageng Wu, Bowen Gu, Ren Zhou, Kevin Xie, Doug Snyder, Yixing Jiang, Valentina Carducci, Richard Wyss, Rishi J Desai, Emily Alsentzer, et al.
Bridge: Benchmarking large language models for understanding real-world clinical practice text.
*arXiv preprint arXiv:2504.19467*, 2025.
- Yao et al. (2023a)
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan.
Tree of thoughts: Deliberate problem solving with large language models.
In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), *Advances in Neural Information Processing Systems*, volume 36, pp. 11809–11822. Curran Associates, Inc., 2023a.
URL [https://proceedings.neurips.cc/paper_files/paper/2023/file/271db9922b8d1f4dd7aaef84ed5ac703-Paper-Conference.pdf](https://proceedings.neurips.cc/paper_files/paper/2023/file/271db9922b8d1f4dd7aaef84ed5ac703-Paper-Conference.pdf).
- Yao et al. (2023b)
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
ReAct: Synergizing reasoning and acting in language models.
In *The Eleventh International Conference On Learning Representations*, 2023b.
- Zhou et al. (2023)
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi.
Least-to-most prompting enables complex reasoning in large language models.
In *The Eleventh International Conference On Learning Representations*, 2023.
URL [https://openreview.net/forum?id=WZH7099tgfM](https://openreview.net/forum?id=WZH7099tgfM).

## Appendix A Appendix

### A.1 Implementation details

#### Model Access and Environment

All open-source models (except Deepseek-R1) are executed locally on NVIDIA A100 GPUs (8$\times$80 GB). Deepseek-R1 and all proprietary models are accessed through an institution-managed Azure cloud service, avoiding the potential privacy concerns and security risks.

#### Inference protocol and decoding controls.

All models are evaluated under a unified decoding configuration to ensure compute parity.
Unless otherwise noted, we use deterministic decoding (temperature $=0$; sampling disabled).
The confidence interval is estimated via bootstrap resampling with 1,000 iterations.

### A.2 Model performance

We conducted a comprehensive comparison of 95 state-of-the-art large language models (LLMs) across 87 multilingual clinical tasks derived from the BRIDGE benchmark, encompassing nine languages and eight distinct task categories (classification, extraction, question answering, summarization, and others). The evaluated models comprised proprietary systems (e.g., GPT-4o and Gemini), open-source architectures (e.g., DeepSeek, Llama, and Qwen), and specialized medical domain models (e.g., Me-LLaMA and Meditron).

Our findings, presented in Table LABEL:tab:all_model_results, demonstrate that most LLMs (82 of 95, 86.3%) exhibit degraded performance under CoT inference.

**Table S1: Overall Performance of LLMs Across Different Inference Strategies**
| Model | Model<br>Size | Model<br>Domain | Model<br>Percentile | Zero-shot<br>Score | CoT<br>Score | $\Delta$ Score<br>(CoT - ZS) | $\Delta$ (%)<br>(CoT - ZS) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Gemini-2.5-flash | / | General | 80-100% | 44.8 [44.1,45.6] | 43.3 [42.5,44.1] | $-1.6$ | $-3.5$ |
| Deepseek-R1 | 671 | General | 80-100% | 44.2 [43.5,45.0] | 42.1 [41.3,42.9] | $-2.1$ | $-4.9$ |
| GPT-4o-0806 | / | General | 80-100% | 44.2 [43.4,45.0] | 40.7 [39.9,41.4] | $-3.5$ | $-8.0$ |
| Qwen3-Next-80B-A3B-Thinking | 80 | General | 80-100% | 43.9 [43.1,44.6] | 42.9 [42.1,43.7] | $-1.0$ | $-2.2$ |
| Gemini-1.5-pro-002 | / | General | 80-100% | 43.8 [43.1,44.6] | 40.5 [39.7,41.3] | $-3.3$ | $-7.6$ |
| Gemini-2.0-flash-001 | / | General | 80-100% | 43.0 [42.2,43.8] | 42.0 [41.2,42.8] | $-1.1$ | $-2.4$ |
| Mistral-Large-Instruct-2411 | 123 | General | 80-100% | 42.3 [41.5,43.1] | 38.9 [38.1,39.7] | $-3.4$ | $-8.0$ |
| Qwen3-30B-A3B-Thinking-2507 | 30 | General | 80-100% | 41.8 [41.0,42.6] | 41.4 [40.6,42.2] | $-0.4$ | $-0.9$ |
| Athene-V2-Chat | 72 | General | 80-100% | 41.7 [40.9,42.5] | 39.3 [38.6,40.1] | $-2.4$ | $-5.6$ |
| Qwen3-235B-A22B-Thinking | 235 | General | 80-100% | 41.6 [40.8,42.5] | 40.1 [39.4,40.9] | $-1.5$ | $-3.6$ |
| Qwen2.5-72B-Instruct | 72 | General | 80-100% | 41.6 [40.8,42.4] | 38.9 [38.1,39.7] | $-2.8$ | $-6.6$ |
| Qwen3-32B-Thinking | 32 | General | 80-100% | 41.0 [40.3,41.8] | 34.5 [33.7,35.3] | $-6.6$ | $-16.0$ |
| HuatuoGPT-o1-72B | 72 | Medical | 80-100% | 41.0 [40.2,41.8] | 38.1 [37.4,38.9] | $-2.9$ | $-7.0$ |
| Qwen3-30B-A3B-Thinking | 30 | General | 80-100% | 40.9 [40.1,41.8] | 39.4 [38.6,40.1] | $-1.6$ | $-3.9$ |
| medgemma-27b-it | 27 | Medical | 80-100% | 40.8 [40.0,41.6] | 38.2 [37.5,38.9] | $-2.6$ | $-6.4$ |
| Qwen3-14B-Thinking | 14 | General | 80-100% | 40.2 [39.4,41.0] | 37.1 [36.3,37.9] | $-3.1$ | $-7.7$ |
| Qwen3-4B-Thinking-2507 | 4 | General | 80-100% | 40.0 [39.2,40.8] | 38.8 [38.0,39.7] | $-1.2$ | $-3.0$ |
| Qwen3-8B-Thinking | 8 | General | 80-100% | 40.0 [39.2,40.8] | 37.1 [36.2,37.9] | $-2.9$ | $-7.3$ |
| gemma-3-27b-it | 27 | General | 80-100% | 39.9 [39.1,40.7] | 37.5 [36.8,38.3] | $-2.4$ | $-5.9$ |
| Qwen2.5-32B-Instruct | 32 | General | 60-80% | 39.9 [39.1,40.7] | 37.7 [36.9,38.5] | $-2.2$ | $-5.6$ |
| Llama-3.3-70B-Instruct | 70 | General | 60-80% | 39.9 [39.1,40.7] | 36.8 [36.1,37.6] | $-3.0$ | $-7.6$ |
| Qwen3-Next-80B-A3B-Instruct | 80 | General | 60-80% | 39.8 [39.0,40.6] | 40.5 [39.7,41.3] | $0.7$ | $1.7$ |
| Deepseek-R1-Distill-Llama-70B | 70 | General | 60-80% | 39.8 [39.0,40.6] | 38.9 [38.2,39.7] | $-0.8$ | $-2.1$ |
| Deepseek-R1-Distill-Qwen-32B | 32 | General | 60-80% | 39.7 [39.0,40.5] | 38.7 [37.9,39.5] | $-1.0$ | $-2.6$ |
| Mistral-Small-3.1-24B-Instruct-2503 | 24 | General | 60-80% | 39.7 [38.9,40.6] | 36.2 [35.4,37.0] | $-3.5$ | $-8.8$ |
| QWQ-32B | 32 | General | 60-80% | 39.4 [38.6,40.2] | 37.0 [36.2,37.8] | $-2.3$ | $-6.0$ |
| Qwen3-32B-Non-Thinking | 32 | General | 60-80% | 39.3 [38.5,40.1] | 32.8 [32.0,33.6] | $-6.5$ | $-16.5$ |
| QwenLong-L1-32B | 32 | General | 60-80% | 39.2 [38.4,40.1] | 36.7 [35.9,37.5] | $-2.5$ | $-6.5$ |
| Qwen3-235B-A22B-Non-Thinking | 235 | General | 60-80% | 39.2 [38.4,40.0] | 38.0 [37.2,38.8] | $-1.2$ | $-3.1$ |
| Llama-3.1-70B-Instruct | 70 | General | 60-80% | 39.1 [38.3,39.9] | 35.1 [34.3,35.9] | $-4.0$ | $-10.2$ |
| Qwen3-4B-Thinking | 4 | General | 60-80% | 38.5 [37.7,39.3] | 37.0 [36.2,37.8] | $-1.5$ | $-4.0$ |
| gemma-2-27b-it | 27 | General | 60-80% | 38.2 [37.4,39.0] | 34.2 [33.4,35.0] | $-4.0$ | $-10.4$ |
| Baichuan-M2-32B | 32 | Medical | 60-80% | 38.0 [37.0,39.0] | 38.3 [37.4,39.2] | $0.3$ | $0.7$ |
| Magistral-Small-2506 | 24 | General | 60-80% | 37.6 [36.8,38.4] | 34.2 [33.3,35.0] | $-3.4$ | $-9.1$ |
| Mistral-Small-24B-Instruct-2501 | 24 | General | 60-80% | 37.5 [36.7,38.4] | 31.6 [30.8,32.4] | $-5.9$ | $-15.8$ |
| gemma-3-12b-it | 12 | General | 60-80% | 37.3 [36.6,38.1] | 35.4 [34.6,36.1] | $-1.9$ | $-5.2$ |
| gpt-oss-120b | 120 | General | 60-80% | 37.2 [36.4,38.1] | 32.1 [31.4,32.9] | $-5.1$ | $-13.8$ |
| Deepseek-R1-0528-Qwen3-8B | 8 | General | 60-80% | 36.9 [36.1,37.8] | 36.3 [35.5,37.1] | $-0.7$ | $-1.8$ |
| Qwen3-30B-A3B-Instruct-2507 | 30 | General | 40-60% | 36.9 [36.1,37.6] | 37.5 [36.7,38.3] | $0.6$ | $1.7$ |
| Qwen3-14B-Non-Thinking | 14 | General | 40-60% | 36.8 [36.0,37.6] | 34.6 [33.8,35.4] | $-2.2$ | $-6.0$ |
| Qwen3-30B-A3B-Non-Thinking | 30 | General | 40-60% | 36.2 [35.4,36.9] | 36.9 [36.1,37.7] | $0.7$ | $1.9$ |
| Phi-4 | 14 | General | 40-60% | 36.1 [35.4,36.9] | 32.6 [31.8,33.4] | $-3.5$ | $-9.8$ |
| Baichuan-M1-14B-Instruct | 14 | Medical | 40-60% | 36.1 [35.3,36.9] | 34.4 [33.6,35.2] | $-1.7$ | $-4.8$ |
| gpt-35-turbo-0125 | / | General | 40-60% | 35.3 [34.5,36.1] | 31.6 [30.8,32.4] | $-3.7$ | $-10.4$ |
| Mistral-Small-Instruct-2409 | 22 | General | 40-60% | 35.2 [34.4,36.0] | 31.2 [30.4,32.0] | $-4.0$ | $-11.4$ |
| Llama-4-Scout-17B-16E-Instruct | 109 | General | 40-60% | 35.1 [34.4,35.9] | 29.4 [28.7,30.1] | $-5.7$ | $-16.3$ |
| gemma-2-9b-it | 9 | General | 40-60% | 35.1 [34.3,35.9] | 29.9 [29.2,30.7] | $-5.1$ | $-14.6$ |
| Deepseek-R1-Distill-Qwen-14B | 14 | General | 40-60% | 34.3 [33.5,35.1] | 34.8 [34.0,35.6] | $0.5$ | $1.5$ |
| Qwen3-4B-Instruct-2507 | 4 | General | 40-60% | 34.1 [33.3,34.8] | 36.2 [35.4,37.0] | $2.2$ | $6.3$ |
| HuatuoGPT-o1-70B | 70 | Medical | 40-60% | 33.9 [33.2,34.6] | 32.3 [31.6,33.1] | $-1.5$ | $-4.6$ |
| Qwen3-8B-Non-Thinking | 8 | General | 40-60% | 33.9 [33.1,34.7] | 33.3 [32.5,34.1] | $-0.6$ | $-1.6$ |
| Llama-3-70B-UltraMedical | 70 | Medical | 40-60% | 33.4 [32.6,34.2] | 29.4 [28.7,30.2] | $-4.0$ | $-11.9$ |
| Qwen3-4B-Non-Thinking | 4 | General | 40-60% | 33.3 [32.5,34.1] | 33.2 [32.4,34.0] | $-0.1$ | $-0.3$ |
| Llama3-OpenBioLLM-70B | 70 | Medical | 40-60% | 33.0 [32.2,33.8] | 28.8 [28.0,29.5] | $-4.2$ | $-12.8$ |
| Llama-3.1-Nemotron-70B-Instruct-HF | 70 | General | 40-60% | 32.7 [32.0,33.5] | 24.1 [23.5,24.7] | $-8.7$ | $-26.4$ |
| MeLLaMA-70B-chat | 70 | Medical | 40-60% | 32.3 [31.5,33.1] | 29.2 [28.4,30.1] | $-3.0$ | $-9.3$ |
| Yi-1.5-34B-Chat-16K | 34 | General | 40-60% | 32.1 [31.4,32.9] | 29.6 [28.8,30.3] | $-2.5$ | $-7.9$ |
| QwQ-32B-Preview | 32 | General | 20-40% | 31.7 [31.0,32.5] | 23.3 [22.5,24.1] | $-8.4$ | $-26.6$ |
| Qwen2.5-7B-Instruct | 7 | General | 20-40% | 31.3 [30.5,32.1] | 30.2 [29.5,31.0] | $-1.1$ | $-3.4$ |
| Ministral-8B-Instruct-2410 | 8 | General | 20-40% | 30.4 [29.5,31.2] | 25.9 [25.1,26.7] | $-4.5$ | $-14.7$ |
| HuatuoGPT-o1-7B | 7 | Medical | 20-40% | 29.6 [28.8,30.4] | 26.7 [26.0,27.4] | $-2.9$ | $-9.8$ |
| Phi-3.5-MoE-instruct | 42 | General | 20-40% | 29.5 [28.8,30.3] | 25.3 [24.5,26.0] | $-4.3$ | $-14.5$ |
| medgemma-4b-it | 4 | Medical | 20-40% | 29.4 [28.3,30.5] | 28.5 [27.6,29.3] | $-0.9$ | $-3.1$ |
| gpt-oss-20b | 20 | General | 20-40% | 29.0 [28.3,29.8] | 24.9 [24.1,25.6] | $-4.2$ | $-14.4$ |
| Llama-3.1-8B-Instruct | 8 | General | 20-40% | 29.0 [28.2,29.8] | 29.4 [28.6,30.2] | $0.4$ | $1.4$ |
| Yi-1.5-9B-Chat-16K | 9 | General | 20-40% | 28.8 [28.0,29.6] | 25.4 [24.6,26.2] | $-3.4$ | $-11.9$ |
| gemma-3-4b-it | 4 | General | 20-40% | 28.6 [27.8,29.4] | 28.2 [27.4,29.0] | $-0.4$ | $-1.3$ |
| Deepseek-R1-Distill-Llama-8B | 8 | General | 20-40% | 28.5 [27.7,29.2] | 27.3 [26.6,28.1] | $-1.1$ | $-4.0$ |
| Qwen2.5-3B-Instruct | 3 | General | 20-40% | 26.6 [25.8,27.3] | 25.4 [24.7,26.2] | $-1.1$ | $-4.3$ |
| Qwen3-1.7B-Thinking | 1.7 | General | 20-40% | 26.3 [25.5,27.0] | 27.7 [27.0,28.4] | $1.4$ | $5.4$ |
| HuatuoGPT-o1-8B | 8 | Medical | 20-40% | 25.9 [25.1,26.6] | 22.5 [21.8,23.2] | $-3.4$ | $-13.0$ |
| Phi-3.5-mini-instruct | 4 | General | 20-40% | 25.4 [24.7,26.1] | 23.9 [23.2,24.6] | $-1.5$ | $-5.9$ |
| Deepseek-R1-Distill-Qwen-7B | 7 | General | 20-40% | 25.3 [24.5,26.0] | 23.9 [23.1,24.7] | $-1.4$ | $-5.5$ |
| Llama-2-70b-chat | 70 | General | 20-40% | 25.2 [24.4,25.9] | 19.0 [18.3,19.7] | $-6.1$ | $-24.4$ |
| Phi-4-mini-instruct | 4 | General | 20-40% | 24.5 [23.8,25.3] | 22.5 [21.8,23.2] | $-2.0$ | $-8.3$ |
| Llama-3.2-3B-Instruct | 3 | General | 20-40% | 22.9 [22.1,23.7] | 21.6 [20.9,22.3] | $-1.3$ | $-5.7$ |
| Qwen2.5-1.5B-Instruct | 1.5 | General | 0-20% | 22.2 [21.4,22.9] | 19.5 [18.7,20.2] | $-2.7$ | $-12.1$ |
| Qwen3-1.7B-Non-Thinking | 1.7 | General | 0-20% | 22.0 [21.2,22.7] | 22.8 [22.0,23.5] | $0.8$ | $3.7$ |
| Phi-4-mini-reasoning | 4 | General | 0-20% | 21.3 [20.6,22.0] | 19.8 [19.1,20.5] | $-1.5$ | $-7.1$ |
| Llama-2-13b-chat | 13 | General | 0-20% | 20.9 [20.2,21.7] | 16.2 [15.5,16.9] | $-4.7$ | $-22.4$ |
| MeLLaMA-13B-chat | 13 | Medical | 0-20% | 20.8 [20.0,21.5] | 20.3 [19.5,21.0] | $-0.5$ | $-2.4$ |
| BioMistral-7B | 7 | Medical | 0-20% | 20.4 [19.7,21.2] | 10.8 [10.2,11.5] | $-9.6$ | $-46.9$ |
| Qwen3-0.6B-Thinking | 0.6 | General | 0-20% | 20.4 [19.6,21.1] | 18.9 [18.2,19.7] | $-1.4$ | $-7.1$ |
| MMed-Llama-3-8B | 8 | Medical | 0-20% | 20.4 [19.7,21.1] | 16.2 [15.4,16.9] | $-4.2$ | $-20.6$ |
| Llama-3.1-8B-UltraMedical | 8 | Medical | 0-20% | 20.2 [19.5,20.9] | 18.3 [17.7,19.0] | $-1.8$ | $-9.0$ |
| OpenThinker3-7B | 7 | General | 0-20% | 19.8 [19.0,20.5] | 18.4 [17.7,19.1] | $-1.4$ | $-7.2$ |
| MedReason-8B | 8 | Medical | 0-20% | 18.2 [17.4,18.9] | 18.3 [17.6,19.0] | $0.1$ | $0.7$ |
| Llama-2-7b-chat | 7 | General | 0-20% | 16.5 [15.8,17.2] | 13.7 [13.0,14.3] | $-2.8$ | $-17.1$ |
| gemma-3-1b-it | 1 | General | 0-20% | 15.7 [15.0,16.4] | 13.5 [12.8,14.2] | $-2.2$ | $-14.0$ |
| meditron-70b | 70 | Medical | 0-20% | 15.7 [15.0,16.4] | 13.2 [12.5,13.9] | $-2.5$ | $-16.0$ |
| Qwen3-0.6B-Non-Thinking | 0.6 | General | 0-20% | 15.2 [14.5,15.9] | 16.6 [15.9,17.3] | $1.4$ | $9.1$ |
| Deepseek-R1-Distill-Qwen-1.5B | 1.5 | General | 0-20% | 14.3 [13.6,15.0] | 13.4 [12.7,14.1] | $-0.8$ | $-5.9$ |
| Llama3-OpenBioLLM-8B | 8 | Medical | 0-20% | 14.2 [13.5,14.9] | 13.3 [12.6,14.0] | $-0.9$ | $-6.4$ |
| Llama-3.2-1B-Instruct | 1 | General | 0-20% | 12.7 [12.1,13.4] | 11.9 [11.2,12.5] | $-0.9$ | $-6.7$ |
| Phi-4-reasoning-plus | 14 | General | 0-20% | 10.6 [10.0,11.2] | 10.8 [10.2,11.4] | $0.3$ | $2.5$ |
| Phi-4-reasoning | 14 | General | 0-20% | 10.0 [9.4,10.6] | 10.1 [9.5,10.7] | $0.1$ | $0.9$ |
| meditron-7b | 7 | Medical | 0-20% | 9.5 [8.9,10.2] | 9.5 [8.9,10.1] | $0.0$ | $0.0$ |

### A.3 Model Performance Percentile

We defined a model’s basic capability as its macro-average zero-shot performance across all BRIDGE tasks. To examine whether model capability influences susceptibility to CoT-induced performance degradation, we stratified the 95 models into five equal percentile tiers (19 models each) to analyze this relationship systematically.

As shown in Table [S2](https://arxiv.org/html/2509.21933v1#A1.T2), higher-performing models are more resilient to CoT inference, with top-tier models showing smaller performance drops than lower-performing counterparts.

**Table S2: Performance Drop of CoT Prompting Across LLM Strength Tiers**
| Model Performance Tier | $\Delta$ Score (%) CoT | Observation |
| --- | --- | --- |
| Top 20% | -5.82 | Smallest gap |
| 20%–40% | -6.67 | Moderate |
| 40%–60% | -7.19 | Moderate |
| 60%–80% | -8.63 | Large |
| Bottom 20% | -9.86 | Largest gap |

### A.4 Prompt design and variants

To ensure our findings were not artifacts of specific prompt formulations, we tested four additional prompt variations for both zero-shot and CoT inference using established prompting methodologies. Zero-shot variants differed in formality and constraint specification, while CoT variants used diverse reasoning framings. Complete prompt variations are shown below.

We selected 4 representative models that span different architectures and capabilities to provide a balanced assessment of prompt robustness while maintaining experimental feasibility. As shown in Table [S3](https://arxiv.org/html/2509.21933v1#A1.T3), the performance decrease with CoT inference was consistent across all prompt variations, confirming that our results represent a systematic limitation.

**Table S3: Performance Gap Between Zero-Shot and Chain-of-Thought: $\Delta$score (%)**
| Model | Prompt 1* | Prompt 2 | Prompt 3 | Prompt 4 | Prompt 5 | Overall |
| --- | --- | --- | --- | --- | --- | --- |
| Llama3.3-70B | $-3.10$ (7.8%) | $-2.08$ (5.28%) | $-2.75$ (6.81%) | $-2.89$ (7.25%) | $-2.24$ (5.61%) | $-2.61$ (6.55%) |
| Mistral-Large | $-3.40$ (8.0%) | $-3.03$ (7.14%) | $-3.35$ (7.91%) | $-2.76$ (6.64%) | $-2.99$ (7.10%) | $-3.11$ (7.36%) |
| Qwen2.5-72B | $-2.70$ (6.5%) | $-2.24$ (5.42%) | $-1.47$ (3.54%) | $-3.27$ (8.08%) | $-1.94$ (4.73%) | $-2.32$ (5.65%) |
| Athene-V2 | $-2.40$ (5.8%) | $-1.72$ (4.16%) | $-1.19$ (2.87%) | $-3.11$ (7.63%) | $-1.70$ (4.13%) | $-2.02$ (4.92%) |
| * Prompt 1 is the original task instruction and also the one used in the main experiments. |  |  |  |  |  |  |

### A.5 LLM-as-a-Judge

To investigate LLM failure modes, we employed an LLM-as-a-judge framework implemented with OpenAI-o3 to annotate three binary metrics: hallucination, omission, and incompleteness. For each instance, the LLM judge provided a binary assessment indicating the presence of a failure mode, along with an explanatory statement citing specific examples or rationales. Validation against expert human annotations using the same criteria showed close alignment with the LLM-as-a-judge results.

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 2
presentation: 2
questions: - Can you provide the performance breakdown across the 8 task types?
- Do the results consistently hold across all 9 languages?
- Did you try any standard CoT variants (self-consistency, few-shot, retrieval-augmented prompts)?
- Does controlling CoT length or enforcing structured formats change the outcome?
soundness: 2
strengths: - The evaluation scale (95×87, multilingual, multi-task) far exceeds prior clinical CoT reports and sets a solid empirical reference point for the community.
summary: This paper presents the first large-scale, systematic study investigating the efficacy of Chain-of-Thought (CoT) prompting for large language models (LLMs) on clinical text understanding tasks. The authors conduct an exhaustive evaluation across 95 LLMs and 87 real-world clinical tasks, spanning 9 languages and 8 task types. The work argues that CoT can undermine reliability on noisy, fragmented EHR-like inputs and calls for clinically grounded reasoning strategies.
weaknesses: - The central observation "longer CoT can accumulate errors and degrade performance in noisy settings"  aligns with prior intuition in the hallucination literature. The contribution is primarily scale and domain-specific validation, rather than a new conceptual insight into why CoT fails.

- The paper evaluates only vanilla CoT, without incorporating widely-used practical enhancements such as self-consistency, few-shot prompting, RAG, or structured CoT. In the absence of these comparisons, the work risks overstating the generality of its conclusions.

- The paper aggregates 87 tasks across 8 types but never presents a breakdown of CoT's effect by task type. This is a critical omission. It is highly plausible that tasks like Named Entity Recognition (NER) or Event Extraction do not require multi-step reasoning in the first place. Forcing CoT on such tasks may simply be introducing noise or an inappropriate cognitive pathway, making the resulting performance drop unsurprising and arguably uninformative.

- Although the study claims multilingual evaluation across 9 languages, there is no language-specific analysis.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: - To clarify the length-performance relationship, could the authors provide results if they experimentally constrained the maximum token length of the Chain-of-Thought reasoning trace?
- Were the observed performance drops uniform across the eight clinical task types, or did certain tasks (e.g., NER, QA, classification) demonstrate disproportionately high or low susceptibility to CoT failure?
- Considering the observed lexical signatures related to numerical brittleness, what results would an experiment yield if the prompt were modified to strictly enforce verbatim quotation of numerical and abbreviated inputs in the reasoning trace?
soundness: 3
strengths: - This work provides an unprecedentedly large-scale and systematic empirical assessment of Chain-of-Thought prompting across diverse clinical text tasks and models.
- The paper demonstrates strong originality by moving beyond traditional medical QA benchmarks to evaluate CoT reliability on complex, noisy, real-world EHR narratives.
- Mechanistic analyses clearly link CoT performance degradation to specific factors like reasoning trace length and weak clinical concept grounding.
- Establishing an error taxonomy (hallucination, omission) supported by LLM-as-a-Judge validation offers a crucial new safety evaluation framework for clinical LLMs.
summary: The paper presents the first large-scale systematic investigation into the effectiveness of Chain-of-Thought (CoT) prompting for clinical text understanding, evaluating 95 large language models across 87 real-world, multilingual clinical tasks derived from electronic health records. Contrary to findings in general domains, the study robustly demonstrates that CoT prompting systematically undermines accuracy for the majority of models, particularly lower-capability ones. The authors perform detailed mechanistic analyses showing that performance degradation is linked to increased reasoning length, poor clinical concept grounding, and brittleness in handling numerical data and abbreviations. Finally, an error taxonomy validated by expert clinicians identifies hallucination and omission as the primary failure modes of CoT traces, offering foundational guidance for developing safer and more reliable reasoning protocols in clinical AI.
weaknesses: - Focusing mainly on zero-shot CoT, the findings do not directly address whether more sophisticated reasoning techniques like few-shot CoT or self-consistency would mitigate the observed performance decline.
- Although acknowledged, the correlational relationship between reasoning trace length and performance decline remains ambiguous regarding causality versus input difficulty.
- The lexical signature analysis identifies potential weaknesses in numerical and abbreviation handling but does not include direct experimental confirmation that these are causal failure mechanisms.
- The authors deliberately exclude practical intervention strategies like RAG, yet these are essential for safety-critical grounding and are likely necessary to overcome the identified hallucination issues.
- Relying on cTAKES for concept alignment metrics, rather than an LLM-based grounding metric, might underestimate or inaccurately characterize complex semantic grounding for higher-capability models.
- The methodology for selecting the three error-audited tasks, out of 87 tasks total, needs clearer justification regarding their representativeness of the overall failure modes or maximal CoT degradation.
- Discussion of the few models (13 out of 95) that showed marginal improvements under CoT is insufficient to provide guidance on conditions that might benefit from explicit reasoning.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: - The baseline (zero-shot) itself is unclear — many tasks (e.g., NER, de-identification) normally require few-shot or schema grounding to perform sensibly
- The Appendix tables list 95 models but omit details of prompt instantiation per task, API parameters, and evaluation scripts
soundness: 3
strengths: - The trade of study on accuracy vs. interpretability for CoT in medicine is timely
- The error taxonomy is well motivated and furthermore the failure modes are validated by clinicians for reliability
summary: This paper introduces a study for chain of thought (CoT) effect on medical text understanding. Finding is that CoT degrades performance in most clinical nlp tasks, at an average of 86% of the models being worse than just zero shot, where they benchmarked on 95 models across 87 real world clinical text datasets.
weaknesses: - CoT experiment idea is interesting but it felt the contribution is not substantial. It only compares CoT with zero shot inference, so effectively it's just ask LLMs to 'think step by step'. The novelty of the paper doesn't seem to fully flesh out and a bit over claimed. While the paper’s empirical scale is impressive, the core insight—“CoT reduces accuracy in clinical text tasks”—is primarily observational, not explanatory. The work stops at describing what happens rather than advancing why or how in a mechanistically rigorous way. They gesture at causes (longer reasoning chains, concept misalignment), but these are correlational rather than causal analyses. Mostly “Scale ≠ contribution” evaluating 95 models is extensive, but without a deeper model of reasoning failure, the work risks being seen as descriptive rather than scientific
- The eval metrics don't extend beyond BRIDGE which can be quite limiting. Metrics like accuracy, F1, and ROUGE are purely syntactic; they fail to capture clinical reasoning quality or safety relevance. There’s no human assessment of clinical harm or factual distortion beyond a small 200-sample subset. 
- In addition, the metrics are overly quantitative, neglecting qualitative insight where qualitative examples of CoT failures (hallucinations, omissions) are absent from the main text and no detailed case study or sample reasoning trace is shown to illustrate what these failures look like in practice. Figures are mostly aggregate plots (performance trends, correlations) without clinical interpretability examples—a major gap for a paper claiming safety relevance.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Mitigation at scale. Do lightweight guardrails (max CoT length; require citing extracted CUIs; numeric-field templates) materially reduce the CoT penalty without heavy RAG/few-shot? Any quick wins from your logs?

Multilingual grounding. Did concept-alignment trends hold uniformly across the nine languages, or were certain languages/tasks outliers?

Task sensitivity. Which task families (NER vs. diagnosis vs. QA) benefit the most from “short-CoT + grounding” constraints? Any per-task ablations to share?

Judge robustness. You report very high PPV. Did you also estimate NPV or inter-rater agreement between clinicians to bound false negatives on subtle omissions?
soundness: 3
strengths: Scale & realism. 95 models × 87 tasks from clinical notes/consults across nine languages—rare breadth for clinical NLP. 
Clear, robust main result. 86.3% of models drop under CoT; even top models show small but consistent declines—well beyond anecdote. 
Mechanistic clarity. The reasoning-length penalty and concept-grounding benefit together tell a coherent story about why clinical CoT fails.
summary: This paper presents (to my knowledge) the first large-scale, systematic assessment of Chain-of-Thought (CoT) prompting on real clinical text. The authors evaluate 95 LLMs across 87 clinical tasks in 9 languages under harmonized zero-shot vs. CoT protocols, then dig into why CoT underperforms via analyses of reasoning length, medical-concept grounding, lexical signatures, and an error taxonomy adjudicated by an LLM-as-judge with clinician validation. The headline result is clear and somewhat surprising: 86.3% of models do worse with CoT in this setting, with larger drops for weaker models. The paper then shows that longer chains correlate with bigger drops, and that better concept alignment mitigates the damage, while hallucination dominates the failure modes. Overall, this is a careful, comprehensive audit with actionable guidance for safe use of CoT on EHR-style text.
weaknesses: 1.Scope of interventions omitted. By design, the study doesn’t test RAG, self-consistency, or few-shot, which many practitioners would try first; a brief appendix pilot (even on a subset) could bound how much these mitigate the observed drops. 
2.Grounding extractor choice. Concept alignment relies on cTAKES; acknowledging its limits on non-English notes (and perhaps adding a multilingual UMLS/MedCAT check) would strengthen the multilingual claim. 
3.From correlation to causation. Length and alignment analyses are strong correlates; a small controlled intervention study (e.g., enforced brevity; prompting to cite extracted CUIs) would make the causal story even tighter.


Decision: Reject