StethoLM: Audio Language Model for Cardiopulmonary Analysis Across Clinical Tasks
Authors: Yishan Wang
Tsai-Ning Wang
Mathias Funk
Aaqib Saeed
Venue: Accepted by TMLR ()

Abstract: Listening to heart and lung sounds — auscultation — is one of the first and most fundamental steps in a clinical examination. Despite being fast and non-invasive, it demands years of experience to interpret subtle audio cues. Recent deep learning methods have made progress in automating cardiopulmonary sound analysis, yet most are restricted to simple classification and offer little clinical interpretability or decision support. We present StethoLM, the first audio–language model specialized for cardiopulmonary auscultation, capable of performing instruction-driven clinical tasks across the full spectrum of auscultation analysis. StethoLM integrates audio encoding with a medical language model backbone and is trained on StethoBench, a comprehensive benchmark comprising 77,027 instruction–response pairs synthesized from 16,125 labeled cardiopulmonary recordings spanning seven clinical task categories: binary classification, detection, reporting, reasoning, differential diagnosis, comparison, and location-based analysis. Through multi-stage training that combines supervised fine-tuning and direct preference optimization, StethoLM achieves substantial gains in performance and robustness on out-of-distribution data. Our work establishes a foundation for instruction-following AI systems in clinical auscultation.

Published in Transactions on Machine Learning Research (02/2026)
StethoLM: Audio Language Model for Cardiopulmonary Analysis Across Clinical Tasks
Yishan Wang Eindhoven University of Technology
y. V.
Tsai-Ning Wang
t.n.wang
Eindhoven Universily of Technology
Mathias Funk
m.funk@tue.t ml
Findhoven University of Technology
Aaqib Saeed
a.saeed@tue.w nl
Eindhoven University of Technology
Reviewed on OpenReview: https: // opwrr/veew.weee..m net/ forum? id- 49RuUH9JyJ
Abstract
Listening to heart and lung sounds auscultation - is one of the first and most fundamen- lal steps in clinical examination. Despile being fast and non-invasive, it demands years of experience to interpret subtle audio cues. Recent deep learning methods have made progress in automating cardiopulmonary sound analysis, yet most are restricted to simple classifica- tion and offer little clinical interpretability or decision support. We present StethoLM, the first audio-language model specialized for cardiopulmonary auscultation, capable of per- forming instruction-driven clinical tasks across the full spectrum of auscultation analysis. LL inlegrates audio encoding with medical language model backbone and trained on StethoBench, comprehensive benchmark comprising 77,027 response pairs synthesized from 16,125 labeled cardiopulmonary recordings spanning seven clinical task categories: binary classification, detection, reporting, reasoning, differential diagnosis, com- parison, and location-basee analysis. Through multi-stage training that combines supervised fine-tuning and direct preference optimization, StethoLM achieves substantial gains per- formance and robustness on out-of-distribution data. Our work establishes foundation for instrrcttion-folloiing AT systems in clinical auscultation.
1 Introduction
Auscultation, the practice of listening to heart and lung sounds with a stethoscope, has long been cor- nerstone of clinical examination. It enables physicians to detect murmurs, wheezes, crackles, and other acoustic markers that signal cardiopulmonary discase. Unlike imaging or laboratory testing, anscultation data can be obtained rapidly, non-invasively, and at low cost, making a uniquely scalable modality for scrcening, triage, and longitudinal monitoring, especially low-resource scttings. Yet, interpreting these sounds requires significant clinical expertise, with physicians needing years of training to reliably identify subtle abnormalities and develop sound clinical judgment. This expertise barrier limits the scalability of auscultation: many healthcare settings lack access to experienced clinicians, training is time-intensive, and diagnostic consistency varies with experience level. These human resource constraints motivate the devel- opment of AI systems that could augment. clinical expertise or extend auscultation capabilities to settings with limited specialist access.
To address this challenge, contemporary deep learning approaches have shown that automated analysis of cardiopulmonary sounds is feasible (Ma al. 2022 Acharya & Basu 2020a). However, the vast majority

---

Published in Transactions on Machine Learning Research (02/2026)
of rescarch has been constrained by classification paradigm: models are trained to perform narrow tasks (e.g.; "murmur vs. no murmur") and output single, fixed label (Zhang et al. 2024b). This paradigm fundamentally misaligns with clinical practice. The task of a clinician is to describe acoustic findings, compare rocordings over time, differentiate between similan-sounding pathologics, and reason about underlying causes. Models confined to single-label classification cannot support this diverse range of clinical reasoning lasks, which has limited their adoption in clinical scttings (Labkoff et al. 2024].
Recent advances in multimodal large language models (Xu et al., 2025a) offer opportunity to break free from this restrictive paradigm. By combining auditory perception with auguage-based reasoning, these models can, in principle, perform instruction-driven tasks and generate interpretable, free-text outputs that emulate clinical dialogue (Wang et al. 2023 However, while general-purpose models can process generic audio, they lack the specialized knowledge required for auscultation. Medical audio presents unique challenges distinct from music, speech, or environmental sounds: diagnostically relevant features reside in millisccond- scale temporal patterns (fine crackles <5ms vs. coarse crackles >10ms), subtle spectral characteristics (monophonic vs. polyphonic wheezing), and phase-specific timing (systolic vs. diastolic murmurs) (Kim) et al. 2021: Ma et al. 2022). These fine-grained acoustic distinctions, absent from general audio tasks, require specialized training on medical data. Existing medical multimodal models have focused primarily on vision-language tasks or limited audio Q&AA (Wang et al. 2025 Zhang et al. 2024b), leaving critical gap: no system is capable of performing the diverse clinical reasoning tasks required for expert-level cardiopulmonary auscultation.
Here, we introduce a new approach for AI in auscultation, shifting the focus from classification com- prehensive clinical reasoning. To this end, we present StethoLM (see Figure 1 for overview), the first audio-language model specialized for cardiopulmonary analysis, and StethoBench, benchmark designed to train and evaluate models on multiple clinical reasoning tasks within cardiopulmonary analysis. Our key contributions are:
StethoLM, audio-language model designed specifically for cardiopulmonary auscultation (see Figure D. capable of performing seven distinct, instruction-driven clinical reasoning tasks, from detection and reporting to differential diagnosis and comparisor
StethoBench, new, comprchensive benchmark of 77,027 instruction-respore pairs derived from 16,125 clinical recordings. It is the first benchmark designed to move beyond classification and facilitate the development of generalist auscultation model
Extensive experiments showing that specialized, intruction-basee training is critical. StethoLM substantially outperforms powerful, general-purpose multimodal models on both in-domain and out- of-domain data, establishing a new statc-of-the-art for multi-task clinical audio understanding.
2 Related work
Machine Learning for Cardiopulmonary Auscultation Machine learning has been extensively ap- plied to cardiopulmonary sounds, but historically, this research has been dominated by a classification paradigm. Early efforts used signal processing and shallow classifiers to distinguish between categories such as "normal vs. abnormal" heart sounds or "wheeze vs. crackle" in lung recordings (Ma et al. 2022). With the advent of deep learning, models trained on spectrogram representations achieved strong performance on public benchmarks, including the PhysioNet,/ CCinC Challenge for heart sound classification and the ICBHI Challenge for respiratory sound analysis (Li et al. 2022 More recent work has leveraged transfer learning and sele-supe audio representations to compensate for small dataset sizes, further improving accuracy on these tasks (Gairola et al. 2021 Acharya & Basu 2020a). Current approaches remain fragmented and label-centric, optimized only for narrow diagnostic categories, thus provide little support for broader clinical decision- decision-making.
I (Code: httpp://gttuub.co///ttaaaai//i/
3 Dataset: http:s////agggggoc.ccccc/c////ts
2

---

Published in Transactions on Machine Learning Research (02/2026)
A.
Cardiopulmonary Datasets
StethoBench
B.
Heart
Task Universe
Task Instruction:
Respiratory
Generate medical ceport
Classification
Annotation wheezes
based on this respiratory audio.
Annotation 2: right posterior
Reporting
Diagnosis:
Reasoning
COPD
AF
Identification
Response:
Asthma
N=77027
PVC
The lung sound recording
Covid-19
LVH
Gemini
Dif. Diagnosis
from the right postevinr
Prenumia
LVCD
Qwen
Annotation 1: murmur Diagnosis: Heart Fallure.
Location
chest reveals wheezing.
Bronchiectasis
AV block
Comparison
consistent with asthma.
A
A
R
log Spectrogram
Audio Toue
Tokenize & Embed
backbone
Loss
(LoRA Fine-tune)
encoder
Mapper
Task-Reponse
Overview of StethoLM and StethoBench. A. Automated benchmark creation pipeline, where off-the-shelf LLMs generate 77,027 task response pairs from 16,125 cardiopulmonary recordings and associatod annotations. B. Distribution of audio type and the examples of discase that StethoLM covers. C. MtethoLM architecture integrating audio text alignment, supervised fine-tuning for diverse clinical tasks.
Figure 1:
General-Purpose Audio-Language Models Foundation audio-language models have recently enabled reasoning over complex auditory inputs. Modols such as Pengi (Deshmukh et al. 2023 use audio embeddings as conditioning tokens to perform audio-to-lext lasks, while LTU (Gong el al. 2023) and more recent systems like GAMA (Ghosh al. (222)) and Qwen2.5-Omin (Xu et al. 2025a leverage pre-trained audio encoders to integrate auditory signals with language models, enabling tasks where the model follows natural-language instructions about audio. Yet these models primarily trained on cannera-puurooe audio, speech, environmental sounds. They are not designed for domain-specife tasks such as cardiopulmonary auscultation, where fine-gres gee-ggr acoustic patterns, and clinical metadata are critical. As result, their direct applicability to medical audio limited, motivating the need for domain-spocific audio-language modeling.
Multimodal Large Language Models for Healthcare Instruction-tuned large language models in healthcare, such as MedPaLM (Singhal et al. 2023 and ClinicaGPPT (Wang et al., 2023). demonstrate that LLMs can integrate domain knowledge and generate structured outputs for complex medical tasks. More recent multimodal systems, including MedGemma (Sellergren al. 2025) and MedRAX (Fallahpour al. 2025), extend these capabilities to visual language inputs, supporting applications such as diagnosis, report generation, and multi-step clinical reasoning. Together, these works illustrate the potential of leveraging LLMs to move beyond fragmented, label-centric outputs toward multitask clinical support. In the car- diopulmonary domain, models such as RespLLM (Zhang et al., 2024b) and preliminary Q&A systems have attempted to adapt LLMs for respiratory and cardiac audio (Wang et al. 2025), but they remain limited in task diversity and scope, underscoring the need for a more comprehensive, generalist approach.
Clinical Tasks in Medical AI: From Specialized to Generalist Systems clinical practice, aus- cultation involves multiple interconnected tasks. Physicians must assess whether the sounds are normal or abnormal, localize specific events (c.g., "S1 at mitral area," " "crackles during inspiration"), summarize find- ings in clinical notes, reason about underlying pathology, and compare recordings across visits. Existing work
3

---

Published in Transactions on Machine Learning Research (02/2026)
has trained medical AI models for these specialized tasks. Classification and detection models for cardiopul- monary sounds achieve strong performance specific sound identification (e-g., wheeze vs. crackle) (Kim)
eet al. 2021 Acharya & Basu 2020b). Report generation translating findings into structured text has been explored in radiology (Wang et al. 2024 Ouis & Akhloufi, 2024] but is not yet available for clinical
audio. Differential diagnosis systems generate ranked diagnosis lists from clinical data (McDuff et al., 2025
while explainable AT methods provide reasoning for model predictions (Sadeghi et al. 2024 both critical for clinical trust but typically isolated from other tasks. Longitudinal and comparative analysis, essential for tracking disease progression, has been studied primarily for EHR data (Cascarano et al., 2023 These ca- pabilities reflect distinct aspects of routine clinical work, yet they are rarely integrated. In cardiopulmonary auscultation, this fragmentation is particularly obvious: most systems address only single-label classification, leaving broader clinical lasks and workflows unsupported.
3 Problem Formulation
Binary Classification
Is this heart sound normal abnormal?
The heart sound abnormal.
Detection
The recording exhibits expiratory wheezes.
What the primary sound event this audio?
Report
The lung sound shows coarse crackles and bronchial breathing in the left poszerior region, consistent with bronchiectasia.
Generate summary the lung sound
findings.
Upper infection. Wheezing and dry cough the upper airway.
inflammation
Reasoning
Audio Spectrogram
Provide a diagnosis and explain your reasoning Differential Diagnosis
The wheezing heard could be caused by asthma, Other potential causes, though less likely, could include bronchitis or localized airway obstruction.
potential causes the lung
sound abnormalities.
StethoLM Audio-Language Model
Comparison
Although both audios lack murmurs, Audio 2 suggests an underlying cardiac abnormality, unlike Audio which represents normal heart sound.
Compare Audio 2 to Audio 1, What is the major diagnostic difference?
Location-based Analysis
The lateral right location indicates that the bronchiolkis is affecting the right lower or middle lobe, which may correlate with distribution
What is the significance location lateral right
Figure 2: Diverse clinical tasks supported by StethoLM. Instructions (left) represent realistic clinical queries, while responses (right) provide task-appropriate outputs ranging from binary decisions to complex diagnostic reasoning.
We formulate cardiopulmonary auscultation as conditional language modeling problem. Given an audio recording x E Rº and natural language instruction (gI....,94. .x) consisting of K tokens, the goal is to generate a textual response = (y)...,, yr. of T. tokens that accurately addresses the instruction. Our model, parameterized by 0, learns the conditional probability distribution pe(y|x, q).
Due to the sequential nature of language, this distribution decomposed autoregressively using the chain rule of probability:
L po 9) II po
(1)
where Yej (y1....,9jj1 are the previously generated tokens. This formulation differs fundamentally from traditional label-centric approaches, which learn a fixed mapping x + c C for a predefined set of classes C. Our approach instead enables the model to interpret diverse instructions and generate appropriate free-form responses.
The model is trained on ((x, N where each triplet contains an audio sample, an instruction, and the corresponding ground-truth response. Our objective is to maximize the likelihood of generating
4

---

Published in Transactions on Machine Learning Research (02/2026)
Table Summary of datasets included in StethoBench. The upper section lists in-distribution (TD) datasets used for training, validation, and testing. The lower section lists out-of-distribution (OOD) datasets used for generalization evaluation. OOD datasets differ from ID data in recording devices, patient populations, disease prevalence, clinical contexts, collection protocols, reflecting realistic deployment scenarios. "Samples" indicates the number of audio recordings. "Pairs" refers to the number of corresponding instruction--eesoons pairs.
Dataset Samples (train/val/test) Pairs Minutes Modality
Application
Task
In- Distrribuution (ID) Datasets
ICBHI
539 / 381
9,028
329
Respiratory
Abnormality Detection
KAUH
200 69 / 67
5,842
97
Mixed
Disease Classification
SPR
1,773 / 1,002 / 48
14,183
276
Respiratory
Respiratory Event Detection
CoughVID
1,223 / 830
9,844
283
Cough
COVID-19 Detection
CovidUK
2,000 / 1,000 / 1,000
13,356
378
Cough
COVID-19 Screening
ZCH
1,019 / 240
6,387
258
Cardiac
Heart Sound Analysis
CirCor
1,560 / 1,447
14,166
1,140
Cardiac
Murmur Detection
Dodtt---iistiitttion (OOD) Datasets
TR
120
856
141
Respiratory COPD Severity Assessment
CinC
442
900
174
Cardiac
Abnormality Detection
BMD
565
1,789
198
Cardiac
Heart Sound Classification
FluSense
300
376
Cough
Flu Delection
Total
16,125 125
77,027
3,283
All
Cardiorespiratory Monitoring
correct responses:
0" arg max 9
Σ log (o
(2)
This objective equivalent to minimizing the cross-entropy loss between the model predictions and the target tokens.
We consider instructions spanning seven clinical task categorics listed below. These tasks are grounded in established clinical workflows for auscultation (Labkoff et al. 2024 Fallahpour et al. 2025 Examples of these tasks are illustrated in Figure
Binary Classification: Determining whether sounds are normal or abnormal, whether abnormality (e.g., murmur) is present the most fundamental screening decision in auscultation
Identifying specific acoustic events (e.g., murmurs, crackles) pathological findings within recording
Detection:
Generating structured summaries of auscultatory findings for documentation in modical records, mirroring standard clinical note-taking
Clinical Reporting:
Diagnostic Reasoning: Explaining the relationship between observed acoustic patterns and under- lying pathophysiological mechanisms, with supporting evidence
Differential Diagnosis: Ranking potential conditions that could explain the observed findings, key step in clinical decision-making
Comparative Analysis: Comparing recordings, this task reflects longitudinal monitoring of patients
Location-Basso Analysis: Characterizing sounds at specific anatomical sites (c.g., mitral vs. aortic area), accounting for expected regional variations in normal findings
5

---

Published in Transactions on Machine Learning Research (02/2026)
4
StethoBench
We introduce StethoBench,, comprehensive multimodal benchmark for evaluating audio-language models cardiopulmonary auscultation tasks. StethoBench integrates publicly available datasets covering respiratory sounds, cardiac auscultation, and related conditions, transforming beterogeneous data sources into a unified instruction-eeppore format to enable evaluation across diverse clinical auscultation tasks.
4.1 Data Sources.
StethoBench aggregates seven cardiopulmonary audio datascts (Table 1 partitioned into in-domain datasets for training, validation, and evalualion, and oul-of-domain (OOD) datasets for assessing model robustness and generalization.
In- In-Domain Datasets:
ICBHI Respiratory Sound Database (Rocha et al. 2017): : 6,898 annotated respiratory cy- cles from 126 subjects, with labels for crackles, wheezes, and normal breath sounds across diverse auscultation locations.
KAUH (Fraiwan al. 2021 : Respiratory rocordings from 112 subjects with comprchensive anno- lations including diagnosis, adventitious sound types, chest zones, and demographic information.
COUGHVID (Orlandic et al.) 2021): Large-scale crowdsourced cough sound dataset with over 25,000 recordings from diverse geographic locations, annotated for COVID-19 status, severity, and respiratory health indicators by expert physicians.
CovidUK (Torresani et al. 2024): UK COVID-19 Vocal Audio Dataset, the largest PCR-referenced collection of respiratory audio recordings with 70,565 linked test results, including cough, exhalation, and speech samples alongside demographic and symptom data.
CirCor DigiScope (Oliveira et al. b: The largest pediatric heart sound dataset with 5,282 recordings from 1,568 subjects, featuring detailed murmur characterization (timing, shape, pitch, grading, quality) and manual segmentation of fundamental heart sounds.
SPRSound (Zhang et al. 2022): Pediatric respiratory database containing 2,683 recordings with 9,089 respiratory events from 292 participants, annotated at both record and event levels by expe- rienced pediatric physicians.
ZCHSound (Liu et al. 2024): Pediatric heart sound database from 1,259 participants, including 566 cases of congenital hcart discase (CHD) with precise discase diagnoses across multiple CHD calegories.
Out-of-Domain Datasets:
Respiratory@TR. (Alaan et al. 2020 SespiraaoryDDaaaaseeee dataset containing multi-channel (12 channels) lung sound recordings from 77 subjects diagnosed with asthma, bronchitis, and five COPD severity grades (COPDO-COPD4), validated by pulmonologists with reference to chest X-rays and pulmonary function tests.
CinC (Liu al. 2016): PhysioNet/ CeaaooNet/Computing in Cardiology Challenge 2016 dataset, comprising 4,430 heart sound recordings from 1,072 subjects across eight independent databases worldwide, annotated for normal/abnorman classification.
BMD-HS Ali al. 2024): BUET Multi-discase Heart Sound dataset with 864 recordings across six diagnostic categories (AS, AR, MR, MS, MD, Normal), featuring multi-label annotations for valvular heart. diseases with echocardiographic validation.

---

Published in Transactions on Machine Learning Research (02/2026)
FluSense (Al Hossain et al. 2020): Audio dataset from inffuenza-like illness surveillance platform, containing annotated recordings for nine respiratory event classes (cough, sneeze, sniffle, throat- clearing, speech, breathe, gasp, silence, and other) collected from diverse acoustic environments.
Each dataset includes audio recordings with rich metadata encompassing demographics, clinical diagnoses, auscultation locations, and pathology types. Cardiac datasets provide fine-grained annotations such murmur characteristics and segmentation boundaries, while respiratory datasets include both record-level diagnoses and event-level annotations for adventitious sounds.
Dut-of-Distribution Definition and Selection. We define out-of-distribution (OOD) data recordings that differ from training data in ways that reflect realistic deployment scenarios. These differences encom- pass multiple lypes of distribution shift: (1) Covariale changes in input distribution p(x) arising from different recording devices and protocols (e.g., TR's 12-channel multi-location recording vs. conventional single-point auscultation), patient populations (e.g., BMD's adult valvular disease patients vs. predomi- nantly pediatric populations CirCor and ZCHSound training data), or acoustic environments; (2) Label chant-shanges in outcome distribution p(y) reflecting different discase prevalence across populations; and (3) Semantic shift where the label space task definition changes, requiring the model recognize sound cate- gories absent from training (e.g., FluSense's focus on spontaneous respiratory events like sneezes, sniflles, and chroat-clearing, which are distinct from the pathological sounds crackles, wheezes, murmurs emphasizzed in clinical training data). This evaluation strategy tests whether domain-specialized training generalizes to the breadth of conditions encountered in clinical practice.
4.2 Instruction- Response Pair Generation
To create instruction-responee pairs from the audio metadata and labels, we employ a large language model- based synthetic data generation approach (Tang et al. 2023 [ernandez al.) (024). Specifically, use LLMs such GPT-4o (OpenAl 2024) to generate diverse, clinically grounded instruction-responss pairs from structured metadata.
Single-Audio Analysis. For each audio recording, we provide the LLM with the ground truth annotations (e.g., diagnosis, location, sound characteristies) and prompt it to generate six distinct instructin-respouss pairs covering different clinical tasks: (1) binary classification (e.g., normal vs. abnormal), (2) pathology identification, (3) clinical report generation, (4) diagnostic explanation, (5) differential diagnosis, and (6) location-based analysis.
Pairwise Comparison. In addition to single-audio analysis, we generate pairwise comparison tasks that require models to identify acoustic differences between two recordings. For each pair of audio recordings with different diagnoses, we prompt the LLM to generate two comparative instruction-response pairs that highlight the key acoustic and diagnostic differences between the recordings. Responses are constrained to a maximum of 40 words encourage concise, clinically relevant comparisons similar to real clinical discussions.
The complete prompt templates for both single-audio and pairwise tasks are provided in Appendix A Our approach follows established practices in instruction-tuning literature (Peng et al. 2023 Wang et al. 2022 where GPT-4o generated instruction data has been shown to produce high-quality training signals.
Quality Control. To assess the quality and clinical accuracy of generated instruction-respones pairs, we manually reviewed random sample of approximately 2% of the datasel to evaluate factual correctness, clinical relevance, and instruction diversity. For further identification of error types and error estimation, see sythetic data quality analysis in Appendix
4.3 Dataset Statistics
StethoBench comprises 16,125 audio samples totaling 3,283 minutes (54.7 hours), from which we gener- ated 77,027 instruction-response pairs spanning seven distinct task types (Table 1 The task distribution

---

Published in Transactions on Machine Learning Research (02/2026)
reflects clinical workflow diversity: identification tasks (25.9%), classification tasks (21.5%), report gener- ation (16.7%), clinical reasoning (11.8%), differential diagnosis (11.6%) (1.6%). pairwise comparison (4.6%), and location-based analysis (4.6%). Audio recordings vary substantially in duration across datasets, with in- domain recordings averaging between 5.83 seconds (CovidUK) and 22.89 seconds (CirCor), reflecting diverse clinical contexis from brief cough samples to extended cardiac auscultation recordings.
To ensure balanced representation across datasets, we applied sampling strategies large-scale datasets (e.g., COUGHVID, CovidUK), limiting each dataset's contribution to approximately 1,000-2,000 audio samples while preserving the original train-test splits provided by dataset creators. The final benchmark consists of approximately 13,000 in-domain samples (across 7 datasets) and 2,000 out-of-domain samples (across 4 datasets).
5
StethoLM
5.1
Model Architecture
StethoLM is a multimodal audio-language model composed of three core modules: an audio encoder E a projection network Mp, and language model backbone GLLM-
Given raw audio waveform x R² sampled at 16 kHz, it is first transformed into mel-spectrogram RFXT" with F 64 frequency bins. The audio encoder EA, Efficient.et pretrained on medical
sounds (Wang et al. 2025), processes the spectrogram to produce sequence of feature vectors:
haudio = EA(s) E R Xdd
(3)
where N2 is the number of feature frames and da 1,280 is the audio feature dimension.
The trainable projection network Mp, a multi-layer perceptron, maps these audio features into the language model's embedding space, transforming handoo into k = prefix tokens, each of dimension d = 4,096 matching the LLM's hidden dimension:
Zaddoo = Mp (handio))) E k
2025 as our language model backbone GLLM, chosen for its We use MedGemma-44BB (Sellergren et al. strong medical domain knowledge. Given an instruction q and target response = (B1,..., Bm) of length m, they are tokenized and embedded to produce Zinst E Rxdd and Zresp E mm respectively, where n the instruction length. During training, the complete input sequence is:
z auudoo Zistti Preep] E R (+++++m)
At inference, the model autoregressively generates response tokens yI..... Um conditioned on [zaudio Zinst]
using causal attention.
5.2 Training Methodology
We employ a two-stage training strategy: (1) Supervised Fine-Tuming (SFT) to teach the model the primary clinical tasks, followed by (2) an investigation into Dircct Preference Optimization (DPO) to refine response quality.
Supervised Fine-Tuning (SFT). In the SFT stage, each training sample consists of a triplet (xi, qiyy y:) where y; (y.. :.t............))) m.) is the ground-truth response sequence. The training objective is to maximize the likelihood of response tokens conditioned on the audio and instruction:
m:
(SF
logPo(yu.
W..j.x,, qi)
(6)
where N the number of training samples, 0 denotes all trainable parameters, Vi,<, (......... Mi, repre- sents preceding response tokens, and the loss is computed only over the my response token positions. During
8

---

Published in Transactions on Machine Learning Research (02/2026)
SFT, the parameters of the audio encoder Ελ and projection network Mp are fully updated. For the LLM backbone GLIM, we use parameler-effiiint fine-tuning with LoRA (Hu et al. 2022), which keeps the origi- nal weights frozen and introduces low-rank trainable adapter matrices, significantly reducing computational cost.
Preference Optimization with mDPO. Following SFT, we explored direct preference optimization (DPO) (Rafailov et al., 2024) to refine response quality. DPO requires & preference dataset Ppref
{xx, qi Yu,i, Vt,a)iii where Yw a preferred ("winner") response and yı is a dispreferred (*loser") re- sponse for the same context qx.q).
The DPO loss function directly optimizes the policy model to satisfy the preferences:
LDPO(TE;TTe)) = (. log (Blog
To (Yw x. q)
To x, q)
- B blog
Trr m x,
Trof (y: x, q)
where oof is a frozen reference model, a is the sigmoid function, and / is a temperature parameter controlling the deviation from the reference policy. This objective increases the likelihood of preferred responses Yu while decreasing that of dispreferred responses yt-
For multimodal settings, we adopt mDPO (Yu et al. 2024), which extends DPO to model modality- conditional preferences. The key principle is that preference strength should depend on input quality: when audio is clear, the model should confidently distinguish between responses; when audio is degraded, this confidence should decrease appropriately. mDPO achieves this by adding moddaltt--ondditonall loss term that encourages the model to assign higher likelihood to the same response when paired with clean audio versus degraded audio.
Coneretely, we construct preference pairs by sampling K candidate responses per training sample varying lemperalures (T E 1.3) and ranking them by BERTScore similarity to ground truth (highest Yw: lowest y1). This yields approximately 2400 preference pairs from the training set. During train- ing, we generate degraded audio views x on-the-fly via temporal cropping, frequency masking, or spectral perlurbation. As discussed in Section 7 mDPO did not improve performance despite successful preference learning during training, revealing challenges in applying preference optimization to medical audio-language modeling.
5.3 Implementation Details
We train StethoLM on a single NVIDIA H100 GPU using the AdamW optimizer (Loshchilov & Hutter 2017)
with learning rate of y x 10-5, weight decay À 0.01, and momentum cocfficients 3 (0.9. 0.98). The learning rate is linearly warmed up over the first 100 steps and then held constant. Training is performed for 30 epochs with a batch size of 16, during which the validation loss stabilizes. Model checkpoints are saved based on validation loss, and the best-performing checkpoint is used for evaluation.
Audio inputs are resampled to 16 kHz and segmented into 10-second clips, which are zero pero-padded or truncated as necessary. For pairwise comparison tasks, we concatenate two 4.5-second audio clips with a 1-second silent gap in-between, maintaining the 10-second total input length. During training, we apply random audio augmentations using Augly (Papakipos & Bitton 2022). The combined text input (instruction and response) is limited to maximum of 64 tokens. The audio encoder produces 1,280-dimensional features, which are passed through 3-layer MLP with hidden dimension of 2,560 to generate four audio prefix tokens, each of dimension 4,096. We apply LoRA (Hu et. al. 2022) lo the query (Q), key (K), value (V), and output (O) projection matrices across all attention layers of MedGemma- 4B-IT. The LoRA setup uses a rank of r = 16, scaling factor a = 32, and dropout probability p 0.1. The audio encoder and projection network are trained in full precision (FP32), while the language model trained with mixed precision (FP16/FP3)) for improved memory efficiency.
9

---

Published in Transactions on Machine Learning Research (02/2026)
6 Experiments
6.1
Evaluation Metrics
Evaluating free-text climical responses is challenging, as a response be phrased differently from reference yet remain clinically correct, or be semantically similar but diagnostically wrong. To address this, we employ two complementary metrics to assess both semantic fidelity and clinical correctness.
BERTScore F1. We use BERTScore (Zhang et al. 2019) to measure semantic similarity between gener- ated responses and ground-truth references. BERTScore computes token-level similarities using contextual embeddings from a pretrained language model, providing a more robust measure of semantic equivalence than n n-gram-basee metrics like BLEU or ROUGE. We use the F1 formulation, which balances precision (how much of the generation matches the reference) and recall (how much of the reference covered by the gen- eration). We employ DeBERTa-largo-ma attention mechanism and larger model capacity provide superior semantic understanding, particularly for domain-specific technical language. The MNLI (Multi-Gents Gentr Natural Language Inference) fine-tuning fur- ther improves the model's ability to assess semantic relationships in diverse contexts, making it well-suited for evaluating medical text where precise semantic distinctions matter for instance, distinguishing bet ween 'systolic murmur' and 'diastolic murmur' or 'carly inspiratory crackles' versus "late inspiratory crackles." We report BERTScore as percentages (0-100%)
(He al. 2021) the embedding model. DeBERTa's enhanced
Clinical Accuracy. (LLM-Judged). While BERTScore excels at mcasuring semantic similarity, it can penalize clinically valid responses that use synonyms (e.g., "basilar crepitations" vs. "crackles at the lung bases") or fail to detect responses that are fluent but factually incorrect. To directly evaluate diagnostic mcasure Climical Accuracy using statc-of-thc-art large language modcl, GPT-5 (Ope- Allidity/accurann automated evaluator. For each generated response, the LLM judge is provided with both the model's output and the ground-truth reference. Tt. is then prompted to make a binary determination (ges/no)) is the generated response clinically consistent with the reference? The prompt (see Appendix C is designed to ignore minor stylistic variations and focus strictly on whether the core clinical information is correctly conveyed. We report accuracy as the percentage of responses judged be correct (i.e., yes). This metric serves as a crucial proxy for diagnostic utility in real-world clinical applications. To validate robustness and add transparency to the current metric, we conducted sensitivity analyses across multiple judge models and prompt variations (Appendix D
nAl 2025).
Metric Selection and Complementarity.. We prioritize BERTScore in main results (with ROUGE-1 and METEOR in Appendix F because its contextualized embeddings better capture semantic equivalence in medical language than n-gram metrics recognizing, for instance, that "basilar crepitations" and "crackles at the lung bases" convey the same clinical meaning. This approach aligns with recent medical multimodal models (Sellergren et al. 2025 Fallahpour et al. 2025) and provides contimuous measurement of partial cor- rectness. However, semantic similarity does not guarantee diagnostic validity. response can be fluent and semantically coherent while containing clinical crrors-for example, correctly identifying abnormality (high BERTScore) but misclassifying the specific pathology (low accuracy). We therefore report both BERTScore and clinical accuracy complementary metrics: BERTScore captures linguistic and semantic alignment, while LLM-judged accuracy evaluates diagnostic correctness.
6.2 Baseline Models
We compare StethoLM against both general- purpose audio-language models and a text-only configuration to assess model performance and training effectiveness. All baselines are evaluated using identical experimental protocols and metrics.
Text-Only LLM Bascline. We evaluate McdGenma-4B-IT (Seleeggenn et al. 2025), the same language model backbone used in StethoLM, receiving only text instructions without audio input. This configuration is critical for interpreting BERTScore results, as contextual embeddings can yield relatively high scores
10

---

Published in Transactions on Machine Learning Research (02/2026)
Table 2: Performance comparison on the StethoBench benchmark. BertS denotes BERTScore and Acc denotes Accuracy (both in %). The best score for each metric per task highlighted in bold, and the second-best score is underlined. Note: "Gemini' refers to Demiii-225-Fllas
and 'Qwen" refers to Qwen2.5-Omni. Additional baseline comparisons with Qwen3-Omni (Xu et al. 2025b) and Audio Flamingo 3 (Goel et al. 2025) are provided in Appendix
Supplementary metrics (ROUGE-1, METEOR) and relaxed accuracy evaluation for DDx focusing on clinical plausibility are provided in Appendix
E
Models
Task
LLM
Pengi
LTU
Gama
Gemma3N
Gemini
Qwen
StethoLM
BertS
Acc
BertS
Acc
BertS
Acc
BertS
Acc
BertS
Acc
BertS
Acc
BertS
Acc
BertS
Acc
Classification
48.6
1.9
33.6
7.6
49.0
2.5
46.4
7.5
51.3
31.3
49.3
49.5
58.9
31.0
75.5
66.4
Detection
43.0
2.8
30.8
4.3
43.1
3.6
44.0
4.2
47.1
11.5
45.9
19.2
52.5
22.0
70.4
47.9
Reporting
41.1
2.2
30.7
4.1
45.1
2.0
44.7
2.0
47.5
9.2
49.8
13.2
56.7
12.7
72.8
36.2
Reasoning
41.8
0.0
27.6
1.0
43.7
4.0
46.5
5.3
45.2
9.0
47.6
22.2
55.6
29.0
71.4
44.1
DDx
42.3
3.3
26.6
4.1
44.9
4.0
44.6
4.0
44.5
43.6
13.2
19.8
67.7
30.6
Comparison
45.1
2.3
28.4
1.7
47.2
2.1
45.2
1.9
47.7
12.7
46.9
15.4
57.8
22.7
70.7
40.5
Location
14.4
1.3
26.4
1.1
46.4
4.4
16.0
2.2
49.0
11.0
45.6
14.3
54.6
11.3
72.0
36.8
Overall
43.8
2.8
29.2
3.4
45.6
3.2
45.3
3.9
47.5
12.6
47.0
21.0
56.5
21.2
71.8
47.8
(+s 40+) even without audio information when the model generates fluent, medically plausible text based solcly instruction context.
General Audio-Langaage Models. We compare against three audio-langus lando-l models trained on general audio: Pengi (Deshmukh et al.) 2023), audio-grounded language model using audio embeddings soft prompts for frozen language model; LTU (Gong et al. 2023). which integrates pretrained audio encoders with instruction-tuned LLMs; GAMA (Ghosh et al. 2024). generative audio-language model using audio prefix tokens; and Gemma3N (Google DeepMind 2025), a lightweight multimodal (audio image video text) instruction- tuned model that processes audio via native audio tokens rather than external encoders. These models represent state-of-the-art general audio understanding but lack specializod training on cardiopul- monary sounds.
Large Multimodal Models. We evaluate two foundation models with multimodal capabilities: Qwen2.5- Omni (Xu et al. 2025a an open-source audio-language model supporting diverse audio understanding tasks, and 2emini-2.5--lashh (Comanici al. 2025), Google's multimodal model with audio, video, image, and text understanding. These models with tens to hundreds of billions of parameters, trained on web-scale multi- modal data, represent a fundamentally different approach than specialized audio-language models. Rather than being optimized specifically for audio-text tasks, they are generalist systems designed to handle any combination modalitios. Comparing against these models reveals whether domain-spocializzd training on medical audio can compete with or surpass the emergent capabilities large-scale general-purposs systems.
6.3 In-Domain Performance
We evaluate all models on StethoBench's test set containing approximately 20,000 instruction-respone pairs spanning seven clinical task categories (Table 2. Performance is measured using both BERTScore and ground truth-generation match accuracy, providing complementary views of generation quality and clinical correctness. Additional evaluation using ROUGE-1 and METEOR metrics is provided in Table Appendix
StethoLM Achieves Substantial Gains Across All Tasks. StethoLM attains an overall BeRTScore of 71.8% and accuracy of 47.8% outperforming all bascline models (Table 2 Relative the strongest bascline, Qwen2.5-Omi (56.5% BERTScore, 21.2% accuracy), StethoLM yields absolute improvements of + +15.3 percentage points in BERTScore and +26.6 percentage points in accuracy. The model demonstrates
11

---

Published in Transactions on Machine Learning Research (02/2026)
superior performance across all seven task categories, with notably strong results classification (75.5% BERTScore, 66.4% accuracy) and detection lasks (70.4%, 47.9%). These consistent performance gains across task types ranging from binary classification to complex clinical reasoning provide evidence that domain-specializzed training on cardiopulmonary audio yiclds robust benefits beyond those achicvable through general-purpose audio understanding or large-scale pretraining alone.
Audio Grounding Is Essential for Clinically Meaningful Understanding. The text-only baseline (LLM) attains a moderate BERTScore (43.8%) despite minimal accuracy (2.8%), indicating that linguistic fluency and medical knowledge alone are insufficient for clinically valid reasoning. MedGemma-4B-IT IT, pre- trained extensively on medical text, can produce coherent responses containing appropriate terminology even without access to the audio signal. However, without acoustic grounding, these responses remain generic and fail to reflect the true respiratory and cardiac findings. This contrast highlights that high textual similarity does not equal diagnostic correctness, and incorporating the clinical accuracy metric is necessary.
Task Performance Reflects Inherent Complexity and Evaluation Challenges. Performance varies substantially across task calegories, revealing systematic differences in task difficulty (see Table 2). Binary classification tasks achieve the highest accuracy (66.4%) with relatively small BERTScore-accuraay dis- crepancies (9.1 points), consistent with their objective nature and unambiguous ground truth. Detection (70.4% BERTScore, 47.9% accuracy), reporting (72.8%, 36.2%), and reasoning tasks (71.4%, 44.1%) exhibit moderate accuracy with larger metric divergence. These generative tasks require selecting and synthesizing information: for reporting, the model must choose which acoustic findings to emphasize; for reasoning, which pathophysiological mechanisms to claborate. High BERTScore indicates the model generates clinically ap- propriate content, while lower accuracy reflects variation in how information is organized and emphasized compared to reference annotations synthesized from human labels. Differential diagnosis presents the great- est challenge (67.7% BERTScore, 30.6% accuracy). Beyond identifying relevant conditions, this task requires ranking them by likelihood, which is process with inherent subjectivity. We include this challenging task because differential diagnosis represents essential clinical reasoning. This performance pattern mirrors doc- umented variation in human diagnostic agreement. Studies in diagnostic pathology show that inter-rater concordance varies systematically with lask ambiguity: pathologists achieve 87 96% agreement on objective classifications (benign vs. invasive carcinoma) but only 48 84% on ambiguous intermediate diagnoses (EI-) more et al., 2016]. Our results follow this pattern, with accuracy declining tasks shift from objective classilication to subjective prioritization. The BERTScore-wccurayy divergence thus reflects both model limitations and inherent task subjectivity: the model generates semantically appropriate clinical content (captured by BERTScorre) while showing incomplete alignment with specific reference formulations (mea- sured by accuracy).
Domain Specialization Proves Essential for Clinical Audio Understanding. Models trained on general-purpose audio struggle significantly, with general audio-language models (Pengi, LTU, GAMA) achieving BERTScores of 29-45% and accuracy below 4%. This poor transfer indicates that representations learned from music, environmental sounds, and speech emphasize acoustic features fundamentally different from those critical for auscultation. In medical audio, diagnostically relevant information resides in subtle temporal patterns (crackle duration: <5ms vs >10ms distinguishes fine vs coarse), spectral characteristics (wheeze pitch: monophonic vs polyphonic), and phase-specific variations (systolic vs diastolic timing). These features are not present or relevant to general audio tasks.
Large-scale multimodal models provide some benefit through massive scale and diverse pretraining. Gemini- 2.5-Flash (47.0% BERTScore, 21.0% accuracy) and Qwen2.5-Omin Ommn (56.5% BERTScore, 21.2% accuracy) substantially outperform general audio-languee models despite lacking medical audio training. Qwen2.5- Omni, with explicit audio-language training on diverse audio data, approaches competitive BERTScore performance. However, both models fall substantially short of StethoLM in accuracy (~21% vs 47.8%), demonstrating that domnain-specialized training on cardiopulmonary audio provides irreplaceable benefits that cannot be fully compensated by scale alone. This finding aligns with recent evidence that medical foundation models require domain-specific data achieve clinical-grade performance (Singhal et al. 2023;
Moor et al. 2023
12

---

Published in Transactions on Machine Learning Research (02/2026)
Table 3: Out-of-Domain (OOD) performance comparison across datasets. BertS denotes BERTScore and Acc denotes Accuracy (both in %). The bost score for each metric per dataset high- lighted in bold, and the second-best score is underlined. Additional baseline comparisons with Qwen3-Omni
(Xu et al. (022bb and Audio Flamingo 3 (Goel et al.) 2025 are provided in Appendix Supplementary
metrics (ROUGE-1, METEOR) are provided in Appendix
Models
Dataset
LLM
Pengi
LTU
Gama
Gemma3N
Gemini
Qwen
StethoLM
BertS
Acc
BertS
Acc
BertS
Ace
BertS
Acc
BertS
Acc
BertS
Acc
BertS
Acc
BertS
Acc
TR
44.0
0.5
27.7
1.6
14.2
0.6
14.0
0.3
48.4
5.6
14.8
7.5
60.7
17.7
66.2
25.7
CinC
39.5
1.1
29.4
1.3
42.3
0.4
42.6
0.7
44.5
5.2
45.9
21.5
54.0
12.4
63.3
22.2
BMD
45.0
2.5
27.3
1.4
47.2
1.2
46.7
4.4
48.2
11.7
40.7
20.9
58.6
17.3
67.3
30.4
FluSense
45.8
0.3
30.6
15.6
45.8
9.4
47.3
22.3
50.7
9.4
52.3
14.3
59.4
37.1
61.5
23.2
Overall
43.6
1.1
28.8
5.0
44.9
2.9
45.2
6.9
48.0
8.0
45.9
16.1
58.2
21.1
64.8
25.2
6.4
Out-of-Domain Generalization
(TR), CinC2016, BMD-HS (BMD), and FluSense, testing generalization across different recording devices, acoustic environ- ments, and patient populations (see Table 3p. StethoLM achieves 64.8% BERTScore and 25.2% accuracy out-of-domain data, representing moderate BERTScore decline (7.0 percentage points) but substantial accuracy decrease (22.6 percentage points) relative to in-domain performance (71.8%, 47.8%). This diver- gence reveals partial generalization ion: the model maintains semantic understanding of clinical language and acoustic features but struggles with precise diagnostic mappings under distribution shift. Additional OOD ovaluation using ROUGE-1 and METEOR is provided in Table 10 Appendix
We evaluate StethoLM's robustness on four external datasots:
F
StethoL. achieves the best, overall performance four benchmarks. The model performs optimally TR (66.2% BERTScore, 25.7% accuracy), CinC (63.3%, 22.2%), and BMD-HS (67.3%, 30.4%), which feature high-quality recordings from controlled clinical settings with rigorous quality control protocols (Altan et al.) 2020 Ali et al. 2024). However, Qwen2. 10mn2.55-mmi sur- passes MMethoLM on FluSense accuracy (37.1% vs 23.2%), revealing critical limitations clomain-specialized training when confronted with challenging worldworld conditions. FluSense differs fundamentally from clin- ical datasets in capturing spontancous respiratory events (sneezes, sniffles, throat-clearing, gasps) that are absent from structured clinical auscultation, where protocols focus on controlled breathing cycles (A) Hos- sain et al.) 2020). This creates distribution mismatch: StethoLM's training provides little supervision for these everyday respiratory behaviors, while Qwee2.5-Ommii diverse general audio training encompasses such spontaneous events. Additionally, FluSense recordings exhibit environmental noise, variable quality from crowdsourced collection. Detailed failure analysis examining out-of-vocabulary sound events is pro- vided in Appeudix G
OOD datasets and maintains superiority three of
6.5 Zero-Shot Classification
Beyond instruction--flllwing tasks, we investigate whether StothoLM's Iearned representations can support zero-shot classification, i.e. predicting class labels the model has never been trained on. To do this, first generate descriptive report for each audio imput using SM. We then encode this report using off- the-shelf sentence encoder (text-enbeddinggwv from Sowen--Embedding (Zhang et al. 2025)) and compute cosine similarities with embeddings of all candidate class labels. The audio is assigned the label with highest similarity to its generated report. Accuracy is computed by comparing predicted labels against ground truth.
We compare StethoLM against AudioMAE (Huang et al. 2022). CLAP (Wu al. 2023), and Opera- CE (Zhang al.] 2024a), three representative pre-trained audio models that produce ficed-dieensional embeddings optimized for downstream classification. In contrast, StethoLM trained solely for generative
13

---

Published in Transactions on Machine Learning Research (02/2026)
Covid/Non-cowte UK Covid-19 (Exhale)
Covid/Non-covve covid UK Covid-19 (Cough)
Covid/Non- CoughVID (Cough)
COPD/Hcalthy ICBHI (Lung sounds)
100
93.3
87.2
89.4
69.6
Accuracy
56.5
55.1
62.9
63.5
55.4
59.9
55.66
54.1
40
20
(Linear-probing)
CLAP (Linear-probing)
Opera (Linearpprobing)
Figure 3: Zero-shot classification comparison with classification models. Accuracy across four binary tasks comparing zero-shot inference (StethoLM via report generation + text similarity) against supervised linear probing (AudioMAF, CLAP, Opera-CE... Tasks include COVID-19 detection from exhalation/cough recordings and COPD screening from lung sounds.
instruction-falloving without explicit classification objectives. Figure presents zero-shot accuracy across four classification tasks.
StethoLM demonstrates competitive zero-shot classification performance. Notably, StethoLM excels on exhalation-based COVID detection (69.6% vs. 56.5% CTAP), demonstrates competitive performance on cough-based COVID detection (63.5% vs. 64.8% CLAP), and achieves strong results on COPD detection (89.4% vs. 93.3% CLAP). These results suggest that generative instruction-tan custrcctonnttuiig captures clinically mean- ingful acoustic features transferable to classification tasks, despite explicit classification training.
6.6 Audio Grounding and Model Explainability
We amalyzed attention patterns when StethoLM generates responses to clini- cal instructions paired with cardiopul- monary audio on the lest set. The heatmap (see Figure 4 visualizes the at- tention matrix for layer 3, head 2, show- ing how query tokens attend to key to- kens across three distinct regions: audio embeddings (4 tokens), instruction text (14 tokens), and generated response (20 tokens). Black solid lines separate the au- dio region from instruction tokens, while red dashed lines mark where response generation begins. The pattern reveals that generated response tokens maintain substantial attention to the audio prefix throughout generation, confirming that the model actively processes acoustic in- formation during clinical reasoning at in- ference lime. The allention weights show clear differentiation between the three in- put modalities, with significant attention from response tokens back the audio embeddings.
Audio
67
Instruction (14)
Attention Weight
Query Tokens
-65
63
Response (20)
64
Audio Instruction (14)
Response (20)
(4)
Key Tokens
Figure 4: Attention patterns revealing audio grounding during response generation. Heatmap visualizes how gener- ated response tokens (vertical axis) attend to key tokens (horizon- tal axis). Black solid lines mark the audio-isttruction boundary; red dashed lines mark where response generation begins.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Inference Query
Ground Truth
StethoLM Generation
Classity this lung sound normal abnormal.
Abnormal.
Abnormal. The presence of adventitious suggests respiratory pathology.
type this?
patient likely the exhalation sound.
patient likely based exhalation pattern.
brief summary lung sound analysis.
The recording presents with possible wheezing. suggests potential airway
recording presents possible wheezing suggesting airway constriction.
Explain the reasaning behind the diagnosis of asthma.
Asthma is characterized by reversible airway obstruction and bronchial hyper resporsiiiessssssss manifest as wheezing.
Asthma is indicated by wheezing common finding obstructive airway diseases.
List three differential diagnoses this cough
Upper respiratory infection. Acute bronchitis. Imitant cough (e from smoke allergens).
Upper respiratory infection, 2, Acute bronchitis. Imtaat cough.
Which audio would prompt you to order an echocardiogram, and why?
Audio 2, suggestive of VSD, warrants an echocardiogram to confirm the diagnosis and assess defect's size and hemodynamic impact.
Audio as it suggests ventricular septal defect, potentially requining further cardiac evaluation with an echocardiogram.
Based on the posterior right location, what findings would support diagnosis of
Expected findings include diminished breath sounds, possible wheezes (especially on expiration).
The presence wheezes or prolonged expiration posterior right lung field would support diagnosis of COPD.
Figure 5: Qualitative analysis of StethoLM's clinical reasoning capabilities. Representative outputs demonstrate the model's ability to generate clinically coherent responses across diverse auscultation tasks.
6.7 Qualitative Analysis
To complement quantitative metrics, examine representative examples of StethoLM's responses across seven clinical task categories (see Figure 5). The model consistently generates clinically coherent responses that preserve diagnostic validity while often extending beyond ground-truth references with medically ap- propriate context. For instance, when classifying sounds as abnormal, the model correctly identifies abnor- mality and supplements with clinical reasoning ("The presence of adventitious sounds suggests a respiratory pathology"). Explanation tasks reveal integration of acoustic observations with pathophystological knowl- edge ("characterized by reversible airway obstruction and bronchial exper-responsiveness))) While these examples demonstrate strong performance, also conducted systematic failure case analysis to identify key limitations and improvement directions (Appendix G). Overall, StethoLM's behavior suggests suitability for clinical decision support rather than autonomous diagnosis, with its medical coherence and contextual reasoning valuable for augmenting clinician judgment while requiring expert supervision for high-stakes diagnostic sccnarios.
7 Ablations
To understand which design choices contribute most to StethoLM's performance, we conducted systematic ablation studies by modifying individual components while keeping all others fixed. We evaluate each variant both semantic similarity (BERTScore F1) and clinical accuracy (GPT-5 judged) to understand their complementary effects.
7.1 Component Ablations
Audio Contribution. Removing audio input inference results in substantial performance degradation
(BERTScore: 71.8% 57.8%; accuracy: 47.8% 28.5%), confirming that acoustic information is essential for clinical audio understanding tasks. To further validate this finding, conduct additional sanity check
15

---

Published in Transactions on Machine Learning Research (02/2026)
by shuffling the audio modality to create text-audio mismatch (Madaan et al. 2025 where each instruc- tion receives randomly selected audio recording from different sample in the test set. This maintains the imput modality structure while breaking the semantic correspondence between audio and instruction. Under shuffled pairing, performance remains substantially degraded (BERTScore: 71.8% - 63.6%; accu- racy: 47.8% - 31.9%), confirming that the model relies on meaningful audio-text correspondence rather than exploiting text-only patterns. Notably, while the base medical language model (MedGemma-44-IT) appropriately refuses answer audio-related questions without audio input (2.8% accuracy in Table 2 the wwwltiimoddll---ddpptet model continues generating climical responses even when audio is absent mis- matched. This behavior reflects a broader phenomenon in multimodal adaptation where safety mechanisms can be inadvertently compromised (Lee et al. 2024). While detailed safety analysis is beyond this work's scope, this observation highlights the importance of preserving refusal behaviors in clinical Al systems.
Audio Encoder Selection. We evaluate alternative audio encoders while maintaining all other archi- accuracy) and AST (71.8%, 47.5%) achieve performance comparable to the default encoder (71.%%, 47.8%). This robustness across encoder architec- tures suggests that the benefits of domaan-spoccaalzzd training are largely independent of the specific audio encoding method, provided the encoder caplures sufficient temporal and spectral information.
tectural components. Both CLAP (71.4% BERTScore, 47.0%
Language Model Backbone. The choice of language model backbone substantially impacts performance. The default medical-speciallieed backbone achieves 71.8% BERTScore and 47.8% accuracy. Tlama-3.2 3.2-3BB Instruct, gencral-purpose model, attains 70.4% BERTScore and 45.3% accuracy, representing modest degradation. Mediihi--Instruc, despite medical pretraining, yields 68.5% BERTScore and 43.4% accuracy, while the smaller model capacity and medical domain knowledge contribute to performance, with larger models demonstrating greater capacity to integrate audio-language representations even without extensive medical pretraining.
D---Seel---- Sallll-ewe--1.551 achieves 66.7% and 39.4%. These results indicate that both
Prefix Length Impact. We vary the number of audio prefix tokens (k) to assess the tradc-off between expressiveness and efficiency. The default configuration (k=4) achieves 71.8% BERTScore and 47.8% ac- curacy. Reducing to k=2 yields 70.7% and 45.0%. Increasing to k=8 attains 71.5% and 47.5%, showing minimal difference. These results indicate that k-4 provides an effective balance, with longer prefixes offer- ing diminishing returns while shorter prefixes compromise representational capacity.
7.2 Challenges in Preference Optimization
We investigated multimodal direct preference optimization (mDPO) using on-poliey sampling, where mul- tiple candidate responses were generated at different temperatures and ranked by BERTScore similarity to ground truth annotations. Training metries indicated successful preference learning: the model achieved 78% accuracy in preferring chosen over rejected responses and 71% accuracy in assigning higher likelihood to responses paired with clean versus degraded audio. However, evaluation the test set showed slight performance degradation (BERTScore: 71.8% + 69.8%; accuracy: 47.8% + 45.9%).
This failure likely reflects limitations in using BERTScore as a preference signal and challenges inherent to preference optimization in this domain. BERTScore captures semantic similarity and linguistic fluency but does not reliably reflect diagnostic correctness. may favor responses that are generically plausi- ble yet clinically imprecise (missing severity qualifiers, overgeneralizing findings, omitting anatomical, ttem- poral specificity). Additionally, supervised fine-tuning on 70k pneruccioo-responne pairs already provides strong supervision for audio-text alignment and clinical reasoning, potentially leaving limited headroom for preference-based refinement.
8 Discussion
StethoLM demonstrates that instruction-followin audio-language models can successfully perform diverse clinical reasoning tasks in cardiopulmonary auscultation. Our results reveal three critical insights that ad- vance understanding of multinodal medical AT. First, audio grounding is essential for clinical reasoning. The
16

---

Published in Transactions on Machine Learning Research (02/2026)
Table 4: Ablation study on StethoLM components. Each variant modifies one component of the MtethoLM configuration. BertS denotes BERTScore (%), and Acc denotes accuracy (%). All models are evaluated on the StethoBench benchmark.
Component
Configuration
BertS.
Acc.
StethoLM
Full model: Audio Text, all trainable
71.8
47.8
Audio removed at inference
57.8
28.5
Modality
Audio-text mismatch (shuffled pairing)
63.6
31.9
CLAP
71.4
47.0
Audio Encoder
AST
71.8
47.5
Haana-3.2-3B-Imm
70.4
45.3
LLM Backbone
MeddiPhi-Instrus
68.5
43.4
DeepSeck-R1 R1-Distilll-Qwen-1...5
66.7
39.4
k=2 tokens
70.7
45.0
Prefix Length
k=8 tokens
71.5
47.5
Preference Optimization
Multimodal DPO (mDPO)
69.8
45.9
improvement BERTScore (71.8% vs 43.8%) and in accuracy (47.8% vs 2.8%) when audio is incorporated demonstrates that acoustic information provides irreplaceable diagnostic evidence. Second, domain special- ization proves essential. StethoLM's improvement over the strongest baseline (Qwen2.5-Omn) demonstrates that representations learned from general audio, e.g., music, speech, environmental sounds, do not capture diagnostically relevant features in cardiopulmonary auscultation, such as temporal characteristics distin- guishing fine versus coarse crackles. Third, task characteristics fundamentally shape evaluation. The model excels at objective classification (66.4% accuracy) but struggles with differential diagnosis (30.6%), reflecting not just model limitations but the inherent subjectivity of certain clinical tasks where expert disagreement is common (Elmore al. 2016).
Clinical Decision Support: Capabilities and Constraints. StethoLM's ability to generate natural language explanations alongside diagnostic predictions addresses critical limitation of existing auscultation AI systems that produce only fixed categorical outputs. This interpretability enables several clinical appli- cations that extend beyond narrow classification. For triage and screeming, the model can process recordings to flag potential abnormalities with accompanying descriptions, supporting rapid initial assessment high- volume settings. For longitudinal monitoring, the model's comparison capabilities allow tracking disease progression through serial recordings. However, our results reveal significant constraints that preclude au- tonomous diagnostic deployment. An overall accuracy of 47.8% and the model'ss performance degradation out-of-domain data (25.2% accuracy) presents challenges for real-world deployment. Most concerning is the safety implication we observed during ablation studies: after multimodal training, the model generates diagnostic responses even when audio input is absent. This poses risks in clinical settings where sensor failures or data corruption could result in plausible but incorrect output (Li et al. 2023). These findings suggest a viable clinical role for StethoLM as augmentative decision support rather than autonomous system. In this paradigm, the model generates candidate interpretations and supporting reasoning that expert clini- cians review, verify, and refine. However, this augmentative role still requires prospective clinical validation to assess real-world utility, measure impact on clinician workflow, and identify potential failure modes in deployment settings.
Limitations. We acknowledge several limitations that constrain interpretation and generalization of our findings. First, our evaluation relies on single reference annotations, which may penalize valid alternatives in subjective tasks such as differential diagnosis ranking or clinical description generation. Sccond, StethoBench aggregates datasets of heterogeneous quality, annotation standards, and demographic coverage, with some geographic and population biases and variable recording conditions, which may affect performance and gener-
17

---

Published in Transactions on Machine Learning Research (02/2026)
alizability. Thirdly, our evaluation relies LLM-based judges for clinical accuracy assessment, which, while robust across multiple models and prompts, lacks expert clinician validation. Expert adjudication would be particularly valuable for ambiguous diagnostic tasks and would help calibrate performance expectations against human basclines. Finally, multimodal training can compromise refusal behavior, with the model occasionally producing responses without audio input rather than declining appropriately. These limitations indicate that while StethoLM advances instruction-folloinng AT for clinical auscultation, further work on real-world evaluation and safety validation is necessary before clinical application.
9 Conclusion
We presented StethoLM, the first audio-language model specialized for cardiopulmonary auscultation that performs instruction-driven clinical reasoning across the full spectrum of auscultation tasks. Through multi- stage training StethoBench, our comprehensive benchmark of 77,027 synthesized instruction-response pairs, StethoLM achieves substantial improvements over general-purpose audio-language models, demon- strating that domain-speciallzer training on clinical audio enables robust acoustic reasoning. Our key con- tributions include: (1) StethoLM's architecture and training methodology for audio-languago understanding in clinical auscultation; (2) StethoBench, a diverse benchmark spanning seven clinical task calegories; (3) comprehensive evaluation revealing both capabilities and limitations (e.g., challenges in differential diagnosis ranking, sensitivity to recording quality).
Future Directions. We identify several promising directions emerge from this work. First, multimodal integration: extending StethoLM to incorporate patient demographics and medical history could enable more holistic diagnostic reasoning that mirrors clinical practice. Second, interactive refinement: allowing clinicians to iteratively query the model (*Describe the timing of this sound," " "Could this be pneumonia instead?") would support exploratory clinical reasoning. Third, uncertainty quantification: explicit modeling of epistomic uncertainty would better calibrate the model's confidence and flag cases requiring expert review. Finally, prospective clinical validation: real-world deployment studies are essential to assess clinical utility, safety, and integration into existing workflows.
StethoLM establishes foundation for interuction-oollwiine Al systems in clinical auscultation, demonstrat- ing that oudio-langual models can move beyond narrow classification to support the diverse reasoning tasks clinicians perform. By combining domain-specialized training with interpretable natural language interfaces, this work takes a step toward more general, trustworthy, and clinically useful medical AL. As cardiopul- monary diseases remain among the leading causes of mortality worldwide, scalable AI tools that possess clinicians' diagnostic capabilities hold significant, promise for improving global health outcomes.
18

---

Published in Transactions on Machine Learning Research (02/2026)
Acknowledgments
This work was supported by the NWO AiNed Fellowship Grant awarded to A.S., and in part by Google.org and the Google Cloud Research Credits program through the Gemini Academic Program. We also acknowl- edge the use of the Dutch National Supercomputer Snellius for essential computational tasks.
References
Jyotibdha Acharya and Arindam Basu. Deep neural nelwork for respiratory sound classification in wearable devices enabled by patient specific model tuning. JEEE Transactions on Biomedical Circuits and Systems. pp. 1-1, 2020a. ISSN 1940-9990. doi: http://bbas..20.2981111 URL http:://xxxxxoo dod...doo..oo org/10..//..ooo...ooooo http:://dx.doo.or/111.... 1109/
TECAS. 2981172
Jyotibdha Acharya and Arindam Basu. Deep neural network for respiratory sound classification in wearable devices enabled by patient specific model tuning. IEEE transactions on biomedical circuits and systems, 14(3):535 544, 2020b.
Forsad Al Hossain, Andrew A Lover, George A Corey, Nicholas G Reich, and Tauhidur Rahman. Flusense: A contactless syndromic surveillance platform for influenza-like illness in hospital waiting areas. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 4(1):1-28, 2020.
Shams Nafisa Ali, Afia Zahin, Samiul Based Shuvo, Nusrat Binta Nizam, Shoyad 1bn Sabur Khan Nuhash, Sayeed Sajjad Razin, et al. Buet multi-disease heart sound dataset: A comprehensive auscultation dataset for developing compuler-aided diagnostic systems. ai preprint arXiv:2409.0072,, 2024.
Gökhan Altan, Yakup Kutlu, Ali Ömer Pekmezci, and Sadık Nural. Respiratorydattaasscttt: Copd severity analysis, 2020.
Anna Cascarano, Jordi Mur-Petit, Jeronimo Hernandezz Marina Camacho, Nina de Toro Fadie, Polyxeni Gkontra, Mare Chadeau- Hyam, Jordi Vitria, and Karim Lekadir. Machine and deep learning for longitudinal biomedical data: review of methods and applications. Artificial Intelligence Review, 56 (Suppl 2):1711-1771, 2023.
Gheorghe Comanici, Eric Bieber, Marcol Schackermann, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507. 06261, 2025.
Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. Pengi: An audio language model for audio tasks. Advances in Neural Information Processing Systems, 36:18090-18108, 2023.
Joann G Elmore, Gary M Longton, Patricia A Carney, Berta M Geller, Tracy Onega, Anna NA Tosteson, Heidi D Nelson, Margaret S Pepe, Kimberly H Allison, Stuart J Schnitt, et al. Diagnostic concordance among pathologists interpreting breast biopsy specimens. JAMA, 313(11):1122 1132, 2016.
Adibvafa Fallahpour, Jun Ma, Alif Munim, Hongwei Lyu, and Bo Wang. Medrax: Medical reasoning agent for chest x-ray, arXiv preprint or Xix::2502.02673 2025.
Mohammad Fraiwan, Luay Fraiwan, Basheer Khassawneh, and Ali Tbnian. A dataset of lung sounds recorded from the chest wall using an electronic stethoscope. Data in Brief, 35:106913, 2021.
Siddhartha Gairola, Francis Tom, Nipun Kwatra, and Mohit Jain. Respirenet: A deep neural network for accurately detecting abnormal lung sounds in limited data setting. Annual International Conference of the IEEE Engineering Medicine and Biology Society. IEEF Engineering Medicine and Biology Society. Annual International Conference, 2021:527 530, November 2021. ISSN 2375-7477. doi: 10.1109/
embc46164 20...22222..0 9630091. URL htaps://ddii.rrge/
1111099/EMC611444 2021 9630091
19

---

Published in Transactions on Machine Learning Research (02/2026)
Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. GAMA: A large audio-language model with advanced audio understanding and complex reasoning abilities. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (ods.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 6288 6313, Miami, Florida, USA, November 2024. Association for Computational Linguistics. URL https://aclantholl.0 .or.y..orr.. org 2024. emnlp-main. 361
Arushi Goel, Sreyan Ghosh, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao-Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, et al. Audio flamingo 3: Advancing audio intelligence with fully open large audio language models. arXiv preprint arXix:2507.08. 081888 2025.
Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass. Listen, think, and under- stand. arXiv preprint Xix:22300. 10790, 2023.
Google DeepMind. Gemma 3n c4b. https://huggingface co/ google/ Cartmm--nn-Eaa 2025. Multimodal instruction-tuned model supporting audio, image, video, and text.
Pengeheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations, 2021.
Alistair EW Hernandez et al. Large language models and synthetic health data: progress and prospects.
JAMTA Open, 7(4):00ael14, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojeiech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. Advances in Neural Information Processing Systems, 35:28708-2877 28720, 2022.
Yoonjoo Kim, YunKyong Hyon, Sung Soo Jung, Sunju Lee, Geon Yoo, Chacuk Chung, and Tacyoung Ha. Respiratory sound classification for crackles, wheezes, and rhonchi in the clinical field using deep learning. Scientific reports, 11(1):17186, 2021.
Steven Labkoff, Bilikis Oladimeji, Joseph Kannry, Anthony Solomonides, Russell Leftwich, Eileen Koski, Amanda I. Joseph, Monica LonezeGo, Tee A Fleisher, Kimberly Nolen, et al. Toward responsi- ble future: rocommendations for ai-enabled clinical decision support. Journal of the American Medical Informalics Associalion, (1(1)):2730 2739, 2024.
Seongyun Lee, Geewook Kim, Jiyeon Kim, Hyunji Lee, Hoyeon Chang, Sue Hyun Park, and Minjoon Seo. How does vision- language adaptation impact the safety of vision language models? arXie preprint arXiv:24100 10.07571 2024.
Bo Li, Peng Qi, Bo Liu, Shuai Di, Jingen Liu, Jiquan Pei, Jinfeng Yi, and Bowen Zhou. Trustworthy ai: From principles to practices. ACM Compuling Surveys, 55(9):1 46, 2023.
Zishen Li, Yi Chang, and Bjorn W Schuller. Cun-based heart sound classification with an imbalance- compensating weighted loss function. In Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 4934-4937, United States, 2022. IEEE. doi: 10.1109/ 01BBC44829.2022.9777
Chengyu Liu, David Springer, Qiao Li, Benjamin Moody, Ricardo Abad Juan, Francisco J Chorro, Francisco Castells, José Millet Roig, Ikaro Silva, Alistair EW Johnson, et al. An open access database for the evaluation of heart sound algorithms. Physiological Measurement, 37(1))21181, 2016.
Juntao Liu, Yang Li, Shihua Hu, Yunjic Chen, Yaoqin Wu, Lujie Tang, Yujian Hu, Yan Lin, Fengying Shen, Liang Chen, al. Zchsound: Open-source zju paediatric heart sound database with congenital heart dísease. IEFE Transactions on Biomedical Engineering, 2024.
20

---

Published in Transactions on Machine Learning Research (02/2026)
Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. are preprint arXi::1711.05100 2017.
Zhizhong Ma, Yuanhang Qiu, Feng Hou, Ruili Wang, Joanna Ting Wai Chu, and Christopher Bullen. Determining the best acoustic features for smoker identification. In ICASSP 2022 2022 IEEE Inter- national Conference on Acoustics, Specch and Signal Processing (ICASSP), pp. 8177-8181, 2022. doi: 10.1109/ICASSP 420.5
Divyam Madaan, Varshan Muhunthan, Kyunghyun Cho, and Sumit Chopra. Multi-modal data spectrum: Multi-modal datasets are multi-dimensional. arXiv preprint arXi::200.2234 294999 2025.
Daniel McDulf, Mike Schaekermann, Tao Tu, Anil Palepu, Amy Wang, Jake Garrison, Karan Singhal, Yash Sharma, Shekoofeh Azizi, Kavita Kulkarni, et al. Towards accurate differential diagnosis with large language models. Nature, pp. 7, 2025.
Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelligence. Nature, 616:259-2000 2023.
Jorge Oliveira, Francesco Renna, Paulo Dias Costa, Marcelo Nogueira, Cristina Oliveira, Carlos Ferreira, Alípio Jorge, Sandra Mattos, Thamine Hatem, Thiago Tavares, et al. The circor digiscope dataset: from murmur detection murmur classification. IEEE journal of biomedical and health informalics, 26(6): 2524-2532 2021.
OpenAl. Gpt-4o: Omni-modal language model. http:://oppaa...c//n/eeeeeelllee 9pt 2024.
OpenAI. Introducing gpt-5. http:://opppaa...oon/eeeeerr//// August 2025. Accessed: YYYY-MM-
DD.
Lara Orlandic, Tomas Teijeiro, and David Atienza. The coughvid crowdsourcing dataset, a corpus for the study of large-scale cough analysis algorithms. Scientific Data, 8(1):156, 2021.
Mohammed Yasser Ouis and Moulay A Akhloufi. Chestbiox-gen: contextual biomedical report generation from chest x-ray images using biogpt and co-attention mechanism. Frontiers in Imaging, 3:1373420, 2024.
Zoe Papakipos and Joanna Bitton. Augly: Data augmentations for robustness, 2022. URL https://arxiv: htg/abs/2201.06490
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpl-4. arXiv preprint Xrv::22304. 03277, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
BM Rocha, Dimitris Filos, Lea Mendes, Ioannis Vogiatzis, Eleni Perantoni, Evangelos Kaimakamis, Nat- siavas, Ana Oliveira, C Jácome, A Marques, et al. A respiratory sound database for the development of automated classification. In International conference biomedical and health informatics, pp. 33-37. Springer, 2017.
Zahra Sadeghi, Roohallah Alizadehsani, Mehmet Akif Cifci, Samina Kausar, Rizwan Rehman, Priyakshi Mahanta, Pranjal Kumar Bora, Ammar Almasri, Rami S Alkhawaldch, Sadiq Hussain, al. A review of explainable artificial intelligence in healtheare. Compulers and Electrical Engineering, 118:109370, 2024.
Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, Justin Chen, Fereshteh Mahvar, Liron Yatziv, Tiffany Chen, Bram Sterling, Stefanic Anna Baby, Susanna Maria Baby, Jeremy Lai, Samuel Schmidgall, Lu Yang, Kejia Chen, Per Bjornsson, Shashir Reddy, Ryan Brush, Kenneth Philbrick, Mercy Asiedu, Ines Mezer- reg, Howard Hu, Howard Yang, Richa Tiwari, Sunny Jansen, Preeti Singh, Yun Liu, Shekoofeh Azizi,
21

---

Published in Transactions on Machine Learning Research (02/2026)
Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Riviere, Louis Rouillard, Thomas Mesmard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Elena Buchatskaya, lean-Baptiste Alayrac, Dmitry Lepikhin, Vlad Feinberg, Scbastian Borgeaud, Alck Androev, Cassidy Hardin, Robert Dadashi, Léonard Hussenot, Armand Joulin, Olivier Bachem, Yossi Matias, Katherine Chou, Avinatan Hassidim, Kavi Goel, Clement Farabet, Joelle Barral, Tris Warkentin, Jonathon Shlens, David Fleet, Vic- tor Cotruta, Omar Sanseviero, Gus Martins, Phoebe Kirk, Anand Rao, Shravya Shetty, David F. Steiner, Can Kirmizibayrak,, Rory Pilgrim, Daniel Golden, and Lin Yang. Medgemma technical report, 2025. URL https::/arxxv.orr// org.abs/.rgg.///... 2507. 05201
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, al. Large language models encode clinical knowledge. Nature, 620:172-180, 2023.
Ruixiang Tang, Han Zou, Xiaoying Liu, Kai- Wei Chuang, Xiaotian Han, and Xia Hu Xic. Docs synthetic data generation of Ilms help clinical text mining? arXiv preprint or urXiv:2303.04300 2023.
Stefano Torresani, Mare Chaaean-Hyam, Graham S Cooke, Sara Croxford, Sam Fitzgerald-Smin Susan Hopkins, Srinivasa Vittal Katikireddi, Paul Kellam, Eva Kevei, et al. A large-scale and per-referenced vocal audio dataset for covid-19. Scientific Data, 11(1):664, 2024.
Guangyu Wang, Guoxing Yang, Zongxin Du, Longjum Fan, and Xiaohu Li. Clinicalgpt: Large language models finetuned with diverse medical data and comprehensive evaluation, 2023. URL https://arxiv. org/abs/2306
Tsai-Ning Wang, Lin-Lin Chen, Neil Zeghidour, and Aaqib Saeed. Careaqa: A cardiac and respiratory audio question answering model for open-ended diagnostic reasoning. In Xuhai Orson Xu, Edward Choi, Pankhuri Singhal, Walter Gerych, Shengpu Tang, Monica Agrawal, Adarsh Subbaswany, Flena Sizikova, Jessilyn Dunn, Roxana Daneshjou, Tasmic Sarker, Matthew McDermott, and Irene Chen (cds.), Proceed- ings of the sizth Conference on Health, Inference, and Learning, volume 287 of Proceedings of Machine Learning Research, pp. 231-241226 246. PMLR, 25-22 27 Jun 2025. URL htttp:://prcceeeeeee mlr. mlr wwwpp/r2s7/17111
wang25b.html
Xinyi Wang, Grazzicla Figueredo, Ruizhe Li, Wei Emma Zhang, Weitong Chen, and Xin Chen. A survey of deep learning-baad based radiology report generation using multimodal data. arxie preprint orXiv:2405. 1 12833, 2024.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Han- nanch Hajishirzi. Self-instruct: Aligning language models with self-generatdd instructions. arXiv preprint ar1iv:2212. 10560, 2022.
Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor KorgKKiikpatrick, and Shlomo Dubnov. Large- scale contrastive language-audio pretraining with feature fusion and keyword-to-captine augmentation. In TCASSP 2023-2023 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.200 2025a.
Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, Baosong Yang, Bin Zhang, Ziyang Ma, Xipin Wei, Shuai Bai, Keqin Chen, Xuejing Liu, Peng Wang, Mingkun Yang, Dayiheng Liu, Xingzhang Ren, Bo Zheng, Rui Men, Fan Zhou, Bowen Yu, Jianxin Yang, Le Yu, Jingren Zhou, and Junyang Lin. Qwen3-omni technical report, 2025b. URL https: https:///axxiv.. http://wrgg//bb/00.177555
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Ly, et al. Qwen3 technical report. arXiv preprint arXi:2205.003388 2025.
22

---

Published in Transactions on Machine Learning Research (02/2026)
Fei Yu, Hongbo Huang, Xuyin Zhou, Zhiyuan Zhang, Qiushi Wang, Junjie Wang, Songyang Liu, Kai Chen, Yichuan Shen, and Dahua Lin. mdpo: Conditional preference optimization for multimodal large language models. arXiv preprint arXiv:2406.114 11809. 2024
Qing Zhang, Jing Zhang, Jiajun Yuan, Huajie Huang, Yuhang Zhang, Baoqin Zhang, Gaomei Lv, Shuzhu Lin, Na Wang, Xin Liu, et al. Sprsound: Open-source sjtu pacdiatric respiratory sound databasc. IEEE Transactions on Biomedical Circuils and Systems, 16(5):867 881, 2022.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. Xiv preprint ar orXio:1904.0000 2019.
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2006. 051066, 2025.
Yuwei Zhang, Tong Xia, Jing Han, Yu Wu, Georgios Rizos, Yang Liu, Mohammed Mosuily, J Ch, and Cecilia Mascolo. Towards open respiratory acoustic foundation models: Pretraining and benchmarking, Advances in Neural Information Processing Systems, 01-27024-27055. 2024a.
Yuwei Zhang, Tong Xia, Aaqib Saeed, and Cecilia Mascolo. Resplim: Unilying audio and text with mul- timodal llms for generalized respiratory health prediction, 2024b. URL https: //arxi//avvvv/a/ wwwaaaiiii//aaaaaaaeeeeeaa org/abs/2410.
05361
23

---

Published in Transactions on Machine Learning Research (02/2026)
A
EntruuctionRResponse Generation Prompts
Below are the complete system prompts used to generate instruction-respons pairs from cardiopulmonary audio metadata (See Section 4.2 for details about data generation).
A.1
Lingle-Audio Analysis
L.M prompt for single-andio pair generation.
You are medical expert specializing in cardiopulmonary auscultation.
Based the following ground truth for lung sound recording: ['diagnosis' detagnosis'row[[diaae 'location:: row(location']], generate 6 diverse instruction-response pairs that simulate realistic climical tasks.
Each instruction should reflect a distinct clinical task such as:
1. Classifying audio as normal abnormal
2. Identifying the likely pathology
3. Generating a summary or report
4. Providing diagnosis and explanation Differential diagnosis (What could be causing this?) Location-based analysis
Return the output as JSON list with the following structure:
[
{ "instruction": "What pathology is suggested by this audio?",
"response": "Pneumonia."
{ "instruction": "What is the primary finding in this lung sound?", "response": "Crackles are present." },
Make sure cach pair distinct focus and clinically grounded the diagnosis row" (wwddngnssss"" Be concise and clinically accurate.
24

---

Published in Transactions on Machine Learning Research (02/2026)
A.2 Pairwise Comparison Prompt
L.M prompt pairwise comparison data generation.
You are medical expert specializing in lung sound interpretation.
You comparing two anonymized lung sound recordings. Each has associated ground truth information:
Audio
Diagnosis: {row [' ('dda'nonsis__ _
Audio 2:
Diagnosis: (row[' diagnodssssss__22 __']]
Generate two realistic Instruction-respones pairs simulating real clinical comparative analysis. Instructions should be phrased questions doctor medical traince may ask. The response should be concise (max 40 words).
Example JSON output and format:
[
{
"instruction": "How do the sounds in Audio 2 differ from those in Audio 1?", "response": "Audio features coarse crackles suggestive pneumonia, while Audio has abnormal sound."
{
"instruction": "Compare Audio 2 to Audio 1. What is the key acoustic difference?", "response": "Audio features wheezes."
}
Make sure comparisons are clinically relevant and accurately reflect the diagnostic differences between the two recordings.
25

---

Published in Transactions on Machine Learning Research (02/2026)
B Synthetic Data Quality Analysis
To validate the clinical reliability of our 3LM-generated instruction-respee pairs (Section 4.2) we con- ducted manual review to charactcrize error patterns and assess benchmark quality. This appondix details the methodology, observed patterns, and their implications.
B.1 Validation Methodology
One author with clinical training conducted a two-phase review. First, an exploratory phase examined multiple samples across task types to identify systematic error patterns. Second, a structured validation phase reviewed 10 randomly sampled instruction-response pairs per task type (70 total pairs) to characterize error frequency and severity. For each pair, the reviewer assessed: (1) Factual consistency: Does the response accurately reflect the ground-truth annotations (diagnosis, location, sound characteristics)! plausibility: Is the response medically reasonable given the available information? (3) Task appropriateness: Does the response match the instruction's requirements?
(2) Clinical
Samples were categorized as:
Accurate: fully consistent with ground truth,
Acoustic Hallucination: adds sound features not in annotations,
Clinical Over-Elaboration: adds plausible but unsupported clinical interpretations, or
Contradictions: factually inconsistent with ground truth.
B.2 Representative Examples
Table Examples of error patterns in synthetic instruction-response generation.
Ground Truth
Instruction
Generated Response
Error Type
What could be the differential diagno- Wheezing sis?
Diagnosis: Bron-
While the primary diagnosis is bronchitis, differentials could in- clude asthma pueumonia, given obstructive sounds like wheezing and rhonchi.
Acoustic halluci- nation (rhonchi added)
chitis, Sound:
Diagnosis: Pneu-
Explain your reason- ing for suspecting se- vere pneumonia.
The presence of coarse crackles, consistent with fluid accumulation and alveolar collapse, strongly sug- gests severe pneumonia.
Clinical elaboration (pathoplyysiology added)
over-
monia (severe),
Sounds: Crackles
Murmur:
No, Provide differential diagnosis.
Possible valvular dysfunction, my- ocardial disease, or congenital anomaly.
Contradiction (valvular dys- function typically causes murmurs)
Outcome:
Ab-
normal
B.3 Task-Specific Reliability
Table summarizes validation results. Low-entropy tasks (classification, detection, comparison) show 80- 90% accuracy, while generative tasks (reasoning, differential diagnosis) show 60-70% accuracy, with errors primarily from clinical over-elaboration.
B.4 Error Pattern Analysis
Acoustic Hallucination (2/70, 3%). The most concerning error involves adding acoustic features not annotations. For example, annotations specifying only "wheezing" might generate "wheezing and rhonchi."
26

---

Published in Transactions on Machine Learning Research (02/2026)
Table Synthetic data validation results by task type. Based on 10 samples per task (70 total).
Task
Accurate
Acoustic Halluc.
Clinical Over-elab.
Contradict.
Classification
8/10 (80%)
Detoction
9/10 (90%)
Reporting
8/10 (80%)
0
Reasoning
7/10 (70%)
2
1
DDx
6/10 (60%)
3
0
Comparison
9/10 (90%)
1
Q
Location
9/10 (90%)
0
Total
56/70 (80%)
2 (3%)
11 (16%)
(1%)
Importantly, hallucinated sounds are clinically related to true sounds (both obstructive airway indicators), suggesting the model conflates co-occurring pathological sounds rather than generating arbitrary features.
Clinical Over-Elaboration (11/70, 16%). The most frequent error adds clinically plausible but un- supported interpretations. Given only "crackles, pneumonia," " the model might add "consistent with fluid accumulation and alveolar collapse." " While medically accurate, these explanations extend beyond labels. This is most prevalent in differential diagnosis (3/10) and reasoning (2/10) tasks where instructions explic- itly request clinical explanations.
Contradictions (1/70, 1%). Rare instances include logical inconsistencies (suggesting "valvular dysfunc- tion" despite "no murmur" annotation). This represent occasional failures in logical consistency.
B.5 Implications for Benchmark Reliability
Our validation reveals task-dependent reliability: low-entropy tasks achieve 80-90% accuracy, while high- entropy tasks show 60-70% accuracy. Acoustic hallucination (3%) is rare but concerning. Clinical over- claboration (16%) is more common but less problematic. reflects medically appropriate interpretations that, while unsupported by minimal labels, demonstrate sound medical reasoning. Contradictions (1%) are rare.
27

---

Published in Transactions on Machine Learning Research (02/2026)
C
Clinical Accuracy Judgement prompt
Below is the complete prompts for clinical correctness evaluation (Sec Scction 6.1 for details).
LLM prompt clinical correctness evaluation.
For the given ground-truth and prediction, determine if the prediction reasonably captures the key information.
Minor stylistic differences, small missing details should NOT make it 'No: Respond ONLY in JSON with 'answer' as 'Yos' or 'No.. Do NOT provide any explanations.
Ground-truth: frow. get (eeggeeeeeeeeeeee ('response', 1)} Prediction: (row geot generated', "}}
28

---

Published in Transactions on Machine Learning Research (02/2026)
D Clinical Accuracy Robustness Analysis
To add robustness and transparency to the clinical accuracy metric (section 6.1 we evaluated 200 randomly sampled instances per task under two conditions: (1) varying judge models with fixed prompt, and (2) varying prompts with fixed judge (GPT-5-mini).
D.1 Prompt Variations
We designed three prompts with varying wording. Prompt 1 (baseline, shown in Appendix C instructs the judge to focus on key information while ignoring minor stylistic differences. Prompt 2 emphasizing clinical intent. Prompt 3 applies stricter criteria, requiring that predictions do NOT introduce unsupported clinical claims.
Prompt 2: Lenient (Clinical Intent Focus).
You are evaluating whether model output clinically reasonable. Given ground-truth response and prediction, decide whether the prediction captures the main clinical intent of the ground truth. Answer "Yes" if:
The main diagnosis or conclusion correct, AND
Any missing details would not meaningfully change clinical interpretation.
Ignore stylistic differences, phrasing, minor omissions. Do NOT penalize lack of detail unless it changes the clinical meaning. Respond ONLY in JSON with the key 'answer' set to "Yes" or 'No.. Do NOT provide any explana- lions.
Ground-truth: frow.get response', }}} Prediction: (row. get (' generated', }}}
Prompt 3: Striet (Clinical Consistency Focus).
You are evaluating a medical AI system. Given a ground-truth clinical response and model prediction, determine whether the prediction is clinically consistent with the ground truth.
Answer "Yes" ONLY if:
The core diagnosis, findings, or clinical conclusions are correct, AND
The prediction does NOT introduce unsupported clinical claims, overconfident diagnoses, incorrect reasoning.
Minor stylistic differences are acceptable. Missing or incorrect clinical information SHOULD result in 'No".
Respond ONLY in JSON with the key 'answer' set to "Yes' or 'No'. Do NOT provide any explana- tions.
Ground-truth:
(row.get, g'ete'rrsennee ('response', ")}
Prediction: frow. (row.get( generated', })}
D.2 Judge Model Sensitivity
Table (left) shows systematic variation across judges. Gemini-2.5-Pro (Comanici et al.) 2025) produces consistently higher scores, indicating a more permissive evaluation style, while GPT-5-mini (OpenA] 2025) and Qwen-Plus (Yang et al. 2025) align closely (differences within 0-6 points). Despite absolute differences.
29

---

Published in Transactions on Machine Learning Research (02/2026)
Table 7: Robustness analysis of clinical accuracy metric. Values in parentheses show deviation from the bascline (GPT-5-mini with Prompt 1). All results 200 random samples per task.
Task
Judge Model Variation
Prompt Variation
GPT-5-mini
Qwen- Plus
Gemini-2.5
Prompt
Prompt 2
Prompt: 3
Classification
69.5
73.5 (+10)
81.0 (+11.5)
69.5
64.0 (-5.3)
66.0 (-3.5)
Detection
46.0
46.0 (0.0)
54.5 (+8.5)
46.0
44.5 (-1.5)
44.5 (-1.5)
Report
32.0
34.0 (+2.0)
56.5 (+24.5)
32.0
30.5 (-1.5)
30.0 (-2.0)
Reasoning
45.0
51.0 (+6.0)
64.5 (+19.5)
45.0
45.0 (0.0)
46.0 (+1.0)
DDx
29.0
35.0 (46.0)
53.0 (424.0)
29.0
27.5 (-1.5)
26.0 (-3.0)
Comparison
39.5
41.0 (+1.5)
52.0 (412.55
39.5
36.0 (-3.5)
35.0 (-4.5)
Location
41.5
44.0 (42.5)
59.5 (+18.0)
41.5
37.5 (-4.0)
39.5 (-2.0)
relative task ordering remains consistent: classification achieves highest accuracy, while differential diagnosis remains most challenging.
D.3 Prompt Sensitivity
Table (right) reveals modest prompt sensitivity. Across all tasks, deviations from the baseline are small (typically within +5 points), with task rankings stable: classification highest (64.0 69.5%), differential di- agnosis lowest 096.0-29.0%)
30

---

Published in Transactions on Machine Learning Research (02/2026)
E
Additional Baseline Comparisons
Table 8: Additional baseline comparison. Performance in-domain and out-of-domain test sets. BertS
denotes BERTScore and Acc denotes Accuracy (both in %). The best score is bold, second-best is underlined.
Model
In- In-Domain
Out-of-Domain
Bert$
Acc
BertS
Acc
Qwen2.5-Oni
56.5
21.2
58.2
21.1
Qwen3-Omi
56.1
23.9
57.0
20.4
Audio Flamingo
56.2
21.4
56.7
18.3
StethoLM
71.8
47.8
64.8
25.2
To strengthen our baseline comparisons (Table
3
we evaluated StethoL.M against two additional recent
audio-lanee languala models: Qwen3-Omi (Xu et al. 2025b and Audio Flamingo (Goel et al. 2025
Models.
Qwen3-Omni: A 30B parameter Mixture-of Experts audio- ladio-language model trained on 20 million hours of audio data, supporting multimodal understanding and native real-time speech interaction Xu et al. b025b
Audio Flamingo 3: A fully open, state-of-the-art audio language model that unifies speech, sound, and music understanding with long-audio, multi-turn, and reasoning capabilities.
Results.
Table 8 shows that SLethoLM consistently oulperforms both additional baselines.
Analysis. Both Qwen3-Omen and Audio Flamingo 3 achieve performance comparable to Qwen2.5-Omni, with BERTScores in the 56-57% range and accuracy around 21-24% on in-domain test set. The similar performance levels across these diverse architectures (sequence-to-sequene) cross-attention based, and prefix- tuning approaches) further indicates that gencral-purpose audio pretraining does not capture the fino-grained acoustic patterns critical for clinical auscultation.
31

---

Published in Transactions on Machine Learning Research (02/2026)
F Supplementary Evaluation Metrics
F.1 ROUGE and METEOR
This section provides supplementary evaluation metrics (ROUGE-1 and METEOR) complementing the primary results in Section 6.3 and Section 6.4 These metries offer alternative perspectives on generation quality through u-gram overlap (ROUGE-1) and synonym-aware matching (METEOR).
Table 9: Performance comparison on StethoBench benchmark using ROUGE-1 and METEOR (complement- ing Table 2h. R-1 denotes ROUGE-1 and MET denotes METEOR (both in %). The best score for each metric per lask is highlighted in bold, and the second-best. score is underlined. Note: Gemini-2.. (irnii-255FFFs// and "Qwen" refers to Qwen2.5- Omni
'Gemini" refers to
Models
Task
LLM
Pengi
LTU
Gama
Gemma3N
Gemini
Qwen
StethoLM
R-1
MET
R-1
MET
R-1
MET
R-1
MRT
R-1
MET
R-1
MET
R-1
MRT
R-1
MET
Classification
20.8
15.1
4.0
0.8
31.0
20.7
18.0
12.1
29.7
19.5
14.4
16.4
33.8
22.5
50.2
36.6
Detection
13.9
9.1
2.6
0.8
19.2
10.2
15.0
9.3
22.7
15.7
12.7
14.7
24.7
13.9
43.4
32.3
Reporting
15.0
9.4
3.6
0.9
18.0
9.8
14.8
8.2
20.3
11.5
19.0
15.4
25.6
14.3
50.0
39.3
Reasoning
13.9
6.2
1.7
0.5
18.7
7.3
19.6
9.5
18.6
8.6
10.3
14.2
26.7
14.1
48.8
35.1
DDx
14.3
0.3
18.3
14.0
7.6
14.2
10.0
25.3
15.5
35.4
24.2
Comparison
17.6
11.8
1.3
0.7
23.7
11.0
13.4
7.2
20.1
12.2
11.5
16.7
26.8
16.0
54.5
40.7
Location
20.0
0.1
26.0
19.9
10.6
25.9
12.6
16.8
27.1
15.0
44.4
33.4
Overall
16.2
10.3
2.6
0.7
21.9
12.3
16.1
9.5
22.2
13.7
13.0
14.8
27.4
16.2
46.5
34.4
Out-of-Domain (OOD) performance using ROUGE-1 and METEOR (complementing Table 3 R-1 denotes ROUGE-1 and MET denotes METEOR (both in %). The best score for each metric per datasel is highlighted in bold, and the second-best score is underlined.
Table 10:
Models
Dataset
LLM
Pengi
LTU
Gama
Gemma3N
Gemini
Q Qwnn
StethoLM
R-1
MET
R-1
MET
R-1
MET
R-1
MET
R-1
MET
R-1
MET
R-1
MET
R-1
MET
TR
17.7
10.5
1.2
0.5
24.2
13.8
17.7
10.5
24.1
15.4
11.0
14.6
30.2
20.3
35.2
21.5
CinC
13.8
6.8
1.1
0.4
19.5
8.4
16.5
8.6
20.5
9.8
5.0
5.1
24.0
12.8
32.6
18.0
BMD
18.0
11.3
1.7
0.6
22.7
14.6
19.2
10.7
23.6
13.3
12.4
16.1
27.5
16.0
37.4
25.9
FluSense
16.3
12.5
0.6
23.8
12.9
19.2
12.2
22.8
16.2
14.0
30.5
18.4
25.4
17.6
Overall
16.8
10.2
1.5
0.5
22.3
12.7
18.2
10.5
22.8
13.2
11.5
14.5
27.5
16.3
32.7
20.8
F.2 Relaxed Evaluation for Differential Diagnosis
Differential diagnosis is inherently subjective, as it involves ranking conditions by likelihood, a process where expert agrcement varies substantially (Elmore al.] 20166). To account for this, we use a relaxed prompt that focuses on clinical plausibility rather than exact ranking match. Using this criterion, StethoLM achieves 35.5% accuracy DDx tasks test (compared to 30.6% before). The relaxed prompt. is shown below:
32

---

Published in Transactions on Machine Learning Research (02/2026)
LLM prompt for differential diaguosis (relaxed plausibiltt).
For the given differential diagnosis lask, assess whether the generated response clinically appropri- ate.
A response clinically appropriate if:
includes reasonable differential diagnoses that could explain the findings
2. It does not include clearly inappropriate unrelated conditions
Respond ONLY in JSON with 'answer' 'Yes' or 'No', No explanations.
Ground-truthe frow.get ( ('response', 1)} Prediction: (row. (row.get( (' generated', })}
33

---

Published in Transactions on Machine Learning Research (02/2026)
G Failure Case Analysis
Inference Query
Response
Fallure Analysis
What the diagnosis and explanation the auscultation findings?
Tricuspid regurgitation. Halosystolic,, harsh murmur at tricuspid area.
Diagnosis:
Both conditions produce holosystolic murmurs with similar timing and quality. Differentiation requires radiation patterns. This highlights the value of integrating auscultation with other cardlac findings.
Clinically Similar
Ventricular septal defect, characterized by harsh holosystolic murmur.
Describe the type sound heard.
Crackles and wet rales indicative of mucus in airways.
Fine-Grained Acoustic Discrimination: Wet vs. dry distinction requires subtle amplitude and spectral cues. Model correctly fluid presence. This suggests encoding may under-emphasize time-frequency characteristics distinguishing moisture content
crackles
audio
Crackles and dry. non-productive cough are present.
you determine correlation between these sounds Covid- 19?
Audio 2 correlates with Covid-19 (positive test), Audio is negative.
Systematic Comparison Reversal: Complete test status reversal indicates architectural limitation in comparison tasks, Suggests explicit indexing mechanisms multi-input scenarios.
Audio is associated with positive Covid-19 test, Audio negative.
Give diagnosis and explain the reasoning.
Diagnosis: Mild COVID-19. Dry cough typical.
Non-Specific Symptom Challenge: Dry cough in COVID-19, common cold, allergies, many other conditions, lacks distinctive acoustic markers. This error pattern suggests the model hasn't learned reliable COVID-spectfic acoustic signatures.
Upper infection. Dry caugh and mild severity support this.
What acoustic differences you identify between the two coughs?
Audio shows wet, produrtive cough with crackles, while Audio
Multi-Audio Tracking Limitation: Model correctly identifies wet/dry characteristics but reverses audio attribution. This architectural challenge, l.e. maintalining audio-label binding across sequential Inputs, may be addressed through explicit position encoding or separate encoder pathways.
dry.
Audio 1 has wet, productive sound, while Audio 2 is dry.
What is the primary sound event detected this respiratory recording?
The primary sound event is sniffle
Out-af-Protocol Sound Events:
FluSense includes
respiratory events (sniffles, gasps, sneezes, throat- clearing) largely absent from clinical auscultation training data, which focuses pathological sounds (crackles, wheezes, murmurs)..
The primarysound event expiratory wheezing.
Figure 6: Representative failure cases across major error categories. Each example shows the instruction, ground truth, model prediction, and analysis revealing the underlying failure pattern. These failure patterns complement the successful generation examples in Figure together providing complete characterization of model capabilit ies and limitations.
34


Reviews:
[{"review_id": "7WtSyE3L2u", "reviewer": "Reviewer_fppt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper focuses on cardiopulmonary auscultation beyond classification tasks. It proposes StethoLM, an audio-language model, and StethoBench, a benchmark with 77k instruction-response pairs across seven clinical task categories including binary classification, detection, reporting, reasoning, differential diagnosis, comparison and location-based analysis.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper claims that StethoLM achieves gains in performance on IID and OOD tasks which has been supported by the overall evaluation.", "audience_yn": "Yes", "audience_explanation": "Both the proposed model, which is simple yet effective, and the benchmark will be of interest to the community. It is important to consider tasks that are aligned with clinical practice for this application and this work will be a step towards that direction.", "requested_changes": "Although the overall paper is well-written, I believe the following changes are essential to improve the positioning and evaluation of the paper:\n- It is currently unclear what does out of distribution mean in the evaluated setting and why are the evaluated datasets out of distribution. Is it a shift in $p(x)$ while keeping $p(y|x)$ constant or $p(y|x)$ shifts? How are the in and out of distribution datasets chosen?\n- The claim that audio grounding is essential is not fully supported in the paper. While the paper does evaluate with audio and language baselines and removes audio altogether during inference in ablation, I'd suggest to conduct sanity checks from recent multi-modal papers [1,2]. Removal can often be considered out of distribution by the model and thus shuffling the input modality is a better choice. In addition it is important to consider other associations with the ground truth and label to ensure the necessity of audio modality.  \n- The baseline models can be strengthened to position the work better. I'd suggest to compare with Qwen3-Omni [3], given the strong performance of Qwen2.5-Omni in the experiments. The paper should also include a comparison with the multi-modal large language models discussed in the related work section. \n- The paper should elaborate on why performance optimization does not improve audio language model performance in section 7.2. It'd also be useful to have the ablation comparison between base model, SFT and DPO across multiple tasks to support this discussion. \n- The paper should discuss more on the metric choices. For instance, why were BertScore preferred for main paper while Rouge and Meteor where positioned in the appendix? The paper should also elaborate on the discrepancies in BertScore and accuracy for both IID and OOD tasks. \n- A minor comment is to follow the same template for Table 2 and Table 3. Either having models as the first row or the first column will make it easier for the readers to follow. \n\nReferences.   \n[1] Gu et al. The Illusion of Readiness: Stress Testing Large Frontier Models on Multimodal Medical Benchmarks.  \n[2] Madaan et al. Multi-modal Data Spectrum: Multi-modal Datasets are Multi-dimensional.    \n[3] Xu et al. Qwen3-Omni Technical Report", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YafmVNPnFD", "reviewer": "Reviewer_VQyZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces StethoLM, an audio language model specialized for cardiopulmonary auscultation that moves beyond narrow classification toward instruction-following clinical reasoning. The authors also present StethoBench, a large benchmark of over 77k instruction response pairs derived from diverse heart and lung sound datasets, covering tasks such as reporting, reasoning, differential diagnosis, comparison, and location-based analysis. The work demonstrates that domain-specific audio language training yields substantial gains over general-purpose multimodal models, including stronger performance on out-of-distribution data. Strengths include the clear motivation, comprehensive benchmark design, strong experimental evaluation, and thoughtful discussion of limitations. A key weakness is reliance on synthetic instruction data and automated clinical evaluation, which raises questions about real-world clinical validity.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are well supported by extensive quantitative and qualitative evaluation. The paper includes comparisons against strong baselines, ablation studies, and both in-domain and out-of-domain testing. The results consistently show that StethoLM benefits from domain-specific audio language training and outperforms general audio language models. The authors are careful to contextualize limitations, particularly for subjective tasks such as differential diagnosis, and do not overclaim clinical readiness.", "audience_yn": "Yes", "audience_explanation": "The paper will be of interest to researchers working on multimodal learning, foundation models in healthcare, and audio understanding. Its focus on instruction-following models, benchmark construction, and evaluation challenges in medical AI aligns well with TMLR’s audience, even for readers not directly focused on clinical audio.", "requested_changes": "1. Clarify the clinical reliability of synthetic instruction response generation by expanding the manual validation process or providing additional analysis of error patterns tied to synthetic data generation. This is important for strengthening confidence in the benchmark.\n\n2. Provide more detail on the LLM-based clinical accuracy evaluation, including potential biases of the judge model and sensitivity to prompt design. This would improve transparency around the reported accuracy metric.\n\n3. Consider adding a small expert-annotated evaluation subset or inter-rater comparison to better ground the results in real clinical practice. This would strengthen the paper, but may not be strictly required for acceptance.", "broader_impact_concerns": "The paper appropriately frames StethoLM as a clinical decision support tool rather than an autonomous diagnostic system. A notable concern is the model's tendency to generate confident outputs even when audio input is missing, which could pose safety risks in deployment settings. While this issue is acknowledged, a clearer discussion of mitigation strategies, such as refusal mechanisms or uncertainty estimation, would strengthen the broader impact discussion.", "additional_comments": "This is a well-written and carefully executed paper that makes a meaningful contribution to medical audio language modeling. The benchmark and task framing are particularly valuable and likely to influence future work in the area. With minor clarifications around evaluation and safety considerations, the work would be a strong fit for TMLR.", "extra_scores": {}, "extra_text": {}}, {"review_id": "RuhAFalvx6", "reviewer": "Reviewer_EDpX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces StethoLM, an audio–language model for cardiopulmonary auscultation, and StethoBench, a large synthetic yet systematically assembled benchmark for instruction-following on medical audio. StethoLM links an EfficientNet audio encoder to a MedGemma-4B-IT backbone and treats auscultation as conditional language modeling, generating reports, explanations, and differential diagnoses rather than single labels. StethoBench aggregates seven public datasets into 16k recordings and 77k instruction–response pairs across seven clinically motivated task families.\n\nEmpirically, StethoLM clearly outperforms general audio–language models and large multimodal systems, achieving 71.8% BERTScore and 47.8% accuracy on in-domain data, compared to 56.5% / 21.2% for Qwen-2.5-Omni and near-chance performance for general audio models. Ablations confirm the need for audio input and show benefits from a medical LLM backbone. The model generalizes well to several external clinical datasets and shows competitive zero-shot screening performance, though it struggles on noisy “in-the-wild” audio (FluSense) and on differential diagnosis ranking, and multimodal DPO slightly degrades performance.\n\nI see the main strengths as: (i) a clear formulation of auscultation as instruction-following language modeling; (ii) a substantial and reusable benchmark that unifies diverse datasets into one task space; (iii) thorough comparisons against both audio–language and large multimodal baselines; and (iv) careful analysis of safety and alignment issues, including the observation that multimodal fine-tuning can erode refusal behavior. \n\nThe main weaknesses are: (i) limited robustness to unconstrained real-world audio and non-pathological events; (ii) only partial treatment of how to mitigate the safety issues revealed by ablations; and (iii) a still somewhat under-explored story around the failure of DPO and the evaluation of inherently subjective tasks like differential diagnosis.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Most central claims are well supported. Table 2 shows clear state-of-the-art performance on StethoBench, with StethoLM substantially outperforming both general audio–language models and large multimodal models across all task types, supporting its positioning as a cardiopulmonary “generalist.” The comparison to general-purpose audio models is strong, as those trained on music or environmental audio transfer poorly, consistent with the need for domain-specific auscultation cues.\n\nGeneralization results are balanced: StethoLM is usually best or second-best on external clinical datasets (TR, CinC, BMD), while performing worse on FluSense—a limitation the authors explicitly acknowledge. Ablations further validate design choices, showing audio is essential, medical LLM backbones provide modest gains, and mDPO degrades performance. Zero-shot binary classification results indicate reasonable transfer to downstream screening tasks.\n\nWhile some clinical implications exceed what offline synthetic-label evaluation can prove, the need for prospective validation is appropriately noted. Overall, the evidence is consistent and proportionate to the claims.", "audience_yn": "Yes", "audience_explanation": "This work should interest multiple TMLR audiences. For multimodal and audio–language modeling researchers, it provides a clear case study showing that domain-specific audio and a medical LLM backbone outperform general audio–language and large multimodal models, supported by detailed task-wise results. For medical AI, the shift to instruction-driven clinical reasoning and the introduction of StethoBench offer a solid foundation for developing medical audio agents.\n\nThe paper is also relevant to alignment and safety readers: multimodal fine-tuning’s erosion of refusal behavior and the negative multimodal DPO result provide concrete examples of safety regressions and optimization pitfalls. Overall, the benchmark, methods, and safety observations fit well within TMLR’s scope.", "requested_changes": "1. Expand the analysis of safety and hallucination behavior.\nThe ablation where audio is removed but the multimodal model still produces diagnostic responses is an important safety signal. It would help to unpack this a bit more: for example, whether the projection network or training recipe effectively forces “non-empty” audio features even for silence/null inputs, and whether the refusal degradation appears uniformly across tasks or concentrates in certain instruction types (e.g., explicit “diagnose” prompts). If space permits, a small mitigation experiment, such as adding explicit “null audio → refusal” examples in SFT, or training on corrupted audio with refusal targets would make the claim that StethoLM is a useful foundation for safer downstream systems more convincing.\n\n2. Clarify the DPO setup and discuss why it may have failed.\nThe multimodal DPO experiments are interesting, but currently a bit under-specified. Since the preference dataset is built by ranking sampled responses with BERTScore against the ground truth, it would be good to spell out the size of this dataset, how many samples per prompt were used, and whether any filtering on length/diversity was applied. A short discussion of why BERTScore may be a problematic proxy for “clinical quality” (e.g., favoring generic but semantically close text, ignoring subtle diagnostic errors) and whether an LLM-based judge or task-specific clinical metric might be more appropriate would add value for readers thinking about DPO in medical settings.\n\n3. Provide more quantitative detail on StethoBench quality control.\nThe paper notes that around 2% of the synthetic instruction–response pairs were manually reviewed. It would be helpful to report at least an approximate error rate observed in that sample, and to categorize common error types (e.g., hallucinated findings, incorrect anatomy, inconsistent severity descriptions). This would give readers a clearer sense of how noisy the synthetic labels are and where future improvements to the benchmark might be most impactful.\n\n4. Improve consistency in baseline naming and metrics.\nIn the text, the text-only baseline is described as MedGemma-4B-IT, while in Table 2 it appears under a different label (e.g., “Gemma3N”). Aligning the naming so that readers can immediately see that this column corresponds to the MedGemma backbone used in StethoLM would reduce confusion. Relatedly, the discussion later mentions an “overall accuracy of 44.6%” and “24.6% on out-of-domain data”, whereas Tables 2 and 3 report 47.8% and 25.2% for StethoLM. If these numbers correspond to a slightly different evaluation (e.g., clinical accuracy on a subset of tasks), it would be good to clarify the relationship.\n\n5. Refine evaluation of differential diagnosis.\nThe differential diagnosis task is inherently subjective because it involves ranking plausible conditions, and the paper already notes this. Since the reported top-1 accuracy is only 30.6% while BERTScore is much higher, I would encourage the authors to consider a complementary metric that gives partial credit, such as top-K accuracy (does the reference diagnosis appear anywhere in the list) or a more tailored LLM-judge prompt that rewards clinically reasonable lists even when the exact order differs. Even a small additional analysis in this direction would help readers interpret the current numbers.", "broader_impact_concerns": "StethoLM presents several deployment risks. Multimodal fine-tuning causes a loss of refusal, leading the model to give confident diagnoses even without valid audio, which is unsafe for any autonomous clinical use. Its fluent explanations may amplify automation bias, making incorrect outputs more persuasive. Generalization drops on “in-the-wild’’ audio, limiting reliability in home or low-resource settings. Finally, StethoBench’s mixed data sources introduce potential fairness and demographic biases, which warrant caution and future evaluation.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)