QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Training
Authors: Wei Dai
Peilin Chen
Chanakya Ekbote
Paul Pu Liang
Venue: NeurIPS 2025 oral ()
Keywords: machine learning
healthcare
reinforcement learning
multimodal learning
AI for healthcare
clinical AI
time series modeling
natural language processing
Primary area: applications
TL;DR: We built a generalist clinical foundation model across both time series and vision modalities with a novel RL training algorithm named DRPO.

Abstract: Clinical decision‑making routinely demands reasoning over heterogeneous data, yet existing multimodal language models (MLLMs) remain largely vision‑centric and fail to generalize across clinical specialties. To bridge this gap, we introduce QoQ-Med-7B/32B, the first open generalist clinical foundation model that jointly reasons across medical images, time‑series signals, and text reports. QoQ-Med is trained with Domain‑aware Relative Policy Optimization (DRPO), a novel reinforcement‑learning objective that hierarchically scales normalized rewards according to domain rarity and modality difficulty, mitigating performance imbalance caused by skewed clinical data distributions. Trained on 2.61 million instruction tuning pairs spanning 9 clinical domains, we show that DRPO training boosts diagnostic performance by 43% in macro‑F1 on average across all visual domains as compared to other critic-free training methods like GRPO. Furthermore, with QoQ-Med trained on intensive segmentation data, it is able to highlight salient regions related to the diagnosis, with an IoU 10x higher than open models while reaching the performance of OpenAI o4-mini. To foster reproducibility and downstream research, we release (i) the full model weights, (ii) the modular training pipeline, and (iii) all intermediate reasoning traces.

QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Training
Wei Dai, Peilin Chen, Chanakya Ekbote, Paul Pu Liang MIT Media Lab and MIT EECS
{dvdai, peili, cekbote, ppliang) @mit.
Abstract
Clinical decision decision-making making routinely demands reasoning heterogeneous data, yet existing multimodal language models (MLLMs) remain largely vision-centric and fail to generalize across clinical specialties. To bridge this gap, introduce QoQ-Med-7B/32B, the first open generalist elinical foundation model that jointly reasons across medical images, time-series signals, and text reports. QoQ-Med is trained with Domain-aware Relative Policy Optimization (DRPO), a novel ciinorcement--earmingg objective that hierarchically scales normalized rewards cording to domain rarity and modality difficulty, mitigating performance imbalance caused by skewed clinical data distributions. Trained 2.61 million instruction tuning pairs spanning 9 clinical domains, we show that DRPO training boosts diagnostic performance by 43% in macro-F1 on average across all visual domains as compared to other critic-free training methods like GRPO. Furthermore, with QoQ-Med trained on intensive segmentation data, is able to highlight salient regions related to the diagnosis, with an IoU 10x higher than open models while reaching the performance of OpenAl 04-mini. To foster reproducibility and down- stream research, we release (i) the full model weights, (ii) the modular training pipeline, and (iii) all intermediate reasoning traces at this link
1 Introduction
Clinical diagnosis has evolved significantly over the past decade, with numerous computational mod-
els developed assist clinicians in organizing patient records [87 [5 formulating diagnoses [19 42).
interpreting clinical images [26 83), and other clinical tasks [2] These advancements have substan-
tially improved healthcare efficiency and accuracy across multiple specialties (7]. Recently, the
emergence of powerful generalist reasoning models such as OpenAl o3 61 and Deepseek R1 [29
have inspired efforts to create specialized clinical reasoning systems 46
capable of answering
complex clinical questions and generating comprehensive clinical reports 20 90]. Reasoning allows
models to think explicitly in a more logical and systematic way with evidence from the inputs and
their own knowledge (34, all of which are essential for clinical diagnosis [55 49
However, building effective models support clinical diagnosis presents several significant chal- lenges. First, clinical data spans multiple modalities across ID (ECG, EEG), 2D (Chest X-ray, dermoscopy, mammography). and 3D (CT Scans, MRI). Models like BiomedGPT [90] and Med- Flamingo [56] have integrated 2D and 3D data within one vision encoder, but no existing model has been able integrate both 1D sensor data with 2D/3D images. The heterogeneity across specialties and modalities [28] [7] often leads to settings where modalities compete rather than synergize, leading suboptimal performance 36 52 89). This necessitates careful retraining or fine-tume fine-tuning strategies to balance heterogeneous distributions while enriching these models with clinical knowledge.
Secondly, conventional training methodologies typically constrain models generate single, defini- tive answers without revealing their underlying analytical process [0) 76 45). This "black box" approach significantly impedes the practical adoption of AI systems clinical settings, as healthcare professionals might hesitate to trust diagnostic suggestions without understanding the reasoning that produced them 72 Transparency in the decision-making process merely a preference but
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

necessary component for responsible clinical implementation, regulatory compliance, and effective human-Al collaboration in healthcare environments 66
In this work, we introduce QoQ-Med: a generalist clinical multimodal foundation model with precise reasoning capabilities spanning clinical images, time series data, and textual records across clinical domains. Our work makes two primary contributions:
1. Firstly, to tackle the challenges associated with balancing heterogeneous data for balanced and efficient training across ID to 3D data, we propose Domain-aware Group Relative Policy Optimization (DRPO). DRPO employs hierarchical scaling based on the domain of the input data, which encourages the model's learning on scarce and hard domains, allowing balanced learning across difficulty levels. Our empirical evaluation demonstrates that DRPO consistently outperforms established RI approaches in diverse multi-domain settings, with up to 43% improvement in average F1 score across 8 clinical vision modalities.
2. To tackle the second challenge of expert interpretabillity. we design and release one of the first multimodal clinical reasoning models, namely QoQ-Med-77B/22 (Qwen Omni-Reasoning on Medical Questions), that integrates visual, time series, and textual data for comprehensive analysis of clinical records, facilitating more holistic diagnostic reasoning. QoQ-Med trained to highlight salient regions in the visual input data, advancing the interpretability while allowing the clinician to check the model's 's diagnosis with ease. To the best of our knowledge, QoQ-Med currently the largest open-source multimodal reasoning model for clinical diagnosis, and the only MLLM that integrates time series data (ECG) with traditional clinical vision modalities.
Finally, we publicly release our model, training pipeline, and reasoning traces generated by the model across 2.61 million question-answer pairs at this link This marks one of the largest resources for transparent and reproducible multimodal reasoning in the clinical domain.
2 Related Work
2.1 Multimodal Large Language Models (MLLMs) for clinical diagnosis
Recent work has adapted Axsion-languag interfaces to the medical domain, yielding models such as LLa [48, RadLM [90), and Med-Flamingo [56]. These models couple frozen LLM backbones with image encoders and are trained on radiology or pathology visual-question-ansswee
and report- 86). Although these systems demonstrate impressive zero- shot understanding, their training corpora are dominated by single-institution chest X-rays, retinal photographs, and pathology slides, resulting limited generalization to demographic diversity and poor robustness to real-world distribution 63,63,5511 (1). GEM [47) is the only MLLM incorporating ECG data, but the training focus is purely ECG, which does not provide comprehensive diagnosis aggregating multiple sources. Our work addresses these gaps by assembling a richer corpus spanning imaging, semees, and text, and by designing an architecture that natively models medical time- series alongside traditional modalities.
generation benchmarks [24 39 92
88
2.2
LLM reasoning with reinforcement learning
The introduction of instruction tuning precipitated a rapid shift from supervised fine-tuning to reinforcement learning pipelines. Proximal Policy Optimization (PPO) [74] as popularized by InstructGPT, trains LLMs against a reward model under a KL penalty to a frozen reference, with an auxiliary critic estimating advantages [62 While effective, PPO's critic incurs substantial memory and computation costs and can destabilise multi-task optimization 77 To reduce overhead, critic-free objectives such as Direct Preference Optimization (DPO) [70] and Group Relative Policy Optimization (GRPO) [75] have emerged, matching PPO's s alignment quality with a simple classification loss. GRPO, in particular, has been widely used in the training of recent SoTA models, such as DeepSeck R1 29 and Qwen-3 3 [79.. However, removing the critic also eliminates per-sample re- re-weighting, causing it to overfit on easy, abundant samples [33]. Classic deep-RL RL work explored adaptive rescaling through task-wise normalization in IMPALA 27 and the PopArt [II. However, these techniques have not been adapted to LLMs or extended to capture fine-grained intra-domain differences. We reinstate that flexibility by learning both inter-domain and intra-domain scaling factors within critic-free RLHF pipeline, combining the efficiency advantages of GRPO with the adaptive weighting capabilities offered by critic-based methods
2

---

Multimodal Training Dataset
QoQ-Med
Outputs w/ Reason
Eucoddrr
Profectin
Seriiss
X
Dermo
Prndus
Patho
LLM Backbone
Reasoning Insights
Eucorber
Pesserioa
Visican
Sraun
Salient Region
Alowe check et
Sesaile (2R) preand caa wan?
Text
Cluitca QA Paits
BIR
Clinical Diagnosis
Easy Data, Smaller [Advantage
RL Training
DRPO
Rewardi Modal
GRPO
Sealing
Policy Model
DeseAi 1
Datais
Hard Data, Larger [Advantage]
Nefwrroos Moddl
1000
DRPO Mixed Training
(a) Overview of QoQ-Med. The training corpus spans 11 clinical domains, including structured waveforms (e.g., ECG), diverse imaging modalities, electronic health records, and curated clinical QA pairs. Modality-specific encoders convert inputs into token embeddings that are linearly projected into a common space and interleaved with text tokens before entering the LLM backbone. The model then autoregressively produces (i) explainable chain-of-thought, (ii) bounding-box annotations highlighting salient regions, and (iii) a concise clinical diagnosis. (b) Overview DRPO Training. DRPO builds top of the critic-free RL training method GRPO. The model's answer is first rated by a reward model before going through standard normalization. Then, a clustering- based scaling performed on top of domain-wiss scaling, both of which encourage the model to focus scarce, hard examples across domains.
Figure 1:
3 Method
In this section, first define problem as multimodal diagnosis question answering task, before describing how we integrated time series alongside vision inputs into a single unified model. Finally, we demonstrate detail how we address the domain heterogeneity problem with the Domain-aware Relative Policy Optimization (DRPO) algorithm and design of appropriate reward functions.
3.1 Problem Definition
xxx is patchified image, x (f) RAxXT; is multichannel time- -ime-series data, x!" is text input, and 9i E (1,.... C} denotes the clinical domain (e-g., CT scans, ECG, Chest X-ray). Vision and time-series inputs are optional, which requires the model to handle missing modalities. The learning objectives are to predict: (i) an unsupervisedly learned reasoning trace, (ii) bounding boxes b, = {br,, 1/- with b E R in (z, y, w, hi) format highlighting salient image regions, and (iii) concise diagnosis 9..
Each clinical sample is x, (x{ (v) .x)" x({")
.g)). where x("))

---

Table 1: Comparison of QoQ-Med against other open-source public clinical MILMs. BBox:
Ability to predict salient bounding boxes; CXR: Chest X-ray; Mammo.: Mammorgraphy; Derm: Dermoscopy: Patho.: Histopathology: US: Ultrasound. BBox: Whether the model is able to produce bounding boxes as output. QoQ-Med is currently the largest medical reasoning MLLM in the field, and the only model trained with DRPO, our RL training algorithm we introduced in Sec. 3.3 *
90 56] are trained on some ECG images, but none of them are trained on ECG time series ipput.
ID
2D
3D
Model
Size
Training BBox
ECG
CXR
Mammo. Derm.
Fundus
Patho.
US
MRI
CT
Med 48
7B-13B
SFT
v
X
x
x
y
Med-Flamingo
56
8.3B
SFT
X
o
V
x
x
x
RadFM 85
14B
SFT
X
v
of
v
x
r
Bio
33M-182M
SFT
o"
x
v
Med-R1
46
2B
GRPO
X
x
v
v
QoQ-Med (Ours)
7B-32B
DRPO
V
V
r
v
3.2
Model
We design the model with an aim that it can take data across as many domains as possible, so that it can provide comprehensive diagnosis while correlate and co-train across the most diverse range of clinical domains, with inputs ranging from ID to 3D.
Model Design. As shown in Figure II we initialize QoQ-Med from a large pretrained vi- sion-language model comprising an image encoder, a linear projection that maps each visual patch embedding into the backbone LLM's token space, and the LLM. To ingest temporal data, we prepend pretrained time-series encoder, namely ECG-JEPA [3], whose outputs are passed through newly initialized linear projection of matching dimension. At inference, the projected image patches, time-series tokens, and tokenized text are interleaved in their original temporal order and fed the LLM. The LLM autoregressively generates free-text chain of thought, bounding- box tokens that localize the evidence identified in that reasoning, and outputs a short diagnosis. This design supports heterogeneous modality combinations, allowing the model to skip missing channels while preserving positional consistency across the multimodal sequence.
Training proceeds in two stages. Stage 1: modality alignment. Since we initialize the projection layer from scratch, we first train and align the ECG encoder, the projection layer, and the L.M.. To encourage high-quality reasoning outputs from the beginning. we use the same DRPO training as in Stage 2. Stage 2: multimodal fine-tuning with DRPO. We train on the full multimodal corpus with DRPO, as described in Sec. 3.3] which balances training across different samples in various domains and difficulty. In this stage, we aim simultancously improve the diagnostic accuracy and reasoning quality, with rewards described in Sec. 3.4
Training Process.
Training Data. We train the unified vision and time-series model across 33 datasets using the CLIMB dataset [22]. The dataset contains 2.61 million samples across ID (ECG). 2D (Chest X-ray, Mammography, Dermoscopy, histopathology, Fundus), and 3D (Ultrasound, MRI, CT Scan) data. The exact composition of the data and the training hyperparameters are included in App. Cl D
Comparison with current public clinical MLLMss Table I demonstrates that our model is currently the largest open clinical MLLM in the field. is also the only model that can both take in time series data and output its thinking process, along with the bounding box annotation highlighting the salient region made during the thinking process.
3.3 Domain-aware Relative Policy Optimization (DRPO)
Group Relative Policy Optimization (GRPO) is reinforcement learning method that gained promi- nence following the success of DeepSeek-RI. Unlike Proximal Policy Optimization (PPO), which relies on a separate value network to estimate advantages, GRPO directly computes the advantage A(g.i.)) (Eq. for each response within group of rollouts ((y.4) at given training iteration. A rollout refers to a single sampled trajectory or response generated by the policy in reaction to a prompt. The advantage quantifies how much better particular rollout is compared to others generated for the same prompt, enabling the policy to prioritize relatively high-quality responses without requiring an explicit estimate of expected return.
Each group of rollouts G(g.)) consists of multiple responses sampled for the same prompt q- Let ((q..)) denote the scalar reward assigned to the i-th response o(g.itt at time step t, where

---

each response is a sequence of tokens (g.i,) == (giit))
(ggi.)): ..... m ". and
Doça.4.5) denotes the length of the token sequence. The set of rewards for the group defined as R 1.)) "(g.2.))... :aa....:"aaalGa..a l.t J. where |Gg] is the number responses in the group,
GRPO normalizes these rewards to have zero mean and unit variance, producing the normalized advantage:
T(q,i.0) (..)
AGRPO (g.i.t)
(1)
(Gig, +
where ( and oG denote the empirical mean and standard deviation of the group rewards, respectively, and is a small constant added for numerical stability. These advantage estimates are incorporated into the GRPO clipped surrogate objective, which also includes a per-token KI.
divergence penalty:
A(.i.1):(0)) = min (Pnn.aa)a ACRPO clip (Plani))aa0)) 1 1-E, 1 1+ (g.h.))
AGRPO (q.5.0)
To (g) 9, ((..))))))))
((()).)))(tt)) =
Teae (( q, ((..))))))))))
Goool
1
(0) Eg ( (G(gual (olg.
A(ns.tja( - B DEL (No Tet)
Here, ((..i.))ckk refers to the token subsequence from position to k. 1, and D denotes the dataset distribution. The term (() represents the importance sampling ratio between the current policy Th and the old policy Tew at token position k; A is the normalized advantage estimate for the i-th response in group ((g,)): is small constant used for numerical stability and clipping: (3 is a scalar hyperparameter that controls the strength of the KI. divergence regularization; and (KI(TTe Teat) denotes the Kullback-Leiblrr divergence between the learned policy and a reference policy. GRPO demonstrates strong empirical performance when the input data is relatively homo- geneous. However, in settings with high data heterogeneity, domains with abundant samples tend to dominate the optimization process, while under-represented domains contribute minimally. This imbalance can bias the model and degrade performance on rare but clinically important modalities. while spending too much compute on easy problems on abundant domains.
Domain-aware Relative Policy Optimization (DRPO). While GRPO normalizes reward signals across rollouts that respond to the same compt-thhreby reducing variance within a group and ensuring fairer comparison among responses- it does not address imbalance across domains. As result, domains that appear more frequently in the training data continue to have a disproportionate impact on the learning process. DRPO builds on GRPO by introducing hierarchical scaling mechanism that explicitly balances contributions from different domains. This correction for inter- domain imbalance preserves GRPO's simplicity and value-free free formulation while promoting more equitable learning across heterogeneous data distributions.
Hierarchical Cluster-Based Scaling. The core innovation of DRPO lies in hierarchical scaling strategy that adaptively balances learning signals based on both domain frequency and task difficulty. This mechanism operates two levels: across domains, to mitigate the dominance of overrepresented domains, and within domains, to adjust for variations in response quality or reward magnitude. Concretely, we first cluster question-level reward sets within each domain, treating each set of individual rewards as a feature vector. We then apply a two-stage reward scaling procedure-firs at the cluster level, then the individual reward level-thereby emphasizing learning from rare and challenging questions.
Clustering. At each iteration step f, we begin by sampling independent batch of questions. These questions are then clustered into different domains. Let denote a domain, and let N(y.)) represent the number of questions in domain g at iteration t. Within each domain at iteration f, we first compute the set of rewards for each question. These rewards, collected across multiple rollouts, are concatenated into feature vector per question. Specifically, for each domain g. we construct a set of reward vectors Hy = (vg) 9=1 vy E RlG,, where vy contains the Ro rollout rewards for question g, and N (g.t) is the number of questions in domain g. at iteration step
Stage-1:
5

---

To uncover patterns in question difficulty, we apply K-means clustering to these reward vectors at each time step separately within each domain:
(1..0:)) (22g,,))..... (((G).0)8.0)) = KMeans(Hgy k(g.)))
where C(c.g.) denotes the centroid of cluster e in domain g. and ki(g.) is the number of clusters, which is determined automatically using the elbow method (see Appendix B.T).
Stage-2: Hierarchical Scaling. For each domain and each cluster within that domain, we compute inter-domain temperature factors ((g and intra-domain temperature factors T(c.g.t). These factors capture both the relative size and average difficulty of each domain and cluster. Difficulty is estimated using the mean reward, either per domain or per cluster within the domain, which serves how easy or challenging the questions are within each specific domain and cluster. These temperature factors are then inversely mudriplied with the corresponding advantage functions at both the domain and cluster levels- so that domains and clusters that are smaller or harder receive proportionally greater weight during training. Concretely:
proxy for
T(g.t) max (VN(..)) Mo.n))), Ticg..) max (VNcassoo Micg.)).))
where Ni(..gt) 1) is the size of cluster and M(g.4) and M(cgg.) denote the mean reward for group y and cluster in group g. at iteration
To scale reward advantage with the appropriate temperature factors, we first normalize rewards at the question level as GRPO, then scale by the domain and cluster temperatures, before multiplying by a KL regularization factor ((,t)) Concretely,
scaled (g.A.t)
(i.e)) (gg.50) T(a.t) T(....))
where Pi. is the question d
reward from GRPO. The KL regularization is applied to prevent outliers from dominating the update, detailed in Appendix B.2 Finally, scale the standard deviation back to by dividing each reward by the standard deviation of the reward in the batch A
Speated
DRPO Objective. DRPO maintains the same objective structure as GRPO, maximizing:
((...1):2(() = min (P(o.t))a)) (n.i,))
DRPO
clip (F((,s))))a(0)) 1 1, 1 (e) APRR0 (a1.0))
Gogol
(0) = EggDD botne
Agsit1(()) - DRL (No Tea))
GG(g,0]|
where Pla...t); (0)
x xe(P ((n))0)))))) Te (or ocor)))))k))"
Benefits of DRPO. The cluster-based DRPO approach offers several key benefits:
1. Hierarchical Scaling: DRPO implements two-layer scaling: first at the domain level and then at the cluster level within each domain. This directs optimization toward both underrepresented domains and challenging question subsets, ensuring the model learns effectively across all data types. This approach prevents the model from focusing only on easy or common problems while neglecting rare but important clinical scenarios.
Preservation of Zero Mean and Unit Variance: DRPO scales rewards after GRPO normalization, maintaining the property that the mean reward within each of rollouts remains 0 and the standard deviation is 1. This property is crucial for stable optimization in reinforcement learning,
established in previous works [14 94 [57].
3. Computational Efficiency: DRPO operates with minimal additional complexity of order O(n), primarily from the K-means al algorithm operating on low-dimensional vectors (typically 5-10 elements). This enables efficient training without the overhead of critic networks, making it particularly suitable for large-scale scale LLM fine-tuning.

---

3.4 Reward Design
During the training of QoQ-Med, we employ a combination of two main rewards and two auxiliary rewards that balance diagnostic accuracy with interpretability, a critical requirement for clinical applications where understanding model reasoning.
Accuracy reward. The primary goal of our model is diagnostic accuracy, for which we compute standard accuracy reward rqce We treat prediction Ú: and ground truth yi as unordered sets of labels and assign r = Fl(ū;, (ū: yi). which directly optimizes the model's ability to identify correct diagnoses across diverse clinical scenarios.
Semantic alignment reward. For clinical applications, the ability to identify and highlight relevant regions in medical imagery is crucial for building clinician trust. The semantic alignment reward encourages the model to correctly identify salient regions that support its diagnostic decisions. Let b; {b. / be the set of axis-aligned bounding boxes output by the model and S; [0, 1] x| the pixel-level segmentation mask associated with the ground-truth diagnosis. We define this reward
as the best intersection-ovee ceatiiioon-vve-uree over--unoonn-n score: P.o max K.
area (S..
By optimizing this
area (US)US:))
reward, the model learns visually highlight the specific anatomical regions relevant its diagnosis, providing critical interpretability for clinical decision support.
Auxiliary rewards. We also employ auxiliary rewards that encourage proper formatting and comprehensive reasoning, detailed in Appendix [B3]] These rewards help ensure that the model's outputs are well-structured and sufficiently detailed for clinical use.
Combined reward. The final scalar reward supplied to DRPO weighted combination: r Asce r + λso r
JoU + >aux ritk In our experiments, we set (Aace, λtou, Xmux) = (0.6, 0.2, 0.2).
Experiments
We design experiments to answer the following research questions. Details are included in App. D
RQ1: How does DRPO compare with other critic-free RL methods and models? As detailed in Sec. 3.2 we train and evaluate QoQ-Med on a combination of 30 clinical diagnosis datasets across 9 clinical domains. description of each dataset is included in App. C The models are evaluated with balanced accuracy and macro-F1. We compare our training method DRPO against supervised fine-tuning (SFT), PPO 77 and four popular critic-free RL training methods: GRPO (73. RLOO (2)
Reinforce++ (33] and ReMax [50]. We further compare our trained model QoQ-Med against medical VLMs (Llava-Med 48. Med-RI (46) and closed source VLMs (GPT-4o (37, 04-mini (61).
RQ2: How well does DRPO handle mixed multimodal inputs? We repeat the comparison on MIMIC-IV, where samples contain chest X-ray, 12-lead ECG trace, and accompanying clinical record. We train and evaluate the models on two tasks: length of stay (LOS) prediction, binned into a 4-day interval, and 48-hour in-hospital mortality (48-IHM). We evaluate the model with accuracy and Fl score in the same way as RQI.
RQ3: How is the quality of the reasoning traces and bounding boxes learned by DRPO? We did both a qualitative and a quantitative analysis on QoQ-Med's reasoning and bounding box outputs. We evaluate the bounding box quality via the intersection truth segmentation available in the dataset. We further collaborated with clinicians to annotate the reasoning traces the validation dataset, grading the traces by their relevance to the final diagnosis.
union (IoU) against the ground
4.1 RQ1: Comparison with other RL Training Methods and Models
Comparison with other RL methods. Table 2 shows comparison between DRPO and several critic-free RL training methods across eight medical imaging modalities. The results demonstrate that DRPO consistently outperforms all competing methods in out of vision modalities in terms of F1 score. Overall. DRPO achieves a mean accuracy that is 5.9% higher in percentage points and an F1 score that is 46% higher compared to the best critic-free baseline method. As compared to GRPO in Fig. a), the most substantial increase is observed in datasets from understudied modalities, like ultrasound and mammography, as defined in App. 2 As shown Fig. 1b) QoQ-Med achieves the best performance across all clinical domains compared to current spen- MLLMs. Compared
7

---

Table 2: Performance comparison of medical vision-language models across various medical imaging modalities. Acc: Accuracy, F1: F1 Score, CXR: Chest X-ray. DRPO training outperforms various other RL training methods and SFT across 7 out of 8 medical imaging domains in F1 score. The metrics averaged across four separate training runs. Metries with standard deviation is included in App. Tab.
Model
CXR
Mammo.
Dermoscopy
CT
Scan
Fundus
Ultrusound
MRI
Puthology
Overall
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
Ace
F1
Acc
F1
Aec
Acc
F1
SFT
.888
078
.811
056
.640
.158
.525
.236
.715
.06
.488
.235
.567
.97
6652
.083
.602
.139
PPO
74
.670
064
.738
.215
668
.278
.571
.257
.669
.083
.490
.880
.767
.540
.75
.364
.665
.234
ReMax
(50
.636
.120
.577
.03
.644
.257
.567
.228
.678
.089
.547
.147
.547
.264
.706
.270
.596
.176
RE++
33
.730
.022
.660
.76
.635
.237
.529
.247
.672
.088
.519
.136
.651
.20
.668
.254
.621
.202
R1.00
.752
.86
.471
068
.636
.216
.534
.224
.670
.099
.599
.144
.658
432
669
.216
.611
.189
GRPO 75
.703
.095
.466
0059
.646
.244
.524
.236
.670
.086
.520
.146
.631
.395
.715
.286
.609
.193
DRPO
.693
.086
.751
.213
.679
.251
.571
.257
.669
.083
.480
098
.733
.45
.762
.388
.668
.237
DRPOSOKL
.711
.264
.691
.382
.597
.365
.676
.554
.228
.722
.535
.300
.668
.283
DRPO
.687
.115
.756
.253
.715
.407
.570
.309
.672
.093
.555
.23
.789
.625
.708
.265
.666
.295
Table 3: Ablation studies on cluster size and reward composition. Acc: Accuracy, Fl; Fl Score. Bold values indicate best performance within each ablation group.
CXR
Mammo,
Dermoscopy
CT Scan
Fundus
Ultrasound
MRI
Pathology
Overall
Config
Acc
F1
Acc
Fl
Ace
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
Fl
Cluster Size
1
.694
.085
.746
.211
.678
.286
1571
.257
.669
.083
.544
.200
.757
.505
.773
.449
.679
.259
3
.694
.15
568
.048
.680
.356
.62
.84
.672
.147
.520
.152
.717
.56
.723
.289
.642
.244
10
.691
.125
.759
.253
.707
.400
.580
.321
.670
.088
.568
.240
.806
.652
.707
.303
.686
.286
20
.668
.167
.751
.268
.675
.300
.548
.62
.635
.166
.547
.214
.804
.649
.731
.329
.670
.294
Reward Composition (Acc:loU)
0.6:0.2
.691
.125
.759
.253
.707
.400
.580
.321
.670
.088
.568
.240
.806
.652
.707
.303
.686
.286
0.2:0.6
.660
.147
.563
.185
.668
.290
.576
.38
.681
.136
.573
.218
.768
.561
.6.9
.233
.652
.260
the closed-source commercial models, it achieves the best performance against GPT-4o 337, while surpassing the reasoning model GPT-o4-mini [60] in all domains except MRI.
Ablations. The substantial improvement in F1 score be attributed to two key components of DRPO. First, the introduction of domain wise scaling contributes to a significant 22.8% improvement in Fl score, evidenced by the performance difference between DRPOnemainOnly and vanilla GRPO. Subsequently, after incorporating clustering within each domain and specifically encouraging the model to focus on small, challenging clusters within each domain, the performance further enhanced by an additional 19.4% in terms of Fl score.
Tab. shows further ablations the number of clusters and reward compositions. In general, found that the weight of each reward does not have a significant impact on the final performance. In particular, the auxiliary rewards formatting saturate shortly in the early stages of training. They have effectively no impact on the later stages due to normalization. We tested different combinations of accuracy rewards: semantic alignment rewards. As demonstrated in the table, decreasing the weight of the accuracy reward gives a drop in overall performance and performance in most domains, but results are still significantly better than all baselines, which demonstrates the robustness of DRPO.
The number of clusters in the model is determined automatically via the elbow method, with the possibility set upper limit on the number of clusters. As part of the ablation, tested the model with 1 (no clustering). 3, 10 and 20 clusters, and included the results in Tab. 3 In general, we observe that having cluster performance. A higher cluster limit, however, does not seem to hurt the performance, as the elbow method automatically chooses lower cluster count than the limit. This allows the algorithm to remain efficient under arbitrary cluster limits.
very low cluster limit will cause decrease in
Runtime Efficiency. As shown in Fig. 3e., while DRPO requires clusters to be calculated on each step, has negligible impact on the overall runtime. Across all critic-free RL. methods, reward calculation accounts for less than 2% of the total runtime of a step.
8

---

CT Scan
MedRI
Accurey Delleencs (DRPO (800)
Naval Todk
BCG
Undenspro Regiom
Mallpli Cargatic
GPT-4
015
Ofher Thrason
Pathology
CXR
0,8
-055
Mammo
Raio Tamme?
NarS Iac son
Pata do
Conmiling
Fandis APPI HOst
coVD DELLE
Valk rtsdfoee
RC Mnsic Cle Lotkiie
Fundus
curDa
Uhmasound
Datzser
MRI
Figure 2: (a) Difference accuracy (DRPO GRPO). DRPO brings the most performance gain in understudied modalities as defined in App. C.2 (b) Accuracy comparison of QoQ-Med against
SoTA open source and closed source LLMs. QoQ-Med outperforms all open and closed MLLMs
across 8 domains. The full results are included in App. Table
Avg. Time Per Step
Time Spent on Reward Calculation (s)
RuMax
76.67
Med-R]
0.055
RL. Method
RL Method
RL Method
ReMax
RL.00
74.49
GPT-4o
0.068
RLO0
6.79
386
GRPO
76.01
64-mini
0.375
GRPO
8.28
425
QuQ-Med
0.353
DRPO
9.86
429
200
400
(a)
ECG Diagnosis Accuracy (%)
(b) IoU of Output BBoxes against GT
(c)
Time (s)
Figure 3: (a) Accuracy of ECG Diagnosis. DRPO models reach the best performance among all critic-free RL methods. (b) Intersection over Union (IoU) of model-generater bounding boxes against truth labels. QoQ-Med (Ours) surpasses open source models and has performance on par with 04-mini. (c) Per Step Runtime reward calculation of RI. methods on 8xA 100 GPUs. While DRPO adds hierarchical clustering, the runtime of the reward calculation still accounts for less than 2% of the total runtime per step and has minimal impact on training.
4.2 RQ2: Multimodal Fusion Performance
We tested how the model integrates multiple modalities and how much each modality con- tributes to the final diagnostic accuracy via MIMIC-IV [41 dataset. On the MIMIC-IV dataset, the model has to chest X-rays, and health records. As shown in Tab. found DRPO allows the model to reach a better performance in both tasks as com- pared to GRPO. In addition, taking full inputs across ECG, Chest X-ray images, and electronic health records (EHR) gives better performance than any ablation of these modalities, signaling that QoQ-Med is able effectively aggregate information across all modalities. Specifically, we found vision and texts contirbute more to the
Table 4: Models' Perf. on MIMIC-IV. DRPO- Full with inputs from 3 modalities has the best per- formance for both tasks, time-series only (DRPO- TS+T) and vision only (DRPO- Vision+T) abla- tions having worse performance, and the text only ablation having the worst performance.
across ECGs,
LOS
48-IHM
Algo/Inputs
Acc
F1
Acc
F1
GRPO-Full
0.626
0.105
0.551
0.354
DRPO-TextOnly
0.645
0.195
0.563
0.583
DRPO-ECG+T
0.639
0.204
0.602
0.528
DRPO-Vision+T
0.669
0.223
0.596
0.586
DRPO-Full
0.663
0.283
0.642
0.597
final accuracy and Fl scores than ECG. While QoQ-Med represents first step towards multimodal reasoning models across vision and time series, future works could explore better architecture, data, or training methods that better balances the power of each modalities.

---

Seestion: Absue traze san slise, tsere bomuntngge prevent thas sluee?
is diggosis of the petises the ciinical iragai
Kodel Reasoning:
Kodel Reasaning:
CT. scan cross-asctiomal vaRa the
and slightly elavaced aligntly reised, sinkish, reaah, acely face, whach the Lecture this inage leniun's color. shaps, location are comsisiett with cercinom....
hesorthan
have
hypodemss your
cursaalent
Liseua))
(whiter then surnel cissue sppears uni unifremly without abvious areas that would suggest henorthage...
dark
Ansner: Bosal cell carcincea scan, ttere tarr
(b)
fessium: Jasse
Koryy urage peluens.
presert In this Mil scart
is a
sragel
truth:
tapport dessses
Hodel Reasoning:
Modea Reasoning:
Suror appears as a round, willcctrcanerisee
aodiastinal area asptars to be encarged
distinct, often irrequiar, boundary tunor 1s
redical douice iiloural effusion, wnich is Tluid
tissu which 18 scrong indicator of lexion arain... in a region shere Luoora are common.
staniard for x-rays ineleate the atmormalities
Ansatr:: to findlngg
Qestine telon is ristony of petient. Hew lova vill the natieet stay in the hogelta2?
crich says Nodel Reasontin::
the paliant's histury includes Keray imapus showing abvicus souts pachulory the longs, atable failure other sppeers un sear kabiw, conditions Eoc. The laba show zoreal hentolooiicl parasn anctara, which suppost that no insdiitte Life-threstmming arenis, infection, electrolyte inbslences..
which could indiccatiee scuts
conditions like
Assive:; A. 0-6 days
(c)
Relevance disgnosis
high
low
Figure Model outputs annotated by elinical experts. QoQ-Med correctly reasons from modality- specific clinical knowledge, generates bounding boxes, and outputs the correct predictions in most instances except (c). (e) demonstrates the model's ability to synthesize multimodal inputs with reasoning. The bounding boxes correctly highlight the salient regions related to the reasoning steps when one present.
4.3
RQ3: Quality of Reasoning Traces
Clinician relevance annotations. App. D.4.2 provides a breakdown of dinician-annotaan reasoning traces, revealing that the model mostly generates contents highly relevant to the diagnosis, with minimal output judged as irrelevant. We observe that the model often correctly recalls relevant clinical knowledge, which help guide the model by providing associative context. For example, in Fig 4 4aa, the model correctly recalls different signs of hemorrhage on CT, such as darker or whiter tissues, and relates this context to specific parts of the image to make correct prediction. In Fig. Fe), the model correctly identifies the presence of pacemaker, indicating support device, but subsequently concludes that there are no additional abnormalities, ultimately leading it to predict "No finding". " This suggests that while the model's final predictions may be incorrect, its intermediate reasoning often reflects clinically relevant patterns.
Bounding box quality. Fig. 3(b) demonstrates that the model identifies bounding boxes correlated with the ground truth annotations, with the IoU exceeding the best open source models while reaching a similar performance as the closed-source reasoning model 04-mini. From Fig. 4 we also see that the outputs by the model are sufficiently aligned with the reasoning process, allowing the clinicians to confirm the model's 's predictions while gross-referencing the source image.
Conclusion
We introduced QoQ-Med, a clinical MILMM with reasoning across 9 clinical domains. Our Domain- aware Group Relative Policy Optimization (DRPO) demonstrates superior performance over existing approaches, with up to 43% improvement in average Fl score across clinical modalities and substantial gains in multimodal fusion tasks. The ability of QoQ-Med to process ID time series data alongside traditional 2D/3D clinical images addresses a significant gap in existing medical multimodal systems, while its transparent reasoning process enhances interpretability and clinical trust. By publicly releasing QoQ-Med-7BB333
and our comprehensive reasoning dataset containing 2.61 million question-answer pairs, we hope to contribute valuable resources to advance clinical reasoning AI. A potential limitation is the limited sample efficiency as the reasoning process is not supervised. Moving forward, we hope the community can explore ways to elicit high-quality reasoning with better data efficiency, with a special focus understudied modalities like ECG and ultrasound.
10

---

6 Acknowledgement
This material is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. 2141064. Any opinion, findings, and conclusions or recommendations expressed in this material are those the authors and do not necessarily reflect the views of the National Science Foundation.
We thank the MIT Office of Research Computing and Data (ORCD) for support through ORCD Seed Fund Grants, which provided access to 8xH200 GPUs and additional funding support. We also thank the NVIDIA Academic Grant Program for GPU support.
We also extend our sincere thanks to Haowen Wei (Research Associate, MIT Institute for Medical Engineering & Science, MIT.nano) and Dr. Farzan Vahedifard (Neurologist, Athinoula A. Martinos Center for Biomedical Imaging, Harvard Medical School) for their careful annotation of the model's reasoning outputs and valuable insights that significantly improved this work.
References
[1] Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy. and Luke Zettlemoyer. Scaling laws for generative mixed- modal language models. In International Conference on Machine Learning, pages 265-279, PMLR. 2023.
[2] Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadacc, Julia Kreutzer, Olivier Pictquin, Ahmet Üstün, and Sara Hooker. Back to basies; Revisiting reinforce-style optimization for lcarning from human feedback in llms. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 1 12248-12267. Association for Computational Linguistics, 2024. doi: 10. 10.18653NV12024.AALLL LONG LONG.662666 662.. URL https://doo. org/ 18653/v1/ 2024. aol-long.... 662
[3] Walid Al-Dhabyami, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy. Dataset of breast ultrasound images. Data brief, 28:104,63, 2020.
[4] Erick Perez Alday, Annie Gu, Amit J Shah, Chad Robichaux, An-Kwok Ian Wong, Chengyu Liu, Feifei Liu, Ali Bahrami Rad, Andoni Elola, Salman Seyedi, et al. Classification of 12-lead eegs: the physionet/computing in cardiology challenge 2020, Physiological measurement, 41(12): 2020.
[5] Mohammad Alkhalaf, Ping Yu, Mengyang Yin, and Chao Deng. Applying generative ai with retrieval augmented generation summarize and extract key clinical information from electronic health records. Journal of biomedical informatics, 156: 104662, 2024.
[6] Julia Amann, Alessandro Blasimme, Effy Vayena, Dietmar Frey, Vince I. Madai, and the Precise4Q consor- tium. Explainability for artificial intelligence in healthcare: a multidiscpplinary perspective. BMC Medical Informatics and Decision Making, 20( 1):310, 2020.
[7] Mohamed Amgad, Habiba Elfandy. Hagar Hussein. Lamees Atteya, Mai AT Elsebaie, Lamia S Abo El- nasr, Rokia A Sakr, Hazem SE Salem, Ahmed F Ismail, Anas M Saad, et al. Structured crowdsourcing enables convolutional segmentation of histology images. Bioinformatics, 35(18) (311133346133467 2019.
[8] Asia Pacific Tele Tehh-Ophhhalmology Society. APTOS 2019 blindness detection. https: http:://wwwkkkgll/ kaggle com/ cttpp///aappggaa/e wwweoaoiinnnnneeeeeeeeeiieeeeeee data 2019. [Dataset].
[9] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. et al. Qwen2. 5-vl technical report. arXiv preprint 4xXv:22502.13923 2025.
[10] Sartaj Bhuvaji, Ankita Kadam, Prajakta Bhumkar, Sameer Dedge, and Swati Kanchan. Brain tumor classification (mri), 2020. URL https://t/ kaggle. com dav/ aaddv//111/ /1883168
cww/dav/
[11] Andreea Bodnari and John Travis. Scaling enterprise ai in healthcare: the role of governance in risk mitigation frameworks. npj Digital Medicine, 8(1):272, 2025.
[12] Karsten M Borgwardt, Cheng Soon Ong. Stefan Schönauer, SVN Vishwanathan, Alex J Smola, and Hans-Peter Kriegel. Protein function prediction via graph kernels. Bioinformatics, ([ssupl_1)xi47-ii1 2005.
11

---

[13] Andrew A Borkowski. Marilyn M Bui, L Brannon Thomas, Catherine P Wilson, Lauren A DeLand, and Stephen M Mastorides. Lung and colon cancer histopathological image dataset (lc25000). arXiv preprint urXiv.1912.12142. 2019.
[14] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg. and Dario Amodei. Deep rein- forcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[15] Joseph Paul Cohen, Paul Morrison, and Lan Dao. Covid-19 image data collection. arXiv 1003.11597, 2020. URL https://github. http:////gthuuub.oooeeeeeeee hotp:///e.eeeeee///o/ooeee-eeeee http://vvvdcccetcrareaeaaaaaaaae dataset
[16] Errol Colak, Felipe C Kitamura, Stephen B Hobbs, Carol C Wu, Matthew P Lungren, Luciano M Prevedello, Jayashree Salpathy-Cramet Robyn L. Ball, George Shih, Anouk Stein, et al. The rsna pulmonary embolism ct dataset. Radiology: Artificial Intelligence, 322):2200254, 2021.
[17] Can Cui, Haichun Yang. Yaohong Wang, Shilin Zhao, Zuhayr Asad, Lori A Coburn, Keith T Wilson, Bennett A Landman, and Yuankai Huo. Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review. Progress in Biomedical Engineering. 5(2):022001, 2023.
[18] Chunyan Cui, Li Li, Hongmin Cai, Zhihao Fan, Ling Zhang, Tingting Dan, Jiao Li, and Jinghua Wang. The chinese mammography database (cmmd); An online mammography database with biopsy confirmed types for machine diagnosis of breast. The Cancer Imaging Archive, 2021. doi: 0/55://CIA.EDEE4BBBB URL BRRLhttps:// //do1 org 10. 7w3779337//// /tclaaccaaa. eqde- 4b16
[19] Hejie Cui, Wei Dai, Yanqiao Zhu, Xuan Kan, Antonio Aodong Chen Gu, Joshua I Lukemire, Liang Zhan, Lifang He, Ying Guo, and Carl Yang. Braingh: benchmark for brain network analysis with graph neural networks. IEEE transactions on medical imaging, 42(2):493-506, 2022.
[20] Hejie Cui, Lingjun Mao, Xin Liang, Jieyu Zhang. Hui Ren, Quanzheng Li, Xiang Li, and Carl Yang. Biomedical visual instruction tuning with clinician preference alignment. arXiv preprint arXiv:2406. 13173, 2024.
[21] Wei Dai, Ehsan Adeli, Zelun Luo, Dev Dash, Shrinidhi Lakshmikanth, Zane Durante, Paul Tang, Amit Kaushal, Arnold Milstein, Li Fei-Fei, et al. Developing icu clinical behavioral atlas using ambient intelligence and computer vision. NEJM AI, page Aloa2400590, 2025.
[22] Wei Dai, Peilin Chen, Malinda Lu, Daniel Li. Haowen Wei, Hejie Cui, and Paul Pu Liang, Climb: Data foundations for large scale multimodal clinical foundation models. JCMI., 2025.
[23] Eric Decencière, Claire LaGraize, Pascale Pélégrin, François Benassi, Christian Régér, and Thomas Vautrin. Feedback on a publicly distributed database: the messidor database. Image Analysis & Stereology, 33(3): 231-234, 2014. ISSN 1854-5165. doi: 10.5566/ias. 1as...155. 1155. |RL/htt:://aa h/dt:://dx.... doi org/ 10/g///..00 5566/ias
1155
[24] Franck Dernoncourt and Ji Young Lee. Pubmed 200k ret: a dataset for sequential sentence classification medical abstracts. arXiv preprint wXXvv17700000011.201 2017.
[25] Ashkan Ebadi, Pengcheng Xi, Alexander MacLean, Stéphane Tremblay, Sonny Kohli, and Alexander Wong. Covidx-us-an open-access benchmark dataset of ultrasound imaging data for ai-driven covid-19 analytics. arXiv preprint arXiv:2103. 100.0003, 2021.
[26] Mark Endo, Rayan Krishnan, Viswesh Krishna, Andrew Ng, and Pranav Rajpurkar. Retricval- based chest x-ray report generation using pre-trained contrastive language-imas model. In Machine Learning for Health, pages 209-219. PMLR, 2021.
[27] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harlcy, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pages 1407-1416. PMLR, 2018.
[28] Suparna Ghanvatkar and Vaibhav Rajan. Graph-based patient representation for multimodal clinical data: Addressing data heterogeneity, medRxiv, pages 2023 1223-12,400 2023.
[29] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. ree:seek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint 2r1.:22501.12948. 2025.
12

---

[30] Nicholas Heller, Fabian Isensee, Dasha Trofimova, Resha Tejpaul, Zhongchen Zhao, Huai Chen, Lisheng Wang. Alex Golts, Daniel Khapun, Daniel Shats, et al. The kits21 challenge: Automatic segmentation of kidneys, renal tumors, and renal eysts corticomeduullary phase ct, arXiv preprint arXiv:2307.01984 2023.
[31] Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojeicch Czarnecki, Simon Schmitt, and Hado Van Hasselt. Multi-task deep reinforcement learning with popart. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3796-3803, 2019.
[32] Murtadha Hssayeni, Croock, A Salman, H Al- Al-khafaji. Yahya, and Ghoraani. Computed tomography images for intracranial hemorrhage detection and segmentation. Intracranial hemorrhage segmentation using a deep convolutional model. Data, 5(1):14, 2020.
[33] Jian Hu. Reinforce++: simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.0322 003262,,0.222222......02 2025.
[34] Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey, arXiv preprint arXiv:2212. 10403, 2022.
[35] Shih-Cheng Huang, Zepeng Huo, Ethan Steinberg, Chia-Chun Chiang, Matthew P Lungren, Curtis P Langlotz, Screna Yeung, Nigam H Shah, and Jason A Fries. Inspect: a multimodal dataset for pulmonary embolism diagnosis and prognosis. arXiv preprint
1. 2023.
[36] Yu Huang, Junyang Lin, Chang Zhou, Hongxia Yang, and Longbo Huang. Modality competition: What makes joint training of mulli-modal network fail deep (provaly(..v.bbly..... In International conference on machine learning, pages
PMLR, 2022.
[37] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-40 system card. arXiv preprint arXiv:2410.2122 2024.
[38] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Giurea-Ileus Christopher Chutc, Henrik Marklund, Behzad Haghgoo, Robyn L. Ball, Katic S. Shpanskaya, Jayne Seckins, David A. Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, Curtis P. Langlotz, Bhavik N. Patel, Matthew P. Lungren, and Andrew Y. Ng. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In The Thirty-Third AAAI Conference Artificial Intelligence, AAAI 2019, The Thirty-Fire Innovative Applications of Artifiiial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Havaii, USA, January 27- February 1, 2019. pages 590-597. AAAI Press, 2019. doi: 10. 10.1609/AAAL 133101.3300000000 URL httplstts:://dee org 10. 1609/ aaai v33101. 3301590
[39] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.066 1909.0616,, 2019.
[40] Alistair E. W. Johnson, Tom J. Pollard, Seth J. Berkowitz, Nathanicl R. Greenbaum, Matthew Lungren, Chih-ying Deng, Roger G. Mark, and Steven Homg. MIMIC-CRR: A large publicly available database of labeled chest radiographs. CoRR, abs/1901 .07000022222 2019. URL http::ttt////iiil orgg 1901997700
abs/1901.07042
[41] Alistair EW Johnson, Lucas Bulgarelli. Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao. Benjamin Moody, Brian Gow, et al. Mimic-iv, a freely accessible electronic health record dataset. Scientific data, 10(1):1, 2023.
[42] Xuan Kan, Wei Dai, Hejie Cui, Zilong Zhang, Ying Guo, and Carl Yang. Brain network transformer. Advances in Neural Information Processing Systems, 35:25586-25555 25599, 2022.
[43] Schun Kim. Learning general representation of 12-lead electrocardiogram with a joint-embedding predic- tive architecture. arXiv preprint arXiv:2410.08552 2024.
[44] Adrienne Kline, Hanyin Wang, Yikuan Li. Saya Dennis, Meghan Hutch, Zhenxing Xu, Fei Wang, Feixiong Cheng, and Yuan Luo. Multimodal machine learning in precision health: A scoping review. npj Digital Medicine, 5(1):111, 2022.
[45] Nesaretnam Barr Kumarakulasinghe, Tobias Blomberg, Jintai Liu, Alexandra Saraiva Leao, and Panagintis Papapetrou. Evaluating local interpretable model-agnostic explanations on clinical machine learning classification models. In 2020 IEFE 33rd international symposium computer-based medical systems (CBMS), pages 7-12. IEEE, 2020.
[46] Yuxiang Lai, Jike Zhong, Ming Li, Shitian Zhao, and Xiaofeng Yang. Med-rl; Reinforcement learning for generalizable medical reasoning in vision- models. arXiv preprint MxXX:22503.13933 2025.
13

---

[47] Xiang Lan, Feng Wu, Kai He, Qinghao Zhao, Shenda Hong, and Mengling Feng, Gem: Empowering milm for grounded eeg understanding with time series and images. arXiv preprini N50:::2503000073, 2025.
[48] Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang. Tristan Nau- mann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large language-and and-vision assistant for biomedicine in one day. Alice Oh. Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023.
[49] Stella Li, Vidhisha Balachandran, Shangbin Feng, Jonathan Ilgen, Emma Pierson, Pang Wei W Koh, and Yulia Tsvetkov. Mediq: Question-asking Ilms and benchmark for reliable interactive clinical reasoning. Advances in Neural Information Processing Systems, 37:28858-28888 28888, 2024.
[50] Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effective, and efficient reinforeement learning method for aligning large language models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL BtRphttps:/// /ppp//eveeeeeeeeeeeeeeeeeee
[51] Paul Pu Liang, Akshay Goindani, Talha Chafckar, Leena Mathur, Haofei Yu, Russ Salakhutdinov, and Louis-Philippe Morency. Hemm: Holistic evaluation of multimodal foundation models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[52] Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations & trends in multimodal machine learning: Principles, challenges, and open questions. ACM Computing Surveys, 16(10):1-4,, 2024.
[53] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference computer vision, pages 2017.
[54] Feifei Liu, Chengyu Liu, Lina Zhao, Xiangyu Zhang, Xiaoling Wu, Xiaoyan Xu, Yulin Liu, Caiyun Ma, Shoushui Wei, Zhiqiang Hc, al. An open access database for evaluating the algorithms of clectrocardio- gram rhythm and morphology abnormality detection. Journal of Medical Imaging and Health Informatics, 8(7):1368-133 1373, 2018.
[55] Mary M Lucas, Justin Yang, Jon Pomeroy, and Christopher C. Yang. Reasoning with large language models for medical question answering. Journal of the American Medical Informatics Association, (9)) 1964-1975, 2024.
[56] Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. Med-flamingo: multimodal medical few-shot learner. In Stefan Hegselmann, Antonio Parziale, Divya Shanmugam, Shengpu Tang, Mercy Nyamewaa Asiedu, Serina Chang. Tom Hartvigsen, and Harvineet Singh, editors, Machine Learning for Health, MLAH@Neurie 2023, 10 December 2023, New Orleans, Louisiana, USA, volume 225 of Proceedings of Machine Learning Research, pages 353-367, PMLR, 2023. URL htttps://proceedinngg mlr, press/ / m3.r222. html
[57] Abhishek Naik, Yi Wan, Manan Tomar, and Richard S Sutton. Reward centering. arXiv preprint arXiv:2405.099 099999999099.........99999999999 2024.
[58] Nida Nasir, Afreen Kansal, Feras Barneih, Omar Al-Shaltone, Talal Bonny, Mohammad Al-Shabi, and Ahmed Al Shammaa. Multi-modal image classification of covid-19 19 cases using computed tomography and x-rays scans. Intelligent Systems with Applications, 17:200160, 2023.
[59] Ha Quy Nguyen, Hieu Huy Pham, Tuan Linh Le. Minh Dao. and Khanh Lam. Vindr-exr: An open dataset of chest x-rays with radiologist annotations. PhysioNet, 2021. doi: 10.13026/3kn--223 b287. URL https://áoi. orw/ 1ttp://...10.////// 13026/3akn 3atn/-kknnnn88
[60] OpenAl. Gpt-40 mini: advancing cost-cfficient intelligence. Online technical report, OpenAl, 2025, URL wwt.e/oo//nno/ http://dexxx//e gpt- 40 mini- wwiii-adveen advancing--n..--aa -ettteeffiiiie----tllllggeeeeee intelligence/
[61]
OpenAL. Openai 03 and 04-mini system card. Technical report. OpenAl, April 2025.
URL https: :ttts://ccdn. openai..n....ooooo
33-and-aa 04 mini system- - card.pdf
[62] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 15:27730-27744 2022.
14

---

[63] Ece Ozkan and Xavier Boix. Multi-domain improves classification in out-of-distribtiion and data-limited scenarios for medical image analysis. Scientific Reports, 14(1)24441, 2024.
[64] Andre G.C. Pacheco, Gustavo R. Lima, Amanda S. Salomão, Breno Krohling, Igor P. Biral, Gabriel G. de Angelo, Fáhio C.R. Alves Jr, José G.M. Exgario, Alana C. Simora, Pedro B.C. Castro, Felipe B. Rodrigues, Patricia H.I., Frasson, Renato A. Krohling, Helder Knidel, Maria C.S. Santos, Rachel B. do Espirito Santo, Telma L.S.G. G. Macedo, Tania R.P. Canuto, and Luíz F.S. de Barros. Pad-ufes-20: A skin lesion dataset composed of patient data and clinical images collected from smartphones. Data in Brief. 32: 106221, 2020. doi: 10. 10.1016/j.dib.2022 2020.............. 106221.. URL https://doi. orgg...r//// 20rg//0..or 1016/ 1.16/j.dib. 2020 106221
[65] Jiazhen Pan. Che Liu, Junde Wu, Fenglin Liu. Jiayuan Zhu, Hongwei Bran Li, Chen Chen, Cheng Ouyang. and Daniel Rueckert. Medvim-rl: Incentivizing medical reasoning capability of vision-language models (ylms) via reinforcement learning. arXiv preprint arXiv:2502. 19634, 2025.
[66] Liron Pantanowitz, Matthew Hanna, Joshua Pantanowitz, Joe Lennerz, Walter H. Henricks, Peter Shen, Bruce Quinn, Shannon Bennet, and Hooman H. Rashidi. Regulatory aspects of artificial intelligence and machine learning. Modern Pathology, 37(12):100699, 2024.
[67] João Pedrosa, Carlos Guilherme, Patrícia Márcio, João André, Isabel Eduardo, and Aurélio António. Lndb dataset (version 4). 17th International Conference on Image Analysis and Recognition (ICIAR 2020). Zenodo, 2023, doi: 2.5228/2cnndo..oo / URL httpss::://do org/10. httpg//....81///// zenodo. 8348419
[68] Hieu Huy Pham, Trung H. Nguyen, and Ha Quy Nguyen. Vindr-mammo: A large-scale benchmark dataset for computer-aided detection and diagnosis in full-field digital mammographh. PhysioNet, 2022. URL
httpps://oo/ org/ 10. 13026/br2v-705 bttp/bb2vv-77
[69] Lawyer-Lee R., Gimenez F., Hoogi A., and Rubin D. Curated breast imaging subset of digital database for screening mammography (chis-ddsm) [data set), 2016.
[70] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-53741, 2023.
[71] V. Rotemberg, N. Kurtansky, B. Setz-Stablein, and et al. A patient-centric dataset of images and metadata for identifying melanomas using clinical context. Scientific Data, 8(1)33, 2021. doi: 10. 00.108//41199-001 1008110-- httpshtts:///o.. org/ 10. 1038/ 0038//41597-02111 021-0
[72] Madeline Sagona, Tinglong Dai, Mario Macis, and Michael Darden. Trust in ai-assisted health systems and aï's trust in humans. npj Health Systems, 2(1):10, 2025.
[73] Michael Santacroce, Yadong Lu, Han Yu, Yuanzhi Li, and Yelong Shen. Efficient rihf: Reducing the memory usage of ppo. arXiv preprint arXiv:2309.00554, 2023.
[74] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1707v 2017.
[75] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits mathematical reasoning in open language models. arviv preprint arXiv:2402.03:3 03300,, 2024.
[76] Gregor Stiglic, Primoz Kocbek, Nino Fijacko, Marinka Zitnik, Katrien Verbert, and Leona Cilar. Inter- pretability of machine learning-based prediction models in healthcare. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 10(5):c1379, 2020.
[77] Hong Sun, Kristof Depraetere, Laurent Meesseman, Patricia Cabanillas Silva, Ralph Szymanowsky, Janis Fliegenschmidt, Nikolai Hulde, Vera von Dossow, Martijn Vanbiervliet, Jos De Baerdemacker, et al. Machine learning based prediction models for different clinical risks in different hospitals: evaluation of live performance. Journal of Medical Internet Research, 24(6)=e34295, 2022.
[78] Hidenori Takahashi, Hironobu Tampo, Yusuke Arai, Yuji Inoue, and Hidetoshi Kawashima. Applying artificial intelligence discasc staging: Deep learning for improved staging of diabetic retinopathy. PloS one, 12(6))e01 79790, 2017.
[79] Qwen Team. Qwen3 technical report. Technical report, Alibaba. 2025. URL httpp::/t// com/ DeennK/Qvee3/llobb//ee Saaaa//eeeeeeeeeeeee wwwheeeaalhheppeeeeeec Rdp Online; accessed May 14, 2025.
[80] Qiaoying Teng. Zhe Liu, Yuqing Song, Kai Han, and Yang Lu. A survey on the interpretability of deep learning in medical diagnosis. Multinedia Systems, 28(6):2335 2355,,23555500 2022.
15

---

[81] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The H1AM1000 dataset: A large collection of multi-source dermatoscopic images of common pigmented skin lesions. CoRR, abs/1803. 10417, 2018. BtLLhttp://aaaat
/////1803...00 10417
[82] Patrick Wagner, Nils Strodthoff, Ralf-Dieter Bousseljot. Dieter Kreiseler, Fatima Lunze, Wojciech Samek, and Tobias Schaeffter. Ptb-xl, a large publicly available electrocardiography dataset. Scientific data, 7(1): 1-15, 2020.
[83] Zhongwei Wan, Che Liu, Xin Wang, Chaofan Tao, Hui Shen, Zhenwu Peng, Jie Fu, Rossella Arcucci, Huaxiu Yao, and Mi Zhang. Electrocardiogram instruction tuning for report generation. arXiv e-prints, pages arXiv-2403, 2403, 2024.
[84]
Nina Wie. Covid-blues: A large-scale lung ultrasound dataset for covid-19 diagnosis. lattps://githu: com/ com Www.aai///VVID---- BLUES 2021. Maastricht University Medical Center.
[85] Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang. and Weidi Xie. Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data. arXiv preprint arXiv:2308. 02463, 2023.
[86] Peng Xia, Ze Chen, Juanxi Tian, Yangrui Gong, Ruibo Hou, Yue Xu, Zhenbang Wu, Zhiyuan Fan, Yiyang Zhou, Kangyu Zhu, et al. Cares: A comprehensive benchmark of trustworthiness in medical vision language models. Advances in Neural Information Processing Systems, 37:140334 140365, 2024.
[87] Chao Yan, Yao Yan. Zhiyu Wan, Ziqi Zhang. Larsson Omberg. Justin Guinney, Sean D Mooney, and Bradlcy A Malin. A multifaceted benchmarking of synthetic electronic health record generation models. Nature communications, 13(1):7609, 2022.
[88] Jin Ye, Guoan Wang. Yanjun Li. Zhongying Deng, Wei Li. Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, Benyou Wang, et al. Gmai Gmai-mmbench: comprehensive multimodal evaluation benchmark towards general medical ai. Advances Neural Information Processing Systems, 37:94327-944222 2024.
[89] Liping Yu, Cristiano Cuppini, Jinghong Xu, Benjamin Rowland, and Barry E Stein. Cross-modal competition: the default computation for multisensory processing. Journal of Neuroscience, 39(8); 1374-1385, 2019.
[90] Kai Zhang, Rong Zhou, Eashan Adhikarla, Zhiling Yan, Yixin Liu, Jun Yu, Zhengliang Liu, Xun Chen, Brian D Davison, Hui Ren, et al. A generalist vision-language foundation model for diverse biomedical tasks. Nature Medicine, pages 1-13, 2024.
[91] Sheng Zhang, Qianchu Liu, Guanghui Qin, Tristan Naumann, and Hoifung Poon. Med-rlvr: Emerging medical reasoning from 3h base model via reinforcement learning. arXiv preprint arXiv:2502. 19655, 2025.
[92] Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc- vqa: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305.10411: 10415, 2023.
[93] Jianwei Zheng, Huimin Chu, Danicle Struppa, Jiamming Zhang, Sir Magdi Yacoub, Hesham El-Askary, Authony Chang, Louis Ehwerhemuepha, Islam Abudayyeh, Alexander Barrett, al. Optimal multi-stage arthythmia classification approach. Scientific reports, 10( 1011)8889, 2020.
[94] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 2019.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We explained our method in detail in Sec. then supported each points with extensive experiments in
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed our limitations on how the reasoning is learned in the conclusion paragraph.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-speciiiiatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer: [NA]
Justification: In this work, we describe a novel way of training a reasoning model across heterogeneous domains. We detailed the assumption (that a set of domains must present). However, the effectiveness the method is primarily proved via experiments, not theoreti- cally.
Guidelines:
The answer NA means that the paper does not include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We release our repository containing the code used for all experiments. We also include all the datasets we used.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
18

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We open source training pipeline, model weights and training hyperparam- eters. The dataset used in our model fully public, with little to no license restrictions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
publicc/guides//deetttmi http:///uuuuneeoeeeeeeaaaasaaaaa www.aammmiiiiooooooooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips //iiss.c/pput
guides PoodeSSumiissiooool.....ooo
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We describe our training and test details in App. D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We did 4 separate runs with different seeds for each experiment in Table 2 and included the standard deviation in Appendix Table 7
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We described compute resources in App.
D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https::ts http://neeripppp httpeeps...ppuucceeeeeeee.eeeeee cccppbllc/Bthcccsiieeelllll
Answer: [Yes]
Justification:
We have reviewed the code of ethics and included a impact statement in A
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discussed social impacts and included a detailed discussion in the impact statement under App. A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
20

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: The model is mainly oriented for clinical use. However, before extensive real world testings (like human trials), the model is not suitable for real clinical deployment. We address this on the model weights download page, emphasizing this is a reseach review, not product approved by related federal agencies.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite each dataset used in training in Tab.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
21

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We detail the usage of our models throughout the paper. The code, model weights, training pipeline will be shared publically with no license restriction, to the extent that is allowed by the training dataset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: All training data are available publicly. We did not collect any data with human subjects. All data from the public datasets are properly anonymized.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: All training data are available publicly. We did not collect any data with human subjects.
Guidelines:
22

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: This work proposes a novel approach in training multimodal large language models with reinforcement learning for clincial use cases. We have described the approach in details in Sec. 3 and Sec.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy (https://neure) / for what should or should be described.
23

---

A Impact Statement
This paper presents the development, training, and evaluation of QoQ-Med, multimodal language model for clinical reasoning. QoQ-Med the first foundational model to jointly process and reason medical images, ECG signals, and clinical text. By incorporating underused physiological signals such as ECG, our model moves beyond vision-centric approaches and supports broader applications cardiology. These robust models can then assist clinicians in interpreting diverse forms of patient data, identifying patterns, and generating more complete diagnostic assessments.
As multimodal health care models become increasingly used, transferability and accountability become the focus. It is no longer sufficient for models to produce accurate predictions, but also to justify how those predictions trust and facilitate broader deployment healthcare AIs in real-world settings.
made. By incorporating reasoning component, aim to build
To address data and privacy concerns, we have taken appropriate steps to access consented data. We have anonymized sensitive personal information such as gender and race from the datasets we use.
The model is mainly oriented for clinical use. However, before extensive real world testings (like human trials), the model is suitable for real clinical deployment. We address this during the download of model weights, emphasizing this is a research review, not a product approved by federal agencies.
B
DRPO Implementation Details
B.1 Elbow Method for K-means Cluster Selection
To automatically determine the optimal number of clusters kup for each domain, we implement the elbow method that balances clustering granularity against overfitting. For each domain's question vectors, we:
Compute the inertia (sum of squared distances to assigned centroids) for range of cluster counts k E {1, 11,,,....,imaan kimax where kimax = min (10, Nunique) and Nunique is the number of unique question vectors.
2. Calculate the decrease in inertia A4, = 4-11 4 for each additional cluster.
3. Identify the "elbow point" where adding more elusters yields diminishing returns, specifically where ATe < T A for tolerance parameter 0.10.
This approach prevents excessive fragmentation of the question space while still capturing meaningful patterns in question difficulty. In practice, this typically results in a small number of clusters (2-5) per domain, which is computationally efficient while capturing the primary patterns in question difficulty.
B.2 KL-aware Regularization
The hierarchical scaling in DRPO can increase the variance of advantages, potentially leading to optimization instability where outliers dominate updates. To address this, we implement KL-aware regularization that dampens small percentage of advantages proportionally their KL divergence from the reference model.
For each response Og,i, compute question-level KI. divergence:
k(g.t) = Σ
Te log Teaf g.t
(4)
We then apply an inverse-linear regularizer to the scaled advantages:
Lp soeated m(gait) 1p + max (q3.))
(5)
k(g.t); 0)
where lp the p-th percentile of the values { poated kis} within the mini-batch, and p = 0.9 in our experiments.
24

---

Table 5: Comparison with other general domain and medical MLLMs. Acc: Accuracy, Fl: F1 Score. QoQ-Med-7B is the best model among open-source clinical MLLMs across all categories. rivels closed source models in terms of accuracy but has lower Fl score.
Chest X-Ray
Mammo. Dermoscopy
CT Scan
Fundus
Ultrusound
Pathology
Overall
Model
Acc
F1
F1
Acc
F1
Acc
Ace
F1 Ace
F1
F1
Acc
F1
Open Source Clinical VLMs
LLa Va-Med
.088
.466
.48
434
.448
.363
.000
.287
Med-R1
.641
.002
.996
.029
630
.000
530
.007
.671
.062
.549
.008
.550
.000
.659
.000
.603
.013
Closed Source Commercial VLMs
04-mini
_198
.189
.297
271
.46
.45
441
.389
.378
.354
.66
.205
.725
.775
.514
.522
411
.394
GPT-4o
.244
056
.42
.383
222
.401
229
.24
.231
-908
.364
QuQ-Med-77B
.253
.240
.652
.286
QoQ-Med-32B
.732
.203
.758
.224
.661
.290
.571
.257
.691
.148
.609
.260
.904
.810
.726
.336
.707
.316
This regularization has several advantages:
1. It selectively dampens responses with both high advantage and high KI. divergence, which are likely to represent overconfident "shortcuts" rather than genuine improvements.
2. It requires no additional computation since the KI. divergence k; is already calculated for the main DRPO objective.
3. It preserves most advantages while providing robustness against spurious outliers.
B.3 Auxiliary Rewards
In addition to the primary accuracy and semantic alignment rewards described in Section 3.4, we implement auxiliary rewards to enhance the overall quality and completeness of the model's outputs.
Format reward. Clinical diagnoses require structured and complete explanations. To ensure the model provides comprehensive evidence for each diagnosis, implement a format reward:
fint 1 if bounding box is present for every predicted label, otherwise.
This binary reward penalizes responses that make claims without providing visual evidence, encour- aging the model to explicitly support each diagnostic conclusion with localized visual evidence. This particularly important in clinical settings where unsubstantiated diagnoses can lead improper treatment decisions.
Visual coherence. When the dataset includes multiple images views, we may optionally include a visual coherence reward that encourages consistency in identified regions across different views of the same structure. This is defined as:
.oh
(bu T(b.,)))
V
DEV
where V; is the set of views, bi.e is the bounding box in view and T is a transformation function that maps between coordinate systems of different views. This reward is particularly valuable for modalities like echocardiography where multiple views of the same structure available.
These auxiliary rewards comprise small portion of the total reward signal but play an important role in shaping the model's outputs toward clinically useful formats. In practice, we found that even with small weights, these auxiliary rewards significantly improve the practical usability of model outputs as judged by clinical evaluators.
C Details of the Datasets used in Training and Validation
We use the CLIMB dataset to train our QoQ-Med model. CLIMB a multimodal clinical diagnosis dataset introduced in Dai al. [22] It contains mixture of 44 publicly available datasets across 13
25

---

Model
GRPO-Balanced (Acc)
DRPO-Balanced (Ace)
Accuracy Comparison
1
0.9
8888
Accuracy
0.8
0,823
0.770
0.7
E12 1229
0,722
6:703
089
0, 566
6,689
0.65g
0.6
3.656
0.571
0.5
Chest X- X-Ray
Mammo
Dermoscopy
CT Scan
Fundus
Ultrasound
MRI
Pathology
Overall
F1 Score Comparison
1
0.8
0.81
0:7M0
F1 Score
0.6
0.590
0.4
0.449
0. 366
0.372
0:34
0.320
0.362
0.2
0.229
0.557
0.257
0,302
164
0.145
0.064
6,013
Chest X- R-Ray
Mammo.
Dermoscopy
CT C Scan
Fundus
Ultrasound
MRI
Pathology
Overall
Modality
Figure 5: Comparison of DRPO and GRPO on Balanced Datasets. Acc: Accuracy, Fl: Score. To remove the influence of imbalanced dataset, further conducted experiment subset of the 30 datasets, where each dataset share the same portion in the training dataset mix. This helps compare method with similar methods like loss scaling (i.e. focal loss (53) and upsamplin//dowssampiigg techniques. Thanks to the dynamic weighting based on both difficulty and scarcity, method better captures the changing dynamics throughout the training, allowing to perform better than GRPO even with a perfectly balanced dataset.
balanced
26

---

Table 6: Details of individual datasets used in training of QoQ-Med.
Dataset
# Classes
Classes
PTB-XLI XL 82
7
Normal, Conduction Delay (CD), Hypertrophy (HYP), Myocardial Infare- tion (MI), Sinus Baapaaraae//raaaaaaaaaaaaaeooaea (STTC), Atrial Fibrilla-
tion/Atrial Flutter (A. Fib/AAlutter/ Other
Chapman-San Shamaann- 9
Same as PTB-XI.
Georgia 4
7
Same as PTB-XI.
CPSC [54]
7
Same PTB-XL
CheXpert 38
14
Atelectasis, Cardiomegaly, Consolidation, Edema, Enlarged Cardiomedi- astinum. Fracture, Lung Lesion. Lung Opacity, Pleural Effusion, Pneumonia, Pncumothorax, Plcural Other, Support Devices, No Finding
MIMIC-XXR 40
14
Same as CheXpert
VinDr-CXR (59
6
Lung tumor, Pncumonia, Tuberculosis, COPD, Other diseases, No finding Normal, Bacterial Pneumonia, COVID-19, Viral Pneumonia
COVID-19
4
CoronaHack
Normal, Bacterial Pneumonia, Viral Pneumonia
VinDr-Mammo 68
5
BI-RAD 1-5
CBIS-DDSM 699
6
BI-RAD 0-5
CMMD[18]
2
Benign, Malignant
ISIC-2020
Malignant, Benign
HAM10000
Melanoma (MFL.), Nevus (NV), Basal Cell Carcinoma (BCC), Actinic Kerato-
sis/Intracpithelial Carcinoma (AKTEC), Other (OTHER)
PAD-UFES-200 64
5
Melanoma (MEL), Nevus (NV), Basal Cell Carcinoma (BCC), Actinic Kerato-
sis/Intraepithelial Carcinoma (AKIEC), Other (OTHER)
Messidor- 2
None, Mild DR, Moderate DR, Severe DR, PDR No DR, Mild, Moderate. Severe, Proliferative DR SDR (simple diabetic retinopathy), PPDR (pre-proliferative diabetic retinopa- thy), PDR (proliferative diabetic retinopathy)
APTOS 2019
Jichi 78
3
LNDb
nodule 2 3mm, nodule <3mm, non-nodule
INSPECT
No PE, Acute Subsegmental-onf, PE, Acute PE, Subsegmental-oni PE,
Chronic PE
KiTS23 30
2
Benign, Malignant
Hemorrhage 39
2
No Hemorrhage, Has Hemorrhage
RSPECT
No PE, Chronic Acute PE Normal, Malignant, Benign
BUSI
3
DOVID-BLUES 84
2
Has COVID. No COVID Covid, Pncumonia, Normal No Tumor, Pituitary Tumor, Glioma Tumor, Meningioma Tumor Yes, No (presence of tumors)
COVID-US 25
3
Brain Tumor 10
Brain Tumor
LC25000
5
Colon adenocarcinomas, Benign colon, Lung adenocarcinomas, Lung squa- mous cell carcinomas, Benign lung
BCSS
4
Tumor, Stroma, Lymphocytic infiltrate, Necrosis/debris
domains. In this work, we use the vision (2D and 3D) and ECG subset the CLIMB dataset, which contains 707K 2D, 1.83M 3D, and 78.9K ECG data.
For RQ1, we use the same training/validation split as in the original CLIMB dataset, which largely inherits the splits from the original papers.
list of datasets used in the paper included in Table 6
C.1 QA Pair Generation
In particular, we use the QA version of the datasets, created in the following way:
For every sample n, DR with label set y/ C Vk. paired question and answer (gi a,) is generated as follows:
Question d:t natural-language query formed by combining
a brief task prompt (c.g., "identify the primary abnormality"), a cue about the input modality (e.g., "on the chest X-ray"), the list of allowable labels drawn from Vg.
27

---

Answer a;: the gold label derived from formatted as
a single class a, E V& when categories are mutually exclusive, or subset a₁ S V2 when multiple findings may co-occur.
Formally, we introduce a mapping v (ri, Vc. Vi.) + (gi az) that converts each classification instance into its question-answer analogue without altering the underlying task semantics. As illustration, consider the CheXpert dataset for chest X-ray diagnosis:
Question. "Shown above frontal chest radiograph. State the predominant finding, choosing from: No Finding, Cardiomegaly, Pleural Effusion, Lung Con- solidation, Atelectasis."
Answer. "Pleural Effusion"
For multi-label tasks, predictions are judged with an order-independent match: a response â; is deemed correct iff à; a, irrespective of label ordering. This criterion accommodates scenarios where several conditions may be present simultaneously and listed in any sequence.
C.2 Defintion of Novel Tasks, Understudied Modalities and Underrepresented Regions
We follows the original work's definition of Novel Tasks, Understudied Modalities and Underrepre- sented Regions:
Novel Tasks. Novel tasks are defined as new diseases or conditions not present previously, Examples include COVID.
Understudied Modalities. Building on the criteria set forth in the original study, we characterize each modality along two axes: (i) research attention, estimated by the number of Google Scholar hits returned for standardized queries such as "[modality] classification and (ii) public data availability counted as the total number of openly released samples. Based on both criteria, mammography and ECG appear under-studied in the literature, whereas ultrasound is chiefly limited by data scarcity.
Underrepresented Regions. The underrepresented regions are defined as datasets collected from regions historically underrepresented regions (like Africa, South America and Parts of South and Southeast Asia), regions with less economic developments (like India, Vietnam, Iraq and Brazil).
D Details of Training and Evaluation
D.1 Training Hyperparameters
Unless mentioned otherwise, we the same set of hyperparameters train the model across different training methods. The models are trained for 1 epoch on an 8xNVIDIA A100 and H200 GPU instances. For 7B models, use per-device batch size of 4, and rollout batch size of 512. The maximum context length is 8192. To ensure consistency throughout the training, we shuffle the data with seed 42 beforehand, and disable shuffling throughout the training process. To save compute, we employ early stopping, which stops training when the accuracy converges and stops improving. Most 7B model trainings converge within days of training. The training of 32B model takes more than weeks to train on an 8x A 100 machine, so a 8xH200 machine used to speed up the training process of 32B model via faster interconnect.
We build our training pipeline based on the FSDP and VeRL framework, with vLLM to speed up reasoning training with KV Cache. We use learning rate of le-6, weight decay of le-2, and KL coefficient of le-4. We use Adam W full model training at 32-bit precision for all 7B models, and at 16-bit precision for the training of the 32B model.
Throughout the training and evaluation, the 3D images and videos are sliced uniformly into frames, before getting concatenated into the model's 's input. Images are downsized so that they have max pixel count of 524,288. For all RI. training methods, we use Qwwn2.55-VL-77 [9] as the base model. For QvQ-Med, an ECG encoder named ECG-JEPA 43 is prepended.
28

---

Environmental Impacts. Experiments on 7B/32B models exhibit minimal environmental impacts. Our A100 SXM has TDP of 400W. Assuming CPUs and other components have combined TDP of 600W, training of 7B model requires electricity of 182.4kWh. Under the same assumption, training 32B model requires an electricity of 571.2 kWh. We would also like to point out that releasing public models as we did has positive environmental impact, researchers can directly use or finetune the model on the domain data, instead of doing the training repeatedly.
The following sections detail the evaluation metrics used in our experiments to assess the performance of DRPO and baseline methods across various clinical tasks and datasets.
D.2
RQ1: Performance Comparison with Critic-free RL Methods
For comparing DRPO with other critic-free RL methods and models, we formulate each clinical diagnosis task as a multi-label classification problem over all possible labels in dataset. The evaluation metrics are computed as follows:
TP + TN TP FP+
Accuracy label
(8)
TP TP FP
Precision
(9)
TP
Recalliabel
(10)
TP PN
Precisionaael Recallabel
FI label
2
(11)
Precisionaael + Recallaabel
where TP, TN, FP, and FN represent true positives, true negatives, false positives, and false negatives, respectively, for each label across all samples.
To account for potential class imbalance, compute the balanced accuracy for each dataset:
Balanced Accuracy dataset
Accuracy label
(12)
T.
LEL
where L is the all labels in the dataset. Similarly, we compute the macro-F1 score for each dataset:
Macro-Fl Flataset
Fliahel
(13)
JEL
To obtain domain-level metrics, we perform unweighted averaging across all datasets within each clinical domain:
1
Metricdomain
MetriCdaraset
(14)
Daomain
Donasin
where Doomain represents the set datasets in a specific clinical domain, and Metric can either balanced accuracy macro-F1.
Finally, we compute the overall performance by averaging the domain-leve metrics across all clinical domains:
9
MetriCoverall
MetriCaamain,
(15)
29

---

D.3 RQ2: Multimodal Input Handling
For evaluating the model's s ability to handle mixed multimodal inputs on the MIMIC-IV dataset, we similar metrics as in RQI. We formulate two prediction tasks:
D.3.1 Length of Stay (LOS) Prediction
LOS prediction is formulated as a 4-class classification problem. Patient stays are binned into the following categories:
Class A: 0-4 days
Class B: 5-8 days
Class C: 9-12 days
Class D: more than 12 days
D.3.2 48-hour In-Hospital Mortality (48-IHM) Prediction
48-IHM is formulated as binary classification task. A positive label is assigned if either:
The patient's ss death date within 48 hours of admission, or
The patient is discharged to hospice within 48 hours of admission
For both tasks, we compute accuracy and Fl scores. For the multi-class LOS prediction, we compute:
Number of correct predictions
Accuracy,.os =
(16)
Total number of samples
Macro-Fl
Fl (E(A,B,C,D)
Σ
(17)
For the binary 48-IHM prediction:
TP AP+TV TN TP TN + FP FN
Accuracy4s-HHM
(18)
Precision Recall
FI 22 Precision + Recall
(19)
where Precision and Recall are computed as defined RQ1.
The input data for both tasks consists of:
Chest X-ray images from MIMIC-IV-CXR
12-lead ECG traces from MIMIC-IV-ECG
Textual electronic health record data from MIMIC-IV, including vital signs, lab measure- ments, treatments, medications, and demographics collected within 48 hours of admission
Data without timestamps (e.g., diagnoses) and discharge summaries and radiology reports are excluded from the input.
D.4 RQ3: Quality of Reasoning Traces and Bounding Boxes
To evaluate the quality of reasoning traces and bounding boxes produced by DRPO, we employ both quantitative and qualitative measures.
30

---

D.4.1 Bounding Box Quality
For quantitatively assessing bounding box quality, we compute the Intersection over Union (IoU) against ground truth segmentations:
Bprea O Bgl
IoU
(20)
|prred U Bgl
where Bpred the predicted bounding box and By is the ground truth segmentation.
For datasets without available ground truth segmentations, we generate segmentations using the BiomedParse model. Samples with no meaningful segmentations are excluded from this evaluation.
D.4.2 Reasoning Trace Quality
For qualitative assessment reasoning traces, we collaborate with clinicians to evaluate the relevance of each reasoning statement the final diagnosis. The evaluation categorizes statements into three levels:
High Relevance: The statement critical and closely related to the final diagnosis.
Low Relevance: The statement relates to relevant medical concepts but is not necessary to arrive at the diagnosis.
No Relevance: The statement is not helpful to the final diagnosis process.
We compute the distribution of these relevance categories across the validation set of both CLIMB and MIMIC-IV datasets assess the overall quality of the model's reasoning process.
31

---

E Relevance Annotation of Reasoning Traces
To better understand the reasoning process and help model produce better reasoning in the future, we collaborated with clinicians to annotate the reasoning traces on randomly selected samples in the validation set, which is representative of model's real world performance on unseen samples. We present unfiltered, original model reasoning traces here with color-coded highlights indicating relevance. Phrases marked yellow represent information low clinical relevance, pink highlights indicate high clinical relevance, and aqua denotes content judged have no relevance the diagnosis.
Question:
Above a brain MRI scan patient. What type tumor, any, is present this brain MRI scan?
Answer with one of the following: No Tumor, Glioma Tumor, Menin- gioma Tumor, Pituitary Tumor
Ground Truth Label: No Tumor
Model's Prediction: No Tumor
Reasoning: To analyze the brain MRI scan, we should consider the
following steps:
1. Understanding MRI Scans:
MRI (Magnetic Resonance Imaging) is diagnostic tool that uses powerful magnet, radio waves, and a computer to create detailed images of organs and structures inside the body, including the brain.
It is particularly useful for identifying tumors because it can show differences in tissue density. which can indicate whether mass solid, liquid, blood-filled, and whether benign or malignant.
2. Identify Key Features of Tumor on MRI:
Hypointense//soineense on T1:
Most normal brain tissue appears dark on a Tl Tl-weighted MRI. so tumor that is the same color normal brain tissue called "isointense" "hypointense."
This means it is often harder to spot a T1 image, but can be seen on the contrast-en- hanced images.
A tumor that the color brain tissue T1-weighted MRI is often sign of glioma, which is type of brain tumor that originates from glial cells in the brain.
Hyperintense on T2:
A tumor that appears bright on T2-weighted MRI is called "hyperintense.
This means is often easier to see on T2 image, hut is a less specific sign. It be seen in many brain tumors, including gliomas, meningiomas, and pituitary tumors.
The brightness T2 images often associated with edema, which sign of inflam- mation and swelling around the tumor, which is common feature of many types of tumors.
The image provided is T2 weighted MRI, which usually makes tumors appear bright, and in this case, the tumor appears hyperintense, consistent with typical appearance of meningioma MRI. Meningiomas on T1 images, which what we see here. They are usually henign but can be quite large and cause symptoms. However, without further context, it impossible to determine definitively the type of tumor. The MRI is simply a tool, and definitive diagnosis often requires combination imaging, a patient's medical history, and other tests such as biopsy.
often hyperintense on T2 images and hypointense
32

---

Above is chest X-ray image of patient. What is the diag- nosis of patient the X-ray image?
Question:
Answer with multiple phrases from the following: No Finding, Enlarged Cardiomediastinum, Cardiomegaly, Lung Opacity, Lung Lesion, Edema, Consolication, Pneumonia, Atelectasis, Pneumothorax, Pleural Effusion, Pleural Other, Fracture, Support Devices
Ground Truth Label: Support Devices
Model's Prediction: No Findings
Reasoning to annotate: To analyze the chest x-ray image, need
consider the following steps:
1. Examine the Image for Anomalies:
Heart and Mediastinum: The mediastinum is area in chest that contains the heart, great vessels, trachca, and esophagus. In this chest x-ray, the mediastinal area appears to be enlarged, which a strong indicator of pathologies such as cardiomegaly enlarged heart) or other mediastinal pathologies, but it's essential understand that this is an interpretation and should be confirmed by a medical professional.
Size and Shape
Lung Opacity: The lung fields are evaluated any signs of increased density, which can indicate fluid, air, tissue abnormalities. The x-ray shows increased density in the lung fields, which can suggest consolidation, such as from pncumonia, or other pathology. The appearance of the lung fields is consistent with signs of fluid, which is sign of plcural effusion.
Support Devices: The x-ray shows a pacemaker, which a medical device used to regulate the heartheat. This normal finding some patients and does indicate pathology.
2. Consider the X-ray Protocol and Patient Information:
The patient's 's position (supine) is noted. This position is often used in chest x-rays ensure that the patient is in comfortable position and to ensure that the image is clear and useful for interpretation. The radiographic markers (e.g.. the "P" and "R" signs) standard for x-rays to indicate the position of the patient and to ensure that the image is correctly oriented.
3. Analysis of the Image for Any Indications of Pathology:
Pleural Effusion: The increased density in the lung fields indicative of pleural effusion, which is buildup of fluid in the pleural space, the between the lungs and the chest wall. This is common sign of heart failure. liver disease, or malignancy.
Cardiomegaly: The enlarged heart size, though not definitive without further clinical context, is suggested by the mediastinal widening, though the definitive interpretation is that this is more about the mediastinum and not a direct indicator of cardiomegaly.
Based on the analysis, the most likely findings in this x-ray image suggestive of:
Plcural Effusion due the increased density in the lung fields.
No Finding for other abnormalities, given that there are no signs of fractures, pneumothorax, or pneumonia.
33

---

Question: Above lung ultrasound video. What is the diagnosis based on this lung ultrassunn?
Answer with one word from the following options: covid, pneumo- nia, normal
Ground Truth Label: covid
Model's Prediction: pneumonia
To analyze the image and determine the reasoning process for making diagnosis, we follow these steps:
Reasoning to annotate:
1. Examine the Image Type and Purpose: The image a lung
ultrasound, which is diagnostic tool used to assess lung parenchyma (the part of the lung that contains air and is responsible for exchange) for signs of abnormalitics. Ultrasound is particularly useful in the early stages respiratory disease, as it can detect changes lung tissue that may be indicative of various conditions.
Understanding Ultrasound Features: In ultrasound, different tissue types and pathologies can have distinct appearances. For example:
Hyperechoic (bright) areas often indicate a pathological process, such as air, fluid, or solid tissue. In lung ultrasound, hyperechoic areas might suggest consolidation, which is often associated with pneumonia.
Hypoechoic (dark) areas are usually indicative normal lung tissue, in some cases, they could also be associated with certain pathologies.
Echogenicity patterns can vary in different lung pathologies, these patterns can help identifying the nature of lung pathology. For example, the image shows relatively uniform, hyperechoic pattern, which is often seen in lung ultrasound when it is used to identify areas of consolidation, hallmark of pncumonia.
3. Clinical Correlation: It is essential to consider the patient's 's clinical context, such symptoms, medical history, and other diagnostic tools used in conjunction with the ultrasound. However, since the image just ultrasound, must rely the image's characteristics alone. The image shows single ultrasound frame, and often, series of ultrasound images are analyzed over time to get comprehensive view of lung involvement, but for this exercise, we are only analyzing image.
4. Interpretation of Image: In this case, the image shows pattern that is consistent with lung ultrasound findings that suggest pneumonia. The hyperechoic
characteristic of consolidation, which is sign of fluid or air accumulation the lungs, typically associated with pneumonia. This is further supported by the fact that lung ultrasound is sensitive tool for identifying pncumonia in its early stages, often before radiographic changes are visible.
34

---

Question: Above lung ultrasound video. What is the diagnosis based on this lung ultrasounn?
Answer with one word from the following options: covid, pneumo- nia, normal
Ground Truth Label: covid
Model's Prediction: pneumonia
Reasoning to annotate: To analyze the image and determine the reasoning process understanding what the image might suggest, follow these steps:
The image is lung ultrasound, which is diagnostic imaging technique used in medicine to visualize the lung parenchyma. Ultrasound is often used in emergency settings for quick, non-invasive assessment, especially cases where quick decision-making is needed. Ultrasound images grayscale, with varying shades of gray representing different tissue densities. The image shown here a grayscale ultrasound image, which is typical for ultrasound.
1. Examine the Image Type:
2. Analyze the Image Features:
Grayscale Intensity:
The image is grayscale, with varying shades gray indicating different tissue densities. In ultrasound, the brightness of the image can be indicative different pathologics. For example:
Hyperechoic (brighter than normal tissue): This often indicates air, which is why ultra- sound is used to visualize chest, as air easily detected. this image, the lesion appears hyperechoic, meaning it reflects more ultrasound waves than the surrounding tissue, which is characteristic sign that can suggest the presence of fluid or air, which is often associated with disease such pneumonia.
Hypoechoic (darker than normal tissue): This indicates less reflective tissue and is often associated with soft tissues, such tumors, or areas of fluid accumulation. The lesion this image is not hyperechoic, it is unlikely suggest air, but rather, it suggests a lesion that is darker than the surrounding lung tissue, which could be indicative of pathological condition. Shape, Size, and Border: The lesion in the image is well-defined, meaning it has clear border, which is feature that can be associated with malignancy or inflammation. The shape mostly round, which is common benign lesions. However, the exact nature of lesion would need further investigation to determine if it is benign malignant. The size and location are critical, but these cannot he determined from a single image.
3. Clinical Context:
Ultrasound often used in the context acute respiratory distress, and it is frequently used to identify signs of pneumonia, which can be associated with various signs on ultrasound such as consolidation, pleural effusion, or The image shows lesion, which may suggest pathology, but it is crucial to understand the clinical context, such as the patient's 'ss history. symptoms, and other clinical signs, to make a diagnosis.
Based on the reasoning process above, the analysis of this ultrasound image suggests that it showing signs might be indicative of pneumonia. The hyperechoic nature lesion often associated with consolidation, sign that can occur in pneumonia. However, it's important remember that ultrasound is tool and can only suggest possibilities; the final diagnosis is typically confirmed by further tests, such chest X-rays, CT scans, other medical procedures.
35

---

Question: Above is chest X-ray image of patient. What is the diag- nosis patient X-ray image?
Atelectasis, Cardiomegaly, Consolidation, Edema. Enlarged Cardiomediastinum. Frac- ture. Lung Lesion, Lung Opacity, Finding, Pleural Effusion, Pleural Other, Pneumonia, Pneumothorax. Support Devices
Answer with multiple phrases from the following:
Ground Truth Label: Pneumonia
Model's Prediction: Pneumonia
To analyze the chest x-ray image, need to understand the reasoning process for identifying signs that may sug- gest a particular diagnosis. The reasoning process involves observing the following key features:
Reasoning annotate:
2. Lung Opacity:
The density of the lung fields is evaluated. Normally, lung tissue very acrated and appears very light on the x-ray, almost completely black. there is any increased opacity the lung fields, it may indicate fluid, blood, or abnormal tissue, such consolidation, atelectasis, consolidation, which are suggestive of pneumonia. In this image, the lung fields appear darker than normal, which is sign of increased opacity. This could indicate number of pathologies. such consolidation, which often associated with pncumonia. Atelectasis, which is the collapse of part of the lung, can also appear as darker opacity on x-ray and a common cause of increased lung opacity.
3. Pleural Effusion:
This is where there is an abnormal accumulation of fluid in the pleural space, which can be seen x-rays smooth, curved opacity the base of the lung, often associated with lung collapse.
This isn't visible in this image, as there's no indication of a smooth, curved opacity.
4. Pneumothorax:
This is where air is in the pleural space, and it is usually seen as black, triangular area X-rays, because air is very radiolucent and doesn't 't allow any x-rays to pass through.
This image does not have any signs of pneumothorax, such as a triangular black area.
5. Pleural Other:
This can include other pathologies that affect the pleura, such as malignancy, infection, or inflammation. However, this image does not show signs of malignancy or inflammation, unlikely be the primary focus.
6. No Finding:
This means that the x-ray shows obvious signs of pathology, but this is unlikely in this image, as there are signs that suggest pathology.
7. Lung Lesion:
This is general term for any abnormality in the lung, which can be sign of pathology. The darker, more opaque by variety of reasons including pncumonia, atelectasis, or consolidation. In this image, the areas increased opacity the lung fields are suggestive lung lesion.
the lungs suggestive of lung lesion, which could be caused
Based on the reasoning process, the most plausible findings in this image are those that suggest increased lung opacity, which are suggestive of pathology. The reasoning process points towards lung lesions. which often associated with pneumonia, atelectasis, or consolidation. The signs of increased lung opacity are indicative of pathology could be pneumonia, which a common cause of increased lung opacity in x-rays.
36

---

Question: Above is brain MRI scan. Is there present in this brain MRI scan?
Answer with one of the following: No Tumor, Has Tumor
Ground Truth Label: Has Tumor
Model's Prediction: Has Tumor
Reasoning to annotate: To analyze the image, let's use following
reasoning steps:
The image brain MRI sean, which is a medical imaging technique used to visualize detailed cross-sectional images of the brain. MRI is particularly useful for detecting tumors because provides high contrast between different types tissues, including tumors, which can be seen as distinct from normal brain tissuc.
Identify Type Image:
2. Analyze the Image Features:
The tumor appears as a round. well-cireumscribed lesion, meaning has clear boundary from the surrounding brain tissue. In MRI images, tumors are often character- ized as having a distinet, often irregular, boundary, while normal brain tissue has smooth, the tumor is clearly distinct from the surrounding brain tissue, which is strong indicator of lesion that could be a tumor.
Shape and Size:
well-defined edge. In this
Signal Intensity: The signal intensity refers to how bright or dark the tissue appears on the MRI image. Tumors vary in signal intensity, but often, they appear brighter than normal brain tissue on a T2-weighted MRI (which is the standard MRI sequence used for brain imaging). This is because tumors have different tissue properties, such increased blood flow, which can Icad to a higher signal intensity on MRI. This tumor is indeed bright on this image, suggesting a figh-intenssty lesion, which is often associated with tumors, but this alone is not definitive for diagnosis.
Location and Location: The tumor is located in the brain, and in this case, it's in a region where tumors common. The location and size of the lesion important factors in deter- mining the potential pathology, but they are not definitive on their own. The location in this image is consistent with where tumors commonly appear in brain, and the size is indicative of a lesion that could be lumor.
Clinical Context: While this reasoning relies mostly on the image itself, the context (clinical history of the patient, symptoms, and any other imaging studies) would be considered by medical professional to make a definitive diagnosis. This context is often essential for precise understanding and interpretation of the image.
37

---

Question: Above brain CT sean slice. Is there any hemorrhage present in this CT scan slice?
Answer with one of the following: No Hemorrhage, Has Hemor- rhage
Ground Truth Label: No Hemorrhage
Model's Prediction: No Hemorrhage
To analyze the image, we will go through series of steps to understand what we're looking at and what it might tell about the presence of hemorrhage.
Reasoning to annotate:
1. Understanding CT Scans: CT (computed tomography)
X-rays and computer processing to create detailed images of the body's internal structures. They are particularly useful diagnosing brain injuries because they can show changes in tissue density, which can indicate bleeding, tumors, or other pathologies. The color and density of the tissue CT scan can often give clues about what happening.
2. Anatomy of a CT Scan Image: CT scan is cross-sectional view of the body, and cach slice provides a detailed look at brain representing different densities. The brain tissue usually appears as medium gray, while air or gas appears black, and bone, which is very dense, appears white.
image is typically grayscale, different shades
3. Color Coding on CT Scans: The way tissues appear CT scans can vary depending the type of contrast used and the type of scan (c.g., CT with contrast, CT without contrast), a standard CT scan, brain tissue usually appears shades on CT scans, depending on the type and age of the hemorrhage. Typically, blood is denser than brain tissue and appears white non-contrast CT scan, while contrast-enhanced CT scan, appear darker (hypodense) or have different color depending on its age and location.
medium gray. If there is bleeding, the blood can appear in different
4. Symptoms of Hemorrhage CT: On CT scan, signs of hemorrhage include:
Hypodense (darker than normal brain tissue) non-contrast CT: This is the most common appearance for acute hemorrhage. happens when blood clots and does not allow contrast pass through.
Hyperdense (whiter than normal brain tissue) on non-contrast CT: This is seen with older blood, where the blood has already clotted and is more dense than normal brain tissue. Contrast enhancement: If contrast medium is used, it can help highlight the blood clot, making it stand out even more on the scan.
5. Analyzing the Image:
The image grayscalc, and in this case, the brain tissue appears a medium gray shade, which is normal for brain tissue in CT scan.
There is no evidence of white areas (hyperdense), which would suggest older blood clots contrast enhancement that would indicate bleeding. The entire brain tissue appears uniformly gray, without any obvious bright or dark that would suggest hemorrhage.
Based on these considerations, we are looking signs that would indicate hemorrhage, but there are none in this image. The tissue density is consistent with normal brain tissue, and there is no evidence of blood clots other signs of hemorrhage. The image is consistent with a brain CT without hemorrhage.
38

---

Above is dermoscopy image of patient. What is the diagnosis of patient the dermoscopy image?
Question:
Answer with one word from the following: Melanoma, Nevus, Basal Cell Carcinoma, AKIEC, Other
Ground Truth Label: Nevus
Model's Prediction: Nevus
Reasoning to annotate: To analyze the image, let's go through the
reasoning process step by step:
1. Examine the Image:
The image shows closc-up of skin with a lesion. The lesion appears have somewhat uneven, irregular. and slightly raised surface, which can be indicative of various skin conditions. The coloration is mostly dark brown, which could be sign of pigment-ric lesion, such a mole or nevus. The border of the lesion appears somewhat uneven, which is characteristic feature many nevi, but this is definitive without further context.
2. Consider the Clinical Context:
dermatology, dermoscopy often used to evaluate skin lesions signs may suggest malignancy. The image provided does not have any clinical history or patient data, but based on the visual clues, we can consider following points:
Melanoma typically shows a variety features that can indicative of malignancy, such as changing color, irregular border, asymmetry. The lesion in this image does not show any obvious signs malignancy ulceration, bleeding, or bleeding into the lesion, which often associated with melanoma. However, these signs present, and a definitive diagnosis requires biopsy.
always
Nevus (or mole) common benign growth the skin. These are often characterized by their uniformity, color, and border. In this image, the lesion's 's color is consistent, and border is somewhat irregular, but this is not definitive sign for malignancy. lesion could be nevus, but cannot make certain diagnosis without further clinical context and examination.
Basal Cell Carcinoma (BCC) is usually non-melanoma skin cancer. BCCs often erythematous, ulcerated, and can be very vascular. The lesion in this image lacks these characteristics, so is less likely to be BCC. However, is still possibility, but is less likely based on the image alone.
AKIEC (Actinic Keratoses) are precancerous skin lesions that are usually caused by chronic exposure. They often have rough, scaly, warty appearance, and they usually light tan, brown, or red. This lesion does not have the characteristics of AKIEC, and the image does suggest precancerous lesion.
Other:
There are other possibilitics such as vascular lesion, which could be due to various causes as angiomas, which are benign. However, the image does not provide enough information make definitive diagnosis, and further clinical examination and histopathology are required.
39

---

Question: Above clinical image of patient. What is the diagnosis of the patient in the clinical image?
Answer with word from the following: Mclanoma, Nevus, Basal Cell Carcinoma, AKIEC, Other
Ground Truth Label: Basal Cell Carcinoma
Model's Prediction: Basal Cell Carcinoma
Reasoning to annotate: To analyze the image, let's go through the rea-
soning process step by step:
Observation of the Image:
The skin lesion the image is pinkish in color, which can be indicative of various skin conditions. The color suggests it's a dark mole (nevus) or dark, raised, and unevenly shaped mass (melanoma). The color is relatively uniform. but the pink hue is bit unusual normal skin, indicating something might be amiss.
Color and Texture:
Shape and Size: The lesion appears slightly raised, pinkish, and slightly elevated area, with a somewhat indistinct border. The uneven border is critical feature that could suggest a basal cell carcinoma, these often have
irregular border, while melanoma is known for having a jagged, uneven, or indistinct border. Nevus usually have a more sharply defined border and are often flat, while AKIEC (Actinic Keratosis) lesions have rough, scaly surface, which is more consistent with the texture this image. The lesion's 's size and shape are also consistent with basal cell carcinoma, which is more likely to present as raised, pink, or
flesh-colored lesion on skin, especially on sun-exposed areas.
Skin Lesion Location and Context: The lesion is on the skin, which is a common location for basal cell carcinomas, which are the most common type of skin cancer. They often appear such the face, neck, and back, which aligns with the location of the lesion in this image.
areas exposed to the
Clinical Context: Basal cell carcinomas are typically painless and are often associated with chronic exposure. They usually benign, but they be removed prevent potential malignancy. In this context, the lesion's color, shape, and location consistent with basal carcinoma. is important to rememher that only dermatologist can make definitive diagnosis, but this analysis is based common clinical patterns.
Full Training Metrics
We provide full training metrics across individual datasets for RQI in the following tables:
40

---

Table 7: Performance comparison with standard deviations. Results averaged over 4 runs. Ace: Accuracy, F1: FI Score, CXR: Chest X-ray.
CXR
Mammo.
Dermoscopy
CT Scan
Model
Ace
F1
Acc
F1
Acc
F1
Ace
F1
SFT .6882.030
.078#.013
.411.0055
.0562.000
.640#.029
.158:.000
.5254.000
.336t.000
ReMax
.6362.099
.1202.030
.5774.063
.033+.000
.6444.009
.257..018
5674.026
.228+.001
RE++
.730=.027
.0822.032
.6602.087
.0762.011
.635=.008
.2372.079
.529.0133
.2474.039
RLOO
.7522.015
.086*.003
.471..002
.068*.006
.636=.000
.2162.0028
5344.014
.224+.042
GRPO
.703=.012
.095=.007
.4622.000
.059+.000
.6462.003
.2442.016
.524+.002
.236+.001
DRPO Domainthlly
.6932.001
.086=.001
.751=.008
.213+.005
.679=.016
.251+.042
.571+.00
.257+.000
DRPO
.687=.005
.115+.014
.7562.008
.253+.006
.715=.006
.407+.006
.570+.012
.3092.013
Fundus
Ultrasound
MRI
Pathology
Overall
Model
Acc
F1
Acc
F1
Acc
F1
Ace
F1
Ace
F1
SFT
.715s.051
0662:.000
548t.02. 028
.235s.000
.367..03
.17a..000
6524.000
083+.000
6022.014
.139=.001
ReMax
.678.015
0892.006
5471.044
.5471.012
70661007
.176:.001
RE++
6624.003
098+.013
519+.010
136+.012
651
420+.025
668
226+.026
621+:020
.202+0053
RLOO
670=.003
099:.013
519t.012
.144.014
658+.032
.334.020
699+.011
.216..024
.6112002
.189+003
GRPO
.670z.001
0862 002
520+.009
.46t.011
.611:.027
.352.0330
714
28622 010
.609
.133=:003
DRPO
369=.000
083e 000
AS0+.040
098 031
733++012
.475..012
.762+.009
.388t.028
1666+
004
DRPO
.6722.004
.033.0088
.555s.014
223+.021
.789t.019
.625a.028
.708&.010
.2652.016
.6662.009
.2952.013
41

---

Table 8: Performance metrics of QoQ-Med-7B trained with DRPO DomainOnly (DRPO without
intra-domain scaling) Vision Only Data
Dataset
Accuracy Sensitivity
Specificity
F1
Precision
Recall
Individual Datasets
CT/NSPPET
0.6835
0,2000
0.8000
0.0691
0.0417
0,2000
CTHHEMORRHAGE
0,5000
0,5000
0.5000
0.3333
0.2500
0,5000
CT/KITS23
0.6154
0.5000
0.5000
0.3810
0.3077
0,5000
CT/LNDB
0,5000
0.5000
0.5000
0.3333
0.2500
0,5000
CT/RSPECT
0,5556
0.3333
0.6667
0.1667
0.1111
0.3333
CHEST XRAY/VINDR
0.7465
0.0701
0.9313
0.0075
0.0241
0.0701
CHEST XRAY/MIMIC-OM
0.7694
0.0950
0,9402
0.0516
0.0606
0.0950
CHEST XRAY/CHEXPERT FULL
0.7523
0.1116
0,9342
0.0873
0.0722
0.1116
CHEST XRAY/CORONAHACK
0.5556
0,3333
0.6667
0.1667
0.1111
0,3333
CHEST XRAY/COVID19
0.6481
0,2500
0.7500
0.1143
0.0741
0,2500
FUNDUS/JICHII
0.6250
0.2500
0.7500
0.1000
0.0625
0,2500
FUNDUS/APTOS
0.6800
0.2000
0,8000
0.0671
0.0403
0.2000
FUNDUS/MESSIDOR-2
0.7021
0.2000
0,8000
0.0814
0.0511
0,2000
DERN/HAM10000
0.7067
0.2667
0.8167
0.1536
0.1407
0.2667
DERM/ISIC2020
0.5833
0.5833
0.5833
0.4958
0.7727
0.5833
DERM/AD UFES_20
0.7434
0.3167
0.8341
0.2094
0.2417
0.3167
MAMMOICBIS
0.6694
0.2333
0.8096
0.0935
0.0670
0.2333
MANMO//VINDR
0.6800
0.2000
0.8000
0.0686
0.0414
0,2000
MANMOO/CMM
0.8889
0,5000
0.5000
0.4706
0.4444
0.5000
ULTRASOUND/BUSS
0.6667
0,5000
0.7500
0.4455
0.5965
0,5000
ULTRASSUUD/CCOVID BLUES
0,5000
0.0000
1.0000
0.0000
0.0000
0.0000
ULTRRASOUNDICOVID US
0.4667
0.3667
0.6825
0.1558
0.3889
0.3667
MRI/BRAIN TUMOR
0.6354
0,2708
0.7569
0.1396
0.2312
0,2708
MRI/BRAIN TUMOR 2
0.8780
0.8615
0.8615
0.8703
0.8615
PATHOLOGY//CCSS PATHOL0GY//CC5000
0.6250
0,2500
0.7500
0.1000
0.0625
0,2500
0,9200
0.8000
0,9500
0.7977
0.8054
0,8000
Modality Summaries
CT
0.5709
0.4067
0.5933
0.2567
0.1921
0.4067
CHEST XRAY
0.6944
0.1720
0.8445
0.0855
0.0684
0.1720
FUNDUS
0.6690
0.2167
0.7833
0.0828
0.0513
0.2167
DERM
0.6778
0.3889
0.7447
0,2863
0.3850
0,3889
МАMMO
0.7461
0.3111
0,7032
0,2109
0.1843
0,3111
ULTRASOUND
0.5444
0.2889
0.8108
0,2004
0.3285
0.2889
MRI
0.7567
0,5662
0,8092
0,5050
0.5607
0.5662
PATHOLOGY
0.7725
0.5250
0.8500
0.4488
0.4340
0,5250
Overall Results
Overall
0.6790
0.3594
0.7674
0.2595
0.2755
0.3594
42

---

Table Performance metries QoQ-Med-32B trained with DRPO on Vision Datasets
Dataset
Accuracy
Sensitivity
Specificity
F1
Precision
Recall
Individual Datasets
CT/INSPECT
0.6835
0.2000
0.8000
0.0691
0.0417
0.2000
CTHEEMORHHAGE
0,5000
0.5000
0.5000
0,3333
0.2500
0,5000
CT/KITS23
0.6154
0.5000
0.5000
0,3810
0.3077
0,5000
CT/LNDB
0.5000
0.5000
0.5000
0,3333
0.2500
0,5000
CT/RSPECT
0.5556
0.3333
0,6667
0.1667
0,1111
0.3333
CHEST XRAY/VINDR
0.7407
0.1048
0.9168
0.0411
0.1507
0.1048
CHEST XRAY/MIMIC-XX
0.7382
0.1087
0.9028
0.0825
0.1143
0.1087
CHEST XRAY/CHEXPERT FULL
0.7096
0.1680
0.9283
0.0766
0.1213
0.1680
CHEST_XC XRAY/CORONAHACK
0.7130
0.5694
0.7847
0.4745
0.4474
0.5694
CHEST_XRAY XRAY/COVID11
0.7593
0.4375
0.8289
0.3383
0.2866
0.4375
FUNDUS/JINHII
0.6250
0.2500
0.7500
0.1132
0.0732
0.2500
FUNDUS/APTOS
0.7417
0.3500
0.8396
0.2269
0.3389
0.3500
FUNDUS/MESSIDOR-2
0.7064
0.2083
0.8029
0.1031
0.0754
0.2083
DERN/HA.M1000
0.7400
0.3500
0.8375
0.2725
0.4761
0.3500
DERM/ISIC2020
0.5000
0.5000
0.5000
0.3333
0.2500
0.5000
DERM/PAD UFES_
0.7434
0.3167
0.8341
0,2628
0.2397
0.3167
MAMMO/CBIS
0.7020
0,2083
0.8052
0.1
0.2505
0,2083
MAMMO/VINDR
0.6833
0,2000
0.8042
0.0904
0.0686
0,2000
MAMMO/CMMD
0,8889
0,5000
0.5000
0.4706
0.4444
0,5000
ULTRRASOUNDBBUSI
0.6204
0.4306
0.7153
0.3328
0.3681
0,4306
ULTRASSOUNDCCOVID- BLUES
0,5000
0.0000
1.0000
0.0000
0.0000
0.0000
ULTRASOUNDICOVID US
0.7067
0.6364
0.8095
0.4467
0.3803
0.6364
MRI/BRAIN TUMOR
0,8333
0.6667
0,8889
0.6457
0.7094
0.6667
MRI/BRAIN TUMOR
0.9756
0.9792
0.9792
0,9751
0.9722
0,9792
PATHOLOGY/BCSS
0.6510
0.3021
0.7674
0.1967
0.1501
0.3021
PATHOL0GY//LC25000
0.8017
0.4833
0.8813
0.4754
0.5001
0.4833
Modality Summaries
CT
0.5709
0.4067
0,5933
0.2567
0.1921
0.4067
CHEST CHEST_XRAY
0.7321
0,2777
0.8723
0.2026
0.2241
0.2777
FUNDUS
0.6910
0,2694
0.7975
0.1477
0.1625
0.2694
DERM
0.6611 1
0.3889
0.7239
0.2896
0.3219
0.3889
МАMMO
0.7581
0.3028
0.7031
0.2239
0.2545
0.3028
ULTRASOUND
0.6090
0.3556
0.8416
0.2598
0.2495
0.3556
MRI
0.9045
0.8229
0,9340
0.8104
0.8408
0.8229
PATHOLOGY
0.7264
0.3927
0.8243
0.3360
0.3251
0.3927
Overall Results
Overall
0.7066
0.4021
0.7863
0.3158
0.3213
0.4021
43

---

Table 10: Performance metries 7B model trained with GRPO on Vision Only Data
Dataset
Accuracy
Sensitivity
Specificity
F1
Precision
Recall
Individual Datasets
CT/INSPECT
0.6835
0.2000
0.8000
0.0691
0.0417
0.2000
CTHEEMRRHHAGE
0,5000
0.5000
0.5000
0.3333
0.2500
0,5000
CT/KITS23
0.3846
0.5000
0.5000
0.2778
0.1923
0,5000
CT/LNDB
0.5000
0.5000
0.5000
0,3333
0.2500
0,5000
CT/RSPECT
0.5556
0.3333
0,6667
0.1667
0.1111
0.3333
CHEST XRAY/VINDR
0.7432
0.0947
0,9216
0.0241
0.0224
0.0947
CHEST XRAY/MIMIC-XXX
0.7771
0.1042
0.9428
0.0679
0.0805
0.1042
CHEST XRAY/CHEXPERT FULL
0.7583
0.1128
0,9342
0.1061
0.1209
0.1128
CHEST_XC XRAY/CORONAHACK
0.5556
0.3333
0.6667
0.1667
0.1111
0.3333
CHEST_XRAY XRAY/COVID11
0.6481
0.2500
0.7500
0.1176
0.0769
0.2500
FUNDUS/JINHHI
0.6250
0.2500
0.7500
0.1000
0.0625
0.2500
FUNDUS/APTOS
0.6800
0.2000
0.8000
0.0667
0.0400
0.2000
FUNDUS/MESSIDOR-2
0.7021
0.2000
0,8000
0.0814
0.0511
0,2000
DERN/HA.M1000
0.7200
0.2833
0.8292
0.1998
0.2880
0,2833
DERM/ISIC2020
0.4688
0.4375
0,5000
0.3043
0.2333
0.4375
DERM/PAD UFES_
0.7321
0,2917
0.8268
0,2298
0.2043
0,2917
MAMMO/CBIS
0.6082
0,2000
0,8000
0.0080
0.0041
0,2000
MAMMO/VINDR
0.6800
0,2000
0,8000
0.0667
0.0400
0,2000
MAMMO/CMMD
0.1111
0,5000
0.5000
0.1000
0,5000
ULTRRASOUNDBBUSI
0.5556
0,3333
0.6667
0.1667
0.1111
0.3333
ULTRASSUUNDCCOVID- BLUES
0,5000
0.0000
1.0000
0.0000
0.0000
0.0000
ULTRASOUNDICOVID US
0.4400
0.3333
0.6667
0.0920
0.0533
0.3333
MRI/BRAIN TUMOR
0.6354
0.2708
0.7569
0.1437
0.3502
0.2708
MRI/BRAIN TUMOR
0.6585
0.7083
0.7083
0.6483
0.7742
0.7083
PATHOLOGG//BCSS PATHOL0GY//LC25000
0.6771
0.3542
0.7847
0,2464
0.2224
0.3542
0.8467
0.6167
0.9042
0.5609
0.5281
0.6167
Modality Summaries
CT
0.5247
0,4067
0.5933
0.2360
0. 0.1690
0.4067
CHEST CHEST_XRAY XRAY
0.6965
0.1790
0.8431
0.0965
0,0824
0.1790
FUNDUS
0.6690
0.2167
0.7833
0.0827
0,0512
0.2167
DERM
0.6403
0.3375
0.7187
0.2446
0.2419
0.3375
МАMMO
0.4664
0.3000
0.7000
0.0582
0.0332
0.3000
ULTRASOUND
0.4985
0.2222
0.7778
0.0862
0.0548
0.2222
MRI
0.6470
0.4896
0.7326
0.3960
0.5622
0.4896
PATHOLOGY
0.7619
0.4854
0.8444
0.4036
0.3752
0.4854
Overall Results
Overall
0.6130
0.3296
0.7492
0.2005
0.1962
0.3296
44

---

Table 11: Performance metrics of 7B model trained with RLOO on Vision Data
Dataset
Accuracy
Sensitivity
Specificity
F1
Precision
Recall
Individual Datasets
CT/INSPECT
0.6835
0.2000
0.8000
0.0691
0.0417
0.2000
CTHEEMRRHHAGE
0,5000
0.5000
0,5000
0,3333
0.2500
0,5000
CT/KITS23
0,4103
0.5000
0.5417
0,2885
0.2027
0,5000
CT/LNDB
0.4583
0.4583
0,4583
0,4222
0.4444
0.4583
CT/RSPECT
0.5556
0.3333
0,6667
0.1667
0.1111
0.3333
CHEST XRAY/VINDR
0.7514
0.0763
0,9305
0.0256
0.0248
0.0763
CHEST XRAY/MIMIC-XXX
0.7626
0.0942
0.9396
0.0495
0.0586
0.0942
CHEST XRAYCCHEXPERT FULL
0.7583
0.1099
0,9370
0.1032
O. 0216
0.1099
CHEST_XC XRAY/CORONAHACK
0.5556
0.3333
0.6667
0.1667
0.1111
0.3333
CHEST_XRAY XRAY/COVID11
0.6481
0.2500
0.7500
0.1143
0.0741
0.2500
FUNDUS/JINHII
0.6250
0.2500
0.7500
0.1026
0.064S
0.2500
FUNDUS/APTOS
0.6933
0.2333
0.8083
0.1252
0.1702
0.2333
FUNDUS/MESSIDOR-2
0.7021
0.2000
0.8000
0.0821
0.0516
0,2000
DERN/HA.M1000
0.6833
0.2083
0.8021
0.1077
0.1415
0.2083
DERM/ISIC2020
0.5000
0.5000
0.5000
0.3333
0.2500
0.5000
DERM/PAD UFES_
0.7208
0.2667
0.8195
0.2047
0.2509
0.2667
MAMMO/CBIS
0.6082
0,2000
0.7975
0.0083
0.0043
0,2000
MAMMO/VINDR
0.6800
0,2000
0,8000
0.0671
0.0403
0,2000
MAMMO/CMMD
0.1111
0,5000
0.5000
0.1000
0.0556
0.5000
ULTRRASOUNDBBUSI
0.6574
0.4861
0.7431
0.3829
0.3261
0.4861
ULTRASSUUNDCCOVID- BLUES
0,5000
0.0000
1.0000
0.0000
0.0000
0.0000
ULTRASOUNDICOVID US
0,4400
0.3333
0.6667
0.0920
0.0533
0.3333
MRI/BRAIN TUMOR
0.6406
0.2813
0.7604
0.1605
0.4819
0.2813
MRI/BRAIN TUMOR
0.6098
0.6667
0.6667
0,5900
0.7576
0.6667
PATHOLOGG//BCSS PATHOL0GY//LC25000
0.6302
0.2604
0.7535
0.1208
0.3132
0.2604
0.7900
0.4750
0.8688
0.3765
0.4051
0.4750
Modality Summaries
CT
0.5215
0,3983
0,5933
0.2559
0,2100
0.3983
CHEST CHEST_XRAY
0.6952
0.1728
0.8448
0.0918
0,0795
0.1728
FUNDUS
0.6735
0.2278
0.7861
0.1033
0,0954
0.2278
DERM
0.6347
0.3250
0.7072
0.2152
0.2141
0.3250
МАMMO
0.4664
0.3000
0.6992
0.0585
0.0334
0.3000
ULTRASOUND
0.5325
0.2731
0.8032
0.1583
0.1265
0.2731
MRI
0.6252
0.4740
0.7135
0.3752
0.6197
0.4740
PATHOLOGY
0.7101
0.3677
0,8111
0.2487
0.3591
0.3677
Overall Results
Overall
0.6074
0.3173
0.7448
0.1884
0.2172
0.3173
45

---

Table 12: Performance metrics 7B model trained with Reinforce++ on Vision Data
Dataset
Accuracy
Sensitivity
Specificity
F1
Precision
Recall
Individual Datasets
CT/INSPECT
0.6835
0.2000
0.8000
0.0691
0.0417
0.2000
CTHEMMRRHHAGE
0,5000
0,5000
0.5000
0.3333
0.2500
0,5000
CT/KITS23
0.3846
0.3667
0.5833
0.2391
0.1774
0.3667
CT/LNDB
0.5000
0.5000
0.5000
0,3333
0.2500
0,5000
CT/RSPECT
0.5556
0.3333
0,6667
0.1667
0,1111
0.3333
CHEST XRAY/VINDR
0.7362
0.0811
0,9201
0.0087
0.0449
0.0811
CHEST XRAY/MIMIC-XX
0.7471
0.0930
0.9355
0.0414
0.1817
0.0930
CHEST XRAY/CHEXPERT FULL
0.7482
0.1269
0,9362
0.0922
0.2212
0.1269
CHEST_XC XRAY/CORONAHACK
0.5556
0.3333
0.6667
0.1667
0.1111
0.3333
CHEST_XRAY XRAY/COVID11
0.6481
0.2500
0.7500
0.1143
0.0741
0.2500
FUNDUS/JINHHI
0.6250
0.2500
0.7500
0.1000
0.0625
0.2500
FUNDUS/APTOS
0.6833
0.2083
0.8021
0.0829
0.0914
0.2083
FUNDUS/MESSIDOR-2
0.7021
0.2000
0.8000
0.0814
0.0511
0,2000
DERN/HA.M1000
0.6933
0.2333
0.8083
0.1515
0.4133
0,2333
DERM/ISIC2020
0.5313
0.5208
0.5417
0.3829
0.7609
0.5208
DERM/PAD UFES_
0.6981
0,2167
0.8052
0.1845
0.2750
0,2167
MAMMO/CBIS
0.6082
0,2000
0.7957
0.0086
0.0044
0,2000
MAMMO/VINDR
0.6800
0,2000
0,8000
0.0671
0,2000
MAMMO/CMMD
0.1111
0,5000
0.5000
0.1000
0,5000
ULTRRASOUNDBBUSI
0.5648
0.3333
0.6806
0.1702
0.1143
0.3333
ULTRASSUUNDCCOVID- BLUES
0,5000
0.0000
1.0000
0.0000
0.0000
0.0000
ULTRASOUNDICOVID US
0,4400
0.3333
0.6667
0.0920
0.0533
0.3333
MRI/BRAIN TUMOR
0.6250
0,2500
0.7500
0.1008
0.0632
0,2500
MRI/BRAIN TUMOR
0.4146
0.5000
0.5000
0,2931
0.2073
0,5000
PATHOLOGY/BCOE
0.6667
0.3333
0.7778
0,2424
0.4238
0.3333
PATHOL0GY//LC25000
0.7367
0.3417
0.8354
0.2108
0,2386
0.3417
Modality Summaries
CT
0.5247
0,3800
0.6100
0.2283
0.1661
0,3800
CHEST CHEST_XRAY XRAY
0.6871
0.1769
0.8417
0.0846
0,1266
0.1769
FUNDUS
0.6702
0.2194
0.7840
0.0881
0,0683
0.2194
DERM
0.6409
0.3236
0.7184
0.2396
0.4831
0.3236
МАMMO
0.4664
0.3000
0.6986
0.0586
0.0334
0.3000
ULTRASOUND
0.5016
0.2222
0.7824
0.0874
0.0559
0.2222
MRI
0.5198
0.3750
0.6250
0.1970
0.1352
0.3750
PATHOLOGY
0.7017
0.3375
0.8066
0.2266
0.3312
0.3375
Overall Results
Overall
0.5890
0.2918
0.7333
0.1513
0.1750
0.2918
46

---

Table 13: Performance metrics of 7B model trained with ReMax on Vision Dafa
Dataset
Accuracy
Sensitivity
Specificity
F1
Precision
Recall
Individual Datasets
CT/INSPECT
0.6835
0.2000
0.8000
0.0691
0.0417
0.2000
CTHEMMRRHHAGE
0,5000
0.5000
0,5000
0,3333
0.2500
0,5000
CT/KITS23
0.3846
0.4667
0.5208
0.2692
0.1892
0.4667
CT/LNDB
0.5000
0.5000
0.5000
0,3333
0.2500
0,5000
CT/RSPECT
0.5556
0.3333
0,6667
0.1667
0.1111
0.3333
CHEST XRAY/VINDR
0.7396
0.0694
0.9244
0.0057
0.0030
0.0694
CHEST XRAY/MIMIC-XXX
0.7548
0.0923
0.9376
0.0450
0.0442
0.0923
CHEST XRAY/CHEXPERT FULL
0.7456
0.1218
0,9376
0.0832
0.0665
0.1218
CHEST_XC XRAY/CORONAHACK
0.5556
0.3333
0.6667
0.1667
0.1111
0.3333
CHEST_XRAY XRAY/COVID19
0.6481
0.2500
0.7500
0.1143
0.0741
0.2500
FUNDUS/JINHHI
0.6250
0.2500
0.7500
0.1008
0.0632
0.2500
FUNDUS/APTOS
0.6800
0.2000
0,8000
0.0671
0.0403
0.2000
FUNDUS/MESSIDOR-2
0.7021
0.2000
0.8000
0.0814
0.0511
0.2000
DERN/HA.M1000
0.6833
0.2083
0.8021
0.1079
0.1340
0,2083
DERM/ISIC2020
0.5625
0.5625
0.5625
0.4589
0.7667
0.5625
DERM/PAD UFES_
0.7094
0,2417
0.8129
0.1550
0.3262
0,2417
MAMMO/CBIS
0.6122
0,2083
0,8008
0.0242
0.2042
0,2083
MAMMO/VINDR
0.6800
0,2000
0,8000
0.0671
0.0403
0,2000
MAMMO/CMMD
0.1111
0,5000
0.5000
0.1000
0.0556
0.5000
ULTRRASOUNDBBUSI
0.6481
0.4722
0.7361
0.3810
0.3300
0,4722
ULTRASSUUNDCCOVID- BLUES
0,5000
0.0000
1.0000
0.0000
0.0000
0.0000
ULTRASOUNDICOVID US
0.4667
0.3636
0.6825
0.1508
0.3889
0.3636
MRI/BRAIN TUMOR
0.6510
0,3021
0.7674
0.1929
0.4841
0.3021
MRI/BRAIN TUMOR
0.4146
0.5000
0.5000
0,2931
0.2073
0,5000
PATHOLOGY/BCGE
0.6771
0.3542
0.7847
0,2575
0.4475
0.3542
PATHOL0GY//LC25000
0.7533
0,3833
0.8458
0.2903
0,3582
0.3833
Modality Summaries
CT
0.5247
0,4000
0,5975
0.2343
0.1684
0.4000
CHEST CHEST_XRAY XRAY
0.6887
0.1734
0.8433
0.0830
0,0598
0.1734
FUNDUS
0.6690
0.2167
0.7833
0.0831
0,0515
0.2167
DERM
0.6518
0.3375
0.7258
0.2406
0..0889
0.3375
МАMMO
0.4678
0.3028
0.7003
0.0638
0.1000)
0.3028
ULTRASOUND
0.5383
0.2786
0.8062
0.1772
0.2396
0.2786
MRI
0.5328
0.4010
0.6337
0.2430
0.3457
0.4010
PATHOLOGY
0.7152
0.3688
0.8153
0.2739
0.4028
0.3688
Overall Results
Overall
0.5985
0.3098
0.7382
0.1749
0.2221
0.3098
47

---

Table 14: Performance metrics of 7B model trained with PPO on Vision Data
Dataset
Accuracy
Sensitivity
Specificity
F1
Precision
Recall
Individual Datasets
CT/INSPECT
0.6835
0.2000
0.8000
0.0691
0.0417
0.2000
CTHEEMRRHHAGE
0,5000
0.5000
0.5000
0,3333
0.2500
0,5000
CT/KITS23
0.6154
0.5000
0.5000
0,3810
0.3077
0,5000
CT/LNDB
0.5000
0.5000
0.5000
0,3333
0.2500
0,5000
CT/RSPECT
0.5556
0.3333
0,6667
0.1667
0.1111
0.3333
CHEST XRAY/VINDR
0.7327
0.0833
0.9166
0.0063
0.0033
0.0833
CHEST XRAY/MIMIC-XX
0.7242
0.0751
0.9298
0.0117
0.0752
0.0751
CHEST XRAY/CHEXPERT FULL
0.6908
0.0784
0,9233
0.0193
0.0476
0.0784
CHEST_XC XRAY/CORONAHACK
0.5556
0.3333
0.6667
0.1667
0.1111
0.3333
CHEST_XRAY XRAY/COVID11
0.6481
0.2500
0.7500
0.1143
0.0741
0.2500
FUNDUS/JINHII
0.6250
0.2500
0.7500
0.1000
0.0625
0.2500
FUNDUS/APTOS
0.6800
0.2000
0.8000
0.0671
0.0403
0.2000
FUNDUS/MESSIDOR-2
0.7021
0.2000
0,8000
0.0814
0.0511
0,2000
DERN/HA.M1000
0.7217
0,3000
0.8271
0.2075
0.3442
0.3000
DERM/ISIC2020
0.5313
0.5000
0.5625
0.3782
0.7614
0,5000
DERM/PAD UFES_
0.7509
0,3333
0.8390
0,2482
0.2643
0,3333
MAMMO/CBIS
0.6449
0,2750
0.8051
0.0770
0,2750
MAMMO/VINDR
0.6800
0,2000
0.8000
0.0667
0.0400
0,2000
MAMMO/CMMD
0,8889
0,5000
0.5000
0.4706
0.4444
0,5000
ULTRRASOUNDBBUSI
0.5556
0.3333
0.6667
0.1667
0.1111
0.3333
ULTRASSUUNDCCOVID- BLUES
0,5000
0.0000
1.0000
0.0000
0.0000
0.0000
ULTRASOUNDICOVID US
0.4133
0,2500
0.6381
0.0741
0.0435
0,2500
MRI/BRAIN TUMOR
0.6563
0,3125
0.7708
0,2030
0.2817
0.3125
MRI/BRAIN TUMOR
0,8780
0.8958
0.8958
0.8778
0.8864
0.8958
PATHOLOGG//BCSS PATHOL0GY//LC25000
0.6276
0.2500
0.7535
0.1008
0.0632
0.2500
0.8633
0.6583
0.9146
0.6280
0.7043
0.6583
Modality Summaries
CT
0.5709
0,4067
0,5933
0.2567
0. 0911
0.4067
CHEST CHEST_XRAY XRAY
0.6703
0.1640
0.8373
0.0637
0,0622
0.1640
FUNDUS
0.6690
0.2167
0.7833
0.0828
0,0513
0.2167
DERM
0.6680
0.3778
0.7429
0.2780
01.4566
0.3778
МАMMO
0.7379
0.3250
0.7017
0.2048
0.1828
0.3250
ULTRASOUND
0.4896
0.1944
0.7683
0.0802
0.0515
0.1944
MRI
0.7671
0,6042
0.8333
0,5404
0.5840
0.6042
PATHOLOGY
0.7455
0.4542
0.8340
0.3644
0.3837
0.4542
Overall Results
Overall
0.6648
0.3429
0.7618
0.2339
0.2456
0.3429
48


Reviews:
[{"review_id": "lo1ENeWys7", "reviewer": "Reviewer_MhFk", "summary": "This paper introduces a QoQ-Med model, a large Multimodal Clinical Foundation Model with Domain-Aware GRPO Training. The model takes medical images, time-series signals, and text reports as inputs. It also introduces a novel reinforcement learning method, Domain-aware Relative Policy Optimization (DRPO), which adjusts rewards based on domain rarity and modality difficulty to address data imbalance. The model is trained on 2.61 million instruction-tuning samples from 9 clinical domains. Experimental results show strong performance.", "questions": "1. The logic in clinical diagnosis varies significantly in different fields. Please discuss more on the applicability and adaptability of QoQ-style training across domains with differing reasoning structures.\n2. Although the model supports images, text, and time-series, there is little analysis on how easily the framework scales to additional modalities (e.g., pathology slides, genomic data, 3D scans). Please discuss how to extend the model on further modality data/inputs.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The multimodal inputs setup closely mirrors real clinical diagnostic reasoning, going beyond standard VQA\n2. The model is trained on high-quality data, including 2.61 million question-answer pairs with intermediate reasoning\n3. The method employs the SOTA GRPO in the model training\n4. Experimental results look very promising.\n\nWeaknesses:\n1. The logic in clinical diagnosis varies significantly in different fields. Please discuss more on the applicability and adaptability of QoQ-style training across domains with differing reasoning structures.\n2. The study lacks expert human evaluation or annotation, such as rating answer helpfulness or diagnostic correctness.\n3. Although the model supports images, text, and time-series, there is little analysis on how easily the framework scales to additional modalities (e.g., pathology slides, genomic data, 3D scans).", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer MhFk for the constructive feedback and suggestions. \n\n\nW1/Q1: The reasoning training from QoQ style training is highly generalizable, as we do not presume any reasoning structure during the training process. The supervision is only on the final diagnosis. This follows the finding from the [Deepseek R1](https://arxiv.org/abs/2501.12948) paper, where the authors demonstrated how simple, flexible supervision elicits models’ pretraining knowledge, helps prevent reward hacking and enables more holistic learning. In Sec. 4.3 and App. E, we detailed how the model reasons across different modalities. In particular, we found that the model is able to focus more on the color for the reasoning of dermoscopy, with examples like “slightly raised, pinkish, and slightly elevated area.” On the other hand, the model focuses more on the brightness, patterns and textures for ultrasound videos, with examples like “The hyperechoic nature of the lesion is often associated with consolidation.” \n\n\nW2: As described in Sec. 4.3, we collaborated with expert clinicians to annotate the relevance of the reasoning traces to the diagnosis. The expert annotation in App. E indicates that around 93.8% of the reasoning traces are relevant to the diagnosis of the disease. In particular, around 45.3% of the reasoning is rated as highly relevant, and 100% of the rated samples contain highly relevant reasoning traces. \n\n| Model | CXR (Acc/F1) | Mammo. (Acc/F1) | Dermoscopy (Acc/F1) | CT Scan (Acc/F1) | Fundus (Acc/F1) | Ultrasound (Acc/F1) | MRI (Acc/F1) | Pathology (Acc/F1) | Overall (Acc/F1) |\n|-------|--------------|-----------------|---------------------|------------------|-----------------|---------------------|--------------|--------------------|--------------------|\n| SFT | .679/.077 | .726/.182 | .624/.158 | .571/.257 | .669/.083 | .561/.235 | .520/.197 | .653/.083 | .625/.159 |\n| PPO | .670/.064 | .738/.205 | .668/.278 | .571/.257 | .669/.083 | .490/.080 | .767/.540 | .745/.364 | .665/.234 |\n| ReMax | .689/.083 | .468/.064 | .652/.241 | .525/.234 | .669/.083 | .538/.177 | .533/.243 | .715/.274 | .599/.175 |\n| RE++ | .687/.085 | .466/.059 | .641/.240 | .525/.228 | .670/.088 | .502/.087 | .520/.197 | .702/.227 | .589/.151 |\n| RLOO | .695/.092 | .466/.058 | .635/.215 | .522/.256 | .673/.103 | .532/.158 | .625/.375 | .710/.249 | .607/.188 |\n| GRPO | .696/.096 | .466/.058 | .640/.245 | .525/.236 | .669/.083 | .499/.086 | .647/.396 | .762/.404 | .613/.200 |\n| DRPO_DomainOnly | .694/.085 | .746/.211 | .678/.286 | .571/.257 | .669/.083 | .544/.200 | .757/.505 | .773/.449 | .679/.259 |\n| DRPO_NoKL | .685/.103 | .711/.264 | .691/.382 | .597/.365 | .676/.085 | .554/.228 | .722/.535 | .710/.300 | .668/.283 |\n| DRPO | .691/.125 | .759/.253 | .707/.400 | .580/.321 | .670/.088 | .568/.240 | .806/.652 | .707/.303 | .686/.286 |\n\nW3/Q2: We would like to clarify that our model QoQ-Med does support pathology slides, as well as 3D scans (CT, MRI) and videos (ultrasound). The performance is demonstrated in Table 2 (attached above for reference). For pathology slides, DRPO achieves an F1 score of 0.449, representing an 11.1% improvement over the next best baseline GRPO (0.404). In 3D imaging modalities, our method shows strong results on MRI with DRPO achieving 0.652 F1 score, a substantial 20.7% improvement over the next best baseline PPO's 0.540. For CT scans, DRPO without KL divergence demonstrates the best performance with a 0.365 F1 score, improving by 42.0% over SFT and PPO's 0.257."}, {"review_id": "4xkgYNEITq", "reviewer": "Reviewer_pph3", "summary": "This paper presents QoQ-Med, a multimodal clinical foundation model that processes medical images (2D/3D), time-series signals (ECG), and text. The model is trained with Domain-aware Relative Policy Optimization (DRPO), a new reinforcement learning objective that extends GRPO. DRPO boosts diagnostic performance and hierarchically scales rewards based on domain rarity and example difficulty to balance training.", "questions": "1. Availability of the Reasoning Traces Dataset: The commitment to release the 2.61 million reasoning traces is a significant and commendable contribution that would greatly benefit the research community. However, upon reviewing the provided anonymous link, I was unable to locate this specific dataset. Could the authors please clarify the status of this resource or provide a direct pointer to it?\n2. Analysis of Reasoning and Failure Modes: Figure 4 provides compelling qualitative examples of QoQ-Med's successful reasoning process, which is a key strength. To offer a more comprehensive and balanced assessment of the model's true capabilities and limitations, it would be highly beneficial to include an analysis of its common failure modes. For instance:\n  - Are there recurring patterns when the generated reasoning is clinically incorrect or irrelevant?\n  - Could the authors provide examples where the reasoning seems plausible but is factually disconnected from the input image?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. Multimodality: The proposed model, QoQ-Med, is significant as it is one of the first open-source foundation models to jointly process 1D time-series data (ECG) alongside 2D/3D medical images and text.\n2. Comprehensive evaluation: The experimental evaluation is thorough and convincing. The results show that DRPO achieves substantial performance improvements (e.g., a 43% increase in macro-F1) over established methods. The ablation studies effectively isolate and validate the contributions of DRPO's key components.\n3. Writing: The paper is well-written and clearly structured. The methodology is explained logically, and Figure 1 provides an intuitive overview of both the model architecture, greatly enhancing the reader's understanding.\n\nWeaknesses\n1. Significance analysis: The primary weakness is a lack of statistical significance analysis for the experimental results. Given the high variance often seen in RL training, reporting error bars or confidence intervals from multiple runs would be crucial to substantiate the robustness of the claimed performance gains.\n2. Hyperparameters: The design of the combined reward function involves several weighting hyperparameters (e.g., $\\lambda_{acc}, \\lambda_{IoU}, \\lambda_{aux}$). The paper does not provide a sensitivity analysis or justification for the specific values chosen, leaving it unclear how performance might be affected by different choices.", "was_revised": true, "final_justification": "My recommendation is to Accept.\n\nI thank the authors for their detailed rebuttal. The responses have successfully addressed the main concerns and questions I raised in my initial review.\n\nResolved issues: the weight of each reward\n\nUnresolved issues: none", "rebuttal": "We thank Reviewer pph3 for finding our multimodality task important, evaluation comprehensive and writing well-structured. \n\n\nWeakness 1: We further ran the main experiments 4 times under each setting for statistical analysis, and included the updated results below:\n\n\n| Model | CXR |  | Mammo. |  | Dermoscopy |  | CT Scan |  |\n|---|---:|---:|---:|---:|---:|---:|---:|---:|\n| | **Acc** | **F1** | **Acc** | **F1** | **Acc** | **F1** | **Acc** | **F1** |\n| **SFT** | .688±.030 | .078±.013 | .481±.015 | .056±.004 | .640±.029 | .158±.000 | .525±.000 | .236±.000 |\n| **ReMax** | .636±.049 | .120±.030 | .577±.063 | .033±.020 | .644±.009 | .257±.018 | .567±.026 | .228±.001 |\n| **RE++** | .730±.027 | .082±.032 | .660±.087 | .076±.011 | .635±.008 | .237±.079 | .529±.013 | .247±.039 |\n| **RLOO** | **.752±.015** | .086±.003 | .471±.002 | .068±.006 | .636±.008 | .216±.028 | .534±.014 | .224±.042 |\n| **GRPO** | .703±.012 | .095±.007 | .466±.000 | .059±.000 | .646±.003 | .244±.016 | .524±.002 | .236±.001 |\n| **DRPO**_DomainOnly_ | .693±.001 | .086±.001 | .751±.008 | .213±.005 | .679±.016 | .251±.042 | .571±.000 | .257±.000 |\n| **DRPO** | .687±.005 | **.115±.014** | **.756±.008** | **.253±.006** | **.715±.006** | **.407±.006** | **.570±.012** | **.309±.013** |\n\n| Model | Fundus |  | Ultrasound |  | MRI |  | Pathology |  | Overall |  |\n|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|\n| | **Acc** | **F1** | **Acc** | **F1** | **Acc** | **F1** | **Acc** | **F1** | **Acc** | **F1** |\n| **SFT** | **.715±.051** | .066±.010 | .548±.028 | **.235±.000** | .567±.038 | .197±.000 | .652±.000 | .083±.000 | .602±.014 | .139±.001 |\n| **ReMax** | .678±.015 | .089±.006 | .547±.044 | .147±.008 | .547±.012 | .264±.018 | .706±.007 | .270±.011 | .596±.002 | .176±.001 |\n| **RE++** | .672±.003 | .098±.013 | .519±.010 | .136±.012 | .651±.022 | .420±.025 | .668±.024 | .254±.026 | .621±.020 | .202±.053 |\n| **RLOO** | .670±.003 | **.099±.013** | .519±.012 | .144±.014 | .658±.032 | .432±.020 | .699±.011 | .216±.024 | .611±.002 | .189±.003 |\n| **GRPO** | .670±.001 | .086±.002 | .520±.009 | .146±.011 | .631±.027 | .395±.030 | .715±.004 | .286±.010 | .609±.005 | .193±.003 |\n| **DRPO**_DomainOnly_ | .669±.000 | .083±.000 | .480±.040 | .098±.031 | .733±.012 | .475±.012 | **.762±.009** | **.388±.028** | .668±.004 | .237±.004 |\n| **DRPO** | .672±.004 | .093±.008 | **.555±.014** | .223±.021 | **.789±.019** | **.625±.028** | .708±.010 | .265±.016 | **.666±.009** | **.295±.013** |\n\n\nThe standard deviation for all methods is around ~1% across runs. In terms of overall performance, our method (DRPO) significantly outperformed all baseline methods (SFT, ReMax, RE++, RLOO and GRPO) with p<0.0001. \n\nWeakness 2: In general, we found that the weight of each reward does not have a significant impact on the final performance. In particular, the auxiliary rewards on formatting saturate shortly in the early stages of training. They have effectively no impact on the later stages due to normalization. We tested different combinations of accuracy rewards: semantic alignment rewards. The results are as follows:\n\n| **Acc:IoU** | **CXR** |       | **Mammo.** |       | **Dermoscopy** |       | **CT Scan** |       |\n|-------------|---------|-------|------------|-------|----------------|-------|-------------|-------|\n|             | **Acc** | **F1** | **Acc**    | **F1** | **Acc**        | **F1** | **Acc**     | **F1** |\n| **0.6:0.2** | **.691**    | .125 | **.759**   | **.253**  | **.707**       | **.400** | **.580**        | **.321**  |\n| **0.2:0.6** | .690    | **.147**  | .563       | .185 | .668           | .290  | .576    | .308 |\n\n| **Acc:IoU** | **Fundus** |       | **Ultrasound** |       | **MRI** |       | **Pathology** |       | **Overall** |       |\n|-------------|------------|-------|----------------|-------|---------|-------|---------------|-------|-------------|-------|\n|             | **Acc**    | **F1** | **Acc**        | **F1** | **Acc** | **F1** | **Acc**       | **F1** | **Acc**     | **F1** |\n| **0.6:0.2** | .670       | .088  | .568       | **.240** | **.806** | **.652** | **.707**          | **.303**  | **.686**    | **.286** |\n| **0.2:0.6** | **.681**   | **.136** | **.573**           | .218  | .768    | .561  | .698      | .233 | .652        | .260  |\n\nDecreasing the weight of the accuracy reward gives a drop in overall performance and performance in most domains, but results are still significantly better than all baselines, which demonstrates the robustness of DRPO.  \n\nQ1: We included a sample reasoning trace in the anonymous repo under datasets/. Due to the size limit of the anonymous repo and the restriction on external links, we are not able to share the entire reasoning traces now, but we will include them in the final manuscript. \n\nQ2/3: We included examples of reasoning traces that lead to incorrect answers in App. E and Sec. 4.3. Generally, there are 2 areas where the current model seems to struggle. First, the model seems to struggle to spot minor objects like support devices. In addition, the model was not particularly good at diagnosing novel diseases like COVID from ultrasound, where the most common misdiagnosis is pneumonia. From our randomly selected samples, there is no instance where the reasoning evidence is not aligned with the image. We will conduct a more thorough error analysis in the appendix of the final manuscript."}, {"review_id": "ZtdJTjHKci", "reviewer": "Reviewer_L5Uq", "summary": "This paper introduces QoQ-Med, a novel multimodal foundation model for clinical applications, uniquely capable of reasoning over diverse data types including medical images (2D/3D), time-series signals (ECG), and text. The core technical contribution is a new reinforcement learning algorithm, Domain-aware Group Relative Policy Optimization (DRPO). DRPO is designed to address the critical challenge of data heterogeneity and imbalance in clinical datasets by introducing a hierarchical reward scaling mechanism. The authors demonstrate that DRPO significantly improves diagnostic performance over standard RL methods, particularly in underrepresented domains. The resulting model, which generates both a final diagnosis and interpretable outputs like reasoning traces and salient region highlighting, is shown to be competitive with or superior to existing open-source and even some closed-source models.", "questions": "N/A", "limitations": "A minor limitation is that while the model handles multiple modalities, the fusion mechanism is a relatively simple concatenation of projected tokens, a more sophisticated fusion mechanism might yield further improvements, though this is more of a future work direction than a flaw.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n---\n\n- The paper tackles a significant and challenging problem, which is creating a generalist clinical foundation model that handles truly heterogeneous data. The integration of 1D time-series data (ECG) with 2D/3D imaging and text is a notable step forward from vision-centric medical MLLMs and addresses a real clinical need for holistic patient data analysis.\n\n- The proposed DRPO algorithm is a novel and well-motivated contribution. It provides a principled way to handle domain imbalance within a critic-free RL framework, which is computationally efficient. The hierarchical scaling based on both domain rarity and intra-domain difficulty is an intelligent design that directly addresses a known problem in training on real-world. The empirical results, showing up to a 43% macro-F1 score improvement, strongly support its effectiveness.\n\n- The experimental evaluation is comprehensive and well-designed. The authors compare their method against a wide range of strong baselines (SFT, PPO ...). They evaluate on a large, multi-domain dataset and include specific experiments to validate multimodal fusion and interpretability. The ablation studies clearly demonstrate the contribution of each component of DRPO.\n\n- Strong comparison against other publicly available systems in the Table 4.\n\n- The authors plan to release the model weights (7B and 32B), the modular training pipeline and the extensive reasoning traces is a major strength. This significantly enhances reproducibility and will provide a valuable resource for the research community, lowering the barrier to entry for research.\n\nWeaknesses:\n---\n\nN/A", "was_revised": true, "final_justification": "Thank you for the rich dialogue, In overall, I am in favour for the acceptance of the paper.", "rebuttal": "We thank reviewer L5Uq for finding our DRPO method interesting, our task important, and our experiments comprehensive. We are excited to see what researchers will build upon our released multimodal disease diagnosis models. We agree that a more sophisticated fusion mechanism may be helpful in aligning multiple modalities. There have been some recent efforts to tackle this challenge. [FUSION](https://arxiv.org/html/2504.09925v1), for example, utilizes similarity loss for the alignment between vision and text embeddings. And [EMMA](https://openreview.net/forum?id=lbrO3bGpeO) proposed a lightweight module for cross-modality alignment. Most of these efforts are centered around texts and images. In our cases, fusion of time series, vision and texts is a non-trivial challenge. While it is out of the scope of this work, we will explore this in our future work, and we welcome community efforts to build better multimodal representations."}, {"review_id": "MxAlgPQdOa", "reviewer": "Reviewer_mAXD", "summary": "This paper presents QoQ-Med-7B/32B, an open-source multimodal clinical foundation model that jointly processes medical images (2D/3D), time-series signals (ECG), and text reports. The key contribution is Domain-aware Relative Policy Optimization (DRPO), a hierarchical reinforcement learning method that scales rewards based on domain rarity and task difficulty to address data imbalance in heterogeneous clinical datasets. The model is trained on 2.61M instruction-tuning pairs across 9 clinical domains and demonstrates significant improvements over existing methods.", "questions": "1. Can you provide confidence intervals through bootstrap sampling of your existing results? What is the variance across different random seeds for at least a subset of experiments?\n2. What is the isolated contribution of cluster-level scaling? How sensitive are results to the number of clusters and reward weights?\n3. How does DRPO perform on the rarest conditions in your datasets? Does the hierarchical weighting risk overfitting to noise in extremely small domains?", "limitations": "The authors briefly acknowledge the limited sample efficiency of unsupervised reasoning learning in their conclusion, but a more comprehensive discussion of limitations is warranted. The most critical limitation is the lack of statistical validation - all experiments are single-run due to computational constraints, preventing any assessment of variance or statistical significance of the reported improvements. This is particularly concerning given that some performance gains are modest and could potentially fall within noise margins. The paper also lacks analysis of potential biases that DRPO's hierarchical weighting might introduce or amplify. By up-weighting underrepresented domains, the method could inadvertently overfit to spurious correlations in small datasets or amplify existing demographic biases present in those domains. The computational requirements (8xA100 GPUs for multiple days of training) severely limit accessibility and reproducibility, yet the environmental impact is not discussed. Additionally, while the model generates reasoning traces, there's no systematic evaluation of their clinical accuracy beyond selected examples, and no prospective clinical validation is provided. The reliance on pseudo-labels from BiomedParse for segmentation evaluation introduces another unvalidated source of error. Finally, the heavy dependence on appendices for critical implementation details means that key aspects of the method cannot be fully evaluated within the peer-review process, potentially hiding important limitations or implementation choices that could affect reproducibility and generalization.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1. DRPO is a well-motivated extension of GRPO with clear theoretical foundation. The hierarchical scaling mechanism (domain-level and cluster-level) effectively addresses the critical problem of performance imbalance in multi-domain clinical learning. The mathematical formulation is rigorous and the algorithm design is sound.\n2. First open-source model to successfully integrate 1D time-series (ECG) with 2D/3D medical images, addressing a significant gap in existing medical MLLMs. The architecture elegantly handles missing modalities.\n3. The model generates reasoning traces and bounding boxes for localization, crucial for clinical trust. Qualitative examples with clinician annotations demonstrate practical utility, though more systematic evaluation is needed.\n\nWeaknesses: \n1. The paper's most significant limitation is the lack of statistical significance testing. All experiments are single-run due to computational constraints (acknowledged in checklist Q7), meaning no error bars or confidence intervals are provided. This fundamentally limits our confidence in the reported improvements, especially given that some gains are relatively modest. The paper also lacks systematic error analysis beyond a few qualitative examples. While Figure 4 shows some failure cases, there's no aggregate analysis of error types, failure modes, or when DRPO provides the most benefit versus baseline methods. The evaluation would be significantly strengthened by at least bootstrap-based confidence intervals on existing results and a more thorough characterization of model failures.\n2. Despite working with clinical data where fairness is critical, the paper provides no analysis of demographic or institutional bias. It's unclear whether DRPO's domain weighting helps reduce disparities or potentially amplifies existing biases by over-emphasizing certain underrepresented domains that may correlate with demographic factors. The paper also doesn't evaluate performance on truly rare conditions beyond the main imaging modalities, leaving questions about robustness in extreme low-resource settings. Without subgroup performance analysis or evaluation on out-of-distribution data, the clinical safety and equity implications remain unclear.\n3. The reasoning quality evaluation, while featuring clinician annotations, lacks methodological rigor. The paper doesn't specify how many clinicians participated, what their qualifications were, or what the inter-annotator agreement was. The evaluation appears to be purely qualitative on selected examples rather than systematic across the validation set. Furthermore, the use of BiomedParse model to generate pseudo-segmentation labels when ground truth is unavailable introduces another source of potential error that isn't validated. The heavy reliance on appendices for critical implementation details also makes it difficult to fully assess the methodology, as appendix content isn't peer-reviewed.", "was_revised": true, "final_justification": "Thank you for your rebuttal. I have ajusted my rating accordingly.", "rebuttal": "We thank Reviewer mAXD for finding our method interesting, the mathematical formulation rigorous and the multimodal architecture novel. \n\nW1/Q1: We further ran the main experiments 4 times under each setting for statistical analysis, and included the updated results below:\n\n| Model | CXR |  | Mammo. |  | Dermoscopy |  | CT Scan |  |\n|---|---:|---:|---:|---:|---:|---:|---:|---:|\n| | **Acc** | **F1** | **Acc** | **F1** | **Acc** | **F1** | **Acc** | **F1** |\n| **SFT** | .688±.030 | .078±.013 | .481±.015 | .056±.004 | .640±.029 | .158±.000 | .525±.000 | .236±.000 |\n| **ReMax** | .636±.049 | .120±.030 | .577±.063 | .033±.020 | .644±.009 | .257±.018 | .567±.026 | .228±.001 |\n| **RE++** | .730±.027 | .082±.032 | .660±.087 | .076±.011 | .635±.008 | .237±.079 | .529±.013 | .247±.039 |\n| **RLOO** | **.752±.015** | .086±.003 | .471±.002 | .068±.006 | .636±.008 | .216±.028 | .534±.014 | .224±.042 |\n| **GRPO** | .703±.012 | .095±.007 | .466±.000 | .059±.000 | .646±.003 | .244±.016 | .524±.002 | .236±.001 |\n| **DRPO**_DomainOnly_ | .693±.001 | .086±.001 | .751±.008 | .213±.005 | .679±.016 | .251±.042 | .571±.000 | .257±.000 |\n| **DRPO** | .687±.005 | **.115±.014** | **.756±.008** | **.253±.006** | **.715±.006** | **.407±.006** | **.570±.012** | **.309±.013** |\n\n| Model | Fundus |  | Ultrasound |  | MRI |  | Pathology |  | Overall |  |\n|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|\n| | **Acc** | **F1** | **Acc** | **F1** | **Acc** | **F1** | **Acc** | **F1** | **Acc** | **F1** |\n| **SFT** | **.715±.051** | .066±.010 | .548±.028 | **.235±.000** | .567±.038 | .197±.000 | .652±.000 | .083±.000 | .602±.014 | .139±.001 |\n| **ReMax** | .678±.015 | .089±.006 | .547±.044 | .147±.008 | .547±.012 | .264±.018 | .706±.007 | .270±.011 | .596±.002 | .176±.001 |\n| **RE++** | .672±.003 | .098±.013 | .519±.010 | .136±.012 | .651±.022 | .420±.025 | .668±.024 | .254±.026 | .621±.020 | .202±.053 |\n| **RLOO** | .670±.003 | **.099±.013** | .519±.012 | .144±.014 | .658±.032 | .432±.020 | .699±.011 | .216±.024 | .611±.002 | .189±.003 |\n| **GRPO** | .670±.001 | .086±.002 | .520±.009 | .146±.011 | .631±.027 | .395±.030 | .715±.004 | .286±.010 | .609±.005 | .193±.003 |\n| **DRPO**_DomainOnly_ | .669±.000 | .083±.000 | .480±.040 | .098±.031 | .733±.012 | .475±.012 | **.762±.009** | **.388±.028** | .668±.004 | .237±.004 |\n| **DRPO** | .672±.004 | .093±.008 | **.555±.014** | .223±.021 | **.789±.019** | **.625±.028** | .708±.010 | .265±.016 | **.666±.009** | **.295±.013** |\n\n\nThe standard deviation for all methods is around ~1% across runs. In terms of overall performance, our method (DRPO) significantly outperformed all baseline methods (SFT, ReMax, RE++, RLOO and GRPO) with p<0.0001. \n\nW2: We evaluated the fairness of the following eight datasets from CLIMB, where the demographic information is available: CheXpert, Vindr-Mammography, HAM10000, Hemorrhage, COVID-BLUES, ISIC2020, PAD-UFES-20. In particular, these datasets contain the age, gender and parent birthplace of the participants. We bin the age into 20-year brackets, treat each demographic information as a group, and evaluate the standard deviation of the accuracy and F1 scores across different groups. The results are as follows (lower is better):\n\n| Method | Age Std_acc | Age Std_f1 | Gender Std_acc | Gender Std_f1 | Parent Std_acc | Parent Std_f1 | Overall Std_acc | Overall Std_f1 |\n|--------|-------------|------------|----------------|---------------|----------------|---------------|-----------------|----------------|\n| GRPO   | **0.0290**  | 0.0345     | **0.0124**     | 0.0207        | 0.2955         | **0.2958**    | **0.1123**      | 0.1170         |\n| DRPO   | 0.0298      | **0.0279** | 0.0164         | **0.0154**    | **0.2927**     | 0.3023        | 0.1130          | **0.1152**     |\n\nIn general, we do not observe a significant difference in fairness between GRPO and DRPO (our method). Our method gives a more homogeneous F1 score across groups, while the standard deviation in accuracy across groups for GRPO is lower. We will conduct a more thorough analysis of the fairness of these models and training methods as a part of our ongoing effort. \n\nQ2: The number of clusters in the model is determined automatically via the elbow method, as described in Sec. 3.3 and App. B. 2, with the possibility to set an upper limit on the number of clusters. In the original experiment, the cluster limit for each domain is set to 10. We tested the model with 1 (no clustering), 3, 10 and 20 clusters. The results are as follows:\n\n\n| **Cluster** | **CXR** |  | **Mam** |  | **Derm** |  | **CT** |  | **Fund** |  | **US** |  | **MRI** |  | **Path** |  | **All** |  |\n|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|\n|     | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 |\n| **1** | **.694**| .085 | .746 | .211 | .678 | .286 | .571 | .257 | .669 | .083 | .544 | .200 | .757 | .505 | **.773**| **.449**| .679 | .259 |\n| **3** | **.694**| .125 | .568 | .048 | .680 | .356 | .562 | .284 | **.672**| .147 | .520 | .152 | .717 | .546 | .723 | .289 | .642 | .244 |\n| **10** | .691 | .125 | **.759**| .253 | **.707**| **.400**| **.580**| **.321**| .670 | .088 | **.568**| **.240**| **.806**| **.652**| .707 | .303 | **.686**| .286 |\n| **20** | .668 | **.167**| .751 | **.268**| .675 | .300 | .548 | .262 | .635 | **.166**| .547 | .214 | .804 | .649 | .731 | .329 | .670 | **.294**|\n\nIn general, we observe that having no cluster or a very low cluster limit will cause a decrease in performance. A higher cluster limit, however, does not seem to hurt the performance, as the elbow method automatically chooses a lower cluster count than the limit. This allows the algorithm to remain efficient under arbitrary cluster limits. \n\nQ3: The core contribution of the hierarchical scaling scheme is that it performs better for rare conditions and rare datasets. As described in Figure 2, compared to GRPO, DRPO performs particularly better in clinical modalities where data is rare. In addition, the gap between training and validation accuracy is similar across all methods and modalities, which indicates that DRPO does not cause the model to overfit to smaller datasets. \n\nW3: We agree that synthetic segmentation labels may introduce errors for some rare cases.  BiomedParse reported an average dice score of 0.91-0.98 across clinical domains. We randomly selected 100 images evenly across the eight imaging domains where ground truth segmentations are not available. We observed that the segmentations for most classes are highly consistent with the true abnormalities, like Lung Opacity, Acute pulmonary embolism and Pituitary Tumor, with an estimated dice score of >0.8. However, the segmentations for some rarely seen or novel classes, like COVID-19 and malignant lesions in breast ultrasound, are less accurate, with an estimated dice score in the 0.3-0.4 range.  Due to the restriction of visual content in the rebuttal, we will provide a detailed visual and quantitative analysis of the generated segmentations in the final manuscript. \n\nIn the final manuscript, we will discuss the environmental impact in App. D.1. Our A100 SXM has a TDP of 400W. Assuming CPUs and other components have a combined TDP of 600W, training of a 7B model requires an electricity of 182.4kWh. Under the same assumption, training a 32B model requires an electricity of 571.2 kWh. We would also like to point out that releasing public models as we did has a positive environmental impact, as researchers can directly use or finetune the model on the domain data, instead of doing the pretraining repeatedly."}]

Author rebuttal:

Meta-review: This paper introduces QoQ-Med, a multimodal clinical foundation model capable of reasoning over diverse data types, including 2D/3D medical images, ECG signals, and text. The core innovation is Domain-aware Group Relative Policy Optimization (DRPO), a novel reinforcement learning algorithm that addresses data heterogeneity and imbalance in clinical datasets through hierarchical reward scaling. Trained on 2.61 million instruction-tuning pairs spanning nine clinical domains, QoQ-Med significantly outperforms other critic-free training methods such as GRPO in diagnostic performance, particularly in underrepresented domains. Additionally, the model produces interpretable outputs, including reasoning traces and highlighted image regions. The authors have released the full model weights, the modular training pipeline, and all intermediate reasoning traces.

The reviewers unanimously recognize this work as a novel, valuable, and significant contribution to the field. The authors’ rebuttal effectively addressed additional concerns, including statistical significance testing, fairness, and hyperparameter analysis. Given the paper’s originality, importance, and overall quality, I recommend it for an oral presentation to highlight its impactful contributions at the conference.
Decision: Accept (oral)