Bridging Radiology and Pathology Foundation Models via Concept-Based Multimodal Co-Adaptation
Authors: Yihang Chen
Yanyan Huang
Fuying Wang
Maximus Yeung
Yuming Jiang
Shujun Wang
Lequan Yu
Venue: ICLR 2026 Poster (2026)
Keywords: multimodal learning
concept-based learning
foundation models
parameter-efficient fine-tuning
medical imaging
survival analysis
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose a novel prompt-tuning method that uses clinical concepts to dynamically co-adapt radiology and pathology foundation models, improving multimodal fusion and interpretability.

Abstract: Pretrained medical foundation models (FMs) have shown strong generalization across diverse imaging tasks, such as disease classification in radiology and tumor grading in histopathology. While recent advances in parameter-efficient finetuning have enabled effective adaptation of FMs to downstream tasks, these approaches are typically designed for a single modality. In contrast, many clinical workflows rely on joint diagnosis from heterogeneous domains, such as radiology and pathology, where fully leveraging the representation capacity of multiple FMs remains an open challenge. To address this gap, we propose Concept Tuning and Fusing (CTF), a parameter-efficient framework that uses clinically grounded concepts as a shared semantic interface to enable cross-modal co-adaptation before fusion. By incorporating task-specific concepts that are relevant across modalities, CTF aligns radiology and pathology representations, thereby enhancing their complementarity and enabling interpretation. We further design a Global–Context–Shared Prompt (GCSP) mechanism, which employs a small set of learnable tokens to capture domain-specific priors, shared patient-level information, and cross-domain context. The resulting concept alignment scores from each modality are then fused to produce a final prediction. Extensive experiments demonstrate that CTF outperforms strong unimodal, latent-fusion, and adapter-based baselines (e.g., AUC 0.903 on TCGA-GBMLGG). Notably, CTF achieves these gains without finetuning the full FMs, requiring only 0.15\% additional parameters, thus highlighting the effectiveness of concept-based multimodal co-adaptation. Our code is available at: https://github.com/HKU-MedAI/CTF.

Published as a conference paper at ICLR 2026
BRIDGING RADIOLOGY AND PATHOLOGY FOUNDA- TION MODELS VIA SACCEPT-BASED MULTIMODAL CO-ADAPTATION
Yihang Chen', Yanyan Huang', Fuying Wang", Maximus Yeung', Yuming Jiang', Shujun Wang', and Lequan Yu!*
'University of Hong Kong Stanford University
*Wake Forest University School of Medicine
'The Hong Kong Polytechnic University
{yihange, yanyanhhe connect..........c.............. @onnect hkkuhk fuyingwestanfore edu, y jianggwfussmm...... edu
shu-jun, wang@polyu, edu.yuueedd.a.. edu.hk,
(mefyeung, 1qyu) hk
ABSTRACT
Pretrained medical foundation models (FMs) have shown strong generalization across diverse imaging tasks, such as disease classification in radiology and tu- mor grading in histopathology. While recent advances in parameter-effie efficientt fine- tuning have enabled effective adaptation of FMs to downstream tasks, these ap- proaches are typically designed for single modality. In contrast, many clinical workflows rely on joint diagnosis from heterogeneous domains, such as radiol- ogy and pathology, where fully leveraging the representation capacity of multi- ple FMs remains an open challenge. To address this gap, we propose Concept Tuning and Fusing (CTF), parameter-efficient framework that uses clinically grounded concepts as shared semantic interface to enable cross-modal co- adaptation before fusion. By incorporating task-specific concepts that are relevant across modalities, CTF aligns radiology and pathology representations, thereby enhancing their complementarity and enabling interpretation. We further design a Prompt (GCSP) mechanism, which employs a small set of learnable tokens to capture domain-spectin mation, and cross-domain context. The resulting concept alignment scores from each modality are then fused to produce a final prediction. Extensive experiments demonstrate that CTF outperforms strong unimodal, latent-fusion, and adapter- Notably, CTF achieves these gains without finetuning the full FMs, requiring only 0. 0.15% additional pa- rameters, thus highlighting the effectiveness of concept- based multimodal co- adaptation. The code available at https: / thub. com/ HKU-MedAT/ CTF
priors, shared patient-level infor-
based baselines (e.g., AUC 0.903 on
INTRODUCTION
Foundation models (FMs) are increasingly demonstrating significant potential in transforming healthcare by enabling the joint analysis of medical images and associated textual information (Cui jt al. 2023 Steyaert et al. 2023 Qian al. 202T Radford et al., 2021). clinical practice, how- ever, a patient's 8 condition is often assessed through multiple diagnostic domains such as radiology scans (e.g., CT, MRI) providing macroscopic structural information and pathology slides revealing microscopic cellular details (Tomaszewski & Gillies) [021; Qi al. 2024p. Integrating information from these diverse sources is crucial for a holistic understanding of disease processes and accu- rate prediction of clinical outcomes like patient survival or tumor grade (Rahaman al.) 2025] Wang et al., 2024). Yet, many current lasion-language FM applications in healthcare operate within siloed domains
for radiology, another for pathology- each with its own "visual language"
"Corresponding author.
We use the term "domain" to refer pathology and radiology, and "modal" refer texts and images.

---

Published as a conference paper at ICLR 2026
(a) Eaisting Method
(b) Our Appronch (CTF)
Rediolegy
Radisiogy PM
Radisigy
Radislop 1M
Labest Fuiien (backkhoss)
Concapt Bridge
Concepe- flowd Fasion
Puthslagy
Puthedggg
Veatures
Stale
Pathelegy Imaege
Pathellgy TM
Law Mide
Medium lire.
L8gh
Prediction
Figure I: Conceptual comparison of multimodal fusion paradigms. (a) Conventional pipeline: radi- ology and pathology FMs are used frozen feature extractors; fusion is performed on static latent features. (b) CTF performs cross-domain co-adaptation of risk-stratified concept semantics before fusion, enabling interpretable predictions.
(Lu et al. 2024; Zhang et al., 2023 Majzoub al. 2025). The central challenge, therefore, is not merely to combine their outputs but to bridge these distinct expert models in way that creates deep, synergistic understanding and maintains the clinical interpretability crucial for high-stakes medical decisions.
Existing attempts at multi-domain integration in medicine face practical and methodological limits. A common strategy uses pre- pre-trained Vision Language Models (VLMs) fixed feature extractors for each domain, followed by a simple fusion mechanism (e.g., concatenation) applied to these static representations (Zhang al., 2024a Xu & Chen 2023). This approach limits the model's adapt- ability to the specific information of the downstream task and the interplay between modalities. Conversely, full fine-tuning of large VLMs computationally expensive and often remains con- fined to the model's pretraining domain, weakening knowledge transfer across domains (Shi et al., 2024). Furthermore, both approaches often result in "black-box" systems where the reasoning be- hind predictions remains opaque, posing challenges for clinical trust and interpretability- a critical requirement in high-stakes medical decision-maie making (Amann al., 20200) Rudin [2019]
| [Doshi- Velez & Kim 2017). This conventional paradigm, which relies on extracting static features from siloed modalities before simple, non-interpretable fusion step, is illustrated Figure
To overcome these challenges, we argue that the key to unlocking synergy is to establish shared, interpretable vocabulary that can bridge the semantic gap between FMs from different domains. Clinically-groundee concepts, such as "tumor necrosis" or "cellular atypia", provide this natural bridge. However, treating concepts as fixed definitions brittle: the prognostic meaning of a con- cept in one domain often depends context from the other. For example, "irregular tumor margins" in radiology is far more alarming when paired with histopathology evidence of "lymphovascular in- vasion." " While recent works have begun aligning expert-derived concepts with images (Zhao al.) 2024 Nguyen et al. 2025 Zhu et al., 2025a), they do not dynamically modulate these concepts based on cross-domain information. Our core insight that concepts should not be static bottle- neck, but a dynamic medium for co-adaptation, where the semantic representation of a concept in one modality is actively tuned by features from the other.
To this end, we introduce Concept Tuning and Fusing (CTF), novel framework that bridges ra- diology and pathology VLMs through -erricel concepts (Figure [p)) Instead of fusing static features, CTF forces each modality to "be aware" of the other during the feature extraction process. The core of our framework the Slohal-Context-Sharee Prompt (GCSP) strategy, prefix tuning method that conditions the interpretation of concepts within one domain (c.g., radiology) on the visual features from the complementary domain (e.g., pathology). This cross-domain condi- tioning allows each VLM to produce richer, contextually the patient case before they are fused for a final prediction. We detail three stages: (i) MI + diver- sity concept selection (Sec 3.1), (ii) GCSP-based cross-domain concept tuning (Sec 3.2), and (iii) concept-score fusion and prediction (Sec 3.3).
concept representations tailored to
Our main contributions are:
2

---

Published as a conference paper at ICLR 2026
We propose CTF, new framework that uses medically-relevant concepts as dynamic and interpretable bridge to fuse distinct medical VLMs, moving beyond conventional "black-box" latent fusion.
We introduce the Global-Context-Shareee Prompt (GCSP) strategy, a novel cross-domain con- ditioning mechanism that uses efficient prompt tuning (#00. 15% trainable parameters) to adapt the semantic meaning of concepts based complementary domains.
We conduct extensive experiments on four public and in-house datasets, demonstrating that CTF significantly outperforms state-of-the-art unimodal and multimodal methods in both survival analysis and cancer grading, achieving C-index improvement of 3.5% and AUC improve- ment of 2.9% over the strongest baselines, respectively.
2 RELATED WORK
Multimodal Fusion for Clinical Prediction. The integration of diverse data, especially macro-
scopic radiology and microscopic pathology, is critical in oncology (Zhao et al. 2022] /hhu et al,, 2025b), with multimodal models consistently outperforming unimodal approaches (Benani et al.
2025). A dominant paradigm is latent-space fusion, where features are independently extracted and then combined using methods like co-attention transformer (Xu & Chen 2023 Lu et al,, 2019 Chen ct al. 2023), or ceermation-theorette disentanglement to separate shared and specific information (Zhang et al.) 2024a). While powerful, these methods primarily fuse latent representations that are already fixed, treating the feature extractors as black boxes and limiting the depth of synergy. Even when powerful pathology foundation models are used as the pathology encoder (Chen et al. [2024a] Ma et al. 2025 Xu et al. 2024), they typically plugged in as static feature extractors within this latent-fusion pipeline. Recent robust MIL methods for WSI analysis. such as cDP-MIL and SIB- MIL (Chen et al. 2024b 2025a), further highlight the importance of handling instance-level noise and heterogeneity in pathology representations. Our work proposes a shift: instead of refining the fusion of static features, enable dynamic dialogue between domains. Unlike latent fusion that combines already-fixee -ixeed features, and fine-tuning that adapts backbones separately, CTF explicitly co-adapts concept semantics using cross-domain prompts before any fusion.
Foundation Model Adaptation and Cross-Domain Guidance. The advent of foundation models (FMs) offers new avenues for realizing this dialogue. While many works use FMs as off-the-shelf feature extractors, Parameter-Efficie methods have emerged to adapt them. Techniques include architectural adapters (Houlsby et al. 2019) and recent medical adapters (Lee et al.) 20255 or prompt-based tuning (Lester et al. 2021) which have been used adapt VLMs for specific domains or tasks (Zhou 2022), often in unimodal fashion (Peng et al. 2025). In computational pathology, knowledge-enhancee compres- sion and prompt-like adaptation have been explored for few-shot WSI classification (Guo et al. 2025), and parameter-effiiien tuning has also been leveraged for medical report generation in fed- crated settings (Che et al.] 2025). CTF innovates by using prompts not just for task adaptation, but as cross-domain conditioning mechanism where one domain dynamically influences the semantic interpretation within another, creating an integrated, context-aware system before fusion.
Fine-Tuning (PEFT) (Hu et al., 2022; Li & Liang] 1
Concept- Based and Interpretable Multimodal Learning. A major barrier to the clinical adop- tion of deep fusion models is their lack of interpretability (Amann et al. 2020). Concept Bottle- neck Models (CBMs) (Koh al. 2020 Ghorbani et al. 2019) address this by forcing predictions through a set of human-understandaal
concepts. While foundational, this rigid bottleneck can limit performance. The field is evolving towards more flexible, interpretable frameworks, such as using multimodal contrastive learning (Nauta et al.) (02T) to find local, explainable correlations between imaging and text or building large-scale, concept-centric FMs like ConceptCL.IP (Nie et al., 2025). CTF is inspired by this philosophy but makes a crucial contribution to the multimodal context. We treat concepts not as static bottleneck, but as the very medium for the cross- cross-domain guidance de- scribed above. Our key innovation, the GCSP strategy, allows each VLM to adjust its understanding of concepts like "tumor invasiveness" based on real-time information from the complementary do- main. This unique synthesis provides the deep synergy of cross- dross-domain guidance while leveraging the transparency of concept-basee reasoning, distinguishing our work from both black-br box fusion techniques and traditional CBMs.
3

---

Published as a conference paper at ICLR 2026
J. Concept Generation and Selection
Rudidagr
Hhakrlogy Visio Sacodar
Workflow of Concept Tuning and Fusling (CTF) framework
Fackdag, Viean eeookkin
V
Whele de of dei
Oldbul-armtett- Shared Prompt furing
Izara
Delyy Tel Bost
Chttall Phomes Medidity spootto
Pg
(
Gureapt
retullaation
Garly muds effect/ekeme
Stared Fronpt
Low
Noe rasty ctud difuuian
Priverey Tedt Ercoder
Second Round
Decressed Call Calldaaity
|+--
Conceet
Smergin Mitoiic Activity
Geingettee Conkel Pommpta
Tecressed Callbariyy
Huttckog Vision
VA
Tasc-miieed Concopra
Figure 2: Overview of the CTF framework.(1) Concept Generation & Selection identifies relevant and diverse medical concepts. (II) Concept Tuning & Fusing freezes radiology/pathology vision encoders. GCSP prepends three prompts to each concept's text tokens-(a global (task adaptation), (b) shared (MLP over [fr. fi.l). and (c) context (gated by the complementary modality) before the frozen text encoders, producing per-domain concept scores that are concatenated for prediction.
METHODOLOGY
Our Concept Tuning and Fusing (CTF) framework enables synergistic integration of radiology and pathology data by creating a shared semantic bridge built on medically relevant concepts. The framework, depicted Figure 2 proceeds in three main stages. First, in Prognostic Concept Se- lection (See 3.1), we generate comprehensive pool of medical concepts and then use principled optimization strategy to select a compact, diverse, and prognostically relevant subset for each do- main. Second, in the core of our framework, Cross-Domain Concept Co-Adaptation (S2c Prompt (GCSP) mechanism to dynamically tune the textual
use our novel Global-Context-Shale embedding of each concept, making it aware of both the downstream task and the cross-domain context of the specific patient case. Finally, in Fusion and Interpretable Prediction (Sec 333). these co-adapted concept scores are fused and fed to prediction head, ensuring that the final output is grounded in a transparent, concept-level rationale. The entire model, with only the lightweight prompt modules being trainable, is optimized end-to-end.
3.1 FEATURE EXTRACTION AND PROGNOSTIC CONCEPT SELECTION
Vision Feature Extraction. Given radiology image Dr and pathology whole-slide image (WSI) Thi we first obtain global feature representations using the vision encoders (V,, Vi)) of powerful, pre-trained medical VLMs. This yields feature vectors f. Vy(x,) E P.. and fa = Vi(xx)) E Pun, where Der and Duh are the dimensions of the respective vision feature spaces. These encoders remain frozen during training preserve their rich, pre-trained knowledge.
Prognostic Concept Selection. A high-quality set of medical concepts forms the foundation of interpretable bridge. Given initial large candidate pool S (generated via LLMs per domain. Details in Appendix C. we aim to select compact subset C of size l that is both prognostically relevant and semantically diverse. This avoids selecting redundant concepts (e.g.. "irregular mar- gins" and "ill-defined borders""). We formalize this as maximizing submodular objective function (Harshaw et al. 2019 Lin & Bilmes 2012), which balances relevance d(-) and diversity:
F(C) = Σ d(c) +
min (1 - (tc,tt)))
cEC e'ECi(C)
CEC
As this is NP-hard, we approximate the solution with two-stage greedy algorithm (shown in Fig.
1. Relevance Ranking: We first score every candidate concept E S based on its prognostic relevance. We define this relevance score d(c) using the Mutual Information (MJ) between the cept's aligmment scores and patient labels (Kraskov et al. 2004). For each image 2;, an alignment score a(x;;c) c) (t_ ft)/([[tell ( is computed, where t. T(c) is the text embedding of
4

---

Published as a conference paper at ICLR 2026
concept from the text encoder T()). We estimate MI via sklearn's kNN-based mutuallinff _nfooclassiiiittt (Ross 2014), which discretizes the continuous alignment internally. We then compute its MI with p(a.y) All concepts in S are then
the label Y as d(c)) = T(A:Y)) = Σyer Σāea, p(ā, v) log p(Atr())) sorted in descending order based on this score.
2. Diversity Maximization: We initialize our final concept set with the top-ranked concept from the first round. We then iteratively add concepts to C. from the sorted list. At each step, we select the next concept c* that maximizes semantic diversity with respect to the concepts already chosen, defined as e" arg esx (mincec - (tter, te))), where d(: is the cosine simi- larity between concept text embeddings from the VLM's text encoder T(.). This process continues until [C] = k, yielding the final, high-quality concept sets Crad and Chist-
This selection performed once with frozen encoders, and tuned prompts (Sec. 3.2) are used only during model trainingginference.
3.2 CROSS-DOMAIN CONCEPT CO-ADAPTATION
Static concept representations fail to capture how the meaning medical finding can shift based on complementary information. To address this, we introduce the Global-Context-Shee Prompt (GCSP) strategy, a parameter-effiient tuning method that generates a dynamic, case-specific prefix pluned for each concept e before it is processed by the frozen text encoder T(-). Unlike methods that aim to modify static latent features, our approach actively conditions the semantic representation of concepts within each domain, creating synergistic dialogue before fusion. We apply GCSP symmetrically: radiology concepts are conditioned on pathology features and vice versa (Fig. 2 For radiology (pathology by symmetry) concept, the prefix piuned RRLDD is concatenation of three specialized components:
pluned = Concat (Pa, Ps(f...))
(2)
Global Prompt (Pa). For each concept we learn dedicated, domain-specins prompt vector Po(c). This prompt is shared across all patients within one domain and provides a general adapta- tion of the concept's 's pre-trained meaning the specific nuances of the downstream task.
Context Prompt (Pc). This prompt is the key to our cross-domain guidance. It allows one modal- ity to influence the interpretation of concepts in the other via ofittur--of-xppeee (MoE) style layer (Shazeer et al.] 2017). For a radiology concept Gr. the context prompt is generated from the pathology image feature Fn. We maintain learnable pool of M basis prompt vectors (Phasis Ji=1
shared across all concepts in that domain. A lightweight gating network, 9r Peh + M takes the complementary feature fı. to produce mixture weights: Pa(fi) = E a = softmax (f))). This mechanism dynamically selects and weights conceptual attributes and provides patient-specifie, cross-domain conditioning of concept semantics (see Appendix A.I for
C, where
more details). The same mechanism is applied symmetrically for pathology concepts.
Shared Prompt (Ps). This prompt captures holistic, patient-specific synergy. First, shared latent vector is produced by small MIP, Øs. that takes the concatenated features from both modalities as input: fs Øs (Concati fr, fr)). This shared feature is then projected by two separate linear layers, to suit different VLMs' context: Ps,((s)) (s,((fs) and Ps. (fs) ss This provides a unified adjustment signal to all concepts for given patient, ensuring coherent refinement across domains.
3.3
CONCEPT-BASER FUSION AND TASK-SPECHIS PREDICTION
After generating the composite prompt Pruned for each concept, we prepend Pruned to the tokenized concept string. Then we obtain set of tuned textual embeddings. (te, and {te, }. from the text en- coder, where t. T [unnde tokens for c). We then compute two concept score vectors, s, E [Gal and 8). IR [Cmal, representing the alignment of each image with its corresponding tuned concepts, where the j-th element of each vector is computed as: Sr,j (f. f
llz) and ll2). These score vectors provide an interpretable representation of
Sh.j = (fT ( cg the patient's condition. The final patient representation is formed by concatenating the scores from
5

---

Published as a conference paper at ICLR 2026
both domains: = Concat(s,, sn,). This interpretable vector is then passed through final predic- tion head, a multi-layer perceptron (MLPpred), to produce the task-specific output o = MLP All FM encoders remain frozen and only the prompt modules and prediction head are trainable, totaling 0.5M (0.15% 15% of 307M across both FMs).
Optimization. The entire framework is trained end-to-end using loss function appropriate for the downstream task. For survival analysis, we use the Cox Proportional Hazards model (Cox) 1972) where is a vector of risk scores. The model is optimized by minimizing the negative log partial
likelihood Loox = - - 6:6.11 (r2 log ZjER. exp(r;)). where 6; 1 if the event (e.g., death) was observed and 6; = if the data is right-censored, r; is the predicted risk for patient and R; = {ilt; 2 f4] is the set of patients still at risk at time & (Katzman et al.) 2018). For a classification task like cancer grading, the prediction head ML.pped pred outputs class logits, and the model is optimized using a standard cross-entropy loss. Note that MI ranking used only offline for concept selection. At inference, we compute cosine-normalized concept scores with tuned embeddings and no labels or MI are used.
4 EXPERIMENTS
We designed a comprehensive set of experiments to validate the core hypotheses of our work. Our evaluation structured to demonstrate that (1) our concept-based fusion paradigm, CTF, surpasses ofthe-t methods that rely on static or independently adapted latent features; (2) the perfor- mance gains are primarily driven by our novel GCSP strategy, which enables dynamic, cross- domain dialogue; and (3) CTF yields predictions that are not only accurate but also interpretable, robust across diverse clinical tasks, and grounded in clinically plausible reasoning.
4.1 EXPERIMENTAL SETUP
Datasets. We evaluate CTF on two distinct and clinically vital predictive tasks: survival analysis and cancer grading. We curated four datasets spanning different cancer types and imaging modalities. For survival analysis, we use three cohorts with paired imaging and clinical data: TCGA-LG TCGA-GBM, and private Centerl-Ge (Gastric Cancer). For cancer grading, we evaluate perfor- mance on three cohorts. including private Center2-CHS (Chondrosarcomai) The specific classifica- tions include 3-way WHO grades for the brain tumor cohorts (LGG and GBM merged) and 5 5-way TNM T-stage for the gastric cancer cohort, a particularly challenging benchmark task. We report msd over 10 stratified train/val/test splits. Besides, we perform paired t-tests between CTF and the strongest baseline per task in Appendix [D.4]
Implementation Details. To instantiate our framework, we selected domain-experi foun- dation models to maximize clinical relevance: BiomedCLIP (Zhang et al. 2023) for radiol- ogy and CONCH (Lu et al. 2024) for pathol- ogy. By keeping their vision encoders frozen, leverage their specialized knowledge bases and focus our method on creating a synergistic dialogue between them. For each domain, we selected A = 256 concepts using the designed 12 tun- able tokens (Sec. 3.2).. The whole framework, including all prompt generators, is trained end- to-end. Full implementation, hyperparameter details, and hardware/GPU usage are in Ap- pendix A.2 and Appendix A.3
Table 1: Survival prediction performance (C- index 1) on three datasets. Best performance is in bold, second-best is underlined.
Model
TOGA-LGG
TOGA-GBM
Centerl-GC
Uninodo! Baxeliner
Radiologa-Dei Only
0.598 00128
0,477
0.055
0.614 = 0.052
ABMIL llse et al. 2018
0.669 = 0.101
0.480
0.93
0,590 0.000
CLAM (ea dizn
0.689
0.188
0.497
0068
0.31 - 0.60
TransMi
2001
0.682 0.121
0.50B
0.115 0.06
Zhin di
0.678
0.519
0.033
Muliinder Lateet Fasion
Concat- Fussit-Fussion
0.674 0.112
0.515
0.770
008
strategy (Sec. 3.I and inserted
Crocs Attezion
0.685 - 00108
0,527
0.068
0.631
0.060
MOTCAT XX & 2033
0.571
0.080
0.563
0.108
0.622
0.640
PIBD Zhulig ar. 20240
0.687
= 0.123
03331
0.06
0.638
0.088
Multimodif Adkentiv Fusion Bareline
M4Survne (Lee al. 2025
0,709
0.112
0,545
0.072
0.642 0.065
0.103
0.579
0.061

---

Published as a conference paper at ICLR 2026
Table 2: Cancer Grading Performance (AUC1 and ACC?). Best performance in bold, second-best is underlined.
TCGAGGBMLGG (3-way)
Center2-CHS (5-way)
CenterI-GC (S-way)
Model
AUC1
ACC1
AUC1
ACC₁
AUC
ACC
Unimodal Baseltnes
Radiology-Onf
0.776 + 0.059
0.624
0.066
0.679 + 0.069
0.429
0.001
0.595 + 0.087
0.341
0.080
ABMIL [sse et al 2018
0.835 + 0.050
0.667
+ 0.076
0.770 0.092
0.493 2: 0.138
0,609 = 0.063
0,384 1: 0.033
CLAM LLnet al 2021
0.860 0.048
0.681 0.077
0.775 + 0.089
0.512+: 0.130
0.628 + 0.055
0.390 0.051
TransMIL Shad et m. 2021
0.864 = 0.050
0.681 + 0.068
0.781 + 0.065
0.518 ::18::00 0.128
0,625 = 0.038
0.388 0.0-1
ACMIL (Zhang at all
2024b
0.853 + 0.046
0.680 + 0.069
0.779 + 0.046
0.515 +: 0.129
0.619 + 0.062
0.389
0.054
Multimodal Latent
Raselines
Concat- Fusion
0,858 +: 0.038
0.687 + 0.062
0.805 + 0.075
0.535
0.115
0.629 + 0.051
0.391
0.048
Cross Attention
0 +: 0.030
0.695 +: 0.059
0.817 +: 0.071
0.581 0.110
0.635 +: 0.048
0.394 0.049
MOTCAT (Xu & Chen 2023
0,865 + 0.025
0.657 + 0.053
0.826 +: 0.078
0.612
0.120
0.641 + 0.050
0.390
0.052
Mudtimodul Adaptive Fasion Baseline
MASurvive (Lee et ul.
2025
0.861 +: 0.031
0.691 + 0.061
0.830 + 0.076
0.626 + 0.115
0.649 + 0.052
0,390 0.051
CTF (Ours)
0.028
0.718 0.063
+ 0.081
0.698 0.164
0.660 0.044
0.401 0.057
4.2 BASELINES
We compared CTF against a comprehensive set of models. Unimodal baselines include several high-performing single-modality methods (ABMIL. (Ilse et al. 2018), CLAM (Lu et al. 2021)
etc.). Fusion baselines include: (1) Simple Fusion methods (Concat-Fusion, Cross-AAtention): (2) State-of-the-Art (SOTA) I Latent Fusion methods that fuse static features (MOTCAT (Xu & Chen 2023). PIBD (Zhang et al. 2024a)): ): and (3) SOTA Adaptive Fusion method that fine-tunes model weights (M4Survive (Lee el al. (025)). Detailed descriptions of each baseline are in Appendix AA.3 A..3 For fairness, all baselines share the same frozen vision encoders and survival/elasiifcation heads, and only fusion modules differ. Adaptation specifics for MOTCAT/PIBD are in Appendix[A.. A.4
4.3 QUANTITATIVE RESULTS
Superior Survival Prediction. As presented in Table T CTF consistently achieves state-of-the-art performance, outperforming all baselines in our study on the three survival prediction cohorts. No- tably, it achieves a C-index (Harrell et al. 1982) of 0.713 on TCGA-LGG, surpassing the strongest adaptive baseline, M4Survive, and the strongest latent fusion baseline, PIBD, by 3.8%. The results reveal a clear hierarchy of fusion strategies. While all multimodal methods generally outperform unimodal approaches, affirming the value of data integration, the key distinction lies in how fusion is performed. Advanced latent fusion models like PIBD show respectable gains over simple con- catenation but are ultimately limited by their reliance on static, pre-extracted features. Adaptive methods like M4Survive improve upon this by fine-tuning architectural components.
However, CTF's superior performance suggests fundamental advantage. Instead of simply fusing latent vectors or adapting architectural blocks, CTF performs semantic co-adaptation. By dynami- cally tuning the meaning of clinical concepts in one domain based on context from the other before fusion, it achieves deeper, more synergistic integration. This consistent performance gain across diverse cancer types aligns with our hypothesis: dynamic, concept- based co-adaptation is a more effective paradigm for multimodal fusion than static latent fusion or independent architectural adap- tation.
Generalization to Cancer Grading. As shown in Table 2 CTF again achieves state-of-the-ae performance, outperforming all baselines across the three datasets. On average, CTF obtains an AUC improvement of 3.6% over the strongest fusion baseline (MOTCAT). This is a significant result, as cancer grading relies on identifying distinct morphological and cellular patterns. CTF's success suggests its ability to learn rich, concept-based dialogue between radiology's 's macro-structural views and pathology's micro-cellllar details is highly effective for this classification task. This robust performance validates the broader utility of our concept-tuning and fusion paradigm.
7

---

Published as a conference paper at ICLR 2026
Table 3: Ablation on the Centerl-Ge dataset for survival prediction and tumor grading. Metrics reported as mean with standard deviation in parentheses. Best numbers are in bold. A denotes absolute change vs, the full CTF model.
Survival Prediction
Tumor Grading
Category
Variant
C-index (1)
AT)
Reference
CTF (Full Model)
0.665 (0.061)
()
0.660 (0.049)
()
w/o Context Prompt (Pc)
0.629 (0.058)
(-0.036)
0.635 (0.047)
(-0.025)
Prompt
Components
w/o Shared Prompt (Ps)
0.653 (0.063)
(-0.11))
0.651 (0.056)
(.0.009)
w/o Globul Prompt (P))
0.642 (0.051)
(-0.023)
0.640 (0.048)
(-0.020)
Tuning
Static Concepts (CBM)
0.586 (0.049)
(-0.079)
0.622 (0.055)
(-0.038)
Strategy
Static Concepts Prompt Tuning
0.638 (0.059)
(-0.027)
0.635 (0.052)
(-0.025)
Concept Selection
Random Selection
0.622 (0.053)
(-0.043)
0.654 (0.055)
(-0.006)
Top-MI Selection (relevance-only)
0.646 (0.060)
(-0.019)
0.642 (0.049)
(-0.018)
CTF (General, CLIP + CLIP)
0.621 (0.055)
(-0.044)
(0.054)
(-0.045)
Backbone
CTF (Hybrid, CLIP + CONCH)
0.639 (0.059)
(-00026)
0.643 (0.050)
(-0.017)
Sensitivity
CTF (Expert, BiomedC1.IP MUSK)
0.680 (0.064)
(+0.015)
0.658 (0.053)
(-0.00Z)
CTF (Expert, BiomedCLIP PLIP)
0.627 (0.063)
(-0.038)
0.636 (0.052)
(-0.029)
4.4 ANALYSIS AND ABLATION STUDIES
The Critical Role of Cross-Domain Dialogue. As shown in Table 3 the most impactful ablation was the removal of the Context Prompt (Pc), which embodies our cross-domain guidance mech- anism. This single change caused the most significant performance degradation, with the C-index plummeting by 0.036. This result provides direct empirical evidence that forcing each modality to be "aware" of the other during concept interpretation is the primary driver of CTF's success. The re- moval of the Global (task-specific) and Shared (synergistic) prompts also led to performance drops, confirming that all components of the GCSP strategy contribute meaningfully to the performance.
Dynamic Tuning Static Concepts. We then tested model using "Static Concepts" without any prompt tuning. which is analogous to a standard Concept Bottleneck Model (CBM) (Koh et al.) 2020). Specifically, the "static concepts (CBM)" variant uses fixed, pre-trained concepts and feeds their scores directly to an MLP head for downstream predictions. As in Table we found the model intermediate
performed poorly (C-index 0.586), demonstrating that simply using concepts layer is insufficient. The dynamic adaptation enabled by GCSP is paramount. Interestingly, this aligns with recent findings that show modern, flexible prototype-base methods are moving beyond the rigid CBM structure (Chen et al. 2025b). Our work contributes a novel cross-modal tuning mechanism to this emerging paradigm.
Concept Selection and Backbone Choice. The inputs to our framework are also critical. We compared our prognostic and diversity-awand concept selection against two alternatives: Random Selection and Top-MI Selection (relevance-only). As shown in Table [ our principled strategy significantly outperforms both, validating the need to select concepts that are both prognostically relevant and semantically diverse. Furthermore, to confirm the value of domain-specific foundation models, we replaced the expert BiomedCLII and CONCH encoders with general-purpose CLIP models. This "Generalist" setup led to substantial performance drop (C-index from 0.665 to 0.621), confirming that CTF's ability to bridge modalities is maximized when it operates on the
Center1-GC
TCGA LGG
TCGA-GBM
0.75
0.713
0.75
0.725
0.70
0.665
0.70
0.665
0.65
0.65
0.60
0.579
0.60
0.579
0.55
0.55
16
20
126 Concepts
512
Number Learnalle Tokens
Figure 3: Hyperparameter sensitivity analysis on all three datasets for survival prediction. (a) Perfor- mance (C-index) versus the number of learnable tokens (L) per prompt component. (b) Performance versus the number of selected concepts (k)) per domain.
8

---

Published as a conference paper at ICLR 2026
Top Pathology Concepts
ROI
Attention Heatmap
CT
Top Rediology Concepts
rengertii
Patten 219, 3..91 Pattent (8,
Permertve of brtege
seseed merolalee
Concept-based based Interpretation of CTF Predictions. Visualization of top 5 pathology and radiology concept scores for (top) a low-risk patient censored at late time point and (bottom) high-risk patient with an early event from the Centerl-GC dataset.
Figure 4: Concept-
rich representations of expert models. When fixing radiology to CLIP and varying pathology VLMs ('Hybrid' rows in Table 3). we observe that pathology-speciin VLMs (CONCH, MUSK (Xiang jet al. 2025), PLIP (Huang et al. 2023) outperform generic CLIP, with MUSK slightly ahead of CONCH on the survival task of CenterT-GC.
Parameter Sensitivity. We analyzed CTF's sensitivity to two key hyperparameters in our GCSP strategy: the prompt length L and the number of concepts k. As shown in Figure 3 we evaluated performance across all three survival cohorts. In Figure Bh. we varied the length of the tunable prompt from 4 to 20. Performance is generally stable, with a slight peak L 12. Shorter prompts may lack expressive power, while longer ones increase parameter counts without clear benefit, validating our choice of L 12 as efficient and effective setting. In Figure Bp. we varied the number of selected concepts per modality from 32 to 512. Performance peaks at A 256. Too few concepts (e.g., h = 64) fail to capture sufficient prognostic information, while too many may introduce noise and larger computation cost without improving, and in some cases slightly degrading, performance. This confirms 256 might be an optimal choice.
512)
4.5 QUALITATIVE ANALYSIS AND INTERPRETABILIT
Beyond quantitative benchmarks, a critical goal of CTF is to provide transparent and trustworthy predictions. We conducted series of qualitative analyses to demonstrate that our model's reasoning grounded clinically relevant patterns.
inspect which radiology and pathology concepts dominate the prediction for given patient. visuaiisssss final concept scores for two Centerl-GC patients: one low-risk censored late, and one high-risk with early event. The low-risk patient (Figure 4 top) suggests a less aggressive phenotype, such as "Well- differentiated tumor". In contrast, the high-risk patient (Figure 4 bottom) shows high scores for aggressive concepts like "Disseminated tumor cells" and "Bad blood vessel morphology", providing a clear and alarming prognostic signal. This demonstrates CTF provides accurate predictions with transparent, clinically relevant rationale based on meaningful concepts.
Patient-Level Rationale. Because is vector of concept scores,
Semantic Drift. We compared the concept embedding before and after tuning. To quantify how GCSP modifies concept semantics, we compute, for each concept the cosine similarity 8, = cos(t, t.; tz) between the tuned embedding f, and original text embedding t, from the frozen encoder. For Center I-C'ss radiology concepts, the distribution of concentrates well above 0.5 with a median of 0.633 (Fig. [A),, indicating moderate, musk-sharpened shifts rather than wholesale redefinition. The t-SNE visualization (Fig. 5)) corroborates this pattern, showing coherent displace- ment without mode collapse. Practically. this bounded drift preserves the interpretability of the original human-readable concepts while enabling measurable gains in downstream prediction.
Visualizing Cross-Modal Influence. We assess whether GCSP captures clinically meaningful de- pendencies by grouping selected concepts into high-level radiology and pathology categories and

---

Published as a conference paper at ICLR 2026
12
KDE
Original Embedding
median 0.633
Tuned Embedding
9
Count
Axis
6
3
0
0.56
0.60 0.64
0.68
0.72
Axis
(a) Similarity histogram
(b) Embedding t:SNE
Figure 5: Semantic drift of concept embeddings. (a) Histogram of cosine similarity between tuned and original concept embeddings (median 0.633) indicates task-sharpened but semantically consis- tent shifts. (b) t-SNE shows systematic displacement without mode collapse.
quantifying how the caaiiology-t-ppathhhoe context prompt Pe alters pathology concept scores (vs. a zeroed Po).
Patholegy Concept Categories (Effect)
The Fig. 6 heatmap shows clinically plausible lationships learned by the model. For instance, radiology concepts related "Tumor Morphol- ogy" and "Invasion & Metastasis" strongly am- plify pathology concepts of "Cellular Atypia" and "Vascular Invasion," mimicking patholo- gist's reasoning process where macroscopic signs of aggression prompt a closer search for micro- scopie evidence (Tomaszewski Besides, radiology's "Necrosis & Cell Death" has minimal influence on pathology's "Mitotic Activ- ity". This is clinically sound: while large-scale necrosis visible on a CT scan, it is poor pre- dictor of the specific rate of cell division (mitotic count), which is only assessable under a micro- scope (Bosman al. 2010)
Atysia 0.55
0.41
0.12
0.11
1.5
Vescular trwasion-
0.30
0.13
0.53
0.18
Nor Norallze Infuree
Milati Activit
Tamor Aerhowectree
0.55
0.60
0.09
0.10
0.0
Gillies 202T).
Tumor Morphiton
Interoperetty
Invesion & Metartass
Mecmete cat Death
Radiolggy Concept Categories (Cause)
Figure 6: Cross-modal influence analysis the Centerl-Ge dataset. The heatmap shows the changes normalized influence score, repre- senting how much the presence of radiology concept category (columns) affects the scores of pathology concept categories (rows) via (Pc).
5 CONCLUSION AND LIMITATIONS
We presented CTF, a parameter-efficcient, concept-based multimodal co-adaptation framework that bridges radiology and pathology FMs. Via the Global-Comtext-Sharee Prompt (GCSP), CTF dy- namically tunes clinically grounded concepts with task, cross-domain, and shared patient context, aligning representations before fusion and yielding transparent, concept-level predictions. Abla- tions confirm the primacy of cross-domain context and principled concept selection, and qualitative analyses reveal clinically plausible rationales and influence patterns.
Limitations include reliance on a predefined concept pool and paired data. Furthermore, the modest absolute performance on the difficult 5-way gastric cancer grading task underscores that it is not ready for clinical deployment in this specific scenario, serving as a benchmark for methodological comparison. Our current concept pool is generated once by an LLM and then fixed throughout train- ing. While GCSP dynamically tunes concept semantics, does not introduce new labels or expand the concept vocabulary. Beyond tuning semantics, future work will (i) periodically refresh/expand the concept pool using model-derived importance and LLM-ddcoodig of tuned embeddings, and (ii) handle missing modalities via partial-paired training and concept imputation.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This work was supported in part by the Research Grants Council of Hong Kong (27206123 17200125, C5055-24G, and T45-401/22-N), the Hong Kong Innovation and Technology Fund (GHP/318/22GD) Guangdong Natural Science Fund (No. 2024A1515011877 1175501117)) 1875), the Start-up Fund of The Hong Kong Polytechnic University (No. PO045999), the Seed Fund of the Research Insti- tute for Smart Ageing (No. P0050946), and Tsinghua-Polys Joint Research Initiative Fund (No. P0056509), and PolyU UGC funding (No. PO053716).
ETHICS STATEMENT
This research utilized both public and private medical datasets. All data from The Cancer Genome Atlas (TCGA) are publicly available and de-identified. The in-house datasets (CenterI-GC and Center2-CHS) were collected under protocols approved by the local Institutional Review Board (IRB), with all patient data fully anonymized before use in this study. Informed consent was obtained from all participants. Our study strictly adheres to data privacy and protection regulations, as detailed in Appendix
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our results, we have made our code anonymously available at: fetts::/anooymeee 4open. Sccierce///CTF-220
The main paper provides detailed description of our methodology in Section The appendix further provides pseudocode for our core algorithms,, comprehensive architectural and training details, including hyperparameters, and specifics on baseline implementations (Appendix A).
REFERENCES
Julia Amann, Alessandro Blasimme, Effy Vayena, Dietmar Frey, Vince Madai, and Precise4Q Consortium. Explainability for artificial intelligence in healthcare: multidisciplinary perspec- tive. BMC medical informatics and decision making, 20:1-9, 2020.
Alaedine Benani, Stéphane Ohayon, Fewa Laleye, Pierre Bauvin, Emmanuel Mcssas, Sylvain Bo- dard, and Xavier Tannier. Is multimodal better? a systematic review of multimodal versus uni- modal machine learning in clinical decision-makinn medRxiv, pp. 2025-03, 2025.
Fred Bosman, Fatima Carneiro, Ralph H Hruban, and Neil D Theise. WHO classification of fumours of the digestive system. Number Ed. 4. 2010.
Haoxuan Che, Haibo Jin, Zhengrui Gu, Yi Lin, Cheng Jin, and Hao Chen. LIm-driven medical report generation via communication-effiicaa heterogeneous federated learning. IEEE Transactions on Medical Imaging, 2025.
Richard J Chen, Tong Ding, Ming Y Lu, Drew FK Williamson, Guillaume Jaume, Andrew H Song, Bowen Chen, Andrew Zhang, Daniel Shao, Muhammad Shaban, et al. Towards general-purpose foundation model for computational pathology. Nature medicine, 30(3):850 862, 2024a.
Yihang Chen, Weiqin Zhao, and Lequan Yu. Transformer-basee multimodal fusion for survival prediction by integrating whole slide images, clinical, and genomic data. 2023 IEEE 20th International Symposium Biomedicul Imaging (ISBI), pp. 1-5. IEEE, 2023.
Yihang Chen, Tsai Hor Chan, Guosheng Yin, Yuming Jiang, and Lequan Yu. cdp-mil: Robust multiple instance learning via cascaded dirichlet process. In European conference computer vision, pp. 232-250. Springer, 2024b.
Yihang Chen, Tsai Hor Chan, Jianning Chen, Liang Li, Guosheng Yin, and Lequan Yu. Sib-mil: Sparsity-induue bayesian neural network for robust multiple instance learning on whole slide image analysis. IEEE Transactions Medical Imaging, 2025a.
11

---

Published as a conference paper at ICLR 2026
Yuxuan Chen, Jiawen Li, Jiali Hu, Xitong Ling. Tian Guan, Anjia Han, and Yonghong He. Cross- modal prototype allocation: Unsupervised slide representation learning via patch-text contrast in computational pathology. arXiv preprint arXiv:2503.22110, 2025b.
David R Cox. Regression models and life-tables. Journal of the Royal Statistical Society: Series (Methodological), 34(2):187-205 1972.
Can Cui, Haichun Yang, Yaohong Wang. Shilin Zhao, Zuhayr Asad, Lori A Coburn, Keith T Wilson, Bennett A Landman, and Yuankai Huo. Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review. Progress in Biomedical Engineering, 5(2):022001, 2023.
Finale Doshi-Velez and Been Kim. Towards rigorous science of interpretable machine learning. arXiv preprint arXiv: 1702.08608 2017.
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. Advances in neural information processing systems, 32, 2019.
Zhengrui Guo, Conghao Xiong, Jiabo Ma, Qichen Sun, Lishuang Feng, Jinzhuo Wang, and Hao Chen. Focus: Knowledge-enhanced adaptive visual compression for few-shot whole slide image classification. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 15590-15600, 2025.
Frank E. Harrell, Robert M. Califf, David B. Pryor, K L Lee, and Robert A. Rosati. Eval- uating the yield of medical tests. JAMA, 247 18:2543-6, 1982. URL https://api.
Gemanticscholl org Corpus ID: 23344910
Chris Harshaw, Moran Feldman, Justin Ward, and Amin Karbasi. Submodular maximization beyond non-negativity: Guarantees, fast algorithms, and applications. In International Conference on Machhne Learning, pp. 2634 2643. PMLR, 2019.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficien transfer learning for nlp. In International conference machine learning, pp. 2790-2799. PMLR, 2019.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3 2022.
Lijie Hu, Songning Lai, Yuan Hua, Shu Yang, Jingfeng Zhang, and Di Wang. Stable vision concept transformers for medical diagnosis. arXiv preprint arXin:2506.05286 2025.
Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas Montine, and James Zou. A visual- language foundation model for pathology image analysis using medical twitter. Nature medicine, 290))230772316,, 2023.
Maximilian Ilse, Jakub Tomezak, and Max Welling. Attention-based deep multiple instance learn- ing. In International conference on machine learning, pp. 2127-2136. PMLR, 2018
Jared L Katzman, Uri Shaham, Alexander Cloninger, Jonathan Bates, Tingting Jiang, and Yuval Kluger. Deepsurv: personalized treatment recommender system using a cox proportional hazards deep neural network. BMC medical research methodology, 18(1):24 2018.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference machine learning, pp- $338-5348. PMLR, 2020.
Alexander Kraskov, Harald Stögbauer. and Peter Grassberger. Estimating mutual information. Phys- ical Review Statistical, Nonlinear, and Soft Matter Physics, 69(6):066138, 2004.
Ho Hin Lee, Alberto Lestammari--aang Jameson Merkov, Matthew Lungren, and Ivan Tarapov. Multi-modal mamba modeling for survival prediction (m4survive): Adapting joint foundation model representations. arXiv preprint arXiv:2503. 10057, 2025.
12

---

Published as a conference paper at ICLR 2026
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-effhcient prompt tuning. arXiv preprint arXiv:2104.0869) 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2001.0 1et.:10000009000 00000000000000 2021.
Hui Lin and Jeff A Bilmes. Learning mixtures of submodular shells with application document summarization, arXiv preprint 1rXivv/2/0.... 48771, 2012.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin-
guistic representations for vision-and- -nd--annua-gee tasks. Advances neural information processing systems, 32, 2019.
Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature biomedical engineering, 5(6))555-577) 2021.
Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Long Phi Le, Georg Gerber, et al. A visual-language foundation model for computational pathology. Nature Medicine, 30:863-8744 884,, 2024.
Jiabo Ma. Zhengrui Guo, Fengtao Zhou, Yihui Wang, Yingxue Xu, Jinbang Li, Fang Yan, Yu Cai, Zhengjie Zhu, Cheng Jin, et al. A generalizable pathology foundation model using a unified knowledge distillation pretraining framework. Nature Biomedical Engineering, pp. 1-20, 2025.
Roba Al Majzoub, Hashmat Malik, Muzammal Naseer, Zaigham Zaheer, Tariq Mahmood, Salman Khan, and Fahad Khan. How good is my histopathology lasnua-la foundation model? a holistic benchmark. arXiv preprint arXiv:2503. 122:9990..1299, 2025.
Meike Nauta, Ron Van Bree, and Christin Seifert. Neural prototype trees for interpretable fine- grained image recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14933-14944 2021.
Anh Tien Nguyen, Keunho Byeon, Kyungeun Kim, and Jin Tae Kwak. Vleer: Vision and language embeddings for explainable whole slide image representation. arXiv preprint 2rXiv:250..20850 2025.
Yuxiang Nie, Sunan He, Yequan Bie, Yihui Wang, Zhixuan Chen, Shu Yang, and Hao Chen. Con- ceptelip: Towards trustworthy medical ai via concept-enhanced contrastive langauge-image pre- training. arXiv preprint 1iX.::2501111579 2025.
Wei Peng, Kang Liu, Jianchen Hu, and Meng Zhang. Biomed-dpt: Dual modality prompt tuning for biomedical vision-language models. arXiv preprint htth:2555505888 2025.
Ying-Jia Qi, Guan-Hua Su, Chao You, Xu Zhang, Yi Xiao, Yi-Zhou Jiang, and Zhi-Ming Shao. Radiomics in breast cancer: Current advances and future directions. Cell Reports Medicine, 5(9), 2024.
Xuejun Qian, Jing Pei, Hui Zheng, Xinxin Xie, Lin Yan, Hao Zhang, Chunguang Han, Xiang Gao, Hanqi Zhang, Weiwei Zheng, et al. Prospective assessment of breast cancer risk from multi- modal multiview ultrasound images via clinically applicable deep learning. Nature biomedical engineering, 566):522 -32,2-53,,00 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763, PMLR, 2021.
Md Mamunur Rahaman, Ewan KA Millar, and Erik Mcijering. Leveraging vision-language embed- dings for zero-shot learning in histopathology images. IEEE Journal of Biomedical and Health Informatccs, 2025.
Brian C Ross. Mutual information between discrete and continuous data sets. PloS one, 9(2): e87357, 2014.
13

---

Published as a conference paper at ICLR 2026
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1(5):206-21) 2019.
Matthias Schmid, Hans A Kestler, and Sergej Potapov. On the validity of time-dependent auc esti- mators. Briefings in Bioinformatics, 16(1):153- 168, 2015.
Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Ad- vances neural information processing systems, 34:2136-2147, 2021.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le. Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gater mixture-of-exppeee layer. arXiv preprint arXiv: 706.6658 2017.
Jiangbo Shi, Chen Li, Tieliang Gong, Yefeng Zheng, and Huazhu Fu. Vila-mil: Dual-scale vision- language multiple instance learning for whole slide image classification. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11248-11258, 2024.
Sandra Steyaert, Marija Pizurica, Divya Nagaraj, Priya Khandelwal, Tina Hernandez-Boussard An- drew J Gentles, and Olivier Gevaert. Multimodal data fusion for cancer biomarker discovery with deep learning. Nature machine intelligence, 5(4):351-362, 2023.
Michal Tomaszewski and Robert J Gillies. The biological meaning of radiomic features. Radiol- ogy, 298(3):505-51)) 516, 2021.
Pengyu Wang, Huaqi Zhang, Meilu Zhu, Xi Jiang, Jing Qin, and Yixuan Yuan. Mgiml: Cancer grad- ing with incomplete radiology-patholle data via memory learning and gradient homogenization. IEEE Transactions on Medical Imaging, 43(6):2113-2124, 2024.
Jinxi Xiang, Xiyue Wang, Xiaoming Zhang, Yinghua Xi. Feyisope Eweje, Yijiang Chen, Yuchen Li, Colin Bergstrom, Matthew Gopaulchan, Ted Kim, et al. A vision-language foundation model for precision oncology. Nature, 638((851):76991))
2025.
Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang. Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier González, Yu Gu, et al. A whole-slide foundation model for digital 2024.
pathology from real-world data. Nature, 630(8015):881-15)
Yingxue Xu and Hao Chen. Multimodal optimal transport-based co-attention transformer with global structure consistency for survival prediction. In Proceedings of the IEEF/CVF interna- tional conference on computer vision, pp. 21241-21251, 2023.
Shin'y ya Yamaguchi, Kosuke Nishida, Daiki Chijiwa, and Yasutoshi Ida. Zero-shot concept bottle- neck models. arXiv preprint arXiv:2502.09000 2025.
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Pre- ston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedelip: multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.0091: 2023.
Yilan Zhang, Yingxue Xu, Jianqi Chen, Fengying Xie, and Hao Chen. Prototypical informa- tion bottlenecking and disentangling for multimodal cancer survival prediction. arXiv preprint arXiv:2401.01646 2024a.
Yunlong Zhang, Honglin Li, Yunxuan Sun, Sunyi Zheng, Chenglu Zhu, and Lin Yang. Attention- challenging multiple instance learning for whole slide image classification. European confer- ence on computer vision, pp. 125-143. Springer, 2024b.
Weigin Zhao, Ziyu Guo, Yinshuang Fan, Yuming Jiang, Maximus CF Yeung, and I Lequan Yu. Align- ing knowledge concepts to whole slide images for precise histopathology image analysis. npj Digital Medicine, 7(1):383, 2024.
Yuanshen Zhao, Jingxian Duan, Zhi- Zhi-Cheng Li, Ningli Chai, and Longsong Li. A radiopathomics model for prognosis prediction in patients with gastric cancer. In 2022 14th Biomedical Engi- neering International Conference (BMEACON), pp. 1-4. IEFE, 2022.
14

---

Published as a conference paper at ICLR 2026
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and patter recognition, pp. 16816-16825 2022.
Chunzheng Zhu, Yangfang Lin, Shen Chen, Yijun Wang, and Jianxin Lin. Medeyes: Learning dynamic visual focus for medical progressive diagnosis. arXiv preprint arXiv:2511.22018, 2025a.
Chunzheng Zhu, Yangfang Lin, Jialin Shao, Jianxin Lin, and Yijun Wang. Pathology-aware proto- type evolution via lim-driven semantic disambiguation for multicenter diabetic retinopathy diag- nosis. In Proceedings of the 33rd ACM International Conference Multimedia, pp. 9196-9205, 2025b.
15

---

Published as a conference paper at ICLR 2026
Appendix
A FRAMEWORK AND IMPLEMENTATION DETAILS
This section provides detailed breakdown of our framework's s implementation, including pseu- docode for core algorithms, expanded architectural descriptions, and specifics on baseline imple- mentations ensure full reproducibility.
A.1 ALGORITHM PSEUDOCODE
To further clarify the mechanics of our proposed method, we provide pseudocode for the two main stages of the CTF framework: the greedy concept selection process (Algorithm [), the end-to-end forward pass for a single patient (Algorithm 2 and the MoE layer to generate context prompts.
Algorithm Greedy Algorithm for Prognostic and Diverse Concept Selection
1: Input: Initial concept pool S, target concept set size k; training data images (æ;), survival outcomes {y,, 5.}.
2: Input: Frozen vision encoder V-)) and text encoder T(.).
3: Initialize: Final concept set + 61
4: Initialize: Relevance scores D <- {).
5: Pre-compute image features f. = (xx,) and concept embeddings t, = T(c) for all i,
6: {- Stage I: Relevance Ranking
7: for each concept e E S do
8: Compute alignment scores A {a(x,, fi
9: Discretize scores Ag to get Å.. Binarize survival outcomes to get Yoin-
10: Compute mutual information d(c) = [(Ac; Yba)).
11: Add (c, d(c)) to D.
12: end for
13: Sort concepts in S descending by their relevance score d(c). Let the sorted list be Swoted-
14: {- SSage 2: Diversity Maximization -}
Add the top-ranked concept from Ssorted C.
16: while |C| < k do
Initialize: "max_diverss diveessitsttt"" -- "ooccee"" null.
18: for each concept c' E Ssoned \ C do
Compute max similarity to current set: max sim max,ec (tor,t)) Diversity score 1 max _sim.
20:
21:
Diversity score > "ixx_iiversity" then "max diversity" Diversity score.
22:
23:
"next concepp" <-
24:
end if
25: end for
26: Add "next to C and remove from Swoted-
27:
end while
Output: Final selected concept set C.
A.2 ARCHITECTURAL AND TRAINING DETAILS
To ensure full reproducibility, we provide an expanded set of implementation details. All experi- ments were conducted on an HPC server equipped with NVIDIA L40s GPUs (48GB VRAM). See Table for detailed hyperparameters.
Image Preprocessing.
Pathology (WSIs): Whole-slide images processed at 20x magnification. We segmented the tissue foreground via Otsu's 's thresholding. The WSI was then tiled into non-overlapping 256 patches. Following the CLAM methodology (Lu et al. 2021). used contour filtering to remove
16

---

Published as a conference paper at ICLR 2026
Algorithm 2 CTF Forward Pass for a Single Patient
1: Input: Radiology image Tr, pathology image Th--
2: Input: Selected radiology concepts Grdd, pathology concepts Chist.
3: Models:
Frozen encoders
Vr, Tr, Thi
Learnable
modules:
Pc: Pi -9r 0s, (. 92 MLPpred
4: {- {- 1. Feature Extraction -
5: fr + V,(x),). fr +- (Z)).
6: 2. GCSP and Concept Tuning (Example for Radiology)
7: for each concept c, E Crad do
8: Global Prompt: Pa, + Look up learnable prompt for Cp.
9: Shared Prompt: fs 4 Øs Concat f1))) Ps.r + Ps. (fs).
10: Context Prompt: a + softmax (fa)). Pc. + E,
oz Phasis C,r,i
11: Combine: Pluned <- Concat Pa., Pc. Ps.r).
12: Tune: Get original tokens for c.. Prepend Pluned
13: ter TT (Tuned concept tokens for c+).
14: end for
15: Repeat symmetrically for each concept eh E Ctis to get tex.
16: { {-3. 3. Concept Scoring and Fusion- }}
17: Compute radiology scores: Sr,j = cosine_sim({ tor,) ). Let 8, [s,,1.., sr,k]-
Compute pathology scores: Sh.j cosine simneessimmpp fh. tes,) Let 8/. [sh,1 kh
19: 2 + Concat $)).
20: { 4. Prediction
21: 0 < (L).
Output: Task-specific output (e.g., risk score for survival analysis).
beckground/whiiespaaae patches before feature extraction with the CONCH vision encoder. A small attention pooling head is learned for feature aggregation.
Radiology (CT/MRI); For the in-house Centerl-GC and Center2-CHS datasets, tumor regions on CT/MRI scans datasets, we used the entire axial slice containing the largest tumor cross-section, as brain tumors typically occupy large and central portion of the image. All radiology images were resized to x244 normalized to [0, IJ. and then normalized using ImageNet statistics before being fed the BiomedCL.IP vision encoder.
manually segmented by an expert radiologist. For the public TCGA
Compute and memory footprint. We report peak GPU memory and full training time (batch size 1). Long denotes the sequence length (all tokens) and krads kipath the number of concepts per domain.
Table GPU memory usage and wall-clock time per epoch for CTF under different configurations. All results are measured with batch size 1. Ley denotes the sequence length (tokens) and kind kpah denote the number of concepts for radiology and pathology, respectively.
Configuration
GPU
(Lany Knd- Kpus)) (128, 256, 256)
Peak memory (GB)
Time (hour)
C-index
Default (recommended) 3090-friendly (A)
NVIDIA L40s 48GB
28.7
18.9
0.665
0.061
RTX 3090 24GB
(61, 256, 256)
19.2
11.6
0.642
0.053
3090-friendly (B)
RTX 3090 24GB
(128, 128, 256)
22.0
15.5
0.656 4 0.066
A.3 BASELINE IMPLEMENTATION DETAILS
For a fair comparison, all baselines were implemented using the same frozen vision encoders (BiomedCLIP, CONCH) as our CTF model to extract initial features.
Unimodal Models: For pathology-based unimodal models (ABMIL, CLAM, TransMIL, ACMIL), used the official publicly available codebases and adapted them the survival prediction task using a Cox loss final layer. For the Madiology-Oniy baseline, the single feature vector passed through MIP identical our prediction head.
17

---

Published as a conference paper at ICLR 2026
Algorithm 3 Noisy Top-k MoE for Context Prompt Generation
1: Input: Complementary-modaale
feature for mini-batch E BBBD, basis prompts (P:) Ix; gating parameters W gate: W noise E DRD, top-k value & (k I M); noise 0; training flag train; loss coefficient A.
2: Compute clean logits: Clean XW gate (shape (B M)}
3: if train and noisy gating enabled then
4: Std + softplus(( W moise) + {(B x M)}
5: Sample noise: ~ (0, Std?)
6: Logits < Clean A
7: else
8: Logits Clean
9: end if
Convert probabilities: P softmax(Logits, dim = [BBx M)}
11: Initialize G <- ORx
12: for b = to do
13: Select top-k indices and values from Pbb, :): (Sp, ub) + Topk k)
Normalize within top-k: at + 1b/ 200 2b 10 10-6) {as E R
15: Set sparse gates: Gbb, Sb] <
Compute context prompt: p(() Eies, ab Pi (P( RAXD]
17:
end for
Importance per expert: Imp < Σか G[b,:] ((M)
19:
if train and noisy gating enabled and k < M then
20:
Compute soft load (expected assignment count) via NoisyTopK (see Alg. Logits(b, :]. Std[b,]] :). P(b.S.) ((M)
4): Load f
ProbIn Clean [b,
else
22:
Hard load: Load + Σ [[ 0] {(M))
23: end if
24: Coefficient of variation squared: CV + Var(z)/(Mcan(( 10-10)
25: Auxiliary loss: Lmos + -CCCV2 + (Vo(()))))
26: Output: Sparse gates E RxM per-sample context prompts (P(() n 1m- auxiliary load- balancing loss Lmoe-
Algorithm 4 ProbInTopK for Noisy Gating (expected soft load)
1: Input: Clean logits E M noisy logits n E IR M noise std E R M (all for one sample),
top-k values topv E R&+1 from n (descending).
2: Let 7in &- topv [k] and Tout + lopv[k: - 1] (thresholds for in/out]
3: for to M do
4: if' r1]i]] > Tin then
5: prob |j] + ((cl]] - [i])
6: else
7:
prob [ + xx [cbi] Tomt)/ oosi)/
8:
end if
9:
end for
Output: Vector prob R with prob[j] = P(j E Top-k).
Simple Latent Fusion: Concat-Fusee involved concatenating fr and fa and feeding them to the prediction head. Cross-Attention used a standard transformer encoder layer where features from one modality formed the query and features from the other formed the key/value, followed by concatenation of the attended features.
SOTA Latent Fusion: For MOTCAT and PIBD, we re-implemented the core fusion mechanisms described in their respective papers, placing them between frozen feature extractors and the final prediction head. We performed hyperparameter search for key parameters, such as the number of attention heads for MOTCAT and the 3 coefficient for PIBD's bottleneck.
18

---

Published as a conference paper at ICLR 2026
Table 5: Hyperparameters for the CTF model components.
Parameter
Value
Description
Optimizer
AdamW
Learning Rate
Initial learning rate.
Weight Decay
Ic-6
Batch Size
I
Epochs
100
With early stopping (patience=ó on val-loss). For all prompt types (Globall Context, Shared).
Tinahle Prompt Length (E)
12
Context Prompt Pool Size (M)
16
Number of busis prompi vectors.
Gating Network 59h
2-layer ML12 128 16)
With ReL. activation.
Shared Prompt Generator (dg)
2-layer (1024 256 128)
ReL.U activation.
Prediction Head (MLPprol
2-layer MLP (512 + 128 #classes)
With ReLU and Dropour (p=0.1).
Submodular A
Balancing factors for concept selection.
SOTA Adaptive Fusion: For fair comparison with M4Survive, which uses Mamba-based adapters, we implemented a similar udapter-based strategy. We inserted lightweight Mamba blocks to process the features fr and fn before a fusion block, and fine-tuned only the adapter weights, keeping the vision backbones frozen.
A.4 ADAPTATION OF MOTCAT/PIBD TO RADIOLOCY-PATHOO.OO
Since MOTCAT and PIBD were originally proposed for genomics-patholog. We describe our adaptation details to radiology-patholon
МОТСAТ. The MOTCAT architecture features a unidirectional co-attention mechanism, where one modality serves "guidance" stream to refine the representation of the other. Since MOTCAT was originally designed for genomics guiding pathology, adapting it to our radiology-patholle setting required selecting a guidance direction. We evaluated both possible configurations: (1) Pathology- Sareey-guidssPaatie
quides-Radiology. and (2)
To determine the optimal configuration for baseline, conducted an empirical comparison both the TCGA-GBMLGG and CenterI-Ge datasets for the cancer grading task. The results are summarized in Table Our evaluation shows that using radiology guidance yields slightly better performance while being more computationally efficient, Hence, we selected the Radiology- guides- Pathology configuration the MOTCAT baseline for all experiments reported in the paper.
PIBD. We replace the (histology, genomics) pair with (pathology, radiology), project each to PIBD's shared width D, and pass the two embeddings to the original bottleneck/diseetaanging fusion. OB- jectives and downstream heads are unchanged.
Table 6: Comparison of MOTCAТ performance with different guidance directions on the cancer grading task. We report mean standard deviation for AUC over 10 stratified splits.
TCGA-GBMLGG
CenterI-GC
Guidance Direction
AUC (t)
Time (h)
AUC (t)
Time (h)
0.858 0.042
4.32
0.638 + 0.046
11.25
0.865 0.025
3.81
0.641 + 0.050
9.13
B DATASET DETAILS
We evaluate our framework on four datasets containing paired radiology and pathology images:
TCGA-LGG: cohort of 173 patients with Lower-Grade Glioma curated from The Can- cer Genome Atlas (TCGA For each patient, we obtained diagnostic whole-slide images (WSIs) and paired pre-operative, multi-parametric MRI seans (post-contrast T1-weighted and T2-FLAIR).
httpp::/ wwtp://........./t/oaaaaaaa
19

---

Published as a conference paper at ICLR 2026
Table Summary of patient cohorts used for experiments.
Survival Task
Grading Task
Dataset
Cancer Type
N
Censorship (%)
Description
Classes
TCGA-LGG
Lower-Grade Glioma
173
83.8%
TCGA-GBM
Glioblastoma
186
18.3%
WHO Tumor Grade
3
Centerl-Ce GGC
Gastric Cancer
683
57.2%
TNM Stage (T-Stage)
5
Center2-CHS
Chondrosarcoma
76
WHO Tumor Grade
5
Similarly, we curated a cohort of 186 patients with Glioblastoma Multiforme (GBM), the most aggressive primary brain tumor. The dataset consists of the same paired pre- operative MRI and WSI data types, matched with clinical survival outcomes from TCGA.
TCGA-GBM:
Center1-GC: An in-house dataset of 683 gastric cancer patients. For each patient, we have pre- operative CT scan and a post-resection WSI, acquired within one month of each other to ensure temporal consistency.
Conter-Chondrosarooma: An in-house dataset of 76 Chondrosarcoma patients with paired pre-operative MRI and WSI. Cohort characteristics are detailed in Table
B. ETHICAL CONSIDERATIONS AND DATA USAGE
All data from TCGA are publicly available and de-identified. The in-house datasets (Centerl-GG Center2-CHS)
collected under protocols approved by the local Institutional Review Board (IRB), with all patient data fully anonymized and de-identified prior to its use in this research. Informed consent was obtained from all participants included in the in-house studies. Our study strictly adheres to data privacy and protection regulations.
B.2 PAIRING WINDOW AND EXCLUSION CRITERIA
We require fully paired pattooogyy examples for all experiments; pairing is performed the patient level (no spatial registration), and only cases satisfying the following rules are retained.
In-house cohorts (Center1-GC, Center2-CHSS We pair the pre operoperavvve CT/MRI study with the diagnostic resection WSI from the same surgical episode. When multiple candidates exist, we select the imaging study and slide whose acquisition dates yield the minimum absolute time gap, and we require a pairing window of At < 30 days. If multiple WSIs are available, we use the slide annotated as "diagnostic" (primary tumor block).
TCGA cohorts (LGG/GBM). We pair MRIs and WSIs by TCGA case ID. When multiple scans/slides are available, we choose the pre-operet operravte MRI closest in time to the diagnostic his- tology slide for that case. Pairs with missing or ambiguous identifiers/metadataa are excluded.
C CONCEPT SELECTION AND DETAILED CONCEPT LISTS
Our framework's 's interpretability is founded on a set of high-quality medical concepts. As described in the main paper (Section 3.T). we use a submodular optimization approach to select a of con- cepts that are both prognostically relevant (high mutual information with survival outcome) and semantically diverse (low cosine similarity between embeddings). This approach aligns with recent trends in building more transparent models by grounding them in human-understandable concepts Yamaguchi et al. 2025).
While Concept Bottleneck Models (CBMs) Koh et al.) 2020) pioneered this direction, they can suffer from performance degradation and instability, limiting their use in high-stakes medical ap- plications (Hu et al. 2025). Our CTF framework avoids these pitfalls not by forcing information through a rigid bottleneck, but by using concepts as dynamic, tunable semantic bridge, as demon- strated in ablation studies.
20

---

Published as a conference paper at ICLR 2026
C.1 LLM PROMPT TEMPLATES AND GENERATION PIPELINE
We use Gemini-2.5-pre to generate concept pools. This appendix documents the exact prompts, generation settings, and post- post-processing used to build the candidate concept pools in Sec. 3.1 from large language models (LLMs).
Overview. For each disease and modality, we query the LLM once per progression stage in = early, intermediate, advanced, metastatic. Each query asks for N 250 short, atomic fea- tures as numbered list in the format "1.; 2.; ......" We then parse the enumeration, clean tokens, and save a ISON file keyed by stage: "early": [...], "intermedi a [...], "advanced": [...]r "metastatic": [...] cancer.(modal), json with modal in (rad, path).
Files are saved as
Prompt templates (exact text). We use the following two templates, differing only by modality. Bracketed fields are filled programmatically.
Template R (Radiology. MRI)
What are the radiological features in Magnetic Resonance images of [CANCER] at the [STAGE] stage of progression (differentiate early, intermediate, advanced, metastatic stages), please describe using keywords or short sentences. Give [N] features and answer the question with the following format: 1.; 2.; ...
Template (Histopathology. WSIs)
What are the morphological features in Whole Slide Images (WSIs) of [CANCER] at the [STAGE] stage of progression (differentiate early, intermediate, advanced, metastatic stages), please describe using keywords or short sentences. Give [N] features and answer the question with the following format: 1.; 2.;
Generation settings. Unless noted otherwise, use default sampling parameters of the LLM- Backend (no explicit temperature or top-p overrides). We issue one request per stage and modality- disease pair, with N=250. Radiology requests specify MRI/CT Pathology requests specify H&E WSIs.
Parsing and cleaning. The LLM returns a numbered list. We extract items using an enumeration- aware regex:
"ss+ As* (.+?) (?= \d+\ (AZ)
applied with multiline and dotall flags. We then:
trim whitespace and punctuation; remove leading bullets/asterisks: drop empty entries;
cap at N concepts per stage; preserve the original order;
save to cancer.(modal] ison as dictionary keyed by stage.
We subsequently merge the four stage lists, deduplicate (exact and fuzzy string match), and pass the merged pool the MI submodular selector (Sec. 3.1
Reproducibility and anonymity. API keys never embedded in the PDF repository. We use environment variables (e.g., GEMINT APT KEY, OPENAT API KEY).
Post-processing. We merge the four stage lists per (disease, modality). deduplicate by (i) ex- match after lowercasing and punctuation stripping and (ii) fuzzy string match, then supply the cleaned pool to the mutual-information ranking and submodular diversity selection described in I.I
21

---

Published as a conference paper at ICLR 2026
C.2 GASTRIC CANCER (CENTERI I-GC) CONCEPTS
Tables [ and 9 list the top 30 (out of 256) selected concepts for the CenterI-GC dataset for radiology and pathology, respectively.
Table 8: 30 selected Radiology concepts for the Centerl-Ge (Gastric Cancer) dataset.
Poorly definedlinegular tumor margins
11. Gastric outlet obstruction
21.
Venous encasemenniivvasion
2. Marked hetergenennus enhuncement
12.
Linitis plastica appearance
22. Infiltration of adjacent organs
3. Asymmetric or eccentric wall thickening
13. Effacemenn perigastric fat planes
23.
Peritomeal cancinomatosis
4. Presence of tumor ulceration
14. Definite serosal involvement
24.
Omental caking
5. Tumor necrosis necrotic evre
15. Tumor spiculation
25.
Distani metastasis ovaries
6. Large, matted regional lymph nodes
16. Air within the tumor
26. Enlarged Virchow's 's node
7. Lyuiphatic spread to regional nodes
17. Solid tumor component
27. Visible fooding vessels (ncovasuularity)
8. Presence of ascites
18. Distortion mucooal folds
28. Arterial encasement
9. Liver metastases with rim enhancement Invasion the muscularis progria
19. Moderate arterial phuse enhancement
29. Adrenal metastases
20. effect on adjacent structures
30.
Presence tumor calcifications
Table 9: 30 selected Pathology concepts for the Centerl -GC (Gastric Cancer) dataset.
1. Loss of glandular architecture
11. High mitotic activity
21. Tumor-infitrating lymphocyyes present
2. Solid growth puttern
12. Areas of tumor necrosis
22. Tumor-associater macrophages present
3. Presence of signct ring cells
13. Incressed nuclear-yytolasse ratio
23. conttelllllmmccneehyyl transition
4. High-grade cellular atypia
14. Prominent ireeular nuclcoli
24. High level of Microsatellite Instability
5. Lymphuvascular invasion (LVT)
15. Discohesive cells presence
25. Increased HER2 expression
6. Perineural invasion
16. Spindle cell morphology
26. of E-cahherin expression
7. High tumor budding
17. Desmoolasiic reaction
27. Increased Ki-67 proliferation index
8. High tumor-stroma ratio
18.
mucin pools
28. Abnormal
morphology
9. Poorly differentiated features
19. Tumar cell upoptosis
29. Increased Cyclin DI expression
10. Disruption of busement membrune
20. Cribriform growth pattern
30. Multinucletionn giant cells present
C.3 BRAIN TUMOR (TCGA-LGG/GBM) CONCEPTS
To demonstrate the adaptability of our concept selection strategy. Tables 10 and TT list the top 30 selected concepts for the TCGA glioma cohorts. These concepts are distinct from those for gastric cancer and reflect the specific pathology of brain tumors.
Table 10: 30 selected Radiology concepts for the TCGA-LGG/GBM datasets, curated from the provided list.
1. Central necrosis
11. Suhependymal spread
21. Obstrietive hydrocephalus
2. Cystic components within the turor
12. New satellite lesions
22. Uncal hemiaaiin
3. Patchy enbancement
13, Multifocal disease
23. Compression of the brainstem
4. IlI-defined or infiltrative margins
14. Dissemination through CSF
24. Superficial cortical involvement
5. Incrensed tamor heterogeneity
15. Brenkdown of the blood-brain barrier
25. Perfusion ahnormalities.
6. Mass effect adjacent structures
16. New areas restricted diffusion (DWT)
26. Positive amino PET
7. Compression of ventricles
17. Lobulated
27. Small area of new enhancemeni
8. Midline shift
18. Elevated cholinc/craaiin ratio (MRS)
28. Incressed periiesional edema
9. Internal septations within cystic areas
19. Decreased NAA (N-acetyl aspartate)
29. Sublle now vascularity
10. Deep gray matter stractures
20. Loss of gray-whiie white matter differentia-
30. Infiltration of tentorium
liom
Table 11: Top 30 selected Pathology concepts for the TCGA-LGG/GBM datasets. These concepts were selected from a larger pool to represent the core histopathological, molecular, and microenvi- ronmental features of high-grade gliomas like Glioblastoma.
1. Pscudopalisading nocrosis
11. Perineuronal satellitosis
21. ATRX loss
2. Microvascular aur proliferation
12. BGFR amplification
22. Lack of 1p/19g co-deletion
3. High-grade cellular atypia
13. TERT promoter mutation
23. Ilypoxia
4. High mitotic activity
14. MGMT promoter methy status
24. Tumee associated macrophages
5. Increased cellularity
15. IDHI mutation status
M2-polarized macrophages
6. Nuclear ploomorphism
16. p53 murations
26. T-cell exhaustion
7. Presence of multimulleaed giant cells
17. PTEN loes
27. PD.LI expression
8. Glomerukidd bodies
18. Chromnosomal gains
28. Aberrant GFAP expression
9. Prominent nuclcoli
19. Chromosonaal losses
29. More expressoon of stem eell markers
10. Diffuse infiltration brain parenchyma
20. Nuclear hyperchromasin
30. Genomic instability
22

---

Published as a conference paper at ICLR 2026
D ADDITIONAL QUALITATIVE AND INTERPRETABILITY ANALYSIS
D.1 PATHOLOGY ATTENTION HEATMAPS
Figure presents a compelling comparison attention maps from our model's pathology stream for representative patients from the Centerl-GC cohort. The heatmaps visualize attention weights from the feature aggregation module, where red indicates regions receiving the highest attention. In the high-risk patient (b), the model correctly localizes its attention dense, disorganized clusters of tumor cells characteristic of poorly differentiated carcinoma. Conversely, in the low-risk patient (a), the attention sparse, indicating the absence of these aggressive features.
(a) Low Risk
(b) High Risk
0088
0.66
0.4
-02
Figure 7: Pathology Attention Heatmaps for High- and Low-Rish Patients. The figure contrasts the model's 's spatial attention for (a) low-risk patients and (b) high-risk patients from the Centerl cohort.
D.2 DEPENDEPENDENT AUC ANALYSIS
The Concordance Index (C-index) provides a single, global measure of model's rank-based dis- criminatory ability. However, in survival analysis, model's predictive accuracy can vary over time. The dime-dependent Area Under the Curve (td-AUC) offers a more granular evaluation by assessing the model's ability distinguish patients who will experience an event before a specific time point t from those who will not (Schmid et al. 2015).
As shown in Figure CTF demonstrates not just superior but strikingly dominant performance over the PIBD baseline on the Center1-GC. dataset. CTF maintains a high td-AUC (consistently 0.75) across the entire follow-up period, achieving an excellent mean AUC of 0.808. stark contrast, the PIBD baseline performs at level comparable random chance, with a mean AUC of 0.483. This significant and persistent performance gap, highlighted by the shaded green area, indicates that while the latent-space fusion model fails to maintain discriminative power over time, CTF's dynamic, concept-bard based fusion provides robust and reliable prognostic predictions for both near-term and longer-term outcomes.
23

---

Published as a conference paper at ICLR 2026
10
08
08
6.7
6.
CTF (Cun) AMeanAU (1000))
PIED (Basotin)) (Mean AUG 0.83
Performanc
6.40
(Monns)
Figure 8: Time-dependent AUC curves on the Centerl-GC dataset comparing CTF the PIBD baseline. CTF (blue) consistently and significantly outperforms the baseline across all time horizons, maintaining robust prognostic accuracy. The shaded green area highlights the large performance gain.
Table 12: Concept intervention analysis on predicted risk scores. For two representative patients from the Centerl-GC dataset, we intervene on the highest-scoring concepts by clamping their score 0.0 and observing the change in the final predicted risk. The percentage change (A Risk) high- lights the causal influence of each concept. C' and Rt represent original concept scores and new risk scores.
Intervened Concept
Modality
C
R,
A Risk (%)
Patient A: High-Risk (Actual Outcome: Event at 25 months)
Initial Predicted Risk: 3.91
Bad blood vessel morphology Lymphovascular invasion Disseminated tumor cells
Radiology 3.35
2.98
-23.8%
Pathology
2.80
3.02
-22.7%
Pathology
2.00
3.22
-17.6%
Patient B: Low-Risk (Actual Outcome: Censored at 54 months)
Initial Predicted Risk: -2.57
Well diffferentiated
Pathology
1.65
-1.94
+24.5%
Circumscribed tumor margins
Radiology
1.20
-2.18
+15.2%
Low sumor-stroma ratio
Pathology
0.85
-2.37
+7.9%
D.3 CONCEPT INTERVENTION
Concept Intervention. To test if the learned concepts have causal impact predictions, per- formed concept intervention experiments by neutralizing oigh-impact concept scores for represen- tative patients. As shown in Table [2] intervening concepts like "Bad blood vessel morphology" for a high-risk patient or "Well-differentiated" for a low-risk patient resulted in significant and clin- ically plausible shifts in the final risk score (a 23.8% decrease and 24.5% increase, respectively). This provides strong evidence that CTF's predictions are causally linked to interpretable concepts.
D.4 STATISTICAL SIGNIFICANCE TESTS
For each dataset and metric, we compare CTF against the strongest competing baseline using a paired t-tests across the 10 stratified splits. We report raw p-values in Table 13 below.
24

---

Published as a conference paper at ICLR 2026
Table 13: Paired one-sided t-test (over 10 stratified splits) comparing CTF with the strongest com- peting multimodal baseline on each dataset and task. Reported are p-values for the null hypothesis that there is no difference in mean performance between CTF and the baseline.
Tisk
Dataset
Metric
CTF
Best buseline
p-value
ICGA-LGG
C-index
0.713 + 0.103
M4Survive (U.709 0.112)
0.13
Survival
TCGA-GBM
C-index
0.579
0.063
MOTCAT (0.563 0.108)
0.09
Centerl-GC
C-index
0.665 = 0.061
M4Survive (0.642 + 0.065)
0.05
TCGA-GBMLGG
AUC
0.903 +: 0.028
Cruss-Attention (0.868 + 0.030)
0.02
Grading (AUC)
Center2-CHS
AUC
0.854 + 0.081
M4Survive (0.830 0.075)
0.06
Centerl-GC
AUC
0.660 0.049
M4Survive (0.649 0.052)
0.05
ICGA-GBMLGG
ACC
0.718 : 0.063
M4Survive (0.691 0.061)
0.05
Grading (ACC)
Center2-CHS
ACC
0.698 + 0.164
M4Survive (0.626 + 0.115)
0.02
Centerl-Ge
ACC
0.401 + 0.057
Cross-Atiention (0.394 + 0.049)
0.06
E BROADER IMPACT AND LIMITATIONS
E. POTENTIAL. FOR POSITIVE IMPACT
The successful development and deployment of the CTF framework could have a significant positive impact on clinical oncology and computational medicine.
1. Improved Prognostic Accuracy:
By creating deeper synergy between radiology and pathology, CTF can provide more accurate and reliable predictions of patient outcomes. This could help clinicians better stratify patients for treatment, identifying high-risk indi- viduals who may benefit from more aggressive therapies and low-risk individuals for whom de-escalation could be considered.
2. Enhanced Clinical Decision Support: The interpretable nature of CTF is a key advantage. By presenting predictions alongside the contributing medical concepts (c.g.. "High score for "Lymphovascullr invasion""). the model can serve as "second-read" tool that not only provides a risk score but also highlights the key evidence, facilitating a more informed dialogue between the AI and the clinician.
3. Accelerated Scientific Discovery: The cross-modal influence analysis (Figure can un- cover novel or subtle correlations between macroscopic imaging features and microscopic cellular patterns. This could generate new hypotheses for translational research into the biological drivers of cancer aggression.
E.2 LIMITATIONS AND FUTURE WORK
Despite its promising results, work has several limitations that open avenues for future research. A primary limitation the framework's is dependence on the quality and comprehensiveness of the initial concept vocabulary. The performance of CTF is fundamentally tied the concepts provided, and while we used LLMs to generate broad list, this process may miss crucial niche concepts or introduce biases. Future work should therefore explore more robust, data-driven methods for concept discovery or involve domain experts in formal human-in-the-loop process to refine and validate the concept library.
Furthermore, our framework leverages powerful, pre-trained Vision Vision-Language Models, inheriting both their extensive knowledge and their potential biases. The model's performance is ultimately capped by the quality of these foundational backbones. While our ablation study confirmed that expert models perform best, valuable future direction would be to investigate methods for jointly fine-tuning the prompt modules and small fraction of the backbone model's weights to achieve even better task-specific adaptation. This must be balanced against computational costs, as the forward pass for CTF, though parameter-efficiet in its training, remains resource- iittencveetenivve due to the multiple large models and the WSI feature extraction bottleneck. Further optimization would be required for real-time clinical deployment.
Perhaps the most significant barrier to immediate clinical translation is the model's requirement for fully-paired data-one one radiology and one pathology image per patient-during training. This
25

---

Published as a conference paper at ICLR 2026
considerable constraint, real-world clinical datasets often incomplete. Extending CTF to gracefully handle missing modalities is therefore a critical next step. Future iterations could investigate learning impute concept scores from the available modality using dynamic attention mechanisms to operate effectively even with an incomplete data stream.
F STATEMENT ON THE USE OF LARGE LANGUAGE MODELS (LLMs)
We disclose the use of Large Language Models (LLMs) this work. LLMs played a role in the following ways:
1. Concept Generation: As detailed in Section 3 and Appendix C.I we utilized an LLM to generate an initial broad pool candidate radiological and pathological concepts. This served as a starting point for prognostic and diversity-based concept selection algo- rithm.
2. Language Polishing: LLMs were used as a general-purpont writing assistant improve the clarity, grammar, and style of the manuscript.
26


Reviews:
[{"review_id": "TyTNoRRVPl", "reviewer": "Reviewer_q1cP", "summary": "This paper proposed a parameter-efficient framework that uses clinically grounded concepts to bridge radiology and pathology, where a set of learnable tokens is employed to learn modality-specific and common knowledge, as well as cross-modal interactions.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed method is well-motivated, and the idea is new but technically complex.\n2. The experiments are comprehensive, including comparisons to SOTA methods, thorough ablation studies and intuitive visualization.\n3. The proposed method outperforms SOTA methods.", "weaknesses": "1. The pool of basic concepts is static by asking LLM to obtain relevant concepts, which may limit the performance and new concept discovery. It would be better if there is a mechanism to update the pool as the model is optimized. \n2. The scalability of LLM can be investigated to see if the performance can be further improved with a stronger LLM.\n3. Since MOTCAT and PIBD are designed for histology-genomics data, how were they applied to the datasets used in this work? Specifically, MOTCAT is unidirectional, which utilizes genomic data to guide the learning of pathological modeling. What about CTF? More details about experimental settings should be uncovered.\n4. Given that the concepts would be updated via a learnable prefix, can the learned concepts be decoded via an LLM decoder? This can facilitate new concept discovery. \n5. The figure of GCSP can be improved. Currently, it is challenging to understand how the embedding of each modality interacts with the mentioned three types of prompts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H7KFGnrPo4", "reviewer": "Reviewer_c1KR", "summary": "This paper proposes a model called Concept Tuning and Fusing (CTF), a parameter-efficient framework that uses clinically grounded concepts to bridge radiology and pathology foundation models (FMs) for survival prediction and cancer grading. Instead of performing multimodal fusion using static latent features, this paper proposes to use concepts as a shared semantic interface and dynamically co-adapt the concept semantics. The framework contains i) concept generation and selection by querying LLMs, and ii) global-context-shared prompt tuning. Empirical evaluations are done using the public TCGA dataset and a private dataset on gastric cancer.", "questions": "- In Section 3.3, after generating $P_{tuned}$, a set of tuned textual embeddings are obtained from the text encoder. Is the text encoder the frozen ones as shown in Fig. 2? How is $P_{tuned}$ used in this text encoder?\n- Please also refer to the weaknesses section as above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The use of clinically grounded concepts as a bridge for multimodal co-adaptation is original. It provides enhanced interpretability and improved performance. The idea of the global-context-shared prompt (GCSP) conditions the interpretation of findings from one modality on another, which looks quite novel and interesting to me.\n- The paper articulates a concept selection strategy that balance prognostic relevance and semantic diversity.\n- Overall, the empirical evaluation is strong, showing consistent improvements across datasets and tasks.\n- The ablation study and hyperparameter sensitivity analysis are performed to allow a better understanding of the architectural design.\n- By and large, the paper is organized and well-written, the logic is smooth and the paper is easy to follow.", "weaknesses": "- One major issue I am not quite following is the motivation of the GCSP strategy. In Section 3.2, it is argued that the prognostic *importance* of findings in radiology may be amplified by a specific histological finding. This implies that the basis of performing GCSP is that correct and good concepts (findings) are identified from each modality. In this case, a naive solution would be to feed all concepts from different modalities to a neural network and let the network to figure out the synergy between them. The GCSP strategy, in contrast, seems to solve this problem with a much more complex design. I am not sure why the naive way would fail and why GCSP could be a better solution than a naive one.\n- As reported in Table 1, M4Survive and CTF both have large standard deviation values. To ensure the improvement does not purely arise from noise, statistical tests should be performed to examine the significance of the improvements. \n- Other minor issues:\n\t- For inline citations, `\\citep{}` should be used instead of `\\citet{}`.\n\t- Some notations are not consistent throughout the paper, for example the prompt is noted by $P^{\\text{tuned}}$ in Section 3.2 and $P_{tuned}$ in Section 3.3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ApR8teSP7l", "reviewer": "Reviewer_LEUR", "summary": "This paper presents CTF (Concept Tuning and Fusing), a framework that bridges radiology and pathology foundation models using clinically meaningful concepts as an interpretable interface. Instead of fusing static features, CTF employs a Global-Context-Shared Prompt mechanism that dynamically adapts concept interpretations based on cross-domain information, allowing each modality to inform the other's concept understanding. Tested on survival prediction and cancer grading across multiple datasets, CTF achieves state-of-the-art performance while training only 0.15% additional parameters.", "questions": "1.\tCould the authors elaborate more on the registration of WSIs and radiology data, such as CT scans? How to make sure the data are paired since they are in different macro- and micro-scales?\n\n2.\tFor the computation of the alignment score for concepts, why is it not normalized or scaled? Meanwhile, it’s then discretized into a variable, but how? Could the authors elaborate more, or specify where the reader can find related details in the Appendix? Moreover, since here the model requires the label to compute the mutual information, what’s the strategy during inference where there is no patient label?\n\n3.\tIn Section 3.3, the authors mentioned that the concept score vectors “provide an interpretable representation of the patient’s condition”. Could the authors elaborate more on this conclusion? \n\n4.\tCTF trains in an end-to-end manner. Although most of the parameters of foundation models are frozen, this still introduces concerns about efficiency. Could the authors provide the least requirement to train CTF compared to uni-modal or late fusion baselines (which, as far as I know, can be trained in one GTX 3090 card)? Also, how many GPUs are used for the actual model training? (Appendix only mentions types of GPU)\n\n5.\tSelection strategy for pathology VLMs. It seems CONCH contributes largely to the performance of CTF from Table 3. But this ablation study only compares CONCH with the general-purpose CLIP.  What about other pathology VLMs such as PLIP [1] and MUSK [2]?\n\n$\\quad$ [1] Huang, Zhi, et al. \"A visual–language foundation model for pathology image analysis using medical twitter.\" Nature medicine 29.9 (2023): 2307-2316.\n\n$\\quad$ [2] Xiang, Jinxi, et al. \"A vision–language foundation model for precision oncology.\" Nature 638.8051 (2025): 769-778.\n\n6.\tCould the author please add these papers in the Introduction or Related Work sections since they are relevant:\n\n$\\quad$ [1] (Pathology Foundation Model) Chen, Richard J., et al. \"Towards a general-purpose foundation model for computational pathology.\" Nature medicine 30.3 (2024): 850-862.\n\n$\\quad$ [2] (Pathology Foundation Model) Ma, Jiabo, et al. \"A generalizable pathology foundation model using a unified knowledge distillation pretraining framework.\" Nature Biomedical Engineering (2025): 1-20.\n\n$\\quad$ [3] (Pathology Foundation Model) Xu, Hanwen, et al. \"A whole-slide foundation model for digital pathology from real-world data.\" Nature 630.8015 (2024): 181-188.\n\n$\\quad$ [4] (Foundation Model Adaptation / Prompt Generation and Selection) Guo, Zhengrui, et al. \"Focus: Knowledge-enhanced adaptive visual compression for few-shot whole slide image classification.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n$\\quad$ [5] (Foundation Model Adaptation / Parameter-efficient learning) Che, Haoxuan, et al. \"Llm-driven medical report generation via communication-efficient heterogeneous federated learning.\" IEEE Transactions on Medical Imaging (2025).\n\n**I will definitely consider raising my score if authors solve my concerns above.**", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tNovel methodology: The Global-Context-Shared Prompt (GCSP) strategy dynamically conditions concept interpretations in one modality based on features from the complementary modality, enabling deeper synergy than traditional static feature fusion approaches.\n\n2.\tParameter efficiency with strong performance: CTF achieves state-of-the-art results while requiring only 0.15% additional trainable parameters by keeping foundation models frozen and learning only lightweight prompt modules.", "weaknesses": "1.\tThe citations in this paper are **all incorrect**. Please correctly use the \\citet and \\citep LaTeX commands to distinguish proper references for different scenarios.\n\n2.\tBaselines such as PIBD, MOTCAT are originally developed for the fusion of pathological and genomic data. The comparison might be unfair for pathology and radiology data as inputs.\n\n3.\tTask “Center-1 GC” in Table 2 shows relatively low performance (all models) in AUC and especially ACC, suggesting this is not a proper task to choose for model evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X0cndTRuf1", "reviewer": "Reviewer_N7n7", "summary": "The authors proposed a feature alignment scheme for bridging the radiology and histopathology image foundation models. A list of concepts is first selected and then used as the linkage feature between radiology and histopathology image features, with separate adaptors trained. The performance evaluation is conducted on two downstream tasks, i.e., survival analysis and tumor grading. Three datasets (two public and one private) are adopted in the experiments, and only marginally improved results of the proposed method are reported in comparison to previous multi-modal feature fusion methods, e.g., 1.6-2.6% increase in C-index for survival analysis. The manuscript is not easy to follow overall. Many details seem to be missing from the introduction, along with several flaws detailed below.", "questions": "see weakness", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ The paper proposed a multi-modal alignment scheme that required light tuning efforts, which could be easily adopted in the downstream tasks.\n+ The downstream in the evaluation is clinically relevant.", "weaknesses": "- There are many details of the proposed methods missing. 1. It is not clear how the 3D volumes of radiology images and WSI\n Images are encoded. It could largely influence the semantics of the image embedding, especially for the alignment task. 2. Why is the context prompt only computed for the histopathology images? 3. Details of the MoE layer? 4. Is P_tuned computed for all the concepts in both sets? 5. How are the images associated with the concepts? 6. Are there any other inputs other than f_r and f_h?\n- The overall lack of clarity in the introduction makes the paper a bit hard to follow.\n- Radiologists and histopathologists speak quite different languages, which is also reflected by the selected concepts shown in the appendix. How often can these concepts be aligned, and it will be helpful to show what percentage of these concepts are matched. \n- The proposed method utilized the global features of the images, while the concepts are composed in more detail. How will this mismatch of the semantic feature levels affect the performance? How different could the features be after the adoption in comparison to the original image feature? It is also related to the following point.\n- The overall performance gain after adopting the proposed method is marginal, e.g.,1.6-2.6% increase in C-index for survival analysis in comparison to the recent baseline. Is the performance gain sourced from the additionally concept injected? What will be the performance using solely the two image features and textual features (e.g., simple concatenation)\n-", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)