Multi-Modal Foundation Models for Computational Pathology: A Survey
Authors: Dong Li
Guihong Wan
Xintao Wu
Xinyu Wu
Xiaohui Chen
Yi He
Zhong Chen
Peter K Sorger
Chen Zhao
Venue: Accepted by TMLR ()

Abstract: Foundation models have emerged as a powerful paradigm in computational pathology (CPath), enabling scalable and generalizable analysis of histopathological images. While early developments centered on uni-modal models trained solely on visual data, recent advances have highlighted the promise of multi-modal foundation models that integrate heterogeneous data sources such as textual reports, structured domain knowledge, and molecular profiles. In this survey, we provide a comprehensive and up-to-date review of multi-modal foundation models in CPath, with a particular focus on models built upon hematoxylin and eosin (H&E) stained whole slide images (WSIs) and tile-level representations. We categorize 34 state-of-the-art multi-modal foundation models into three major paradigms: vision-language, vision-knowledge graph, and vision-gene expression. We further divide vision-language models into non-LLM-based and LLM-based approaches. Additionally, we analyze 30 available multi-modal datasets tailored for pathology, grouped into image-text pairs, instruction datasets, and image-other modality pairs. Our survey also presents a taxonomy of downstream tasks, highlights training and evaluation strategies, and identifies key challenges and future directions. We aim for this survey to serve as a valuable resource for researchers and practitioners working at the intersection of pathology and AI.

Guihong Wan' Published in Transactions on Machine Learning Research (11/2025)
Multi-Modal Foundation Models for Computational Pathology: A Survey
Dong Li*
dong lit @baylor edu
Deportment of Computer Science, Baylor University
gwwn.hmngh.harvord..de
Department of Dermatology, Massachuseits General Hospital, Harvord Medical School
Xintao Wu
zintaoun@uuork.cmmm edu
Electrical Engineering and Computer Science Deportment, University of Arkansas
Xinyu Wu Department of Computer Seience, Baylor University
vingu wul @baylor edu
Xiaohui Chen
riaohui chen I @baylor edu
Department of Computer Science, Baylor University
Yi He
@wm edu
Department Data Science, The College William and Mary
Zhong Chen
zhong. chen Bsiu. edu
School of Computing, Southern Illinois University
Peter K. Sorger
peter peter_ wwwger@hhm..harvrd.cco edu
Department of Systems Biology, Harvard Medical School
Chen Zhaot
chen zhao Thhyoobaylor.om edu
Department of Computer Science, Baylor University
Reviewed on OpenReview: https: // openreview. net/ forum? id- N27GSH9922
Abstract
Foundation models have emerged as powerful paradigm in computational pathology (CPath), enabling scalable and generalizable analysis of histopathoolgical images. While early developments centered on uni-modal models trained solely on visual data, recent ad- vances have highlighted the promise of multi-modal foundation models that integrate hetero- gencous data sources such textual reports, structured domain knowledge, and molecular profiles. In this survey, provide comprehensive and up-to-date review of multi-modaa foundation models in CPath, with particular focus on models built upon hematoxylin and eosin (H&E) stained whole slide images (WSIs) and tile-level representations. We categorize 34 state-of-the-art multi-modal foundation models into three major paradigms: vision-language, vision-knowledge graph, and vision-gene expression. We further divide vision-language models into non-LLMM-bascl and LLM-basod approaches. Additionally, we analyze 30 available multi-modal datasets tailored for pathology, grouped into image-text pairs, instruction datasets, and image-other modality pairs. Our survey also presents a tax- onomy of downstream tasks, highlights training and evaluation strategies, and identifies key challenges and future directions. We aim for this survey to serve as valuable resource for researchers and practitioners working at the intersection of pathology and AT.
"Equal Contribution "Correpponding Author

---

Published in Transactions on Machine Learning Research (11/2025)
1 Introduction
The advent of foundation models has significantly transformed computational pathology (CPath) by enabling scalable and generalizable deep learning solutions for analyzing histopathological images. These models are designed to extract meaningful patterns from vast collections of pathological data, enhancing diagnostic accuracy, prognostic assessments, and biomarker discovery (Ochi et al., 2025). Among various imaging modalitics, hematoxylin and cosin (H&E) stained images remain the most widely used in CPath due to their accessibility and effectiveness in capturing morphological details of tissues (Chanda et al., 2024; Guan et al., 2025). Whole Slide Images (WSIs), obtained from high-resolution scanning of tissue samples, offer compre- hensive histopathological insights but are computationally demanding duc to their large size. To manage this, WSIs are typically divided into smaller tile images, which serve as the fundamental units for training deep learning models (Wang et al., 2024; Ding et al., 2024; Chen et al., 2024d). Existing foundation models for CPath (FM4CPath) can be broadly classified into uni-modal and multi-modal paradigms (Li et al., 2025a), with the former primarily focusing on visual representation learning and the latter integrating additional modalities such text, knowledge graphs, and gene expression profiles for enhanced interpretability and performance.
Early research in CPath predominantly leveraged uni-modal foundation models (Wang et al., 2022b; Chen et al., 2024c; Vorontsov et al., 2023), where deep learning models were trained solely on histopathological images. These uni-modal models have led to significant advancements in classification, segmentation, and prognostic prediction lasks by learning rich visual features from pathology slides. However, despite their success, these models are inherently limited by their exclusive reliance on image data, which often lacks crucial contextual information present in pathology reports, structured knowledge, or molccular profiles. To overcome these limitations, recent efforts have shifted toward multi-modal foundation models (Lu et al., 2024a; Wang et al., 2024; Lu et al., 2024b), which integrate heterogeneous data sources to provide more robust and interpretable insights.
Existing multi-modal foundation models for CPath (MMFM4CPath) can be categorized into three primary paradigms: vision-language, vision-knowledge graph, and vision-gene expression models. A roadmap of up-to-date MMFM4CPath is shown in Figure 1. Vision-language models (Huang et al., 2023; Ikezogwo et al., 2024; Sun et al., 2024e;b) utilize textual annotations, such as WSI reports and tile-level captions, to enrich visual representations, facilitating zero-shot learning and seamless cross-modal integration between images and text. Within this category, models can be further divided into non-LLMMbaasdd and LLM-based approaches, with the latter incorporating large language models (LLMs) for improved natural language understanding and generative capabilities. Vision knowlege graph models (Zhou et al., 2024b;a) integrate structured domain knowledge by leveraging pathology-speciftt ontologies and knowledge graph to guide deep learning models. Vision-gene gene expression models (Xu et al., 2024; Vaidya al., 2025) align visual features with molecular-level insights from RNA sequencing and other omics data, facilitating genolype-phenolype associations for precision medicine.
While existing surveys have explored FM4CPath (Ochi et al., 2025; Chanda et al., 2024; Guan et al., 2025; Bilal et al., 2025; Li et al., 2025a), they often lack a comprehensive analysis tailored to multi-modal approaches. As shown in Table 1, our survey differentiates itself by systematically categorizing 34 of the most up-to-date MMFM4CPath and analyzing 30 available multi-modal datasets for pathology, with an emphasis on modalities beyond lanion-language integration. Additionally, we provide an in-depth discussion evaluation methodologies, training strategies, and emerging challenges in this field. For transparency, we detail our literature search strategy, including databases, search terms, time range, and inclusion /exclusion criteria, in Appendix A. The key contributions of this survey include:
Comprehensive and Up-to-Date Survey. This survey systematically reviews 34 multi- founda- tion models in computational pathology across language, vision-knowledge graph, and rision-gene expression paradigms. It offers detailed comparisons of their architectures, pretraining strategies, and adaptation techniques, providing a broader and more current coverage than prior surveys.
In-Depth Analysis of Pathology-Speeific Multi-Modal Datascts. This survey curates and catego- rizes 30 available datasets into three types: image-text pairs, multi-modal instructions, and image-other
2

---

Published in Transactions on Machine Learning Research (11/2025)
PLoBited
Nitur Necicel
Oiaaro Medchhl
FCLR)
DeartP
IMARI
(ECCA)
PREM
Pururel
TITAN
VILMMPPUK
GLOVER
10/P%)
(DGOM)
Pattsen-Lath
LLMisead
On
Pilture Conmunictiooe
Pehhnsigh
WS--LIWA
KEP (LOCH)
KoinsKoondeie Gragh
SAMGUE
Figure 1: roadmap of multi-mode foundation models for computational pathology (MMFM4CPath),
Table 1: Comparison between our survey and related surveys.
MMPMGCPPlh
Visdon- Language
Viion- Knowlogge Graph Vilsa-Gou Gome Exprcsson
Total
Imugo- Toxt Puir
Inscuction
Imago Image-OOlacs Otler Modalil
Toial
Octa ol, (Oeai 2022))
5.
Charts at al (Chendi et al mon))
10
X
14
Guss et (Gias sk, 2385)
11
14
mit sl., p02i))
aryy
Staney
modality pairs. We emphasize how these datasets enable various training strategies and highlight their roles in aligning modalities and supporting instruction tuning.
Thorough Overview of Multi-Modal Evaluation Tasks. A taxonomy of evaluation tasks is pro- vided, covering six major categories including classification, retrieval, generation, segmentation, predic- tion, and visual question answering. We detail how different MMFM4CPath are evaluated under various settings.
Future Research Opportunities. We outline three promising directions, such integrating H&E images with spatial omics data for deeper biological insight, leveraging H&E to predict MxIF markers for cost-effective virtual staining, and establishing standardized benchmarks to ensure consistent evalu- ation across tasks and datasets. These directions aim to enhance the clinical relevance, scalability, and comparability of future models.
2
Background
2.1 Computational Pathology
Computational Pathology (CPath) is an interdisciplinary field that applies computational techniques, in- cluding machine learning and computer vision, analyze and interpret pathological data. By leveraging digital pathology, CPath enhances diagnostic accuracy, facilitates large-scale biomarker discovery, and sup- ports personalized medicine. Among the various imaging modalities in pathology, Hematoxylin and Eosin (H&E) stained images serve the most commonly used vehicle for studying CPath. These images capture essential morphological charactcristies of tissues, making them fundamental for histopathological analysis. Within the realm of digital pathology, Whole Slide Images (WSIs) and tile images are two primary forms of data representation. WSIs, generated from high-resolution scanning of entire tissue slides, provide compre- hensive visual information gigapixel scale, allowing pathologists examine cellular structures in detail. However, due their enormous size and high computational demands, WSIs pose significant challenges terms of storage, processing, and analysis. To mitigate these challenges, WSIs often divided into smaller,
3

---

Published in Transactions on Machine Learning Research (11/2025)
Vislen-Kowleddge Graph Multi Modallty
Vision-Language MuttiModallty
Vision-Gene Expression Mult-Modally
Tile Caption:
This Image displys lytoph fealuring prominent
emeloped manti zone...
Report: adeeccarrdoomm Gleason score 4 (Grado 2) confinedd prostate.
Whole-Slid Image (V50)
Socanding
Figure 2: (Teft) Illustration of whole-slide image and its corresponding tile images from H& H&E-stiied tissue. (Right) The three primary types of multi-modal approaches in computational pathology.
more manageable tile images, which serve as the primary unit of analysis in many computational pathology studies.
While visual analysis remains central to CPath, researchers increasingly rely on multi-modal data to enhance interpretability and improve model performance. One major auxiliary modality is language, which includes both tile-level captions that describe specific regions of tissue and WSI-level pathology reports that pro- vide global contextual information about slide. Integrating text data with images enables vision-language models to learn richer feature representations and facilitate interpretability. Another important modality is structured domain knowledge, often represented in knowledge graphs, which encode relationships between diseases, biomarkers, and tissue structures, guiding AI models toward more biologically plausible interpreta- tions. Additionally, molecular data, such as gene expression profiles, offer complementary insights by linking histopathooogical features to underlying genetic mechanisms. By aligning visual data with gene expression information, vision-gene expression models enable the discovery of novel Figure 2 illustrates examples of WSIs and tile images alongside the three major multi-modal paradigms in CPath. The synergy of these multi-modal approaches, including vision-language, vision-knowledge graph, and vision-gene expression, has proven crucial in advancing the field of CPath, enabling more robust, gener- alizable, and interpretable Al-driven pathology models.
associations.
2.2 Pre-training Objective for Multi-Modal FMs
Unlike uni-modal models, which are primarily pre-trained through self-supervised contrastive learning (SSCL), multi-modal FMs, due to their cross-modal nature, involve more diverse set of self-supervised learning (SSL) objectives during their pre-training process. Furthermore, when fine-tuning LLMs to enable conversational abilities, supervised instruction tuning is usually required.
The primary pre-training objective for multi-modal FMs is SSCL. CLIP (Radford et al., 2021), as a pioncer in this field, ensures that the embeddings generated by the image encoder and text encoder are as similar as possible for paired image-text data by utilizing contrastive loss. CoCa (Yu et al., 2022) builds upon CLIP by adding multi-modal encoder and an additional captioning loss to enable the mapping from the visual space to the language space. BLIP-2 (Li et al., 2023) trains lightweight Querying Transformer (Q-Former) using a two-stage strategy. In the first stage, frozen image encoder bootstraps vision-language representation learning, while in the second stage, visual features are mapped to the language model input space, leveraging a frozen LLM for text generation. Additionally, next word prediction (NWP) is a text- specific SSL task commonly used for fine-tuning LLMs. aims to predict the most. likely next token based on the given text sequence. Morcover, cross-modal aligmment (CMA) multi-modal domaín-specific task, which aims to build a unified semantic space where the embedding vectors from different modalities can reflect the same semantic content. In addition to contrastive learning, generative reconstruction and prediction are also commonly used SSL proxy tasks for CMA.
Instruction luning (TT) is a method for fine-tuning LLMs to enable them to better understand and execute the instructions or task requirements provided by users. Unlike traditional pretraining objectives like NWP, the goal of IT is to enable the model generate meaningful responses or actions basod on specific instructions or questions. In Instruction Tuning, the model not only learns how to generate language but also learns how to adapt and generate different outputs according to various task requirements. This typically involves
4

---

Published in Transactions on Machine Learning Research (11/2025)
supervised training using large number of instructions, ensuring that the model can understand the intent of the tasks and effectively perform them. Such tasks can include text generation, question answering, and conversation.
3 Multi-Modal Foundation Models for CPath
The power of multi-modal data has been repeatedly validated not only in the general machine lcarning com- munity (Wang, 2021; Wang et al., 2023; Wu et. al., 2023) but also in the field of computational pathology (Ochi et al., 2025; Chanda et al., 2024; Guan et al., 2025). Given the high cost of pathology image data, leveraging other modalities, particularly textual data, as auxiliary information to learn more robust tile or WSI rep- resentations has become a dominant approach in developing foundation models for pathology (FM4CPath). Based on the modalities used, we categorize existing multi-modal FM4CPath (MMFM4CPath) into three major groups: vision-language, vision-knowledge graph, and vision-gene expression models. Additionally, we comprehensively summarize their network archilectures and pre-training details across different stages, as shown in Table 2. The rapid advancements in LLMs have enabled cration and conversational capabilitics. We further categorize vision-language models into non-LLM-basod and LLM-based approaches. The goal of these methods is typically to learn robust representations of tiles or WSIs for a wide range of downstream tasks.
to possess enhanced gen-
3.1 Non-LLM-Based Vision-Language FM4CPath
Vision-language FM4CPath enhance the models' understanding of pathological images by aligning paired imago-text data under vision-language SSL frameworks, such as CLIP (Radford et al., 2021) and CoCa (Yu et al., 2022), enabling them to learn robust visual representations while also supporting zero-shot and cross- modal tasks. These methods typically use an off-the-shelf trained vision encoder before performing joint risual-language pre-training, which has been shown to improve the performance (Zimmermann et al., 2024). They also leverage existing LLMs or Visual LLM (V-LLMs, a.k.a. MLLMs), typically in two ways: (i) fine-tuning them to serve as text encoders, or (ii) leaving them untuned, solely utilizing their capabilities for generation and conversation.
CLIP-based Vision-Language FM4CPath. The success of CLIP on natural images has inspired some works to apply it in the CPath domain. PLIP (Huang et al., 2023), PathCLIP (Sun et al., 2024e) and QuiltNet (Tkezogwo et al., 2024) all fine-tune CLIP model pre-trained composed of paired tiles and their captions. CHIEF (Wang et al., 2024) uses an image encoder pretrained for CPath domain (Wang et al., 2022b) to encode the tile sequence extracted from WSIs to obtain WS1-level features and CLIP's text encoder to encode anatomical site information (WSI-level label). A weakly super- vised aggregation network then combines both modalities to generate rich multi-modal WSI representations. Unlike previous methods that rely on out-of-shelf vision encoders, CaL (Dai al., 2024) leverages the generative capabilities of V-LLMs to obtain high-quality tile-caption pairs and uses them to train an OpenAl CLIP (Radford et al., 2021) framework from scratch, followed by fine-tuning on tilc-caption pairs from public datasets. PathAlign-R (Ahmed et al., 2024) is also trained from scratch on pathology data using the CLIP framework, but it focuses on the WSI-level. M.L.P (Zhou et al., 2025) leverages V-L.Mss as the backbone to generate universal multi-modal embeddings for CPath, integrating images and text within a single framework to better understand their complex relationships.
natural images using datasets
CoCa-based Vasiun-Language FM4CPath. CoCa's multi-modal decoder serves as a crucial bridge between visual and linguistic information. By transforming encoded image features into text-aware rep- resentations, it significantly boosts the moosl integration of MMFM4CPath, thereby enhancing its performance in advanced pathology applications. CONCH (Lu et al., 2024a), PRISM (Shaikovski et al., 2024), and Lucassen et al. (Lucassen et al., 2025) all pro-train an image encoder on pathology datascts, and then further conduct joint vasion-language pre-training within the CoCa framework. The difference is that PRISM and Lucassen et al. extend the image encoder to the WSI-level using a Perceiver network (Jaegle et al., 2021) and employ WSIs along with their corresponding clinical reports for training. MUSK (Xiang et al., 2025) first independently trains image and text encoders on unpaired pathology images and text tokens via masked data modeling within the BETT-3 framework (Wang et al., 2022a). Using Masked Image
5

---

Published in Transactions on Machine Learning Research (11/2025)
Table 2: Overview of architecture and pre-training details of MMFM4CPath (Due to space constraints, the references for the mentioned LLMs, V-LLMs, and off-the-shelf architectures are provided in the footnote of this table.)
[Asslaaliit))
Model
Your
Vision (VY)
Leuagenge (k) Knoeledge Graph
(RG) / Gone Exprasion (CE)
Netwock Ardiltecture
Multi-Modal
Objective
Stesieay"
O
Preiraining Datalla
M
Dato Short Deseription
tsput
Type
Quittes (Broegssuet al, se61
Ry
Trandorrer
SS (CT.P))
<28R TIle oud 597
PLP (llwee al, 3023) /
1. Vsneteeer lagees
StL (ULJP)
D
D
Zelk Coppio Palre
PIP st si. 313e)
2004
T: VIE-B/52
L. Unuatraar Lagers
SSL (CLIP)
D
D
ani TepCppkk Pairs
10k
(Sulkowki 9031
VIE-R/14
Nei,
RRCIPT (M1-18)
BIGPPT (LU3 wis Orns-biiaztn
Spertenes
PotaAlig- 2 (Mbred et si, 101 X
2014
V/T-5/16
L: Torarr
851. (M85))
(CT.1))
5X
Thee Prees 301 WSt
WSI Repirt
PattAliggG G [Mbroed et al, 2B1) X
2014
V/T-5/16
Qornee (1MM)
MEP
200 (1858)
188
5X
KMN
SN
N
"Ele Fivez 354 Wilt
43 WSS Haport ihaks
WSk
(U35A)
Now 22t--
DRLLOY (Wragg 2031:
2001
Augovaator Not.
E: Svar?
Thunttener Lagerx
XLL
WSS (C3U1))
0,5
GIK Wiit whs Labels
SEL (801))
UMM Tile Saspphel Prees 218 WSl
CONCH CC el al, 200M)
2004
T. W/1
L: Treasbezze Lagers
Tranderta:: Legms
851, (KWP))
N
SORAE Puckchay Teal Kainis
Tike
851, (CsCk)
Til Oup Patrs
HEAN (2iaz 2034)
2014
WLLL VITS
Druakezarr Lapera
Trandoraas Layaza
SSL (7931)
SEL (C26k)
3.D
F.S
N
421K Ekil-Captiss Patx
2543: WSp
(C24k)
MUSE (Xiogg 9035)
V-P7N
Stand Micsie Layers
I. L.P m]]]]
Don Attvadion Deeode
SSL, (BR-175
(CsCC)
IB Toe. Tikana nnal lild Take
LOB Tile Oppaie Patrs
Pall (Sta st, andd) x
2005
T: V11-
I. Uhuaabrnzre Lagers
SST. JCT
SST. (CT,UP)
1/5M Flih Quality Th Caytina Parte
7066: TTe Captkes Paare
11km
MELSHPUU [Zace 2005)
2005
I Upwe: 1.5
MIL
SML (U3.IP)
D
SHil Copilo Palre
11k
2005
3006
W: Nei,
VIU46/14
BuCPP (LI
BixGP 12:3 waS
8EC, (CsCK)
41K
ISK Bapocts
Pathdtt (Bas 201))> y
2014
VII-8/33
(LLM) Viaaa1388
XLP
SRL (CMAA)
Dovepptoce Parl of Parstsyrtcon Seuppo Froes PertileTRACT
TeETWW (See et st, ammua /
2011
TV ym
I. (TIM)) Vioee
St. (IT) & RF
D
Mivtl ure Disloggs To in Iik Tile
Tike
2011
TV Wyy
(I(MM) GPT:4
50.
(USIA)
y
UITK Poriclegy Speilli Inttractioes
The
SSL (C343))
1.54M Iip-Captias Pate
PackClax (t al. 302661
(LL)) 30G
Poliley
8L
457k lsstnnttize wiih 5668 Vigt Surar
Haa5CPF28/24 act)
2024
Wi Perstoe Net.
(LLNG BuCIPE3 ! HhGIP
SSL (8W15
0/0
IX.R with
12.1 WSL Palee
BaccGPT (Thu el al, 200/
2024
VID-L/t
(LLM) BwGPTLL
SEC, (NWP))
F5
B&1K WSL-Bepeet Pars
C1IOVTOR (Chea vi. 2006)
EVA V/T
L. Q-Pomue (LLMI:: Vaua /B/ FIaTT
QpPorte::
XIP
NSL (BLIEP-3)
7
5 8N
cook T.ls and sHtk Capcano
VQJA haainatias
Puthheige (W0 al, Xps/ /
2011
J- VHTIMG VIAVA / Queen VT.28 / (IiweEM
S0 (TT)
1/1/1
428 Tustaces Conviing 6 Puitelay Tadis
Tike
E2M--boo
Longlee
XLL
(UMIA)
1788 butrodtice likkoling Pair
VIL-S;
Patble dBEaL -iBit
2004
W: Eraadonam Layezs
HaaNe=31 HIPT
An Babeiilng Mayciag
Taaadiraa: (NWP)
Wi 5:
WSS 7.14k VIAA
Tie-Coptaan Parts
PA-LLAV (Dmi al., 2004
2014
VIT-8/32
with LaRA
Trandorta::
SL (TT)
(CMA))
D
56.0K VIQA Pars
WSELLAN (Tāvag X
NPM
Tangleee
L ChasSBBRT (LIA) Reeisa/bbeiia
P
SS1 (CMA)
F.F
KP
0,55K WS5 Paire
MIP
S1. (1T)
F.F
K.
D
175K: VIgA Paiie
(aab Oree (Sen al, X05B) x
Wi SilaPhree
1/T VIT 1.
L. Qweit 1-
851, (CCO))
SE (IT)
D.D.D
F.D.D
D.D.D
D
F
JueaK WSL-Boqeer Parrs
911X Tixs Teatin lass
S1X Instretion
Pattiee HLMA (S0 al 33000 anC
TV. VIT Ryax
I. Tauaabezzr Laper
1 [LLMI Viena
MPP
SSL (CMA
551. (CT.IP
SN
NP
N
TUE66 The Cigtten Paive
T166: Tie Cogties Phire
TEP-LLAVA Bay is at, 365; x
2004
1. V21-0/36
I: Se Ti-tbemmxt
MLL
Nu (12)
y
375K VIJA Paite
(Bhorc 2020 /
2004
L.MMMeBBlR KG: PuldkelBEET
SSL (CEIP)
SEL, (PNE)
D
0.P
THEK Tie-hapka Piirs
ehor 3015)
951, (PKE)
4IK Gusaye 863
Patladogr wibh
Vaaio-GE
TANGLE Larre si., 2004)
2014
(thonan)
VIT (But), Seds
A Laym MLP
581. (800))
jCT.1)
S/MNN
F/7.5
15M Ras Tike Pien WSlk
AFTAR (K. ni., 2
2014
Tow Lager Vruealttl.
1t: VI1-4/15
L. SEULLLE Hawvi.2 GK willsti
SoL, (U3.I))
350. (800
N.N
T.S3K Wil--eeper- finas patee 7.83k While
WXL
THREADS (Vubb v., 2005
2006
We ASMII
T: Vil-i
GE: sKIPT IRNA), A BapsLaom MLP (DOA)
SSC, (CEIP)
F8
20.8 WalkGise (RNA4) Pazz
29AK Wib-loaa (DNA) Paiz
WSD
JP (C
VIT-B/18
GE. MLP
MEL
SSC,
2155 like
Netovel sechiteerro rypie Thx Fandee W: WST Fandee I. Test Raedrr, LLM: Large Langugg Movel V-M Matii meddd LEM, RG Krowdstge Glyph Devedrr, CR Oree Reganddos Funder.
Muti redid Sensiatis vodes on sylialy pertlldd in rattiil etge with eoch row take column repooeetting distect pootrahiing place.
(TT), Wist bedatw Knarid lutter Laniie (MIL), and WE1 Weedy Sspented Leonler (WSL.,
catoyetont
(S8L.)
Oppoclaee
Comatiio Locaing ICE. Madaed Hansoo Netwer (MSN)) Sed Wer Proditoe (NWP). Chae Modu Abyeune (CMA), Pethodigy leaonidgg Busscing (FSE), und sditbaiittto iSD) Cl fariee dreakel lead in ta ounaite ekeedivs nu CLIP. CoCs BLIF-2 BBOT and BELTS.
Protmasing sos opi dikeas aedaleecurs Yaass
Madaltis, 36 KabaaSodal Frowss Fron Serich. D. Dannn-Spon: Turra, Insesriau Trurs (Claezz
in Eesol.
GPT4 (eeaae al. 2026 Lhoea 3.13: [Doee 200b Vieasa iB (Clam al, a00 Fla (lomg al, 200 LLAVA (la d di, 2001), Vien- IB (Bar dl 2925), Inveall (Zhang 2023a), QemalaiB (Ying ei al, 2004, L Marrst comiiaaion it 3220. LoRt Hi al, 2000 Viemae/bo (Baas e sh., 2003: Qee dt 2020). Vaase (Cansn al. 2020
(135
Takie BcCPP
P
Virur-
2019. L-P0 dlllllee incriiction
2015 (2h Prenser (leepe di 2005 PEET
(t 2020) BVA-KIT-//1 (Paag si., 2083: QuEiezer clas sl, 2013) BiCR (ou
it 2002), ABMI al, 20n V-EEN (Kase
al, 201) BuBERT- Baov 2 (Lov 2000), WOPT (ha et al, 2000 nEEI (Fang 2002)
Modeling (MIM) (He et al., 2022), it leverages ViT's patch structure to predict missing patches and learn robust representations, and then aligns the two modalities within the CoCa framework. TITAN (Ding et al., 2024) proposes a novel foundational framework for whole-slide imaging analysis through three progressive
6

---

Published in Transactions on Machine Learning Research (11/2025)
Initially, the WSI encoder is optimized via the iBOT framework (Zhou et al., 2021) enhanced with positional encoding; this is followed by dual-scaled refinement under the CoCa framework leveraging tile-level features and WSI-level contexts, where pathology-specciaized V-LLMs generate diagnostic captions and structured reports.
training stages:
Other Vision-Languag FM4CPath. Unlike previous methods that use CLIP or CoCa framework, PathAlign-G (Ahmed al., 2024) first pre-trains a VIT-S using Masked Siamese Networks (MSN) (Assran et al., 2022), and then fine-tunes the model using the BLIP-2 framework. This enables PathAlign-G to utilize shared pathology image-text embedding space, enhancing its cross-modal capabilities and making it more suitable for generative tasks.
3.2 LLM-Based Vision-Language FM4CPath
The powerful reasoning ability (Li et al., 2025b; Zhao et al., 2025) and the fusion of vision and language modalities (Radford et al., 2021; Yu et al., 2022) provides an extra perspective for MMFM4CPath, where pathological visual representations aligned with language signals latent space can assist L.LSs in under- standing pathology knowledge, thereby contributing to the construction of generative foundation AI assis- lants for pathologists (Lu et al., 2024b). These methods acquire pathology-speciallle pre-trained image encoder with an LLM via simple multi-modal module for mros-mmdall feature alignment, then fine-tuning the LLM. Beyond contrastive learning, they employ diverse pre-training objectives, from supervised to self-supervised learning. We classify them into instruction-based and non-instrrcction-baeed methods based on LLM fine-tuning approaches.
V-LLMs by pairing a
Instruction- Based V-LLMs for CPath. Most V-LLMs for CPath undergo instruction tuning on carcfully curated datasets, refining general-purpose LLMs for the pathology domain while enhancing their pross-modal understanding. PathAsst (Sun et al., 2024e) builds a pathological V-LLM using PathCTII as the visual backbone. It aligns the image encoder with the LLM via trained layer on QA-based instructions, then fine- tunes the LLM with limited instructions. Following the same pre-training process, Quilt-LLaVA (Seyfioglu et al., 2024) and PA-LLaVA (Dai el al., 2024) are fine-tuned on their publicly available instruction-luning datasets, while PathChat (Lu et al., 2024b) undergoes instruction tuning on its carefully designed and diverse instructions. SlideChat (Chen et al., 2024d) and WSI-LLaVA (Liang et al., 2024) go beyond the tile-level and create V-LLMs capable of handling gigapixel WSIs, and are fine-tuned on corresponding WSI- level instruction datasets. TCP-LLaVA (Tyy et al., 2025) is the first pathology WST VQA model based token comprossion. By introducing modality compression module with trainable compression tokens, it compresses tens of thousands of patch tokens into a few hundred before feeding them into the LLM, significantly reducing computational cost while maintaining accuracy.
Instead of relying on separate image encoders or vasion-language architectures, PathInsight (Wu et al., 2024) directly fine-tunes existing V-LLMs using instructions covering six pathology lasks. In addition to ruction tuning, Dr-LLaVA (Sun al., 2024a) employs reinforcement learning (RL) with an automated reward function that assesses the clinical validity of responses during multi-turn interactions. CLOVER (Chen et al., 2024a) aims to develop cost-effective V-LLM for conversational pathology. It employs BLIP-2 with lightweight Q-former (Li et al., 2023), keeping both the visual encoder and LLM frozen to avoid full LLM tuning. CLOVER combines generation-based instructions from GPT-3.5 (Achiam et al., 2023) with template-based instructions, forming hybrid instructions that improve understanding. As one of the most powerful models currently, CPath-Omni (Sun et al., 2024b) aims to build a unified model that can process tile-level and WSI-level inputs separately through proprictary framework, and integrate LLMs to enable generation and conversational capabilities. It undergoes four stages of training on three proposed datasets of different types: tile-caption pairs, tile-level instructions, and WSI-level instructions.
V-LLMs for CPath. PathGen-LLaVA (Sun et al., 2024d) is trained from scratch on the CLIP architecture using tile-caption pairs, then a fully connected (FC) layer is trained to ensure the features extracted by the image encoder are understandable by the LLM, Finally, it employs a supervised image captioning task rather than instruction tuning, as PathGen-LLaVA is specifically designed for generating pathology image descriptions. W2T (Chen al., 2024b) utilizes four frozen visual extractors (including those trained on natural images and pathology images) and three text extractors in various
Non-Instruction-
7

---

Published in Transactions on Machine Learning Research (11/2025)
combinations. It is trained on its proposed WSI-VQA instruction dataset using next word prediction (NWP) to interpret WSTs through generative visual question answering. HistoGPT (Tran et al., 2025) is designed with three model sizes: small, medium, and large. Among them, HistoGPT-S and HistoGPTTM first train a Perceiver Network (Jaegle et al., 2021) as a WSI encoder using multiple instance learning (MIL), followed by fine-tuning the LLM with NWP. HistoGPT-I, on the other hand, employs a graph convolutional network (GNN) (Kipf Welling, 2016) to encode WSI-level positional information, eliminating the need for a pre-trained WSI encoder. HistoGPT is capable of simultaneously generating reports from multiple pathology images and provides prompts that allow for expert knowledge guidance.
3.3 Enhancing FM4CPath with Other Modalities
Due to the high costs, pathology-specifie datasets are typically small and sourced from diverse origins, such as websites or videos (Huang et al., 2023; Ikezogwo et al., 2024). This often results in noisy data with limited quality, making it unstructured and lacking domain knowledge. Meanwhile, massive multi-modal data aligned with clinical practices, along with domain-specific knowledge, such as gene expression profiles, remain underutilized for pretraining. Based on these, some studies have explored incorporating modalities beyond vision and language to enhance the training signal.
Knowleeee Graph FM4CPath. To integrate structured domain-specific knowledge, KEP (Zhou et al., 2024b) constructs a pathology knowledge graph and encodes it using knowledge encoder, which then guides lisiuanlanguage pretraining. They design pathology knowledge encoding (PKE) method to align semantic groups in the latent space for training the knowledge encoder. Similarly, KEEP (Zhou et al., 2024a) builds a disease knowledge graph for encoding and employs wwwlledg-guideed dataset structuring to generate tile-caption pairs for pretraining within the CL.IP framework, incorporating strategios such as positive mining, hardest negative sampling, and false negative elimination.
Desion-Gene Expression FM4CPath. Serving WSI-level information, gene expression profiles pro- vide insights into quantitative molecular dynamics, complementing the qualitative morphological perspec- tive of a WSI and capturing biologically and clinically significant details. TANGLE (Jaune et al., 2024) is a transeriplomix--uidee WSI representation ion learning framework that aligns image signals with RNA se- quences encoded by multi-layer perceptron (MLP) in the latent space using contrastive loss, similar to CLIP. It extends beyond human tissucs, incorporating a specialized architecture and datasct for rat tissue pre- training. THREADS (Vaidya et al., 2025), like TANGLE, utilizes molecular profiles from next-generation sequencing for WSI representation learning but uniquely integrates WSI-RNA and WSI-DNA sequence pairs. mSTAR (Xu et al., 2024) integrates three modalities within an extended CLIP framework, training WSI-report-geet expression pairs via inter-modality and inter-cancer contrastive learning. It then employs self-distillation to transfer multi-modal knowledge to the patch extractor. Recently, OmiCLIP (Chen et al., 2025) is a visual omics foundation model that builds on CoCa-style cross-moda contrastive learning to align H&E slides with spatial transcriptomics data, enabling gene expression prediction and cross-modal retrieval, thereby bridging pathology and molecular omics.
Note that some methods do not solely focus on pathology images but also encompass multi-modal medical imaging data such Computed Tomography (CT), Magnelic Resonance Imaging (MRI), and X-ray from various organs (Zhang et al., 2023b; 2024; Zhao et al., 2024; Xia et al., 2024). However, since their goal is not to leverage other medical image modalities to enhance pathology image represontation but rather to develop a universal medical image model, these studies exceed the scope of our survey.
4 Multi-Modal Datasets for CPath
Larger, more diverse, and higher-quality datasets for CPath have been proven to be the key to the success of FM4CPath ( (Vorontsov et al., 2023; Zimmermann et al., 2024), and MMFM4CPath is no exception. Curating pathology-speciff public datascts has long been a challenge in this field, driving extensive rescarch efforts. Many well-designed datasets have been developed address various pathology-related questions, continuously advancing CPath. We summarize existing multi-modal datasets for CPath, highlighting high-
8

---

Published in Transactions on Machine Learning Research (11/2025)
quality datasets those that have demonstrated success in current models. Based on data types, we categorize them into three groups as shown in Table 3.
Image-Text Pair Datasets for CPath. This category includes tile-level tile-caption pairs and WSI-level WSI-report pairs. Training on these datasets within a self-supervised contrastive learning framework enables FM4CPath to learn richer image embeddings while gaining zero-shot and cross-modal capabilities. Due to the expensive expert annotation and the preference of many research institutions for in-house data, several datasets have been constructed by collecting pathology tile images and text data from online sources, books, and publicly available educational resource.. For example, QUILT (Ikezogwo et al., 2024), OPENPATH (Huang et al., 2023). ARCH (Gamper & Rajpoot, 2021a) and MI-ZERO (Lu et al., 2023) leverage data from YouTube, Twitter, pathology textbooks, and educational resources, respectively. Due to the lack of a unified format, the collected data undergo standardized processing pipelines to ensure high quality. Image data is filtered for non-pathology images, followed by sub-figure segmentation. Text data refined with LLMs, including sub-caption segmentation and token-bascd filtering. Finally, multinodal models align figures with captions. Additionally, QUILT (Ikezogwo et al., 2024) uses speech recognition to extract text. from videos.
Other tile-caption pair datasets primarily expand existing datasets or utilize internal datasets to enhance scale and diversity (Sun et al., 2024e; Lu et al., 2024a; Sun et al., 2024d;b; Dai et al., 2024). Notably, ARCH (Gamper & Rajpoot, 2021a) is a multiple-instance captioning CPath dataset, where each image bag is associated with a single caption. Furthermore, datasets such as PATHGEN (Sun et al., 2024d), HISTGEN (Guo et al., 2024), and MAsS-34KK (Ding et al., 2024) generate WSI-report pairs by leveraging generative models or processing WSI descriptions using LLMs.
Multi-Modal Instruction Datasets for CPath. These datasets incorporate diverse instructions for tun- ing LLM-based vision-language FM4CPath, training them Al assistants in the pathology domain. Since mamually designing instructions is typically expensive, instruction construction often directly relies on LLMs to generate cost-effective instruction datasets. The most common instruction type is VQA, which typically includes closed-ended and open-ended question-and-answer (Q&A) sessions to develop the model's conversa- tional abilities. Due to prompt flexibility, different datasets create various instructions based on their needs. For example, PATHINSTRUCCTSSu et al., 2024e) provides cestruction-olllvviii call upon other pathology models for problem-solving. Lu et al. (L. et al., 2024b) developed six instruction types to adapt the model to diverse pathology conversation scenarios. CLOVER INSTRUCTION(Chen et al., 2024a) generates instructions both through LLMs and by matching template questions with original text captions for cost-effectiveenes. PATIMMU(Sun et al., 2024c) uses enhanced descriptions with images to prompt GPT-4V (GPT, 2023), generating professional multi-modal pathology Q&As with detailed expla- nations. Due to the scarcity of large-scale multi-modal pathology datasets for training WST interpretation assistants, WSI-level instructions have emerged, typically creating VQAs from WSI reports and advanced LLM-generated prompts.
samples that enable LLMs to
Image-Other Modality Pair Dataset. There is still a lack of extensive exploration of datasets involving vision and other modalities. Zhou et al. (Zhou et al., 2024a;b) constructed two different discasc knowledge graphs and, guided by one of them, created well-struetured semantic groups linked through hierarchical relations. The MBTG-47K dataset (Vaidya et al., 2025) includes paired data of DNA and RNA gene soquences with WSIs. Xu et al. (Xu et al., 2024) publicly released a datasct with pairs containing three modalities. These are bold atlempts at constructing datasets that integrate pathology images with other modalities.
5
Evaluation Tasks
5.1 Taxonomy of Evaluation Tasks in MMFM4CPath
Unlike uni-modal FM4CPath, data from other modalities not only enhance MMFM4CPathis understanding of pathology images but also enable MMFM4CPath to perform zero-shot learning and cross-modal tasks. When MMFM4CPath are combined with LLMs, they gain the ability to engage dialogue and genera- tion, allowing them to adapt to more diverse tasks. We have summarized the evaluation tasks used by MMFM4CPath, as shown in Figure 3, and categorized them into six main types from a machine learning
9

---

Published in Transactions on Machine Learning Research (11/2025)
Table 3: Multi-Modal Datasets for CPath.
(fwiisbiiit)
Datsot!
Datas Type
Desssipiion
Suaiiing Dotoort loweibrt
Pooili
Data Siurce
Privite
Mochol
Assbted
LISE
tike pard with stils4
Contersg Quisr
(Rasppo cl, 2011
The Captins Pak
VoeTule
quiliic (Troegoo ctal., 9030)
Videe
scuion Beia 35 pair.
ParaCar (bee sl, 2006
CeCClppt Pot
pulis
pallsdagy
1.O
Pubbied (G sl, 2011
PHICUUP (2a 202k)
PertLLATON 2000
Captios
spptii yoirs
drssaal tia cpttbe pars the LAIG detoct (Srei socm au. 3520
WSS Tuttter Rapios
(floog Xra)
DUNCE et 1ibaa
Caplin Pak
PMK (Burate 9350)
COSCE5 (1e a610
Gas
WSL-Itppon
UUS oxpart
Mare bInK (DKag S090
C35 WSls Shaggie
151100 nitt aall 182802 Wikkopprr peiss.
QTTS (Soanttios SE--)
IIW (Paay al, 5930
(PennPoncc (Su di N004
Carroos
x
Tho Captiis Pai
TO0 liecapiko
1,1.0
Paraaaar Guaan-3hi Ooesthenn
CPhck Drasi (Stvet at, X3b0))
Pennti (tea ot al, 200601
TepCapaa Par
explie pake bous 2000 WSt..
15 celkes Ligrepalts die
H
optiia para thes FacaCarr
Pasttitolua
TaK the-
TOGA CTexx it al, 201)
PritGaesUU (Bux in sl, 2044)
Mosam (lou 201)
WSS-ltppet Pair
11:33 paavd WSi and perhes-
Hasoiii (hna al, p0t)
PoSurnss sik menwing PCarman-L.5 6,36; nurikee the est pars with cit work
PCaamnssC (803)
Hepoapttnn 1,30J00 tioegrane
B,1,0
PMALLAN 3120)
ARCII (Chrper & Bajose, Stttaa/
Bag Cuptiss Pai
with rucl bag modiliing wabi
12,08 lear and 15,154
Pritded "Gu al. 20ti), palssker lestbook
x
Militas et di, 3125)
TepCaptia Pat
Divme decintt $5,000 the
B.1.O
mlastseel ioverm
Parsfooraces al, (800)
Towl
INIK pulledegy welll rustal
B.1.O
Pobdoa (ux al, (838))
somple,
CPem Vigh costel be pou autieg (Bbext 251), vart tonalionn doeiscoto klck beidtz optiis..
CPunsParc (Paa et sl, 2004 x
palos labicdd iseate to tha-
ViJ
II
Pantlia, Cieers PerminCamoon Paraliomee
Chackllas (Sha al, 23248)
Mons-Modal Invercctis
Inncanccion mo al, St300)
CPeon WSI
MSE Look Instrrctioss
reglengn VOS and ckins.
HI
datect le proppiin GPT C'Aclis alo))
CPack Orm (Sar al, 3200)
QUiT viga e QUA laseik foo Yostiee
QuT (Syyigh ot al, 3000) /
VQN Pice
3EL Q&A pars.
1.1.0
genva know why qpedices QUILL--VAAea QUIL- the BDt tee pelkengr keage.
Youlibbe
Qch-ltac (Syyigg al. 2800)
oxcated beuiasakk ligs qui lass PorsiAAtiou Prace PorsAABeects
Pacallast it al, 2001
Testratiios
589.212
quastioa
PMC-DA 2820. TOOL (Tene at di, 2015)
PaiaChat (lare al., 23200)
Tia-Land
QUEE-YQA PallID anst
9926u)
Willeedd ihe faviie and ineovsstioo
Evnismens (Wi of
II
PAlWOO (Ih d. M00 oec
Pattiaaigh (Whi al, (xt
Suteiksso Wel line sking witt a ederiiaiall uxnder cherdiin VQS pams oeuilia sollatiis suck
Sume (Che me0a
WSF Look
18) WSI snike, palls and UiS
TOGA (Teex et el, (00)
SaaCCtr (Chm 30100)
Vigh (Cee /
Plyk
102 N477 guo
WOSTA (Trecekk el (913)
Wil (Chee g969
PAALAWA (D a. StA6/
ViAA Paoe
30,043 quotinnezneer pato.
PaaAAA (He mal 208)
DAelLan (Daret d. 2209)
Suaeeiasses: 334 WSI ox-- abrng sandur cheddat VIgA aniilia avakation beack-
ViAA pain.
Wit-Bloes [Lang i.. 2000)
VIDA Plic
vidth parkiloptoll
HI
TOGA (Texcak at al, 201)
WSI LLAN (traze et al, 2830
eppaidiiii witk takia
Suuddasoo 734 WSI uander chendont VIJA paza enaliia evatattaa betca-
PAMDEMA (Bua al. 3Gid of
VIDA Plzz
pullodigy
Qlcto with
Pritdei (Dret a0m1) Quaa--b
Acka (Mlber 2000) Dresthom
DEPSLLANA Gara al, 2000)
VQSA Dlaz
139.97 Qlck parn Ivee 39 db-
B
TOSA (heeonde eral. ant)).
ICPsEtaVA (aos e a01)
Putlodogy
205 contata 11.45 toman otr
DO (Sduiid 201)
KIEP el st, 333le /
KEEP (Zae et du, 33Bla]
Sutserie Group
Isded thrregs theme
Pallsdag
16,77 inforaatio vitebatte
ESTARR (Nu al. 201)
WSS 11S- Soy Pix
lsnge 1ss and IISA
H
TUGA (Zooook it 203)
aSTAA [3x (S al, 201)
moluiiin be peotuilaing
MBTG-4TK (hatps sl, a0t) y
ANS-RAASIE Pii: JaAiS WS1-0OA URLA-iv Per. 29/06 WS6-INA perrs
and
HI
GTEe (Deaetions al, 2010)
1004 (Hoernd id. 2018).
TIIIADS (aaipa et al, 20251
ST. RADE (Che it vl, 2005)
Tile Tocaedso
leekk Parr
Mb-trraaeegeeeees pars
II
Inu Valiee (Mi al, 5803)
Diait.IP (Chen it al, (86))
Suoce werlade detooss wiikon swulng
entioel and soaled with wrte
Stakkin 1pp. H.. R. IlC, O Gilve..
perspective: classification, retrieval, generation, segmentation, prodiction, and visual question answering (VQA). Furthermore, we classify them based on the type of pathology image input they target (tile or WSI). For MMFM4CPath, their pre-training dataset and model design are closely tied to their evaluation tasks.
10

---

Published in Transactions on Machine Learning Research (11/2025)
Supervised
QuittNet, TITAN, MUSK, Patniniight
Tile-level
Zero-Shot
PathGen-CLIP KEP, KEEP, OmiCLIP
QuitNNt,
PatCCLIP. CONCH, TITAN, MUSK, MILMMPUEE, PA-LLaVA, CPath-Omnl
Few-Shot
CONCH, TITAN, CLOVER, CPath-Omri, PathGen-CLIP
Ciassifiction
(Weakly) Supervised
PRISM, PathAAggn-R, CHIEF, MUSK, HistoGPT, W2T, CPath-Omni, PathGen-CLIPP
MSTAR, THREADS
Zero-Shot
PRISM, CONCH, KEP, KEEP, MISTAR
WSI-level
Few-Shot
CONCH, TANGLE mSSTAR
Goneralization
HistoGPT, CPath-Omni,
CONCH
Tile-to-Tile
PLIP, MUSK
Tll-to-Caption
QuitNet, PattCLIP. CONCH, MUSK, MULM4PUE, KEP, KEEP
Caption-o-TTee
QuitNet, PLIP, PathAign-R, CONCH, MUSK, MLLMMPUE, KEP, KEEP
Tile-level
Tle-to-Gene
OmiCLIP
Disease
KEP
Composed
MILMAPUE
WSI-toWSI
TITAN, TANGLE, THREADS
WSi-to-Report
TITAN, Lucassen
WSI-ieval
Report-ta-
TITAN, Lucassen ot
Tasks
Gene-to-WSI
THREADS
Tile-level
Captioning
PathAlign-G, CONCH, Quil-LLaVA, Pethinsight, CPsth-Omni
Generation
WSS--level
Report Generation
PRISM, TITAN, Lucassen al., HistoGPT, SlideChat, WSI LLaVA OPath-Omni, mSTAk
Segmentation
WSS--levll
Zero- Zero-Shot
CONCH, KEEP
Tile-level
Gene Expression
Prediction
OmiCLIP
Prediction
Survival Prediction
CHIEF, MUSK, WZT, mSTTAR
WSI-leevll
Tumor Thickness
HistoGPT
Prediction
PR Prediction
W2T
Closed-Ended Open-Ended
MUSK, PathAsst, Dr-LLaVA, Quit-LLavA, PathChat, CLOVER, PA-LLaVA, PathGen-
LLaVA
Diegnostic
Referring Prompling
CPath-Omnl
VQA
WSI-level
Closedi Ended Open-Ended
SiidChaat, W2T, WSILLaVA, CPath-Omni, TCP.LLaVA
Figure 3: A comprehensive taxonomy of MMFM4CPath, categorized according to evaluation tasks. Non- LLM-based Dision-languas LLM-based vision-language vision-knowledee graph, and vision- gene expression models are highlighted in different colors, respectively.
For example, benefiting from multi-scale and more diverse training instruucions, CPath-Omni (Sun et al., 2024b) has been evaluated across the widest range of tasks.
Classification is the most common evaluation task for MMFM4CPath, as many pathology-related tasks, such cancer subtyping and biomarker prediction, are fundamentally classification problems. Most MMFM4CPath are evaluated on classification tasks across various scttings. Models using tile-level inputs can perform WSI-level classification via multiple instance learning (MIL), treated as weakly supervised due to the lack of detailed region annotations. Multi-modal data enables zero-shot or few-shot classification with minimal reliance on costly annotations. Some methods also assoss out-of-distribution (OOD) generalization handle distribution shifts between training and test data (e.g., data collected from different institutions). Additionally, CONCH (La et al., 2024a) evaluates classification on diseases with imbalanced data.
11

---

Published in Transactions on Machine Learning Research (11/2025)
Table 4: Comparison of MMPM4CPath for zero-shot Lile classification task on multiple pathology datasets. Bold is the best and underline is the second best.
Method
WSSSILAAD
LC22500LLUG
LC2500COLOO
BACH
CRC100K
OSTRO
SECAPP2
POAM
SKINCANCRR
Fi
Accuray
F1
F1
Accurary
F1
Aurarey
Fi
Avcurary
F1
Accomnery
Accurary
Aurmmry
F1
Aecurary
QuinNNat (Lansogoo ol. 2024)
0.827
0.706
0.775
0.530
0.541
0.335
0.418
0.053
0.495
0.565
B.538
0.373
0.587
0.411
0.464
PLJP (Honng 9023)
0:361
0.022
0.502
0.515
0.528
0.529
0.425
0.518
0.155
0.125
Pe4CLPP (Sau
2024))
0.851
0.943
0.943
0.468
0.053
0.502
0.483
0.725
0.351
CONCH (Im al, 2023a]
0,798
0.527
0.006
0.500
0.785
0.413
MEZMIPUF (2hon et al. 3025)
0.591
0.900
0.223
0.562
0.522
0.638
0.305
Cpaih-Daani (Sin st al., 30240)
0.871
0.971
1.000
0.725
0.780
0.507
0.631
0.959
0.742
PachGen CLP (Son al. 2021d)
0,822
0,808
0.05
0.715
0.780
0.265
0.635
0,582
0.706
KEEP (Zhon 997.a]]
0,802
0.930
Table 5: Comparison of MMFM4CPath for tile-to-caption (inagge-l-text)) and caption-to-tile (lext-to-image) retrieval on multiple pathology datasets. Bold is the best and underline is the second best.
Tile-to-Caption (125)
Caption-to Tile (+26)
Meshod
ARcH- ARCH-PUgMED
ARCH-BoOK
ARCH- PUBMED
Bocn-Book
R155
R-110
R350
R455
R@10
R0050
R:25
Re210
R@50
R655
R$10
R050
QuilkNess (Desrogwo et al., 2024)
0.069
0.111
0.273
0.116
0.168
0.384
0.056
0.092
0.237
0.100
0.152
0.389
PLIP (Huang et al., 2023)
0,037
0.067
0.185
0.096
0.132
0.393
0.037
0.067
0,181
0.112
0.161
0.119
PathCLIP (Sun et al., 2024e)
0.275
0.388
0,680
0.152
0.234
0.482
0,236
0.348
0.630
0.137
0.196
0.445
MLLM/PUE (Zhou et al. 2025)
0.372
0.195
0.782
0.192
0,283
0.603
0.297
0.399
0.688
0.185
0.277
0.555
KEP (Zhou al., 20245))
0.196
0.421
0.282
0.564
0.176
0.404
0.340
0.621
KEEP (Zhon et al., 2024s)
0,180
0.248
0,492
0.298
0.404
0.732
0.182
0.228
0.491
0.322
0.434
0.781
In addition to basic image-to-image retrieval, non-T.MM-bssed MMFM4CPath are widely used for cross- modal retrieval tasks, such as text-to-image and image-to-text retrieval. KEP (Zhou et al., 2024b) performs one-to-many discase retrieval, retrieving captions or tiles with the same discase label using discase names. MLLM4PUE (Zhou et al., 2025) enables many-to-one composed retrieval by using potlology images and questions as queries. Moreover, due to its capability to understand gene expression data, THREADS (Vaidya et al., 2025) generates class prompts from gene expression profiles for WSI retrieval.
The integration of LLMs, whether by fine-tuning them as part of the model's architecture or by directly uti- lizing existing models, enables MMFM4CPath to generate captions captions/reports from tiles/ tiles/WSIs. WSSs. CONCH (Lu et al., 2024a) and KEP (Zhou et al., 2024b) evaluate the segnentation capabilities of these models. Some MMFM4CPath have also been tested for prediction tasks, using WSIs to generate continuous value predic- tions.
LLMebased MMFM4CPath models focus on evaluating their diagnostic VQA ability. Compared to tradi- tional QA tasks, VQA incorporates pathology images into its questions, challenging the image understand- ing capabilities of V-LLMs. Typically, VQA tasks involve answers from a fixed set, usually in the form of closed-ended questions, such multiple-choice (single or multiple answers) or true/false questions, as well as open-ended questions with no predefined answer options. These tasks can also be divided into multi- and single-turn dialogues. The initial LLM-based MMFMACPath only performed tile-level VQA tasks (Sun et al., 2024e; Lu et al., 2024b), but recently, conversational abilities on WSI have gained increasing at- tention (Chen et al., 2024d; Liang et al., 2024). Additionally, "Path-Omni (Sun et al., 2024b) has been validated on the visual referring prompting task, where the regions of interest (ROTs) are highlighted, and both the question and answer are based on these regions. It is worth noting that, due to its flexible format, the VQA task offers high adaptability: tasks like classification and generation can be transformed into VQA tasks via prompt engineering (Wu et al., 2024; Sun et al., 2024b). Thus, LLM-based MMFM4CPath also encompass evaluation capabilities typical of honsLL models. In addition to the quantitative analysis above, qualitative analysis is also frequently used to assess the performance of MMFM4CPath, especially their VQA and generation abilities. This is done by directly observing or through evaluation by professional pathologists to assess the quality of the generated text. For a more detailed discussion of the risks associated with automatic metrics and LLM-as-judge in generation tasks, as well as evidence from recent studies that employ pathologist-base evaluations, please see Appendix B.
12

---

Published in Transactions on Machine Learning Research (11/2025)
5.2 Comparative Analysis of MMFM4CPath on Evaluation Tasks
Currently, research on MMFM4CPath lacks unified benchmark that specifies consistent protocol (e.g.. evaluation datasets and metrics) for assessing different models. In this work, we have compared the evaluation results of all available methods covered in the survey as comprehensively as possible, in order to provide a useful reference for model selection. As noted in Section 5, although different studies may employ the same type of evaluation task, their choices of datasets and metrics are rarely standardized. To address this, we have collected two widely adopted and relatively consistent evaluation tasks and reported results across multiple datasets and metries in Table 4 and Table 5.
Table 4 presents the Fl-score and accuracy of eight MMFM4CPath models across nine datasets, namely WSSS4LUAD (Han et al., 2022), LC25000LUNG (Borkowski ct al., 2019), LC25000COLON (Borkowski et al., 2019), BACH (Aresta et al., 2019), CRC100K (Kather et. al., 2018), OSTEO (Arunachalam et al., 2019), SICAPV2 (Silva-Rodríguez et al., 2020), PCAM (Veeling et al., 2018), and SKINCANCER (Kriegs- mann et al., 2022). The results indicate that even when evaluated on the same dataset and metric, differ- ent MMFM4CPath exhibit their own tendencies. Among them, CPath- Oann, PathGen-CIIN CLIP, and KEEP demonstrate superior and more robust performance across datasets. Table 5 shows the performance of six MMFM4CPath models on the ARCH-PUBMED (Gamper & Rajpoot, 2021b) and ARCH-BOOk (Gamper & Rajpoot, 2021b) datasets for tile-to-caption retrieval and caption-to-tile tile retrieval, measured by Top-K Recall (R@K, { {5, 10, 50}). We observe that KEEP achieves the best results on Book, PathCLIP shows stronger performance on ARCH-PUBMED, while MLLM4PUE delivers more robust and overall stronger re- trieval ability across both datasets. Establishing a unified evaluation standard that specifies consistent tasks, datasets, and metrics for benchnarking different MMFM4CPath remains an urgent and important direction for future research, which we further discuss in Section 6.
6 Future Directions
Developing MMFM4CPath Integrating H&E Images with Spatial Omics. The integration of H&F-stained histopathology images with spatial omics data, such as spatial transcriptomies and proteomics, represents a promising frontier in computational pathology. By coupling morphological context with spatially resolved molecular signatures, future multi-modas foundation models could enable precise cellular localization of gene and protein expression, bridging the gap between tissue architecture and molecular mechanisms. Developing such models would roquire addressing challenges like data sparsity, spatial resolution mismatch, and aligument between modalities, but could significantly enhance our understanding of disease heterogeneity and microenvironmental interactions.
Developing MMFM4CPath to Predict MxIF Markers from H&E Images. A compelling dircc- tion involves using H&E images to predict marker expressions captured by multiplexed immunoftuurescencee (MxTF), enabling cost-effective and scalable estimation of protein-level biomarkers. This line of research leverages the morphological cues from H&E to infer high-dimensional proteomic data, potentially reducing the need for expensive MxIF experiments. Multi-modal foundation models trained with paired H&IE-MxIF data could facilitate virtual staining or marker imputation, supporting downstream tasks such as subtyping, immune landscape assessment, and therapy response prediction in a non-invasive manner.
Toward Multi-Modal Integration Beyond Pairs. Most existing FM4CPath focus on pairwise modality seiion-geee gene expression. An important future direction is to extend this paradigm toward integration of three or more modalities (c.g., langon-lnngago-gge gene expression or vi- sion knowledge graph omies), which would enable more comprehensive modeling of pathological phenotypes by jointly capturing morphological, molecular, and semantic signals. While this higher-order integration cur- rently faces significant challenges, including data sparsity, alignment complexity, and increased computational cost, it represents a natural evolution for the field. Progress in multi-modal pretraining frameworks, data harmonization, and scalable optimization techniques may gradually make such integration feasible.
alignment, such as vision language or vision
Advancing Trustworthy Multi-Modal CPathFMs. An important yet underexplored direction is im- proving the trustworthiness of multi-modal foundation models in computational pathology. Beyond safe- guarding patient privacy, especially when models incorporate clinical text or genomic data, future work must
13

---

Published in Transactions on Machine Learning Research (11/2025)
also address fairness concerns (Shao et al., 2024; Lin et al., 2024), such performance disparities (Li et al., 2024) arising from imbalanced datasets across demographics or disease subtypes. Incorporating privacy- preserving learning, rigorous de-identification, and fairness-aware evaluation will be essential for ensuring responsible and reliable deployment of multi-modal CPathFMs.
Standardized Benchmarking for MMFM4CPath, As the field matures, there a pressing need establish standardized metries and unified benchmarks for evaluating MMFM4CPath Current evaluations are fragmented across tasks, modalities, and datasets, limiting comparability and reproducibility. Future work should focus developing comprehensive evaluation protocols that span classification, retrieval, gen- eration, and VQA across tile- and WSI-level inputs. Such efforts would guide model development, ensure fair comparisons, and accelerate the translation of multi-modal models into clinical practice.
7 Conclusion
In this survey, we have systematically reviewed the recent advances in multi-modal foundation models for computational pathology, focusing on three major paradigms: vision-language, vision-knowledge graph, and vision-genc expression models. We categorized 34 statc-of-the-art models, analyzed 30 multi-modal datasets, and summarized key ream lasks and evaluation strategies. Our comprehensive comparison highlights the growing impact and promise of integrating diverse data modalities in computational pathology.
Acknowledgment
Guihong Wan is supported by the National Cancer Institute of the National Institutes of Health (NTH) under Award Number K99CA286966. The work done by Yi He has been supported in part by the National Science Foundation (NSF) under Grant Numbers TIS- 11S-2905,,,, TTS-2441449, TOS-2446522, and the Commonwealth Cyber Initiative (CCI). The work done by Zhong Chen has been supported in part by an Illinois Innovation Network (IIN) sustaining Illinois seed funding grant.
References
Gpt-4v(ision) system card. 2023. URL https://api. http:://app..smmmaiicccchooooo oww.poouaa...oorrrouuI.uou.. 263218031..
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, el al. Gpt-4 technical report. arXie preprint arXiv:2303.08771 2023.
Faruk Ahmed, Andrew Sellergren, Lin Yang, Shawn Xu, Boris Babenko, Abbi Ward, Niels Olson, Arash Mohtashamian, Yossi Matias, Greg S Corrado, et al. Pathalign: A vision-language model for whole slide images in histopathology. arXiv preprinl 2rv0 19578, 2024.
Maximilian Alber, Stephan Tietz, Jonas Dippel, Timo Milbich, Timothée Lesort, Panos Korfiatis, Moritz Krügener, Beatriz Perez Cancer, Neelay Shah, Alexander Möllers, et al. A novel pathology foundation model by mayo clinic, charit''e, 'e, and aignostics. arXiv preprint XrXi:22511005000 2025.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint wrXiv:2305.10400 2023.
Guilherme Aresta, Teresa Araújo, Scotty Kwok, Sai Saketh Chennamsotty, Mohammed Safwan, Varghese Alex, Bahram Marami, Marcel Prastawa, Monica Chan, Michael Donovan, et al. Bach: Grand challenge breast cancer histology images. Medical image analysis, 56:122-1333 139, 2019.
Harish Babu Aruncchalam, Rashika Mishra, Ovidiu Daescu, Kevin Cederberg, Dinesh Rakheja, Anita Sen- gupta, David Leonard, Rami Hallac, and Patrick Leavey. Viable and necrotic tumor assossment from whole slide images of ostcosarcoma using machino-learning and deep-learning models. PloS one, 14(4): e0210706, 2019.
14

---

Published in Transactions on Machine Learning Research (11/2025)
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In European conference on computer vision, pp. 456 473. Springer, 2022.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vi: A versatile vision-language model for understanding. localization. Text Reading, and Beyond, 2, 2023.
Mohsin Bilal, Manahil Raza, Youssel Altherwy, Anas Alsuhaibani, Abdulrahman Abduljabbar, Fahdah Almarshad, Paul Golding, Nasir Rajpoot, et al. Foundation models in computational pathology: A review of challenges, opportunities, and impact. arXiv preprint AXXu:2502208333 2025.
Olivier Bodenreider. The unified modical language system (umls): integrating biomedical terminology. Nucleic acids research, 32(suppl HED266-DD700 2004.
Andrew A Borkowski, Marilyn M Bui, L Brannon Thomas, Catherine P Wilson, Lauren A DeLand, and Stephen M Mastorides. Lung and colon cancer histopathological image dataset (Ic25000). arXiv preprint arXi::191..12142, 2019.
Dibaloke Chanda, Milan Aryal, Nasim Yahya Soltani, and Masoud Ganji. A new era in computational pathology: A survey on foundation and vision-language models. arXiv preprint arXiv:2408. 14496, 2024.
Kailao Chen, Mianxin Liu, Fang Yan, Lei Ma, Xiaoming Shi, Lilong Wang, Xiaosong Wang, Lifeng Zhu, Zhe Wang, Mu Zhou, et al. Cost-effective instruction learning for pathology vision and language analysis. arxiv preprint ar orXic:2407.17334 2024a.
Pingyi Chen, Chenglu Zhu, Sunyi Zheng, Honglin Li, and Lin Yang. Wsi-vqa: Interpreting whole slide images by generative visual question answering. In European Conference on Computer Vision, pp. 401- 417. Springer, 2024b.
Richard J Chen, Chengkuan Chen, Yicong Li, Tiffany Y Chen, Andrew D Trister, Rahul G Krishnan, and Faisal Mahmood. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In Proceedings of the IEEE/CVE /CF conference on computer vision and pattern recognition, pp. 16144-16155, 2022.
Richard J Chen, Tong Ding, Ming Y Lu, Drew FK Williamson, Guillaume Jaume, Andrew H Song, Bowen Chen, Andrew Zhang, Daniel Shao, Muhammad Shaban, et al. Towards general-purposc foundation model for computational pathology. Nature Medicine, 2024c.
Weiqing Chen, Pengzhi Zhang, Tu N Tran, Yiwei Xiao, Shengyu Li, Vrutant V Shah, Hao Cheng, Kristo- pher W Brannan, Keith Youker, Li Lai, et al. A visual- omics foundation model to bridge histopathology with spatial transcriptomics. Nature Methods, pp. 1-15, 2025.
Ying Chen, Guoan Wang, Yuanfeng Ji, Yanjun Li, Jin Ye, Tianbin Li, Bin Zhang, Nana Pei, Rongshan Yu, Yu Qiao, et al. Slidechat: A large vision-language assistant for whole-slide pathology image understanding. arXiv preprint arXio:2410..1110 11761, 2024d.
Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chaigpt quality. See htlpss//vicuna. lmsys. org (accessed 14 April 2023), 2(3):6, 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling Journal of Machine Learning Research, 25(70):1 53, 2024.
language models.
GTEx Consortium, Kristin G Ardlic, David S Deluca, Ayellct V Segrè, Timothy J Sullivan, Taylor R Young, Ellen T Gelfand, Casandra A Trowbridge, Julian B Maller, Taru Tukiainen, et al. The genotype-tissue expression (gtex) pilot analysis: multitissue gene regulation in humans. Science, 348(6235):648 660, 2015.
15

---

Published in Transactions on Machine Learning Research (11/2025)
Haotian Cui, Chloe Wang, Hassaan Maan, Kuan Pang, Fengning Luo, Nan Duan, and Bo Wang. segpt: toward building a foundation model for single-cell multi-omics using generative ai. Nature Melhods, 21 1470 1480, 2024.
Dawei Dai, Yuanhui Zhang, Long Xu, Qianlan Yang, Xiaojing Shen, Shuyin Xia, and Guoyin Wang. Pa-llava: A large language-vision assistant for human pathology image understanding. In 2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 3138 3143. IEEE, 2024.
Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng, and
Furu Wei. Longnet: Scaling transformers to 1,000,000,000 tokens. arViv preprinl arXix:2307.022 02486, 2023.
Tong Ding, Sophia J Wagner, Andrew H Song, Richard J Chen, Ming Y Lu, Andrew Zhang, Amurag J Vaidya, Guillaume Jaume, Muhammad Shaban, Ahrong Kim, et al. Multimodal whole slide foundation model for pathology. arXim:4411 11 1. 19666, 2024.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning scale. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19358-19369, 2023.
Jevgenij Gamper and Nasir Rajpoot. Multiple instance captioning: Learning representations from histopathology textbooks and articles. In Proceedings of the IEEE/CVF conference on computer vision and pollern recognition, pp. 16549- 16559, 2021a.
Jevgenij Gamper and Nasir Rajpoot. Multiple instance captioning: Learning representations from histopathology textbooks and articles. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp. 16549-16559, 2021b.
Rushin H Gindra, Yi Zheng, Emiiy Green, Mary E. Reid, Sarah A Mazzilli, Daniel T Merrick, Eric J Burks, Vijaya B Kolachalama, and Jennifer E Beane. Graph perceiver network for lung tumor and bronchial premalignant lesion stratification from histopathology. The American Journal of Pathology, 194 (7):1285-122 1293, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The lama 3 herd of models. arXiv preprint Xic:24407.21188 2024.
Yu Gu, Robert Timn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing. ACM Trunsactions on Computing for Healtheare (HEALTH), 3(1):1 23, 2021.
Xianchao Guan, Zheng Zhang, Yifeng Wang, and Yongbing Zhang. A systematic review multimodal large language models (mllms) in computational pathology. Authorea Preprints, 2025.
Zhengrui Guo, Jiabo Ma, Yingxue Xu, Yihui Wang, Liansheng Wang, and Hao Chen. Histgen: Histopathol- ogy report generation via local-global feature encoding and modal context interaction. In Inter- national Conference on Medical Image Computing and Computer-Assistar Intervention, pp. 189-199. Springer, 2024.
Chu Han, Xipeng Pan, Lixu Yan, Huan Lin, Bingbing Li, Su Yao, Shanshan Lv, Zhenwei Shi, Jinhai Mai, Jiatai Lin, et al. Wsss4hadd Grand challenge on weakly-supervised tissue semantic segmentation for lung adenocarcinoma. arXiv preprinl 2x44:2204.00455, 2022.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Xuehai He, Yichen Zhang, Luntian Mou, Erie Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint 2tt::2033112880 2020.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
16

---

Published in Transactions on Machine Learning Research (11/2025)
Zhi Huang, Federico Bianchi, Mert Yuksekgomul, Thomas Montine, and James Zou. A visual languaee foundation model for pathology image analysis using medical twitter. Nature medicine, 2023.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, ct al. Qwen2. 5-coder technical report. arXie preprint orXiv:2409. 12180, 2024.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. or Xiv preprint 2xx::2210.21227 2024.
Wisdom Ikezogwo, Saygin Seyfioglu, Fatemch Ghezloo, Dylan Geva, Fatwir Sheikh Mohammed, Pavan Ku- mar Anand, Ranjay Krishna, and Linda Shapiro. Quilt-Im: One million image-text pairs for histopathol- ogy. NeurIPS, 2024.
Maximilian sss,, Jakub Tomezak, and Max Welling. Attention-based doep multiple instance learning. In International conference on machine learning, pp. 2127 2136. PMLR, 2018.
Ana-Maria Istrate, Donghui Li, Dario Taraborelli, Michaela Torkar, Boris Veytsman, and Ivana Williams. A large dataset of software mentions in the biomedical literature. arXiv preprint Xix: 00699, 2022.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In ICML. PMLR, 2021.
Guillaume Jaume, Lukas Oldenburg, Anurag Vaidya, Richard J Chen, Drew FK Williamson, Thomas Peeters, Andrew H Song, and Faisal Mahmood. Pranseritoomics-uuddee slide representation learning in computational pathology. In CVPR, 2024.
Jakob Nikolas Kather, Niels Halama, and Alexander Marx. 100,000 histological images of human colorectal cancer and healthy tissue. (No Title), 2018.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. ar preprinl XrXiv: 02907, 2016.
Katharina Kriegsmann, Frithjof Lobers, Christiane Zgorzelski, Joerg Kriegsmann, Charlotte Janssen, Rolf Rüdinger MeliB, Thomas Muley, Ulrich Sack, Georg Steinbuss, and Mark Kriegsmann. Deep learning for the detection of anatomical tissue structures and neoplasms the skin on scanned histopathological tissue sections. Frontiers in Oncology, 12:1022967, 2022.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinfor- matics, 36(4):1234- 1240, 2020.
Dong Li, Chen Zhao, Minglai Shao, and Wenjun Wang. Learning fair invariant representations under covariate and correlation shifts simultaneously. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp. 1174-1183, 2024.
Dong Li, Guihong Wan, Xintao Wu, Xinyu Wu, Ajit J Nirmal, Christine G Lian, Peter K Sorger, Yevgeniy R Semenov, and Chen Zhao. A survey on computational pathology foundation models: Datasets, adaptation strategics, and ovaluation tasks. arXiv preprint NNi:250111572. 2025a.
Dong Li, Xujiang Zhao, Linlin Yu, Yanchi Liu, Wei Cheng, Zhengzhang Chen, Zhong Chen, Feng Chen, Chen Zhao, and Haifeng Chen. Solverllm: Leveraging test-time scaling for optimization problem via Ilm-guided search. arXiv preprint Xrxim:2510. 16916, 2025b.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping languago-imago pre-training with frozen image encoders and large language models. In International conference on machine learning. pp. 19730-191 19742. PMLR, 2023.
17

---

Published in Transactions on Machine Learning Research (11/2025)
Yuci Liang, Xinheng Lyu, Meidan Ding, Wenting Chen, Jipeng Zhang, Yuexiang Ren, Xiangjian He, Song Wu, Sen Yang, Xiyue Wang, et al. Wsi-llava: A multimodal large language model for whole slide image. arXiv preprint arXie:2412.02141 2024.
Yujie Lin, Dong Li, Minglai Shao, Guihong Wan, and Chen Zhao. Fade: Towards fairness-aware gen- cration for domain generalization via classifier-guides score-based diffusion models. arXiv preprint WwXiv:2406009490 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lec. Visual instruction tuning. Advances in neural information processing systems, 36:34892-33 34916, 2023.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Ming Y Lu, Bowen Chen, Andrew Zhang, Drew FK Williamson, Richard J Chen, Tong Ding, Long Phi Le, Yung-Sung Chuang, and Faisal Mahmood. Visual language pretrained multiple instance zero-shot transfer for histopathology images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19775, 2023.
Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Tvy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Long Phi Le, Georg Gerber, et al. A visual-lanuuage foundation model for computational pathology. Nature Medicine, 2024a.
Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Melissa Zhao, Aaron K Chow, Kenji Ikemura, Ahrong Kim, Dimitra Pouli, Ankush Patel, et al. A multimodal generative ai copilot for human pathology, Nature, 634(8033):4664411) 2024b.
Ruben T Lucassen, Sander PJ Moonemans, Tijn van de Luijtgaarden, Gerben E Breimer, Willeke AM Blokx, and Mitko Veta. Pathology report generation and multimodal representation learning for cutancous melanocytic lesions. Xiv preprint or arx:2250 0025502.192900 19293, 2025.
Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tic-Yan Yan Liu. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in bioinformatics, 23(6):bbac409, 2022.
Weimin Lyu, Qingqiao Hu, Kehan Qi, Zhan Shi, Wentao Huang, Saumya Gupta, and Chao Chen. Efficient whole slide pathology vqa via token compression, arXiv preprint XXhv22507.14497, 2025.
Reza Mirzazadeh, Zaneta Andrusivova, Ludvig Larsson, Phillip T Newton, Leire Alonso Galicia, Xesús M Abalo, Mahtab Avijgan, Linda Kvastad, Alexandre Denadai-Souza, Nathalie Stakenborg, et al. Spatially resolved transcriptomic profiling of degraded and challenging fresh frozen samples. Nature Communica- tions, 14(1):509, 2023.
Mieko Ochi, Daisuke Komura, and Shumpei Ishikawa. Pathology foundation models. JMA journal, 8(1): 121-130 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML. PMLR, 2021.
Lynn Marie Schriml, Cesar Arze, Suvarna Nadendla, Yu-Wei Wayne Chang, Mark Mazaitis, Victor Felix, Gang Feng, and Warren Alden Kibbe, Disease ontology: a backbone for disease semantic integration. 2012.
Nucleic acids research,
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open largo-scale dataset for training next generation image-text models. Advances in neural information processing systems, 35:25278-25285 2022.
18

---

Published in Transactions on Machine Learning Research (11/2025)
Mehmet Saygin Seyfioglu, Wisdom O Ikezogwo, Fatemch Ghezloo, Ranjay Krishna, and Linda Shapiro. Quilt-Ilava:
Visual instruction tuning by extracting localized narratives from open-source histopathology videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
13192, 2024.
George
Shaikovski, Adam Casson, Kristen Severson, Eric Zimmermann, Yi Kan Wang, Jeremy D Kunz, Juan A Relamero, et al. Prism: A multi-modal generalive loundation model for slide-level histopathology. arXiv:2405 10254, 2024.
Minglai Shao, Dong Li, Chen Zhao, Xintao Wu, Yujie Lin, and Qin Tian. Supervised algorithmic fairness in distribution shifts: A survey. arXiv preprint 200::2402.01327, 2024.
Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, et al. Transmil: Trans- former based correlated multiple instance learning for whole slide image classification. Advances in neural information processing systems, 34:2136-211 2147, 2021.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The maasely-ggted mixture-of-experts layer. arXiv preprint arXiv:1701.0633 2017.
Julio Rodrígez,,, Adrián Colomer, María A Sales, Rafael Molina, and Valery Naranjo. Going deeper through the gleason scoring scale: An automatic end-to-end system for histology prostate grading and cribriform pattern detection. Computer methods and programs in biomedicine, 195:105637, 2020.
Shenghuan Sun, Alexander Schubert, Gregory M Goldgof, Zhiqing Sun, Thomas Hartvigsen, Atul J Butte, and Ahmed Alaa. Dr-llava: Visual instruction luning with symbolic clinical grounding. arXiv preprinl arXiv:2405. 195667 2024a.
Yuxuan Sun, Yixuan Si, Chenglu Zhu, Xuan Gong, Kai Zhang, Pingyi Chen, Ye Zhang, Zhongyi Shui, Tao Lin, and Lin Yang. Cpath-omni: A unified multimodal foundation model for patch and whole slide image analysis in computational pathology. arxiv preprint arXiv:24 12. 12077, 2024b.
Yuxuan Sun, Hao Wu, Chenglu Zhu, Sunyi Zhong, Qizi Chen, Kai Zhang, Yunlong Zhang, Dan Wan, Xiaoxiao Lan, Mengyue Zheng, et al. Pathmmu: A massive multimodal expert-level benchmark for understanding and reasoning in pathology. In European Conference on Computer Vision, pp. 56-73. Springer, 2024c.
Yuxuan Sun, Yunlong Zhang, Yixuan Si, Chenglu Zhu, Zhongyi Shui, Kai Zhang, Jingxiong Li, Xingheng Lyu, Tao Lin, and Lin Yang. Pathgen-1.6 m: 1.6 million pathology image-text pairs generation through multi-agent collaboration. arXiv preprint Xix:2 00.00000,00008, 2024d.
Yuxuan Sun, Chenglu Zhu, Sunyi Zheng, Kai Zhang, Lin Sun, Zhongyi Shui, Yunlong Zhang, Honglin Li, and Lin Yang. Pathasst: A generative foundation ai assistant towards artificial general intelligence of pathology. In AAAT, 2024e.
Katarzyna Tomezak, Patrycja Cverwińska, and Maciej Wiznerowicz. Review the cancer genome atlas (toga) ): an immeasurable source of knowledge. Contemporary 2015.
Onkologia, 2015(1):68 77,
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothéc Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXim:2302.1199 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. Xév preprint arXiv:2307.09288, 2023b.
Manuel Tran, Paul Schmidle, Ruifeng Ray Guo, Sophia J Wagner, Valentin Koch, Valerio Lupperger, Brenna Novotny, Dennis H Murphree, Heather D Hardway, Marina D' Amato, et al. Generating dermatopathology reports from gigapixel whole slide images with histogpt. Nature Communications, 16(1):4886, 2025.
19

---

Published in Transactions on Machine Learning Research (11/2025)
Anurag Vaidya, Andrew Zhang, Guillaume Jaume, Andrew H. Song, Tong Ding, Sophia J Wagner, Ming Y Lu, Paul Doucet, Harry Robertson, Cristina Almagro-Perez, et al. Molecular-driven foundation model for oncologic pathology. arXiv preprint 0rXi::2501.1652,, 2025.
Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling, Rotation equivariant cmns for digital pathology. In International Conference on Medical image computing and computer-assistiee intervention, pp. 210-218. Springer, 2018.
Eugene Vorontsov, Alican Bozkurt, Adam Casson, George Shaikovski, Michal Zelechowski, et al. Virchow: A million-slide digital pathology foundation model. arXiv:2309007778 2023.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, et al. Image as foreign language: Beit pretraining for all vision and vision-language tasks. arXiv:2208.104 10442 2022a.
Xiao Wang, Guangyao Chen, Guangwu Qian, Pengcheng Gao, Xiao Yong Wei, Yaowei Wang, Yonghong Tian, and Wen Gao. Large-scale multi-modal pre-trained models: A comprehensive survey. Machine Intelligence Research, 20(4):447-422, 2023.
Xiyue Wang, Sen Yang, Jun Zhang, Minghui Wang, Jing Zhang, Wei Yang, Junzhou Huang, and Xiao Han. Transformer-based unsupervised contrastive learning for histopathological image classification. Medical image analysis, 2022b.
Xiyue Wang, Junhan Zhao, Eliana Marostica, Wei Yuan, Jietian Jin, Jiayu Zhang, Ruijiang Li, Hongping Tang, Kanran Wang, Yu Li, et al. A pathology foundation model for cancer diagnosis and prognosis prediction. Nature, 2024.
Yang Wang. Survey on deep multi-modal data analyties: Collaboration, rivalry, and fusion. ACM Transac- tions on Multimedia Computing, Communications, and Applications (TOMM), 17(1s):1-25, 2021.
Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and S Yu Philip. Multimodal large language models: A survey. In 2023 IEEE International Conference on Big Data (BigDala), pp. 2247 2256. IEEE, 2023.
Xiaomin Wu, Rui Xu, Pengehen Wei, Wenkang Qin, Peixiang Huang, Ziheng Li, and Lin Luo. Pathinsight: Instruction tuning of multimodal datasets and models for intelligence assisted diagnosis in histopathology. arXiv preprint 2ix:224 08.07777 2024.
Peng Xia, Kangyu Zhu, Haoran Li, Tianze Wang, Weijia Shi, Sheng Wang, Linjun Zhang, James Zou, and Huaxiu Yao. Mmed-rag: Versatile multimodal rag system for medical vision language models. arxiv preprint arXiv:2410. 13085, 2024.
Jinxi Xiang, Xiyue Wang, Xiaoming Zhang, Yinghua Xi, Feyisope Eveje, Yijiang Chen, Yuchen Li, Colin Bergstrom, Matthew Gopaulchan, Ted Kim, et al. A vision- language foundation model for precision oncology. Nature, 2025.
Yingxue Xu, Yihui Wang, Fengtao Zhou, Jiabo Ma, Shu Yang, Huangjing Lin, Xin Wang, Jiguang Wang, Li Liang, Anjia Han, et al. A multinodal knowledggeenhanceed whole-slide pathology foundation model. arXiv preprint ar X4:: 15362, 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint or xixi:24412.15115, 2024.
Fan Yang, Wenchuan Wang, Fang Wang, Yuan Fang, Duyu Tang, Junzhou Huang, Hui Lu, and Jianhua Yao. scbert as a large-scale pretrained deep language model for cell type annotation of single-cell rna-seq data. Nature Machine Intelligence, 4(10):852-86) 2022.
Jiahui Yu, Zirui Wang. Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Xiv preprint 2io:2205500117. 2022.
20

---

Published in Transactions on Machine Learning Research (11/2025)
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- training. In Proceedings of the JEEE/CVT international conference on compuler vision, pp- 11975-11988 2023.
Kai Zhang, Rong Zhou, Fashan Adhikarla, Zhiling Yan, Yixin Liu, Jun Yu, Zhengliang Liu, Xun Chen, Brian D Davison, Hui Ren, et al. A generalist vision-language foundation model for diverse biomedical lasks. Nature Medicine, pp. 1 13, 2024.
Pan Zhang, Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Haodong Duan, Songyang Zhang, Shuangrui Ding, et al. Interniinxxcomppser:: vision-language large model for advanced text-image comprehension and composition. arXiv preprint arXi::2300.15112, 2023a.
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, al. Biomedelip: a multimodal biomodical foundation model pretrained from fifteen million scientifie image-text pairs. ar preprinl orXiv:2303.000 00915, 2023b.
Qiwei Zhao, Dong Li, Yanchi Liu, Wei Cheng, Yiyou Sun, Mika Oishi, Takao Osaki, Katsushi Matsuda, Huaxiu Yao, Chen Zhao, et al. Uncertainty propagation on Ilm agent. In Proceedings of the 63rd Annual Meeting of the Association for Compulational Linguisties (Volume 1: Long Papers), pp. (064-6073, 2025.
Theodore Zhao, Yu Gu, Jianwei Yang, Naoto Usuyama, Ho Hin Lee, Sid Kiblawi, Tristan Naumann, Jianfeng Gao, Angela Crabtree, Jacob Abel, et al. A foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalitics. Nature methods, pp. 1-11, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Newral Information Processing Systems, 36:46595 -46123,, 2023.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXi::211.00832, 2021.
Qifeng Zhou, Thao M Dang, Wenliang Zhong, Yuzhi Guo, Hehuan Ma, Saiyang Na, and Junzhou Huang. Milm4puc: Toward universal embeddings in computational pathology through multimodal Ilms. arXiv preprinl Xr0:: 07221, 2025.
Xiao Zhou, Luoyi Sun, Dexuan He, Wenbin Guan, Ruifen Wang, and Lifeng others Wang. A knowledge- enhanced pathology vision-language foundation model for cancer diagnosis. 24W:2412.11106 2024a.
Xiao Zhou, Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Weidi Xie, and Yanfeng Wang. Knowledge-enhancel visual-language pretraining for computational pathology. In ECCV. Springer, 2024b.
Eric Zimmermann, Eugene Vorontsov, Julian Viret, Adam Casson, Michal Zelechowski, al. Virchow2: Scaling self-supervised mixed magnification models in pathology. arXiv:2408. 007388 2024.
21

---

Published in Transactions on Machine Learning Research (11/2025)
Appendix
A Survey Methodology
To cnsure transparency and reproducibility of our survey, we briefly describe how the reviewed papers and datasets were identified and selected. We outline the literature databases searched, the main search terms used, the time period covered, and the inclusion /exclusion criteria applied. This methodological overview clarifies the scope of our survey and allows readers to assess its completeness and potential sources of bias.
Databases scarched: arXiv, medRxiv, bioRxiv, Google Scholar, ACM Digital Library, PubMed, DBLP, Europe PMC.
Search terms: "computational pathology foundation model", "multi-modal computational pathol- ogy", "vision-language foundation model for computational pathology", "vision-omics foundation model for computational pathology", "large language model for computational pathology", "foun- dation model for computational pathology", etc.
Date range: Jamuary 2022 September 2025.
Inclusion/exclusion criteria: This survey focuses on multi-modal foundation models (FMs) de- veloped specifically for computational pathology (CPath), with an emphasis models built upon hemaloxylin and eosin (H&E) stained whole-slide images (WSIs) and tile-level representalions. We review 34 state-of-the-art models that integrate pathology images with auxiliary modalities such as textual reports, knowledge graphs, and molecular profiles, categorizing them into vision language, vi- Noon-knowledge graph, and vision-gens expression paradigms. In addition, we analyze 30 pathology- specific multi-modal datasets, grouped into image-text pairs, instruction datasets, and image other modality pairs, and summarize the evaluation tasks and strategies most relevant to CPath foundation models. Several related directions are excluded from the scope of this survey. Specifically, hods that extend beyond pathology to broader biomedical imaging, including Computed Tomography (CT), Magnetic Resonance Imaging (MRI), and X-ray (Zhang et al., 2023b; 2024; Zhao et al., 2024;
Xia et al., 2024), are not covered in detail, as their primary goal is to build universal medical imaging models rather than enhance pathology image representation. Similarly, we do not comprehensively review purppse multi-modal large language models (MLLMs) that incorporate pathology data only as a small subset of training, since their emphasis lies in broader generative AI capabilities rather than pathology-speciff representation learning. By clearly defining these boundaries, we aim to provide a focused and coherent review of foundation models for computational pathology while acknowledging related but out-of-scope rescarch directions.
B Risks of Automatic and LLM-based Evaluation in Generation Tasks
In generation tasks such diagnostic VQA and pathology report generation, conventional automatic metrics (e.g., METFOR, BLEU, ROUGE) or even L.Maas-judge evaluations present several risks. Biases can arise because these metrics tend to reward surface-form fluency or frequent stylistic templates, leading to inflated scores for outputs that look plausible but misrepresent rare or clinically critical findings. Hallucinations, the fabrication of entities such as tumor grade, margin status, or molecular markers, may be overlooked by lexical overlap-based metrics and even tolerated by LLM judges if phrased fluently, despite being clinically unsafe. Dataset leakage and contamination pose further risks when near-duplicate reports or Q&A items exist across training and evaluation sets, when LLM judges share training sources with the systems under evaluation, potentially inflating performance through circular validation. These limitations highlight the need for complementary expert review and contamination evaluation protocols to ensure that generation tasks are assessed for clinical factuality rather than superficial textual similarity.
Beyond automatic and LLM-bbaod metries, several recent studics have incorporated pathologist based eval- uations for diagnostic VQA and report generation. For example, PathChat was evaluated on 260 open-ended diagnostic VQA questions by a panel of seven board-certified pathologists, who graded the correctness and
22

---

Published in Transactions on Machine Learning Research (11/2025)
ranked the quality of model responses, revealing substantial gaps between human judgment and automatic metrics. Similarly, HistoGPP, a model for whole-slide report generation, was assessed by certified der- matopathologists, who judged whether generated reports captured the essential clinical findings and flagged factual inaccuracics. Both studies underscore that human experts prioritize clinical factuality and action- ability over surface overlap, frequently identifying omission errors such as missing tumor stage, commission errors such as hallucinated findings, or semantic inversions such as "invasive" vs. "in situ." While pathologist review is costly and less scalable, remains the most reliable safeguard against subtle yet clinically critical errors. Taken together, these observations highlight the necessity of integrating structured expert review into future benchmarks, where automatic metrics and human-in-the-lopp evaluation should be reported jointly to provide a more faithful assessment of generative models in computational pathology.
23


Reviews:
[{"review_id": "8GcDxRdsAa", "reviewer": "Reviewer_8caN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors provide a review of large-scale (ie. \"foundation model) deep-learning methods for computational pathology. In particular, this work provides a taxonomy of models and training methods for these tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I am not familiar with the field of computational pathology, but at least to me it seems that it provides an extensive list and description of other studies and datasets, tabulated with annotations about methodology and public availability. The work also provides an overview of the different methods, often focusing on LLMs or models trained with cross-modal contrastive learning (ie. visual-language models, like CLIP).", "audience_yn": "Yes", "audience_explanation": "Yes, the content of this work is clearly relevant for certain groups of ML practitioners, as well as researchers interested in foundation models, since they are often benchmarked across many modalities.", "requested_changes": "The article does not discuss in sufficient detail the performance of these foundation models in practical tasks. Given that the purpose of foundation models is to be pretrained on a large body of data to improve performance on the actual desired task, I am wondering (as, again, I am unfamiliar with the field) if it is possible to include what the performance of the referenced models are on various downstream tasks, so that readers can get an overview of how well the different approaches do. For example, is there a standard benchmark task (or set of tasks) that papers commonly report performance on? If so, it may be useful including their reported performance. If not, it may still be valuable for a survey to inform readers of the scope and/or limitations of the other works referenced here, for a better broad picture of the state of the field (since Fig. 3 already includes information about which models were used for what tasks, it seems reasonable to suggest that the performance on each task be listed). \n\nIncluding information about performance might be important because self-supervised learning and foundation models are often employed to improve performance over direct training on the desired task, which means that readers would obviously be interested to see this information. Because of this, it may be convenient to review some works that use approaches without foundation models as a \"baseline\" for progress in the field.", "broader_impact_concerns": "N/A", "additional_comments": "Due to my unfamiliarity with the field of computational pathology, I am unable to provide a strong recommendation for this work. However, it does appear to provide a fairly comprehensive coverage within this domain of foundation/ssl-model techniques that are familiar to me.", "extra_scores": {}, "extra_text": {}}, {"review_id": "APwLYRglEK", "reviewer": "Reviewer_Qd8m", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary:**\n\nThis survey reviews multimodal foundation models for computational pathology, focusing on H&E whole-slide images and tile-level inputs. It does four things. First, it groups thirty-two recent models into three families: vision and language (both non-LLM and LLM based), vision and knowledge graph, and vision and gene expression, and compares their architectures, pretraining goals, and adaptation strategies. Second, it compiles twenty-eight datasets, including image-text pairs, instruction datasets, and image-other-modality pairs. Third, it proposes a task taxonomy covering classification, retrieval, generation, segmentation, prediction, and visual question answering at both tile and WSI levels. Fourth, it highlights future directions such as integrating spatial omics, predicting multiplexed immunofluorescence markers from H&E, and creating standardized benchmarks. The survey also provides a visual roadmap of models (Fig. 1), a comparison to prior surveys highlighting broader coverage (Table 1), a large model pretraining/architecture table (Table 2), a dataset catalogue (Table 3), and a task taxonomy diagram (Fig. 3).\n\n**Strengths:**\n- S1: The work covers image-and-text models, and it also includes models that link images to medical knowledge graphs and to gene-expression data. It reviews 32 models and 28 datasets, which is quite exhaustive.\n- S2: This survey clearly explains how models are trained (for example, CLIP, CoCa, BLIP-2, iBOT), how instruction tuning is done, and how they are evaluated (zero-shot or few-shot, WSI or tile), which makes it easier to compare papers that use different setups.\n- S3: The paper groups evaluations into six task types (classification, retrieval, generation, segmentation, prediction, and VQA) and shows whether results are at the tile level or the whole-slide image (WSI) level, with Figure 3 and the nearby text guiding new readers.\n- S4: The paper lists useful resources, including image and text pairs, instruction datasets, and collections that link images with other data such as pathology knowledge graphs and WSI with omics. This provides readers an easy starting point to find training data and benchmarks.\n\n**Weaknesses:**\n- W1: The survey is well organized, but it has little quantitative comparison across common tasks and datasets. Since it calls for standard benchmarks, add a small, curated table with harmonized metrics on datasets such as SlideBench, WSI VQA, or TCGA, even if the comparisons are imperfect.\n- W2: VQA and report generation are scored mainly by automatic metrics or an LLM-as-judge, and although pathologist reviews are mentioned, the paper lacks a discussion on biases, hallucinations, and dataset leakage.\n- W3: Please include the works _[1,2,3]_ and give the manuscript a careful proofread to remove typos and improve consistency.\n- W4: It’s not explained how the papers were found and chosen, so readers can’t judge completeness or potential bias.\n---\n_[1] Chen, Weiqing, Pengzhi Zhang, Tu N. Tran et al. \"A visual–omics foundation model to bridge histopathology with spatial transcriptomics.\" Nature Methods (2025): 1-15._\n\n_[2] Lyu, Weimin, Qingqiao Hu, Kehan Qi et al. \"Efficient Whole Slide Pathology VQA via Token Compression.\" arXiv preprint arXiv:2507.14497 (2025)._\n\n_[3] Tran, Manuel, Paul Schmidle, Ruifeng Ray Guo et al. \"Generating dermatopathology reports from gigapixel whole slide images with HistoGPT.\" Nature Communications 16, no. 1 (2025): 4886._", "claims_and_evidence_yn": "Yes", "claims_explanation": "The survey paper’s claims are supported by clear tables and figures, including a catalog of 32 models and 28 datasets and a well-structured task taxonomy.", "audience_yn": "Yes", "audience_explanation": "This paper offers a clear, well-organized survey that can orient both newcomers and practitioners. It also highlights concrete future directions and benchmarking needs, making it a handy reference for researchers building models or datasets in this area.", "requested_changes": "1. Add a small table with consistent metrics on common datasets such as SlideBench, WSI VQA, and TCGA to complement the narrative taxonomy.\n2. Provide a brief Methods section listing databases, search terms, date range, and inclusion/exclusion rules, so readers can assess completeness and bias.\n3. Analyze risks with automatic metrics and LLM-as-judge; the paper currently notes qualitative, pathologist-based checks without deeper analysis.\n4. Specify metric choices and default data splits to improve cross-paper comparability alongside the task taxonomy.\n5. Please include the works [1,2,3] in your figures and tables.\n6. Correct typos: \"visio-language\" (in CoCa-based Vision-Language FM4CPath), \"VAQs\" (in Multi-Modal Instruction Datasets for CPath), \"based on the these regions\" (in Evaluation Tasks), \"Instrution\" (Table 1), \"ECVA\" (Table 2 footnote), \"BILP-2\" (Table 2, CLOVER row), \"Qulit\" (multiple places), \"TGCA\" (Table 3).", "broader_impact_concerns": "Not very applicable here.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xIIIGJR3Uc", "reviewer": "Reviewer_9HxA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors list, describe and organize multi-modal foundation models and datasets for computational pathology, which is concerned with using ML and other computational techniques to understand, classify and organize diseased tissues, often by analyzing images of stained tissues. The paper organizes these foundation models into 3 broad types, depending on which pair of modalities are combined: vision-language, vision-knowledge graph and vision-genomic. There is also a distinction made between LLM and non-LLM language modeling Architectures and other modelling detailed of these foundation models are described. The various tasks the models can be used for (classification, prediction,  retrieval, generative applications, etc) , Datasets are organized into image-text, instruction, and image-other modality categories. Future directions are suggests, such as combining these methods with spatial omics so as to understand the locality of the phenotypic consequences of gene expression in particular areas of tissue.\n\nAlthough I'm neither an expert in computational pathology nor in foundation models, my impression is that this survey is quite thorough and comprehensive and the subcategorization of methods should be useful to various researchers.\n\nThe weaknesses I noticed were fairly minor. The paper seems to me to be a little light on analysis of trends of what seems to work well in this area (although perhaps that's because of the lack of consistent evaluation standards in the field, noted by the authors at the end of the paper). The paper leans a little bit too much in the direction of being a laundry list of models and datasets for my taste, at the expense of analysis of what's most important and what seems to work. However, I suppose a survey paper is not the same as a meta-analysis, so I think the paper generally achieves the goals of a survey paper.\n\nThe future directions mentions leaves out what (to me) seems to be an obvious future direction: combining 3 or more modalities (e.g. vision-language-gene expression or vision-knowledge graph-gene expression).\n\nThere are also a few typos which I will list below.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Claims are not really at the center of what makes a survey paper valuable, but the claim that the field of multi-modal foundation model computational pathology is organized into vision-language, vision-knowledge-graph and vision-gene expression seems well supported, with many specific examples of each subtype provided.", "audience_yn": "Yes", "audience_explanation": "Computational pathology is a well-established and important field. Foundation models are a fast-moving and highly active field. As such, it is worthwhile to to collect current research into a survey paper, as the authors have done. This paper will help researchers entering the field to get caught up quickly.", "requested_changes": "Some discussion of the possibility of 3-or-more-modalities as a future direction would be nice. Maybe that idea is unrealistic in the near future, but it's such an obvious idea that I think the paper would benefit from some discussion of it, regardless. \n\nA couple of typos:\n\nIn Table 1, Instrution should be Instruction\n\nAt the bottom of page 9, I'm pretty sure 'VAQ' should be 'VQA'.\n\nThe first sentence of Section 2.2 on page 4 should have a comma rather than a period following \"(SSCL)\".", "broader_impact_concerns": "I don't have serious concerns here.  I suppose that there could be some risk that poorly designed LLMs could reveal confidential details of patients inappropriately ? Maybe some discussion of that would be helpful. I think the paper is OK as-is, though.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)