PathGen-1.6M: 1.6 Million Pathology Image-text Pairs Generation through Multi-agent Collaboration
Authors: Yuxuan Sun
Yunlong Zhang
Yixuan Si
Chenglu Zhu
Kai Zhang
Zhongyi Shui
Jingxiong Li
Xuan Gong
XINHENG LYU
Tao Lin
Lin Yang
Venue: ICLR 2025 Oral (2025)
Keywords: Image-text pairs generation
Vision-language models
Multi-agent collaboration
Primary area: datasets and benchmarks
TL;DR: We present PathGen-1.6M, an open-source large-scale pathology dataset with 1.6M high-quality image-caption pairs, enabling the creation of powerful multimodal models for pathology analysis.

Abstract: Vision Language Models (VLMs) like CLIP have attracted substantial attention in pathology, serving as backbones for applications such as zero-shot image classification and Whole Slide Image (WSI) analysis. Additionally, they can function as vision encoders when combined with large language models (LLMs) to support broader capabilities. Current efforts to train pathology VLMs rely on pathology image-text pairs from platforms like PubMed, YouTube, and Twitter, which provide limited, unscalable data with generally suboptimal image quality. In this work, we leverage large-scale WSI datasets like TCGA to extract numerous high-quality image patches. We then train a large multimodal model (LMM) to generate captions for extracted images, creating PathGen-1.6M, a dataset containing 1.6 million high-quality image-caption pairs. Our approach involves multiple agent models collaborating to extract representative WSI patches, generating and refining captions to obtain high-quality image-text pairs. Extensive experiments show that integrating these generated pairs with existing datasets to train a pathology-specific CLIP model, PathGen-CLIP, significantly enhances its ability to analyze pathological images, with substantial improvements across nine pathology-related zero-shot image classification tasks and three whole-slide image tasks. Furthermore, we construct 200K instruction-tuning data based on PathGen-1.6M and integrate PathGen-CLIP with the Vicuna LLM to create more powerful multimodal models through instruction tuning. Overall, we provide a scalable pathway for high-quality data generation in pathology, paving the way for next-generation general pathology models. Our dataset, code, and model are open-access at https://github.com/PathFoundation/PathGen-1.6M.

Published as a conference paper at ICLR 2025
1ATHGEN-1.6M: 1.6 MILLION PATHOLOGY IMAGE- TEXT PAIRS GENERATION THROUGH MULTI-AGENT COLLABORATION
Yuxuan Sun
Yunlong Zhang
Yixuan Si² Chenglu Zhu Xinheng Lyu?
Kai Zhang
Zhongyi Shui Jingxiong Li Xuan Gong
Tao Lin
Lin Yang
1 Zhejiang University Westlake University
3 Ohio State University 4 Harvard University
5 Center for Interdisciplinary Research and Innovation, Muyuan
ABSTRACT
Vision Language Models (VLMs) like CLIP have attracted substantial attention in pathology, serving as backbones for applications such as zero-shot image classifi- cation and Whole Slide Image (WST) analysis. Additionally, they can function vision encoders when combined with large language models (LLMs) to support broader capabilities. Current efforts train pathology VLMs rely on pathology image-text pairs from platforms like PubMed, YouTube, and Twitter, which provide limited, unscalable data with generally suboptimal image quality. this work, we leverage large-scale WSI datasets like TCGA to extract numerous high-quality image patches. We then train a large multimodal model (LMM) to generate cap- tions for extracted images, creating PathGen-I 1.6M. dataset containing 1.6 million high-quality image-caption pairs. Our approach involves multiple agent models col- laborating to extract representative WSI patches, generating and refining captions to obtain high-quality image-text pairs. Extensive experiments show that integrat- ing these generated pairs with existing datasets train pathology-spede model, PathGen-CLA, CLIP, significantly enhances its ability to analyze pathological im- ages, with substantial improvements across nine pathology-related zero-shot image classification tasks and three whole-slide image tasks. Furthermore, we construct 200K instruction mnstruction-tuning data based on PathGen-1e 1.6M and integrate PathGen-CLA with the Vicuna LLM to create more powerful multimodal models through in- struction tuning. Overall, we provide a scalable pathway for high-quality data generation in pathology, paving the way for next-generation general pathology models. Our dataset, code, and model are apen-access at PathGen-I
CLIP
FLP Quitteet FathcLiP BlomesC.JP PathGen-CC
LLWA 1.5 138
IC kong
Gamera Pro visio
Owen VL. MAX
PathGen 1.6M
IC-Cala
GPT-80-1 10G
PudMod-Al AI
Futhöin
CRCLOOK
SectaParh-T Tims
Quilt 0.7M
Skanaanoe
OpenPath
From
PathCap
NASSHING
Cieritatton
Dsteo
Figure 1: Illustration of the scale of the PathGen dataset (left), the performance of the proposed PathGen-CLIP (middle), and the PathGen-LLaVa (right), both derived from training on PathGen.
"Corresponding authors.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Pathology plays a crucial role in modern medicine as it is the gold standard for disease diagnosis and the selection of treatment methods (Kumar et al. 2014). With the rapid growth of artificial intelligence, there is an increasing interest in developing robust ceneral-purpose models to assist physicians, particularly in pathology. Pathology-speciig CLIP models have demonstrated exceptional
performance in zero-shot image classification (Radford et al. 2021 Jia et al., 2021), multimodal
understanding (Liu et al. 2024 2023; Bai et al. 2023 Li et | 2022 2023b Dai et al. (024))
robustness to various perturbations (Radford et al. 202 Shu et al. 2223 Zhou al. [0022). 2). and
scalability across diverse tasks (Rombach et al. 2022 Lin et al. 2023b Esmacilpour et al. (022))
However, training such models typically requires vast amounts of data. For instance, general CLIP models are trained using massive amount of data from sources like WIT (Radford et al. 2021) and LAION (Schuhmann et al. 2022a), at scales of millions even billions. In pathology, researchers similarly focused on amassing large collections of pathology image-text pairs from various sources, including academic articles from PubMed (Lin et al.
2023a [kezoowo al. 2024 Sun al. 2024b), social media (Huang et al. 2023 Ikezogwo et al. 2024), and books (Caamer & Rajpoot 2021).
Despite these efforts, the largest datasets do not exceed one million samples, which is significantly smaller compared to the scale of natural image datasets. This underscores the challenges in the availability and scalability of pathology-related datasets. Key limitations include: (1) The available pool of image-text pairs from the internet and books are quickly exhausted, hindering scalability. (2) Many collected images suffer from significant quality loss during the acquisition process. For example, images from PubMed articles often undergo compression, while educational content from platforms like YouTube typically consists of screenshots at 1080p resolution, further degraded by video compression, making them incomparable to the high-resolution images used in practical scenarios. (3) Unpaired image-text pairs frequently appear on social media, where users may post pathological images with comments focusing on non-pathological features like aesthetic appeal.
Fortunately, The Cancer Genome Atlas (TCGA) is a comprehensive, publicly funded project that pro- vides clinical data across various cancer types. This dataset includes numerous WSIs, tigh-resolution scans from patient tissue samples. These WSIs inherently contain an immense amount of detailed information-suce
cellular structures, tissue organization, and morphological natternscrrucaa for cancer diagnosis and research. However, these datasets typically only provide labels at the slide level, leaving patches within WSIs without detailed textual annotations. This lack of specific annotations hinders models from learning rich semantic information from such high-quality image data.
In this study, we aim to harness high-quality images from WSIs construct a large-scale image-text dataset. We develop cascaded approach involving multiple agent models that collaborate to extract the most representative patches from WSIs and generate captions describing the visual details within each patch. This process enables us to compile dataset of 1.6 million image-caption pairs, designed to train and enhance
multimodal models. Our main contributions are as follows: (1) We propose PathGen-1.5 1.6M, which is currently the largest and highest-quality pathology image- text dataset, as depicted in Figure [ (2) The construction approach of PathGen- 1.6M offers a scalable solution to expand the currently limited pool of pathology image-text pairs, addressing critical need in the field. (3) Experiments confirm that PathGen-
can significantly boost the performance of existing multimodal models like CLIP and LLaVA in the pathology domain, while scaling up PathGen presents a promising potential as a future pretraining strategy for pathology models.
2 RELATED WORK
Existing Dision-language Datasets. Training vision-lant language models like CLIP (Radford et al. 202T) requires large and high-quality image-text pairs to capture the richness of visual and semantic in- formation. In the general domain, notable datasets have been constructed such as LAION-5B (Schuh mann et al. 2022a WIT-400M (Radford et al.) 202I). 1). In pathology, ARCH (Gamper & Rajpoot 2021) and PathCap (Snn et al. 2024b) datasets are collected from PubMed and medical textbooks, containing 11,816 and 207.000 ,000 pathology image-caption pairs, respectively. The OpenPath (Huang et al. 2023) dataset includes 208,414 pairs from Twitter posts, while the Quilt- 1M (Ikezogwo et al. 2024) dataset gather 768,226 histopathology image-text pairs from video frames and corresponding subtitles on YouTube. These datasets are primarily gathered from social media platforms and text-
2

---

Published as a conference paper at ICLR 2025
books. Although fine-tuning CLIP on these specialized pathology datasets significantly enhances its adaptation to pathology tasks, challenges such as low image quality, image-text misalignment, and poor data scalability continue to hinder the further development of pathology-specoe
CLIP models.
Pathology-specific CLIP Models. CLIP (Radford et al. 202T) is a powerful model that Iearns visual concepts through natural language supervision. In general domains, it demonstrates remarkable capabilities in zero-shot classification, image retrieval, and multimodal understanding by training vast image-caption datasets. This approach leverages language's semantic richness to enhance visual recognition, making it adaptable across various applications without task-specific training. In pathology domain, CLIP's potential is being increasingly recognized as solution to major challenges, such as the scarcity of labeled data and the requirement for domain- expertise. By leveraging natural language descriptions, CLIP facilitates the identification and classification of intricate features that traditionally pose significant challenges for manual annotation. Recently, there has been a notable increase in pathology CLIP variants, including PubMedCLIP (Eslami et al. 2023).
BiomedCLIP (Zhang et al. 2023a), PMC-CLIP (Lin et al. 2023a), Quilt-Net (Ikezogwo et al. 2024 PathCLIP (Sun et al. 2024b). PLIP (Huang et 2023)) and CONCH (Lu et al. 2024).
Large Multimodal Model (LMM). The integration of large language models (LLMs) like GPT- 3 (Brown et al.) 2020). T5 (Raffel et al. 2020), and GPT-4 (OpenAl, 2023a) with vision capabilities has spurred the development of sophisticated multimodal models (LMMs). These LMMs. such as Flamingo (Alayrac et al. 2022). BI.IP-2 (Li et al. 2023b), and Fuyu (Bavishi et al. 2023). excel
in multimodal understanding by utilizing pretraining techniques. Additionally, instruction-tuningg derived from NLP, has been adapted for LMMs, enabling them to generate more controllable and task-specific outputs through datasets like those used in GPT-4V (OpenAI 2023b), Gemini Pro Vision (Team et al. 2023), Qwen-VL (Bai et al., 2023), and InstructB Dai et al.) (023). The application of LMMs in pathology is particularly promising. Models such as PathAsst (Sun et al. 2024b), LLaVA-Med et al. (2023)). Quilt-LLaVA VA (Seyfioglu et al.. (2023) have been developed using curated pathology-specifie instruction-tunii and educational YouTuue videos. These advancements facilitate effective analysis and the generation of descriptive texts for pathological images. Consequently, we leverage this capability to generate corresponding descriptions for image patches within WSIs. By creating high-quality image-text pairs, we aim to enhance the foundational vision-language models in pathology.
datasets sourced from resources like PubMed
Multi-Agent Collaboration. With the advancement of large models (LMs) and the development of specialized models for various tasks, recent research has explored the use of multi-agent collaboration.
This approach allows these models to work together, achieving tasks that are beyond the capabilities of any single model alone. For instance, leveraging L.MSs for role-playing can be used to accomplish
tasks such as software development (Hong al.
2023 Qian al. [023). societal simulation (Park
ct al. 2023 20222). ) policy simulation (Xiao et al. 2023 Hua et al. 2023), game simulation (Xu et al.
2023
Wang et 2023b) and video generation Yuan et 2024
3
PATHGEN DATASET CONSTRUCTION
The entire data construction pipeline is illustrated in Figure 2 We employ multiple agents working collaboratively to generate high-quality pathology image-text pairs. This process involves extracting representative WSI image patches through CLIP-based image retrieval and clustering. These patches then described by a trained pathology LMM agent, followed by another LMM agent and LLM agent that revises and summarizes the descriptions. In this section, we detail the construction of the agent model and explain their collaborative workflow in the data generation process.
3.1 AGENT MODEL PREPARATION
PathGen-CLIP-L General models like OpenAl's CLIP underperform in the pathology domain, necessitating a specialized model for tasks such as cross-modal retrieval we employ in section 3.2 For this purpose, we combine portions of existing datasets, including PathCap (200K), as well as cleaned versions of Quilt- Muilt-IM (400K) and OpenPath (100K), where misaligned image-caption pairs removed from the latter two datasets, resulting in total of 700K samples. We refer to this dataset as PathGen,,.tt- Utilizing the OpenCLIP framework (Tharcoo et al., 2021), we train a CLIP-L version of the pathology-specific CLIP model with 336 input image size, designated as MithGen--I.I-LL...
3

---

Published as a conference paper at ICLR 2025
Representative patch extraction
Description generation
CLIP Image Encoder
Patches
Glaleting
Inital descriptions
Agent
Patch embcddings Cross: modal Retrieval
Similarity arity Filter
Finding Soaiing--eee retrieual
Revised descriptions
Finding embeddings
embeddings
Topı
Agent
Text Encoder
butes
Prompt Agent
report
Altibbute baned reir eval
Summarite Agent
Image refined descriptian pairs
Topk
Representative patch embeddings
Figure 2: Illustration of the multi-agent collaboration pipeline for generating pathology image- text pairs. This process comprises two main components: (1) Representative Patches Extraction, which utilizes prompt-based cross-modal retrieval and clustering; and (2) Description Generation, where multiple LMM and L.M agents employed to generate, revise, and summarize descriptions.
To generate high-quality pathology image-text pairs, require patholoyy-specfiic LMM capable of producing accurate and detailed image captions. Existing image- caption pairs are often too simplistic comprehensively describe these details. hindering the training of more effective description generation models. Inspired by the PathMMU benchmark (Sun et al. 2024a), we sample 30,000 image-caption pairs, with 10,000 pairs each from PathCap, OpenPath, and Quilt- 1M. We provide each image along with its corresponding caption to GPT-4V, allowing the model enhance and refine the original captions by incorporating details observed in the images. This approach enables to generate 30,000 detailed image descriptions. We build upon LLaVA-v1.5. 13B (Liu et al. 2024) by replacing ATenA--CLIP vision encoder with our PathGen-CU MiihGen-CLII---istt We then trained this modified architecture PathGen- LLa dA our matholoyy-specinng description LMM agent. We compare the pathology image description generation capabilities of PathGen-LLLaV..a with those of LLaVA-Medd (Li et al.)
Description LMM Agent:
curated pathology image-descriptions pairs to create
a023a) and Quilt-LLaVA (Seyfioglu et al. 2023) in Appendix B.1
Revise LMM Agent: The Revise Agent is pathology LMM built the LLa frame- work (Liu et al., 2024). designed with error-correction capabilities. While existing pathology LMMs trained only for limited tasks, such as multiple-choice questions, dialogue, and description, they lack the capabilities to perform self-correction. To achieve this, we leverage descriptions generated by the description LMM Agent and prompt GPT-4 to systematically introduce controlled inaccuracies through three defined operations: add, delete, or edit. By reversing these operations caciattioonndddinn becomes deleting, deleting becomes adding, and edits are inverted enables us to create a multimodal caption editing dataset of (image, caption, editing operation) triplets. These derived operations, pre- and post-modification descriptions, and corresponding images are then employed to train the Revise LMM Agent, thus equipping it with robust multimodal error-correction capabilities.
Summarize Agent:
Due the CLIP model's limitation of accepting only tokens as input, data generated by PathGen-L. VA often exceeds this length. To address this, we prompt GPT-4 to generate instruction- tuning data for summarizing these descriptions. We then fine-tune Llama-2 as summary agent produce concise summarics for each generated description of WSI patches.
3.2 DETAILS OF DATA CONSTRUCTION PIPELINES
Source Data:
We source approximately 7,500 WSIs with paired reports from TCGA. Since these reports often contain information unrelated to WSIs. such as gross findings and measurements. Inspired by HistGen (Guo et al. 2024), prompt GPT-4 to extract only observable morphological
4

---

Published as a conference paper at ICLR 2025
Table 1: Human evaluation of image captions from PathGen-1. 1.66 and OpenPath. The accuracy
represents the proportion of correct findings in each caption on average.
PuthGen 1.41
OpenPuth
Pathologist ID
Number of Cuptions
Total Findings
Findings Captioa
Cornect Findings
Incerrett Findings
Accuracy
Number Cuptions
Total
Findings Cuption
Correct Findlngs
Incorrect Findings
Accuracy
Pathologist
200 200
1,099 1,059
5.30
956
103 122
90.3%
200
412 412
2.06
319
93
77.4%
Parhologist
5.30
957
88.5%
200
2.06
312
100
75.7%
and diagnostic features in reports. As some reports contain substantial information and may exceed the 77-token input limit for models like CLIP, we design GPT-4 prompts (Figure (8] to split longer reports into 2-3 concise sentences, while preserving critical information. With these WSIs and cleaned reports, we develop five-step meticulous pipeline, shown in Figure 2 spanning from representative patch extraction to description generation, to produce high-quality image-text pairs.
Step 1: Representative Patch Extraction: This step identifies representative patches from WSIs using prompt-based retrieval and k-means clustering. Prompt-based retrieval focuses on diagnostically relevant patches using WSI reports and predefined prompts, while k-means clustering ensures feature diversity by capturing distinct morphological patterns. Prompt-based retrieval uses PathGen-CLIP imir- L. to identify relevant patches through two prompt types: (1) previously cleaned WSI reports, formatted as 1-3 concise paragraphs, as prompts. (2) GPT-4 generated prompts based on the WSI's tissue origin (e.g., lung, colon), describing potential attributes like enlarged nuclei or lymphocyte infiltration. For both report-based and attribute-based prompts, we identify the top 64 relevant patches in each WSI, resulting in 128 patches in total. Detailed GPT-4 prompts are provided in Appendix C.2 (2) K-means clustering: Since the prompt- based retrieval primarily focuses on patches with higher degree of pathological changes, it may overlook patches with other morphological variations. To address this, to extract features from WSI patches and apply k-means clustering on these features. Each cluster represents patches with distinct morphological characteristics. The number of clusters is determined by the square root of the total number of patches, as larger WSI typically contains more distinct morphological features. We sample 256 patches from each WSI using clustering, ensuring uniform sampling within each cluster across the WSI to guarantee more diverse selection of samples. Finally, we combine the extracted patches from prompt- prompt-based based retrieval and clustering, resulting total of 384 representative patches sampled from each WSI.
PathGen-CLIIP-LA
Since the representative samples may still contain highly similar patches, which may impede the subsequent training CLIP contrastive learning. We utilize PathGen- CLIP iit to compute patch similarities and exclude redundant patches within each WSL. Using a similarity threshold of 0.88, we employ a probabilistic filtering approach where patches exceeding this threshold are removed with probability proportional to their similarity score.
Step2: Similar Patch Filtering:
Step3: Description Generation: We utilize the trained PathGen- VA model with the prompt: "This is histopathology image from source, describe this image in detail." where source indicates the origin of the wSI (c.g., lung, colon). This approach generates detailed descriptions for all representative patches extracted from the WSIs, thereby creating initial image-description pairs.
To mitigate potential errors and hallucinations L.MM-geneatte descriptions, implement trained revision agent that reviews image-descriptee pairs and refines the description through additions, modifications, or removals, while preserving descriptions that are already accurate. This step ensures the overall quality of descriptions.
Step4: Description Revision:
Step5: Description Summarization: Since the deseriptions generated by PathGien-LJ LLa VAdesss are usually lengthy and often exceed the CLIP 77-token limit, we use a summarization agent extract the key information from these descriptions, ensuring that no essential details are lost.
Through these steps, we generate total of 1.6 million high-quality image-text pairs from 7,500 WSIs, sourced from 27 different tissue types. These diverse pathology image-text pairs are utilized for the contrastive learning pre-training of the CLIP model. This effort aims develop a more robust pathology-speciff CLIP model, which enhances support for downstream pathology tasks.
3.3 EXPERT EVALUATION OF CAPTION QUALITY IN PATHGEN-1 1.6M
To validate the accuracy of our model's captions, we conduct evaluations by randomly selecting 200 generated captions and breaking them down into individual pathological findings. Two expert
5

---

Published as a conference paper at ICLR 2025
Table 2: Comparison of different CLIP models zero-shot pathology image classification datasets with accuracy (%). The top performance is highlighted in bold, with the second-best underlined.
Model
LC-Lung
LC-Colon
CRC100K
SkinCancer
Peam
BACH
Osteo
WSSSLUAD
SICAP42
Average
OLPP 33.1
75.7
26.2
9.6
53.9
21.7
46.9
64.6
32.8
40.6
OpenAL-CLIP-1
70.4
81.1
40.3
19.4
55.5
34.3
53.9
81.2
25.4
51.3
PLIP
87.9
90.2
52.8
42.5
51.8
34.3
52.9
73.1
42.5
58.6
PubmedCLIP
80.5
11.3
PMC-CIIP
33.3
51.9
8.7
11.4
53.8
21.3
29.2
65.2
31.5
34.0
58.7
PathCLIP
88.9
94,3
53.3
35.1
72.5
46.8
69.2
85.1
48.3
56.2
CONCH
74.7
59.4
63.2
78.7
73.5
33.0
68.7
BiomedCLIP
48.8
94.3
29.9
31.7
84.0
39.8
36.7
73.7
32.2
52.9
PathiGon-CLIP
63.3
65.6
89.2
73.5
PathGen-CT MnhGen-CLLIP.I.
89.8
99.3
78.0
70.6
88.2
71.3
74.6
82.2
63.5
79.7
physicians assess each finding, with Pathologist A and Pathologist assigning average accuracy ratings of 90.3% and 88.5% per caption, respectively. For comparison, we evaluate image-caption pairs from online platforms like Twitter, which demonstrate less than 80% accuracy in human assessments. Moreover, the number of findings per caption is significantly lower, as Twitter captions tend to be brief, less detailed, and often include irrelevant features such as aesthetic aspects rather than directly corresponding to the image. Consequently, the overall quality of mathGen-1.6M, is substantially higher than that of existing datasets.
EXPERIMENTS
In this section, we describe the training process of PathGen-LLaaWdes and PathGen-CLIP utilizing our generated dataset. We then evaluate PathGen-CLIP's effectiveness through various downstream pathology tasks, comparing its performance against state-of-the-ar baseline models. These evalua- tions encompass zero-shot image classification, few-shot image classification, and whole slide image classification. Finally, we demonstrate that by integrating the PathGen-CI. vision encoder with LLMs and leveraging dataset, achieve superior performance in pathology LMMs.
4.1 IMPLEMENT DETAILS OF MODEL TRAINING PROCESS
desp dAdep? PathGen-LLa VA adopts LLaVA's model structure and training ap- proaches, divided into two stages, Initially, we align image-text pairs from PathGen,nse- facilitated by fully connected (FC) layer. Subsequently, we fine- tune both the FC layer and the Vicuna component using me-geneeatted detailed image descriptions. This process equips
PathGen-LLa Va
with Vicuna LLM using
with the capability to generate image descriptions.
The data generated using MaahGenn-LLaVAAee morphological descriptions. Additionally, the pretrained LLM component (Vicuna) has undergone human value alignment, often avoiding direct diagnostic outputs and frequently recommending consultation with professional pathologist for definitive diagnoses. Therefore, we utilize PathGen- 1.6M for first-stage training to help the model learn key morphological and tissue structural features. Subsequently, PathGen employed in the second stage of training, which enhances the model's diagnostic comprehension capabilities while building upon the strong morphological understanding established in first stage training. In the Appendix [B.3] conduct ablation studies compare the effects of merging PathGeninit and PathGen-I. 1.6M for simultaneous training versus using PathGen- 1.6M for the first stage of training followed by PathGen the second stage.
PathGen-CLIP:
predominantly features extensive
4.2 ZERO-SHOT IMAGE CLASSIFICATION
Due to the training of CLIP-based models on image-text pairs through contrastive learning, these models achieve an intrinsic alignment between textual descriptions and visual content. This align- ment facilitates zero-shot image classification, which is particularly effective in scenarios with no annotations. To underscore the capabilities of the PathGen- CLII series, we evaluate its zero-shot nine pathology classification datasets, including PatchCamelyon
image classification performance (Peam) (Veeling et al.) 2018), CRC- 100K (Kather et al.] 2018)). SICAPv2 Valla-Rodriguez et al. 2020). BACH (rrsta et al. 2019 Osteo (Arunachalam et al. 2019). SkinCancer (Kriegsmann

---

Published as a conference paper at ICLR 2025
1.00
0.95
0.99
0.90
0.85
0.80
Model
Model
0.95
BiomedCLip
0.94
LC-Colon
PUP
QuiltNee
Camelyon17
Quithett
PLIP
0.33
PathGen-CLLPP
PathGen CLIP
0.60
PathGen-CUPL
PathGer-CUP
0.92
16
32
128
32
128
A0000
Model
Model
0.85
LC-Lung
WSSSLUAD
BlomedCL
PLIP
PLIP
Quittiet
Quikteet
0.80
PathYGen CUP
PathGen-CLL CUP
PathGen CUP-L
PathGen CUP
0.75
12 Training labels per
16
128
32 Training labels class
128
Figure Comparison of few-shot classification accuracy (%) across different CLIP models on various pathology image classification datasets, visualized with box plots.
et al. 2022), WSSSLUAD (Han et al. 2022), LC-Lung, and LC-Colon (Borkowski et al. 2019) For each dataset, design class-speciff prompts, such as "an H&E image of class", and calculate the similarity between each class's 's text prompt and the image. The class prompt with the highest similarity score is assigned as the predicted label. We compare the performance of PathGen-CLIP with eight previous CLIP models, including OpenAI-CLIP. OpenAI-CLIP-L, PLIP, PMC-CLIP, PubMedCLIP, QuiltNet, PathCLIP, BiomedCLIP and CONCH.
Results: The PethGen-CLIP series significantly outperforms previous SOTA models in zero- shot classification tasks, with PathGen-CLiP-L emerging as particularly advanced model. As demonstrated in Table 2 PathGen-CLIP exceeds QuiltNet by 30.5% on the Pcam dataset and by 19.2% on the SkinCancer dataset. On average performance across all datasets, PathGen-CL.I also far surpasses the previously SOTA model, CONCH, by 6.4%. Moreover, our stronger variant, PathGen-CLIPA model, exhibits exceptionally consistent performance across various datasets, achieving remarkable results even in datasets where all other models perform poorly. For instance, the BACH dataset, MaahGen-CLIP.. already surpasses previous models by a large margin (13.2%). Similarly, it exceeds the average performance of PathGen-CC CathGen-CIIP by 5.4%. The high performance of both PathGin and MathGen-CLIPA. underscores the effectiveness our PathGen- 1.6M dataset, offering potential for clinical utility in scenarios where annotated data is available.
4.3 FEW-SHOT IMAGE CLASSIFICATION WITH LINEAR PROBING
Traditional image classification tasks generally require extensive labeled data to achieve high accuracy. which impractical in many real-world applications due to resource constraints, particularly in the pathology domain. In this study, we explore the effectiveness of PathGen-CIn setting, where the model undergoes fine-tuning through linear probing on its pre-extracted feature representations. We assess the model on four representative datasets: LC-Colon, Camelyon17 LC-Lung, and WSSSLUAD, with fine-tuning performed using various training sizes (2, 8, 16, 32, 64 and 128 shots). Each training size is randomly sampled 10 times and conducts 10 runs. The results are displayed box plots to illustrate the model's 's performance across different conditions.
in few-shot
The PathGen-Cle CLIP series achieves notable performance with minimal samples, making the model viable in climical settings with scarce annotations. As illustrated Figure with only 2 shots, MathGen-CLIPP reaches close 92% accuracy the WSSSLUAD dataset, significantly
Results:
7

---

Published as a conference paper at ICLR 2025
Table 3: Performance of various CLIP models across three datasets, MIL. methods, and two evaluation metrics. The top performance is highlighted in bold, with the second-best underlined. Additionally, the last row in gray indicates the performance of current state-of-the-an vision-only pre-trained pathology model.
Performance
CAMELYON-17
CAMELYON-16
BRACS
Average
Method
F1-score
AUC
F1-score
AUC
F1-score
AUC
Fl-score
AUC
OpenAI-CLIP
23.5+4m
60.7+3a
62.8
61.4s29
46.8eu5
78.5+a0
44,4
OpenAI-CLIP-L
37.7
76.7
75.820s
69.7=11
$1.6239
78.9
55.0
75.1
ABMIL
PI.IP
45.6118
82.811a
86.611A
90.01za
51.7120
78.5104
61.3
83.8
Quilt-Net
44.3+20
84.2+10
82,9=
87.1+12
54.7=30
82.3+16
60.6
84.5
BiomedCLIP
55.5
84.
82.6214
83.6234
57.4
80.6214
65.2
82.8
PathCLIP
45.2
82.6114
80.2
85,0
56.4
83.8104
60.6
83.8
CONCH
55.0=29
86.4+n3
93.9=L1
95.2+07
62.0=14
90,0=06
70.3
90.5
PathGen-CT.n (ours)
58.2
87.5
93.5
96.9
62.6219
85.8
71.4
90.1
PathGen-CLIPA. (ours)
$8.6
87.9
94.3=18
95.8+14
66.6:44
87.2+22
73.2
90.3
GigaPath-GG G
54.4
88.3+1a
98.0=04
98.3ama
62.3s49
84.2
71.6
90.2
OpenAI-CLIP
25.4
59.4
63.7=45
67.3+43
52.2+45
76.7+16
47.1
OpenAT-CLIP-II
34.914
78.5110
78.7
73.4
55.5216
78.8120
56.4
76.9
ACMIL
PLIP
46.0
86.1
90.4=22
94.8+11
57.0=33
80.8+ea
64.5
87.2
Quilt-Net
44.4s10
86.0=as
84.8=35
90.5#34
60.8
82.0=18
63.3
86.2
BiomedCLIP
53.614m
83.6111
82,9
84.8133
63.2
81.91s
66.6
83.4
PathCLIP
44.2*06
82.9+13
84,4=16
87,3+16
58.0=76
83.0+24
62.2
84,4
CONCH
56.3216
87.5ans
94,4g1n
97.2201
66.1216
88.7
72.7
91.1
PathGen-CLe CLIP (ours)
53.3246
89,4
92.6216
97.2=0s
66.9=3a
87.0204
71.0
91.2
PathGen-CLIP--L. (ours)
58.41s2
92.0
94.5
97.4114
66.9
88.4114
73.3
92.6
GigaPath-G
55.9114
89.8114
95.7
99.21s4
61.0145
83.6
70.9
90.9
outperforming competitors like PLIP, Quilt-NET, BiomedCLIP and CONCH which record accuracies below 90%. Furthermore, as the number of shots increases, PathGin demonstrates faster performance improvements and convergence compared to other models. Additionally, the narrower box plot widths of the PathGen-CLIP model highlight its superior task adaptation capabilities and robustness.
4.4 WHOLE SLIDE IMAGE CLASSIFICATION
Whole slide image classification is essential for automating disease identification and classification from high-resolution pathological slides by analyzing high-resolution pathological slide images, which are typically larger than 100,000 100,000 pixels. This task is particularly valuable for clinical practitioners as it boosts clinical accuracy and efficiency significantly. The standard approach for WSI classification involves segmenting WSIs into image patches, extracting instance embeddings using frozen image encoder, and employing Multiple Instance Learning (MIL) to convert these embeddings into slide-level predictions. Superior patch representations provided by the image encoder are crucial, they significantly influence WSI prediction performance. Therefore, assess the efficacy the PathGen-CLIP series in comparison with other prominent models, including OpenAI-CIIIP, OpenAI-CLIP-LL, PLIP, BiomedCLII, PathCL IIP,, QuiltNet and CONCH. For the MIL. method, we utilize the widely adopted ABMIL (lsse et al. 2018) and the current SOTA method, ACMIL (Zhang et al. (2023b). Additionally, we involve the state-of-art vision-only encoder, GigaPath- G (Xu et 20044). which is four times larger and trained on 800 times the amount of vision-only private data for comparison. Our evaluations span three datasets: CAMELYONI6 (Litjens al.) 2018), CAMELYON17 (Litjens et al. 2018), and BRACS (Brancati et al. 2022). excluding any TeGA-related datasets. For detailed experimental setup, please refer to the Appendix
Results: PathGen-CLiP series consistently outperform existing pathology-specift CLIP models Across three key datasets leveraging both MIL methods. For instance, as shown Table 3 the PathGen-CL.I model, employing the ABMIL architecture, achieves remarkable average AUC of 96.9 on the CAMELYONI6 dataset. This substantially exceeds PLIP (90.01), BiomedCLIP (83.6), Quilt-Net (87.1) and CONCH (95.2). Overall, PathGen-CLIP-L models demonstrate significant improvements with an average AUC of 92.6% across datasets using ACMIL- outhrr-roooooninngggggg PLIP at 87.2%, BioMedCLIP at 83.4%, Quilt-Net at 86.2%, and CONCH at 91.1%. Despite CONCH being powerful model trained on proprictary datasets, our PathGen- CLIP series trained on PathGen-1.6M, open-source pariant-signfficant outperforms these established models.
8

---

Published as a conference paper at ICLR 2025
Table 4: Overall results of models on the PathMMU test set. The best-performing LMM in each subset for general and pathology domain LMMs is in-bold, and the top-performing LMM is underlined.
Test Overall
PubMed
SocialPath
EduContent
Atlas
PathCLS
Tiny
ALL
Tuy (281)
ALL
Tay
All (1855)
Tiny (255)
All
Tiay
ALL (1007)
Tiay
ALL
(1156)
(9677)
(3068)
(235)
(1938)
(208)
(177)
(1809)
Expert performance
71.8
72.9
71.5
69.0
68.3
78.9
General Large Multimodal Models
BLIP-2 TSXL-
33.3
33.5
37.0
37.4
35.7
34,6
30.2
34.5
39.4
40.7
19.8
20.6
InstrucBBLIP XXX
34.3
33.9
39.1
37.2
33.6
34.3
34.5
36.0
38.5
39.3
22.6
22.7
LLaYA-1.5-138
38.8
37.6
44.5
41.0
40.4
40.4
34.1
39.4
47.1
44.3
24.9
23.5
Qwen VL-MAX
49.2
45.9
53.0
50.9
53.6
49.3
52.2
47.9
51.4
49.8
30.5
29.6
Gemini Pro Vision
42.8
42.7
43.8
44.9
42.4
42.0
43.5
43.7
49.5
49.4
32.8
34.7
GPT-4V-100
53.9
49.8
59.4
53.5
58.7
53.9
60.4
53.6
48.1
52.8
36.2
33.8
Pathology-specife Large Multimodal Models
LLaVA-Med
25.3
26.2
28.5
27.7
28.9
27.3
22.7
27.2
22.6
30.7
22.6
20.3
Quilt LLVV
45.6
41.5
47.3
42.6
46.4
46.6
51.8
45.3
46.2
42.7
32.2
29.2
PathGe-LLLaV
60.1
58.4
60.1
60.9
58.8
60.8
63.5
64.9
54.2
Stronger models have the potential to be trained with fewer data through vision-language
approach, compared to traditional vision-only pretraining. Notably, PathGen-CLIPA. outperforms GigaPath-G on WSI classification tasks, achieving 2.4% higher Fl-score and 1.7 AUC improvement with ACMIL, despite GigaPath-G having significantly more model parameters and training data. This underscores the substantial advancements of PathGen-1.6.. contributes the analysis of WSIs.
4.5 INTEGRATING WITH LARGE LANGUAGE MODELS
LLMs possess extensive knowledge and common sense due to their larger model sizes and diverse training datasets. Models like CLIP, which are already aligned with language models, are often used for integration with LLMs to develop powerful LMMs. In this work, to train such LMMs, we construct 200K instruction- -uning samples based on PathGen data by prompting GPT-4 using the prompt shown in Figure 16] and Figure i7]in the appendix to generate choice QAs and 105K multi-round of dialogues. We follow the training methodology of LLaVA to train our LMM, which we refer to as PathGen-LLLa VA.. We evaluate the performance of PathGen- LLaVA on the PathMMU dataset, which includes expert annotations from multiple sources and discases. Additionally, such as GPT-4V, Gemini-Pi Pro Vision, Qwen-VL-Max, as well as previous comain-specifie LMMss, including LLaVA-Med and Quilt-LLaVA, to validate the capabilities of model. For detailed information regarding training data and model training details, please refer to the Appendix C..5
setecciionttuning data, including 95K multi-
compare MannGennLLaaa with the most advanced -ooeralldomain models,
CC.5
Results: PathGen-LLaV VA significantly outperforms previous SOTA pathology LMMs, even surpass- ing the leading general model, GPT-4V. As shown in Table PathGen-LLaVA consistently exceeded Quilt-LLaVA by 17.5%, 12.2%, 15.4%, and 22.2% across the PathMMU's PubMed, SocialPath, Atlas, and PathCLS subsets respectively, and also outperformed the top seneral-domain model GPT-4V by a significant margin. Specifically, in overall test performance, PathGen- LeaGGenn-LLVV VA achieved 58.4%. surpassing GPT-4V's 49.8 49.8%. These results demonstrate the superiority of the PathGen dataset and the effectiveness of PathGen-CLII as a backbone. It marks a step closer to the potential application of pathology LMMs in assisting medical professionals in practice.
4.6 EXPLORATION OF SCALABILITY AND VERSATILITY OF PATHIGNN-1.6N.
Scaling up with Non-WSI Report Paired Data: In our approach, we require WSIs paired with corresponding reports. However, many WSIs in datasets like TCGA lack associated reports. To explore whether expanding our dataset without paired reports can enhance performance. We conduct experiments by adding 6,800 additional WSIs from TCGA that lack associated reports and extract a total of 1, 1,238,78 750 patches using source prompts (247,750) and k-means clustering (991,000). These newly collected samples, derived from source prompts and k-means clustering, were then incorporated into the PathGen- dataset to train the CLIP model, referred to as PathGen- PathGen-CLIP-LL* CL.P-L*
Results: Scaling up with non- WSI report paired data using PathGen's 's generation approach and incorporating it into PathGen- 6M still improves zero-shot performance on 7 out of tasks (as shown in Table 5) confirming the scalability of our method regardless of WSI report availability.
9

---

Published as a conference paper at ICLR 2025
Table 5: Results of scaling up data (PathGen-CLLIP-L.)) scaling up the model (PathGen-CLLIP-H),, and
utilizing PathGen-I.6M. to adapt vision-only model to vision-language model (PathGen-GiaaPut-GG)
Method
LC-Lamg
LCCColon
CRC-100K
SkinCancer
Peam
BACH
Osteo
WSSSLUAD
SICAP22
PathGen-CLIP-L.
90.0
99.3
78.0
70.6
88.2
71.5
74.6
82.9
63.5
PathGen-CLLin CLIP-L
92.5
99.6
78.1
72.1
89.9
67.9
77.5
84.4
59.5
PathGen-LLIP-E
94.4
98.5
78.1
76.4
89.4
66.5
81.4
85.3
56.5
PathGien GigGaaa-GG
94.6
98,1
75.0
74.2
89.0
58.5
80.2
84.5
46.6
Table 6: Comparison of GigaPath- G and PathGen PathGen-GigaPattee GigaPPth--G for the few-shot classification task using the Camelyon17 and WSSSLUAD datasets. The top accuracy is highlighted in bold.
Number Shots
Datasets
Models
Camelyon17
GigaPath-G
60.30% # 13.55%
82.93% + 15.10%
94.24% # 4.15% 94.02% 6.25%
95.83% = 3,30%
PathiGnn
GigaPath-G G
72.71% # 23.35%
84.11 13.95%
97.40% 0.75%
GigaPath-G
67.76% + 12.55%
76.55% 9.55%
87.76% + 5.10%
93.33% + 1.15%
WSSSLUAD
PathGcn-Gigg GigaPath-G
77.96% # 23.30%
91.06% 3.35%
93.79% + 1.20%
94.82% = 0.50%
Scaling up with Larger Model: In where researchers continually scale up models to achieve greater capabilities, the adaptability of data for larger models is crucial. Therefore, we conduct further experiments by scaling up MachGen-CLIP... effective for larger models.
PathGen-CLIPP. validate that our data remains
Results: MathGen-CLIP-H achieves better results than PathGen-CLIP-L on 6 out of 9 zero-shot classification datasets, as shown in Table 5 This clearly demonstrates that PathGen- 1.6M effectively supports larger models in achieving enhanced performance.
To demonstrate the versatility of PathGen-1 1.6M,, we transition from vision-language pretrained models the state-of-the-ar vision-only pathology pretrained encoder, GigaPath-G (Xu et al. 2024), trained on over billion private patch images. We combine it with the language encoder BioClinical-BERR (Alsentzer al. 2019) create new CLIP model, PathGen-GiggaPan GigaPath-G.
Incorporation with Advanced Vision-only Model:
Results: PaihGen-GigaPathh-G not only acquires vision-language capabilities but also enhances its vision-only performance. As shown in Table[5] PathGen-Gige GigaPath--G achieves comparable overall performance on zero-shot tasks, despite not being pre-trained on billion-scale datasets like OpenAl- CLIP. Furthermore, our linear probing evaluations on two datasets, presented in Table 6 reveal substantial performance improvements, particularly with a very limited number of samples. These results indicate that PathGen-1.6M is not only effective for training CLiP-based models but also suitable for vision-only models, underscoring its substantial potential impact.
CONCLUSION
In this work, we propose a novel multi-agent collaboration approach that generates 1.6 million high-quality pathology image-text pairs from whole slide images. Utilizing these generated data alongside existing datasets, we train two advanced models: PathGen-CLIP and PathGen-CLIIP-L. These models achieve significant advancements in zero-shot image classification, few-shot image classification, and whole slide image classification, even attaining comparable or better results to much larger vision-only models in terms of parameters and training data. Additionally, We integrate PathGen-CLIP. with an LLM to create powerful pathology-specing LMM, PathGen-LLaVA, By leveraging the PathGen-1 PathGen-LLaVa VA. Our experiments demonstrate that PathGen-LLaVA exhibits strong pathology image understanding capabilities, significantly outperforming previous pathology LMMs on the large-scale PathMMU dataset by large margin and surpassing the performance of powerful closed-source GPT-4V. Furthermore, we investigate the scalability of PathGen-1 Mifeen-1166MM. 1.6M by scaling up the training data, scaling up the model size, and combining it with advanced vision-only pre- pre-trained pathology models, showcasing the extensibility of the PathGen approach. Extensive evaluations confirm the superiority and promising potential of the PathGen-1 1.6M dataset.
dataset, we generate 200,000 instruction-tunin samples to train
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
Our research is solely based on image-text pairs, with no involvement of human subjects. The dataset used this study is publicly released and is maintained for long-term availability on GitHub. The data consists of descriptive content related to pathology images, without any inclusion of potentially harmful insights or conclusions. There are no potential conflicts of interest or sponsorship associated with this research. The PathGen dataset utilized in this study is derived from publicly available data provided by TCGA, ensuring full compliance with privacy, security, and legal requirements. All relevant ethical standards and research integrity guidelines have been thoroughly observed throughout the development and use this dataset.
7 REPRODUCIBILITY STATEMENT
The datasets proposed in this study are publicly released and maintained for long-term accessibility at GitHub SaaiuubbPaaheee--11 I.6M, Additionally, we have made PathGen CLIP, Bethhen-CLIIPAA and PathGen- LLaVA openly available, enabling the research community to fully reproduce our results. Furthermore, detailed parameter settings for all experiments and comprehensive descriptions downstream datasets provided in the appendix. These resources support the verification and extension of our work by other researchers.
8 ACKNOWLEDGEMENTS
This study was partially supported by Zhejiang Provincial Natural Science Foundation of China (Grant XHD2320000,, the National Natural Science Foundation of China (Grant No.92270108), foundation of Muyuan Laboratory (Program ID: 14106022401, 1410002200,,) the Research Center for Industries of the Future (RCIF) at Westlake University, the Westlake University Joint Research Institute (Program ID: WU22022MY003 and the Westlake Education Foundation.
REFERENCES
Jean- Cean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In Advances Neural Information Processing Systems, 2022.
Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. Publicly available clinical bert embeddings. arXiv preprint arXiv:1904.03323. 2019.
Guilherme Aresta, Teresa Araújo, Scotty Kwok, Sai Saketh Chennamsetty, Mohammed Safwan, Varghese Alex, Bahram Marami, Marcel Prastawa, Monica Chan, Michael Donovan, et al. Bach: Grand challenge on breast cancer histology images. Medical image analysis, 56: 122-139, 2019.
Harish Babu Arunachalam, Rashika Mishra, Ovidiu Daescu, Kevin Cederberg, Dinesh Rakheja, Anita Sengupta, David Leonard, Rami Hallac, and Patrick Leavey. Viable and necrotic tumor assessment from whole slide images of osteosarcoma using machine-learning and deep-learning models. PloS one, 14(4):e0210706, 2019.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: frontier large rision-language model with versatile abilities. arXiv preprint ar 230.::2308.2966666 2023.
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye. Augustus Odena, Arushi Somani, and Sağnak Taşırlar. Introducing multimodal models, 2023. URL htllhttt://weee adept ai/
b blog/fuyu-8b
Andrew A Borkowski, Marilyn M Bui, L Brannon Thomas, Catherine P Wilson, Lauren A DeLand, and Stephen M Mastorides. Lung and colon cancer histopathological image dataset (Ic25000). arXiv preprint 1912 2019.
11

---

Published as a conference paper at ICLR 2025
Nadia Brancati, Anna Maria Anniciello, Pushpak Pati, Daniel Riccio. Giosuè Scognamiglio, Guil- laume Jaume, Giuseppe De Pietro, Maurizio Di Bonito, Antonio Foncubierta, Gerardo Botti, et al. Bracs: A dataset for breast carcinoma subtyping in h&e histology images. Database, 2022: baac093, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan. Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33; 33:1877-1901, 2020.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose purpose vision-language models with instruction tuning, 2023.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies. Volume I (Long and Short Papers), pp. 4171-4186, 2019.
Sedigheh Eslami, Christoph Meinel, and Gerard De Melo. Pubmedclip: How much does clip benefit visual question answering in the medical domain? In Findings of the Association for Computational Linguistics: EACL 2023, pp. 1 1181-1192 2023.
Sepideh Esmaeilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot -id---i-iitiittionn detection based on the pre-trained model clip. In Proceedings the AAAI conference on artificial intelligence, volume 36, pp. 6568-6576, 2022.
Jevgenij Gamper and Nasir Rajpoot. Multiple instance captioning: Learning representations from histopathology textbooks and articles. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16549-16559, 2021.
Zhengrui Guo, Jiabo Ma. Yingxue Xu, Yihui Wang, Liansheng Wang, and Hao Chen. Histgen: Histopathology report generation via local-global feature encoding and cross-modal context interaction. arXiv preprint arXiv:2403.00 053966 2024.
Chu Han, Xipeng Pan, Lixu Yan, Huan Lin, Bingbing Li, Su Yao, Shanshan Lv, Zhenwei Shi, supely-- tissue semantic
Jinhai Mai, Jiatai Lin, et al. Wsss4luad: Grand challenge segmentation for lung adenocarcinoma. arXiv preprint drXiv:2204.06455 2022.
Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.003:: 2023.
Wenyue Hua, Lizhou Fan, Lingyao Li, Kai Mei, Jianchao Ji, Yingqiang Ge, Libby Hemphill, and Yongfeng Zhang. War and peace (waragent):: Large language model-based multi-agent simulation of world wars, 2023.
Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J Montine, and James Zou. A visual- language foundation model for pathology image analysis using medical twitter. Nature medicine, 290)):307-23166, 2023.
Wisdom Ikezogwo, Saygin Seyfioglu, Fatemeh Ghezloo, Dylan Geva, Fatwir Sheikh Mohammed, Pavan Kumar Anand, Ranjay Krishna, and Linda Shapiro. Quilt-Im: One million image-ter text pairs for histopathology. Advances Neural Information Processing Systems, 36, 2024.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openelip, July 2021. URL https: /ttttss:/dol. or./10.5281. If you use this software, please cite it as below.
zenodo. 51437731337
12

---

Published as a conference paper at ICLR 2025
Maximilian llse, Jakub Tomezak, and Max Welling. Attention- based deep multiple instance learning. In Interaational conference on machine learning, pp. 2127-2136. PMLR, 2018
Chao Jia, Yinfei Yang. Ye Xia. Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig, Scaling up visual and vision- Asion-language representation learning with noisy text supervision. In International conference machine learning, pp. 4904-4916. PMLR, 2021.
Jakob Nikolas Kather, Niels Halama, and Alexander Marx. 100,000 histological images of human colorectal cancer and healthy tissue. Zenodol0, 5281, 2018.
Katharina Kriegsmann, Frithjof Lobers. Christiane Zgorzelski, Joerg Kriegsmann, Charlotte Janssen, Rolf Ruedinger Meliss, Thomas Muley, Ulrich Sack, Georg Steinbuss, and Mark Kriegsmann. Deep learning for the detection of anatomical tissue structures and ncoplasms of the skin scanned histopathological tissue sections. Frontiers in Oncology, 12:10229 2022.
Vinay Kumar, Abul K Abbas, Nelson Fausto, and Jon C Aster. Robbins and Cotran pathologic basis of disease, professional edition e-book. Elsevier health sciences, 2014.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large language-and-ving assistant for biomedicine in one day. arXiv preprint arXiv:2306.00S: 2023a.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International conference machine learning, pp. 12888-12900. PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- imaae pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19730-19744 19742. PMLR, 2023b.
Xianhang Li, Zeyu Wang, and Cihang Xie. An inverse scaling law for clip training. In NeurIPS, 2023c.
Weixiong Lin, Ziheng Zhao, Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, and Weidi Xic. Pme-clip: Contrastive language-ing pre-training using biomedical documents. In International Conference on Medical Image Computing and Computer-Assiee Intervention, pp. 525-536.
Springer, 2023a.
Yuqi Lin, Minghao Chen, Wenxiao Wang, Boxi Wu, Ke Li, Binbin Lin, Haifeng Liu, and Xiaofei He. Clip is also efficient segmenter: A text-driven approach for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15305-15314, 2023b.
Geert Litjens, Peter Bandi, Babak Ehteshami Bejnordi, Oscar Geessink, Maschenka Balkenhol, Peter Bult, Altuna Halilovic, Meyke Hermsen, Rob Van de Loo, Rob Vogels, et al. 1399 hteestained sentinel lymph node sections of breast cancer patients: the camelyon dataset. GigaScience, 7(6): giy065, 2018
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03774 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference computer vision and paltern recognition, pp. 11976-11986, 2022.
Ming Y Lu. Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood. Data-efficien and weakly supervised computational pathology on whole-slide images. Nature biomedical engineering, 5(6):555- 2021.
13

---

Published as a conference paper at ICLR 2025
Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Long Phi Le. Georg Gerber, et al. A visual-language foundation model for computational pathology. Nature Medicine, 30(3):863-877 2024.
OpenAI. Introducing chatgpt. https//appnnd..commboocccca.a..p 2022.
OpenAl. Gpt-4 technical report, 2023a.
OpenAl. Gpt-4v(ision) system card. httpps///cchhpppee....cc//p/pppp Syyts Card. 2023b.
OpenAI. Hello gpt-4o. 2024-05-26.
2024. Accessed:
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprini arXiv:2304.07193, 2023.
Joon Sung Park, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael Bernstein. Social simulacra: Creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pp.
1-18, 2022.
Joon Sung Park. Joseph C O'Brien, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint 2XX1:2200403442. 2023.
Chen Qian, Xin Cong, Wei Liu, Cheng Yang, Weize Chen, Yusheng Su, Yufan Dang, Jiahao Li, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, (11))54855555) 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pullern recognition, pp. 10684-10695, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25285 2022a.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski. Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022b. URL
ew. net./ forum? 6-M337 wwt..ee/eeeeeooout///033333 4vmsMcy
Mehmet Saygin Seyfioglu, Wisdom O Ikezogwo, Fatemeh Ghezloo, Ranjay Krishna, and Linda Shapiro. Quilt-llava: Visual instruction tuning by extracting localized narratives from open-source histopathology videos. arXiv preprint arXiv:2312.047746 2023.
Yang Shu, Xingzhuo Guo, Jialong Wu, Ximei Wang, Jianmin Wang, and Mingsheng Long, Clipood: Generalizing clip to distribtissi In International Conference on Machine Learning, pp. 31716-31731. PMLR, 2023.
14

---

Published as a conference paper at ICLR 2025
Julio Silva-Rodriguez, Adrián Colomer, María A Sales, Rafael Molina, and Valery Naranjo. Going deeper through the gleason scoring scale: An automatic and-to-end system for histology prostate grading and cribriform pattern detection. Computer methods and programs in biomedicine, 195: 105637, 2020.
Yuxuan Sun, Hao Wu, Chenglu Zhu, Sunyi Zheng, Qizi Chen, Kai Zhang, Yunlong Zhang, Xiaoxiao Lan, Mengyue Zheng, Jingxiong Li, al. Pathmmu: A massive multimodal expert-level benchmark for understanding and reasoning in pathology. arXiv preprint arXiv:2401.16335 2024a.
Yuxuan Sun, Chenglu Zhu, Sunyi Zheng, Kai Zhang, Lin Sun, Zhongyi Shui, Yunlong Zhang, Honglin Li, and Lin Yang. Pathasst: A generative foundation assistant towards artificial general intelligence of pathology. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 5034-5042, 2024b.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant cnns for digital pathology. In Medical Image Computing and Computer Assisted Intervention-MICCAA 2018: s1 International Conference, Granuda, Spain, September 16-20, 2018, Proceedings, Part 11, pp. 210-218. Springer, 2018.
Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models not fair evaluators. arXiv preprint arXiv:2305.17922 2023a.
Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang. Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. Avalon's game of thoughts: Battle against deception through recursive contemplation. arXiv preprint arXiv:2310.01 2023b.
Bushi Xiao, Ziyuan Yin, and Zixuan Shan. Simulating public administration crisis: A novel generative agent-based simulation system to lower technology barriers in social science research. arXiv preprint ar arXiv:2311.069577 2023.
Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang, Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier González, Yu Gu, et al. A whole-slide foundation model for digital pathology from real-world data. Nature, pp. 1-8, 2024.
Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Yi Wu. Language agents with reinforcement learning for strategic play in the werewolf game. arXiv preprint arXiv:2310. 18940, 2023.
Zhengqing Yuan, Ruoxi Chen, Zhaoxu Li, Haolong Jia, Lifang He, Chi Wang, and Lichao Sun. Mora: Enabling generalist video generation via a multi-agent framework. arXiv preprint arXiv:2403. 13248, 2024.
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint 05X1v:2303.00915. 2023a.
Yunlong Zhang, Honglin Li, Yuxuan Sun, Sunyi Zheng, Chenglu Zhu, and Lin Yang. Attention- challenging multiple instance learning for whole slide image classification. arXiv preprint arXiv:2311.07125, 2023b.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision- language models. International Journal of Computer Vision, 13009):2337-2348, 2022.
15

---

Published as a conference paper at ICLR 2025
A OVERVIEW OF PATHGEN-1.6M
In this section, we present details of the PathGen-1.6M dataset, including quality assessment and statistical information, illustrate its high quality and significant contribution the open-source pathology dataset community. Additionally, we provide further details about the derivative dataset, BechGen-Instrutt-22 introduced in Section and specifically designed for training large multi- modal models. Finally, we outline the procedure for accessing our datasets.
A.1 IMAGE QUALITY
As mentioned in the main paper, the quality of previous pathology datasets is significantly degraded due to sereenshots and various compression issues. We conduct comparative image quality analysis between PathGen and previous Catasets-Patthap (Sun et al. 2024b), Quilt (Ikezogwo et al. [2024), and OpenPath (Huang et al., 2023). We randomly sample 500 images from each dataset. These images assessed using the existing image clarity mode and and through human evaluations conducted by pathologists.
In evaluating image clarity with clarity assessment models, the field of view and size significantly affect scores. To standardize assessments, we implement two approaches: (1) cropping the original images to uniform size of 672x672, and (2) resizing the original images to 336x336 which is the imput image size for PathGen-CLIP-L. For the pathologists human evaluation, we present the original images without resizing cropping. The pathologists are tasked with categorizing the image quality into five levels: low, moderately low, medium, moderately high, and high, corresponding to scores of 1-5.
214
PathCap
Quilt
6.7
OpenPath
PathGen
XE5
B/W-
PothCap
Quift
OpenPath Pathien
AwthCap
Quili
OpenPath PathGen
Score
Clarity model score with crop
Clarity model score with resize
<) Distribution of human evaluation score
Figure 4:
Comparison of image quality with the previous model using the clarity assessment model and human evaluation.
The image quality of PathGen is notably superior to other datasets, significantly higher than those from PathCap, Quilt, and OpenPath. As illustrated in the left and middle sections of Figure we display the clarity scores of each dataset's images in a boxplot format. The overall image quality from PathGen is significantly higher than those from earlier datasets, especially after resizing to 336x Additionally, the shorter length of the box for PathGen images indicates more consistent image quality. This consistency likely due to the images being directly cropped from WSIs with minimal compression and distortion. Similarly, as shown in the right part of Figure 4 the human evaluation experiments further confirm that PathGen consistently delivers higher image quality scores. This demonstrates that our image quality more closely approximates the images used by clinicians during diagnostic readings. This highly significant in reducing the domain gap between training images and those used in clinical scenarios.
Results:
al httpss//wwgggaaa.....coo//rr/ree
16

---

Published as a conference paper at ICLR 2025
The image shows section prostate tissue with glandular structures. glands are by single layer of epithelial cells with round to oval nuclei, which are fairly uniform in size and shape, The cytoplasm these cells appears easinophilic. The stroma between the glands is composed of connective tissue, which is pink and fibrous in appearance. There is no evidence of significant cellular atypia, mitotic figures, architectural distortion that would suggest malignancy. The overall morphology consistent with benign prostatic tissue.
PathGen
Pt in 30s w/ lung mass, eff, nite swts, loss, hx MM 15 yrs ago #pathologists thhemepthh #tsarcoma
A testis from HIR patient displaying weak PRDM9 cytoplasmic staining the germ cells (arrow).
Horizontal bar; 30 μ..
OpenPath
PathCap
It seemed that prominent pathology besides membranous nephropathy, in this case, was segmental sclerosis. So putting everything together, we favor that. This not a crescent, and this collapsing FSGS. But it is not common. In primary FSGS, as you saw the other group coming from the other paper that author showed, coming from Vancouver, although they had not at that time tested for PLA2R, but at least they looked for other causes, such as Summarrzee caption: Diagnosis is collapsing FSGS which is not common.
Quilt
Comparison of caption quality with previous datasets, with different pathological descrip- tions from various aspects of PathGen marked in different background colors.
Figure 5:
A.2 CAPTION QUALITY
There are several issues with captions from existing datasets: (1) Brevity: Captions are often too brief comprehensively outline the details within an image. This lack of detail can models to overlook important features in the image. This issue is prevalent datasets like OpenPath, PathCap, and Quilt. (2) Mismatch between text and image: On social media platforms like Twitter, the text accompanying an image may not necessarily describe the pathological features of the image but might focus on aesthetic qualities instead. On YouTube, a current frame might describe content from previous frames, leading to mismatches. (3) Colloquial language: Captions are often informal and random on social platforms. In contrast, the PathGen-1.66 1.6M dataset employs captions generated by LMM, which are further refined by various agents to ensure relevance between the text and the image and detailed deseriptions. As illustrated Figure 5 evident that the captions in PathGen are significantly more detailed and precise. It recognizes cell morphology ('round to oval nuclei, fairly uniform in size and shape"), tissue structure ("The glands are lined by a single layer of epithelial cells"), and diagnosis ("overall morphology is consistent with benign prostatic tissue"). In contrast, OpenPath's 's caption contains many abbreviations, which are not the vocabulary of models like BERT (Devlin al. 2019) and thus identified '[UNK]' tokens, leading to the loss of important information. PathCap only includes a single piece of staining information, while Quilt, despite containing a lengthy text, mostly discusses irrelevant details. The effective description related to the image boils down to only one sentence.
pathology-specine
A.3 DATASET STATISTICS
Image Sources: PathGen sources its image data from TCGA (The Cancer Genome Atlas), which provides comprehensive collection of pathology whole-slide images from various participating institutions, encompassing a diverse range of tissue and cancer types. We illustrate the data distribution of different image sources in the left panel of Figure 6 Our dataset originates from 27 different institutions, with the highest number of images obtained from breast tissue, totaling 217,025 images, and the fewest from lymphoid tissue, containing 4,543 images.
17

---

Published as a conference paper at ICLR 2025
Breast
217925
lung
195486
Kidrey
140335
Endometrium
1:2327
9.8%
Head neck
59230
Prostato
Blodder
Colon
Uver
64463
Brain
52688
Cervs
50305
gland
Rectum
50325
Esophagus
2671L
Stomach
25439
bisues
Thymus
21392
Skin
21372
Eye (uvea)
13044
Overy
Mesothli.m
11772
Bile duct 7531
6849
Lymphoid
Word Count (Distribution)
%
Figure 6: Visualization of the distribution of sample quantities from different tissue sources (top right) and the distribution of caption lengths (bottom right).
We conduct a distribution analysis of the caption lengths for all samples, as shown in the lower right corner of Figure The distribution of caption lengths is relatively symmetrical. Captions shorter than are extremely rare, indicating that our captions tend to describe the details of pathological images. The largest proportion of captions is around 50 words, accounting for 9.8%, while captions exceeding 70 words are similarly uncommon. This is because we employ a summary agent to control the length, ensuring compatibility with CLIP's maximum input length of 77 tokens.
Word Count:
Word Frequency: We also analyze the word frequency distribution within our dataset using word clouds, shown in the upper left corner of Figure 7 Overall, the captions in PathGen heavily focus on morphological features such as "disorganized," "disrupted," "fibrous," "prominent," and diagnostic characteristics like "inflammatory." "neoplastic," and "atypia." The other three panels in Figure display word frequency analyses for samples drawn from three distinct tissue types. Each tissue type exhibits distinct word frequency distribution, closely related to the morphological and diagnostic features typical of that tissue, such as "alvcolar" in lung tissue and "colloid" in thyroid tissue. This diversity samples and captions enables our model learn broader range of feature knowledge.
A.4 STATISTICS OF DERIVATIVE DATASET IITHGGN-INSTRUCT-20O
As mentioned in the main text, to integrate with large language models and construct large multimodal model, we extract samples from PathGen-1.666 to build BathGen-Instruet220 This dataset includes data for multi-choice QA-based instruction-tunin instruction- -ttrtingontninn..... We present the average question length, option length, answer length, and the number of questions with single image for these two types of data, shown in Table 7
and dialogue-based open-ended
Table 7: Statistical Information for PathGen-IIntruut-2000
Source
Avg. question length
Avg. option length Avg. answer length
# Questions / # Images
Multi-choice QA Open-ended QA
14.9
3.7
1.2
17.1
33.2
5.0
B
ADDITIONAL EXPERIMENTS AND DISCUSSION ABOUT THE PATHGEN-1.6M
B.
COMPARISON BETWEEN PATHGEN-LLAVA desp- LLAVA-MED-VIS. AND QUILT-LLAVA
Since the quality of description generation largely determines the effectiveness of PathGen, we compare BathGen-LLLaA.... with previous pathology L.MSs, including Quilt-LLaVA (Ieezogwo
18

---

Published as a conference paper at ICLR 2025
Tung
phin
nuc
inflanntory
ama
inf Ialanaattoon
m geggnnnnccy mow dénse
Fibr ous
Cal
noma
d aiagnosi
D sucess
Sept the
Hallla
preast
feature
Lawe cel lula
cytop aa
e
derse
PJ Tonorr phisp
aiiinnncy cy
piecoer rasoerpaaa
do optral all
eos
inophi
DEOPLe
p cin mentt da dAs
Franno
suggesting
REypL
seet ion
anyara
Likely
norma
hyper chromatmmmatooo
pppplastis
L
Thigh
Le
car
high
hy
berchromat
Loee
Features collular
proeinent
etterial
Indicative
sight
disergenian
Overall
Breast
follicle es
foulc
Muelerr
Aucisoli Peliooliibres ibrosis
infl
inflamma tory
hyper chromat
oma ti
Feeturrre es Ser
stypia
disrupted
in Pecserphirr
wathooogie
displays
maliignanc gnancyy
fea tur es eytoplate
normal
MP
neop
Troma
ignns Porpoolagy W
dia
inf
Lmmma
Indication
conne C
tive
nucler
e-ee
Tiorous
nula nostare
e
diagnosi
i
cuboida
section on
GOL I
cytop lash cellut
d
bithelia
normal
likely
suEfes
Ling
care Ih
eos moosinophiii Ic
Fir
charge
a vee
a
prosine hight
pppons dense
Thyroid
Lung
Figure 7: Visualization of word clouds derived from the overall dataset captions, along with separate word clouds for three distinct tissue sources.
et al. 2024) and LLa (Li et al., 2023a), in terms of description generation. Our evaluation includes extensive testing with the of LMM GPT-4o (OpenAl, 2024), as well as smaller-scale human evaluation.
For the GPT-4o testing, we randomly select 1000 images from our complete set of images, using GPT-4o as the evaluator to judge the quality of descriptions generated by both models. Previous research (Wang et al., 2023a) reveals that earlier versions of GPT-4 (OpenAT 2023a) and Chat- GPT (OpenAl) 2022) exhibit position bias, where the model presented first is often judged more favorably. Since we are unsure if the latest versions of GPT-4 and GPT-4o still exhibit this behavior, conduct two rounds of testing to ensure fairness. In one round, the description generated by PathGen-LLa LLa is presented first, and in the other round, the compared model's description is presented first.
For the human evaluation, due to the high cost and difficulty of involving professional pathologists, randomly select 100 images from our dataset and invite a professional pathologist evaluate which model performs better.
The quality of descriptions generated by PathGen-LLaVA.s significantly surpasses that of LLaVA-Med-vL. and Quilt-LLaVA in both GPT-4o and human evaluations. As depicted in Figure in the GPT-4o evaluation, MathhennLLLVAAdess outperforms Quilt-LLaVA over 85% of cases and exceeds LLaVA-Med-vi 1.5 in more than 94% of cases. In the human evaluation, PathGen- VLa outperforms Quilt-LLaVA and LLa VA-Med-- viddv.l.5 85% and 98% of cases, respectively. initial version of PathGen-LLaaA, to validate the quality of the dataset's descriptions. The performance gap might be more substantial if the more advanced PathGen-LLa's VL is used.
Results:
Note that this comparison uses MathGen-LLLaAr
Interestingly, in the GPT-40 evaluation, there appears to be a slight preference for the model whose re- sponse is positioned second. This bias is observed in both comparisons, whether PathGen-LLaVa is compared to Duilt-LLaVe or LLa LLLVA-MMd-vv 1.5.. When the position is reversed (i.e., the comparison model's 's response is presented first and our model's 's response second), the winning rate of our model is slightly higher, although the difference is not substantial. We hypothesize that OpenAl may have optimized this position bias issue.
19

---

Published as a conference paper at ICLR 2025
Evaluator: GPT-4o
PathGen-LLaVAAdep vs Quilt-LLaVA
1.9%
85.0%
13.1%
Evaluator: GPT-4o
1.0%
(Position reversed)
92.3%
6.7%
Evaluator: Pathologist
Win
85.0%
6%
9%
Tie
Evaluator: GPT-4o
PathGen-LLaVAdeey vs LLaVa-Med v1.5
0.8%
Lose
94.0%
5.2%
Evaluator: GPT-40
0.2%
(Position reversed)
97.4%
2.4%
Evaluator: Pathologist
98.0%
2.0%
Figure 8: Comparison of performance in generating pathology image descriptions among PathGen-
LLa VAdepp Quitt-LLaVA, and LLaVA-Med-v1.55 evaluated through GPT-4o and human assessments.
B.2 PERFORMANCE COMPARISON ACROSS DIFFERENT ZERO-SHOT CLASSIFICATION PROMPTS
Existing medical pathology-specific CLIP models usually design prompts that particularly suited to their models to enhance zero-shot image classification capabilities. However, this specificity can skew direct comparisons between models due to prompt variability. To address this, we collect substantial variety of prompts from existing literature on pathology-specift CLIP models and conduct a comparative test across eight mainstream datasets: LC-Lung, LC-Colon, Peam, WSSSLUAD, CRC- 100K, SkinCancer, BACH, and Osteo. We visualize performance variations using box plots to highlight the fluctuations in model responses different prompts.
Results: PathGen-CLIP and PathGen-CLIP-L consistently outperform previous models across all datasets and various prompts. As illustrated in Figure 9 the box plots for both models are significantly higher than those of other models. Additionally, the shorter box lengths indicate greater robustness of PathGen-CLIP CLIP and L to prompt variations. This superior performance and robustness make our models more suitable for practical clinical applications.
B.3 WHY PATHGEN- 1,6M IS RECOMMEND FOR FIRST-STAGE TRAINING?
As mentioned in the main paper, the captions generated by PathGen-LLa VA tend describe morphological content, such as cellular and tissue structures, rather than direct diagnoses. This due the value alignment the LLM, which avoids making definitive pathological diagnoses. Additionally. the training dataset for multimodal description generation contains relatively few diagnostic data entries. Therefore, using PathGen alone is insufficient for equipping PathGen-CLL with diagnostic capabilities. In our study, we conducted experiments comparing the training of models using various scales of data solely from PathGen, as well as combining data from different scales of PathGen with PathGen;nit-
Results: We observe notable decline the performance of PathGen-CLIP across four datasets when the data scale exceeds 0.7 million. As depicted in Figure 10 this trend is consistent most datasets whether the training utilizes data exclusively from PathGen or a combination of PathGen and PathGen,,st datasets. We hypothesize that when the scale of PathGen data surpasses that of PathGen,.at- the model's excessive focus on morphological features of cells and tissues detracts from its ability make direct pathological assessments. This shift in focus adversely affects the
20

---

Published as a conference paper at ICLR 2025
1.0
0.8
0.6
Accuracy
PUR
0.2
FNCCUP
DowNN-CCUP
0.0
LC-Lung
LC-Colon
Pcam
WSSSLUAD
1.0
Mathod
Qustiet PNC-Cu
Qpenis-LIN Spers-CUP
0.6
Accuracy
0.4
0.2
CRC-100K
SkinCancer
BACH
Osteo
Figure 9: Boxplot visualization of performance variation between PathGen-CLe CLIP and previous related CLIP models across various zero-shot classification prompts.
model's capability for zero-shot image classification, as it becomes overly specialized in recognizing structural details at the expense of broader diagnostic accuracy.
21

---

Published as a conference paper at ICLR 2025
WSSLUAD
CRC-100K
85
75
65
60
55
55
90
48
0.2
0.2
0-7
LC-Colon
LC-Lung
-W-WA www.leieiiii...o
Pww.oooiieennnnoooooo ose eage
-Wo! weehcosnkooaeeeo stge
PuthCereet stge
100
100
96
21
81
65
80
00
0.3 PathGen Dataset Scale
PathGen Dataset Scale
Figure 10: Illustration of different data construction strategies and comparisons of data scales. "wo/ PathGengnat stage" represents training solely with various scales of PathGen data using a single-stage training approach. "w/ PathGen, one stage" denotes the combination of different scales of PathGen and PathGeninit data for single-stage training, scale of 0 means all the training data from PathGin, "w/ PathGen two-stage" serves as reference constant line, representing the final result our current method, which employs PathGen-I.cn data for the first stage of training and then utilizes PathGen for the second stage. The horizontal axis represents the data scale in millions.
22

---

Published as a conference paper at ICLR 2025
B.4 PERFORMANCE COMPARISON BETWEEN ZERO-SHOT AND FULLY SUPERVISED APPROACHES
To demonstrate the extent of the gap between zero-shot classification and full fine-tuning in the field of pathology, we conduct a comparative analysis of their results. As shown in Table there remains notable gap between zero-shot performance and full fine-tuning, However, the advancements of CLL-G----LP-LL. significantly narrow this gap, showcasing its capability to adapt pathology-specing knowledge effectively. Notably, PathGen-CL.I supervised setting, outperforming OpenAI-CLIP and approaching the results of GigaPath-G. This remarkable given that GigaPath-G leverages much larger training datasets and model sizes.
achieves impressive performance in the fully
Table 8: Performance comparison between zero-shot classification and fully supervised approaches.
Model
WSSSLUAD
Lung
Colon
PatchCamelyon
Previous SOTA (Zero-shot)
85.1
88.9
94.3
72.5
PathGen-CLIP-L (Zero-shot)
82.2
89.8
99.3
88.2
OpenAI-CLIP-L (Full Fine-tuning)
95.9
99.7
100.0
93.5
GigaPath-G (Pull fine-tuning)
97.1
100.0
100.0
96,9
PathGen-CT CL.IP-I. (Full Fine-tuning)
97.7
100.0
100.0
97.0
B.5 ADDITIONAL ABLATIONS
B.5.1 ABLATIONS ON THE DATA CURATION PROCESS
We conduct an ablation study to evaluate the contribution of each step in our data curation process. As shown in the following table, removing either the clustering- musseriig-bbsed retrieval method or the prompt- based retrieval leads to performance decline. The removal of clustering based retrieval results in a more substantial average performance drop of 4.3%, likely because this method enhances dataset quality by sampling patches with diverse features. Eliminating the revise agent causes an average performance decrease of 2.3%, indicating its importance in correcting errors or hallucinations generated by large multimodal models. These findings underscore the critical role of each component in our data curation pipeline. Furthermore, by combining these approaches, PathGen-CLIPA achieves an average performance improvement of 6.8% over ManheenCCLI-Lii.n demonstrating the effectiveness of the PathGen-1.66 1.66 dataset.
Table 9: Impact of each step in the PathGen- 1.6M data curation process on the final performance of PathGen-CLIP, where RA represents the Revise Agent, PBR denotes prompt-based retrieval and CBR represents the clustering-
retrieval.
Model
LC-Lang
LC-Colon
CRCIDOK
uncer
WSSSLUAD
Averuge
Parhcien-CC.IP.. P
89.1
96.9
60.3
55.7
83.2
65.3
71.9
85.1
48.3
72.9
Cerbbien-CLIP//. wlo RAA
91.1
98.9
76.2
67.2
87.3
68.8
70.8
$1.5
54.9
77.4
PubbGen-CLA CLIP-L wlo PBR
91.0
99,1
77.0
68.0
86.8
65.0
72.8
83.2
59.5
78.0
PathGen-CIIPP
90.5
97.5
72.7
71.2
PathGen-CLIP.LI
89.8
99.3
78.0
70.6
88.2
71.5
74.6
82.2
63.5
79.7
B.5.2 ABLATIONS APPLYING PATHGEN-1.6M TO DIFFRRENT CLIP MODELS
To further demonstrate the broad applicability of PathGen- 1.6M, we adapt it to other CLIP-like models, including the Conn/Next-base-w.333 w320 (Liu et al. 2022), convolution-baset CLIP model trained by LAION (Schuhmann al. (022b), and a CLIPA-L. [i et al., 2023c), more efficient architecture for training ViT-based CLIP models. As shown in Table I0 incorporating the PathGen- 1.6M dataset significantly enhances the performance of both models, with improvements of 6.8% and 7.1%, respectively. These results further validate the quality and effectiveness of the PathGen-1.6M dataset.
B.5.3 ABLATIONS ON THE SOURCE OF PATHGEN-L 1.6M's EFFECTIVENESS
To explore whether the effectiveness of PathGen- 1.6M arises from its high- image-text pairs, the images themselves, the WSI reports used during its creation, conduct two complementary
23

---

Published as a conference paper at ICLR 2025
Table 10: Performance improvement of different CLIP models after continued training with PathGen-
1.6M. Here, Model represents models trained without PathGen-1.66M while other models incor-
porate the PathGen-I.6ke dataset during training.
Model
LC-Lang L.CColon
CRCIOOK
SkinCancer
Pram
BACH
Osteo
WSSSLUAD
SICAPY2
Average
PathGen- PuthGen-CLIPA-L Lijie
79.3
98.4
61.5
30.5
86.1
56.5
59.8
78.3
48.0
68.7
PuthGen.
(+6.8)
PuthGinnCCnnNeett Cannn-CCeaneethee...a..
8288
94.6
99.9
48.0
74.9
53.0
62.2
84,1
56.1
68.4
PuthGen-Con/Neetee Conv-extt-ase
89.1
95.9
67.9
60.2
80.7
66.2
77.9
87.2
54.3
75.5 (+7.1)
PahhGenCLLIP-Le CLIP-Lajus
89.1
96.9
60.3
55.7
83.2
65.1
71.9
85.1
48.3
72.9
PathGenCIIIP.I.
89.8
99.3
78.0
70.6
88.2
71.5
74.6
82.2
63.5
79.7 (+6.8)
0.55
0.00
0.8
Camelyon17
0.7
WSSSLUAD
0.20
LC-Lung
LC-Colon
Model
Model
0.70
Model
Model
DINOvO
DINOV2
DINOV2
DIND22
0.5
FLth-Gn-CLIP-L
PathGensCa P-L
0.55
Gat-GannEUIN.
16
32
128
128
Training labols per class
Figure 11: Few-shot classification performance comparison between PathGen-CLII-L trained with PathGen-1.6M, and DINO-V2-L trained using only the images from PathGen-1.6M.
experiments. In approach, replace all image captions in PathGen-1.6M, with their correspond- ing WSI reports, creating "image-report" pairs to train PathGen-CLIP-L. Meanwhile, we also further fine-tune the pre-trained DINO-v2 (Oquab al., 2023) by continuing pre-training the images from PathGen-1.6N, for 20 epochs as a comparison.
As shown in Table T1. the model trained with "image-report" pairs performs worse than PathGen- CLIP-Linit- This occurs because WSI reports provide global summary that fails to accurately capture the localized features of individual patches. The mismatch between patch-level features and the shared global captions introduces significant challenges for CLIP's contrastive learning, as the model struggles establish meaningful connections between image and embeddings. addition, Figure II reveals that DINO-v2 pretrained on PathGen-1.6., images significantly underperforms PathGen-CLIP-LL in few-shot experiments. This likely because DINO-v2 requires substantially larger datasets-on on the scale of tens even hundreds of millions of images-to to achieve optimal performance. The 1.6 million images in PathGen- 1.6M are insufficient for this method.
These experiments demonstrate that the superior performance of BathGen-CLIPA primarily arises from the high-quality image-text pairs PathGen-1.66 1.6M, underscoring the dataset's effectiveness.
Table 11: Zero-shot classification performance comparison by replacing PathGen-1.6M original captions with WSI reports. Model
LC-Luung
14-Coolon
CRC100K
SkinCancer
Peam
BACH
Osteo
WSSSLLAD
SICAP2
Average
PalliGen-UL CLII-L (Image-report poirs)
88.9
97.2
57.2
58.6
70.1
48.8
63.4
82.0
56.3
69.1
PathGessCI.i CIP---
96.9
61.3
83.2
65.3
71.9
48.3
72.9
PLthGes.CLLPPL.
89.8
99.3
78.0
70.6
88.2
71.5
74.6
$2.2
63.5
79.7
24

---

Published as a conference paper at ICLR 2025
C EXPERIMENTAL DETAILS
C.I MORE DETAILS OF DATA CONSTRUCTION PROCESS
C.1.1 PATCH EXTRACTION DETAILS
When extracting image patches from the WSI of the TCGA, since many WSIs contain multiple layers with different magnifications, we always opt for the highest magnification layer to ensure the finest details and resolution are captured. The patches consistently cropped size of 672x672 pixels.
C.1.2 DETAILS OF THE TRAINING DATA CONSTRUCTION PROCESS FOR THE REVISION AGENT
As we initially lack multimodal training samples for description revision (from incorrect to correct), need to first construct such samples. As illustrated in Figure 12 begin with caption sentence presumed to be correct. We then employ a prompt with an LLM to generate a revision operation that intentionally introduces errors into the caption, creating incorrect version. Next, reverse this revision operation: additions are converted to deletions, deletions become additions, and for modifications, the sentences before and after the change swapped. This process not only yields an incorrect caption but also the reverse operation that corrects the caption back to its original form, along with the corresponding image. These elements constitute our triplet revise agent training data.
"before": 'There is prominent presence of colls with daridy stained nuclel, indicative loukacytes, which are consistent inflammatory infiltrate', "after': 'There a prominent presence cells with lightly stained nudee, indicative of tymphacytes, which are consistent
"mode": "edit".
content": "Changed Idarkly
'lightly
'leuktocyte" 'lymphocytes'
and Inflammatory infiltrate' to 'normal tissue' to introduce discrppancy."
Edit
Correct caption
lightly stained nuclei
There is prominent presence of cells with darkly stained nuclei, indicative of leukocytes, which are consistent with an inflammatory infiltrate,
Reverse Edit
+>
lymphocytes
normal tissue
"before": 'There prominent presence of lightly indicative ymphocytts, which are consistent with normal tissue", "after":"
After editing
prominent presence of darkly nuclel, indicative which
are consistent with an infiammatory infiltrate", "mode": "edit".
"edit content": "Changed 'lightly stained nuclel' 'darkly stained nuciei' and 'lymphocytes to 'loukocytes'
'normal tissue iinllammatoyy infiltrate"
Incorrect caption
There is prominent presence cells with lightly stained nuclei, indicative of lymphocytes, which are consistent with normal tissue
Generated triplet training data consisting of (Image, Incorrect Caption, Edit Operation)
Figure 12: Illustration of the process for generating data on revision operations, using 'edit' as an example.
25

---

Published as a conference paper at ICLR 2025
C.2 PROMPTS
This section presents all the prompts used in our dataset and experimental process, including: (1) The
prompt in Figure 13 is used with GPT-4V to add details and enhance existing image captions. (2)
The prompt in Figure 14 is employed with GPT-4o to evaluate the quality of descriptions generated by two comparative multimodal models. (3) The prompt in Figure [5] is used to transform generated captions into erroneous versions and to specify the corresponding operations (add, delete, edit). By reversing this process, it generates sequence of corrections that transition from erroneous to correct captions, serving as training data for the Revise Agent. For specific examples, please refer to Section C.1.2 (4) The prompt in Figure [6 generates open-ended QA instruction-tuning data based on existing descriptions. Additionally, Figure 17 is used to create multiple-choice QA instruction-tume tuuning data. (5) The prompt in the left part of Figure 18 is used with GPT-4 to generate attribute-based prompts for PathGen-CLIPeont (e.g., nuclear atypia, pleomorphism, stroma in lung WSI), and the right part of Figure 18 prompts GPT-4 to extract information related to pathological features from WSI reports, aiding in the generation of finding-base -bsee retrieval prompts for PathGen-CLn snit- (6) We use the prompt "This is histology image from the {source of tissue]. Describe this image in detail." to prompt PathGen-LLaa dess in generating detailed descriptions of the images. In this prompt, "[source of tissue)" specifies the origin of the tissue sample.
C.3 DOWNSTREAM TASKS AND DATASETS
C.3.1 DATASETS FOR ZERO-SHOT AND FEW-SHOT IMAGE CLASSIFICATION
We employ nine datasets for tasks related to zero-shot and few-shot image classification, which include: CRC100K (Kather et al. 2018) This dataset comprises image patches taken from H&E stained histological samples and spans both colorectal cancer tissues and normal tissues. is di- vided into nine distinct categories: Adipose, Background, Debris, Lymphocytes, Mucus, Smooth Stroma, and Colorectal Adenocarcinoma Ep- ithelium. WSSS4LUAD (Han et al.) 2022); Featuring patch-level annotations from 87 whole slide images, this collection is focused on distinguishing between tumor and normal tissue classes. LC25000 (Borkowski et al. 2019): This dataset includes samples of lung and colon adenocarcino- mas, organized into two subsets: LC-lung, which encompasses lung adenocarcinomas, lung squamous cell carcinomas, and benign lung tissues; and LC-colon, which contains colon adenocarcinomas and benign colonic tissues. PatchCamelyon (Veeling et al.] 2018): Originating from histopathological scans of lymph node sections, each image in this dataset carries a binary label indicating the presence or absence of metastatic tissue. SICAPv2 RSulya-Rodriguez et al. 2020): This dataset features images of prostate pathology magnified 10 times, classified as non con-canc,rous, and Grades 3-5 according to the Gleason grading system. BACH (Aresta et al. 2019): Incorporating H&E stained breast histology images, this dataset categorizes images into four groups based on the dominant cancer type: normal, benign, in situ carcinoma, invasive carcinoma. Osteo (Arunachalam al., 2019): Sourced from whole slide images, this dataset aims to classify different tissue regions as viable tumors, necrotic tumors, or non-tumors, capturing the diverse responses of osteosarcoma to chemotherapy. SkinCancer (Kriegsmann et al., 2022):: Consisting of tissue patches from skin biopsies across 12 anatomical compartments and 4 types of neoplasms, this dataset provides a comprehensive view of skin cancer variations.
Muscle, Normal Colon Mucosa, Cancer-Associaa
C.3.2 EXPERIMENT DETAILS OF LINEAR PROBING
The linear probe experiment designed to evaluate the feature representation of a pre-trained model by adding linear layer to its output. This linear layer maps the model's 's output vector to the number of classes for classification. We conduct the experiment using a batch size of 32 and run it for 20 epochs. The optimizer used is AdamW with a learning rate of 1 x 10-2 To ensure robustness and reproducibility, we utilize 10 different seeds. The procedure involves randomly selecting 256 samples from each class to form the training set. If an official test set is unavailable or lacks labels, the remainder of the dataset serves as the test set. Throughout the 20 epochs, we select the best-performing model based its accuracy the test set.
26

---

Published as a conference paper at ICLR 2025
C.3.3 DATASETS FOR WSI CLASSIFICATION
This paper provides three commonly-used WSI classification datasets, CAMELYONI6 (ittens et al.
2018), CAMELYON17 (Litjens et al. 2018), and BRACS (Brancati al. 2022), for evaluating PathGen-CL.J representation quality for WSI classification, with rigorous and standardized splits ensure robust model training and validation.
To be specific: CAMELYON16 consists of 400 WSIs, with 270 assigned for training and 130 for testing. To enhance model validation, the training is further divided into training and validation subsets in a 9:1 ratio, following the methodology in recent studies. CAMELYONI7 comprises 1,000 WSIs sourced from five different hospitals. These slides are categorized based labels such as Normal, isolated tumor cells, metastsstsssssss and Macro-metastases. Due to the lack of labels in the official test set, the dataset's training set of 500 WSIs reallocated to assess out-of-distribution (OOD) performance. Specifically, 200 WSIs from the fourth and fifth hospitals are designated as the test set, while the remaining 300 WSIs are split into training and validation sets 9:1 ratio. BRACS (BReAst Carcinoma Subtyping) includes 547 WSIs stained with Hematoxylin and Eosin (H&E), representing three lesion types: benign, malignant, and atypical. These are further divided into seven subcategories. Due to the limited number of WSIs, only three-class subtyping (benign, atypical, and malignant) is performed. The WSIs are segmented into non-overlapping patches of 224 x 224 at 20x magnification. The dataset is officially split into 395 training images, 65 validation images, and 87 test images, and this split is adhered to in subsequent analyses.
C.3.4 ADDITIONAL EXPERIMENTAL DETAILS FOR WSI CLASSIFICATION
Data Pre-processing, For the data pre- processsign we employ the method described CLAM (Lu et al. (0021), which involves threshold segmentation and filtering to identify tissue regions within each whole-slide image (WSI). From these identified regions, we extract non-overlapping patches at a magnification of x20. Specifically, the crop size for the ViT-Base model 224 224, whereas for the ViT-Large model, the crop size is 336 x 336.
Feature Extraction. This paper selects OpenAl CLIP (Radford et al. 2021 several of its variants specifically developed for the biomedical domain, and our PathGen-Cue CLIP as a feature extractor. Features are extracted and saved with a dimension of 512 for ViT-Base and 768 for ViT-Large. In line with the regular practice of MIL methods, the feature extractor is frozen during training to save computational resources and storage.
Model Architecture. The MIL framework commonly used for WSI classification includes three learnable components: (1) fully-connected layer reduce the dimensionality of features to 256 for the ViT-Base model and 384 for the ViT-Large model. (2) An attention network to aggregate and transform the instance features. (3) A final fully-connected layer for making predictions. ABMIL (lsse) et al.,,22000 2018)) and ACMIL (Zhang et al., 2023b) share the same fully-connected layers for reducing feature dimensionality and making predictions. For the attention network, ABMIL uses the gated attention network, and ACMIL introduces Multiple Branch Attention (MBA) and Stochastic Top-K Instance Masking (STKIM) based on the gated attention network.
Training. The models are trained for 50 epochs using cosine learning rate decay schedule. The initial learning rates are determined through grid search within the range [0.0001, 0.0002, 0.0005). based validation performance. The training process utilizes the Adam optimizer with a weight decay of 0.0001, and the batch size is consistently set to 1. Otherwise, we set M = 5. K = 10, and p 0.6 for ACMIL.
C.4 HARDWARE
We utilize 24 NVIDIA A100-80G GPUs for caption generation, 8 NVIDIA A100-80 G GPUs for training the PathGen-La VL model, 4 NVIDIA A100-80G GPUs for fine-tuning LLaMA, and NVIDIA A 4100-40 G GPUs for training and testing downstream datasets.
27

---

Published as a conference paper at ICLR 2025
C.5 TRAINING HYPERPARAMETERS
For the CLIP training, we adhere to the open_elip framework and use OpenAl CLIP as initialization. We learning rate of 3e-5 with Adam optimizer that includes weight decay of 0.1. We batch size of 96 across NVIDIA A100 GPUs, resulting in an effective batch size of 384. In the first stage of training using PathGen-1.66 we limit the training only one epoch. For the second stage of training with PathGen,mit, we conduct two epochs.
For the training of PathGen-LLate LLaVA-v1.5-133 (Liu et al., 2024) as the LLM component. We fully adhere to the training framework and parameters provided in LLaV: VA framework 2 The training follows a two-stage process: in the first stage, we align the LLM with PathGen-CLIP-L using the PathGen dataset, and in the second stage, we train using the PathGeen-Instruct22000 dataset. We employ 8 NVIDIA A A100-80G cards for the training of PathGen-LLaVa
we use our trained PathGen-CLIP-A as the vision encoder and
This microscopic image of cells tissues. Please first describe the morphological characteristics of the cells or tissues in image, and then supplement and correct according to the additional description provided, but only include features that observable in the image.
Be aware that some details in the additional description might not present or identifiable in the image. Focus solely on the characteristics of the cells tissues. Da not mention you have additional description such 'considering the additional description', 'based the given description'. Do not ask for other information or give unrelated response.
Image
Additional description (description)
Figure Prompt for GPT-4V to add details and enhance existing image captions.
need to evaluate the performance two models generating descriptions for pathological image. The task both models is to meticulously describe the visual pathological features evident in the provided image. Below are the responses from the two models. You are required to:
1. Carefully examine the provided pathological image and compare the responses from both models. Please focus the description of the visual pathological features in the image and avoid placing much emphasis purely textual pathological expressions that unrelated to the image.
Image
Provide comparison the responses based the given image, and analyze which model performed better. Conclude by stating which model superior in the line using the format "Model better" "Both models equally good" you believe both models performed equally well.
Model A: (Caption generated by A}
Model B: (Caption generated by B}
Figure 14: Prompt for GPT-4o to evaluate the quality of descriptions generated by two comparative multimodal models.
28

---

Published as a conference paper at ICLR 2025
You need to modify the description generated by multimodal model to intentionally introduce discrepancies that are inconsistent with the original image content described. The changes should adhere to the following requirements:
1. Modifications must be logically coherent and consistent with general pathological knowledge, but they should contradict or omit details relative to the described Image content.
2. You need specify the changes you make. For additions, include the sentence preceding the new content. Format modifications JSON array, with each change detailed follows:
edits: l'before': original text', 'after': 'modified text', 'mode': 'edit', cedit_conteneee 'description of change')
For additions: ('before': ", 'after': 'added text', 'mode': 'add',
'text before addition))
For deletions: f'before': 'original text', 'after': 'mode': 'delete")
Ensure there are 3-4 additions and deletions. Present the modifications in JSON format like:
I"changes": [change1, change2,
Description: (description)
Figure 15: Prompt for GPT4 to transform generated captions into erroneous versions and to specify the corresponding operations (add, delete, edit).
This image pathological microscopic view of cells/tissues, Below is the image description.
Description:
(description)
Please develop 2-5 high-quality, logically structured, challenging Q&A questions. Keep following guidelines in mind:
1. Focusing cellular and tissue feature http///mooooo/iaanoooooooo/ diaagnois.
Frame questions resemble professional pathology exam questions, and ensure each question has an answerable solution.
3. The questions should designed that the answers are discernible only through careful observation of the Image.
4. Ensure the answers are in detail and cannot be easily guessed.
5. Besides asking questions like "xxx feature aatpeest//indicatie///oooo
with xxx," xox,"yon you must include questions that directly ask about the characteristics observed in the objects within the pathology image, such as "What features does the cell/tissue/xxx exhibit?"
6. Provide the questions along with their respective answers. Do not mention the answers are based any provided description.Inil "n\n"" The answer should fallow JSON format: "questions""("qqusttiiii (["uquetto"""" xxx, "answer": xox]]]"
Figure 16: Prompt for GPT4 to generate open-ended QA-based instruction-tuning data based on existing descriptions.
This image pathological micrascopic view of cells/tissues. Below is the image description.
Description:
(description)
Please develop 1-3 high quality multiple-chote questions. Keep the following guidelines mind:
1. Focusing the cellular and tissue feature/more http///ooooooooolooissssss
Frame questions resemble professional pathology exam questions, and that each question answerable solution.
3. Besides asking questions like "xox feature suggests/iniicativ//coosss with xxo," you must include questions
that directly ask about the characteristies observed in the objects within the pathology image, such as "What features does the cell/tissue/xxx exhibit?"
4. The questions should be designed that the answers are discernible only through careful observation of the image.
5. The correct answer choices need to be deceptive; they cannot be easily guessed by just taking shot in the dark.
6. Provide the questions along with their respective answers and explanations. Do not mention that the answers are based on any provided description.
The should follow JSON format:
[[" (("question": xxox, "options": xxx'.....], "answer": xxx, "explanation": xox]]]"
Figure 17: Prompt for GPT4 to generate multi-choice QA-based instruction-tumi -tnning data based on existing descriptions.
29

---

Published as a conference paper at ICLR 2025
Your task extract descriptions of histomorphologica characteristics evident whole slide image from the pathology report. The summary should:
This image is tissue patch from a (source of tissue) whole slide image of TCGA- (source TCGA). Please list 20 pathological features or attributes that might appear in this type of whole slide image.
Concentrate solely the features visible in the microscopic image, disregarding any additional, non-visible details like descriptions the gross morphology the spteilnn
specimen, clinical data, and measurements in centimeters (cm) and grams (g)-
2. Retain the original terminology with high fidelity.
like 'mentioned, "description; or 'report.' Should the summary surpass 50 words, must divided into multiple segments, each not exceeding 50 words.
Refrain from using
5. The summary must be presented in JSON format, structured
coammary_parttlle "scontent>", "summary_pare "scontent>")
Report: (WSI report)
Figure 18: Prompt for generating attribute- based and finding andeng-based text prompts for PathGen-CLAn used to retrieve the most representative patches from WSIs.
Table 12: Classes for each dataset on zero-shot image classification. Note that we used the same prompt templates for each dataset. The templates used are: ['An H&E image of (} 'this is an image of presented in image', "An H&E patch of {
Dataset
Classes
PatchCamelyon
'Iymph node', 'lymph node metastasis"
NCK-CRC
Addipose., 'Debris', "Lyyppocytes', 'Mucus', 'Smooth
muscle", 'Normal colon mucosa', "Cancer-associatee stroma', 'Colorectal adenocarcinoma epithelium'
LC25000Lung
'Lung adenocarcinoma" 'benign lung tissue", 'lung squamous cell carcinomas'
LC22000Colon
'Colon adenocarcinoma', 'normal colon tissue'
BACH
'Benign tissue', 'In-situ carcinoma", 'Invasive carcinoma', 'Nommal tissue'
SICAPv2
NNon-cancerous', 'Atrophic well differentiated and dense
glandular regions', CCibriform, ill-formed, large-fused and papillary glandular patterns', 'Isolated cells or file of cells, nests of cells without lumina formation and pseudo-rosetting patterns'
Osteo
NNon-tmoor, Necroticc tumor', 'Viable tumor'
SkinCancer
Non-tumor chondral tissue", 'Non "Non-tumor dermis', Non-tumor elastosis', 'Non-tumor epidermis',
NNnnttumrr hair follicle', 'Non-tumor skeletal muscle', 'Non-tumor necrosis', 'Non-tumor nerves'. 'Non-tumor sebaceous glands', 'Non-tumor subcutis', 'Non-tumor sweat glands', 'Non-tumor vessel', 'Tumor epithelial basal cell carcinoma', 'Tumor epithelial squamous cell carcinoma', 'Tumor melanoma', 'Tumor naevus'
WSSS 'tumor', 'normal'
30

---

Published as a conference paper at ICLR 2025
Table 13: Datasets used in our study and their corresponding source links
Dataset
Link
PatchCamelyon
Detps://patchcanl yon, grand
org. load/
CRC-100K
CCLDa://2eno20 .0gg/ /ecordB/L2:1135
SICAPv2
h=tps:/
mence Ley, asess? 9dve3/1
BACH
hctDa://LO1ar/U1 cha Lenge grand chal lenge org Dataset/
Osteo
/ jol.nass...d..........
02:0706
SkinCancer
hLLps:/ (ebps///heidee hww.hheidaaaa...
10.11598/cata/ TOCKB8
MHIST
h htt////mr/... http://mrra...hhhbbbbb 16/MHT87
WSSS4LUAD
nstps:/ h htfpt://wasxuusc.raaaa chal wtgge.9o.
LCC5000 (1.C Lung and L.C
hstps:
Colon)
cetthuu..omm/tampppath. lung 00.0n on mage set 7tab- reacmeeee-reer-eeeee -oeeeooo--ll
BRCAS
h=tps:/ braas. 1ea7, ChF.it/
Camelyon17 17
h2 h Toame Von grard -ha lenge.org/Date
16
hstps://cee /eeme Vonlo. wwa.e- cha wenccoogg.... Data/
PathMMU
31

---

Published as a conference paper at ICLR 2025
D EXAMPLES
D.1
GENERATED IMAGE-CAPTION PAIRS
Figures 20 to 24
showcase the descriptions we generated and the captions after summarization.
D.2 GPT-4o EVALUATION EXAMPLES
showcase the evaluation of description quality generated by PathGen-LLaVA, Quilt-LLaVA, and LLaVA-Med-v1.5 using GPT-4o. These figures intricately demonstrate the specific strengths of our generated descriptions.
Figures 25 to
D.3 PATHGEN-INSTRUE EXAMPLES
Figures 29 and 30 showcase the multi-choice QA-based and open-ended-based instruction-tuning samples in PathGeen-nstruett2200
OpenPath
PathCap
Quilt
PathGen(Ours)
Figure 19: Comparison of image quality from different datasets: OpenPath, PathCap, Quilt, and PathGen.
32

---

Published as a conference paper at ICLR 2025
Tissue origin: Liver
The liver tissue section displays hepatocytes with large, round nuclei and eosinophilic cytoplasm, some binucleated. Notable vacuoles lipid droplets in the cytoplasm. Sinusoids contain red blood cells; the architecture is largely intact with signs of fibrosis, inflammation, or neoplastic changes. Further testing required for definitive diagnosis.
Summarized caption:
Generated description: The image shows section of liver tissue with numerous cells exhibiting various morphological characteristics. The cells have large, round to oval nuclei with prominent nucleoli, which is typical for hepatocytes. Some of the cells contain multiple nuclei, suggesting they may be binucleated or multinucleated, which can be normal variation in liver tissue. The cytoplasm of the cells appears eosinophilic, which pinkish color due the staining technique used.
There are clear spaces within the cytoplasm of some cells, which could represent vacuolation lipid droplets, common feature certain liver pathologies. overall architecture the tissue appears to be preserved, with no immediate signs of fibrosis or significant disruption of normal lobular pattern.
The presence of red blood cells can observed within sinusoidal spaces, which are the small blood vessels within the liver. There is obvious evidence of inflammatory infiltrate neoplastic changes
particular field of view. However, without additional context or staining, it is difficult to provide definitive diagnosis to identify subtle changes that might be present.
Figure 20: An example generated description and summarized caption for a given pathology image. Different pathological descriptions from various aspects marked in distinct background colors.
33

---

Published as a conference paper at ICLR 2025
Tissue origin: Thyroid
Summarized caption: The histological section shows pink-staining collagenous stroma and cells with dark nuclei, likely lymphocytes, indicating inflammation. Larger cells suggest histiocytes. Absence of thyroid follicles suggests atrophy. These features, combined with the clinical history, may indicate Hashimoto's thyroiditis patient with Graves' disease and diffuse goiter. Confirmation requires further clinical and immunohistochemical evaluation.
Generated description: The image shows section of tissue with cells that have somewhat irregular architecture. The cells appear be follicular in nature, which typical thyroid tissue, They are organized in pattern that is somewhat disrupted, with variable follicle sizes and shapes. The nuclei of the cells enlarged and hyperchromatic, meaning they darker than usual, suggesting high -uclear-to-oyooplasmic ratio. There are also some nuclei that appear have irregular contours and some with prominent nucleoli. stroma, or connective tissue, between the follicles seems to be increased, which could indicate fibrotic process. These features are suggestive of neoplastic process, and given the clinical context a thyroid nodule, could be consistent with thyroid carcinoma. However, definitive diagnosis would require clinical correlation and possibly additional immunohistochemica staining molecular testing.
Figure 21: An example generated description and summarized caption for a given pathology image. Different pathological descriptions from various aspects are marked in distinct background colors.
34

---

Published as a conference paper at ICLR 2025
Tissue origin: Lung
Summarized caption: The lung tissue image displays cells with a high pucleus-to-cytopee hyperchromatic and pleomorphic nuclei, suggesting cellular atypia. Cells form glandular patterns,, indicative of adenocarcinoma. The stroma is fibrous with inflammation. Differential diagnosis points to lung carcinoma, possibly adenocarcinoma, pending further clinical and molecular analysis for confirmation,
ratio,
Generated description: The microscopic image shows section of lung tissue with several morphological features. The tissue is composed of cells with high nucleus-to-cyton to-cytoplasm ratio, which indicative of high cellular density. The nuclei appear hyperchromatic and pleomorphic, suggesting variability in size and shape, which can be sign of cellular atypia. There are also areas where the cells arranged glandular or acinar pattern, which typical of certain types of lung tissue, such as adenocarcinoma. The stroma, connective tissue, appears fibrous with some inflammatory scattered throughout.
Based on the morphological characteristics observed in the image, a differential diagnosis could include form of lung carcinoma, such as adenocarcinoma, given the glandular structures cellular atypia. However, a definitive diagnosis would require clinical correlation and possibly additional immunohistochemica staining or molecular studies.
Figure 22: example of generated description and summarized caption for given pathology image. Different pathological descriptions from various aspects are marked in distinct background colors.
35

---

Published as a conference paper at ICLR 2025
Tissue origin: Bladder
Summarized caption: The image shows malignant urothelial carcinoma cells with high nucleus-to- cytoplasm ratios, pleomorphic nuclei, and prominent nucleoli on the left. These cells infiltrate the organized, spindle-shaped cells of the muscularis propria on the right. The irregular cell shapes and disrupted tissue architecture indicate advanced disease, reflecting the cancer's penetration into deeper bladder layers.
Generated description: The image displays section of tissue with various cellular and structural features. On the left side of the image, there is a cluster cells with a high nucleus- to-cytoplasm ratio, pleomorphic nuclei, and prominent nucleoli, which are indicative of malignant cells. These cells are irregularly shaped and vary in size, suggesting a lack of normal cellular differentiation and tissue architecture.
The side of the image shows more organized structure with elongated cells that have regular, spindle-shaped morphology. These cells are arranged in parallel fashion, which is characteristic of smooth muscle fibrous tissue. The boundary between the two sharply defined, indicating possible infiltration of the malignant cells into the adjacent tissue.
not
The overall features are consistent with urothelial carcinoma, which common type of bladder cancer. The presence of atypical cells infiltrating into the muscularis propria suggests higher stage of disease, as this indicates that the cancer has penetrated beyond the urothelial lining into the deeper muscle layer bladder wall.
Figure 23: An example of generated description and summarized caption for given pathology image. Different pathological descriptions from various aspects are marked distinct background colors.
36

---

Published as a conference paper at ICLR 2025
Tissue origin: Colon
Summarized caption: The colonic tissue image shows mucosal layer with columnar epithelial cells and goblet cells, and lamina propria with dense inflammatory infiltrate. Crypt architecture distorted. These findings suggest chronic colitis. Differentiation between ulcerative colitis and Crohn's disease requires further clinical correlation and specific pathological characteristics like ulcers or granulomas.
Generated description: The image shows section colonic tissue with several notable features. The mucosal layer is visible with columnar epithelial cells lining the lumen. Goblet cells, which are mucus seccutsn cells, can seen interspersed between epithelial cells. The lamina propria, which is the connective tissue layer beneath the epithelium, appears to have dense infiltrate inflammatory cells, including lymphocytes plasma cells.
There also evidence crypt architectural distortion, with some crypts appearing irregular and branching, which sign of chronic inflammation. The submucosa appears to be expanded and contains more inflammatory cells, and there is loss af the normal orderly arrangement of the crypts.
The presence of these features, particularly the crypt distortion and the dense inflammatory infiltrate in the lamina propria, consistent with chronic colitis. The specific diagnosis of ulcerative colitis (UC) Crohn's disease (CD) would typically require clinical correlation and additional features not visible in this image, such as ulcers or granulomas respectively.
Figure 24: An example of generated description and summarized caption for given pathology image. Different pathological descriptions from various aspects are marked distinct background colors.
37

---

Published as a conference paper at ICLR 2025
Model A (Quilt-LLaVA):
Upon examining the histopathology image the breast,
observe a lobule that is not densely packed with cells. The cells within this lobule predominantly epithelial cells, which breast tissue.
the primary constituents
In the upper left quadrant of the image, there is significant presence collagenous stroma. stroma, which the supportive framework of the breast tissue, appears to be interspersed with fibroblasts. These fibroblasts are cells that produce collagen and other fibers, contributing the structural integrity the tissue.
Moving towards the center and slightly to the right of the image, there is a region that is almost entirely composed collagenous stroma. This area characterized by dense, fibrous structure, indicative of a high collagen content. The fibroblasts this region appear to less numerous, suggesting mature and well cell-struutured stromal component.
Model (PathGen-LLavA):
The microscopic image shows section of breast tissue with several distinct features. There are multiple adipocytes, which are the large, white, empty-looking spaces that characteristic of breast tissue. These cells have thin, peripheral nuclei due large vacuole stored that pushes the nucleus to the side.
Interspersed between the adipocytes fibrous stromal elements, which appear pink, thread-like structures. These are collagen fibers providing structural support to tissue.
Within the stroma, there clusters of cells with enlarged, hyperchromatic nuclei and some degree pleomorphism, which suggests increased nuclear-to-cytoplasm ratio. These cells are irregularly distributed and vary in size and shape, which indicative neoplastic changes.
The presence of these atypical cells within the breast tissue, along with the architectural disruption of the normal glandular structures, is suggestive an invasive carcinoma. The cells' morphology, including their nuclear atypia and disorganized growth pattern, supports this diagnosis.
GPT-4o: Comparison of Responses Based on the Given Image
1. Descriiptin Lobule and Epithelial Cells:
1. Mentions labule that is not densely packed with cells and notes the presence of epithelial cells.
2. description does not seem directly correspond given image, where no clear lobules or evident.
epithelial
2. C.lllagenous Stroma:
1. Describes significant presence collagenous stroma in the upper quadrant and centrally.
2. Correctly identifies pink, structures as collagenous stroma notes the presence fibroblasts.
Model B:
1.Adipocytes:
1. Accurately identifies large, white, empty-looking spaces as adipocytes, which characteristically contain
vacuoles
2. This visible in the provided image, corresponding to the large spaces with thin peripheral nuclei.
2. 2.Fibrous Stromal Elements:
Apprapriately describes pink,
structures as stromal elements (collagen fibers).
3.Cellular Features and Neoplastic Changes:
1. Mentions clusters of cells with enlarged, hyperchromatic nuclel and some degree of pleomorphism, Indicating
nuclear atypia and neoplastic changes.
2. Suggests the presence of invasive carcinoma based the description these atypical cells.
3. The given image daes show irregular nuclei, which aligns with this description.
Analysis:
image. mention epithelial cells lobules reflect actual contents depicted. Additionally, while correctly identifies collagenous stroma, adds details about fibroblasts which are not prominently seen or inferable from the image alone.
-Model provides
description
-Model accurately describes the adipocytes and fibrous stroma, which clearly visible in the image. key pathological features such enlarged, hyperchromatic nuclei and their disorganized pattern, indicative of neoplastic changes, possibly pointing towards invasive carcinoma. This description aligns closely with the visible histological features in the given image.
identifies
Conclusion:
Model better.
Figure 25: An example of GPT--40's evaluation and comparison of generated descriptions by PathGen-
LLa dAde and Quilt-LLaVA.
38

---

Published as a conference paper at ICLR 2025
Model (PathGen-LLavA):
The microscopic image displays section of renal tissue with several notable features. The cells exhibit a high degree of pleomorphism, with some cells having large, hyperchromatic nuclei and prominent nucleoll, which are indicative of malignant transformation. There is also evidence of abnormal mitotic figures, which are consistent with neoplastic activity. The architecture of the tissue disrupted, lacking the organized structure typically seen in normal renal parenchyma. The presence of clear cells, which are large cells with abundant clear cytoplasm and distinct cell borders, characteristic clear cell renal carcinoma (RCC). The stroma appears to be sparse and vascularized, which common in clear cell RCC. The overall features observed are consistent with diagnosis clear cell renal cell carcinoma.
Model (Quilt-LLaVA):
The histopathology image reveals kidney tissue sample with notable presence of inflammatory cells. These cells scattered throughout the tissue, indicating ongoing inflammatory response. cells are predominantly located the interstitial spaces, which common inflammation in the kidney.
addition the interstitial infiammation, there also evidence inflammation within the tubules. This tubular inflammation characterized by the infiltration inflammatory cells into the tubular lumen. The tubules appear to be affected in a patchy manner, suggesting non-uniform distribution of the inflammatory process.
The image also shows few areas where the inflammation seems to more pronounced. These areas are characterized by higher density of inflammatory cells, indicating more severe or active phase of inflammation. The overall picture is suggestive of chronic inflammatory process, possibly due ongoing disease condition.
Upon examining the provided pathological image, the following comparisons and analysis can be made about the responses from both models:
GPT-4o:
Model
DDeccriess the tissue
with specific emphasis malignant features.
*entions high pleomorphism, hyperchromatic nuclel, and prominent nucleoli Indicative of malignancy.
*Notes presence abnormal
figures disrupted iidentifies clear cells with abundant dear cytoplasm, linking these features clear cell renal cell carcinoma (RCC).
+Highlights sparse, vascularized stroma, characteristic of clear cell RCC, and concludes with diagnosis of clear cell RCC. B:
Describes the tissue kidney mentions
of infiammatory cells.
Focuses on the distribution of inflammatory cells in interstitial spaces and tubules.
Describes the inflammation as patchy with some areas having higher density infiammatory cells. possibly
Concluues suggesting chronic inflammatory
ongoing disease condition.
Comparison and Analysis:
*Accuracy in Describing Visual Features:
Model Accurately identifies features such pleamorphism, hyperchromatic nuclei, prominent nucleoli, abnormal mitotic figures, clear cells with abundant cytoplasm, and vascular stroma which are all visible in the image. provides cohesive description that matches the pathological feetures seen.
Model Mentions inflammatory cells and their distribution in interstitial spaces and tubules. While there scattering of cells, the description does match the image which heavily leans towards features of malignancy, specifically characteristics of clear cell RCC rather than chronic inflammation.
Overall Observation:
detalls Model properly identified the specific of pathology, matching description with the visual details in the
image such clear cells and pleomorphism.
inflammatory processes that indicate malignant characteristics.
Model provides detailed
align image
Based on the accuracy and relevance to the provided image:
Figure 26: An example of GPT-4o's evaluation and comparison of generated descriptions by PathGen- LL dedep and Quilt-LLaVA.
39

---

Published as a conference paper at ICLR 2025
Model (Quilt-LLaVA):
The histology image of the prostate reveals number of intriguing features. The most striking aspect is the presence of nucleoli within the nuclei of cells. These nucleoli are clearly visible, indicating high level of cellular activity. However, it's important to note that the presence of nucleoli not necessarily indicative malignancy. fact, it's quite common to observe nucleoli in prostate cells, even in healthy state.
Model (PathGen-LLaVA))
The image shows section of prostate tissue with glandular structures that are irregularly shaped and lined by epithelial cells. The cells exhibit some variability in size and shape, with prominent nucleoli, which is suggestive of neoplastic changes. The stroma between the glands appears fibrous with pinkish hue, indicating connective tissue. There is no clear evidence of invasion into the surrounding stroma by the glandular structures, which would be feature suggestive of malignancy. However, the presence of architectural distortion and cellular atypia could consistent with prostatic intraepithelial neoplasia (PIN), which precursor to prostate cancer.
GPT-4o: Upon examining the provided histology image of the prostate and comparing the responses, the following analysis can be made:
Model
*Deccribes the presence of nucleoli within the nuclei of the cells, which accurate upon observation the
*Highlights that nucleoli are indicative a high level of cellular activity but not necessarily malignancy. LLccks detail about other features present in the image, such as glandular structures, stroma, or variability cell size and shape.
Model
*Offers more comprehensive description of the pathological features evident in the image. *Acuurateyy describes the presence irregularly shaped glandular structures lined by epithelial cells and notes variability shape.
.Commntts prominent nucleoli, suggestive of neoplastic changes, correlating well with the image.
-includes observation of the fibrous stroma with pinkish hue, indicating connective tissue presence.
Mentions the lack of evidence of invasive malignancy yet observes architectural distortion and cellular atypia, suggesting possible prostatic intraepithelial neoplasia (PIN).
Given the image and the required detailed pathological description, Model has provided thorough and accurate lysis of the visual features in the image. it covers more aspects of the tissue structure and offers insights into potential
pathological conditions.
Model 8 is better.
Figure 27: An example of GPT-4o's evaluation and comparison of generated descriptions by PathGen- LLaVA and Qui.t-LLaVA.
40

---

Published as a conference paper at ICLR 2025
Model (Quilt-LLaVA):
Upon examining the histopathology image of the breast, I observe a lobule that is not densely packed with cells. The cells within this lobule are predominantly epithelial cells, which are the primary constituents the breast tissue.
In upper left quadrant presence of collagenous stroma. This stroma, which the supportive framework of the breast tissue, appears to be interspersed with fibroblasts. These fibroblasts cells that produce collagen and other fibers, contributing to the structural integrity of the tissue.
image, there significant
Moving towards the center and slightly to the right of the image, there is a region that almost entirely composed of collagenous stroma. This area is characterized by dense, fibrous structure, indicative of high collagen content. The fibroblasts in this region appear to be less numerous, suggesting mature well-structured stromal component.
Model (PathGen-LLaVA):
The microscopic image shows section of tissue with several distinct features. are multiple adipocytes, which are the large, white, empty-looking spaces that are characteristic of breast tissue. These cells have thin, peripheral nuclei due to the large vacuole of stored fat that pushes the nucleus side.
Interspersed between the adipocytes fibrous stromal elements, which appear pink, thread-like structures. These are collagen fibers providing structural support the tissue.
Within the stroma, there clusters of cells with enlarged, hyperchromatic nuclei and some degree of pleomorphism, which suggests increased nuclear-to-cyee seddear-tocctoollasmic ratio. These cells irregularly distributed and vary in size and shape, which is indicative of neoplastic changes.
The presence of these atypical cells within the breast tissue, along with the architectural disruption of the normal glandular structures, is suggestive of an invasive carcinoma. The cells' morphology, including their nuclear atypia and disorganized growth pattern, supports this diagnosis.
-oo Comparison of Responses Based on the Given Image
Model A:
1.Description Lobule Epithelial
1. Mentions lobule that is not densely packed with cells and notes the presence of epithelial cells.
2. This description does not seem to directly correspond to the given image, where no clear lobules or
concentrated epithelial cells are evident.
Collaagenous
Describes significant presence of collagenous stroma the upper left quadrant and centrally.
2.
Correctly identifies the pink, fibrous structures as collagenous stroma and notes the presence of fibroblasts.
Model B:
1.Adipocytes:
1. Accurately identifies the large, white, empty-looking spaces adipocytes, which characteristically contain vacuoles pushed to the side.
2.
visible the provided image, corresponding large spaces thin peripheral nuclei.
2. 2.Fibrous Stromal Elements:
1.
Appropriately describes the pink, thread like structures fibrous stromal elements (collagen fibers).
3. 3.Cellular Features and Neoplastic Changes:
1. Mentions clusters cells with enlarged, hyperchromatic nuclel some degree pleomorphism, indicating nuclear atypia and neoplastic changes.
2. Suggests the presence of invasive carcinoma based on the description of these atypical cells.
3. The given image does show irregular nuclei, which aligns with this description.
Analysis:
-Model provides less accurate description of the image. The mention of epithelial cells and lobules does not reflect the actual contents depicted. Additionally, while correctly identifies collagenous stroma, it adds details about fibrablasts which are not prominently seen or inferable from the image alone. fibrous
-Model accurately describes adipocytes key pathological features such enlarged, hyperchromatic nuclei and their disorganized pattern, indicative of neoplastic changes, possibly pointing towards invasive carcinoma. This description aligns closely with the visible histological features in the given image.
which clearly
image. identifies
Conclusion:
Model better.
Figure 28: An example of GPT-4o's evaluation and comparison of generated descriptions by PathGen- LLa VAde and LLa LLaVA-Med-v11.5
41

---

Published as a conference paper at ICLR 2025
Multi-choice QA
Question: Which of the following cellular features is observed the epithelial cells lining duct-like structures?
Option:
(A) Necrosis
(B) Increased mitotic figures
(C) Hyperchromasia
(D) Presence of prominent nucleoli
Answer: C
Question: Which feature the image is atypical for normal liver architecture?
Option:
(A) Presence of hyperchromatic nuclei
(B) Scattered inflammatory cells in the surrounding
stroma
(C) Presence of pink-staining collagen fibers the stroma
(D) Hepatocytes arranged in trabecular pattern
Answer:
Figure 29: An example of a multi-choice QA-based instruction tuning sample from MallGen-Instruck 200K.
42

---

Published as a conference paper at ICLR 2025
Dialogue-based Open-ended QA
What characteristic feature of the nuclei is observed in the spindle-shaped cells this microscopic image?
The nuclei the spindle-shaped cells are elongated and vary in size and shape, with some showing hyperchromasia.
Describe the arrangement of the cells within the tissue section. What does this arrangement suggest about the type of cells present?
The cells are arranged in interlacing fascicles, which is characteristic of smooth muscle myofibroblastic proliferation.
What type of cytoplasmic staining is observed in the cells, and what might this indicate about the cellular composition?
The cells have variable amount of eosinophilic cytoplasm, which indicates the presence smooth muscle cells or myofibroblastic cells.
What evidence there the image that the cells might experiencing cellular atypia, despite the absence of mitotic figures and necrosis?
The nuclei vary size and shape, with some showing hyperchromasia, which indicates degree of cellular atypia, though there are mitotic figures necrosis visible.
Figure 30: An example of a dialogue based open-ended QA instruction tuning sample from PathGen- InStruct-200K.
43


Reviews:
[{"review_id": "GtBAlGkXfi", "reviewer": "Reviewer_wDqp", "summary": "PathGen-1.6M represents a significant advancement in pathology AI, introducing the largest high-quality pathology image-text dataset created through multi-agent collaboration. The approach leverages whole slide images from TCGA to extract representative patches and generate accurate, detailed captions, achieving 88-90% accuracy validated by pathologists. The resulting models, PathGen-CLIP and PathGen-LLaVA, demonstrate superior performance across various tasks including zero-shot classification, few-shot learning, and whole slide image analysis, outperforming existing models including GPT-4V. This work provides a scalable pathway for generating high-quality pathology data and developing more capable AI models for clinical applications.", "questions": "- How many real pathology report findings did you'll extract in section 3 and how did you verify the quality of it\n- How do you evaluate the performance of the images retrieved for the prompts in section 3.1?\n- How many prompts do you use for each WSI?\n- How did you arrive at the design choice of using both prompt-based and clustering-based retrieval?\n- Why chose a threshold of 0.88 for similarity?\n- Is similarity computed globally in all the extracted patches across slides?\n- Does the revision agent have a no-operation capability as well? What happens when a correct description is passed to the revision agent?\n- What does first and second stage training in PathGen-CLIP mean?\n- Why did you use different datasets for zero and few shot experiments? As an example, Camelyon is not included in zero-shot examples", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Well written introduction\n- Good literature survey on LMMs and Pathology--CLIPs\n- Brilliant idea on how to develop the revise LLM agent\n- Representative patch selection using GPT-4 is an excellent idea\n- Interesting approach to select patches through clustering but modulating the number of clusters as the square root of the size of a slide\n- Strong evaluation pipeline across a range of tasks", "weaknesses": "- It's an overkill to say that this is an agent based system. Agent based systems are autonomously operating on a set of predefined rules and behaviors, while this approach appears to be more of a sequential pipeline with different models performing specific tasks rather than truly autonomous agents interacting with each other.\n- Lit survey on multi-agent architectures could be expanded\n- Usage of GPT-4s internal knowledge about the morphology of an organ is a good idea but deters diversity of patches collected. Also unclear if those prompts are well represented in the CLIP training dataset. Additional details on how many prompts obtained for each WSI will help the reader. It's unclear if each patch is matched against 2 prompts (report and attribute based) or more?\n- Added details on motivations for design choices such as including both prompt and image retrieval will help.\n- The methods section needs more details and fleshing out the writing will help; I think this is also generally true for most of the paper\n- While not the goal of the paper, it will help to include fully supervised baselines as well to educate the readers of the gap with CLIP like models\n- Details on how the instruction tuning data was curated are not provided", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xl5Thjkj3R", "reviewer": "Reviewer_TdEJ", "summary": "The paper proposes a new 1.6 million dataset of paired image-caption pathology data.\nThe authors propose a scalable way to construct the dataset using existing LMMs and refinement strategies.\nThey show this dataset is useful in developing a pathology specific CLIP model which performs better than existing domain specific and general purpose CLIP style models in zero-shot an few-shot problems\nFinally they show that scaling dataset size and model size can lead to improvements in some tasks.", "questions": "Dataset Construction:\nIts unclear why and how the subset of 700K samples was choosen from existing datasets to create PathGen_init. \nFor training the description agent, the authors mention using 10K initial image-caption pairs to generate 30K dataset, so are 3 captions generated per image?\n\nRevision Agent:\nWhat model is used for training the revision agent? Its also unclear what the inputs for revision agent are at inference? Does it take the generated caption and produce possible edits?\n\n\nIn figure 10, its unclear the w/ PathGen_init two stage performance doesnt vary when scale of PathGen data is varied? When scale is 0 it means all data is PathGen_init is that correct?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\nThe paper proposes a new way to generate quality image-caption pairs for pathology data. \nIt uses existing publicly available pathology image-caption data and the ability of newer general purpose LMMs like GPT-4 to generate detailed descriptions to create an initial dataset for training a LLaVA style captioning model.\nIt then uses this captioning model in conjunction with a revision and summary agent to generate synthetic image-caption data.\nThis approach of using existing data and LMMs to build a model and refine its outputs using other agents is interesting and not explored in the context of pathology.\nGiven the large nature of WSI images and significant redundancy and similarity of image content, the authors propose ways to construct a dataset of diverse image patches\n\nEvaluation:\nThe authors evaluate the model on various zero-shot and few-shot patch-level tasks and on WSI-level prediction tasks and compare its performance against various existing models. The results with GigaPath a vision encoder are promising and show we can use existing vision encoders and improve their language capabilities using the captioning data.\nThey also qualitatively evaluate a small subset of the generated captions using expert pathologists. \n\nExperiments:\nThe authors show various ablations around dataset construction which are valuable in understanding some of the limitations of using captioning models. The section scaling dataset and model size is interesting and shows some evidence around usefulness of scaling datasets and models using this approach.\n\nAll the data and models are publicly available which is great for the pathology community and this will also be the largest publicly available image caption dataset for pathology.", "weaknesses": "Clarity:\nWhile I appreciate the authors covering a lot of ablations and experiments and describing the prompts, many of the design choices aren't clearly explained well. I've added some in the questions below. \n\nEvaluation:\nWhile the authors do compare with many older pathology VLM models, its unclear why they couldn't get access to the more recent CONCH which is publicly available on HuggingFace. \nFor WSI tasks while its helpful they added Gigapath, they don't compare against better publicly available pathology vision encoders like \nH-Optimus.\n\nImprovements:\nGiven the setup, its unclear how much of the improvements are coming from the new PathGen data and the refinement through revision agent, given they are one of the main contributions of the paper. i.e There isnt a comparison of the performance of the original PathGen-CLIP-L_init encoder with the improved PathGen-CLIP-L. Another comparison which would be useful is understanding with/out data generated using the revision agent.\nIt would be useful to highlight these as it helps understand how well such a setup can scale in generating synthetic data for iterative refinement. \nQuality and Scale of Initial Dataset:\nIts also unclear how important the scale of the detailed caption dataset 30K used for training PathGenLlava and the quality of it. Does scaling the dataset size and having some refinement here help? Have the authors checked the quality of captions generated by GPT-4 here and can provide some insight.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fS2DeOuXB0", "reviewer": "Reviewer_Y3n6", "summary": "This paper introduces PathGen-1.6M, a large-scale dataset containing 1.6 million high-quality image-caption pairs extracted from Whole Slide Images (WSI).  It also developed a scalable approach for high-quality pathology image-text data generation by multiple agent models collaborating, paving the way for next-generation general pathology models.", "questions": "1.\tThe writing of the paper requires improvement, as there are several shortcomings. For example, there is a typo in “PathGen-LLaVAdesp” in Section 4.1, and the conclusion summarizes, “we train two advanced models: PathGen-CLIP and PathGen-CLIP-L.” It appears there is an additional model, “PathGen-LLaVA,” which needs clarification.\n2.\tA significant concern arises regarding potential data leakage in the downstream tasks and benchmarks evaluated. To my knowledge, many tasks and benchmarks are derived from TCIA pathology data, which raises suspicions about the homogeneity of the constructed dataset.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper presents PathGen-1.6M, a dataset containing 1.6 million high-quality image-caption pairs. Based on this dataset, the authors develop PathGen-CLIP, a pathology-specific CLIP model, which achieves substantial improvements across nine pathology-related zero-shot image classification tasks and three whole-slide image tasks.\n2.\tThe authors propose a data construction pipeline that employs multiple LLM agents for description, revision, and summarization. This multi-agent collaboration approach generates more accurate image-caption pairs, validated through human evaluation.\n3.\tThe experiments conducted in the paper are solid and well-executed.\n4.\tThe release of the dataset, code, and model contributes significantly to the advancement of the pathology image research community.", "weaknesses": "1.\tThe Revise LMM appears to have limited utility, primarily providing editing capabilities such as additions, deletions, or modifications. It seems ineffective in addressing common pathological inaccuracies in the generated descriptions.\n2.\tOnly evaluated in CLIP model. Can be work in more pretrained Vision-Language model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jh9TRwsT3q", "reviewer": "Reviewer_33XD", "summary": "Built upon domain-specific WSI-Reports data from TCGA, this paper proposed the largest high-quality patch-text pairs dataset for Computational Pathology (CPath) by prompting LMM to generate text descriptions for pathology patches. To this end, a scalable data curation method is proposed by leveraging several LMM agents to describe, revise and summarize the generated descriptions.", "questions": "1. In Step 5 of data construction, how do authors ensure that no essential details are lost?\n2. How can be validated the effectiveness of each step in data curation? This should be discussed in the ablation study.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. To address data scarcity in CPath for pretraining large models, a scalable data curation method is proposed to expand the limited image-text dataset.\n2. The largest patch-text pairs dataset is curated and used for pretraining to enhance the foundational power in CPath tasks.\n3. The scalability to more WSI-only data is interesting, which has the potential to greatly expand the data scale for large pathology models.", "weaknesses": "1. To support the superiority of the proposed data construction method by introducing extra patch-level supervision, the performance of directly utilizing original report data along with patch images to pretrain a pathology foundation model should be presented.\n2. During data construction, some representative patches were filtered out. These patches are supposed to align well with pathology reports, as they are retrieved based on the report data. To validate the extra contribution of generated patch descriptions, the proposed model should be compared with the one trained on these selected representative patches as well as its corresponding report data.\n3. The model’s generalizability to out-of-domain data has not been validated. The authors tried to scale the model to non-WSI report paired data. However, these data are still from TCGA.\n4. Some SOTA pathology foundation models are missing, especially in experimental comparison, such as UNI [1], CONCH[2], mSTAR[3] (which is also a CLIP-style model trained on TCGA data as well), etc.\n5. Few-shot’s capability significantly relies on how well-aligned the vision and text spaces are. The proposed method is supposed to compare with VLM like CONCH, instead of vision-only GigaPath.\n6. Details in EXPERT EVALUATION are missing. For example, how do authors define what correct or incorrect findings are?\n\nMinor concerns:\n\n- It is hard to recognize different models according to their colors. Please choose ones with higher contrast.\n- The citation of GigaPath seems to be lost.\n\n[1] Towards a general-purpose foundation model for computational pathology, Nature Medicine, 2024\n\n[2] A visual-language foundation model for computational pathology, Nature Medicine, 2024\n\n[3] A Multimodal Knowledge-enhanced Whole-slide Pathology Foundation Model, arxiv, 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)