Quilt-1M: One Million Image-Text Pairs for Histopathology
Authors: Wisdom Oluchi Ikezogwo
Mehmet Saygin Seyfioglu
Fatemeh Ghezloo
Dylan Stefan Chan Geva
Fatwir Sheikh Mohammed
Pavan Kumar Anand
Ranjay Krishna
Linda Shapiro
Venue: NeurIPS 2023 Datasets and Benchmarks Oral (2023)
Keywords: Vision-Language
Dataset
Histopathology
Medical
Video
Image-Text

Abstract: Recent accelerations in multi-modal applications have been made possible with the plethora of image and text data available online. However, the scarcity of analogous data in the medical field, specifically in histopathology, has slowed comparable progress. 
To enable similar representation learning for histopathology, we turn to YouTube, an untapped resource of videos, offering $1,087$ hours of valuable educational histopathology videos from expert clinicians.
From YouTube, we curate QUILT: a large-scale vision-language dataset consisting of $802, 144$ image and text pairs.
QUILT was automatically curated using a mixture of models, including large language models, handcrafted algorithms, human knowledge databases, and automatic speech recognition.
In comparison, the most comprehensive datasets curated for histopathology amass only around $200$K samples.
We combine QUILT with datasets from other sources, including Twitter, research papers, and the internet in general, to create an even larger dataset: QUILT-1M, with $1$M paired image-text samples, marking it as the largest vision-language histopathology dataset to date. 
We demonstrate the value of QUILT-1M by fine-tuning a pre-trained CLIP model. 
Our model outperforms state-of-the-art models on both zero-shot and linear probing tasks for classifying new histopathology images across $13$ diverse patch-level datasets of $8$ different sub-pathologies and cross-modal retrieval tasks.

Quilt-1M: One Million Image-Text Pairs for Histopathology
Wisdom O. Ikezogwo"
Mehmet Seyfioglu Fatemeh Ghezloo ~
Dylan Geva
Fatwir S. Mohammed Pavan K. Anand
Ranjay Krishna Linda G. Shapiro
University of Washington
{wisdomik, fghezloo, fatwirguw edu
fibeel, dgeva, pka4, shapiy,shpis www.apprrooos...aahhnnoooo edu
Abstract
Recent accelerations in multi-modal applications have been made possible with the plethora image and text data available online. However, the scarcity of analogous data in the medical field, specifically in histopathology. has slowed comparable progress. To enable similar representation learning for histopathology. we turn YouTube, untapped resource of videos, offering 1, 087 hours of valuable educational histopathology videos from expert clinicians. From YouTube we curate QUILT: large-scale vision-language dataset consisting of 802, 144 image and text pairs. QUIII was automatically curated using a mixture of models, including large language models, handerafted algorithms, human knowledge databases, and automatic speech recognition. In comparison, the most comprehensive datasets curated for histopathology amass only around 200K samples. We combine QUILT with datasets from other sources, including Twitter, research papers, and the internet in general, create an even larger dataset: QUILT- IM, with 1M paired image- text samples, marking it as the largest vision- language histopathology dataset to date. We demonstrate the value of QUILT- M by fine-tuning a pre- mr--triined CLIP model. Our model outperforms state-of of- -h-hhhrrt models on both zero-shot and linear probing tasks for classifying new histopathology images across 13 diverse patch-level datasets of 8 different sub-pathologies and cross-modal retrieval task
Introduction
Whole-slide histopathology images are dense information, and even individual image patches can hold unique, complex patterns critical for tissue characterization. Summarizing this information into a single label is an oversimplification that fails to capture the complexity the field, which covers thousands of evolving disease sub-types (55] This highlights the need for more expressive, dense, interconnected representations beyond the reach of singular categorical label. As such, natural language descriptions can provide this comprehensive signal, linking diverse features of histopathology sub- sub-pateh -atch structures (20 24].
If there were large-scale vision-language dataset for histopathology, researchers would be able to leverage the significant advancements in self-supervised vision and language pre-training to develop effective histopathology models [46]. Unfortunately, there is significant scarcity of comprehensive datasets for histopathology. Notable open-source contributions have been made with datasets like ARCH (20] and OpenPath (24) Yet, these sources still somewhat limited due to their size, the
RRach corresponding author at wisdomik @cs wossswwsiiggoo.... Equal contribution.
?The data and code will be available at Quilt-IM
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

former has only = 8K samples and the latter (the largest histopathology vision-language dataset to date) has about 200K samples. Although recent efforts (e-g. PMC-15M
67 curated 15M image-text
pairs across a variety of different biomedical domains from Pubmed 48 whether their samples are specific histopathology remains ambiguous; worse, their dataset is not openly available.
Image
Extrast hetepehhekyyy mages...
Devirisn by semoving
Dupicates
Data
Pair
Search
Blury imagie
Medical text
cC
ROI text
Test conection
Text extraction
UMLS entiles
Speech to text
Sub-pathoiogy
LLM
LLM
Mediccll
RDI
Magrification
P UMLS
TEXT
Partology
Figure 1: Overview of QUILT curation pipeline. We identify relevant histopathology You'ubbe videos in Search. For Image extraction, we find and de-noise histopathology frames using trained models. In Text section, we rely on a conventional Automatic Speech Recognition (ASR) model and leverage Unified Medical Language System (UMLS) and large language models (LLMs) for post- processing and ASR eтror correction. Relevant sub-pathology, medical and region-of-interest (ROI) text are extracted using an LLM. Finally, domain-specins algorithms are used to Pair images and text, eliminating duplicates to yield QUILT, richly annotated image-text dataset for histopathology.
To address the need for large-scale rision-language dataset in histopathology, we introduce QUILT: containing 437, 878 images aligned with 802, 144 text pairs across multiple microscopic magnifi- cation scales covering from 10x to 40x. We draw on the insight that publicly available educational YouTube histopathology content represents an untapped potential. We curate QUILT using 1, 087 hours of valuable educational histopathology videos from expert pathologists on YouTube. To extract aligned image and text pairs from the videos, we utilize a mixture of models: large language models (GPT-3.5), handcrafted algorithms, human knowledge databases, and automatic speech recognition. QUILT does not overlap with any current open-access histopathology data sources. This allows us to merge our dataset with other open-source datasets available. Therefore, to create even larger and more diverse dataset, we combine QUILT with data from other sources, such as Twitter, research papers, and the Internet, resulting in QUILT- 1M. The larger QUILT- 1M contains million image-text pairs, making it the largest public vision-language histopathology dataset to date.
Using QUILT and QUILT- 1M. we finetune vision-languas models using contrastive objective between the two modalities. We extensively evaluate it on 13 external histopathology datasets taken across different sub-pathologie We report zero-shot classification, linear probe, and image-to-text and text-to- -o-aage retrieval tasks. Against multiple recently proposed baselines (CLIP (46) PLIP (2), and BiomedCLIP (67]), models trained with QUILT- LM outperform all others. Our ablations identify importance of QUILT.
QUILT offers three significant advantages: First, QUILT does not overlap with existing data sources; ensures a unique contribution to the pool of available histopathology knowledge. Second, its rich textual descriptions extracted from experts narrating within educational videos provide more expressive, dense interconnected information. Last, the presence of multiple sentences per image fosters diverse perspectives and a comprehensive understanding of each histopathological image. We hope that both computer scientists and histopathologists will benefit from QUILT'S potential.
Related work
We built upon a growing literature applying self-supervised learning and other machine learning methods to medical image understanding.

---

Machine learning for histopathology. Early representation learning work in computational pathol- ogy primarily relied on supervssp learning. with each whole-slide image (WSI) receiving single label. The limited nature (single label to many patches) has produced sub-optimal models 12 26] at the patch level. Lately, a self-supervised learning approach, which learns useful representa-
tions from unlabeled data, has shown some success (26 13 12 Most of this work has been unimodal.
They use image augmentations similar to those used for natural images [[4], mostly differing by way of consciously injecting domain knowledge. For example, they leverage the compositional nature of H&E stain information of whole-slice images (26), or inject hierarchical morphological information at different magnificattion [33,, or combine with other modalities like genomic features [12] with descriptive text [20]. When text data is used, the objectives similarly use augmentations seen in natural language [50]. By contrast, explore self- self-supervised mechanisms that learn better histopathology information representations that go heyond a single label, aided by language descriptions.
Medical vision-language datasets. Learning vision-language representations demands a large dataset of images aligned with descriptive text, a resource that is notably lacking in histopathology. The MIMIC-CXR CIMIC-CXR-JPG v2.0.0 dataset (28), for example, consists of de-identified hospital-sources chest radiographs and reports. For histopathology. The Cancer Genome Atlas provides de-identified PDF-reports for a limited number of WSIs. Despite this resource, the enormous size of this data (reaching up to 120, 000 pixels) makes processing challenging, limiting its use to a small number of focused studies [39]. A majority of medical rision-langusa datasets concentrated in the radiology sub-domain, due to the relatively straightforward process of collecting validated multimodal data 28
Many models are trained on subset of PubMed 48] or comparable radiology datasets [68] 23 18 43
PMC-15M 67 a recent subset of PubMed not specific to histopathology. was used to train multiple models. While the models themselves are public, PMC-15M is not, making it hard to determine what portion of it is bistopathology--elewee
Vision-language pairs histopathology. One of the first histopathology vision-language datasets, ARCH, contains only 7, 614 accessible image-text pairs (20 22]. Later on, (24] released OpenPath, a dataset of 200K image-text pairs extracted from Twitter. This was the largest histopathology dataset until QUILT-INU 1M.
Video data for self-supervision. Numerous recent studies have started to tap into video data. For instance, millions of publicly accessible YouTube videos were used to train a vision-language model 65 66]. Similarly, causal video model was trained by using sequential gaming videos [6] provide another example of dense, interconnected supervision for single image. Despite the potential of video content, video often yields noisier datasets compared to static sources. Recently, the enhanced capabilities of automatic speech recognition models streamlined the curation of large-scale cleaner datasets from videos [65] 6 67). Furthermore, the growing versatility of large language models has shown promise as data annotators, information text correctors 663 and as tools for medical information extraction and
Localized narratives 55 44
extractors [33 59 15 21),
reasoning 56
3 Curating QUILT: Overview
Creating vision-language dataset from videos is a significant undertaking, as not all videos are suitable for our pipeline. Many either lack voiced audio, are not in English, fail to contain medically relevant content, or have insufficient medical relevance- example, vidcos that present static images of histopathology content on a slide deck, those that briefly cover histopathology images in pursuit of a different objective. Conventional automatic speech recognition (ASR) systems also struggle with the specialized requirements of histopathology transcription, necessitating non-trivial solution. The de-noising of text and image modalities adds further complexity as the videos are typically conversational and, therefore, inherently noisy. Instructors pan and zoom at varying speeds, recording a mix of relevant and irrelevant bistopathological visual content in their videos. As such, trivially extracting frames static intervals fails to capture the data appropriately. To collect QUILT we trained models and handcrafted algorithms that leverage the nuances in the instructors' textual and visual behavior, ensuring accurate collection and alignment of both modalities.
hhaps:/wwww.cnncrr.ovgggg
3

---

lumen and underising INPUT: epithelum and then festures detail
INPUT:" send to see sorn infaer metionn harr you. when you notiow this thene is derde
INPUTT: " of deughter syot where you see so this was an spitheial isl island with showing central
connective
amount of infartmation here yosh you see this
korafinzation the calla producing lst kanain and the same time you have the opihallum with
this lumen tumen has amount keealin, so delinitlly this is kendinous cpel have oyst that is actually producing keralinn lefs move into the opthellum and the epthhllum i carefully
Sity blue dots these are the hymphocylis and
these colls are your rilammmtoory colls and they see kind of hawing an effect on the eprheekum you see speiial wamm becousse of infermmation the opthvilum starts.
limon. so this farms tiny daug' ter cyst and this one the moet importan
why this kemooryst
hos high amount recarmnncn rale, one important festure adantagenic karalocyst upithallum.
observe most of the areas the epthelial appears unioorm
thicneess right from here.
OUTPUT:
OUTPUT:
[MEEItTThe cyst ls produc ng koratin, indicading is
[Ueense
tymphegyyes present region
(MME2E11 presence daughter cysts loeratocysts"
Moratinous cyst.
"Epiteilsi high keratininiion high
"The op thtthllal ining appears have uniform thicknoss]]
Pealiiideeeee intlammatonn here".
production kerstooysia]]
"Sparse amount keradin laman.".
"ymphosytes"
HRDIlll epvinotaal island
uniform dickeess the optholial Eningl)
"nfammmtory ceilss))
"central kerstiiization "daughter eyer")
Figure 2: QUILT examples. Input is the corrected ASR caption for the representative image. Output are the medical and ROI extracted text(s) paired with the image (see Section 3.1). In histopathology. understanding tissue characteristics often involves views from varying magnitication levels. Thus, in QUIIIT we estimate an image's s magnification (indicated by the relative size of the microscope icon).
3.1 QUILT: Collecting medical image and text pairs from YouTube
Our proposed dataset curation pipeline involves (1) gathering channel and video data covering the histopathology domain, (2) filtering videos based certain "narrative style", (3) extracting and denoising image and text modalities from videos using various models, tools, and algorithms, (4) postprocessing denoised text by LLMs extract medical text and finally, (5) splitting and aligning all modalities for curating the final vision-language pre-training (VLP) data. See Figure (and supplemental material) for a detailed overview of the pipeline.
Collecting representative channels and videos. Our pipeline begins by searching for relevant channels and video ids on YouTube, focusing the domain of histopathology. Using keywords spanning 18 sub- sub-pathology fields (see supplement for more details), we search among channels before searching for videos expedite discovery, considering that video searches time-consuming and the APIS pose limitations on numerous requests [65]. Channels with subscriber count 2 300 are excluded to avoid large general science channels, educational histopathology channels often have fewer subscribers. We then download low-resolution versions of all identified videos, with the lowest resolution at 320p.
Filtering for narrative-style medical videos. For each video within each channel, we exclude videos that are shorter than 1 minute, noiced,, have mon-English audio. For videos meeting these heuristics, two decisions are made:
(A) Do they have the required medical content, i.e., histopathology image-text pairs?
(B) If so, are they in narrative style - videos wherein the presenter(s) spend a significant time panning and zooming on the WSI, while providing vocal descriptions of image content?
For (A) we automatically identify the relevant videos by extracting keyframes from a video. These keyframes are automatically extracted using FFmpeg [ (frames containing significant visual changes). The software requires a threshold that determines the minimum amount of visual change required to trigger keyframe. Through experimentation. we set different thresholds for various video durations, with smaller thresholds for longer videos. Next, we train and use ensemble of three histopathology image classifiers to identify videos with histopathology images (See supplement for more details).
marking the beginning or end of scene
For (B), in which we identify narrative-style videos, we randomly select keyframes predicted to be histopathology. For each such selected frame, we extract the next three histopathology key-frames and compute the cosine similarity between the selected frame and each of the subsequent three
'https:/ttimpeg.or/

---

frames. If all three have similarity scores 2 a preset threshold of 0.9, we count it as a narrative streak. A video is identified as narrative style if at least 10% of the selected frames exhibit a narrative streak. Consequently, we download all narrative-style videos at high-resolution. Narrative-style videos typically cover WSIs at various magnifications, hence, we train a iissu---iag-maannffciaee classifier to predict the following three scales: {(1 10)x, (> 10 - 20)x, (> 20 This provides relevant metadata for downstream objectives.
Eyorm Proppt
Cotcact saciuilyy Oatertes IECoERITiAN fre OH SELSNNNE TE0,, SNOSSEEn MY SENDOL Cancraptanca
Frosp::
Dail are
Trree Feoept
Dox Are Aasmilic apensl resouattinns satay motical lest
THE Ventece
tuar Pumpl:
persiotartly the pectagie fhee the apothatic cell of salener, in rentss
prlp
peedtiing in dendggean catta
USE2 Matectar sclass settas the pecsodeen) Juganes space, pecticdaety the
lee- eaaples:
sast cORso Jr redinia on sirndy diacened
sptthaiss macald
deats puly acoeding und the persiccca catis I
Lavelopess
s.ssLs persangioni cass, the
becliwing the datt the puly bcoading soca the
dectacaca cayy vacce D
radia",
soika
(RP0) 130
Jour sos
Okay Ciea, thace as tenurcinge but thyy ceuilly
Jann
chey
gornn
the
DRTINT
OEDE:
Lon
pigneted
Frocakersu (815)) '",
Figure 3: Prompting the LLM by providing few-shot examples to perform the following tasks: (Left) correcting noisy ASR text within its context. We highlight the probable list of misspelled keywords in yellow and their corrections by the LLM in gray. Additional missed errors/misspelled entries identified by the LLM are highlighted in blue. (Right) extracting medical (MED) and ROI text from a given text. We highlight the definition of medical and ROI text in blue and gray respectively.
Text Extraction using ASR and text denoising. The high costs associated with private medical ASR APIs 5 necessitated the use of a more conventional ASR model: Whisper [47). As anticipated, this model often misinterprets medical terms, thus requiring the use of post-processing algorithms to minimize its error rates.
We propose a four-step text de-noising and quality control pipeline: i) We utilize the Rake key- word extraction algorithm to extract keywords or key-phrases up to four words and refine them by eliminating stopwords [49]. ii) We then cross-check each refined entry against UMLS 7 using the SciSpacy entity linking package E.. If an entry is not found within UMLS, we check for misspelled words within the entry using a spell-checking algorithm instantiated with a specialized list of histopathology terms curated from various histopathology ontology labels and definitions. iii) With this probable list of misspelled keywords, we condition and prompt the LLM with examples to correct the misspelled entry within its context (sentence), and secondly, we task the LLM with identifying additional unconditioned errors/misspelled entries. For both, we leverage a set of manually curated examples to prompt the L.MM in-context Figure 3] For more examples and failure cases, see supplement for more details. iv) Finally, to de-noise the text, we resolve the output mapping of incorrect + correct entries by verifying the corrected words against UMLS and our curated list of histopathology words/phrases. Entries that pass this double-validation process are used to replace the initial noisy transcription. Leveraging domain-specific databases to extract the text and filter out noise allows us to bypass the correction of repetition errors and filler words, such as 'ah', ' 'uhm', 'the', etc. in tandem, using LLMs allows us to concentrate correcting medically-relevant misspelled words, rather than correcting non con--ediically-ebeeee relevvant terms.
From the ASR-corrected text, we extract medical text which describes the image(s) as a whole. Also, when the speaker descrines/gestures at visual regions-of-inters we extract the text entity being described as ROI text. To filter relevant medical text and ROI text from the ASR-corrected text, we utilize LLMs (See supplement for more details), a decision rooted in
through statements like "look here
'nuance. www.wwwwww..ttoaiooooooooooooonn htaps:/ggihib.ccommmmm/ss/gggggp
ooneicalllooehmmmm html
5

---

a few compelling reasons: 1) Curating pre- pre-training datasets at scale that can tolerate higher levels of noise, LLMs are more cost-effective than expert-human (medical) labor. 2) The task does not require LLMs to generate new information but instead they discriminate useful versus irreeevnn signals, serving to improve the signal-to-noise ratio of the data. To extract relevant text, we prompt L.LMs to filter out all non-medically relevant text, providing context as necessary. See Figure for some example image-text pairs. Lastly, we instruct the LLMs to refrain from introducing any new words beyond the corrected noisy text and set the model's temperature to zero. Finally, we use LLMs to categorize our videos into one of the 18 identified sub-pathology classes. Similar to the previous tasks, this categorization is done by conditioning with few examples and prompting the LLM to predict the top three possible classes given the text. More details, prompts, and additional examples are presented in the supplemental material.
Image frame extraction and denoising. For each video, we employ a similar method to that described in Filtering for narrative-style medical videos subsection to extract histopathology key-frames; our method leverages these frames' times as beacons to break the entire video into time-intervals called chunks from which to extract representative image(s). Next, we extract the median image (pixel-space) of stable (static) frames in each chunk if they exists, else we de-duplicate the histopathology keyframes (beacons of the chunk). In essence, we use the extracted histopathology scene frames as guides for data collection, exploiting the human tendency in educational videos to pause narration during explanation, and we extract the relevant frame(s).
Aligning both modalities. For each narrative-style video, we perform the following steps to align image and text modalities: First, we compute histopathology time chunks denoted as ((1,,))) (43, t4), ... (1,-1, (n) from keyframes after discriminating histopathology frames using the histopathology ensemble classifier- - (scene_ ccthess)) Each scene_ crene_chunk is padded with pad _time to its left and right; see supplement for more details.
1. Text: we use the ASR output extract the words spoken during each chunk in scene chunks. Using the method described in Text Extraction using ASR and text denoising subsection, we extract the Medical and ROI caption for this chunk.
2. Image: we extract representative image(s) for every chunk/time-ing inter-llt in scene scene_chunks as described in Filtering for narrative-style medical videos subsection above.
Finally, each chunk in scene scene_chunks is mapped to texts (both medical and ROI captions) and images. Next we map each medical image one or more medical text. Using the time interval in which the image occurs, we extract its raw text from ASR and then correct and extract keywords using the Rake method, which we refer as raw vaw_keywonds. We extract keywords from each medical text returned using the LLM,, and we refer to these as keywords. Finally, if the raw_key mw__eywords occur before or slightly after a selected representative image, and overlap with the keywords in one of the Medical/ROI texts for that chunk, we map the image to the medical/ROP text. Example. keywords: psammoma bodies, match with raw weywooyord: psammoma bodies within the ASR-corrected text Meningiomas typically have meningothelial pattern with lobular-lik arrangements and psammoma bodies.' Refer Figures in the supplement for detailed explanation of the method and examples of aligned image and text.
3.2 QUILT-IM: Combining QUILT with other histopathology data sources
To create QUILT-IM we expanded QUILT by adding other disparate histopathology image-text open-access sources: LAION, Twitter, and PubMed.
PubMed Open Access Articles. We searched the PubMed open-access from 2010-2022, extracting 59,371 histopathology image-text pairs, using our histopathology classifier and multi-plane figure cropping algorithm. The images are categorized into (I) images that are fully histopathology. (2) multi-plane images that contain histopathology sub-figures, and (3) histopathology sub-figures cropped from (1) and (2). See supplement for more details.
Histopathology Image Retrieval from LAION. The Large-scale Artificial Intelligence Open Net- work (LAION-5B) (22) curated over 5 billion pairs of images and text from across the Internet, including a substantial volume of histopathology-relace
data. We tapped into this resource by retrieving 22,682 image and text pairs. See supplement for more details.

---

Twitter Data from OpenPath. We utilized a list of tweets curated by Huang et al. [24]. which totaled up to 55,000 unique tweets and made up 133, 511 unique image-text pairs. This exhibits a one-to- many relationship where many images were matched with multiple captions; this differentiated our work from the OpenPath approach. To maintain comparability, we followed their text pre-procentt pre-processing pipeline 24 See supplement for more details.
Sub Potto sgy Categer es Over All Videos
Top 20 Samurd: Tpp $1.2%
Sacial
D
inf
Ieanatationn change
case
astructure
Img-Text Pairs over Sub-Pathology
SiO THE show....
e
benign
observed
160000
seen
COLL POLLigenn
Lej
140000
Descri
vessel
1
t
type
smal
Inchiaiin
blood
De
iff
dermis
paaer
gland
rappearance ance
60009
dLscase
40300
susee
cytoplase
intiitrate
P
resent
20003
Saarrttor
diagnos
ticken
Herri Pattering
Reuropaltio
Heall lan Caddar Ginternee
or Ceeeiiee Saste Ceterii
Larg
may
Sppeer
nuclei carcinomaaatten
a
e
(c)
(d)
Figure 4: (a) Distribution of all videos over sub-pathology types. (b) across top 20 UMLS semantic types. (c) Number of image-text pairs within each potpphthhlggy type. (d) word cloud of all the text QUILT.
Distribution of our entities
3.3 Quality
To evaluate our pipeline's performance, we assess several aspects. First, we calculate the precision of our LLM's corrections by dividing the number of conditioned misspelled errors replaced (i.e., passed the UMLS check) by the total number of conditioned misspelled words found, yielding an average of 57.9%. We also determined the unconditioned precision of the LLM, similar to the previous step, and found it be 13.8% Therefore, we replace our detected incorrect words with the LLM's s correction 57.9% of the time, and 13.8% of the time we replace the LLM's 's detected errors with its correction (see supplement for more details). To estimate the ASR model's transcription performance, we compute the total number of errors replaced (both conditioned and unconditioned) and divide it by the total number of words in each video, resulting in an average ASR error rate of 0.79%. To assess the LLM's sub- nab-pathology classification, we manually annotated top-k (k 1, 3,) pub-pathology types for 100 random videos from our dataset. The LLM's accuracy for top-3, top-2, and top-1 was 94.9%, 91. 1.9% and 86.8% respectively. Also note that, by prompting the LLM to extract only medically relevant text, we further eliminate identifiable information, such as clinic addresses, from our dataset.
3.4 Final dataset statistics
We collected QUILT, from 4475 narrative videos spanning over 1087 hours with over 437K unique images with 802K associated text pairs. The mean length of the text captions 22.76 words, and 8.68 words for ROI text, with an average of 1.74 medical sentences per image (max=5.33, min=1.0). Our dataset spans total of 1.469M UMLS entities from those mentioned in the text (with .8.5K unique). The images span varying microscopic magnification scales (0-10x, 10-20x, 20-40x), obtaining (280K,
7

---

75K. 107K) images from each scale respectively with an average height and width of 882 x 1468 pixels, as we leverage the max image resolution of videos. Figure (a, c) plots our dataset's diversity across multiple histopathology sub-domains. This plot shows that the captions cover histopathology- relevant medical subtypes: findings, concepts, organs, neoplastic processes, cells, diseases, and a mix of laboratory and diagnostic procedures. Overall, across all 127 UMLS semantic types, our entities cover 76.2% of medically-related semantic types (e.g., findings, disease, or syndrome) and 23.75% non-medical (e.g., geographic area, governmental regulatory activity).
QUILTNET: Experiments training with QUILT-IM
Clip
Biomedklip
Plip
QuittNet
72
62
37
10
30
25
12
SunCercer Pasc ParceCarelyonnn
VOKORC
MI1ST
UC25003Lung
LC3t00oCalón
BAC1
SICAN22
Dacabies
RenalCal
Deteo
Sh nTumar
Figure 5:
QUILTNET, outperforms oft-of-domain CLIP baseline and state-of of-t-e--rrhrrr histopathology models across 12 zero-shot tasks, covering 8 different sub-pathologies (accuracy percentage provided).
We use the Contrastive Language-Image Pre-training (CLIP) objective 46 to pretrain QUILTNET using QUILT- 1M. CLIP takes a batch of N (image, text) pairs and optimizes contrastive objective to create joint embedding space. The optimization process involves concurrent training of both image and text encoders to increase the cosine similarity of embeddings from aligned pairs, while decreasing it for unaligned pairs. The objective is minimized via the InfoNCE loss, expressed as:
1 C -= 2N
Σ log
log
coa(II.TT.)
N Dj=1 (I,n(
N Dj=1 ,cos(L,
where 4, and Tj are the embeddings for the aligned i-th image and text, respectively. For the image encoder, we use both ViT-B/32 and ViT-B/16 architectures [16). For the text encoder, we use GPT- 2 [45] with a context length of 77, and PubmedBert [67). We train QUILTNET by finetuning an OpenAl pre-trae trained CLIP model [46] on QUILT- 1M to enhance its performance in histopathology. Once finetuned, we conduct experiments on two types of downstream tasks: image classification (zero- shot and linear probing) and cross cross-modal retrieval (zero-shot). We also compare the performance of fine-tuning a pre-trained CLIP model versus training it from scratch.
Downstream histopathology datasets. We evaluate the utility of QUIL.NNET on 13 downstream datasets: PatchCamelyon (57] contains histopathologgy scans of lymph node sections labeled for metastatic tissue presence binary label. NCT-CRC-HE-IOUPE (31] consists of colorectal cancer images and is categorized into cancer and normal tissue. For SICAPv2 [53] the images are labeled as non-cancerous, Grade 3-5. Databiox [8] consists of invasive ductal carcinoma cases of Grades I-III. BACH 4 consists of breast tissues labeled as normal, benign, in-situ, and invasive carcinoma. Osteo [5] is set of tissue patches representing the heterogeneity of osteosarcoma. RenalCell [10] contains tissue images of clear-cell renal cell carcinoma annotated into five tissue texture types. SkinCancer (34 consists of tissue patches from skin biopsies of 12 anatomical compartments and 4 neoplasms that make up the Skin Tkior Subset. MHIST (60) contains tissue patches from Formalin-Fixe Fixed Frmeeddd-meeedddd WSIs of colorectal polyps. I.C20000 (. which we divide into LC25000 (Lung) and LC25000 (Colon), contains tissue of lung and colon adenocarcinomas. For more details see supplemental material.
Results using zero-shot learning. Given the vast diversity of cancer sub-types histopathology. it is critical that model maintains comprehensive understanding without requiring specific data
8

---

Table 1: Linear probing- Classification results, denoted as accuracy % (standard deviation). Came- lyon denotes the PatchCamelyon dataset. Supervised results are from each dataset's s SOTA models.
ViT-B/32
ViT-B/16
Datasct
%shoot
CLIP
PLIP
QUILTNET
CLIP
QUILTNET BiomedCLIP QUILTNET
Supervised(%)
GPT/77
GPT/77
GPT/77
GPT/77
GPT/77
PMB/256 PMB/256
91.0 (0.10)93.75 (0.09) 94.64 90.96 (0.10)93.36 (0.23) 92. 14 (0.12) 93.55 (0.25)
NCT-CRC 31
10
92.02 (1.30)93.83 (0.06)95.30 (0.03) (0. (0.12)93.85 (0.04) 92.90 (0.07) 93.72 (0.08)
(94.0)
100
91.83 (0.01)94.16 (0.01)95.22 (0.0))9
(0.09)93.76 (0.02) 92.97 (0.05) 93.60 (0.01)
80.38 (0.16)87.26 (0.23)87.62 (0.35) (0.20)84.78 (0.14) 83.63 (0.44) 83.48 (0.18)
Camelyon 57
82.67 (0.19)87.48 (0.08)87.55 (0.03)82.20 (0.04)86.77 (0.09) 84.18 (0.15) 84.42 (0.10)
(97.5)
100 82.80 (0.01)87.34 (0.01)887.48 (0.01) (0.02)86.81 (0.04) 84.23 (0.01) 84.44 (0.02)
84.27 (0.22)91007 (0.25)90.93 (0.25) (0.16)88.29 (0.15) 87.53 (0.21) 88.06 (0.20)
SkinCancer 34
89.0 (0.02)93.39 (0.05)92.99 (0.02)90.28 (0.01)91.20 (0.0) 89.23 (0.03) 90.03 (0.02)
(93.3)
100 89.02 (0.02)93.29 (0.01)93.03 (0.02)90.29 (0.03)91.20 (0.0) 89.16 (0.02) 89.91 (0.01)
52.45 (2.41)65.76 (2.65)69.92 (1.02) 56.01 (0.66)66.86 86 (6.16) 69.43 (1.03) 68.49 (1.06)
SICAPv2 [53
10 62.24 (0.65)69.23 (0.43)74.14 (0.38) 63.70 (0.69)72.37 (0.65) 71.61 (0.31) 72.48 (0.42)
(67.0)
100 65.75 (0.16)73.0 (0.14)75.48 (0.104)74 (0.16) 74.57 (0.04) 74.60 (0.17)
for retraining. Thus, we evaluate our model's zero-shot performance against three state-of-the-ar models: CLIP, BiomedCLIP, and PLIP. Our model demonstrates superior performance, illustrated in Figure 5 where it outperforms the other models in all but two datasets, in which BiomedCLIP performs marginally better. See supplemental material for UMap visualizations and modal attention visualization comparison. The prompts used for these evaluations are presented in the supplemental material. To ensure fair comparison with BiomedCLIP, which uses a VYT-B/16 and PMB/256 (pre-trained with (67), we trained three different variants of our model. For detailed insights into the results, please refer to supplemental material.
Results using linear probing. We assess the few-shot and full-shot performance of our model by conducting linear probing with 1%, 10%, and 100% of the training data, sampled with three different seeds; we report the average accuracy and their standard deviation in Table [ We deploy our evaluation across four distinct datasets, specifically those with dedicated training and testing sets among our external datasets. Remarkably, our model, utilizing the ViT-B/32 architecture with GPT/77, outperforms its counterparts, BiomedCLIP, PLIP, and CLIP, in most datasets. On the NCT-CRC and SICAPv2 datasets, our model surpasses even the fully supervised performance using only 1% of the labels. Also, note that for some results 10% does better than 100%; this because we are sampling from each class equally, and thus the 10% subset contains more balanced training set than 100%, for datasets that are very imbalanced, resulting sub-optimal performance at 100%.
Results using cross-modal retrieval. In our study, we evaluate cross-modal retrieval efficacy by examining both zero-shot texl-to-image and image-to-text retrieval capabilities. We accomplish this by identifying the nearest neighbors for each modality and then determining whether the corresponding pair is within the top N nearest neighbors, where E {1, 50, 200). Our experiments are conducted on two datasets: our holdout dataset from QUILT- 1M and the ARCH dataset. Results are in Table 2
5 Discussion
Limitations. Despite the promising results, QUILT was curated using several handcrafted algorithms and LLMs. Such curation methods, while effective, introduce their own biases and errors. For instance, our histopathology classifier had occasional false positives (== 5%) confirmed by human evaluation. Occasionally, ASR can misinterpret a medical term and transcribe it as different existing term, such as transcribing 'serous carcinoma" as 'serious carcinoma'. Unfortunately, such errors are not rectifiable using our current pipeline (see supplement for more details). While not directly a limitation of our dataset, training CLIP model trained from scratch underperformed compared

---

Table Cross- modal retrieval results on the QUILT- 1M holdout set and ARCH dataset. each cell, the results are displayed in the format (%/%), with QUILT- results on the right. The best-performing results are highlighted in bold text.
IM 1M holdout results on the left and ARCH
Text-to-lmage (%)
Image-to- -oxx (%)
model
config
R@1
@@50
R@200
R@D 1
R@50
R@200
CLIP ViT-B/32IGPI/77
0.49/0.07
4.73/2.42
10.15/7.21
0.39/0.05
3.99/2.52
8.80/7.22
PLIP VIT-B/32IGPT/77
1.05/0.56
10.79/13.10
21.80/29.85
0.87/0.74
11.04/13.75
21 21.63/29.46
QUILTNET ViT-B/32IGPT/77
1.17/1.41
16.31/19.87
31.99/39.13
1.24/1.35
14.89/19.20
28.97/38.57
CLIP ViT-B/16IGPT/77
0.83/0.09
5.63/2.73
11 11.26/8.72
0.66/0.13
5.02/3.09
10.82/9.00 04
QUILTNET ViT-B/161GPT/77
2.42/1.29
22.38/20.30
41.05/40.89
2.00/1.01
21.66/16.18
39. 39.29/34.15 15
BiomedCLIP VIT-B/16G224MMMBBB256
4.34/8.89 14.99/53.24 24 25.62/71.43
3.88/9.97
13.93/52.13
23.53/68.47 47
QUILTNET VIT-B/16G224PMMB/255
6.20/8.77 30.28/55.14 50.60/77.64
6.27/9.85 31.06/33.06 50.86/73.43
to fine-tuning a pre-trained CLIP (see supplement for more details). This suggests that a million image-text pairs may still not be sufficient. Future works may explore other self-supervised objectives.
Data Collection and Societal Biases Aligning in strategies with 65 we release QUILT derived from public videos, taking structured steps to limit privacy and consent harms (see supplement for more details). Complying with YouTuube's privacy policy, we only provide video IDs, allowing users opt-out of our dataset. Researchers can employ our pipeline to create QUIIT. Regarding societal biases, significant portion of our narrators originate from western institutions, a situation that is further amplified by our focus on English-only videos. Consequently, QUILTNET may exhibit inherent biases, potentially performing better on data associated with these demographies, while possibly underperforming when applied to other cultural linguistic groups.
Conclusion. We introduced QUILT- 1M, the largest open-soureed histopathology dataset to date. Empirical results validate that pre-training using QUILT is valuable, outperforming larger state-of- the-art models like BiomedCLIP across various sub-pathology types and tasks including zero-shov few-shot, full-shot, and cross-modal retrieval. We established new state-of-the-an in zero-shot, linear probing, and cross-modas retrieval tasks in the field of Histopathology.
Acknowledgments
Research reported in this study supported by the National Cancer Institute under Awards No. R01 CA15130, R01 CA225585, and RO1 CA201376 and the Office the Assistant Secretary of Defense for Health Affairs through the Melanoma Research Program under Awards No. W8IXWH-20-1-0799 and W81 WW8-00220-10079. Opinions, conclusions, and recommendations are those of the authors.
References
[1] M. Agrawal, Hegselmann, H. Lang, Y. Kim, and D. Sontag. Large language models are zero-shot clinical information extractors. arXiv preprint arXiv:2205. /2689, 2022.
[2] M. Amith, L. Cui, K. Roberts, H. Xu, and C. Tao. Ontology of consumer health vocabulary: providing formal and interoperable semantic resource for linking lay language and medical terminology. In 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 1177-1178. IEEE, 2019.
[3] A. Araujo, J. Chaves, H. Lakshman. R. Angst, and B. Girod. Large-scale guery-by-imaan video retrieval using bloom filters. arXiv preprint arXiv: 1604.06040000 07939, 2016.
[4] G. Aresta, T. Araújo, S. Kwok, S. Chennamsetty, M. Safwan, V. Alex, B. Marami, M. Prastawa, M. Chan, M. Donovan, et al. Bach: Grand challenge on breast cancer histology images. Medical image analysis, 1661222139 2019.
[5] H. B. Arunachalam, R. Mishra, O. Daeseu, K. Cederberg, D. Rakheja, A. Sengupta, D. Leonard, R. Hallac, and P. Leavey. Viable and necrotic tumor assessment from whole slide images of
10

---

osteosarcoma using machine-learning and deep-learning models. PloS one, 14(4):e0210706, 2019.
[6] B. Baker, I. Akkaya, P. Zhokov, J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639- 24654,, 2022.
[7] O. Bodenreider. The unified medical language system (umls): integrating biomedical terminol- ogy. Nucleic Acids Res., 22sDaatbbase--sssue)2677--222222 2004. URL httt:////bblpp uni-trier
waaaaaallaaaaaraaaaa hawpuaaaaaaaaaaaaaa3.3.... http:////naa33.tttt...oeoeeeeeee htm1#BodenreiderOf4
[8] H. Bolhasani, E. Amjudi, M. Tabatabueian, and S. J. Jassbi. A histopathological image dataset for grading breast invasive ductal carcinomas. Informatics in Medicine Unlocked, 19:10034,, 2020.
[9] A. A. Borkowski, M. M. Bui, L. B. Thomas, C. P. Wilson, L. A. DeLand, and S. M. Mas- torides. Lung and colon cancer histopathological image dataset (lc25000). arXiv preprint arXiv: 1912. /212, 2019.
[10] O. Brummer, P. Polonen, S. Mustjoki, and O. Bruck. Integrative analysis of histological textures and lymphocyte infiltration renal cell carcinoma using deep learning. bioRxiv, pages 2022-08. 2022.
[11] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, Bojanowski, and A. Joulin. Emerging properties in self sufe-supe vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages (650-9660) 2021.
[12] R. J. Chen, M. Y. Lu, W.-H. Weng, T. Y. Chen, D. F. Williamson, T. Manz, M. Shady, and Mahmood. Multimodal co-attention transformer for survival prediction gigapixel whole slide images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015-4025, 2021.
[13] R. J. Chen, C. Chen, Li, T. Y. Chen, A. D. Trister, R. G. Krishnan, and F. Mahmood. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 16144-16155, 2022.
[14] T. Chen, S. Korblithh, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[15] B. Ding, C. Qin, L. Liu, L. Bing, Joty, and B. Li. Is gpt-3 a good data annotator? arXiv preprint arXiv:2212. 10450, 2022.
[16] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 1 16x 16 words: Transformers for image recognition at scale. urXiv preprint arXiv:2010. 1929, 2020.
[17] D. Doukhan, J. Carrive, F. Vallet, A. Larcher, and S. Meignier. An open-source speaker gender detection framework for monitoring gender equality. In Acoustics Speech and Signal Processing (ICASSP), 2018 IEFF International Conference on. IEEF, 2018
[18] S. Eslami, G. de Melo, and C. Meinel. Does clip benefit visual question answering in the medical domain much as it does in the general domain? arXiv preprint arXiv:2112.13906, 2021.
[19] G. Fragoso, S. de Coronado, M. Haber, F. Hartel, and L. Wright. Overview and utilization of the nei thesaurus. Comparative and functional genomics, 5(8):648-6) -654, 2004.
[20] J. Gamper and N. Rajpoot. Multiple instance captioning: Learning representations from histopathology textbooks and articles. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 16549-16559, 2021.

---

[21] F. Gilardi, M. Alizadeh, and M. Kubli. Chatgpt outperforms crowd-workers for text-annotation tasks, arXiv preprint arXiv:2303. 15056, 2023.
[22] X. He, Y. Zhang, L. Mou, E. Xing, and P. Xie. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003. 10286, 2020.
[23] S.-C. Huang, L. Shen, M. P. Lungren, and S. Yeung. Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3942-3951, 2021.
[24] Z. Huang, F. Bianchi, M. Yuksekgonul, T. Montine, and J. Zou. Leveraging medical twitter to build visual-language foundation model for pathology ai. bioRxiv, pages 2023-03, 2023.
[25] E. Hussain, L. B. Mahanta, H. Borah, and C. R. Das. Liquid based-cytology pap smear dataset for automated multi-class diagnosis of pre-cancerous and cervical cancer lesions. Data in brief, 30:105589, 2020.
[26] W. O. Ikezogwo, M. S. Seyfioglu, and L. Shapiro. Multi-modal masked autoencoders learn compositional histopathological representations. arXiv preprint arXiv:2209.01334 2022.
[27] K. Jobin, A. Mondal, and C. Jawahar. Docfigure: A dataset for scientific document figure classification. In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), volume I, pages 74-79. IEEE, 2019.
[28] A. Johnson, T. J. Pollard, N. R. Greenbaum, M. P. Lungren, C.-y. Deng, Y. Peng. Z. Lu, G. Mark, S. J. Berkowitz, and S. Horng. Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs. arXiv preprint arXiv: 2019.
[29] S. Jupp, J. Malone, T. Burdett, J.-K. Heriche, E. Williams, J. Ellenberg, H. Parkinson, and G. Rustici. The cellular microscopy phenotype ontology. Journal of biomedical semantics, 1-8, 2016.
[30] Z. Karishma. Scientific document figure extraction, clustering and classification. 2021.
[31] N. Kather, N. Halama, and Marx. 100,000 histological images human colorectal cancer and healthy tissue. Zemodo 10, 5281, 2018.
[32] A. Kembhavi, M. Salvato, E. Kolve, M. Seo, Hajishirzi, and A. Farhadi. A diagram is worth a dozen images. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 235-251. Springer, 2016.
[33] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205. 1916, 2022.
[34] K. Kriegsmann, Lobers, C. Zgorzelski, J. Kriegsmann, C. Janßen, R. R. MeliB, Muley, U. Sack, G. Steinbuss, and M. Kriegsmann. Deep learning for the detection of anatomical tissue structures and neoplasms of the skin on scanned histopathological tissue sections. Frontiers in Oncology, 12, 2022.
[35] Liu, Q. Wang. H. Fan, Wang. W. Li, Y. Tang, D. Wang, M. Zhou, and L. Chen. Automatic label correction for the accurate edge detection of overlapping cervical cells. arXiv preprint arXiv:2010.01919 2020.
[36]- S. Liu, C. Zhu, Xu, X. Jia, Z. Shi, and M. Jin. Bei: Breast cancer immunotistochemical image generation through pyramid pix2pix. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1815-1824, 2022.
[37] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[38] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. A convnet for the 2020s. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 11976-11986, 2022.
12

---

[39] N. Marini, S. Marchesin, S. Otálora, M. Wodzinski, A. Caputo, M. Van Rijthoven, W. As- wolinskiy, J.-M. Bokhorst, D. Podareanu, Petters, al. Unleashing the potential of digital pathology data by training computer-aided diagnosis models without human annotations. NPJ digital medicine, 5(1):102, 2022.
[40] D. Morris, E. Budack, and R. Ewerth. Slideimages: dataset for educational image classification. In Advances in Information Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14-17, 2020, Proceedings, Part II 42, pages 289-296. Springer, 2020.
[41] M. Neumann, D. King. I. Beltagy, and W. Ammar. ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 319-327, Florence, Italy, Aug. 2019. Association for Computational Linguistics. doi: 11111663/1/111150000 URL htttps:/wwxx www.:w.wlwebbbb.m org/ anthology/ W19-5034
[42] N. F. Noy, M. A. Musen, J. L. Mejino Jr, and C. Rosse. Pushing the envelope: challenges in a Trame-based representation of human anatomy. Data & Knowledge Engineering, 48(3): 335-359, 2004.
[43] O. Pelka, S. Koitka, J. Rückert, F. Nensa, and C. M. Friedrich. Radiology objects in context (roco): multimodal image dataset. Intravascular Imaging and Computer Assisted Stenting and Large-Scale Anmotation of Biomedical Data and Expert Label Synthesis: 7th Joint Interna- tional Workshop. CVII-STENT 2018 and Third International Workshop, LABELS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings 3, pages 180-189. Springer, 2018.
[44] J. Pont-Tuset, J. Uijlings, S. Changpinyo, R. Soricut, and V. Ferrari. Connecting vision and language with localized narratives. In ECCV, 2020.
[45] A. Radford, J. Wu, Child, D. Luan, D. Amodei, I. Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[46] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages
PMLR, 2021.
[47] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:22/2.043550 2022.
[48] R. J. Roberts. Pubmed central: The genbank of the published literature, 2001.
[49] S. Rose, D. Engel, N. Cramer, and W. Cowley. Automatic keyword extraction from individual documents. Text mining: applications and theory, pages 1-20, 2010.
[50] Santos, A. Tariq, S. Das, K. Vayalpati, G. H. Smith, Trivedi, and I. Banerjee. Pathologybert- pre-trained vs. a new transformer language model for pathology domain. arXiv preprint arXiv:2205.06885, 2022.
[51] P. N. Schofield, J. P. Sundberg, B. A. Sundberg, C. McKerlie, and G. V. Gkoutos. The mouse pathology ontology, mpath; structure and applications. Journal of biomedical semantics, 4(1): 1-8, 2013.
[52] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprini arXiv:22 10. 08402, 2022.
[53] Silva- Silva- Rodríguzz, A. Colomer, M. A. Sales, R. Molina, and Naranjo. Going deeper through the gleason scoring scale: An automatic end-to-end system for histology prostate grading and cribriform pattern detection. Computer methods and programs biomedicine, 195:105637 2020.
13

---

[54] A. Singh, V. Natarajan, M. Shah. Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317-8322 2019.
[55] H. Singh and M. L. Graber. Improving diagnosis in health care-the the next imperative for patient safery. The New England journal of medicine, 373(2)):24332299 2015.
[56] K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani,
H. Cole-Lewis, S. Pfohl, et al. Large language models encode clinical knowledge. arXiv
preprint arXiv:2212.13138, 2022.
[57] B. S. Veeling, J. Linmans, J. Winkens, T. Cohen, and M. Welling, Rotation equivariant cnns for digital pathology. In Medical Image Computing and Computer Assisted Intervention-MICCAl 2018: s1 International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11, pages 210-218. Springer, 2018.
[58] P. Voigtlaender, S. Changpinyo, J. Pont-Tuset, R. Soricut, and V. Ferrari. Connecting Vision and Language with Video Localized Narratives. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[59] S. Wang, Y. Liu, Y. Xu, C. Zhu, and M. Zeng. Want to reduce labeling cost? gpt-3 can help. arXiv preprint arXiv:2 108. / 13487, 2021.
[60] J. Wei, A. Suriawinata, B. Ren, X. Liu, M. Lisovsky, L. Vaickus, C. Brown, M. Baker, N. Tomita, L. Torresani, et al. A petri dish for histopathology image analysis. In Artificial Intelligence in Medicine: 19th International Conference on Artificial Intelligence in Medicine, AIME 2021, Virtual Event, June 15-18, 2021, Proceedings, pages 1 11-24. Springer, 2021.
[61] P. Weitz, M. Valkonen, L. Solorzano, C. Carr, K. Kartasalo, C. Boissin, S. Koivukoski, A. Kuusela, D. Rasic, Y. Feng, et al. Acrobat-a multi-stain multbst-amlt--staiin breast cancer histological whole- slide-image data set from routine diagnostics for computational pathology. arXiv preprint arXiv:2211.1362) 2022.
[62] P. S. Wright, K. A. Briggs, R. Thomas, G. F. Smith, G. Maglennon, P. Mikulskis, M. Chapman, N. Greene, B. U. Phillips. and A. Bender, Statistical analysis of preclinical inter-species concordance of histopathological findings in the etox database. Regulatory Toxicology and Pharmacology, 138:105308 2023.
[63] H. Wu, W. Wang. Y. Wan, W. Jiao, and M. Lyu. Chatgpt or grammarly? evaluating chatgpt on grammatical error correction benchmark. arXiv preprint arXiv:2303. 13648, 2023.
[64] W. Wu, S. Mehta, S. Nofallah, Knezevich, J. May, O. H. Chang, G. Elmore, and L. G. Shapiro. Scale-aware transformers for diagnosing melanocytic lesions. IEEE Access, 9; 163526-163541, 2021.
[65] R. Zellers, X. Lu, J. Hessel, Y. Yu, J. S. Park, J. Cao, A. Farhadi, and Y. Choi. Merlot: Multimodal neural script knowledge models. Advances in Neural Information Processing Systems, 34:23634-236511 2021.
[66] R. Zellers, J. Lu, X. Lu, Y. Yu, Y. Zhao, M. Salehi, A. Kusupati, J. Hessel, A. Farhadi, and Y. Choi. Merlot reserve: Neural script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16375-16387, 2022.
[67] Zhang, Y. Xu, N. Usuyama, Bagga, R. Tinn, S. Preston, R. Rao, M. Wei, N. Valluri, C. Wong. et al. Large-scale Nooainn-specing pretraining for biomedical vision-language processing. arXiv preprint arXiv:2303. 00915, 2023.
[68] Y. Zhang, H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference, pages 2-25. PMLR, 2022.
[69] B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba, Places: A 10 million image database for scene recognition. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2017.
14

---

A Data curation models, algorithms and parsing pipelines
A.1 Curating QUILT: Overview
Creating a densely annotated vision-language dataset from videos is a significant undertaking, as involves various handcrafted algorithms and machine learning models. In the following sections, present more detailed information about the challenges of the data curation pipeline and algorithms used address these challenges. To download QUILT-IM and its metadata and access the code to recreate the dataset and trained models, refer to our website
Collecting representative channels and videos. The first challenge lies in obtaining relevant histopathology videos. We used a set of keywords (obtained from online histopathology glossaries F to search for videos, resulting in 65K potential matches. 6 the word cloud of all keywords used for searching YouTube. However, filtering histopathology content based on thumbnail and title yields many false positives, often including general pathology videos. To address this, process the frames of lower-resolution versions of each video to differentiate between histopathology and pathology content, narrowing the selection to R 9K videos.
Pancreatic
yndrrome
Large
mar gina hhhma
PROSTACA Acute
Eriettive
uutuuuus
v
Aorreeae aud Lan Jaa
adenona Foll
Squamous
tcular
tumo
glioma
cu
carch
sal
sstrocyton
coma
Iung
oma
Non
B
Markel
eyeoold
Hodgkin
carc
farciooid
Invasive
leukemiar
Dirnnd
apr
Small Teaur
the
S
T zone
porten
Fungoudes
Instrountestinaa
Figure 6: Word cloud of all keywords used for searching YouTube
Filtering for narrative-style medical videos. Among the z 9K videos, we sought videos with a "narrative style" where narrators freely explain whole slide images and streaks of similar frames occur, indicating an educational performance. To identify such content, we used a model that analyzed randomly sampled frames to determine if they maintained consistent style over time. This process resulted in the selection of ss 4K videos. Non-voiced videos are also filtered by using inaSpeechSegmenter (17) where the video endpoint does not provide the video language or transcript. To identify the audio language of a video, we first check You Tubesss's API. If the information is unavailable through the API, we use OpenAl's Whisper model 47 the first minute of audio from
the video.
To identify videos containing medical content, employ a keyframe extraction process with specific threshold to determine the minimum visual change required to trigger keyframes. For a new video, the thresholds for keyframe extraction are determined by linearly interpolating between the lowest threshold, 0.008 (5-minute video) and the highest 0.25 (200-minute video). Following the keyframe extraction process, we utilize histopathology image classifier identify histopathology content within the extracted keyframes. See S.e/A.3 for more details. To identify narrative-style videos, we randomly select min(num @fuum_o_rrr ofsssttt___a___eeeeeeeeeoee hht_hhttttct______e_ee___e_eee__ _ceen_ffaaess, 20) keyframes from video and utilize pre-trained CLIP & (ViT-B-32) model to embed and compute a cosine similarity on the next three keyframes. If all three have similarity scores 2 threshold of 0.9, we count the video as narrative streak.
Text extraction using ASR and text denoising. Another challenge involves automatic speech recognition (ASR), as YouTube captions are often inadequate for medical vocabulary. To address this issue, employed the Large- V2 open-source Whisper model [47] for speech-to-text conversion. However, gew.gale ASR models like Whisper can misinterpret medical terms, particularly
htw.://iieeeeeeoeeeeeeeeeeeeeene ViT-B-32
15

---

Table 3: Salvagable and Non-salvagable cases for ASR correction using an LLM.
Error due to
Raw output
Sahagable
Non-solvagable
(beacnuse LLM
losses all possible medical context and can lead wrong entries)
(because
can rephrase and/or
extract contextually similar correction)
Unlinetuned
expect HP. They actually look more sarzoidal to me. The reason I say that is they, there's a kind of positive of inflammatory cells associated with them. They really light und well formed. They's re very easy to see low power. And HP in the differential hypersensium nitose, but would be more worried about sarcoidosis.
.o.ook cramialocias
hypersen- stum nitose: ltype--
positive: paucity
sensitivity pacurnoni- lis, cranialomas: granu- lomas
LLM
high- high-larbidin-sinal lymphadenocarhy
returns lymphatic partem dis- tribution instead of a more instead of appropriate lymphangitic pat- tern distribution
returss hilar lym- phadenopathy
bymphin-giatie paltern distribution
appropriate bilar mediastinal lym- phudenoputhy
Incomplete checker
UMLS
...picnotic
LLM correctly returs py- knotic bowever, UML (/200(200) does have the word py- knotie if fails to pass the check.
when the speaker's voice is choppy accented. There straightforward trivial solutions due to: 1) the absence of openly available medical ASR models or data for fine-tuning in the medical domain; 2) the inadequacy of medical named entity recognition models in detecting transcription errors, because these models are typically trained on correctly spelled words; 3) the ineffectiveness of methods like semantically searching over medical glossary, such as UMLS. which only prove effective when the erroneous text has significant similarity to the correct terms; and 4) the inability of simpler methods like finding the longest common substring, which might work in finding a match in the glossary/ontology for replacement, but cannot identify the wrong words/phrased in the first place. To rectify ASR errors, we employed UMLS knowledge database) and LLM (GPT-3.5). This, however, introduces a new challenge of identifying incorrectly transcribed words and determining which words were mistakenly "corrected" and correctly formatted by the LLM after error correction and resolving unintended parsing errors [1. See Figure 3 in the main paper for LLM prompt examples of ASR correction and medical and ROI text extraction from the corrected ASR text. Refer to Table for error examples of ASR correction using the L.M.M
Image frame extraction and denoising. The image processing aspect of this task adds its com- plexity, as it requires static frame detection, quality control for frames, and histology magnification classification. Each model utilized it these steps introduces its own biases and errors. We extract time-intervals (chunks) from each video from which we extract representative image(s). For each of the extracted chunks (tn, fn+1), the static chunk detection algorithm time-interval. with static frames within the chunk. If found, we save the median (in pixel space to prevent blurry outputs) of the stable frames, else (i.e no stable duration of frames) we leverage the structural similarity index (SSIM) method on histopathology key-frames to find the most dissimilar histopathology image to make up the representative images for the chunk, essentially de-duplicating the frames. Figure demonstrates this process.
used to extract sub-
16

---

Stable frames
Stable frames
Stable frames
Unstable frames
Unstable frames
Figure 7: Representative Frame Identification. If Stable frame is found by Algorithm I within the candidate regions, we use it as the representative frame. If not, we use the most dissimilar frames among unstable frames.
Algorithm Static Video Chunk Detection Algorithm
1: procedure DETEECTSTATIIFRRMAEE AMESS starttime, endtime)
2: video itriteofstartimmeeendinn
3: firedFrames + 0
4: VSI alddatedaaaadsssss
5:
pree + first frame in video
6:
for frame E rest of frames in video do abs + absolute difference between frame and prevFrame absDif f fThresh t apply adaptive thresholding using Gaussian filter Diff mean V al +- mean value of absDif] JbsDiffThress [Thresh mean al < then
7:
8:
9:
11:
FizedFramee <- fizedFrames U frame
else
13:
if length of fizeeFrammes > minimum duration then
14:
subelip + extract sub-clip of frames with constant background from fireeFFrmmes for patch E randomly selected patches in each frame of subclip do SSIMVal + calculate SSIM of patch if SSIMV al > threshold then SSIMV alidated Frames + SlslMaaaidtedFFrroee ramess U frame end if end for end if fixedFrames t- 0 end if
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
DrevFrame + frame end for
25:
26:
staticT iesiicTimestmppss + extract start and end times from SSIMValidatedFrames
return SraticTimestone
28:
end procedure
Aligning both modalities. The alignment of the images with their corresponding text requires the implementation of unique algorithms. These algorithms are designed to reduce duplicate content and ensure accurate mappings between image and text. See Figures 8 and 9 and Table 4 for
17

---

demonstration of image-text alignment process. See Figure 10 for sample images and their corresponding medical and ROI texts and the uub-pathlogyy classification provided by the LLM.
Chunked
time
3
10)
X
Mapped
Keywords/Entrier from text within
X Not Mapped
Figure 8: Overview of use of timing and keywords for Alignment Images within video chunk, i.c (A, B, C), Ia at in are aligned with medical texts extracted within the same chunk. The raw raw_keywords within each example chunk is colour coded to illustrate matches with keywords extracted from the medical texts and only matching keywords allow for the pairing of texts containing said keywords to image frames with frame-times around raw keywords times.
Transitions
B
A
C
X
X
X
Image Chunks
Neffee
C
L3
fs
Te Tu
7--6-5 1220
Text Chunks
Figure 9: Video Chunking algorithm illustrate. With each transition tag explained in Table we leverage predicted histopathology frames times / ..... to segment videos into chunks. Chunks at are minimum are Tp in duration, this value is estimated based on the with minimum 20 words being captured per chunk. Images within chunk, unlike texts, not overlapping with other chunks. Text overlap is done to provide needed context for LLM text correction and extraction.
of the video
A.2
Other data sources
A.2.1 PubMed Open Access Articles
We searched the PubMed open-access from 2010 2022 with keywords (pathology, histopathology. whole-slide image, H&E, and 148 keywords from histopathology glossary ") We utilized Entrez to retrieved the top 10,000 most relevant articles for each keyword. This query yielded 109.518 unique articles with PMCIDs. We extracted 162, 307 images and their corresponding captions. Using our histopathology classifier and cropping multi-plane figures as described in A.4 we extracted 59, 371 histopathology image and caption pairs with an average caption length of 54.02 tokens. Figure []] demonstrates the pipeline of collecting data from PubMed.
http://iiillllliooooooooooooooon "http:/www.acbb.crk alm nih. niw.gov/EEttrrezz
18

---

Table 4: All (six) transition states for chunking narrative style videos. p P(H),, i. is the binary histo image classifier prediction at the current frame's s time t, and p( ( is the prediction at next frame's 's time 4,,1, where Tn is the cumulative running time and Tp the estimated minimum chunk time for the video, determined by the words per second of the video. Text and image chunks are implemented as an ordered list of time intervals and image indexes.
P( P(
ba fn-
To
T. >
T.
Text chunk
Image chunk
Tag
A
end tni appendíe,
append index chunk if state is empty append prior in- dex; reset state
B
0
start = max((nn-1) Ip)
append index to chunk state C
end = tai appendíe, e): reset state; start l, 2p
append index to chunk state; re- set state
D
end fai append(s, e) reset state; start = la - 2p
append index to chunk state; rel
E
0
set state
0
append index to chunk state
F
A.2.2 Histopathology Image Retrieval from LAION
The Large-scale Artificial Intelligence Open Network (LAION-5B) [52] curated over 5 billion pairs of images and text from across the Internet, including a substantial volume of histopathology-reen related data. We tapped into this resource by retrieving the 3000 most similar LAION samples for each of the 1, 0000 pairs of images and text sampled from PubMed and QUILIT, using a CL.IP model pre-trained on the LAION data. The retrieval process utilized both image and text embeddings, with cosine similarity serving as the distance metric. Subsequently, we eliminated the duplicate images and removed all non-English pairs from the remaining pairs using LangDetec Consequently, the process yielded 22, 682 image and text pairs.
A.2.3 Twitter Data from OpenPath
We utilized list of tweets curated by Huang al. 24 which totaled up to 55, 000 unique tweets and 133, 511 unique image-text pairs. This exhibits a one-to-many relationship that leans towards the image side, differentiating our work from the OpenPath approach, where we had one image matching with multiple captions (as in the case of MS-COCO captions). In order to maintain comparability with OpenPath, we followed their text pre-processing pipeline given in 24
A.3 Histopathology and Magnification classifier
We use an ensemble of three histopathology image classifiers. To ensure robustness, our ensemble approach consists of two small Conv-NeXt models (38) and one linear classifier fine-tuned with DINO features [11]. This combination is necessary due to the homogenous appearance of histopathology images and the risk of false positives from similar pinkish-purple images. One Conv-NeXt model is trained in detecting E Immunohistochemisty (IHC) stained tissue images, while the other models are trained to handle all IHC stains and tissue types. The training data includes eight sub- groups of the TCGA WSI dataset and a mix of general-domain images, PowerPoint (slide) images, and scientific figure datasets. See Table 5 for details of these datasets.
For the magnification classifier, we finetune a pretrained ConyNeXt-Tiin model 338 with standard preset hyperparameters for few epochs and select the best performing model the validation set. To generate a training set for the magnification model, TCGA subsets were segmented into patches using a method similar to (64). These patches were generated at various magnifications. which were then categorized into three labels: O:{ 1.25x, 2.5x. 5x, 10x), 1: 1:(20x), (:(40x). The TCGA subsets were chosen to ensure diverse representation of tissue morphologies and cancer types, thereby ensuring robust and comprehensive model training. The model was also trained on cytopathology microscopy images and various IHC stains beyond H&E to enhance the model's generalizability
19

---

Medical TEXT
ROI Text
Sub-pathology
(Theree are clusters of cels with micro follicul formations: pseude incuusion, nudlej, nuclear grooves, and small nutleoli are present some cells.)
I'clustrrs of cells', 'micro follicular formations', 'nuclear inclusions', 'oval nuclet', 'nuclear grooves', 'mmal nucleolf]
[Endocrine': 'Cycopathology' 'Heed Neck]
['Cluster of macrophages and T colls characteristic rheumatic fever.; 'Aschoff body a characteristi feature acute rheumatic fever "Macrophages with elongated dhromatin are called Aritchiow cells and are commonly seen Aschoff bodies., 'Pancorditis with Aschoff bodies present.]
I'Cluster macrophages and cells' Aschotf 'Macrophages elongated chromatin,, 'Anitchkww cells', PPancaddtii]]
[Cardllc'' 'Hematopzhologr/ 'Endocrine"]
['An 80 year-old old man has scar-ike plaque on called malignant blopsy!! "The tissue affected by plaque extends from the apidermis to the galea aponeurotica, near the periosteum of the skull:; TThe skin, dermis, and subrutis affected by the process.]
I'scar-llke plaque on the scaip', malignant blopsy,, denmis, subcuts affected by process))
[Dermtoopshollgy
tissue', 'Hematopatollosy
[loflammatory calls surrounding cartilage can indicate acute chandritis, with neutrophil being principal type., 'chronic chondritis may diagnossd tymphocytes are the predo minot infoommatory cell type.!
I'cartllgg', inflammatory cells]]
(Hematopathology
'Bone',
'Dermatosathologyy
('Largo histlocytas abundant cytoplasm identified as Rosei- Dorfman histionytes!/ '5100 stain showed perivoscular cuffing', 'tnitial diagnosis of infammatory poeudotumor arbit'', 'Rosal Dorfman disease may burn out and leave behind fibrotic packets.] ]
I'Lagg histiocytes', perrivasuular cuffing' 'forotic pockets"
[Dematopatholagy 'Soft tissue', 'emstopathologyy]
('Epidermal meldermal acanthosi and papillmmaosi's
l'Epidermal acanthosis and papillomatasis flrrge sebaceous glands', mite!)
[Dematoppthollgy 'Soft tissue', 'Henatopstholayy]
seborrheic keratosis!" glands directly through their duct out to the skin surface, which abnormal.; PPresence of demodex
resembling
'Presence large
('Histologcal description of glandular tissue with little atypia but locared a place where not belong
l'standulaa tisque', ppancreas]]
Gaarrolntesstnal 'Pancreatic";
criteria presence of maligrancy 'Glandd Iocated on the pariphery and infilrating into adventitis and peripancrentic tissue may [alignntt]]
'Homatopathologyy
Figure 10: A collection of sample images from dataset, accompanied by corresponding medical text, ROI text, and the top three sub-pathology classifications derived from the ASR text using the L.IM.
across different conditions. Only the ACROBAT and TCGA datasubsets preprocessed to divide the WSIs into patches at various scales.
A.4 Support Models, Ontology Databases and Algorithms
This section describes the support models, ontology databases and handcrafted algorithms utilized within our pipeline for both searching and parsing our data.
Ontology databases. We employ various ontologies, both specific to histopathology and general ones. Among them are OCHV (2), FMA [42,, BCGO 12 NCIT [19), MPATH (5T) HPATH [22. and CMPO 299. These ontologies serve dual purpose. First, we used ceooppathoooy-ssecciii ontologies (HPATH. MPATH, BCGO, and CMPO) to provide words/phrases to condition the LLM, enabling to identify incorrect words. Second, all ontologies, in conjunction with UMLS, are used to obtain terms or phrases for validating the output the LLM.
!2 Patg::/hhoorndbbbooooooogggggggg
20

---

Pub Med
Separate pibiigres
Captin:: Endencna
candinoma. miE sain Ubs.
Neywords
Magriftarton-
PMA WMC 71C77240822
English
Magrifictioo Conothrr
Relevance
(a)
(b)
(c)
(d)
(e)
Figure 11: (a) Search PubMed open access database, filter based on keywords, date, language and sort by relevance. (b) Download paper and media for each search result. (c) Extract and pair figures and captions. (d) Separate multi-plane figures, find histopathology images and their magnification. (c) Final result.
Table 5: Datasets used train the histopathology image classifier. [um per pixel MPP]
Data Source
Subset
#WSI
#pathees
Train-Test
Mugnification
Image-size
GBM
19
89,022
LUSC
20
LIHC
20
TOGA (H&E Stain)
169,431
84715-16943
57,671 20x
384 x 384
KIRC
16
16,660 10x
KICH
4,748 5x
BRCA
17
1,465 2.5x
SKCM
19
466 - 1.25xx
ACROBAT Weitz
61
H&E K167
99
50589
28105 22484
(I0k, Sx, 2.5x)
384 x 384
HER2
BCT Liu al. t0
4,870
20x (0.46 MPP)
1024 x 1024
CCESD al
35
686
100x/400x
2048 1536
Smear Hussain
25
963
400x
2048 1536
Celeh Liu
202,599
8,103-1,944
Places Zhou al. (0)
36,550
2,109.1,372
A12D Kembhavi ct
4,903
0.7-0.3%
Doctig Jobin et
27
33,004
0.8-0.2%
Scilig pilot Karishma
60
1.671
0.8. 0.2%%
SlideImages Moris al. 400
8,217
0.8-0.2%
TeaVQA Singh et al.
54
28,472
0.8 0.2%%
SlideShare- IM Araujo
0.8-0.2%.
Sub- Sub-pathology types. The list of all 18 sub-pathology types used to prompt LLM on the text classification task are: Bone, Cardiac, Cyto, Dermato, Endocrine, Gastrointestinal, Genitourinary, Gynecologic, Head and Neck, Hemato, Neuro, Ophthalmic, Pediatric, Pulmonary, Renal, Soft tissue, and Breast Histopathology. Figure 12 provides the LLM prompt to retrieve the top three sub-pathology classification based on a given text.
Pre-processing multi-plane figures. Many figures in academic papers are multi-plane, which means number of sub-figures (Charts, graphs, histopathology and honhiistopatholog sub-figures) are placed next to each other to make larger figure. We extracted individual images from multi-plane figures create multiple instance bags. To locate boundaries and white gaps between sub-figures, we utilized Sobel filters. Binary thresholding was then used to find the contours surrounding the figuuis We employ image size and image ratio thresholds remove undesirable sub- sub-figures and our histopathology classifier to maintain just histopathology sub-figures. We supply the histological sub-figures individually for this type of figure by appending "_[0-9++" " to the end of the multi-plane figure id. If figure is divided into more than 5 sub-figures, we preserve the original image to ensure that the resolution of these sub-figures remains reasonable. Figure K shows an overview of this pre- rre-processing step in different scenarios of successful and unsuccessful crops.
21

---

System Prompt:
You histopathology text classifier
User Prompt:
Inagine you are a text classifier. Classify the given text into one of the following surgical pathology types nanely: Bane, Cardize, Cytopathology, Dermatapatholog. Endocrine, Gastrointestinal, Genitourinary, Gynecologic, Benatopatbology, Heuropathology, Dpathalnic, Pediatric, Pulnonary, Renal, Soft tisaue, Breast pathology. Output only the top pathology typea in an ordered python 1ist
Neck,
Few-shot
"Radicular cyst arises within the periodontal 1igament space, particularly the periapex from the epithelial cell of malassez. These radicular cysts are caused inflammation following the death of the pulp extending into the periapical radix. Radicular cysta caused by inflammation are always associated with non vital tooth."
examples:
* S''Sftt tisaue', 'Dermatopathologyy. 'Henatopathology""
INPUT:
lesion with slight thickening the muscularis mucosa and submucosa. There is subtle change in the lamina propria that doeen's look quite like normal stromal cells. Description of slight thick thickening of the muscularis mucosa and submucosa with subtle changes in the lamina propria. Highlighted field shows the changes cells lamina propria appear pink and spindly."
dramatically. Abnormal
OUTPUT: "['Gastrointeatinal"
'Soft tissue', 'Hematopathology ']
Figure 12:
Prompting LLM with few-shot examples to extract the top three sub-pathology classifica- tion of given text.
(a)
(b)
(c)
(d)
Figure 13: (a), (b), and (c) successfully cropped sub- sub-figures where histopathology images (green box) kept and non- histopathology (red box) images are removed. (b) histopathology crops are kept as not separated because the individual crops don't 't meet the size threshold so the original figure is kept. (d) Unsuccessful crop due to minimal gap between sub-figures. Original image is stored.
22

---

A.5 Privacy preserving steps
In order to ensure privacy while handling the dataset, several steps were taken to protect sensitive information. These steps include:
Reduction of Signal Noise using a LLM: To protect privacy of dataset, a LLM was utilized to reduce the signal-to-noise ratio. By applying the LLM, irrelevant or sensitive information was masked or removed.
Exclusion of Videos Not Fully in Narrative Style: Videos that did not adhere fully narrative style were intentionally left out of the dataset. This step was taken to minimize the risk of including any potentially private or sensitive content that could compromise individuals' privacy.
Release of Video IDs and Reconstruction Code: Instead of directly releasing the complete dataset, only video IDs from YouTTuhe were made public. Additionally, the code is provided to enable researchers to recreate the dataset.
Collection from Diverse Channels: Data collection was performed from wide range of sources, including both large and small channels. This approach aimed to decrease the risk overfitting to specific channel types, thereby mitigating privacy concerns associated with potential biases linked to partieular channels.
23


Reviews:
[{"review_id": "fHrc6IAFya", "reviewer": "Reviewer_MJNv", "summary": "", "questions": "", "limitations": "While the authors did not sufficiently address the limitations, it's somewhat understandable given the limitations of existing tools in comprehending and interpreting medical terminology within the histopathology domain. Further data refinement could potentially benefit from the inclusion of expert human input.\n\nAdditionally, based on existing literature, training CLIP tends to require much larger datasets compared to other pretraining architectures. I would recommend exploring other self-supervised algorithms such as ViLT and BEiT. Pretraining these models from scratch with the proposed dataset could potentially yield superior results compared to simply fine-tuning the pretrained CLIP model.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The paper addresses the scarcity of analogous data in the medical field, specifically in histopathology, which has halted comparable progress. The introduction of QUILT-1M can help overcome this limitation and enable researchers to develop better models for histopathology. The dataset can have significant implications for the field of histopathology, which can ultimately benefit patients by enabling better diagnosis and treatment.\n\n2. This paper provide a comprehensive pipeline about the data preprocessing and post-processing with the a mixture of models, including large language models, handcrafted algorithms, human knowledge databases, and automatic speech recognition, to curate the datase. This information is pretty helpful for the research community to create more open-sourced datasets in other domains.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "xUxsA9IcTJ", "reviewer": "Reviewer_8biP", "summary": "", "questions": "", "limitations": "Yes, the authors discussed the limitations of the dataset due to the involvement of automation tools for data collection. In addition, the study is also aware of the social bias in data collection process.", "rating": 10, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper is well written. The dataset generation process is well described, which ensures the quality of the dataset. Compared to prior histopathology text-image datasets, QUILT-1M contains more sample pairs. The limitation and potential social ethic issues are also discussed. I have no doubt that the QUILT-1M dataset and the multimodal model QUILTNET have significant contribution to digital histopathology.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "LShsRpBKMx", "reviewer": "Reviewer_7iJR", "summary": "", "questions": "", "limitations": "- It is unclear to what extent human experts verified the curated text-image pairs and the level of involvement i.e., how many expert pathologists confirmed the false positive rates.  \n\n- A CLIP model was pre-trained on the 1M dataset, did the authors evaluate the potential of the fine-tuned model on downstream tasks when trained with less data (<< 1M). This may be useful to assess the utility of this dataset in low-resource environments (hardware etc.).", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The paper is well written and easy to follow and motivates the need for such a dataset in coherent manner.  \n\n- The authors target histopathology images, this is very relevant and timely. As opposed to many existing vision-language models in medical literature that focused on different modalities, the curation of this dataset tailored to pathology is truly a commendable effort. \n\n- The use of Youtube videos is novel and interesting.  The internet is considerably an unlimited source of information, and the designed curation procedure considers several aspects from filtering to modalities alignment after text-extraction from frames. \n\n- The authors extensively evaluated the curated dataset via downstream tasks after pre-training, achieving state of the art performance on several tasks i.e., zero/few-shot and cross modal retrieval.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "HNjncC7ohM", "reviewer": "Reviewer_rAj5", "summary": "", "questions": "", "limitations": "The authors adequately address limitations and social biases.", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "Please find a list of strengths below:\n\n1. The authors were clever to mine YouTube for this dataset. This is indeed a large resource, and I myself have watched several histopathology videos on YouTube.\n2. The size of the QUILT dataset is a strength. The authors write that QUILT includes 419,780 images and 768,826 matched text pairs. \n3. QUILT includes multiple sentences per image, which allows multiple descriptions of images.\n4. The authors provide a detailed description of the method they used to create QUILT.\n5. The experimental results (Section 4) are comprehensive.\n6. Limitations and biases are discussed.\n7. The code to create the dataset and models are publicly available. Thank you for hosting your models on HuggingFace. I am not affiliated with HuggingFace but I appreciate that your model is available for reuse. Please see point 11 in the “opportunities for improvement” for a related suggestion about the dataset. A critical weakness is that the final dataset is not publicly available.\n8. The authors have created an effective website to describe the dataset.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "U9Umf4kP4k", "reviewer": "Reviewer_jVw5", "summary": "", "questions": "", "limitations": "- It would be interesting to see to what extent including different image-text sources such as LAION, Twitter, and PubMed can improve the accuracy of the proposed model. Could the authors conduct an ablation study to assess the impact of incorporating LAION, Twitter, and PubMed datasets on the QUILT dataset? This study would determine whether these additional sources are beneficial in enhancing the model's performance or if they potentially lead to a decrease in performance.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The authors employed a novel approach to collect this dataset, integrating various state-of-the-art techniques. These include text extraction using ASR, text correction utilizing the LLM, and employing image classifiers to identify videos containing histopathology images.\n\n- The distribution of all the codes used in constructing this dataset represents a substantial engineering endeavor.\n\n- The release of this artifact holds significant importance for the research community. Recent scientific progress has underscored the value of comparable datasets containing numerous image and text pairs. However, existing sources of such datasets are often limited in size or not openly accessible (e.g., PMC-15M).\n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The paper presents a multimodal image-text dataset comprising one million histopathology image-text pairs, highlighting innovative data collection methods involving ASR and LLM for text extraction and correction. While concerns were raised regarding dataset accessibility, overlap with existing sources, and the performance gap between fine-tuning and training from scratch, the reviewers were highly enthusiastic about the merits of this work, including the comprehensive distribution of code, the dataset's significance for the research community due to its size, clear and well-written content, and ethical considerations. I believe this work will be of significant value to the research community.
Decision: Accept (Oral)