SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wisdom O. Ikezogwo\* Mehmet S. Seyfioglu Fatemeh Ghezloo
Dylan Geva Fatwir S. Mohammed Pavan K. Anand
Ranjay Krishna Linda G. Shapiro
{wisdomik,msaygin,fghezloo,dgeva,pka4,ranjay,shapiro}@cs.washington.edu fatwir@uw.edu
Recent accelerations in multi-modal applications have been made possible with the plethora of image and text data available online. However, the scarcity of analogous data in the medical field, specifically in histopathology, has slowed comparable progress. To enable similar representation learning for histopathology, we turn to YouTube, an untapped resource of videos, offering 1,087 hours of valuable educational histopathology videos from expert clinicians. From YouTube, we curate QUILT: a large-scale vision-language dataset consisting of 802,144 image and text pairs. QUILT was automatically curated using a mixture of models, including large language models, handcrafted algorithms, human knowledge databases, and automatic speech recognition. In comparison, the most comprehensive datasets curated for histopathology amass only around 200K samples. We combine QUILT with datasets from other sources, including Twitter, research papers, and the internet in general, to create an even larger dataset: QUILT-1M, with 1M paired imagedataset samples, marking it as the largest vision-language histopathology dataset to date. We demonstrate the value of QUILT-1M by fine-tuning a pre-trained CLIP model. Our model outperforms state-of-the-art models on both zero-shot and linear probing tasks for classifying new histopathology images across 13 diverse patch-level datasets of 8 different sub-pathologies and cross-modal retrieval tasks $^{2}$ .
Whole-slide histopathology images are dense in information, and even individual image patches can hold unique, complex patterns critical for tissue characterization. Summarizing this information into a single label is an oversimplification that fails to capture the complexity of the field, which covers thousands of evolving disease sub-types [55]. This highlights the need for more expressive, dense, interconnected representations beyond the reach of a singular categorical label. As such, natural language descriptions can provide this comprehensive signal, linking diverse features of histopathology sub-patch structures [20, 24].
If there were a large-scale vision-language dataset for histopathology, researchers would be able to leverage the significant advancements in self-supervised vision and language pre-training to develop effective histopathology models [46]. Unfortunately, there is a significant scarcity of comprehensive datasets for histopathology. Notable open-source contributions have been made with datasets like ARCH [20] and OpenPath [24]. Yet, these sources are still somewhat limited due to their size, as the
*Reach corresponding author at wisdomik@cs.washington.edu; Equal contribution.
2The data and code will be available at Quilt-1M
former has only  $\approx 8\mathrm{K}$  samples and the latter (the largest histopathology vision-language dataset to date) has about  $200\mathrm{K}$  samples. Although recent efforts (e.g. PMC-15M [67]) curated 15M image-text pairs across a variety of different biomedical domains from Pubmed [48], whether their samples are specific to histopathology remains ambiguous; worse, their dataset is not openly available.
Figure 1: Overview of QUILT curation pipeline. We identify relevant histopathology YouTube videos in Search. For Image extraction, we find and de-noise histopathology frames using trained models. In Text section, we rely on a conventional Automatic Speech Recognition (ASR) model and leverage Unified Medical Language System (UMLS) and large language models (LLMs) for post-processing and ASR error correction. Relevant sub-pathology, medical and region-of-interest (ROI) text are extracted using an LLM. Finally, domain-specific algorithms are used to Pair images and text, eliminating duplicates to yield QUILT, a richly annotated image-text dataset for histopathology.
To address the need for a large-scale vision-language dataset in histopathology, we introduce QUILT: containing 437,878 images aligned with 802,144 text pairs across multiple microscopic magnification scales covering from  $10\mathrm{x}$  to  $40\mathrm{x}$ . We draw on the insight that publicly available educational YouTube histopathology content represents an untapped potential. We curate QUILT using 1,087 hours of valuable educational histopathology videos from expert pathologists on YouTube. To extract aligned image and text pairs from the videos, we utilize a mixture of models: large language models (GPT-3.5), handcrafted algorithms, human knowledge databases, and automatic speech recognition. QUILT does not overlap with any current open-access histopathology data sources. This allows us to merge our dataset with other open-source datasets available. Therefore, to create an even larger and more diverse dataset, we combine QUILT with data from other sources, such as Twitter, research papers, and the Internet, resulting in QUILT-1M. The larger QUILT-1M contains one million image-text pairs, making it the largest public vision-language histopathology dataset to date.
Using QUILT and QUILT-1M, we finetune vision-language models using a contrastive objective between the two modalities. We extensively evaluate it on 13 external histopathology datasets taken across different sub-pathologies. We report zero-shot classification, linear probe, and image-to-text and text-to-image retrieval tasks. Against multiple recently proposed baselines (CLIP [46], PLIP [24], and BiomedCLIP [67]), models trained with QUILT-1M outperform all others. Our ablations identify the importance of QUILT.
QUILT offers three significant advantages: First, QUILT does not overlap with existing data sources; it ensures a unique contribution to the pool of available histopathology knowledge. Second, its rich textual descriptions extracted from experts narrating within educational videos provide more expressive, dense interconnected information. Last, the presence of multiple sentences per image fosters diverse perspectives and a comprehensive understanding of each histopathological image. We hope that both computer scientists and histopathologists will benefit from QUILT's potential.
We built upon a growing literature applying self-supervised learning and other machine learning methods to medical image understanding.
Machine learning for histopathology. Early representation learning work in computational pathology primarily relied on weakly-supervised learning, with each whole-slide image (WSI) receiving a single label. The limited nature (single label to many patches) has produced sub-optimal models [12, 26] at the patch level. Lately, a self-supervised learning approach, which learns useful representations from unlabeled data, has shown some success [26, 13, 12]. Most of this work has been unimodal. They use image augmentations similar to those used for natural images [14], mostly differing by way of consciously injecting domain knowledge. For example, they leverage the compositional nature of H&E stain information of whole-slice images [26], or inject hierarchical morphological information at different magnifications [13], or combine with other modalities like genomic features [12] or with descriptive text [20]. When text data is used, the objectives similarly use augmentations seen in natural language [50]. By contrast, we explore self-supervised mechanisms that learn better histopathology information representations that go beyond a single label, aided by language descriptions.
Medical vision-language datasets. Learning vision-language representations demands a large dataset of images aligned with descriptive text, a resource that is notably lacking in histopathology. The MIMIC-CXR-JPG v2.0.0 dataset [28], for example, consists of de-identified hospital-sourced chest radiographs and reports. For histopathology, The Cancer Genome Atlas<sup>3</sup> provides de-identified PDF-reports for a limited number of WSIs. Despite this resource, the enormous size of this data (reaching up to 120,000<sup>2</sup> pixels) makes processing challenging, limiting its use to a small number of focused studies [39]. A majority of medical vision-language datasets are concentrated in the radiology sub-domain, due to the relatively straightforward process of collecting validated multimodal data [28]. Many models are trained on a subset of PubMed [48] or comparable radiology datasets [68, 23, 18, 43]. PMC-15M [67], a recent subset of PubMed not specific to histopathology, was used to train multiple models. While the models themselves are public, PMC-15M is not, making it hard to determine what portion of it is histopathology-relevant.
Vision-language pairs on histopathology. One of the first histopathology vision-language datasets, ARCH, contains only 7,614 accessible image-text pairs [20, 22]. Later on, [24] released OpenPath, a dataset of 200K image-text pairs extracted from Twitter. This was the largest histopathology dataset until QUILT-1M.
Video data for self-supervision. Numerous recent studies have started to tap into video data. For instance, millions of publicly accessible YouTube videos were used to train a vision-language model [65, 66]. Similarly, a causal video model was trained by using sequential gaming videos [6]. Localized narratives [58, 44] provide another example of dense, interconnected supervision for a single image. Despite the potential of video content, video often yields noisier datasets compared to static sources. Recently, the enhanced capabilities of automatic speech recognition models streamlined the curation of large-scale cleaner datasets from videos [65, 6, 67]. Furthermore, the growing versatility of large language models has shown promise as data annotators, information extractors [33, 59, 15, 21], text correctors [63], and as tools for medical information extraction and reasoning [1, 56].
3 Curating QUILT: Overview
Creating a vision-language dataset from videos is a significant undertaking, as not all videos are suitable for our pipeline. Many either lack voiced audio, are not in English, fail to contain medically relevant content, or have insufficient medical relevance—for example, videos that present static images of histopathology content on a slide deck, or those that briefly cover histopathology images in pursuit of a different objective. Conventional automatic speech recognition (ASR) systems also struggle with the specialized requirements of histopathology transcription, necessitating a non-trivial solution. The de-noising of text and image modalities adds further complexity as the videos are typically conversational and, therefore, inherently noisy. Instructors pan and zoom at varying speeds, recording a mix of relevant and irrelevant histopathological visual content in their videos. As such, trivially extracting frames at static intervals fails to capture the data appropriately. To collect QUILT we trained models and handcrafted algorithms that leverage the nuances in the instructors' textual and visual behavior, ensuring accurate collection and alignment of both modalities.
<sup>3</sup>https://www.cancer.gov/tcga
Figure 2: QUILT examples. Input is the corrected ASR caption for the representative image. Output are the medical and ROI extracted text(s) paired with the image (see Section 3.1). In histopathology, understanding tissue characteristics often involves views from varying magnification levels. Thus, in QUILT we estimate an image's magnification (indicated by the relative size of the microscope icon).
INPUT: "...so here we have a huge lumen and we have the lining epithelium and then we have the underlying connective tissue capsule. let's see the features in detail one by one. if you see this lumen lumen has sparse amount of keratin. so definitely this is a keratinous cyst. so we have a cyst that is actually producing keratin. let's move into the epithelium and the epithelium if you carefully observe most of the areas the epithelial appears uniform thickness right from here.
{MED:"TThe cyst is producing keratin, indicating it is a keratinous cyst."
"The epithelial lining appears to have uniform thickness" [ROF]
"Sparse amount of Keratin in the lumen." 'uniform thickness of the epithelial lining']
INPUT:"...tend to see some inflammation here yes. when you notice this there is a dense amount of inflammation here yeah you see this tiny blue dots these are the lymphocytes and these cells are your inflammatory cells and they are kind of having an effect on the epithelium. if you see this epithelium because of the inflammation the epithelium starts OUTPUT:
'MED':
Dense inflammation is present in the region with lymphocytes and other inflammatory cells
having an effect on the epithelial cells]
"lymphocytes", "inflammatory cells")
INPUT: "...of a daughter cyst where you see so this was an epithelial island with showing central keratinization the cells producing lot of keratin and at the same time you have the epithelium with the lumen. so this forms a tiny daughter cyst and this is one of the most important trees in why this keratocyst has a high amount of recurrence rate, one more important feature of odontogenic keratocyst is lies within the epithelium."
[MDE: The presence of daughter cysts is one of the reasons for the high recurrence rate of keratocysts, "Epithelial islands with central keratinization and high keratin production are seen in keratocysts"]
ROI: "epithelial island",
"central keratinization", "daughter cyst")\}
3.1 QUILT: Collecting medical image and text pairs from YouTube
Our proposed dataset curation pipeline involves (1) gathering channel and video data covering the histopathology domain, (2) filtering videos based on a certain "narrative style", (3) extracting and denoising image and text modalities from videos using various models, tools, and algorithms, (4) postprocessing denoised text by LLMs to extract medical text and finally, (5) splitting and aligning all modalities for curating the final vision-language pre-training (VLP) data. See Figure 1 (and supplemental material) for a detailed overview of the pipeline.
Collecting representative channels and videos. Our pipeline begins by searching for relevant channels and video ids on YouTube, focusing on the domain of histopathology. Using keywords spanning 18 sub-pathology fields (see supplement for more details), we search among channels before searching for videos to expedite discovery, considering that video searches are time-consuming and the APIs pose limitations on numerous requests [65]. Channels with subscriber count  $\geq 300K$  are excluded to avoid large general science channels, as educational histopathology channels often have fewer subscribers. We then download low-resolution versions of all identified videos, with the lowest resolution at  $320\mathrm{p}$ .
Filtering for narrative-style medical videos. For each video within each channel, we exclude videos that are shorter than 1 minute, non-voiced, or have non-English audio. For videos meeting these heuristics, two decisions are made:
(A) Do they have the required medical content, i.e., histopathology image-text pairs?
(B) If so, are they in narrative style – videos wherein the presenter(s) spend a significant time panning and zooming on the WSI, while providing vocal descriptions of image content?
For (A) we automatically identify the relevant videos by extracting keyframes from a video. These keyframes are automatically extracted using FFmpeg  $^{4}$ , marking the beginning or end of a scene (frames containing significant visual changes). The software requires a threshold that determines the minimum amount of visual change required to trigger a keyframe. Through experimentation, we set different thresholds for various video durations, with smaller thresholds for longer videos. Next, we train and use an ensemble of three histopathology image classifiers to identify videos with histopathology images (See supplement for more details).
For (B), in which we identify narrative-style videos, we randomly select keyframes predicted to be histopathology. For each such selected frame, we extract the next three histopathology key-frames and compute the cosine similarity between the selected frame and each of the subsequent three
4https://ffmpeg.org/
frames. If all three have similarity scores  $\geq$  a preset threshold of 0.9, we count it as a narrative streak. A video is identified as narrative style if at least  $10\%$  of the selected frames exhibit a narrative streak. Consequently, we download all narrative-style videos at high-resolution. Narrative-style videos typically cover WSIs at various magnifications, hence, we train a tissue-image-magnification classifier to predict the following three scales:  $\{(1 - 10)\mathrm{x}, (>10 - 20)\mathrm{x}, (>20)\mathrm{x}\}$ . This provides relevant metadata for downstream objectives.
Figure 3: Prompting the LLM by providing few-shot examples to perform the following tasks: (Left) correcting noisy ASR text within its context. We highlight the probable list of misspelled keywords in yellow and their corrections by the LLM in gray. Additional missed errors/misspelled entries identified by the LLM are highlighted in blue. (Right) extracting medical (MED) and ROI text from a given text. We highlight the definition of medical and ROI text in blue and gray respectively.
Text Extraction using ASR and text denoising. The high costs associated with private medical ASR APIs necessitated the use of a more conventional ASR model: Whisper [47]. As anticipated, this model often misinterprets medical terms, thus requiring the use of post-processing algorithms to minimize its error rates.
We propose a four-step text de-noising and quality control pipeline: i) We utilize the Rake keyword extraction algorithm to extract keywords or key-phrases up to four words and refine them by eliminating stopwords [49]. ii) We then cross-check each refined entry against UMLS [7] using the SciSpacy entity linking package [41]. If an entry is not found within UMLS, we check for misspelled words within the entry using a spell-checking algorithm<sup>6</sup>, instantiated with a specialized list of histopathology terms curated from various histopathology ontology labels and definitions. iii) With this probable list of misspelled keywords, we condition and prompt the LLM with examples to correct the misspelled entry within its context (sentence), and secondly, we task the LLM with identifying additional unconditioned errors/misspelled entries. For both, we leverage a set of manually curated examples to prompt the LLM in-context Figure 3. For more examples and failure cases, see supplement for more details. iv) Finally, to de-noise the text, we resolve the output mapping of incorrect  $\mapsto$  correct entries by verifying the corrected words against UMLS and our curated list of histopathology words/phrases. Entries that pass this double-validation process are used to replace the initial noisy transcription. Leveraging domain-specific databases to extract the text and filter out noise allows us to bypass the correction of repetition errors and filler words, such as 'ah', 'uhm', 'the', etc. in tandem, using LLMs allows us to concentrate on correcting medically-relevant misspelled words, rather than correcting non-medically-relevant terms.
From the ASR-corrected text, we extract medical text which describes the image(s) as a whole. Also, when the speaker describes/gestures at visual regions-of-interest through statements like "look here ..., we extract the text entity being described as ROI text. To filter relevant medical text and ROI text from the ASR-corrected text, we utilize LLMs (See supplement for more details), a decision rooted in
5 nuance.com/en-au/healthcare/provider-solutions/speech-recognition/dragon-medical-one.html
<sup>6</sup>https://github.com/barrust/pyspellchecker
a few compelling reasons: 1) Curating pre-training datasets at a scale that can tolerate higher levels of noise, LLMs are more cost-effective than expert-human (medical) labor. 2) The task does not require LLMs to generate new information but instead they discriminate useful versus irrelevant signals, serving to improve the signal-to-noise ratio of the data. To extract relevant text, we prompt LLMs to filter out all non-medically relevant text, providing context as necessary. See Figure 2 for some example image-text pairs. Lastly, we instruct the LLMs to refrain from introducing any new words beyond the corrected noisy text and set the model's temperature to zero. Finally, we use LLMs to categorize our videos into one of the 18 identified sub-pathology classes. Similar to the previous tasks, this categorization is done by conditioning with a few examples and prompting the LLM to predict the top three possible classes given the text. More details, prompts, and additional examples are presented in the supplemental material.
Image frame extraction and denoising. For each video, we employ a similar method to that described in Filtering for narrative-style medical videos subsection to extract histopathology key-frames; our method leverages these frames' times  $t$  as beacons to break the entire video into time-intervals called chunks from which to extract representative image(s). Next, we extract the median image (pixel-space) of stable (static) frames in each chunk if they exist, else we de-duplicate the histopathology keyframes (beacons of the chunk). In essence, we use the extracted histopathology scene frames as guides for data collection, exploiting the human tendency in educational videos to pause narration during explanation, and we extract the relevant frame(s).
Aligning both modalities. For each narrative-style video, we perform the following steps to align image and text modalities: First, we compute histopathology time chunks denoted as  $\left[(t_1,t_2),(t_3,t_4),\dots (t_{n - 1},t_n)\right]$  from keyframes after discriminating histopathology frames using the histopathology ensemble classifier - (sceneChunks). Each scene_chunk is padded with pad_time to its left and right; see supplement for more details.
1. Text: we use the ASR output to extract the words spoken during each chunk in sceneChunks. Using the method described in Text Extraction using ASR and text denoising subsection, we extract the Medical and ROI caption for this chunk.
2. Image: we extract representative image(s) for every chunk/time-interval in sceneChunks as described in Filtering for narrative-style medical videos subsection above.
Finally, each chunk in sceneChunks is mapped to texts (both medical and ROI captions) and images. Next we map each medical image to one or more medical text. Using the time interval in which the image occurs, we extract its raw text from ASR and then correct and extract keywords using the Rake method, which we refer to as raw_keyword. We extract keywords from each medical text returned using the LLM, and we refer to these as keywords. Finally, if the raw_keyword occur before or slightly after a selected representative image, and overlap with the keywords in one of the Medical/ROI texts for that chunk, we map the image to the medical/ROI text. Example. keywords: psammoma bodies, match with raw_keyword: psammoma bodies within the ASR-corrected text 'Meningiomas typically have a meningothelial pattern with lobular-like arrangements and psammoma bodies.' Refer to Figures in the supplement for a detailed explanation of the method and examples of aligned image and text.
3.2 QUILT-1M: Combining QUILT with other histopathology data sources
To create QUILT-1M, we expanded QUILT by adding other disparate histopathology image-text open-access sources: LAION, Twitter, and PubMed.
PubMed Open Access Articles. We searched the PubMed open-access from 2010-2022, extracting 59,371 histopathology image-text pairs, using our histopathology classifier and multi-plane figure cropping algorithm. The images are categorized into (1) images that are fully histopathology, (2) multi-plane images that contain histopathology sub-figures, and (3) histopathology sub-figures cropped from (1) and (2). See supplement for more details.