SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Histopathology Image Retrieval from LAION. The Large-scale Artificial Intelligence Open Network (LAION-5B) [52] curated over 5 billion pairs of images and text from across the Internet, including a substantial volume of histopathology-related data. We tapped into this resource by retrieving 22,682 image and text pairs. See supplement for more details.
Twitter Data from OpenPath. We utilized a list of tweets curated by Huang et al. [24], which totaled up to 55,000 unique tweets and made up 133,511 unique image-text pairs. This exhibits a one-to-many relationship where many images were matched with multiple captions; this differentiated our work from the OpenPath approach. To maintain comparability, we followed their text pre-processing pipeline [24]. See supplement for more details.
Sub-Pathology Categories Over All Videos
Img-Text Pairs over Sub-Pathology
(c)
(d)
Figure 4: (a) Distribution of all videos over sub-pathology types. (b) Distribution of our entities across top 20 UMLS semantic types. (c) Number of image-text pairs within each sub-pathology type. (d) word cloud of all the text in QUILT.
3.3 Quality
To evaluate our pipeline's performance, we assess several aspects. First, we calculate the precision of our LLM's corrections by dividing the number of conditioned misspelled errors replaced (i.e., passed the UMLS check) by the total number of conditioned misspelled words found, yielding an average of  $57.9\%$ . We also determined the unconditioned precision of the LLM, similar to the previous step, and found it to be  $13.8\%$ . Therefore, we replace our detected incorrect words with the LLM's correction  $57.9\%$  of the time, and  $13.8\%$  of the time we replace the LLM's detected errors with its correction (see supplement for more details). To estimate the ASR model's transcription performance, we compute the total number of errors replaced (both conditioned and unconditioned) and divide it by the total number of words in each video, resulting in an average ASR error rate of  $0.79\%$ . To assess the LLM's sub-pathology classification, we manually annotated top-k ( $k = 1,2,3$ ) sub-pathology types for 100 random videos from our dataset. The LLM's accuracy for top-3, top-2, and top-1 was  $94.9\%$ ,  $91.9\%$ , and  $86.8\%$ , respectively. Also note that, by prompting the LLM to extract only medically relevant text, we further eliminate identifiable information, such as clinic addresses, from our dataset.
3.4 Final dataset statistics
We collected QUILT, from 4475 narrative videos spanning over 1087 hours with over  $437\mathrm{K}$  unique images with 802K associated text pairs. The mean length of the text captions is 22.76 words, and 8.68 words for ROI text, with an average of 1.74 medical sentences per image (max=5.33, min=1.0). Our dataset spans a total of 1.469M UMLS entities from those mentioned in the text (with 28.5K unique). The images span varying microscopic magnification scales (0-10x, 10-20x, 20-40x), obtaining (280K,
75K, 107K) images from each scale respectively with an average height and width of  $882 \times 1468$  pixels, as we leverage the max image resolution of videos. Figure 4 (a, c) plots our dataset's diversity across multiple histopathology sub-domains. This plot shows that the captions cover histopathology-relevant medical subtypes: findings, concepts, organs, neoplastic processes, cells, diseases, and a mix of laboratory and diagnostic procedures. Overall, across all 127 UMLS semantic types, our entities cover  $76.2\%$  of medically-related semantic types (e.g., findings, disease, or syndrome) and  $23.75\%$  non-medical (e.g., geographic area, governmental or regulatory activity).
4 QUILTNET: Experiments training with QUILT-1M
Figure 5: QUILTNET, outperforms out-of-domain CLIP baseline and state-of-the-art histopathology models across 12 zero-shot tasks, covering 8 different sub-pathologies (accuracy percentage provided).
We use the Contrastive Language-Image Pre-training (CLIP) objective [46] to pretrain QUILTNet using QUILT-1M. CLIP takes a batch of  $N$  (image, text) pairs and optimizes a contrastive objective to create a joint embedding space. The optimization process involves concurrent training of both image and text encoders to increase the cosine similarity of embeddings from aligned pairs, while decreasing it for unaligned pairs. The objective is minimized via the InfoNCE loss, expressed as:
$$
\mathcal {L} = - \frac {1}{2 N} \left(\sum_ {i = 1} ^ {N} \log \frac {e ^ {\cos (I _ {i} , \boldsymbol {T} _ {i})}}{\sum_ {j = 1} ^ {N} e ^ {\cos (I _ {i} , \boldsymbol {T} _ {j})}} + \sum_ {i = 1} ^ {N} \log \frac {e ^ {\cos (\boldsymbol {I} _ {i} , \boldsymbol {T} _ {i})}}{\sum_ {j = 1} ^ {N} e ^ {\cos (I _ {j} , T _ {i})}}\right)
$$
where  $I_{i}$  and  $T_{i}$  are the embeddings for the aligned  $i$ -th image and text, respectively. For the image encoder, we use both ViT-B/32 and ViT-B/16 architectures [16]. For the text encoder, we use GPT-2 [45] with a context length of 77, and PubmedBert [67]. We train QUILTNET by finetuning an OpenAI pre-trained CLIP model [46] on QUILT-1M to enhance its performance in histopathology. Once finetuned, we conduct experiments on two types of downstream tasks: image classification (zero-shot and linear probing) and cross-modal retrieval (zero-shot). We also compare the performance of fine-tuning a pre-trained CLIP model versus training it from scratch.
Downstream histopathology datasets. We evaluate the utility of QUILTNET on 13 downstream datasets: PatchCamelyon [57] contains histopathology scans of lymph node sections labeled for metastatic tissue presence as a binary label. NCT-CRC-HE-100K [31] consists of colorectal cancer images and is categorized into cancer and normal tissue. For SICAPv2 [53] the images are labeled as non-cancerous, Grade 3-5. Databiox [8] consists of invasive ductal carcinoma cases of Grades I-III. BACH [4] consists of breast tissues labeled as normal, benign, in-situ, and invasive carcinoma. Osteo [5] is a set of tissue patches representing the heterogeneity of osteosarcoma. RenalCell [10] contains tissue images of clear-cell renal cell carcinoma annotated into five tissue texture types. SkinCancer [34] consists of tissue patches from skin biopsies of 12 anatomical compartments and 4 neoplasms that make up the SkinTumor Subset. MHIST [60] contains tissue patches from Formalin-Fixed Paraffin-Embedded WSIs of colorectal polyps. LC25000 [9], which we divide into LC25000 (Lung) and LC25000 (Colon), contains tissue of lung and colon adenocarcinomas. For more details see supplemental material.
Results using zero-shot learning. Given the vast diversity of cancer sub-types in histopathology, it is critical that a model maintains comprehensive understanding without requiring specific data
Table 1: Linear probing. Classification results, denoted as accuracy % (standard deviation). Camelyon denotes the PatchCamelyon dataset. Supervised results are from each dataset's SOTA models.
Dataset	%shot	ViT-B/32	ViT-B/16
CLIP GPT/77	PLIP GPT/77	QUILTNET GPT/77	CLIP GPT/77	QUILTNET GPT/77	BiomedCLIP PMB/256	QUILTNET PMB/256
Supervised(%)
NCT-CRC [31]	1	91.0 (0.10)	93.75 (0.09)	94.64(0.22)	90.96 (0.10)	93.36 (0.23)	92.14 (0.12)	93.55 (0.25)
10	92.02 (1.30)	93.83 (0.06)	95.30 (0.03)	92.58 (0.12)	93.85 (0.04)	92.90 (0.07)	93.72 (0.08)
(94.0)	100	91.83 (0.01)	94.16 (0.01)	95.22 (0.01)	92.26 (0.09)	93.76 (0.02)	92.97 (0.05)	93.60 (0.01)
Camelyon [57]	1	80.38 (0.16)	87.26 (0.23)	87.62 (0.35)	80.28 (0.20)	84.78 (0.14)	83.63 (0.44)	83.48 (0.18)
10	82.67 (0.19)	87.48 (0.08)	87.55 (0.03)	82.20 (0.04)	86.77 (0.09)	84.18 (0.15)	84.42 (0.10)
(97.5)	100	82.80 (0.01)	87.34 (0.01)	87.48 (0.01)	82.55 (0.02)	86.81 (0.04)	84.23 (0.01)	84.44 (0.02)
SkinCancer [34]	1	84.27 (0.22)	91.07 (0.25)	90.93 (0.25)	85.62 (0.16)	88.29 (0.15)	87.53 (0.21)	88.06 (0.20)
10	89.0 (0.02)	93.39 (0.05)	92.99 (0.02)	90.28 (0.01)	91.20 (0.0)	89.23 (0.03)	90.03 (0.02)
(93.3)	100	89.02 (0.02)	93.29 (0.01)	93.03 (0.02)	90.29 (0.03)	91.20 (0.0)	89.16 (0.02)	89.91 (0.01)
SICAPv2 [53]	1	52.45 (2.41)	65.76 (2.65)	69.92 (1.02)	56.01 (0.66)	66.86 (1.16)	69.43 (1.03)	68.49 (1.06)
10	62.24 (0.65)	69.23 (0.43)	74.14 (0.38)	63.70 (0.69)	72.37 (0.65)	71.61 (0.31)	72.48 (0.42)
(67.0)	100	65.75 (0.16)	73.0 (0.14)	75.48 (0.12)	68.74 (0.10)	74.14 (0.16)	74.57 (0.04)	74.60 (0.17)
for retraining. Thus, we evaluate our model's zero-shot performance against three state-of-the-art models: CLIP, BiomedCLIP, and PLIP. Our model demonstrates superior performance, as illustrated in Figure 5, where it outperforms the other models in all but two datasets, in which BiomedCLIP performs marginally better. See supplemental material for UMap visualizations and cross-modal attention visualization comparison. The prompts used for these evaluations are presented in the supplemental material. To ensure a fair comparison with BiomedCLIP, which uses a ViT-B/16 and PMB/256 (pre-trained with [67]), we trained three different variants of our model. For detailed insights into the results, please refer to supplemental material.
Results using linear probing. We assess the few-shot and full-shot performance of our model by conducting linear probing with  $1\%$ ,  $10\%$ , and  $100\%$  of the training data, sampled with three different seeds; we report the average accuracy and their standard deviation in Table 1. We deploy our evaluation across four distinct datasets, specifically those with dedicated training and testing sets among our external datasets. Remarkably, our model, utilizing the ViT-B/32 architecture with GPT/77, outperforms its counterparts, BiomedCLIP, PLIP, and CLIP, in most datasets. On the NCT-CRC and SICAPv2 datasets, our model surpasses even the fully supervised performance using only  $1\%$  of the labels. Also, note that for some results  $10\%$  does better than  $100\%$ ; this is because we are sampling from each class equally, and thus the  $10\%$  subset contains a more balanced training set than  $100\%$ , for datasets that are very imbalanced, resulting in sub-optimal performance at  $100\%$ .
Results using cross-modal retrieval. In our study, we evaluate cross-modal retrieval efficacy by examining both zero-shot text-to-image and image-to-text retrieval capabilities. We accomplish this by identifying the nearest neighbors for each modality and then determining whether the corresponding pair is within the top  $N$  nearest neighbors, where  $N \in \{1, 50, 200\}$ . Our experiments are conducted on two datasets: our holdout dataset from QUILT-1M and the ARCH dataset. Results are in Table 2.
Limitations. Despite the promising results, QUILT was curated using several handcrafted algorithms and LLMs. Such curation methods, while effective, introduce their own biases and errors. For instance, our histopathology classifier had occasional false positives ( $\approx 5\%$ ) confirmed by human evaluation. Occasionally, ASR can misinterpret a medical term and transcribe it as a different existing term, such as transcribing 'serious carcinoma' as 'serious carcinoma'. Unfortunately, such errors are not rectifiable using our current pipeline (see supplement for more details). While not directly a limitation of our dataset, training a CLIP model trained from scratch underperformed compared
Table 2: Cross-modal retrieval results on the QUILT-1M holdout set and ARCH dataset. In each cell, the results are displayed in the format (%/\%), with QUILT-1M holdout results on the left and ARCH results on the right. The best-performing results are highlighted in bold text.
model	config	Text-to-Image (%)	Image-to-Text (%)
R@1	R@50	R@200	R@1	R@50	R@200
CLIP	ViT-B/32|GPT/77	0.49/0.07	4.73/2.42	10.15/7.21	0.39/0.05	3.99/2.52	8.80/7.22
PLIP	ViT-B/32|GPT/77	1.05/0.56	10.79/13.10	21.80/29.85	0.87/0.74	11.04/13.75	21.63/29.46
QUIILTNET	ViT-B/32|GPT/77	1.17/1.41	16.31/19.87	31.99/39.13	1.24/1.35	14.89/19.20	28.97/38.57
CLIP	ViT-B/16|GPT/77	0.83/0.09	5.63/2.73	11.26/8.72	0.66/0.13	5.02/3.09	10.82/9.04
QUIILTNET	ViT-B/16|GPT/77	2.42/1.29	22.38/20.30	41.05/40.89	2.00/1.01	21.66/16.18	39.29/34.15
BiomedCLIP	ViT-B/16(224)|PMB/256	4.34/8.89	14.99/53.24	25.62/71.43	3.88/9.97	13.93/52.13	23.53/68.47
QUIILTNET	ViT-B/16(224)|PMB/256	6.20/8.77	30.28/55.14	50.60/77.64	6.27/9.85	31.06/53.06	50.86/73.43
to fine-tuning a pre-trained CLIP (see supplement for more details). This suggests that a million image-text pairs may still not be sufficient. Future works may explore other self-supervised objectives.
Data Collection and Societal Biases Aligning in strategies with [65], we release QILT derived from public videos, taking structured steps to limit privacy and consent harms (see supplement for more details). Complying with YouTube's privacy policy, we only provide video IDs, allowing users to opt-out of our dataset. Researchers can employ our pipeline to create QILT. Regarding societal biases, a significant portion of our narrators originate from western institutions, a situation that is further amplified by our focus on English-only videos. Consequently, QILTNET may exhibit inherent biases, potentially performing better on data associated with these demographics, while possibly underperforming when applied to other cultural or linguistic groups.
Conclusion. We introduced QUILT-1M, the largest open-sourced histopathology dataset to date. Empirical results validate that pre-training using QUILT is valuable, outperforming larger state-of-the-art models like BiomedCLIP across various sub-pathology types and tasks including zero-shot, few-shot, full-shot, and cross-modal retrieval. We established a new state-of-the-art in zero-shot, linear probing, and cross-modal retrieval tasks in the field of Histopathology.
Research reported in this study was supported by the National Cancer Institute under Awards No. R01 CA15130, R01 CA225585, and R01 CA201376 and the Office of the Assistant Secretary of Defense for Health Affairs through the Melanoma Research Program under Awards No. W81XWH-20-1-0797 and W81XWH-20-1-0798. Opinions, conclusions, and recommendations are those of the authors.
[1] M. Agrawal, S. Hegselmann, H. Lang, Y. Kim, and D. Sontag. Large language models are zero-shot clinical information extractors. arXiv preprint arXiv:2205.12689, 2022.
[2] M. Amith, L. Cui, K. Roberts, H. Xu, and C. Tao. Ontology of consumer health vocabulary: providing a formal and interoperable semantic resource for linking lay language and medical terminology. In 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 1177-1178. IEEE, 2019.
[3] A. Araujo, J. Chaves, H. Lakshman, R. Angst, and B. Girod. Large-scale query-by-image video retrieval using bloom filters. arXiv preprint arXiv:1604.07939, 2016.
[4] G. Aresta, T. Araújo, S. Kwok, S. S. Chennamsetty, M. Safwan, V. Alex, B. Marami, M. Prastawa, M. Chan, M. Donovan, et al. Bach: Grand challenge on breast cancer histology images. Medical image analysis, 56:122-139, 2019.
[5] H. B. Arunachalam, R. Mishra, O. Daescu, K. Cederberg, D. Rakheja, A. Sengupta, D. Leonard, R. Hallac, and P. Leavey. Viable and necrotic tumor assessment from whole slide images of
A Data curation models, algorithms and parsing pipelines
A.1 Curating QUILT: an Overview
Creating a densely annotated vision-language dataset from videos is a significant undertaking, as it involves various handcrafted algorithms and machine learning models. In the following sections, we present more detailed information about the challenges of the data curation pipeline and algorithms used to address these challenges. To download QUILT-1M and its metadata and access the code to recreate the dataset and trained models, refer to our website.
Collecting representative channels and videos. The first challenge lies in obtaining relevant histopathology videos. We used a set of keywords (obtained from online histopathology glossaries 7) to search for videos, resulting in  $\approx 65\mathrm{K}$  potential matches. Figure 6 shows the word cloud of all keywords used for searching YouTube. However, filtering histopathology content based on thumbnail and title yields many false positives, often including general pathology videos. To address this, we process the frames of lower-resolution versions of each video to differentiate between histopathology and pathology content, narrowing the selection to  $\approx 9\mathrm{K}$  videos.
Figure 6: Word cloud of all keywords used for searching YouTube
Filtering for narrative-style medical videos. Among the  $\approx 9\mathrm{K}$  videos, we sought videos with a "narrative style" where narrators freely explain whole slide images and streaks of similar frames occur, indicating an educational performance. To identify such content, we used a model that analyzed randomly sampled frames to determine if they maintained a consistent style over time. This process resulted in the selection of  $\approx 4\mathrm{K}$  videos. Non-voiced videos are also filtered by using inaSpeechSegmenter [17] where the video endpoint does not provide the video language or transcript. To identify the audio language of a video, we first check YouTube's API. If the information is unavailable through the API, we use OpenAI's Whisper model [47] on the first minute of audio from the video.
To identify videos containing medical content, we employ a keyframe extraction process with a specific threshold to determine the minimum visual change required to trigger keyframes. For a new video, the thresholds for keyframe extraction are determined by linearly interpolating between the lowest threshold, 0.008 (5-minute video) and the highest 0.25 (200-minute video). Following the keyframe extraction process, we utilize a histopathology image classifier to identify histopathology content within the extracted keyframes. See A.3 for more details. To identify narrative-style videos, we randomly select a min(num_of_histo_structure frames, 20) keyframes from a video and utilize a pre-trained CLIP  $^{8}$  (ViT-B-32) model to embed and compute a cosine similarity on the next three keyframes. If all three have similarity scores  $\geq$  a threshold of 0.9, we count the video as a narrative streak.
Text extraction using ASR and text denoising. Another challenge involves automatic speech recognition (ASR), as YouTube captions are often inadequate for medical vocabulary. To address this issue, we employed the Large-V2 open-source Whisper model [47] for speech-to-text conversion. However, general-purpose ASR models like Whisper can misinterpret medical terms, particularly
<sup>7</sup>https://lab-ally.com/historyopathology-resources/historyopathology-glossary
<sup>8</sup>https://huggingface.co/sentence-transformers/clip-ViT-B-32
Table 3: Salvable and Non-salvable cases for ASR correction using an LLM.
Error due to	Raw output	Salvagable
(beause LLM can rephrase and/or extract contextually similar correction)	Non-salvagable
(because the error losses all possible medical context and can lead to wrong entries)
Unfinetuned ASR	...look like the cranialomas I would expect in HP. They actually look more sarcoidal to me. The reason I say that is they, there's a kind of positive of inflammatory cells associated with them. They're really tight and well-formed. They're very easy to see a low power. And so HP is in the differential hypersensium nitose, but I would be more worried about sarcoidosis.	differential hypersen-sium nitose: hyper-sensitivity pneumoni-tis, cranialomas: granu-lomas	positive: paucity
LLM	high-larbidia-stinal lymphadenocathy
lymphin-giatic pattern distribution	returns hilar
lym-
phadenopathy instead of a more
appropriate mediastinal
lym-
phadenopathy	returns lymphatic pattern dis-
tribution instead of a more
appropriate lymphangitic pat-
tern distribution
Incomplete UMLS checker	...picnotic	-	LLM correctly returns py-knotic however, UMLS(2020) does not have the word py-knotic if fails to pass the UMLS check.
when the speaker's voice is choppy or accented. There are no straightforward trivial solutions due to: 1) the absence of openly available medical ASR models or data for fine-tuning in the medical domain; 2) the inadequacy of medical named entity recognition models in detecting transcription errors, because these models are typically trained on correctly spelled words; 3) the ineffectiveness of methods like semantically searching over a medical glossary, such as UMLS, which only prove effective when the erroneous text has significant similarity to the correct terms; and 4) the inability of simpler methods like finding the longest common substring, which might work in finding a match in the glossary/ontology for replacement, but cannot identify the wrong words/phrases in the first place. To rectify ASR errors, we employed UMLS (a knowledge database) and a LLM (GPT-3.5). This, however, introduces a new challenge of identifying incorrectly transcribed words and determining which words were mistakenly "corrected" and correctly formatted by the LLM after error correction and resolving unintended parsing errors [1]. See Figure 3 in the main paper for LLM prompt examples of ASR correction and medical and ROI text extraction from the corrected ASR text. Refer to Table 3 for error examples of ASR correction using the LLM.
Image frame extraction and denoising. The image processing aspect of this task adds to its complexity, as it requires static frame detection, quality control for frames, and histology magnification classification. Each model utilized it these steps introduces its own biases and errors. We extract time-intervals (chunks) from each video from which we extract representative image(s). For each of the extracted chunks  $(t_n, t_{n+1})$ , the static chunk detection algorithm 1 is used to extract sub-time-intervals with static frames within the chunk. If found, we save the median (in pixel space to prevent blurry outputs) of the stable frames, else (i.e. no stable duration of frames) we leverage the structural similarity index (SSIM) method on histopathology key-frames to find the most dissimilar histopathology image to make up the representative images for the chunk, essentially de-duplicating the frames. Figure 7 demonstrates this process.
Figure 7: Representative Frame Identification. If a Stable frame is found by Algorithm 1 within the candidate regions, we use it as the representative frame. If not, we use the most dissimilar frames among unstable frames.
Algorithm 1 Static Video Chunk Detection Algorithm
1: procedure DETECTSTATICFRAMES(video, starttime, endtime)  
2: video = video[startTime:endtime]  
3: fixedFrames ← ∅  
4: SSIMValidatedFrames ← ∅  
5: prevFrame ← first frame in video  
6: for frame ∈ rest of frames in video do  
7: absDiff ← absolute difference between frame and prevFrame  
8: absDiffThresh ← apply adaptive thresholding using a Gaussian filter to absDiff  
9: meanVal ← mean value of absDiffThresh  
10: if meanVal < 10 then  
11: fixedFrames ← fixedFrames ∪ frame  
12: else  
13: if length of fixedFrames ≥ minimum duration then  
14: subclip ← extract sub-clip of frames with constant background from fixedFrames  
15: for patch ∈ randomly selected patches in each frame of subclip do  
16: SSIMVal ← calculate SSIM of patch  
17: if SSIMVal > threshold then  
18: SSIMValidatedFrames ← SSIMValidatedFrames ∪ frame  
19: end if  
20: end for  
21: end if  
22: fixedFrames ← ∅  
23: end if  
24: prevFrame ← frame  
25: end for  
26: staticTimestamps ← extract start and end times from SSIMValidatedFrames  
27: return staticTimestamps  
28: end procedure
Aligning both modalities. The alignment of the images with their corresponding text requires the implementation of unique algorithms. These algorithms are designed to reduce duplicate content and ensure accurate mappings between image and text. See Figures 8 and 9 and Table 4 for a
a demonstration of image-text alignment process. See Figure 10 for sample images and their corresponding medical and ROI texts and the sub-pathology classification provided by the LLM.