SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 8: Overview of use of timing and keywords for Alignment Images within a video chunk, i.e  $\{\mathrm{A},\mathrm{B},\mathrm{C}\}$ ,  $I_{n}$  at  $t_n$  are aligned with medical texts extracted within the same chunk. The raw_keys within each example chunk is colour coded to illustrate matches with keywords extracted from the medical texts and only matching keywords allow for the pairing of texts containing said keywords to image frames with frame-times around raw_keys times.
Figure 9: Video Chunking algorithm illustrate. With each transition tag explained in Table 4, we leverage predicted histopathology frames at times  $/t_1, \dots, t_n/$  to segment videos into chunks.Chunks at are minimum are  $T_P$  in duration, this value is estimated based on the word-per-second of the video with a minimum of 20 words being captured per chunk. Images within a chunk, unlike texts, are not overlapping with other chunks. Text overlap is done to provide needed context for LLM text correction and extraction.
A.2 Other data sources
A.2.1 PubMed Open Access Articles
We searched the PubMed open-access from 2010 - 2022 with keywords (pathology, histopathology, whole-slide image, H&E, and 148 keywords from a histopathology glossary<sup>9</sup>). We utilized Entrez<sup>10</sup> to retrieved the top 10,000 most relevant articles for each keyword. This query yielded 109,518 unique articles with PMCIDs. We extracted 162,307 images and their corresponding captions. Using our histopathology classifier and cropping multi-plane figures as described in A.4, we extracted 59,371 histopathology image and caption pairs with an average caption length of 54.02 tokens. Figure 11 demonstrates the pipeline of collecting data from PubMed.
<sup>9</sup>https://lab-ally.com/historyopathology-resources/historyopathology-glossary
10http://www.ncbi.nlm.nih.gov/Entrez
Table 4: All 6 (six) transition states for chunking narrative style videos.  $p(H)_{t_n}$  is the binary histo image classifier prediction at the current frame's time  $t_n$  and  $p(H)_{t_{n-1}}$  is the prediction at next frame's time  $t_{n-1}$ , where  $T_R$  is the cumulative running time and  $T_P$  is the estimated minimum chunk time for the video, determined by the words per second of the video. Text and image chunks are implemented as an ordered list of time intervals and image indexes.
P(H)@tn	P(H)@tn-1	tn-tn-1>TP	Tr>Tp	Text chunk	Image chunk	Tag
0	0	-	-	-	-	A
0	1	-	-	end = tn; append(s, e); reset	append index to chunk state, if state is empty append prior in dex; reset state	B
1	0	-	-	start = max(tn-1, tn-Tp)	append index to chunk state	C
1	1	1	-	end = tn; append(s, e); reset state; start = tn-Tp	append index to chunk state; reset state	D
0	1	end = tn; append(s, e); reset state; start = tn-Tp	append index to chunk state; reset state	E
0	-	append index to chunk state	F
A.2.2 Histopathology Image Retrieval from LAION
The Large-scale Artificial Intelligence Open Network (LAION-5B) [52] curated over 5 billion pairs of images and text from across the Internet, including a substantial volume of histopathology-related data. We tapped into this resource by retrieving the 3000 most similar LAION samples for each of the 1,000 pairs of images and text sampled from PubMed and QUILT, using a CLIP model pre-trained on the LAION data. The retrieval process utilized both image and text embeddings, with cosine similarity serving as the distance metric. Subsequently, we eliminated the duplicate images and removed all non-English pairs from the remaining pairs using LangDetect<sup>11</sup>. Consequently, the process yielded 22,682 image and text pairs.
A.2.3 Twitter Data from OpenPath
We utilized a list of tweets curated by Huang et al. [24] which totaled up to 55,000 unique tweets and 133,511 unique image-text pairs. This exhibits a one-to-many relationship that leans towards the image side, differentiating our work from the OpenPath approach, where we had one image matching with multiple captions (as in the case of MS-COCO captions). In order to maintain comparability with OpenPath, we followed their text pre-processing pipeline given in [24].
A.3 Histopathology and Magnification classifier
We use an ensemble of three histopathology image classifiers. To ensure robustness, our ensemble approach consists of two small Conv-NeXt models [38] and one linear classifier fine-tuned with DINO features [11]. This combination is necessary due to the homogenous appearance of histopathology images and the risk of false positives from similar pinkish-purple images. One Conv-NeXt model is trained in detecting non-H&E Immunohistochemistry (IHC) stained tissue images, while the other models are trained to handle all IHC stains and tissue types. The training data includes eight subgroups of the TCGA WSI dataset and a mix of general-domain images, PowerPoint (slide) images, and scientific figure datasets. See Table 5 for details of these datasets.
For the magnification classifier, we finetune a pretrained ConvNeXt-Tiny model [38], with standard preset hyperparameters for a few epochs and select the best performing model on the validation set. To generate a training set for the magnification model, TCGA subsets were segmented into patches using a method similar to [64]. These patches were generated at various magnifications, which were then categorized into three labels: 0:{1.25x, 2.5x, 5x, 10x}, 1:{20x}, 2:{40x}. The TCGA subsets were chosen to ensure a diverse representation of tissue morphologies and cancer types, thereby ensuring robust and comprehensive model training. The model was also trained on cytopathology microscopy images and various IHC stains beyond H&E to enhance the model's generalizability
<sup>11</sup>https://github.com/fedelopez77/langdetect
Image	Medical TEXT	ROI Text	Sub-pathology Classification
['There are clusters of cells with micro-follicular formations.'] 'Nuclear pseudo-inclusions, oval nuclei, nuclear grooves, and small nucleoli are present in some cells.']	['clusters of cells', 'micro-follicular formations', 'nuclear pseudo-inclusions', 'oval nuclei', 'nuclear grooves', 'small nucleoli']	['Endocrine', 'Cytopathology', 'Head and Neck']
['Cluster of macrophages and T cells is characteristic of acute rheumatic fever.'] 'Aschoff body is a characteristic feature of acute rheumatic fever.'] 'Macrophages with elongated chromatin are called Anitchkow cells and are commonly seen in Aschoff bodies.'] 'Pancarditis with Aschoff bodies is present.']	['Cluster of macrophages and T cells', 'Aschoff body', 'Macrophages with elongated chromatin', 'Anitchkow cells', 'Pancarditis']	['Cardiac', 'Hematopathology', 'Endocrine']
['An 80-year-old man has a scar-like plaque on the scalp that has been called malignant on a biopsy.'] 'The tissue affected by the plaque extends from the epidermis to the galea aponeurotica, near the periosteum of the skull.'] 'The skin, dermis, and subcutis are all affected by the process.']	['scar-like plaque on the scalp', 'malignant on a biopsy', 'skin, dermis, and subcutis affected by the process']	['Dermatopathology', 'Soft tissue', 'Hematopathology']
['Inflammatory cells surrounding cartilage can indicate acute chondritis, with neutrophils being the principal cell type.'] 'Chronic chondritis may be diagnosed if lymphocytes are the predominant inflammatory cell type.']	['cartilage', 'inflammatory cells']	['Hematopathology', 'Bone', 'Dermatopathology']
['Large histiocytes with abundant cytoplasm identified as Rosai-Dorfman histiocytes.'] 'S100 stain showed perivascular cuffing.'] 'Initial diagnosis of inflammatory pseudotumor of the orbit.'] 'Rosai-Dorfman disease may burn out and leave behind fibrotic pockets.']	['Large histiocytes', 'perivascular cuffing', 'fibrotic pockets']	['Dermatopathology', 'Soft tissue', 'Hematopathology']
['Epidermal acanthosis and papillomatosis resembling a wart or seborrheic keratosis.'] 'Presence of large sebaceous glands that drain directly through their duct out to the skin surface, which is abnormal.'] 'Presence of a demodex mite.']	['Epidermal acanthosis and papillomatosis', 'large sebaceous glands', 'demodex mite']	['Dermatopathology', 'Soft tissue', 'Hematopathology']
['Histological description of glandular tissue with little atypia but located in a place where it does not belong can be a helpful criteria to discern the presence of malignancy.'] 'Glands located on the periphery and infiltrating into adventitia and peripancreatic tissue may be malignant.']	['glandular tissue', 'pancreas']}	['Gastrointestinal', 'Pancreatic', 'Hematopathology']
Figure 10: A collection of sample images from our dataset, accompanied by corresponding medical text, ROI text, and the top three sub-pathology classifications derived from the ASR text using the LLM.
across different conditions. Only the ACROBAT and TCGA datasubsets are preprocessed to divide the WSIs into patches at various scales.
A.4 Support Models, Ontology Databases and Algorithms
This section describes the support models, ontology databases and handcrafted algorithms utilized within our pipeline for both searching and parsing our data.
Ontology databases. We employ various ontologies, both specific to histopathology and general ones. Among them are OCHV [2], FMA [42], BCGO  $^{12}$ , NCIT [19], MPATH [51], HPATH [62], and CMPO [29]. These ontologies serve a dual purpose. First, we used histopathology-specific ontologies (HPATH, MPATH, BCGO, and CMPO) to provide words/phrases to condition the LLM, enabling it to identify incorrect words. Second, all ontologies, in conjunction with UMLS, are used to obtain terms or phrases for validating the output of the LLM.
$^{12}$ https://bioportal.bioontology.org/ontologies/BCGO
Figure 11: (a) Search PubMed open access database, filter based on keywords, date, language and sort by relevance. (b) Download paper and media for each search result. (c) Extract and pair figures and captions. (d) Separate multi-plane figures, find histopathology images and their magnification. (e) Final result.
Table 5: Datasets used to train the histopathology image classifier. [μm per pixel - MPP]
Data Source	Subset	#WSI	#pathces	Train-Test	Magnification	Image-size
TCGA (H&E Stain)	GBM	19	89,022 - 40x
LUSC	20
LIHC	20	57,671 - 20x	384 x 384
SARC	23	169,431	84715-16943
ACROBAT Weitz et al. [61]	KIRC	16	16,660 - 10x
KICH	4	4,748 - 5x
BRCA	17	1,465 - 2.5x
SKCM	19	466 - 1.25x
H&E KI67	99	50589	28105-22484	(10x, 5x, 2.5x)	384 × 384
ER , PGR , HER2
BCI Liu et al. [36]	-	-	4,870	20x (0.46 MPP)	1024 × 1024
CCESD Liu et al. [35]	-	-	686	100x/400x	2048 × 1536
Smear Hussain et al. [25]	-	-	963	400x	2048 × 1536
Celeb Liu et al. [37]	-	-	202,599	8,103-1,944	-	-
Places Zhou et al. [69]	-	-	36,550	2,109-1,372	-	-
AI2D Kembhavi et al. [32]	-	-	4,903	0.7-0.3%	-	-
DocFig Jobin et al. [27]	-	-	33,004	0.8-0.2%	-	-
SciFig-pilot Karishma [30]	-	-	1,671	0.8-0.2%	-	-
SlideImages Morris et al. [40]	-	-	8,217	0.8-0.2%	-	-
TextVQA Singh et al. [54]	-	-	28,472	0.8-0.2%	-	-
SlideShare-1M Araujo et al. [3]	-	-	49,801	0.8-0.2%	-	-
Sub-pathology types. The list of all 18 sub-pathology types used to prompt LLM on the text classification task are: Bone, Cardiac, Cyto, Dermato, Endocrine, Gastrointestinal, Genitourinary, Gynecologic, Head and Neck, Hemato, Neuro, Ophthalmic, Pediatric, Pulmonary, Renal, Soft tissue, and Breast Histopathology. Figure 12 provides the LLM prompt to retrieve the top three sub-pathology classification based on a given text.
Pre-processing multi-plane figures. Many figures in academic papers are multi-plane, which means a number of sub-figures (Charts, graphs, histopathology and non-histopathology sub-figures) are placed next to each other to make a larger figure. We extracted individual images from multi-plane figures to create multiple instance bags. To locate boundaries and white gaps between sub-figures, we utilized Sobel filters. Binary thresholding was then used to find the contours surrounding the sub-figures. We employ image size and image ratio thresholds to remove undesirable sub-figures and our histopathology classifier to maintain just histopathology sub-figures. We supply the histological sub-figures individually for this type of figure by appending "[0-9]+" to the end of the multi-plane figure id. If a figure is divided into more than 5 sub-figures, we preserve the original image to ensure that the resolution of these sub-figures remains reasonable. Figure 13 shows an overview of this pre-processing step in different scenarios of successful and unsuccessful crops.
System Prompt:
You are a histopathology text classifier
User Prompt:
Imagine you are a text classifier. Classify the given text into one of the following surgical pathology types namely: Bone, Cardiac, Cytopathology, Dermatopathology, Endocrine, Gastrointestinal, Genitourinary, Gynecologic, Head and Neck, Hematopathology, Neuropathology, Ophthalmic, Pediatric, Pulmonary, Renal, Soft tissue, Breast pathology. Output only the top 3 pathology types in an ordered python list
Few-shot examples:
"Radicular cyst arises within the periodontal ligament space, particularly the periapex from the epithelial cell of malassez. These radicular cysts are caused by inflammation following the death of the pulp extending into the periapical radix. Radicular cysts caused by inflammation are always associated with a non vital tooth."
"[Soft tissue', 'Dermatopathology', 'Hematopathology']"
Figure 12: Prompting LLM with few-shot examples to extract the top three sub-pathology classification of a given text.
"There is a lesion with slight thickening of the muscularis mucosa and submucosa. There is a subtle change in the lamina propria that doesn't look quite like normal stromal cells. Description of slight thickening of the muscularis mucosa and submucosa with subtle changes in the lamina propria. Highlighted field shows the changes more dramatically. Abnormal cells in the lamina propria that appear pink and spindly."
OUTPUT: ['Gastrointestinal', 'Soft tissue', 'Hematopathology']
Figure 13: (a), (b), and (c) successfully cropped sub-figures where histopathology images (green box) are kept and non-histopathology (red box) images are removed. (b) histopathology crops are kept as not separated because the individual crops don't meet the size threshold so the original figure is kept. (d) Unsuccessful crop due to minimal gap between sub-figures. Original image is stored.
A.5 Privacy preserving steps
In order to ensure privacy while handling the dataset, several steps were taken to protect sensitive information. These steps include:
- Reduction of Signal to Noise using a LLM: To protect the privacy of the dataset, a LLM was utilized to reduce the signal-to-noise ratio. By applying the LLM, irrelevant or sensitive information was masked or removed.
- Exclusion of Videos Not Fully in Narrative Style: Videos that did not adhere to a fully narrative style were intentionally left out of the dataset. This step was taken to minimize the risk of including any potentially private or sensitive content that could compromise individuals' privacy.
- Release of Video IDs and Reconstruction Code: Instead of directly releasing the complete dataset, only video IDs from YouTube were made public. Additionally, the code is provided to enable researchers to recreate the dataset.
- Collection from Diverse Channels: Data collection was performed from a wide range of sources, including both large and small channels. This approach aimed to decrease the risk of overfitting to specific channel types, thereby mitigating privacy concerns associated with potential biases linked to particular channels.