Large-Scale Annotation of Histopathology Images from Social Media

Large-scale annotated image datasets like ImageNet and CIFAR-10 have been essential in developing and testing sophisticated new machine learning algorithms for natural vision tasks. Such datasets allow the development of neural networks to make visual discriminations that are done by humans in everyday activities, e.g. discriminating classes of vehicles. In computational pathology, such machine learning algorithms are applied to the highly specialized vision task of diagnosing cancer or other diseases from pathology images. Importantly, labeling pathology images requires pathologists who have had decades of training, but due to the demands on pathologists’ time (e.g. clinical service) obtaining a large annotated dataset of pathology images for supervised learning is difficult. To facilitate advances in computational pathology, on a scale similar to advances obtained in natural vision tasks using ImageNet, we leverage the power of social media. Pathologists worldwide share annotated pathology images on Twitter, which together provide thousands of diverse pathology images spanning many sub-disciplines. From Twitter, we assembled a dataset of 2,750 images from 1,576 Tweets from 13 pathologists from 8 countries; each message includes both images and text commentary. To demonstrate the utility of these data for computational pathology, we apply machine learning to our new dataset to test whether we can (i) accurately identify different stains, (ii) discriminate between five tissue types, and (iii) differentiate nontumoral, benign/low grade malignant potential [low grade], and malignant diseases. Using a Random Forest, we report (i) 0.960 ± 0.012 Area Under Receiver Operating Characteristic [AUROC] when differentiating between human hematoxylin and eosin [H&E] stained microscopy images from all other types of images e.g. natural scenes, and (ii) 0.996 ± 0.003 AUROC when distinguishing H&E from immunohistochemistry [IHC] stained microscopy images. Though a Support Vector Machine found color features to be important, a Random Forest surprisingly found texture features to be important, for these stain tasks. Additionally, we distinguish all pairs of breast, dermatological, gastrointestinal, genitourinary, and gynecological pathology tissue types, with mean AUROC for any pairwise comparison ranging from 0.783 to 0.873. We report 0.803 ± 0.059 AUROC when all five tissue types are considered in a single multiclass classification task. Finally, for our most difficult and clinically relevant task of distinguishing low grade from malignant tumors, we report 0.703 ± 0.058 AUROC, which marginally drops to 0.683 ± 0.056 for the 3-class classification task of distinguishing nontumoral diseases, low grade tumors, and malignant tumors. We hope this inspires other groups to use our dataset, to (i) improve performance, (ii) build upon our definition of nontumoral, low grade, and malignant in terms of diagnosis text keywords, or (iii) use our data as an independent test set for nontumoral, low grade, and malignant disease classification tasks. We provide a tool, called the Interactive Pathology Annotator, for pathologists and data scientists to browse, search, and validate the dataset. Our goal is to make this large-scale annotated dataset publicly available for researchers worldwide to develop, test, and compare their machine learning methods, an important step to advancing the field of computational pathology.