SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Satoshi Tsutsui, Winnie Pang, Bihan Wen
{satoshi.tsutsui, winnie.pang, bihan.wen}@ntu.edu.sg
The examination of blood samples at a microscopic level plays a fundamental role in clinical diagnostics. For instance, an in-depth study of White Blood Cells (WBCs), a crucial component of our blood, is essential for diagnosing blood-related diseases such as leukemia and anemia. While multiple datasets containing WBC images have been proposed, they mostly focus on cell categorization, often lacking the necessary morphological details to explain such categorizations, despite the importance of explainable artificial intelligence (XAI) in medical domains. This paper seeks to address this limitation by introducing comprehensive annotations for WBC images. Through collaboration with pathologists, a thorough literature review, and manual inspection of microscopic images, we have identified 11 morphological attributes associated with the cell and its components (nucleus, cytoplasm, and granules). We then annotated ten thousand WBC images with these attributes, resulting in 113k labels (11 attributes x 10.3k images). Annotating at this level of detail and scale is unprecedented, offering unique value to AI in pathology. Moreover, we conduct experiments to predict these attributes from cell images, and also demonstrate specific applications that can benefit from our detailed annotations. Overall, our dataset paves the way for interpreting WBC recognition models, further advancing XAI in the fields of pathology and hematology.
The microscopic examination of human blood samples is essential in clinical diagnostics, providing valuable insights into a wide range of health conditions. For example, white blood cells (WBCs) or leukocytes can serve as markers for various blood-related diseases in pathology and hematology. Accurate recognition of basic WBC types (neutrophils, eosinophils, basophils, monocytes, and lymphocytes) forms a fundamental component of pathological diagnostic methods [1], used to identify various blood-related conditions such as leukemia and anemia [2]. Automating the cell recognition process can significantly enhance diagnostic efficacy, and as such, multiple datasets have been proposed for WBC categorization [3, 4, 5, 6, 7]. However, these existing datasets only annotate the cell types without any morphological characteristics of each cell, a key explanatory factor in how hematologists recognize WBCs. This lack of information may limit the development of explainable artificial intelligence (XAI) for cell image analysis, which is particularly significant in the medical domain where interpretability is crucial – clinicians cannot reliably diagnose serious illnesses without clear explanations of a model's conclusions.
In this paper, we introduce WBCAtt (Figure 1), a novel dataset for WBCs that is densely annotated with morphological attributes (Figure 2). Each cell image, obtained from the PBC dataset [6], is annotated with 11 attributes, which were determined through a comprehensive process involving discussions with pathologists, literature review, and manual inspection of cell images. One example
(a) Cell Structure
(b) Microscopic Image
Cell Size: Big Shape: Round
(c) Explanatory Attributes
Figure 1: We construct a new dataset annotating 11 morphological attributes of microscopic images of white blood cells. Our set of attributes, grounded on medical literature, and plays critical role when hematologists recognize cells. Our dataset can facilitate the XAI in blood cell recognition.
of such an attribute is the presence of small cytoplasmic holes known as vacuoles, as depicted in Figure 2-(t)(u). Vacuoles provide insights into the functional state of a cell and are typically observed in monocytes and neutrophils due to their phagocytosis mechanism [8]. However, vacuoles found in other cell types, like lymphocytes and eosinophils, may indicate certain disorders [9, 10]. Based on these references, we define the presence of cytoplasmic vacuoles as one of the attributes. In the end, we establish 11 attributes, each supported by at least one medical reference. These attributes are discussed based on their association with different cellular components: overall cell (Sec. 3.1), nucleus (Sec. 3.2), cytoplasm (Sec. 3.3), and granule (Sec. 3.4). We annotated a total of 10,298 WBC images with these attributes, resulting in 113k labels (11 attributes x 10.3k images). To the best of our knowledge, this is the first public dataset to include such an extensive set of annotations, addressing the current gap in XAI for WBC analysis. Moreover, we conduct experiments to explore the capability of standard deep learning models in recognizing these attributes (Sec. 4). Furthermore, we believe that our detailed attributes can improve the interpretability of machine learning models for recognizing WBCs and related blood disorders. To illustrate this, we showcase specific applications that can be developed using our newly-introduced dataset (Sec. 5).
In summary, our contributions are as follows. We construct the first public dataset for WBCs annotated with comprehensive morphological attributes, addressing the current gap in developing interpretable models for WBC analysis. We also conduct experiments to automatically predict attributes from images, in addition to outlining specific applications. We hope that our dataset will foster advancements in XAI in pathology and hematology.
White Blood Cell Recognition. WBC recognition is crucial for the diagnosis of various diseases in hematology, leading to numerous studies focusing on the development of automatic WBC classifiers [11, 12, 13, 14] and the release of publicly available datasets [3, 4, 5, 6, 7]. Prior to 2020, public WBC datasets were limited in size, containing only hundreds of images [4, 3, 5], which proved insufficient for leveraging state-of-the-art image classification models, such as Convolutional Neural Networks (CNNs). Recently, the introduction of larger datasets [6, 7], containing around ten thousand images, has enabled subsequent studies [11, 12, 13, 14] to develop novel deep learning models that improve classification performance. However, despite the inherent requirement for explainability in medical applications, no prior datasets annotate morphological attributes, which are essential explanatory factors when hematologists recognize WBCs. As such, our work aims to address this significant yet overlooked aspect.
Attribute Datasets. Many attribute datasets have played a pivotal role in various fields of computer vision, inspiring a diverse range of applications and fostering methodological advances. These include fashion (e.g., DeepFashion [15]) for e-commerce applications [16], human faces (e.g., CelebA [17]) for the security of facial recognition systems [18], animals (e.g., Caltech-UCSD Birds [19]) for zero-shot learning [20], and image aesthetics (e.g., AADB [21]) to investigate the inherent subjectivity of human artistic perceptions [22]. Attributes are often considered key interpretable elements in computer vision systems. For example, interpretable autonomous driving systems can be developed using explainable attributes [23, 24]. In medical domains, attributes of X-ray images [25, 26] and
(e) High
(f) Low
(i) Segmented multilobed
(g) Densely
(h) Loosely
(I) Unsegmented round
(m) Unsegmented Indented
(n) Irregular
(o) Clear
(p) Frosted
(q) Light Blue
(t) Yes
(u) No
(v) Pink
(w) Red
(x) Purple
(y) Yes
(z) No
(aa) Small
(bb) Round
(cc) Coarse
Figure 2: Sample images of each attribute. See Sec. 3 for descriptions.
skin disease images [27] have been annotated to support the development of interpretable medical AI, sharing a similar motivation with our work. However, no existing dataset provides attribute annotations for the morphological characteristics of WBCs despite their importance in hematology. Our work addresses this gap by presenting a densely-annotated dataset for WBC recognition.
3 WBCAtt: White Blood Cell Attribute Dataset
The attributes are categorized into four primary groups based on cell structure: overall cell, nucleus, cytoplasm, and granules, as described in Sec. 3.1-3.4, respectively. The examples of these attributes are summarized in Figure 2. We utilized all images of typical WBCs from the PBC dataset [6], which encompassed 1,218 basophils, 3,117 eosinophils, 1,420 monocytes, 3,329 neutrophils, and 1,214 lymphocytes. We annotated 11 attributes for these 10,298 images, resulting in 113,278 image-attribute pairs, with the distribution for each attribute shown in Figure 3.
Attribute Definition Process. Since there was no formally established set of attributes or ontology, we initiated our work with discussions with pathologists working in a laboratory at a healthcare company that develops digital cell imaging analyzers [28]. They provided five prominent attributes (see Coarse Morphological Attributes in Appendix) often used in identifying the five major WBC types, but also noted that the list was not exhaustive. Based on these initial keywords, we conducted a thorough review of relevant textbooks and research papers focusing on the morphological characteristics of WBCs. Subsequently, we refined the attribute set through further discussions with the pathologists and by manually inspecting approximately a thousand WBC images. This process yielded a total of 11 attributes, each supported by at least one medical literature reference. While the samples we inspected were the five major WBC types from healthy individuals, we expect the resulting attributes to sufficiently describe the morphological characteristics that may emerge in response to certain diseases, such as COVID-19 [29]. Moreover, these attributes provide pathologists with valuable insights into significant morphological abnormalities during microscopic examination, in accordance with the guidelines outlined in [30].
Annotation with Quality Control. To ensure reliable annotations across over  $10\mathrm{k}$  images, we devised a rigorous, iterative process involving pathologists, research scientists, and biomedical students who had strong knowledge of cell structures. In the initial stage, the students annotated the images, while being informed of the specific WBC category for each image. Following this, our research scientists meticulously examined each image and its assigned attributes, meaning that every image was inspected by at least two individuals. When ambiguities arose, we discussed with the pathologists who defined the attributes with us, ensuring a consensus on labeling. Further details on the quality control are available in the Appendix. We assessed the reliability of our annotation process by replicating it on a subset of 1,000 images with different annotators. Out of the 11,000 attribute annotations, 10,569 were consistent with the original annotations, giving an agreement rate of approximately  $10569 / 11000 \approx 96.1\%$ . This high agreement rate demonstrates the robustness and reliability of our annotation process.
Figure 3: The distribution of values per attribute. The distribution represents the results of annotating all typical WBCs from the PBC dataset, which is the image source we utilized. We did not actively control or manipulate the distribution. See Sec. 3 for the definitions of the attributes, and Figure 2 for example images. See Appendix for the extract numbers.
3.1 Cell
Cell Size. Cell size refers to the overall dimensions of a WBC. Generally, the size of a WBC can often be indicative of its function, maturation stage, and activation state. For differentiating between various WBC types, lymphocytes usually have a small cell size compared to other WBC types, such as monocytes and granulocytes [31]. Estimating cell size is typically done by comparing the WBC to neighboring red blood cells (RBCs; usually  $6 - 8\mu \mathrm{m}$ ) within the same blood sample, as RBCs provide a consistent reference for size comparison. A WBC is classified as big if its diameter is larger than twice the diameter of the RBCs. Examples of big and small cells can be found in Fig. 2-(a)(b).
Cell Shape. The cell shape of WBCs is a significant morphological feature that offers insights into the cell type. WBCs can display a range of shapes, from round to irregular, depending on the cell type and its interactions with the surrounding microenvironment, including red blood cells. Irregular shapes are more prevalent in neutrophils or monocytes due to their unique functions or maturation stages [32, 33]. Additionally, WBCs can be irregular in shape due to their interactions with adjacent red blood cells. In our definition, WBCs with circular or oval shapes are categorized as round, while any other shapes are classified as irregular. Examples of round and irregular cells are shown in Fig. 2-(c)(d).
3.2 Nucleus
Nucleus Shape. The shape of the nucleus can provide crucial information about the WBC types [34]. Segmented nuclei are typical characteristic of neutrophils and eosinophils, with neutrophils displaying a multilobed nucleus and eosinophils often exhibiting a bilobed nucleus [35]. Band-shaped nuclei can be found in immature neutrophils, also known as band neutrophils [36], which are the intermediate stage in the maturation process of segmented neutrophils. Unsegmented nuclei can be observed in lymphocytes and monocytes, with lymphocytes typically having a round nucleus and monocytes having an indented or kidney-shaped nucleus [31]. An "irregular" class is included to accommodate the nucleus shape that does not fall under the defined classes, often for the nucleus of a basophil or monocyte. In total, we have six nucleus shapes (Fig. 2-(i)-(n)): segmented-multilobed, segmented-bilobed, unsegmented-band, unsegmented-round, unsegmented-indented, and irregular.
Chromatin Density. The density of nuclear chromatin, which refers to the compactness of chromatin within the nucleus, is an important factor for distinguishing between different types of WBCs, such as lymphocytes and monocytes. In general, lymphocytes exhibit denser, heterochromatic nuclear chromatin, while the nucleus of monocytes appears as a "rough mesh", which contains more loosely packed, euchromatic chromatin [35, 37]. For the benefit of individuals who are not familiar with clinical terminology in using our annotations, we describe the chormatin density as loosely packed (euchromatic) and densely packed (heterochromatic), as illustrated in Fig. 2-(g)(h).
Nuclear cytoplasmic (NC) Ratio. The NC ratio refers to the proportion of the cell's volume occupied by the nucleus relative to the cytoplasm, and can provide valuable information regarding the cell type. A WBC with a high NC ratio is typically a lymphocyte, while a lower NC ratio is characteristic of other types. In this dataset, a WBC with an NC ratio greater than 0.7 [38] is considered to have a high NC ratio. This characteristic is often associated with a thin rim of cytoplasm surrounding the nucleus. Fig. 2-(e)(f) depicts examples of WBCs with low and high NC ratios.
3.3 Cytoplasm
Cytoplasm Color. The color of the cytoplasm can offer valuable information regarding the cell type, as different WBCs often exhibit varying cytoplasm colors due to differences in granule content and staining affinity. Furthermore, the cytoplasmic color can offer insights into granulocyte maturation, as it tends to vary across different stages of WBC development [39]. In light of this, we have annotated the cytoplasm colors ranging from light blue to purple blue [40], as shown in Fig. 2-(q)(r)(s).
Cytoplasm Texture. Cytoplasm texture also contributes to the classification of WBCs, as different cell types may exhibit unique textures due to variations in intracellular content, such as granules and other organelles. For example, tiny, dust-like purplish granules that sometimes appear in lymphocytes and monocytes give the cytoplasm a frosted or ground glass appearance [41] while the cells lacking these dust-like granules have a clear or transparent cytoplasm, as shown in Fig. 2-(o)(p).
Cytoplasm Vacuole. Discrete cytoplasmic vacuoles are often found in monocytes and sometimes in neutrophils due to their phagocytosis mechanism [8], where pathogens and cell debris are engulfed and digested. The presence of these vacuoles can help identify cell types and provide insights into the cell's functional state. Vacuoles found in other cell types, such as lymphocytes [9] and eosinophils [10], might be indicative of certain diseases and can aid in the diagnostic process. Examples of cells with and without a cytoplasmic vacuole are shown in Fig. 2-(t)(u).
3.4 Granule
Granularity. In this dataset, granularity means the presence of prominent stainable cytoplasmic granules [35] that distinguish between granulocytes (neutrophils, eosinophils, basophils) and agranulocytes (monocytes, lymphocytes). While granules are not entirely absent in agranulocytes, they are generally found in smaller quantities and are less noticeable compared to their presence in granulocytes [42]. Agranulocytes are categorized as having no granularity unless prominent granules are observed. Fig. 2-(y)(z) shows a granulocyte (granularity: yes) and an agranulocyte (granularity: no).
Granule Color. Granule color is a distinguishing factor among granulocytes. The granules within neutrophils, eosinophils, and basophils have different colors due to their distinct compositions, which reflect their unique immune functions. Neutrophil granules are typically pink, eosinophil granules are red, and basophil granules are purple [40, 42]. This attribute is particularly useful for distinguishing eosinophils from other granulocytes, as eosinophilic granules are uniquely stained red by eosin. This is due to the presence of cationic proteins within the eosinophilic granules, which bind to the eosin dye and give the cell its characteristic red color [43]. Fig. 2-(v)(w)(x) shows examples of them.
Granule Type. Granule type describes the morphological characteristics of granules in granulocytes. Neutrophils, eosinophils, and basophils each have their roles in the immune response, possessing distinct granule types filled with different substances. Neutrophils contain small, fine granules that are packed with antimicrobial proteins and enzymes, such as myeloperoxidase, lysozyme, and lactoferrin. The granules of the eosinophils are usually round and membrane-bound that compose of major basic protein. Basophils, on the other hand, possess conspicuous and coarse granules filled with histamine, heparin, and other mediators of inflammation [35]. Fig. 2-(aa)(bb)(cc) shows examples of them.
4 Attribute Prediction Experiments
While various applications are possible with our attributes (Sec. 5), a preliminary question emerges: How well do standard visual recognition models perform in predicting these attributes? This is a crucial question because if the model cannot even recognize these attributes, we cannot reliably develop interpretable models on top of it. To investigate this, we conduct experiments to predict the 11 attributes from WBC images.
Data Split. We randomly divided the dataset into 6,179 training images, 1,030 validation images, and 3,099 test images. The random division ensures that the cell-type distributions are the same in each set. The exact split is available in Appendix.
Model. Our baseline model predicts attribute values from image representations extracted by an image encoder of convolutional neural networks. We employ an ImageNet-pretrained ResNet50 as our choice of image encoder. The attribute prediction model comprises multi-task heads of per-attribute linear layers, with each layer corresponding to a specific attribute. We intentionally keep the model simple and provide a foundation for future work, such as designing more complex model architectures or integrating domain knowledge of WBCs (e.g., cell structure). The further details about the model (including other backbones) and its training is in Appendix and the source code.
Evaluation Metrics. Due to the imbalance of attribute values, we use macro F-measure, which calculates the harmonic mean of precision and recall, instead of plain accuracy for the evaluations. We run the same code three times with different seeds and report  $95\%$  confidence intervals.
Results. We present the results in Table 1. The baseline model achieves an average macro F-measure of  $91.20 \pm 0.06\%$ . Some attributes, such as granularity, granule type, and granule color, exhibit particularly high F-measures of over  $98\%$ . On the other hand, nucleus shape is the most challenging attribute to predict, with the lowest F-measure of  $76.13 \pm 0.59\%$ , which could be due
Table 1: Macro F-measure (%) for Attribute Prediction.
Cell Size
83.81 ± 0.33	Cell Shape
90.66 ± 0.36	Nucleus Shape
76.13 ± 0.59	Nuclear Cytoplasmic Ratio
96.35 ± 0.06
Chromatin Density
86.39 ± 0.32	Cytoplasm Vacuole
89.57 ± 0.47	Cytoplasm Texture
94.49 ± 0.51	Cytoplasm Color
87.99 ± 0.47
Granule Type
99.44 ± 0.07	Granule Color
98.76 ± 0.08	Granularity
99.61 ± 0.02	(Average)
91.20 ± 0.06
(a) Cytoplasm vacuol
e GT: Yes Pred: Yes
(b) Cell shape GT: R
Sound Pred: Irregular
(c) Cell size GT: Big Pred: Small
Figure 4: Examples of correct and incorrect predictions with Grad-CAM. (a) Vacuoles are correctly highlighted. (b) The model incorrectly identifies leaked cellular substances as a part of the cell. (c) The model overlooks the WBC and focuses on the red blood cell instead.
Monocyte
Cytoplasm color: Blue Cytoplasm vacuole: Yes
(i) Cells from COVID-19 Patient
Figure 5: Examples of correct and incorrect predictions when the predictor is applied for cell images beyond our dataset: (i) Covid-19 Patients, (ii) Bone Marrow Cells, and (iii) Animal Cells. While not all attributes can be predicted correctly, especially those sensitive to staining, our attribute predictor can still recognize key attributes. See Broader Applicability in Sec. 4 for further details.
Monocyte
Nucleus shape: Unsegmented-identented Cytoplasm color: Purple blue (ii) Bone Marrow Cells
Neutrophil
Granule color: Pink
Granule type: Small
(iii) Animal Cells
to the complexity of the nucleus shape. Figure 4 highlights some prediction results along with Grad-CAM[44] to visualize the areas the model focuses on. More examples are reported in Appendix.
Broader Applicability. Although we established attributes based on typical WBCs from healthy human individuals, we anticipate that our attribute definitions can be applied in diverse contexts. For example, Zini and d'Onofrio [29] discuss the morphological features of WBCs in COVID-19 patients, which align with the attributes we have established. To explore how well our classifier can recognize them, we briefly inspected a small number of images involving cells from COVID-19 patients [29], bone marrow samples [45], and swine (a non-human species) [46]. While the attributes are still applicable, we observe that the cytoplasm color, which is sensitive to staining conditions, sometimes cannot be predicted correctly. As we can see in Figure 5 in comparison to Figure 2, colors look different. Nonetheless, we find that attributes less sensitive of colors (e.g., cytoplasm vacuole) can be predicted correctly for the majority of the cases. We discuss this further in Appendix.
One immediate practical application is to automatically recognize morphological features from cell images, which is investigated in Sec. 4. The attribute recognition model can be incorporated into software that automatically analyzes cell morphology [28], which provides valuable assistance in clinical diagnostics, as certain morphological features of WBCs may indicate specific diseases or conditions [47, 48]. It can also assist hematologists in searching for cells with specific morphological characteristics within massive datasets that would be impractical to examine manually. Beyond the cell analyzer, we demonstrate three applications that can contribute to XAI.
Figure 6: (a) As discussed in Sec. 5.1, our dataset enables training a cell type classifier solely based on attributes. We can ask questions like, "What would be the predicted cell type if this cell had purple granules instead of pink?" (b) We can retrieve corresponding images as counterfactual examples.