SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We established the attributes by analyzing five major types of WBCs derived from peripheral blood samples of healthy human individuals. Nevertheless, our attribute definitions are applicable in broader contexts. Moreover, we anticipate that the attribute predictor trained on our dataset will demonstrate generalizability to other domains in the majority of cases. However, formally exploring this aspect requires constructing datasets and conducting rigorous evaluations, which constitute future work. As a preliminary step, we conducted small-scale case studies to evaluate the applicability of the attribute predictor to cell images beyond our dataset. Specifically, we manually examined a small number of cell images of (i) peripheral blood samples from COVID-19 patients [29], (ii) bone marrow instead of peripheral blood [45], as well as (iii) peripheral blood samples from a non-human species, namely juvenile Visayan warty pigs [46].
(i) Cells from COVID-19 Patients. As shown in Figure 14-(i), the majority of predictions made by our model, which was trained on healthy peripheral blood cells, are consistent with the descriptions provided in [29] (listed at the bottom of respective image, in italic). However, there was one exception, as the prediction related to cytoplasm color in Figure 14-(i)(a) was inaccurately determined. This discrepancy could be attributed to the difference in staining compared to that of our training data.
(ii) Cells from Bone Marrow. As illustrated in Figure 14-(ii), our attribute predictor successfully identified the attributes that related to the cytoplasm and granules, showcasing its effectiveness in investigating blood cells in bone marrow samples. However, incorrect predictions were observed for cell and nucleus-related attributes, such as cell size and nucleus shape. The poor prediction
Figure 13: Potential factors contributing to incorrect attribute predictions. Please refer to Appendix 7.6 for more detailed descriptions. Correct predictions are highlighted in green, while incorrect predictions are highlighted in red.
performance in cell size may be due to the nature of the bone marrow dataset, which crops the WBCs at a higher magnification level. (To observe the difference in magnification levels, compare the red blood cells in Figure 14-(ii) to Figure 14-(i) and Figure 14-(iii).) Since we define the size of a cell based on its relative size compared to the red blood cells, all WBCs in Figure 14-(ii) are small cells, even though they occupy a larger number of pixels in the image. This could also mean that our model did not actually learn the size in the way we defined it; it may be simply checking the absolute size in the image rather than checking the size relative to the red blood cells. Exploring this further is future work.
(iii) Cells from Pigs. To investigate the applicability beyond human blood samples, we explored animal blood smears, as shown in Figure 14-(iii). We observe that their staining and smear preparation methods, which are different from the PBC dataset, have a substantial impact on the prediction performance, particularly for color-related attributes. In Figure 14-(iii)(b), for instance, the model fails to predict the presence of eosinophil granules due to significant differences compared to eosinophils in other datasets. (Please refer to Figure 14-(ii)(c) and Figure 13-(iv)(a) for the granule color of eosinophils in the bone marrow dataset and the PBC dataset, respectively.) Moreover, the nucleus and granules of Figure 14-(iii)(a) is hardly recognizable, even to human observers. This difficulty in recognition leads to incorrect predictions for nucleus shape.
7.8 Details of Attribute-based WBC Classifier in Section 5.1
We simply use the predicted probability (ranging from 0 to 1) of each attribute value to predict the cell types. This (lazy) formulation can cause the multicollinearity problem, so we employ the L1 regularizer. The train/val/test split is the same as our dataset split. The specific linear classifier we use is sklearn.linear_model.SGDClassifier(max_iter=1000, loss="log_loss", penalty="l1") in scikit-learn<sup>1</sup>.
<https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGClassifier.html>
Figure 14: The prediction results of our attribute predictor were evaluated for characterizing cells obtained from: (i) peripheral blood smears of patients with COVID-19, (ii) bone marrow smears, and (iii) peripheral blood smears of animal (juvenile Visayan warty pigs). The descriptions of (i)(a) and (i)(b) were summarized from [29]. Correct predictions are highlighted in green, while incorrect predictions are highlighted in red.
7.9 Details of StyleGAN-based Image Editing in Section 5.2
We trained StyleGAN- $\nu 2^2$  [51] and subsequently pixel2style2pixel  $(\mathrm{pSp})^3$  [68] encoder. These models enable us to embed cell images into the latent space of the trained StyleGAN, allowing for GAN inversion. Theoretically, this approach allows us to embed any cell image into the GAN latent space. By manipulating the latent space, we were able to edit the generated images, which is shown in the main paper. Specifically, we explored two techniques: GANSpace[52], which applies Principal Components Analysis (PCA) to the latent space, and StyleSpace[53], which can identify the subspace of the GAN latent space corresponding to certain attributes based on example images. Using these methods, we discovered that a specific principal component can control the cell size and NC ratio, as depicted in Figure 7 in the main paper. Additionally, although not presented in the main paper, we found subspaces to edit nucleus shape (Figure 15-(a)) and cytoplasm texture (Figure 15-(b)). We acknowledge that we did not rigorously evaluate controllability and image quality of these two techniques, as our primary aim was to showcase the usability of our attribute dataset. The code used for these experiments was obtained from the authors of the referenced papers (see the corresponding footnotes).
Figure 15: Applying StyleSpace [53] on our attribute annotation, we discovered the way to control the nucleus shape and cytoplasm-texture.
7.10 Details of APL Detection in Section 5.3
We used the images provided by the dataset [59] and followed their proposed data split except that we removed the duplicated and corrupted images that we identified. We publicly reported the  $\text{duplicates}^6$  and the  $\text{corruptions}^7$ , but we have not received any comments thus far. Once ResNet50 was trained to classify APL or non-APL, we applied our attribute predictors to the promyelocyte images in the test set. Subsequently, we visualized the probability distribution of each attribute for the four groups (true positive, false negative, true negative, and false positive). Upon manual investigation of all attributes, we discovered a significant bias in the blue cytoplasm, as illustrated in Figure 8-(a)(b) in the main paper. It is worth noting that the inherent morphological differences between APL and non-APL promyelocytes may lead to distinct attribute distributions. In other words, if an attribute is a crucial feature for diagnosing APL, the bias is not necessarily undesirable. However, in this specific case, we found that the correlation only existed in the training set, and we could not find any literature supporting this bias. Consequently, we concluded that the correlation was spurious.
7.11 Attribute-based Acute Lymphocytic Leukemia Prediction
Our dataset provides annotations solely for images of healthy patients. However, we anticipate that the attributes we've identified can be exploited for downstream tasks with direct clinical implications, such as certain disorders. In this context, we investigate the potential use of these attributes for the detection of Acute Lymphocytic Leukemia (ALL), which is a significant challenge in the field of pathology. We employ the ALL-IDB [4] dataset, specifically the ALL-IDB2 subset, which includes cropped cell images. We follow the train/val/test split and baselines reported by Genovese et al. [69], who are from the same research group that developed the ALL-IDB dataset.
$^{2}$ https://github.com/NVlabs/stylegan3
3https://github.com/eladrich/pixel2style2pixel
<sup>4</sup>https://github.com/harskish/ganspace
<https://github.com/betterze/StyleSpace>
<sup>6</sup>https://github.com/sidhomj/DeepAPL/issues/32
<sup>7</sup>https://github.com/sidhomj/DeepAPL/issues/31
Table 2: Results for Predicting Acute Lymphocytic Leukemia (ALL) on the ALL-IDB [4].
Method	Accuracy (%)
VGG16	87.54 ± 3.15 [69]
ResNet18	88.69 ± 2.67 [69]
ResNet18 Pre-trained on Additional Histopathological Images	97.92 ± 1.62 [69]
ViT-Base	91.28 ± 2.17
Logistic Regression on Predicted Attributes	81.03 ± 1.48
Gradient Boosting on Predicted Attributes	83.85 ± 1.88
Logistic Regression on Attributes Predicted with Domain Adaptation	86.92 ± 2.13
Gradient Boosting on Attributes Predicted with Domain Adaptation	90.77 ± 1.23
Due to the lack of attribute annotations for the ALL-IDB dataset, we employ our model, trained on our dataset, to predict attributes. We utilize the ViT-Base backbone for this task, as our preliminary examination of a small number of cases indicates that the predicted attributes are more accurate than those from other backbones. This observation aligns with previous research [70], which suggests that ViTs typically yield higher cross-dataset accuracy than CNNs. We also implement a straightforward domain adaptation technique to concurrently minimize the classification loss and the Maximum Mean Discrepancy (MMD) [71] loss between the labeled domain (our dataset) and the unlabeled test domain (the ALL-IDB dataset).
After obtaining the predicted attributes on ALL-IDB images, we trained two types of binary classifiers for ALL detection: L1-regularized multiclass logistic regression and a gradient boosting classifier. This is done using the predicted attributes with and without domain adaptation. We ran the code three times with three different seeds and reported the mean and  $95\%$  confidence intervals. We compared these results with the image-based classifiers using ViT-Base, as well as ResNet18 and VGG16 (the two baselines reported in [69]).
Table 2 summarizes the results. The accuracies of attribute-based prediction methods are comparable to those of image-based prediction methods, suggesting the feasibility of using our attributes as interpretable features for clinical downstream tasks.
7.12 Peer Loss for Potential Annotation Noise
In response to a suggestion from a reviewer, we have explored the peer loss [72], which is robust against label noise and does not require to specify the noise rate. We conducted the following experiments using the ResNet50 backbone and executed the code three times to obtain the mean and the  $95\%$  confidence intervals.
Table 3: Impact of Peer Loss on Average Macro F1 Score
Dataset	Utilize Peer Loss?	Average Macro F1 Score (%)
Training Set with 10% Label Noise	No	85.31 ± 0.24
Training Set with 10% Label Noise	Yes	88.86 ± 0.02
Training Set without Added Label Noise	No	91.20 ± 0.06
Training Set without Added Label Noise	Yes	91.17 ± 0.03
To assess the efficacy of the peer loss on our dataset, we intentionally introduced noise by perturbing  $10\%$  of the annotations within the training data. Specifically, from the pool of 67,969 categorical values (6,179 images  $\times 11$  attributes), we randomly selected 6,797 values  $(10\%)$  and replaced each with an alternative value chosen uniformly from the remaining possibilities. Using this perturbed data, we trained our model both with and without incorporating the peer loss. The outcomes, presented
<sup>8</sup>https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGClassifier.html
<sup>9</sup>https://scikit-learn.org/stable/modules/generated/sklearn.ensembleGradientBoostingClassifier.html
in the upper section of Table 3, reveal that training on the noisy data alone yields an average macro F1 score of  $85.31 \pm 0.24$ . However, integrating the peer loss increases this score to  $88.86 \pm 0.02$ , demonstrating the efficacy of the peer loss.
After confirming the effectiveness of peer loss, we applied it to our training data without any artificially introduced noise. As shown in the lower part of Table 3, utilizing peer loss yields an average macro F1 score of  $91.17 \pm 0.03$ , while training without it results in a score of  $91.20 \pm 0.06$ . The nearly identical scores indicate that the impact of peer loss is not significant, suggesting that the presence of annotation noise within our dataset is limited.
7.13 Other Tables and Figures
- Table 4: Frequencies of attribute values, illustrating content similar to Figure 3 in the main paper.
- Table 5: Initial attributes used as a starting point to differentiate the five types of WBCs.
- Table 7: Precision, recall, and F-measure values for each attribute value.
- Figure 16: GradCAM heatmaps for different seeds and backbones that are not reported in the main paper.
- Figure 17: Visualization of how staining conditions and imaging equipment influence the attributes.
Table 4: Attribute Dist. The distribution represents the results of annotating all typical WBCs from the PBC dataset, which is the image source we utilized. We did not actively control or manipulate the distribution.
Attribute	Value (Count)
Cell-Size	Big (4,997), Small (4,271)
Cell-Shape	Round (7,173), Irregular (2,095)
Nucleus-Shape	Segmented-Bilobed (2,806), Unsegmented-Band (2,356), Unsegmented-Indented (1,205), Segmented-Multilobed (1,143), Unsegmented-Round (967), Irregular (791)
Nuclear-Cytoplasmic-Ratio	Low (8,148), High (1,120)
Chromatin-Density	Densely (8,443), Loosely (825)
Cytoplasm-Vacuole	No (8,559), Yes (709)
Cytoplasm-Texture	Clear (7,429), Frosted (1,839)
Cytoplasm-Color	Light Blue (7,011), Blue (1,273), Purple Blue (984)
Granule-Type	Small (3,003), Round (2,801), Nil (2,374), Coarse (1,090)
Granule-Color	Pink (2,925), Red (2,803), Nil (2,373), Purple (1,167)
Granularity	Yes (6,896), No (2,372)
Table 5: Coarse Morphological Attributes
Cell Type	Nucleus Structure	NC Ratio	Granularity	Granularity Color	Cell Size
Basophils	Segmented	Low	Yes	Blue / Black (dense)
Eosinophils	Segmented	Low	Yes	Red
Lymphocytes	Unsegmented	High	No	Small
Monocytes	Unsegmented	Low	No
Neutrophils	Segmented	Low	Yes	Blue
Table 6: Macro Precision (Pre.), Macro Recall (Rec.), and Macro F-measure (F-m.) of VGG16, ResNet50, ViT-Base, and ConvNeXt-Tiny (CNXT-T) for the task of attribute prediction.
Cell Size	Cell Shape	Nucleus Shape	Nuclear Cyto-plasmic Ratio	Chromatin Density	Cytoplasm Vacuole	Cytoplasm Texture	Cytoplasm Color	Granule Type	Granule Color	Granularity (Average)
VGG16	83.45	89.08	74.88	96.78	83.97	91.24	92.29	84.19	99.28	98.72	99.57	90.31
Pre.	±0.61	±1.80	±0.81	±1.01	±1.26	±0.83	±0.47	±0.32	±0.10	±0.31	±0.09	±0.51
VGG16	83.44	90.13	74.18	95.12	86.71	85.96	95.21	83.93	99.42	98.54	99.60	90.20
Rec.	±0.43	±0.67	±0.50	±0.94	±2.56	±1.39	±1.38	±0.96	±0.12	±0.11	±0.05	±0.70
VGG16	83.44	89.52	74.10	95.91	85.18	88.36	93.61	83.95	99.35	98.62	99.58	90.15
F-m.	±0.52	±0.78	±0.39	±0.35	±0.74	±0.63	±0.35	±0.27	±0.10	±0.18	±0.07	±0.31
ResNet50	84.21	90.73	77.08	97.47	84.55	92.71	93.22	88.24	99.36	98.89	99.60	91.46
Pre.	±0.61	±0.90	±0.47	±0.18	±0.48	±1.76	±0.87	±0.30	±0.10	±0.08	±0.01	±0.30
ResNet50	83.69	90.64	75.70	95.30	88.52	87.08	95.95	88.06	99.52	98.63	99.62	91.16
Rec.	±0.44	±0.80	±0.99	±0.05	±0.59	±2.09	±0.42	±0.45	±0.10	±0.10	±0.03	±0.26
ResNet50	83.81	90.66	76.13	96.35	86.39	89.57	94.49	87.99	99.44	98.76	99.61	91.20
F-m.	±0.33	±0.36	±0.59	±0.06	±0.32	±0.47	±0.51	±0.47	±0.07	±0.08	±0.02	±0.06
ViT-B	83.58	89.53	77.21	96.87	84.73	90.62	92.86	87.62	99.32	99.16	99.64	91.01
Pre.	±0.14	±0.64	±0.27	±0.75	±2.74	±0.97	±0.16	±0.18	±0.13	±0.06	±0.02	±0.30
ViT-B	83.01	90.08	75.88	95.91	84.61	90.68	95.18	88.04	99.45	98.70	99.66	91.02
Rec.	±0.52	±0.52	±2.03	±0.76	±2.93	±0.70	±0.96	±0.60	±0.10	±0.19	±0.05	±0.32
ViT-B	83.19	89.80	75.94	96.37	84.51	90.63	93.94	87.70	99.39	98.92	99.65	90.91
F-m.	±0.43	±0.51	±1.24	±0.29	±1.78	±0.19	±0.38	±0.45	±0.11	±0.11	±0.02	±0.21
CNXT-T	83.50	91.00	78.51	96.80	85.56	92.84	93.93	88.05	99.49	99.22	99.70	91.69
Pre.	±0.43	±0.61	±1.27	±0.42	±0.68	±0.61	±0.54	±0.19	±0.09	±0.16	±0.06	±0.26
CNXT-T	83.44	91.54	77.68	95.65	85.87	88.04	95.00	88.28	99.64	98.75	99.71	91.24
Rec.	±0.35	±0.39	±1.89	±0.29	±0.85	±0.26	±0.77	±0.77	±0.04	±0.08	±0.03	±0.26
CNXT-T	83.46	91.26	77.97	96.21	85.70	90.26	94.44	88.15	99.56	98.98	99.71	91.43
F-m.	±0.37	±0.50	±1.58	±0.34	±0.31	±0.40	±0.27	±0.46	±0.06	±0.12	±0.04	±0.26
Figure 16: Grad-CAM heatmaps from different seeds (ResNet50 with seed 100) and various backbones (including VGG16, ConvNeXt-Tiny, and Vit-Base), while images are the same as Figure 4
Figure 17: WBC appearances under different staining and microscopy conditions. WBCs sourced from the PBC Dataset [6] (used in this work) have undergone May Grünwald-Giemsa staining and were acquired through the automated digital cell morphology analyzer, CellaVision DM96. WBCs from the APL Dataset [59] are stained using Wright's method and observed through the CellaVision DM100 cell analyzer. On the other hand, the Raabin-WBC [7] images, stained with Giemsa, are captured using smartphones mounted on both Olympus CX18 and Zeiss microscopes.
[62] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[63] Bec Crew. Google Scholar reveals its most influential papers for 2019. Nature Index, 2019.
[64] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2014.
[65] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[66] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[67] Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. Label Studio: Data labeling software, 2020-2022. URL https://github.com/heartelabs label-studio. Open source software available from https://github.com/heartelabs/label-studio.
[68] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: a stylegan encoder for image-to-image translation. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[69] Angelo Genovese, Mahdi S Hosseini, Vincenzo Piuri, Konstantinos N Plataniotis, and Fabio Scotti. Histopathological transfer learning for acute lymphoblastic leukemia detection. In IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications (CIVEMSA), 2021.
[70] Yutong Bai, Jieru Mei, Alan L Yuille, and Cihang Xie. Are transformers more robust than cnns? Advances in neural information processing systems, 34:26831-26843, 2021.
[71] Mahsa Baktashmotlagh, Mehrtash Harandi, and Mathieu Salzmann. Distribution-matching embedding for visual domain adaptation. Journal of Machine Learning Research, 17, 2016.
[72] Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. In International Conference on Machine Learning, 2020.