dataesr/scientific-paragraphs-categorization huggingface dataset:
Data and Labels The task we consider is to classify samples between malignant or non-malignant (including normal cases and benign lesions). In most cases, we used the Breast Imaging-Reporting and Data System (BI-RADS) risk rating from the American College of Radiology (ACR)  [50]  to determine malignancy, where the scores are categorised as follows: (1) negative or normal, (2) benign, (3) probably benign, (4) suspicious for malignancy, (5) highly suggestive of malignancy, and (6) known biopsy-proven malignancy. We binarised the rating for each image taking scores 1, 2, and 3 as non-malignant, and BI-RADS 4, 5, and 6 as malignant. In the case of the CMMD dataset these ratings are not available but each breast has been confirmed by biopsy to be benign or malignant, so we used these binary labels.
We used several heterogeneous public datasets, and a private dataset, each based on different geographical locations and with different levels of class imbalance, to assess the performance of class imbalance techniques. Each of the datasets contains high-resolution Full Field Digital Mammography (FFDM) images. This reflects current clinical practice, and is the reason we chose not to use the popular Digital Database for Screening Mammography (DDSM) dataset, given that those images are scanned and digitised Screen-Film Mammograms. A full breakdown of the number of images in each dataset is given in Table  1. The first dataset used for training was a private dataset of 3847 images, containing mammograms from four vendors, namely, Fujifilm, GE, Hologic, and Planmed. Benign images account for 70% of the training set.
We used a test split for this dataset containing 504 images selected randomly. We refer to the datasets as D HMI train and D HMI test. The first of the public datasets used for training was VinDr-Mammo  [11] , consisting of 20,000 images from 5000 patients. The images were collected in two hospitals in Vietnam, Hanoi Medical University Hospital and Hospital 108, using mammography systems from three vendors (Siemens, Planmed, and Giotto). This dataset presents a large class imbalance, where only 988 (5%) images are labelled as malignant, being close to a typical mammography screening scenario where most of the images acquired are normal or benign. The dataset authors have already split the data into training (16,000) and test (4000) images to ensure consistency in results reported. Hereafter, these will be referred to as D VinDR train and D VinDR test , respectively.
The second public dataset used for training was the relatively new Chinese Mammography Database (CMMD)  [10] , which contains 3744 images of 1775 patients. The dataset was published by the South China University of Technology, and the images were obtained using a GE Senographe DS mammography system. In this dataset there are more malignant images than benign cases, reflecting the diagnostic clinical context, where suspicious findings have been identified and further imaging is required to diagnose if they are malignant. We split the dataset into training (2998 images-80%) and test (746 images-20%) sets, stratifying by class and ensuring that the images for a single patient remained in a single set. We refer to these datasets as D CMMD train and D CMMD test. For each of the datasets, we further divided the training set into separate training and validation sets, using the validation set to tune hyper-parameters and to find the epoch of the "best" model during training.
These splits were stratified based on class, and images for each patient appeared in only one group. For each assessed technique tackling class imbalance, the models were separately trained on each of the three above datasets, and the performance was assessed on each of the test sets. Taking a model trained on D VinDR train and testing it on D CMMD test , for example, allows us to assess the generalisation capability on datasets not seen at training time. Finally, the popular public INBreast  [17]  dataset was used solely for testing performance across all experiments, allowing a comparison to other studies in the area  [42]. This dataset consists of 410 FFDM images taken with a Siemens mammography system.
Software All of the image processing and experiments in this study were carried out using Python  [51]. Image processing was done using functions developed by the Hera-MI team on top of the OpenCV  [52]  and scikit-image  [38]  packages. Keras  [53]  and Tensorflow  [54]  were used to build and train the classification model. The Matplotlib  [55]  and Seaborn  [56]  packages were used for plotting.
Breast-Wise AUC Performance When training on D HMI train , and testing on the corresponding test set D HMI test , we do not observe significant differences between simply training on the imbalanced dataset (AUC = 0. 776) and applying standard class imbalance techniques. Whereas the class weighting leads to a slight increase in AUC (0. 786), models trained with over-sampling see a decrease of 0. 014 to 0. 762. There is a wider range of results when considering outof-distribution generalisation. Of these four initial experiments, the imbalanced training leads to the best performance on D VinDR test and INBreast, whereas the models trained with class weighting perform slightly better on D CMMD test (0. 660 vs. 0. 646), although the generalisation to this dataset remains quite poor. The models trained on D HMI train using our synthetic lesions (Artifacted) method achieve the best results on the related D HMI test , with an improvement of ∆ = 0.
021 over the next best AUC score. Moreover, this method achieves a noticeable increase in generalisability to out-of-distribution test sets, with improvements over the next best result for D VinDR test (∆ = 0. 060), D CMMD test (∆ = 0. 070), and INBreast (∆ = 0. 027). Most notably, this gener-alisability results in the same performance on out-of-distribution datasets as when training directly on images from those distributions (D VinDR test : 0. 760 vs. 0. 757, D CMMD test : 0. 730 vs. 0. 727). Turning our attention to the models trained on D VinDR train , we see minor drops in the AUC scores for D VinDR test when comparing the imbalanced training (0. 757) to over-sampling (0. 752), and class weighting (0. 739). However, a more significant decrease occurs for the under-sampling experiment (0.
691, ∆ = -0. 066), when we remove a large portion of the benign samples from the dataset. The over-sampled model achieves the best results on INBreast (0. 824), which is the second highest result on INBreast among all experiments, perhaps due to the high representation of Siemens images in D VinDR train. The models trained on D VinDR train deliver relatively poor generalisation to both D HMI test and D CMMD test , again due to the characteristics of images generated by different mammography systems, as well as an insufficient number and variety of malignant samples in D VinDR train. The Artifacted method, trained on D VinDR train , demonstrates an improvement in AUC over the next best method on D VinDR test , i. e. , the imbalanced training (∆ = 0. 011), as well as better generalisability to D HMI test (∆ = 0.
040). It also yields the second best AUC score on INBreast (0. 799) among the five results in this second set of experiments. In contrast, this method delivers the lowest result for D CMMD test , with a difference of ∆ = -0. 041 compared with class weighting. The models in the final set of experiments are trained on D CMMD train , and here we do not have an Artifacted experiment as the nature of the imbalance is different in the CMMD dataset, i. e. , there are more malignant samples than benign (see Table  1 ). The imbalanced (0. 727) and over-sampled (0. 719) experiments offer similar AUC performance when tested on D CMMD test , with a gap over the next two results, under-sampled (0. 690) and class weighting (0. 681). Most of the trained models perform very poorly when applied to D HMI test and D VinDR test , with AUC scores close to 0.
5, which would be achieved by a predictor making random guesses. The over-sampling experiment is an exception to this with the highest AUC scores for D HMI test (0. 614), D VinDR test (0. 667), and INBreast (0. 711). These results are still poor, however, and indeed the CMMD dataset is associated with the lowest results across all experiments, both on the in-distribution test set, and the ability of a model trained on D CMMD train to generalise to any of the other three datasets.
The only exceptions to this were the application of class weighting on D HMI , where sensitivity reduced from the imbalanced case, and over-sampling with D CMMD , which saw a decrease in specificity. Despite the gap between sensitivity and specificity when training on the imbalanced datasets, the results for MCC are in fact higher than when applying the three standard class imbalance techniques (under-sampling, over-sampling, and class weighting) for both D HMI and D CMMD. However, this observation does not hold in extreme cases when the class imbalance is more severe as for D VinDR , where the near-zero sensitivity contributes to a very low MCC of 0. 033. The Artifacted method achieves the highest MCC score for D HMI (MCC = 0. 443), as well as the second best result for D VinDR (0. 253). The highest result for the latter case is given by the over-sampled experiment (0.
378); however, this coincides with a low sensitivity (0. 263), which indicates that the model may have difficulty generalising to malignant samples different from those in the training set. Figure  2  demonstrates the behaviour of the models trained on D VinDR train with different approaches. With an imbalanced dataset, the model has a strong bias towards the benign class, meaning that almost every sample is given a low probability of malignancy at testtime, resulting in the low sensitivity and high specificity seen at an operating point of 0. 5. However, the trained model remains capable of achieving some separation between malignant and benign samples, leading to the second highest AUC score among the VinDR experiments. For example, over 70% of truly benign samples are given a malignancy score between 0 and 0.
Research Questions In this section, we refer to the three Research Questions (RQ) listed in the introduction (see Section 1) and discuss the insights with regard to each of these questions. (RQ1) How does an imbalanced training set impact cancer classification performance in mammography? For each of the three imbalanced datasets, the class imbalance caused the classifier to bias towards the majority class. As might be expected, a higher imbalance resulted in a higher bias, i. e. , a standard classifier trained on the VinDR dataset predicted every test sample to be benign. On the other hand, models trained on imbalanced datasets unexpectedly achieved comparable AUC-ROC scores to models trained with common class imbalance techniques, indicating that despite predictions shifting towards the majority class, the model still learns to separate the two classes.
This lends credence to setting a new threshold on the output scores post-training as an effective technique of dealing with class imbalance, applied in  [7]  (see also  [18] ). New thresholds can also be set in practice, regardless of class imbalance, to achieve an acceptable trade-off between sensitivity and specificity for a given task. Therefore, threshold setting may be the simplest approach to tackling class imbalance, while remaining relatively effective. (RQ2) How do common techniques for tackling class imbalance compare for cancer classification in mammography? Among over-sampling, under-sampling, and class weighting, no single technique consistently achieved the best AUC-ROC nor MCC scores across all experiments. Oversampling performed best of these techniques on the VinDR and CMMD experiments, in the latter case also leading to a significant improvement when testing on out-of-distribution datasets.
However, over-sampling performed the worst on three out of four test sets when trained on HMI. Therefore, there is clearly no one single best approach for every mammography dataset; however, one lesson that may be gained from these results is that experimenting with other techniques may allow for an improved out-of-distribution generalisation without sacrificing in-distribution performance. There is a notable drop in classification performance in the VinDR and CMMD experiments when under-sampling is applied. This aligns with expectations that using only a subset of data may remove informative samples and thus may hinder performance, particularly in the case of VinDR where 90% of the dataset is removed. Furthermore, the selection of the subset for training affects performance, as the model began to learn only after several attempts with different random seeds for the initialisation of model weights and taking a different random sample of benign images.
The training process was not straightforward either for the imbalanced or class weighting experiments on the VinDR dataset. We hypothesise that these difficulties might be due to the low frequency with which malignant samples were seen during training. In our case, training loss began to decrease after we ensured that approximately an equal number of malignant samples appeared every 0. 2 epochs, although it remains unclear whether this was the direct cause. Over-sampling, together with the Artifacted method, were the only VinDR experiments that worked without further adjustments. Moreover, the model trained on VinDR with over-sampling performed well on the test set relative to the other methods according to our AUC-ROC and MCC metrics (see Section 4), so over-sampling could be seen as a promising approach for cases with high imbalance between classes.
Lesions appear in only one of the two views (CC or MLO) for approximately 10% of malignant breasts in the VinDR dataset, so the other image will be included as malignant during training despite containing no visible signs of malignancy. Predicting these images to be benign during training would lead to a high contribution to the loss, perhaps causing the model to generally avoid assigning a low malignancy score to images. We study the effect of "artifacting" observing the latent representation of the test samples from the HMI dataset, as well as when we insert synthetic lesions into these images (see Figure  3 ). When we introduce "artifacted" samples to a model that has not seen them at training time (top-right pane), multiple tight clusters are formed in the latent space, and the representation of these synthesised examples is closely related to real examples in the test set, both benign and malignant.
When the model has encountered "artifacted" samples during training, a broader latent space representation appears, and some of the synthesised examples occupy their own regions of the space (bottom-right pane). This indicates that some of the synthetic lesions are quite different to the real lesions in the HMI dataset. This helps the model to learn from a broader data distribution and helps to explain the significant increases in out-of-distribution performance seen by the Artifacted method. A further illustration of this effect is seen by comparing the top-left and bottom-left panes, where the addition of synthesised malignant images during training has the effect of pushing the real malignant images closer together in the latent space.
However, the false overconfidence of our model trained on the VinDR dataset with over-sampling indicates that overfitting may still remain a problem. Stadnick et al. [42]  tested several state-of-the-art models  [19, 23, 24, 58]  on multiple public datasets, including INBreast and CMMD. The authors used only 26% of the INBreast images in their test set, and test on the full CMMD dataset, so we re-evaluated our Artifacted model trained on D HMI train , yielding an AUC-ROC of 0. 850 on this INBreast subset and 0. 718 on the full labelled CMMD dataset of 3728 images. The AUC-ROC scores reported by  [42]  vary between 0. 612-0. 980 for INBreast and 0. 449-0. 831 for CMMD, and our results outperform three of the seven reported methods on INBreast, and two on CMMD. Our results are most similar to the model of Wu et al.
[23] , which scores 0. 802 on INBreast and 0. 740 on CMMD. Remarkably, Wu et al. [23]  rely on a significantly larger dataset with both a higher amount of malignant samples and a substantially larger imbalance. On the other hand, we note that the dataset is composed of images coming from four different vendors, similar to our HMI dataset. This leads us to two observations: firstly, the amount of malignant samples is more likely to determine the model performance and the "artifacting" method, used in this work, allows us to cope with insufficient data; secondly, the dataset heterogeneity can positively influence the performance of the training. The VinDR and CMMD datasets are both relatively new, and so are not as well studied as INBreast or DDSM, for example. Thus, our work provides a useful reference and baseline for other researchers using these datasets.
We note the lower results achieved on the CMMD dataset, both in this study and in  [42] , and one potential cause of this is the composition of the dataset. All breasts in the CMMD dataset have been biopsied, indicating that there are suspicious abnormalities in both the malignant and benign images, making the task of separating malignant from non-malignant more difficult. Using a different dataset, Wu et al. [23]  found lower classification performance on a biopsied sub-population compared to their overall screening population. A goal of this paper was to fairly compare popular class imbalance techniques using several datasets and consistent processing pipelines and metrics. Too often, new techniques are proposed for coping with class imbalance, or for generally improving breast cancer classification and segmentation, where the studies provide no comparison to existing methods or they compare methods using proprietary datasets or specific, non-standard performance metrics.
For example, while the AUC-ROC metric is generally used in research, studying specific operating points is more relevant from an industrial application perspective. Further standardisation of evaluation and performance measurement will help researchers in the field to compare the methods from the large body of research in this field and more efficiently find techniques applicable to their problems, and the current work as well as other comparative studies such as  [42]  help in this regard. Access to public datasets is also crucial for these purposes, so the release of two FFDM datasets (VinDr-Mammo  [11]  and CMMD  [10] ) in the last two years is very welcome.
For cases with a higher class imbalance, our experiment with VinDR indicates that over-sampling may yield the best separation between malignant and benign, but the output distributions of the predictions should be assessed to ensure the model does not overfit on the small number of minority samples. Finally, if feasible within the context of the study, whether on mammography or medical imaging more broadly, generating synthetic lesions could provide a good way of balancing the classes while also introducing prior knowledge from domain experts, not only reducing the bias towards the benign class, but also improving the overall model performance. Overall, our experiments tend to demonstrate that better performance is achieved when training on more heterogeneous and augmented data. The best results are achieved on the HMI multi-vendor (five vendors) dataset with synthesised lesions.
VinDR is globally a runner-up with three vendors included. Finally, CMMD-trained classifiers perform the worst having only one vendor in the dataset. Certification of deep learning solutions for breast cancer detection is usually based on demonstrating efficacy on a particular dataset collected by the vendor. It is well-known in the machine learning community that performance may decrease if the distribution of data used for training is different to the cases on which the model is applied (known as a "domain shift")  [59]. The results in this study provide further evidence of this, for example, with models trained on CMMD data failing to generalise to the other datasets. The regulatory process ought to account for this performance degradation under the presence of dataset shift, requiring systems to demonstrate safety and efficacy on sample images from all "domains" (in this case, mammography imaging systems) where the deep learning solution will be applied.
Limitations Due to the large number of experiments, and the time for each training run (1-8 days), it was not feasible to conduct extensive hyper-parameter tuning. While the application of techniques such as Batch Normalisation or Dropout have been shown to increase robustness with respect to hyper-parameters, better results may have been possible for each individual experiment with more in-depth investigation. In our case, for the sake of simplicity and a fair comparison, we chose to use the same setup (learning rate, number of epochs, etc. ) for all experiments. We focused on studying class imbalance in the context of deep-learningbased classifiers, reducing the complexity of the studied neural networks. More advanced solutions may yield different results on the same datasets. There are many proposed techniques for tackling class imbalance, and only a subset of the most commonly used are considered here.
Conclusions This study reviews several popular class imbalance techniques as applied to breast cancer classification in high-resolution mammography using the latest public datasets. The results show that a higher class imbalance when training a deep learning classifier causes a greater bias towards the majority class, which is generally reduced by applying any of the standard class imbalance techniques. However, these techniques often provide no benefit to overall model quality as measured by AUC-ROC, indicating that setting a new threshold or operating point after training may well be sufficient in many use cases. Moreover, under-sampling in particular led to a significant drop in AUC-ROC in cases of higher imbalance between classes. Furthermore, the study demonstrates the usefulness of synthetic malignant lesions to balance classes, both in eliminating the bias towards the majority class, and in achieving higher AUC-ROC performance, with some significant improvements to out-of-distribution generalisation.
Data Availability Statement: Three of the datasets used in this study are publicly available: IN-Breast  [17]  is available on request; the VinDr-Mammo  [11]  dataset is available through PhysioNet at https://www. physionet. org/content/vindr-mammo/1. 0. 0/ (accessed on 15 November 2022); and CMMD  [10]  is available from the Cancer Imaging Archive at https://wiki. cancerimagingarchive. net/ pages/viewpage. action?pageId=70230508 (accessed on 15 November 2022).
Collection of Feeding Behavior Data Direct observation was used as a method to evaluate the composition of the browsed diets of the goats  [10, 23]. Before starting direct observation, a three-day familiarization period was necessary to accustom the flocks to the presence of observers, as detailed by Bonnet et al. [24]  and Meuret and Provenza  [25]. Measurements concerned the number of bites and the botanical composition of each part of the plant consumed by goats. Data were collected by observing each goat for 10 min, three times a day. To determine the average bite mass (BM), 100 hand-plucked samples per plant species, similar to those consumed by goats, were imitated seasonally  [26]. The hand-plucked samples were collected in special bags and dried in an oven at 40 • C to constant weight to obtain the average dry matter of each BM per consumed plant species.