WHEN HIGH-PERFORMING MODELS BEHAVE POORLY IN PRACTICE: PERIODIC SAMPLING CAN HELP
Year: 2021

Abstract:
Training a deep neural network (DNN) for breast cancer detection from medical images suffers from the (hopefully) low prevalence of the pathology. For a sensible amount of positive cases, images must be collected from numerous places resulting in large heterogeneous datasets with different acquisition devices, populations, cancer incidences. Without precaution, this heterogeneity may result in a DNN biased by latent variables a priori independent of the pathology. This may be dramatic if this DNN is used inside a software to help radiologists to detect cancers. This work mitigates this issue by acting on how mini-batches for Stochastic Gradient Descent (SGD) algorithms are constructed. The dataset is divided into homogeneous subsets sharing some attributes (e.g. acquisition device, source) called Data Segments (DSs). Batches are built by sampling each DS periodically with a frequency proportional to the rarest label in the DS and by simultaneously preserving an overall balance between positive and negative labels within the batch. Periodic sampling is compared to balanced sampling (equal amount of labels within a batch, independently of DS) and to balanced sampling within DS (equal amount of labels within a batch and each DS). We show, on breast cancer prediction from mammography images of various devices and origins, that periodic sampling leads to better generalization than other sampling strategies.

heading: 1 INTRODUCTION
text: Breast cancer screening leads to an earlier detection of breast cancer resulting in an improved prognosis and a reduced mortality (Marmot et al., 2013). During a screening exam, Full-Field Digital Mammography (FFDM) images of the breasts of a patient are acquired. They are then analyzed by 1 or 2 radiologist(s) who look(s) for signs of malignancy. If any sign of malignancy is observed, the patient is recalled to assert the final diagnosis with complementary exams. Despite constant technological progress, cancers may still be missed while the recall rate remains high with a limited fraction of recalled patients actually diagnosed with cancer (Rawashdeh et al., 2013).
Machine Learning (ML) based screening systems may help radiologists detecting cancers and improve their sensitivity, specificity and reading time, (Pacilè et al., 2020; McKinney et al., 2020; Schaffter et al., 2020; Shen et al., 2019; Kooi et al., 2017). Indeed, Deep Neural Networks (DNNs) can help to solve this problem initially formalized as a binary classification problem. Recent approaches build upon breasts symmetry (Brhane Hagos et al., 2018; Kooi & Karssemeijer, 2017), the patient history (Kooi & Karssemeijer, 2017) or the whole set of exam images (Geras et al., 2017) to enhance classification performances. Other approaches further aim at localizing the malignant lesion(s) in the breast (Shen et al., 2021; Pedemonte et al., 2020) or improving the generalization of DNNs trained on FFDM images but used on Digital Breast Tomosynthesis (DBT) images, 3D images of the breast (Matthews et al., 2021; Singh et al., 2020).
It is hard to train a DNN on FFDM images that aims to be used in a software delivering nearly real time recommendations for a radiologist investigating screening exams. First, it requires a large quantity of labeled training data but medical data sources are typically small and partially labeled. Transfer Learning (TL) may help to partially cope with this problem (Morid et al., 2021; Karimi et al., 2021) while Domain Adaptation (DA) may help to reduce the potential distribution mismatch between multiple source data distributions and a test data distributions (Crammer et al., 2008; BenDavid et al., 2010; Ganin et al., 2016; Guan & Liu, 2021). Yet, TL may deliver little performance
gain (Raghu et al., 2019), DA may not be appropriate for the setup considered here due to the potentially large number of training sources and the infinite number of testing sources and the ”nearly real time” prediction constraint may be incompatible with the common offline setup most DA solutions embrace as they often build upon source and target domains available at the same time (Ben-David et al., 2007; 2010; Ganin et al., 2016). Second, each medical institution exhibits its own characteristics (patients, radiologists experiences, sets of acquisition devices, disease incidences, ...), thus the DNN trained on data from such sources may still be biased: its predictions being modulated by latent uncontroled variables. If not addressed, this may result in dramatic consequences once the model is used in production (Barocas & Selbst, 2014). Finally, training a DNN on FFDM implies fighting class imbalance (the disease incidence being hopefully small) to avoid technical issues for ML models (He & Garcia, 2009) that could amplify the potential bias issue highlighted above.
The following experiment exhibits the complexity of training a DNN from only 2 sources of data. The goal was to train 2 DNNs to classify benign vs malignant FFDM images on 2 sets of data, DS1 (7258 malignant / 13341 benign images) and DS2 (160 malignant / 21871 benign) and to evaluate them on a left out set of data DS3 (1106 malignant / 14192 benign) from another medical institution. A train / validation split was used patient-wise and source-wise: 80% (resp. 20%) of patients of a source were used for training (resp. validation). The proportion of patients with cancers was kept constant in both train / val sets. The first DNN, dnn1, was trained on DS1, the second one, dnn1∪2 on DS1 ∪ DS2 with a Stochastic Gradient Descent (SGD) like algorithm. Balancing benign and malignant images within mini-batches was used during training to cope with class imbalance. The Weighted Logloss per image (WLL / image), log loss with class weights, was monitored on a validation set during the training, see Figure 1 - left. The model dnn1∪2 exhibits a low global validation WLL / image on DS1 ∪DS2, but a higher WLL / image on DS1 compared to dnn1. The validation WLL / image on DS2 further indicates that it was over-fitted. Plus, dnn1∪2 shows a lower Area Under the ROC Curve (AUC) on DS3 than dnn1, though it was trained on more data, see Figure 1 - center. Finally, the distributions of models predictions on DS3 show that dnn1∪2 predictions are shifted towards low values compared to dnn1 predictions and that even for the malignant images, see Figure 1 - right. Hardly, does dnn1∪2 benefit from data brought by DS2.
This paper identifies 2 main locks to train efficiently a DNN for breast cancer prediction from mammography images: class imbalance and a non homogeneous learning set. Class imbalance can be addressed via an appropriate sampling strategy (He et al., 2008; Chawla et al., 2002), sample weighting (Byrd & Lipton, 2018) or a learnable sampling (Hu et al., 2019; Ren et al., 2018). The sampling strategy may add further benefits such as stratified sampling which may help to efficiently select samples from low variance clusters to speed up the convergence of mini-batch optimization algorithms (Csiba & Richtárik, 2018; Zhao & Zhang, 2014).
This paper focuses on training a DNN with a mini-batch SGD like optimization algorithm on a training set, divided into a finite ensemble of independent and homogeneous sets of samples, sharing an explicit and unique set of properties, called Data Segments (DSs). A Periodic Sampling strategy is proposed to build mini-batches of data by sampling each DS periodically with a frequency propor-
tional to its rarest label and by balancing the labels within a batch and a DS to fight class imbalance. This paper, first introduces formally the proposed Periodic Sampling, then provides numerical evidence of its benefits and finally discusses its main limitations and directions for improvement.

heading: 2 METHOD
text: This paper focuses on binary classification and introduces a method to train a DNN, with a SGD like algorithm, from several segments of data while reducing the risk of using an identified bias. It aims to build appropriately the mini-batches of data with a dedicated Periodic Sampling strategy.

heading: 2.1 NOTATION AND PROBLEM STATEMENT
text: Let X stand for the input space, i.e. the space of FFDM images of a given shape. Let Y = {0, 1} stand for the space of labels, 0 standing for a benign image, 1 for a malignant image. x, y ∈ X × Y is a learning sample: an image with its label. Let i,Ni ∈ N, then DSi = {(xj , yj) ∈ X × Y : j = 0, · · · , Ni − 1} refers to a Data Segment (DS) i.e. a subset of homogeneous learning samples. Let M ∈ N be the number of DSs at hand. The learning set is composed of M homogeneous and independent DSs: ∪M−1i=0 DSi. With a DNN: f : X −→ Y , and with ` standing for the binary cross-entropy, the empirical optimization problem to solve reads:
f∗ ∈ arg min f 1∑M−1 i=0 Ni M−1∑ i=0 ∑ (x,y)∈DSi `(y, f(x)) (1)

heading: 2.2 PERIODIC SAMPLING
text: General description First, the DNN should work on all the DSs of interest. It should learn at the same pace on the training DSs to avoid the issue seen in Figure 1 and not overfit to quickly a DS with few malignant images. Third, the DNN should not be biased toward a specific label on any DS.
Periodic Sampling is a two step process introduced to meet these requirements. First, it builds a sampling pattern that indicates how often samples from the DSs should be included in a mini-batch. Second, it iterates over the sampling pattern and selects a sample from the designated DS while alternating in a balanced way the labels to ensure label balance in a DS.
Sampling proportion With the above notation, we define the number of samples of the least represented label ymini , inside DSi by n min i = ∑Ni−1 j=0 1yj=ymini . Now, let Nmin = ∑M−1 i=0 n min i be the sum of the samples of the least represented label inside each DS. The main idea is to feed the DNN with the samples from DSi with a proportion pi = nmini /N min during the training process.
Sampling pattern Periodically selecting samples from DSi with a period 1/pi allows to feed the DNN with samples from DSi with a proportion pi. This way, even if pi is dramatically small, samples from DSi would still be selected on a regular basis. To do so, a sampling pattern S is built. This is a sequence of DS ids i ∈ {0, · · · ,M − 1} of length Nmin, where the number of occurrences of DSi is given by nmini . To build it, one first sorts the DS by their numbers n min i , in the ascending order. Then, one iterates over the sequence of i, nmini , to fill in the available slots of a list of length Nmin and places the indices i at nmini locations, equally spaced while avoiding potential collisions with already filled slot. This is described in Algorithm 1.
Construction of training batches Then, one builds batches of b ∈ N samples by iterating sequentially (and infinitely) over the sampling pattern S to get, for each sample to be included in a batch, the index i of the DS from which it should be drawn. This way, for every nearly Nmin/b + 1 seen samples by the DNN, the DNN has been trained on nearly pi × (Nmin/b + 1) samples from DSi. To ensure that the labels are presented in equal proportions to the DNN, the mechanism alternates between 0 and 1 labels when selecting a sample from DS i. The batch construction pipeline is summarized in Algorithm 2. An example of the selection of samples according to a sampling pattern with the alternating label mechanism is provided in Figure 2
Algorithm 1: Build the sampling pattern Data: N , {DSi : i = 0, · · · ,M − 1} Result: Sampling pattern S
1 step 1: Compute the numbers of samples with the least represented label for every DSi:{ nmini : i = 0, · · · ,M − 1 } 2 step 2: Compute the length of the sampling pattern L← ∑M−1 i=0 n min i 3 step 3: Build the sampling pattern: S ← array of size L filled with -1 values for i← 0 to M − 1 do
pos← linspace(0, Nmin, nmini ) ; for j ← 0 to nmini − 1 do
p← pos[j] while S[p] 6= −1 do
p← p + 1 end S[p]← i
end end
1 = 5, n min 2 = 10 and n min 3 = 20.
Algorithm 2: Build batches from sampling pattern Data: b, S, Nmin, DS = {DSi : i = 0, · · · ,M − 1} Result: A collection of batches of size b Y ← {0, 1}M randomly initialized idx← 0 while training do
B ← [· · · ] empty list for j ← 0 to b− 1 do
i← S[idx] yi ← Y [i] (x, y)← next samples (x, y) ∈ DSi such that y = yi B ← B + (x, y) idx = idx + 1 (mod Nmin) Y [i] = Y [i] + 1 (mod 2)
end yield B
end
Practically, some Deep Learning (DL) libraries ensure that the order of the samples indices returned by a sampling function is maintained inside the mini-batches while others do not. This may not be an issue as long as the proportions of DSs are ensured at a scale of a few (hundreds) learning steps / batches.

heading: 3 EXPERIMENTS

heading: 3.1 EXPERIMENTAL SETUP

heading: 3.1.1 DATA
text: Image modality and labeling process The considered data are FFDM images e.g. gray-scale images, with pixel values in [0, 4095], although other ranges can be used depending on the FFDM device manufacturer. Every full view (not zoom) FFDM image is considered for training and evaluating the models: Cranio-Caudal (CC), Medio-Lateral Oblique (MLO), ...
Images labels (0 and 1) are assigned as follows. A positive image (label 1) is an image where a suspicious lesion was localized and for which the malignant status was confirmed by a biopsy shortly after the mammogram was done (within 3 months). Each malignant lesion is localized as precisely as possible with a bounding box drawn by trained radiologists. A positive breast is a breast with at least one positive image. When the malignant lesion is not visible inside a specific view of a malignant breast the image is discarded from the dataset e.g. a lesion not visible in CC view while visible in MLO view. Images anterior and posterior to a positive breast are discarded as well. On the other hand, a negative image (label 0) is an image of a negative patient i.e. a patient without a positive breast. In addition, a one-year negative follow-up is required to confirm absence of signs of malignancy in the image. Additional data information are provided in Appendix.
Data segments The learning set is composed of 2 sources of data, S1 and S2. Each source contains images obtained with acquisition devices of 2 manufacturers Ma and Mb. Thus, 4 DSs can naturally be defined: (S1,Ma), (S1,Mb), (S2,Ma), (S2,Mb). The testing sets are composed of up to 3 sources of data, S3, S4 and S5, containing images obtained with acquisition devices from 4 manufacturers, Ma, Mb, Mc and Md, leading to 5 DSs. The numbers of malignant and benign images from all train and test DSs are given in table 1.
Cross-validation scheme For each experiment, the learning set is divided into training and validation data using a source-wise and patient-wise split: 80% (resp. 20%) of the patients of each source are used for training (resp. validation). Furthermore, the split is made such that the ratio of malignant / benign patients is similar for the validation and the training sets.
Image pre-processing and data augmentation Each image is extracted from its native DICOM storing file (Pianykh, 2010) and resized to a shape of (1152, w), the width w being adjusted accordingly to keep the same image proportions. The mask of the breast is extracted from the reshaped image using automatic thresholding and morphological operations.
Normalization and preprocessing techniques are used to standardize the FFDM images. First, the images are cropped / padded to fit a shape of 1152 ∗ 832. Then, the pixels inside the breast mask are re-normalized to have values in [0, 1]. The pixels outside mask are set to a constant value such that the average pixel value of the image is always the same. A standard augmentation pipeline
(random rotation, zoom, shearing... ) is used to augment images used for training while ensuring the malignant lesion remains visible in the augmented image.

heading: 3.1.2 MODEL TRAINING AND EVALUATION
text: Metrics As mammography data usually comes with strong class imbalance, the evaluation metrics need to be independent of labels proportions. The Weighted Logloss per image (WLL / image), the standard logloss with samples weights estimated on the evaluation (here validation) set, the Receiver Operating Characteristic curve (ROC curve) and the AUC are considered. Metrics are computed globally i.e. on the whole (validation or test) set, and also locally i.e. on the DSs taken individually. To make sure a model exhibits a similar behavior on every DS, one also considers the Operating Points (OPs) at threshold in {0.2, 0.4, 0.6, 0.8} on the ROC curve of this DS. The AUC of a DNN and the difference in AUC between 2 DNNs, ∆ AUC, are evaluated measured using 2000 Efron’s bootstraps with uniform sampling with replacement over the test set (or test data-segment). Then 95% Confidence Interval (CI), given in brackets, and P-value are provided as well for the ∆AUC, (Altman & Bland, 2011). A negative CI upper bound of ∆AUC, with p-value < 0.05 is regarded as a significant decrease in performance. A non negative CI lower bound of ∆AUC, with p-value < 0.05 is regarded as a significant increase in performance.
Neural network The considered DNN processes every image independently, it makes image wise predictions and assigns for each image a probability of malignancy. It follows a VGG-like architecture pattern (Schaffter et al., 2020) and is composed of 2 parts: a backbone and a classification head, the backbone being first pre-trained to classify lesions annotated on a subset of images from (S1,Ma), see Appendix for more details. The full model is fine-tuned to perform binary classification on 2 Tesla P100, for 150000 steps with Adam optimizer (Kingma & Ba, 2014) and a learning rate equal to 10−4. Exponential moving average with decay 0.99 of all trainable parameters are tracked, their averaged versions are then used for inference on validation and test data. A batch size of 24 samples is used (12 per GPU). A validation step is made every 1000 training steps. The checkpoint achieving the best WLL / image on the validation set is kept for inference. Tensorflow 2.3 (Abadi et al., 2015) was used for the experiments.

heading: 3.1.3 BASELINES AND COMPARED METHODS
text: Baselines Two baselines are considered and are obtained by training the model on a single DS: (S1,Ma) for baseline (1), (S1,Mb) for baseline (2). Training is done with standard balanced sampling to cope with class imbalance. Baseline 1 (resp. 2) is used as a reference in experiments where models are evaluated on (S3,Ma), (S4,Mc), (S4,Md), (S5,Mc) (resp. (S3,Mb)) to compute the differences of AUC with the other methods (c.f. ∆ AUC).
Compared methods Three strategies are benchmarked. They all aim at training the considered DNN on 2 or more DSs. The first one is called ”balance labels only” and is referred to by ”Bal. labels”. It aims at sampling equally the labels 0, 1 independently of the DSs. This boils down to consider that all the training samples come from a unique DS as often assumed in ML. The second one is called ”balance every data-segments equally” and is referred to by ”Bal. segments”. It aims at sampling equally the DSs and the labels 0, 1. In other words, it can be seen as a sampling pattern S built as if the proportion pi of every DS DSi was given by 1/M , with M the number of DSs. This way, negative and positive samples from DSi are sampled as often as negative and positive samples from DSj , with i 6= j. The last one is the proposed Period Sampling, referred to by ”P. sampling”.

heading: 3.2 EXPERIMENT 1: TRAINING ON 2 DATA-SEGMENTS (2 SOURCES, 1 MANUFACTURER)
text: In this experiment, the training / validation set is divided into 2 DSs: (S1,Ma) and (S2,Ma): images have been obtained with acquisition devices of the same manufacturer but come from 2 different institutions. Performances on the test set (S3,Ma) are reported in Table 2.
The DNNs trained with ”Bal. labels” and ”Bal. segments” do not beat the baseline on DS (S3,Ma) whereas the DNN trained with ”P. sampling” brings a small gain of performance. Such a small gain may be due to the limited amount of malignant images of the DS (S2,Ma) compared to the large number of malignant images in the DS (S1,Ma).
The global metrics and the local metrics on (S1,Ma) and (S2,Ma) exhibit a significantly large discrepancy for the DNN trained with ”Bal. labels”, see Figure 3 - blue curves. Indeed, this DNN presents a very low global WLL / image and a very (suspiciously) high global AUC / image while it also has much higher WLL / images and much lower AUCs on the DSs (S1,Ma) and (S2,Ma). It is as if the DNN trained with ”Bal labels” learned a wrong task: classify the malignant images from (S1,Ma) vs the benign images of (S2,Ma). Moreover, the WLL / image (resp. the AUC) on (S2,Ma) indicates that the DNN has over-fitted very quickly this DS and does not learn from it after 5000 learning steps. Everything appears as if this second DS (S2,Ma) is never used in the end. The DNN trained with ”Bal. segments” seems to over-fitting more slowly the DS (S2,Ma) as suggested by the local metrics on this DS, Figure 3 - green curves. On the contrary, the DNN trained with ”P. sampling” demonstrates much more coherent and reliable global and local metrics, Figure 3 - yellow curves. The global metrics look like an average of local metrics both for the WLL / image and the AUC / image. Plus, the DNN does not seem to quickly over-fit the segment (S2,Ma): it seems to learn at the same pace on both DSs. Last but not least, as a (small) gain of AUC is observed on the test set, this implies that, the DNN trained with ”P. sampling” is able to learn from both DSs and to leverage the second DS (S2,Ma) to improve the test performances.
The ROC curves obtained on the validation sets of DSs (S1,Ma) and (S2,Ma) are represented for each model as well as the OPs at thresholds 0.2, 0.4, 0.6, 0.8, see Fig 4. On the whole validation set, the DNNs trained with ”Bal. labels” and ”Bal. segments” seem globally better than the DNN trained with ”P. sampling”. Indeed, they reach higher global AUC, see black curves in Fig 4. Yet, on the DSs taken individually the DNN trained with ”P. sampling” performs better than the others and reduces the discrepancy between OPs of the ROC curves of both DSs. This likely indicates that the DNN trained with ”P. sampling” works similarly on both DSs. On the contrary, the DNN trained with ”Bal. labels” leads to un-synchronized OPs between the ROC curves of the 2 DSs. Plus the OPs of (S1,Ma) look like opposite to the ones of (S2,Ma) suggesting that this DNN may actually classify the malignant images from (S1,Ma) vs the benign images from (S2,Ma).

heading: 3.3 EXPERIMENT 2: TRAINING ON 4 DATA-SEGMENTS
text: In this experiment, the training / validation set is composed of 4 DSs: (S1,Ma), (S1,Mb), (S2,Ma) and (S2,Mb). 2 baselines are considered: a DNN trained on (S1,Ma) (resp. (S1,Mb)) which is used when evaluating / comparing the methods on (S3,Ma) (resp. (S3,Mb)). The models performances on the test sets (S3,Ma) and (S3,Mb) are reported in Table 3.
The DNNs trained with ”Bal. labels” and ”Bal. segments” do not beat the baseline on (S3,Ma) and exhibit a negative ∆ AUC which is statistically significant. On the other hand, the DNN trained with ”P. sampling” brings a small gain of performance which is is statistically significant. On (S3,Mb), all DNNs beat the baseline, but the one trained with ”P. sampling” gives a larger gain of AUC. Synchronization of OPs on the ROC curves is again impacted, c.f. Figure 6 in Appendix.
Last but not least, the DNN trained with the proposed ”P. sampling” seems to better generalize on new manufacturers than the DNNs trained with other compared approaches and seems slightly better than the baseline trained on DS (S1,Ma) although the gain is not significant, see Table 4.

heading: 4 DISCUSSION
text: Periodic sampling exhibits some drawbacks that are now discussed.
The main drawback of the proposed method is related to the segmentation of the learning set into DSs as we have little information on how building DSs. Two questions arise: (1) ”when does one have to segment the learning set ?” and (2) ”how ?”.
By experimenting, we came up with 3 empirical ways to know whether a set of data should be divided into DSs. First, a poor generalization performance on a left out source of data is likely to indicate that the learning set should be split into DSs as in experiments 1 and 2. Second, a very surprisingly nice validation metrics is also a good indicator that the learning set should be divided into DSs, see Figure 3. Finally, good knowledge of the learning set and a careful look at positive and negative samples numbers in natural subsets of data may indicate a required division of the learning set into DSs, see Table 1. An unbalance between negative samples from a subset and positive samples from another one is likely to indicate a needed segmentation.
Through an iterative process, one can define the DSs relevant to the the learning set and the task to perform: (a) consider N = 1 DS (b) train a model (c) evaluate the models on the potential test DSs of interest, (d) consider partitioning the learning set into N > 1 DSs according to some known characteristics (e.g. source, manufacturer, acquisition device version, population characteristics). Only accumulated knowledge of the data led us to consider the sources and the manufacturers as a segmentation levels to split the learning set into DSs. A potential segmentation of the learning set according to the ethnicity (which may be already covered by the segmentation by source), the age, the biological sex of patients stand as different levels which may be worth investing to improve fairness. Still, addressing such segmentation levels implies dealing with sensible information raising concerns about data privacy and data security. The proposed approach still works when there are more than 2 segmentation levels (not shown) but the segmentation into DSs needs to be defined on some categorical criteria. Binning a continuous variable could allow one to tackle a continuous source of bias (e.g. age). Further investigation are needed to cover this aspect.
This work builds upon the assumption that 2 DSs DSi and DSj do not have some shared information and are independent. Yet, this may be wrong at some point. Indeed, some patients may have data in different DSs when for instance a hospital has used different acquisition devices over time or when a patient goes to another hospital. Adapting the sampling proportions used to generate the sampling pattern could help to cope with this issue. Future work should investigate this subject.
The proposed approach relies on a fixed sampling pattern that cannot change with the dynamics of the training process (Hu et al., 2019; Ren et al., 2018). A dynamic and learnable sampling could be an advantage and cope with issues when 2 DSs are not independent. Alternatively, the problem addressed in this work could be tackled with sample weighting (Shimodaira, 2000) and may lead to similar performances provided that one samples all labels and DSs in equal proportions to avoid having batches containing samples of a single label or a single DS.
This paper focuses on binary classification through the application of breast cancer prediction, a concrete but also restricted scenario. Yet, the formalism is general enough to be tried with other modalities of data and / or other learning tasks. This shall be covered in follow up studies

heading: A APPENDIX

heading: A.1 ADDITIONAL INFORMATION ABOUT DATA
text: This section contains additional information about the training and testing data. Table 5 provides information about the countries of origin of the sources and the number of images per status and type of view. Table 6 provides information about the types of the malignant lesions in each DS. Table 7 provides information about the age of patients in the different DSs.

heading: A.2 MODEL ARCHITECTURE

heading: A.2.1 GENERAL ARCHITECTURE
text: The considered DNN is a fully convolutional neural network predicting image wise probability of malignancy on FFDM images. The DNN is thus trained on whole FFDM images of shape (1152, 832) with image-wise labels as explained in the main body of the paper.
The DNN exhibits a VGG-like architecture (Simonyan & Zisserman, 2015) and adopts a 2-level architecture as depicted in Figure 5. The first part of the DNN is referred to as the Backbone. The Backbone aims at creating an intermediate representation of the full image which facilitates prediction of the final image-wise probability of malignancy. The second part of the model is the classifier and it is in charge of predicting the final image wise probability of malignancy. The Backbone and the classifier are detailed below.

heading: A.2.2 BACKBONE
text: The Backbone is built to take as input a patch of size (224, 224) and to predict its associated label. As 5 labels are possible, the Backbone outputs a vector of size 5. It is designed in a fully convolutionnal way and thus can take as input, images of arbitrary size to form an output feature map. In the case of images at resolution (1152, 832), the Backbone outputs a feature map of size (16, 11, 5). This returned feature map plays the role of the intermediate representation mentioned above. This can be seen as a score vector of size 5 for each spatial position in the grid (16, 11).
The backbone architecture is given in Figure 5. It builds upon a succession of convolution blocks of the form: (a) batch normalization Ioffe & Szegedy (2015), (b) 2D convolution with (3, 3) kernels and ReLU activation Nair & Hinton (2010), (c) 2D max pooling with (2, 2) pool size and stride of (2, 2). The numbers of kernels in each convolution block are given in Figure 5. Finally, 3 2D convolution layers with resp. 1024, 512 and 5 kernels of shapes resp. (2, 2), (1, 1) and (1, 1) and ReLU activations play the role of the patch classifier to output a vector of size 5.
A.2.3 IMAGE CLASSIFIER
To predict the final image-wise probability of malignancy, the intermediate representation built by the Backbone is fed into the classifier that processes it with convolutional blocks to obtain the final image wise prediction.
The classifier builds upon a MaxPooling layer with poolsize of (5, 5) and stride of (5, 5). Then, it is composed of 2 sub-parts which process the max-pooled feature maps. The first sub-part builds on 3 successive blocks of convolution of the form: (a) batch normalization layer, (b) ReLU activation (c) 2D convolution with resp. 16, 8 and 2 kernels of size resp. (3, 2), (1, 1) and (1, 1). This sub-part returns a vector of size 2. The second sub-part is composed of a batch normalization layer and a convolution layer with 2 kernels of size (3, 2). This sub-part returns a vector of size 2 as well. The outputs of the sub-parts are finally summed up to form the final image wise prediction.

heading: A.2.4 REMARK ON TRAINING
text: The 2-level architecture is also reflected in the training procedure which is a two-step process. First, the Backbone is pre-trained on patches to predict their labels. Then, the Backbone weights are used to initialized the full DNN weights and the DNN is then fine-tuned on whole FFDM images to make image wise predictions of malignancy.

heading: A.3 BACKBONE TRAINING
text: The training process of the Backbone is detailed below, the training / fine-tuning process of the DNN being described in the main body of the paper. Practically, the Backbone is fed with patches centered on annotated lesions and is trained to predict which label among 5 possible labels should be assigned to the patch.
A.3.1 DATA
Lesions have been annotated on some images from the DS (S1,Ma) by trained radiologists. Lesions positions and lesions labels have been assigned. Lesions have been categorized into 5 classes: healthy tissue, benign soft tissue, benign calcification, malignant soft tissue, malignant calcification. The numbers of annotated lesions and images are provided in Table 8.
Cross-validation scheme Similarly as described in the main paper, the learning set is divided into training and validation data using a source-wise and patient-wise split: 80% (resp. 20%) of the
patients of each source are used for training (resp. validation). Furthermore, the split is made such that the ratio of malignant / benign patients is similar for the validation and the training sets.
The same train-validation split as described in the paper is used to train the Backbone. This means that the learning samples of a patient used for training (resp. validation of) the BackBone, are also used for training (resp. validation of) the DNN.
Image pre-processing and data augmentation Similarly as in the paper, normalization and preprocessing techniques are used to standardize the FFDM images. A patch of size nearly equal to (224, 224) (see below why) is first cropped from a FFDM image. The pixels inside the breast mask are re-normalized to have values in [0, 1]. The pixels outside mask are set to a constant value such that the average pixel value of the image is always the same. A standard augmentation pipeline (random rotation, zoom, shearing... ) is used to augment images used for training and to return an augmented patch of size (224, 224). No augmentation technique is used for validation patches.
The augmentation transform is always pre-computed, to know exactly the dimension of the patch to crop. This way, every pixel in the augmented patch is an output, through the augmentation transform, of a pixel from the original patch: the transform is surjective. This is why the cropping step, for training patches, does not return patches with an exact shape of (224, 224) but with a shape of nearly (224, 224).

heading: A.3.2 TRAINING SETUP
text: The Backbone is trained to minimize the categorical cross-entropy defined for 5 classes. The training process is done on 20000 learning steps, a sufficiently large number of steps to see convergence. Samples labels are balanced within a batch to fight potential issues related to class imbalance. The training relies on Adam optimizer (Kingma & Ba, 2014) and a learning rate equal to 10−3. Exponential moving average with decay 0.99 of all trainable parameters are tracked, their averaged versions are then used for inference on validation data. The training is done on 2 Tesla P100 and a batch size of 64 samples is used (32 per GPU).
A validation step is made every 500 training steps. The Weighted Logloss per lesion (WLL / lesion), the standard logloss with samples weights estimated on the validation set was used to monitor the training progress. The checkpoint achieving the best Weighted Logloss per lesion (WLL / lesion) on the validation set is kept to initialize the DNN weights.
The models and the experiments were carried out with Tensorflow 2.3 (Abadi et al., 2015).

heading: A.4 ADDITIONAL FIGURE FOR EXPERIMENT 2
text: On top of AUC gain - loss training a DNN with ”Bal. labels” leads to a poor synchronization of OPs on the ROC curves obtained on (S3,Ma) and (S3,Mb), see Figure 6 - left. This suggests that the DNN works in 2 different regimes on the 2 test DSs: a non desired property which stresses the manufacturer bias the DNN has integrated. On the other hand, the DNN trained with ”P. sampling” gives visually better synchronized OPs on the ROC curves of test DSs. This suggests that this DNN behaves quite similarly on them and that it is not influenced by the manufacturer to take its decisions.

Review summaries:
The authors present priodic sampling, that allows systematic sampling from multiple dataset to construct mini-batches for stochastic gradient descent algorithm, such that balances labels overall and within each batch. The periodic sampling is based upon construction of data segments, sharing similar latent properties. Periodic sampling comes to tackle skewed and heterogenous data that's typically shown in medical imaging applications and is specifically targeted in the paper in the form of breast cancer classification based on FFDM images. The authors compared their proposed sampling to 2 other sampling methods to demonstrate its benefit in improving performance. 

The authors in this paper mitigate the issue of DNN bias and difficulty in training of heterogeneous datasets by using mini-batches for SGD algorithms. They focus on the use of periodic sampling, as compared to balanced sampling within each batch but independent of data segments (DS) and balanced sampling within a batch and each DS. DSs are homogenous subsets of a dataset that share some attributes. In this paper, the authors primarily aim at binary classification and efficiently training a Deep Neural Network (DNN) for breast cancer prediction from mammography images (Full-Field Digital Mammography (FFDM) dataset that has been created from multiple sources of data and manufacturers . They ensured the ratio of malignant to benign is consistent throughout the validation and training sets. Another contribution of this paper is the benefits of (periodic) sampling. The authors also train a DNN with a mini-batch SGD-like optimization algorithm. The authors use AUC and ROCs for metric evaluations. They have set 3 benchmarks to compare the performances - “Bal. labels” (balance labels only), “Bal. segments” (balance every DS equally) and “P. sampling” (periodic sampling). In experiment 1, they identified that DNNs trained with Bal. labels and Bal. segments performed globally better than the DNNs trained with P. sampling. However, in experiment 2,  DNNs trained with P. sampling showed better performance gain than the other two DNNs. Also, according to this paper, DNNs trained with P. sampling generalize better on new manufacturers. The main motivation behind using periodic sampling was achieving a better performance gain using DNNs. In order to identify exactly when data segmentation should be used, the authors came up with 3 ways - a) when generalization performance is poor, b) validation metrics to identify the goal, c) good knowledge of the learning set. The authors concluded this paper by discussing the performance of periodic sampling on the dataset (with the claim that it improves the performance of the DNN models), the ways to identify when segmentation should be done and the assumptions they made during this.

This paper looks at the problem of training deep neural models with mammography data from various sources and manufacturers of acquisition device. It shows that naive approaches of training models by sampling from the various subsets perform poorly. The authors propose an algorithm for mini-batch sampling that overcomes this challenge and evaluate it on a large unspecified mammography dataset.

This paper presents a method to build mini-batches of training data using data segments (DS) and periodic sampling. The idea is to define DSs over the training set that correspond to a real split of the data. For example, in the case of mammogram images, the manufacturer of the imaging equipment could be used to split a dataset in 2 DS (all images obtained using equipment from manufacturer#1, same for manuf#2). The author's proposed method for sampling mini-batches using DSs is a form of periodic sampling, where both labels and DS are alternated to form a mini-batch. The paper then investigates the differences in performance when using 3 sampling methods for the binary classification of mammogram images: 1) BL (balanced labels) is a baseline that ignores DSs 2) BS (balanced segments) alternates labels and DS equally 3) PS (periodic sampling) alternate labels and DS based on the proportion of the least represented label in each DS. Results show that PS outperforms BL and BS for 2 experiments with different number of DSs.