When high-performing models behave poorly in practice: periodic sampling can help

1 INTRODUCTION . Breast cancer screening leads to an earlier detection of breast cancer resulting in an improved prognosis and a reduced mortality ( Marmot et al. , 2013 ) . During a screening exam , Full-Field Digital Mammography ( FFDM ) images of the breasts of a patient are acquired . They are then analyzed by 1 or 2 radiologist ( s ) who look ( s ) for signs of malignancy . If any sign of malignancy is observed , the patient is recalled to assert the final diagnosis with complementary exams . Despite constant technological progress , cancers may still be missed while the recall rate remains high with a limited fraction of recalled patients actually diagnosed with cancer ( Rawashdeh et al. , 2013 ) . Machine Learning ( ML ) based screening systems may help radiologists detecting cancers and improve their sensitivity , specificity and reading time , ( Pacilè et al. , 2020 ; McKinney et al. , 2020 ; Schaffter et al. , 2020 ; Shen et al. , 2019 ; Kooi et al. , 2017 ) . Indeed , Deep Neural Networks ( DNNs ) can help to solve this problem initially formalized as a binary classification problem . Recent approaches build upon breasts symmetry ( Brhane Hagos et al. , 2018 ; Kooi & Karssemeijer , 2017 ) , the patient history ( Kooi & Karssemeijer , 2017 ) or the whole set of exam images ( Geras et al. , 2017 ) to enhance classification performances . Other approaches further aim at localizing the malignant lesion ( s ) in the breast ( Shen et al. , 2021 ; Pedemonte et al. , 2020 ) or improving the generalization of DNNs trained on FFDM images but used on Digital Breast Tomosynthesis ( DBT ) images , 3D images of the breast ( Matthews et al. , 2021 ; Singh et al. , 2020 ) . It is hard to train a DNN on FFDM images that aims to be used in a software delivering nearly real time recommendations for a radiologist investigating screening exams . First , it requires a large quantity of labeled training data but medical data sources are typically small and partially labeled . Transfer Learning ( TL ) may help to partially cope with this problem ( Morid et al. , 2021 ; Karimi et al. , 2021 ) while Domain Adaptation ( DA ) may help to reduce the potential distribution mismatch between multiple source data distributions and a test data distributions ( Crammer et al. , 2008 ; BenDavid et al. , 2010 ; Ganin et al. , 2016 ; Guan & Liu , 2021 ) . Yet , TL may deliver little performance gain ( Raghu et al. , 2019 ) , DA may not be appropriate for the setup considered here due to the potentially large number of training sources and the infinite number of testing sources and the ” nearly real time ” prediction constraint may be incompatible with the common offline setup most DA solutions embrace as they often build upon source and target domains available at the same time ( Ben-David et al. , 2007 ; 2010 ; Ganin et al. , 2016 ) . Second , each medical institution exhibits its own characteristics ( patients , radiologists experiences , sets of acquisition devices , disease incidences , ... ) , thus the DNN trained on data from such sources may still be biased : its predictions being modulated by latent uncontroled variables . If not addressed , this may result in dramatic consequences once the model is used in production ( Barocas & Selbst , 2014 ) . Finally , training a DNN on FFDM implies fighting class imbalance ( the disease incidence being hopefully small ) to avoid technical issues for ML models ( He & Garcia , 2009 ) that could amplify the potential bias issue highlighted above . The following experiment exhibits the complexity of training a DNN from only 2 sources of data . The goal was to train 2 DNNs to classify benign vs malignant FFDM images on 2 sets of data , DS1 ( 7258 malignant / 13341 benign images ) and DS2 ( 160 malignant / 21871 benign ) and to evaluate them on a left out set of data DS3 ( 1106 malignant / 14192 benign ) from another medical institution . A train / validation split was used patient-wise and source-wise : 80 % ( resp . 20 % ) of patients of a source were used for training ( resp . validation ) . The proportion of patients with cancers was kept constant in both train / val sets . The first DNN , dnn1 , was trained on DS1 , the second one , dnn1∪2 on DS1 ∪ DS2 with a Stochastic Gradient Descent ( SGD ) like algorithm . Balancing benign and malignant images within mini-batches was used during training to cope with class imbalance . The Weighted Logloss per image ( WLL / image ) , log loss with class weights , was monitored on a validation set during the training , see Figure 1 - left . The model dnn1∪2 exhibits a low global validation WLL / image on DS1 ∪DS2 , but a higher WLL / image on DS1 compared to dnn1 . The validation WLL / image on DS2 further indicates that it was over-fitted . Plus , dnn1∪2 shows a lower Area Under the ROC Curve ( AUC ) on DS3 than dnn1 , though it was trained on more data , see Figure 1 - center . Finally , the distributions of models predictions on DS3 show that dnn1∪2 predictions are shifted towards low values compared to dnn1 predictions and that even for the malignant images , see Figure 1 - right . Hardly , does dnn1∪2 benefit from data brought by DS2 . This paper identifies 2 main locks to train efficiently a DNN for breast cancer prediction from mammography images : class imbalance and a non homogeneous learning set . Class imbalance can be addressed via an appropriate sampling strategy ( He et al. , 2008 ; Chawla et al. , 2002 ) , sample weighting ( Byrd & Lipton , 2018 ) or a learnable sampling ( Hu et al. , 2019 ; Ren et al. , 2018 ) . The sampling strategy may add further benefits such as stratified sampling which may help to efficiently select samples from low variance clusters to speed up the convergence of mini-batch optimization algorithms ( Csiba & Richtárik , 2018 ; Zhao & Zhang , 2014 ) . This paper focuses on training a DNN with a mini-batch SGD like optimization algorithm on a training set , divided into a finite ensemble of independent and homogeneous sets of samples , sharing an explicit and unique set of properties , called Data Segments ( DSs ) . A Periodic Sampling strategy is proposed to build mini-batches of data by sampling each DS periodically with a frequency propor- tional to its rarest label and by balancing the labels within a batch and a DS to fight class imbalance . This paper , first introduces formally the proposed Periodic Sampling , then provides numerical evidence of its benefits and finally discusses its main limitations and directions for improvement . 2 METHOD . This paper focuses on binary classification and introduces a method to train a DNN , with a SGD like algorithm , from several segments of data while reducing the risk of using an identified bias . It aims to build appropriately the mini-batches of data with a dedicated Periodic Sampling strategy . 2.1 NOTATION AND PROBLEM STATEMENT . Let X stand for the input space , i.e . the space of FFDM images of a given shape . Let Y = { 0 , 1 } stand for the space of labels , 0 standing for a benign image , 1 for a malignant image . x , y ∈ X × Y is a learning sample : an image with its label . Let i , Ni ∈ N , then DSi = { ( xj , yj ) ∈ X × Y : j = 0 , · · · , Ni − 1 } refers to a Data Segment ( DS ) i.e . a subset of homogeneous learning samples . Let M ∈ N be the number of DSs at hand . The learning set is composed of M homogeneous and independent DSs : ∪M−1i=0 DSi . With a DNN : f : X −→ Y , and with ` standing for the binary cross-entropy , the empirical optimization problem to solve reads : f∗ ∈ arg min f 1∑M−1 i=0 Ni M−1∑ i=0 ∑ ( x , y ) ∈DSi ` ( y , f ( x ) ) ( 1 ) 2.2 PERIODIC SAMPLING . General description First , the DNN should work on all the DSs of interest . It should learn at the same pace on the training DSs to avoid the issue seen in Figure 1 and not overfit to quickly a DS with few malignant images . Third , the DNN should not be biased toward a specific label on any DS . Periodic Sampling is a two step process introduced to meet these requirements . First , it builds a sampling pattern that indicates how often samples from the DSs should be included in a mini-batch . Second , it iterates over the sampling pattern and selects a sample from the designated DS while alternating in a balanced way the labels to ensure label balance in a DS . Sampling proportion With the above notation , we define the number of samples of the least represented label ymini , inside DSi by n min i = ∑Ni−1 j=0 1yj=ymini . Now , let Nmin = ∑M−1 i=0 n min i be the sum of the samples of the least represented label inside each DS . The main idea is to feed the DNN with the samples from DSi with a proportion pi = nmini /N min during the training process . Sampling pattern Periodically selecting samples from DSi with a period 1/pi allows to feed the DNN with samples from DSi with a proportion pi . This way , even if pi is dramatically small , samples from DSi would still be selected on a regular basis . To do so , a sampling pattern S is built . This is a sequence of DS ids i ∈ { 0 , · · · , M − 1 } of length Nmin , where the number of occurrences of DSi is given by nmini . To build it , one first sorts the DS by their numbers n min i , in the ascending order . Then , one iterates over the sequence of i , nmini , to fill in the available slots of a list of length Nmin and places the indices i at nmini locations , equally spaced while avoiding potential collisions with already filled slot . This is described in Algorithm 1 . Construction of training batches Then , one builds batches of b ∈ N samples by iterating sequentially ( and infinitely ) over the sampling pattern S to get , for each sample to be included in a batch , the index i of the DS from which it should be drawn . This way , for every nearly Nmin/b + 1 seen samples by the DNN , the DNN has been trained on nearly pi × ( Nmin/b + 1 ) samples from DSi . To ensure that the labels are presented in equal proportions to the DNN , the mechanism alternates between 0 and 1 labels when selecting a sample from DS i . The batch construction pipeline is summarized in Algorithm 2 . An example of the selection of samples according to a sampling pattern with the alternating label mechanism is provided in Figure 2 Algorithm 1 : Build the sampling pattern Data : N , { DSi : i = 0 , · · · , M − 1 } Result : Sampling pattern S 1 step 1 : Compute the numbers of samples with the least represented label for every DSi : { nmini : i = 0 , · · · , M − 1 } 2 step 2 : Compute the length of the sampling pattern L← ∑M−1 i=0 n min i 3 step 3 : Build the sampling pattern : S ← array of size L filled with -1 values for i← 0 to M − 1 do pos← linspace ( 0 , Nmin , nmini ) ; for j ← 0 to nmini − 1 do p← pos [ j ] while S [ p ] 6= −1 do p← p + 1 end S [ p ] ← i end end 1 = 5 , n min 2 = 10 and n min 3 = 20 . Algorithm 2 : Build batches from sampling pattern Data : b , S , Nmin , DS = { DSi : i = 0 , · · · , M − 1 } Result : A collection of batches of size b Y ← { 0 , 1 } M randomly initialized idx← 0 while training do B ← [ · · · ] empty list for j ← 0 to b− 1 do i← S [ idx ] yi ← Y [ i ] ( x , y ) ← next samples ( x , y ) ∈ DSi such that y = yi B ← B + ( x , y ) idx = idx + 1 ( mod Nmin ) Y [ i ] = Y [ i ] + 1 ( mod 2 ) end yield B end Practically , some Deep Learning ( DL ) libraries ensure that the order of the samples indices returned by a sampling function is maintained inside the mini-batches while others do not . This may not be an issue as long as the proportions of DSs are ensured at a scale of a few ( hundreds ) learning steps / batches .

Summary: The authors in this paper mitigate the issue of DNN bias and difficulty in training of heterogeneous datasets by using mini-batches for SGD algorithms. They focus on the use of periodic sampling, as compared to balanced sampling within each batch but independent of data segments (DS) and balanced sampling within a batch and each DS. DSs are homogenous subsets of a dataset that share some attributes. In this paper, the authors primarily aim at binary classification and efficiently training a Deep Neural Network (DNN) for breast cancer prediction from mammography images (Full-Field Digital Mammography (FFDM) dataset that has been created from multiple sources of data and manufacturers . They ensured the ratio of malignant to benign is consistent throughout the validation and training sets. Another contribution of this paper is the benefits of (periodic) sampling. The authors also train a DNN with a mini-batch SGD-like optimization algorithm. The authors use AUC and ROCs for metric evaluations. They have set 3 benchmarks to compare the performances - “Bal. labels” (balance labels only), “Bal. segments” (balance every DS equally) and “P. sampling” (periodic sampling). In experiment 1, they identified that DNNs trained with Bal. labels and Bal. segments performed globally better than the DNNs trained with P. sampling. However, in experiment 2,  DNNs trained with P. sampling showed better performance gain than the other two DNNs. Also, according to this paper, DNNs trained with P. sampling generalize better on new manufacturers. The main motivation behind using periodic sampling was achieving a better performance gain using DNNs. In order to identify exactly when data segmentation should be used, the authors came up with 3 ways - a) when generalization performance is poor, b) validation metrics to identify the goal, c) good knowledge of the learning set. The authors concluded this paper by discussing the performance of periodic sampling on the dataset (with the claim that it improves the performance of the DNN models), the ways to identify when segmentation should be done and the assumptions they made during this.