When high-performing models behave poorly in practice: periodic sampling can help

Abstract: Training a deep neural network (DNN) for breast cancer detection from medical images suffers from the (hopefully) low prevalence of the pathology.
For a sensible amount of positive cases, images must be collected from numerous places resulting in large heterogeneous datasets with different acquisition devices, populations, cancer incidences.
Without precaution, this heterogeneity may result in a DNN biased by latent variables a priori independent of the pathology.
This may be dramatic if this DNN is used inside a software to help radiologists to detect cancers.
This work mitigates this issue by acting on how mini-batches for Stochastic Gradient Descent (SGD) algorithms are constructed.
The dataset is divided into homogeneous subsets sharing some attributes (\textit{e.g.} acquisition device, source) called Data Segments (DSs).
Batches are built by sampling each DS periodically with a frequency proportional to the rarest label in the DS and by simultaneously preserving an overall balance between positive and negative labels within the batch.
Periodic sampling is compared to balanced sampling (equal amount of labels within a batch, independently of DS) and to balanced sampling within DS (equal amount of labels within a batch and each DS).
We show, on breast cancer prediction from mammography images of various devices and origins, that periodic sampling leads to better generalization than other sampling strategies.

Reviews and author responses:

The paper proposes a new sampling procedure, which works by dividing the dataset into homogeneous subsets and sampling each subset periodically with a frequency proportional to the rarest label in the subset while preserving class balance within each batch. On the task of breast cancer prediction from mammography images, the experiments demonstrate that the proposed periodic sampling procedure leads to better generalization performance than existing sampling procedures. Strengths:
- Overall, the paper takes an interesting approach to an important problem – the heterogeneity in datasets makes it challenging for models to generalize, especially in medical settings where the images are often generated with different acquisition devices. The general idea is to apply periodic sampling to construct mini-batches for SGD in tackling the challenge of dataset heterogeneity.

Weaknesses:
- The empirical evaluations are not sufficiently convincing – the work should demonstrate its utility and generalization beyond the task by evaluating on other standard tasks and public datasets.
- The paper is also lacking a more thorough performance analysis, such as detailed ablation studies to understand the utility of different components in the procedure. Moreover, given that the proposed procedure has many heuristic components, I would also expect to see more discussion on why the proposed procedure works better in terms of generalization than existing ones and why the existing ones fail, which could help shed light on the significance of their contribution compared to previous work.
 This paper proposes a new sampling procedure to tackle the challenge of dataset heterogeneity in medical imaging and illustrates its generalization performance on breast cancer prediction with mammography images. However, the paper is limited in its narrow evaluation and lacks a thorough analysis to demonstrate its significance or novelty, especially given the fact that the proposed procedure has many heuristic components, so I would suggest a rejection for the paper.

The authors present priodic sampling, that allows systematic sampling from multiple dataset to construct mini-batches for stochastic gradient descent algorithm, such that balances labels overall and within each batch. The periodic sampling is based upon construction of data segments, sharing similar latent properties. Periodic sampling comes to tackle skewed and heterogenous data that's typically shown in medical imaging applications and is specifically targeted in the paper in the form of breast cancer classification based on FFDM images. The authors compared their proposed sampling to 2 other sampling methods to demonstrate its benefit in improving performance.  The paper tackles a most relevant topic in the applicative ML field, it is well written and structured including a thorough review of the relevant literature and the experiments are well described, providing high reproducibility.

However, I have major concerns regarding the contribution of this paper based on the experimental evidence:
1) The authos rightfully defined the major challenges for the classification problem at hand in the introduction section: class imbalance and a non homogeneous learning set. In the following, they described different approach to tackle these challenges. However, they only provided theoretical considerations for why methods other than sampling (i.e. transfer learning / domain adaptation) are not applicable. If this is not demonstrated with scientific/ experimental evidence, the suggested method cannot be claimed to offer a better solution to this general problem. The paper is then limitted to the narrow scope of sampling methods. Additionally, for this narrow scope, it's important to present the state-of-the-art method as a comparision to the proposed one, and I'm not sure that is the case here, or at least it was not well supported. This generally questions contribution of this work in the wider context.

2) Impaired conclusions regarding the impact/ benefit of the proposed sampling is presented again in the experiment section 3.2, page 7, figure 3. I'm not sure I follow the line of thought that's presented there, and it should at least be better clarified in the text: The authors admit that the global metrics are inferior for the periodic sampling - since the problem is to tackle generalization, these are the important metrics. The authors state that "a ( smalll) gain of AUC is observed on the test set" - where in the figure is that shown? From what I see it's only demonstrated on (S1, Ma), i.e. on local metric. While the authors state that "it seems to learn at the same pace on both DSs" - but importantly, this is without reaching a better performance on (S2, M2) and significantly worse on the global one.
My conclusions from the presented figure is differs for the authors'. To me it seems that (M2, Sa) is simply a "bad" dataset, i.e. one that you can not learn much from (because it's too small or with unhelpful distributions). The bal. labels mehtod seems to be better than the proposed periodic sampling to buffer the negative effect of the this dataset in the global performance.

Minor comments:
- The criteria for splitting into DSs provide an important discussion, yet are not very clear and therefore not very generalizable.
- The authors mention that a "surprisingly nice validation metrics" is a good indicator to divide into DSs. Can you exapain/ support this conclusion?
- page 3, 2.2, general description, typo: to quickly --> too quickly
- page 9, second paragraph, typo: the the learning --> the learning The authors present a well written paper of a sampling method to tackle the prevalent issue of skewed heterogenous data in medical images classification.
However, they do not provide comparison of their method to a so called state of the art to handle these robustness issues. Their conclusion of the contribution based on their experimental evidence is lacking and therefore questions the contribution of this paper to the field.

The authors in this paper mitigate the issue of DNN bias and difficulty in training of heterogeneous datasets by using mini-batches for SGD algorithms. They focus on the use of periodic sampling, as compared to balanced sampling within each batch but independent of data segments (DS) and balanced sampling within a batch and each DS. DSs are homogenous subsets of a dataset that share some attributes. In this paper, the authors primarily aim at binary classification and efficiently training a Deep Neural Network (DNN) for breast cancer prediction from mammography images (Full-Field Digital Mammography (FFDM) dataset that has been created from multiple sources of data and manufacturers . They ensured the ratio of malignant to benign is consistent throughout the validation and training sets. Another contribution of this paper is the benefits of (periodic) sampling. The authors also train a DNN with a mini-batch SGD-like optimization algorithm.
The authors use AUC and ROCs for metric evaluations. They have set 3 benchmarks to compare the performances - “Bal. labels” (balance labels only), “Bal. segments” (balance every DS equally) and “P. sampling” (periodic sampling). In experiment 1, they identified that DNNs trained with Bal. labels and Bal. segments performed globally better than the DNNs trained with P. sampling. However, in experiment 2,  DNNs trained with P. sampling showed better performance gain than the other two DNNs. Also, according to this paper, DNNs trained with P. sampling generalize better on new manufacturers. The main motivation behind using periodic sampling was achieving a better performance gain using DNNs.
In order to identify exactly when data segmentation should be used, the authors came up with 3 ways - a) when generalization performance is poor, b) validation metrics to identify the goal, c) good knowledge of the learning set.
The authors concluded this paper by discussing the performance of periodic sampling on the dataset (with the claim that it improves the performance of the DNN models), the ways to identify when segmentation should be done and the assumptions they made during this. The authors in this paper attempt to improve the performance on Breast Cancer Dataset images using DNNs and periodic sampling. They use two data sources and 2-4 manufacturers for training in their experiments.
Some of their strong points have been highlighted below:
- As far as literature research is concerned, there are only few works that have done periodic sampling on such datasets until now. The idea of the proposed method seems to be novel and well-motivated, however, there is insufficient information on a larger scale of experiments.
- The paper has proper definitions for new terms and explanations for the dataset, methods and evaluations. 

Some points of improvement are:
- The paper lacks clarity in some sections. The notations were ambiguous owing to the use of multiple sources and manufactures. Additionally, it requires multiple rereads to grasp what the authors are trying to say.
- The authors did not explain why they used a heterogeneous dataset, with multiple sources and manufacturers. What happens if the dataset contains only 1 source, 1 manufacturer?
- The authors could have expressed class or source wise accuracies for their experiments, for additional and clear comparisons. 
- They state they are using SGD-like optimizers, however, do not explain what “SGD-like” means. 
- The proposed method only discusses the observations of periodic sampling and then the benefits. The authors do not mention if this could be extrapolated to various other datasets.
- The authors have done some previous study but have not compared their results/performance with other datasets or relevant existing methods. Because of this, I am not sure if periodic sampling will globally improve the performance of DNNs on different datasets. 

There were also some typos/questions throughout the paper.
- In the sentence “This paper identifies 2 main locks” on page 2, is “locks” a typo?
- In Section 2.2, for general description, the authors state that the “DNN should work on all DSs of interest”. What do they mean by data segments “of interest”? 
- There are some incomplete sentences in the paper such as Section 4, “we have little information on how building DSs. Two questions”. The authors in this paper propose a novel approach of using periodic sampling with DNNs to efficiently train a DNN for breast cancer prediction from mammography images. They also perform binary classification with an SGD-like optimizer. 
There have been only a few research works in using periodic sampling, which helps in understanding the novelty of this approach. The idea is well-motivated as well. The authors have described the dataset, methods used, and evaluation well. 
The authors could improve some sections by providing additional explanation on their experiments and choice for the heterogeneous dataset. Also, additional evaluation metrics would help the reader understand just how beneficial periodic sampling could be. There are some typos in the paper that would need to be corrected.
All in all, the paper carries a novel approach to an existing problem, however, the authors could have expanded some sections with a more precise explanation, such as why they performed only 2 large experiments, instead of multiple smaller ones or why periodic sampling has not been compared to other relevant existing methods, even though the approach is novel but the method is well known. 

This paper looks at the problem of training deep neural models with mammography data from various sources and manufacturers of acquisition device. It shows that naive approaches of training models by sampling from the various subsets perform poorly. The authors propose an algorithm for mini-batch sampling that overcomes this challenge and evaluate it on a large unspecified mammography dataset. Strengths

1. This is a generally clearly written and interesting paper. It highlights an important problem in the medical imaging domain, not limited to mammography.
2. The problem set up and algorithm description are clear and the example in figure 2 of the sampling approach helpful.
3. Image preprocessing and labelling seem quite reasonable. Research methodology is good. The metrics, generation of confidence intervals and statistical approaches to evaluate the models is strong.

Weaknesses

1. There are several typographical errors and odd uses of English that should be fixed.
2. On page 2, when introducing the problem, it would be helpful to use the same notation in the paragraph above figure 1 as in the figure and perhaps to refer to the colour (green and blue) in the text. It's hard to see how $dnn_1$ and $dnn_{1 \cup 2}$ match to figure 1 on a first quick reading.
3. The related work motivating the research is very weak. That said, the motivating example once understood in nice.
4. No details are given of the source of the datasets or manufacturers of the data acquisition equipment, which affects the reproducibility of the research. Apart from this, the description of the data is generally clear. Results are given only on one unavailable dataset (or set of datasets). It would be interesting to see if the results generalise to other mammography datasets, clinical image problems, image problems, and more generally to other problems. 
5. Are the differences in experiment 1 are influenced more by the differences in class imbalance between the datasets or are the issues more about manufacturer? It would be nice to see some experiments that try to explore this.
6. Last sentence of page 7. I found this difficult to understand. What is meant by the OPs looking opposite?
7. The paper would be enhanced with an ethics statement.
 Clear well-written paper on an important problem. However, empirical results are on an unspecified and unavailable dataset which reduces reproducibility and it is unclear whether the results generalise to other clinical imaging problems or other problems in general. It in unclear what is the state of art being compared to.

This paper presents a method to build mini-batches of training data using data segments (DS) and periodic sampling. The idea is to define DSs over the training set that correspond to a real split of the data. For example, in the case of mammogram images, the manufacturer of the imaging equipment could be used to split a dataset in 2 DS (all images obtained using equipment from manufacturer#1, same for manuf#2). The author's proposed method for sampling mini-batches using DSs is a form of periodic sampling, where both labels and DS are alternated to form a mini-batch. The paper then investigates the differences in performance when using 3 sampling methods for the binary classification of mammogram images: 1) BL (balanced labels) is a baseline that ignores DSs 2) BS (balanced segments) alternates labels and DS equally 3) PS (periodic sampling) alternate labels and DS based on the proportion of the least represented label in each DS. Results show that PS outperforms BL and BS for 2 experiments with different number of DSs. This paper argues that training mini-batches should not only be label-balanced, but also balanced in terms of biases that may exist in the dataset itself. This is a good observation that stems from their experience with mammogram data. They show that modest improvements can be obtained by identifying these biases and then using their PS method to balance the minibatches. 

Overall, the paper is well-written and easy to follow. The proposed PS method is simple and easily reproducible, so most of the paper is devoted to describing the data and the experiments. 

In principle, the approach is general and could be applied to any type of data, as long as relevant DS can be identified. It provides a  way to avoid that important but rare examples become overwhelmed by the masses, a situation that tends to arise in medical settings. Naturally, this type of biased sampling also has a drawback in that it may over-favor some rare examples, bringing the accuracy down for the more common examples. In the end, the gain in performance highly depends on the data. The paper offers some hints in the discussion section on how to identify whether a dataset would benefit from this type of sampling. However, those are only vague guidelines and it is unclear how to apply them in practice. 

This method seems a little bit like a desperate attempt to fight biases in the dataset. In a perfect world, the training, validation and test set can be sampled uniformly without bias from the pool of data. In that case, the proposed sampling method is not useful. As it turns out, in real settings (and in particular in the medical world), the train/valid/test are often not sampled uniformly and do exhibit biases. For example, a test set was obtained using a different acquisition device than the training/validation set. And, if it turns out that a small part of the training set has been acquired on a similar device, one could over-emphasize that small part during training, which would naturally improve the results on that particular test set. This type of scenarios do occur in real life, however, it is not a desirable situation. Since labeled data is hard to come by in the medical world, once biases are identified, it is natural to try to combat them. The proposed segmentation method is a way to combat such biases.  

One aspect that i found missing, is the effect of label imbalance within the segments. What happens if the labels are balanced within the segments but the segment size is imbalanced ? It seems there may be a bound on the possible gains that would depend on the label imbalance within segments. This could be explored analytically or empirically with a toy dataset.

The paper only experiments the method on one mammogram dataset. In that dataset, the ratio of positive examples varies greatly among the DSs (from 44% to 0.34%). This extreme variance is probably what makes it possible to eke out a benefit when using the proposed approach. Other datasets with less extreme imbalance would presumably benefit much less. 
 A simple approach for balanced sampling for mini-batch SGD training is shown to provide modest gains in the case of extreme label imbalance across segments of the data. More experiments are needed to quantify its benefits for other datasets. Still, in some real-life scenarios (not the ideal, unbiased case) it provides a principled way to favor rare but 'deemed-important' examples. However, it lacks an analytical and empirical study to quantify the pro and cons of the method based on the label imbalance within data segments.



Meta-review: This submission has been withdrawn. 

The reviews are of good quality. The authors should consider writing two separate papers: one about the problem and solution from an ML perspective, and the other about the application to radiology. Papers that provide a new method in the context of a single application domain run the risk of making a contribution to neither, and of being evaluated by reviewers who are not experts in both.

Decision: Reject