Dataset Curation Beyond Accuracy

1 INTRODUCTION . Neural networks often require large amounts of labeled data to perform well , making data curation a crucial but costly aspect of deployment . Thus , researchers have studied dataset properties such as distributional shift ( Miller et al. , 2020 ) and the bias in crowd-sourced computer vision datasets ( Tsipras et al. , 2020 ) among others . Often , the evaluation criteria in such studies is downstream prediction accuracy . However , neural networks are increasingly deployed in sensitive real-world applications such as medicine ( Caruana et al. , 2015 ) , self-driving cars ( Bojarski et al. , 2016 ) , and scientific analysis ( Attia et al. , 2020 ) , where not only accuracy matters but also calibration . Calibration is the extent to which model certainty reflects the actual correctness likelihood . Calibration can be important when costs of false positives and false negatives are asymmetric , e.g. , for a deadly disease with cheap treatment , doctors might initiate treatment when the probability of being sick exceeds 10 % . Beyond simple classification , calibration can be important for beam-search in NLP ( Ott et al. , 2018 ) and algorithmic fairness ( Pleiss et al. , 2017 ) . Calibration in machine learning has been studied by e.g Zadrozny & Elkan ( 2001 ) ; Naeini et al . ( 2015 ) . Niculescu-Mizil & Caruana ( 2005 ) have shown that small scale neural networks can yield well-calibrated predictions . However , it has recently been observed by ( Guo et al. , 2017 ) that modern neural networks are ill-calibrated , whereas the now primitive Lenet ( LeCun et al. , 1998 ) achieves good calibration . In this work , we take a complementary approach ; instead of focusing on network architecture , we study how calibration is influenced by dataset properties . We primarily focus on computer vision and perform extensive experiments across common benchmarks and more exotic datasets such as satellite images ( the eurosat dataset ( Helber et al. , 2019 ) ) and species detection ( the iNaturalist dataset ( Van Horn et al. , 2018 ) ) . We consistently find that dataset properties can significantly affect calibration , causing effects comparable to network architecture . For example , we consider the ubiquitous problem of class imbalanced datasets , a common issue in practice ( Van Horn et al. , 2018 ; Krishna et al. , 2017 ; Thomee et al. , 2016 ) . For such datasets , the miscalibration is not uniform but instead varies across the different classes . This problem persists even when common strategies to mitigate class imbalanced are employed . Another practical concern is generating high-quality labels via e.g . crowdsourcing ( Karger et al. , 2011 ) . We demonstrate how labeling quality affects calibration , with noisier labels resulting in worse calibration . Additionally , we show that just the size of the dataset has a strong effect on calibration . This also holds when one artificially increases the dataset size by data augmentation . We motivate our findings by considering the geometry of the cross-entropy loss and utilizing recent results on network expressivity ( Yun et al. , 2019 ) . If the dataset is sufficiently small compared to the number of parameters , we argue that the lack of minimizer for the cross-entropy loss biases the network to high confidence and poor calibration . Our results highlight an underappreciated aspect of calibration and suggest that for sensitive applications , one should measure calibration during dataset curation . 2 BACKGROUND . Calibration . Calibration has a traditional place in machine learning ( Zadrozny & Elkan , 2001 ; Naeini et al. , 2015 ) . Before the advent of modern deep learning , Niculescu-Mizil & Caruana ( 2005 ) showed that neural networks can yield well-calibrated predictions for classification . However , Guo et al . ( 2017 ) showed that modern neural networks are ill-calibrated . Modern neural networks are modeled as e.g . resnet ( He et al. , 2016 ) or densenets ( Huang et al. , 2016 ) . It is important to note that accuracy and calibration do not necessarily follow each other , but can move independently – modern neural networks are ill-calibrated , but still yield excellent accuracy . Beyond image classification , the importance of calibration in NLP has further been studied by Ott et al . ( 2018 ) and its relationship to fairness by Pleiss et al . ( 2017 ) . Metrics for Calibration . We let { xi } ∈ Rn×dx be a dataset of n datapoints with dx features and take { yi } to be the labels . Following Guo et al . ( 2017 ) , we assume that a neural network h outputs h ( xi ) = ( p̂i , ŷi ) , where ŷi is the predicted class and p̂i is the estimated probability that the prediction is correct . For evaluating calibration , we divide the interval [ 0 , 1 ] into M equally sized bins and assign predictions to bins based upon p̂ . Within each bin Bm we define the accuracy as acc ( Bm ) = 1 |Bm| ∑ i∈Bm 1 ( ŷi = yi ) . Similarly , we define the confidence as conf ( Bm ) = 1 |Bm| ∑ i∈Bm p̂i . For a well-calibrated model , we would expect the confidence and accuracy of each bin to be close to each other . Calibration error can be measured by their difference , evaluated on the test set . The resulting metric is known as the expected calibration error ( Naeini et al. , 2015 ) , often abbreviated as ece . Mathematically , it is defined as follows : ece = M∑ m=1 |Bm| n ∣∣∣∣acc ( Bm ) − conf ( Bm ) ∣∣∣∣ ( 1 ) 3 EXPERIMENTS . Experimental setup . We consider the following computer vision datasets . Cifar10 & Cifar100 ( Krizhevsky & Hinton , 2010 ) which contains 50,000 RGB images spanning ten or hundred classes respectively . Classes are balanced . Eurosat ( Helber et al. , 2019 ) , which is a dataset of satellite images over continental Europe ; there are ten balanced classes and 27,000 images in total . iNaturalist ( Van Horn et al. , 2018 ) which is a dataset for species detection . We use the FGVC6 version ( FGVC6 , 2019 ) , compromising over 260,000 images and an imbalanced hierarchical class system compromising e.g. , species and phylum . We perform classification at the ” class ” level , resulting in nine classes . Across all datasets , we use the same architecture , Resnet50 ( He et al. , 2016 ) . We use hyperparameters from the original resnet paper ( He et al. , 2016 ) : cross-entropy loss optimized with SGD using a learning rate at 0.1 and decreased by a factor 0.1 after 50 % and 75 % of the training , a batch size of 128 , a weight decay of 0.0001 , and momentum of 0.9 . For the cifar/eurosat/inaturalist , networks are trained over 62/30/331 × 103 gradient steps , corresponding to 160 epochs for each dataset . We use randomized cropping and random horizontal flipping for data augmentation , see Appendix A for data preprocessing . Experiments are repeated five times , with mean and standard deviations reported . Calibration error is measured in expected calibration error ( ece ) as in eq . ( 1 ) , using M = 15 and evaluated on the test set . Inbalanced dataset . A common problem in practice , not necessarily found in benchmark datasets , is class imbalance ( i.e. , the number of available samples varies between classes ) , see Van Horn et al . ( 2018 ) ; Krishna et al . ( 2017 ) ; Thomee et al . ( 2016 ) . Here we study how imbalanced datasets affect the calibration error for individual classes . Whereas the iNaturalist dataset is naturally imbalanced , the cifar and eurosat datasets are not . For these datasets , we simulate long-tailed class imbalance following Cao et al . ( 2019 ) . We randomly reorder the classes from = 1 to n , and only keep a αi−1 fraction of examples for class i . Given some desired ratio ρ between the class with the most and fewest samples , one picks α such that αn−1 = ρ . Following Buda et al . ( 2018 ) , we also consider a step-imbalance , where half of the classes are downsampled by a factor ρ . We consider ρ = 100 as done by e.g . Cao et al . ( 2019 ) and keep the test set balanced . For cifar/eurosat , we randomly chose what classes to subsample to eliminate class-specific properties . Since iNaturalist is already imbalanced , we keep it as it is , but note that the class-specific properties are correlated with class-specific imbalance . After this procedure , we train the DNNs as normal and give the average calibration error for individual classes . The results are shown in Figure 1 . Generally , classes with fewer examples have significantly higher ECE , showing how imbalance can have a significant effect on model calibration . For the iNaturalist dataset , we have some outlier classes which is likely due to class-specific effects , e.g. , the class with the most labels might be unusually hard to calibrate . Methods for Imbalanced Datasets . As class imbalanced is a problem of practical importance , there is ample work on mitigating this issue . One common strategy is to sample the dataset unevenly when generating mini-batches , attempting to obtain a roughly balanced dataset . One can both oversample minority classes ( Buda et al. , 2018 ) and undersample the majority class ( Japkowicz & Stephen , 2002 ) . Another strategy is instead to weight the objective function to give all classes approximately the same weight in the objective function . A common strategy is weight classes inversely proportional to their frequency ( Wang et al. , 2017 ) . Recently , Cui et al . ( 2019 ) has proposed to reweight based upon the ” effective ” number of samples , which is defined per a mathematical formula . We here investigate if the calibration issues of an imbalanced dataset persist when using such mitigation strategies . Thus , we consider the standard cross-entropy ( original ) , resampling inversely proportional to the frequency ( sampling ) , reweighting inversely proportional to the frequency ( weighted ) , the weighting scheme of Cui et al . ( 2019 ) ( CB ) , and the focal loss of Lin et al . ( 2017 ) ( focal ) . Additionally we consider label smoothing ( Szegedy et al. , 2016 ) ( label smooth ) . We construct imbalanced datasets as in the previous section . Due to limited computational resources , we only consider the three smallest datasets for these experiments . Calibration errors are given in Table 1 , and we give the largest and smallest calibration error among classes and the average ratio of the two . We see that issues of imbalanced calibration errors , while sometimes improving , still persist . Standard deviations are given in Table 2 in Appendix A. Label Quality . When collecting labels , for example via crowdsourcing , a common issue is label quality ( Patterson & Hays , 2012 ; Su et al. , 2012 ; Callison-Burch & Dredze , 2010 ) . For example , workers might have poor incentives to perform well or lack the necessary skills for quality labeling . To study the effects of potentially mislabeled data , we artificially inject symmetric noise into the training set . This is done by selecting a random subset of the training set corresponding to some fixed fraction , and then shuffling the labels of this set . This setup follows conventions in label noise literature ( Patrini et al. , 2017 ; Han et al. , 2018 ) . Given these noisy labels , we train the networks and evaluate them on the original test-set ( which has no noise ) . We consider five levels of label noise in increments of 0.1 , starting at 0.0 . The resulting calibration errors for various noise levels are given in Figure 2 , where we see that label noise increases the calibration error across all datasets . Additionally , we consider the effects of non-uniform noise , studied by e.g . Crammer et al . ( 2006 ) . For class i , we linearly increase a noise level pi from 0.0 to 0.5 . Classes are randomly ordered . For each image with original class i , with probability pi we assign it to a new random class . The reassignment probability to class i is proportional to pi . Results are given in Figure 3 , where noisier classes suffer from worse calibration . This underscores how label quality control in e.g . crowdsourcing ( Su et al. , 2012 ) can be not only important for accuracy , but also for calibration of downstream models . Dataset size . The perhaps most common concern for collecting data is the dataset size , and model accuracy typically grows with this size ( Hestness et al. , 2017 ) . Crowdsourcing labels and bounding boxes for images is common practice , with many researchers investigating strategies to reduce needed queries ( Su et al. , 2012 ) In practice , dataset size can be limited by costs of labeling , but also by obtaining the actual data ( Suram et al. , 2017 ) . Motivated by this , we study the effect of dataset size upon the calibration error . We simply subsample the training sets of the datasets uniformly at random and thereafter train on them , comparing different sizes of the resulting dataset . We consider subsampled sizes , measured in fractions of the original size , from 1.0 to 0.2 in increments of 0.2 , and also consider 0.1 . The test set is not subsampled . The results of these experiments are given in Figure 4 . We see that smaller datasets have substantially larger calibration errors , demonstrating the dramatic effect that dataset size can have not only on accuracy , but also on calibration error . Augmentations . Beyond actual dataset size , it is common to artificially increase the size of the dataset by augmenting it , e.g. , randomly cropping the images or slightly shifting the color balance ( Cubuk et al. , 2018 ) . We have seen that the size of the dataset influences calibration , and now consider the effect of increasing the effective dataset size via augmentations . We use both randomized cropping and horizontal flipping for our training and consider removing these components while keeping other training parameters fixed . The outcome of this experiment is shown in Figure 5 , and we see that removing data augmentations significantly increases the calibration error . Viewing data augmentation as a strategy of extending the training set , we again see how smaller training sets increase the calibration error , just as in Figure 4 . NLP . While we primarily focus on computer vision , we here consider experiments in NLP . Language models often generate text via beam-search , and it has been observed that calibration is important for this process ( Ott et al. , 2018 ) . Here we investigate the effect of dataset size on calibration in NLP . We considering translation of the IWSLT ’ 14 German to English dataset ( Cettolo et al. , 2014 ) using transformer architectures ( Vaswani et al. , 2017 ) with code and default hyperparameters from the publicly available fairseq codebase ( Ott et al. , 2019 ) . As before , we simply subsample the training set uniformly at random for variable sizes and train the transformer with all other training parameters fixed . Figure 6 shows how the mean calibration error , with standard deviations as error bars , varies during training . Again we see how the dataset size influences the calibration . It is natural to guess that there might be word-level calibration issues too , e.g. , rare words might have worse calibration .

Summary: This work is an empirical survey of the calibration problem with convnets. The authors use several existing benchmark datasets and create synthetic class-imbalance for datasets that are initially balanced. They then extend the well-known results on higher prediction error of minority class, to its calibration error. The work investigates several existing methods that alleviate prediction error in imbalanced datasets and examine their effect on calibration error. At last, the effect of dataset size and data augmentation on calibration error is reported. Later on, the effect of random label noise is also examined. The observations, although not surprising, have not been reported before