Dataset Curation Beyond Accuracy

1 INTRODUCTION . Neural networks often require large amounts of labeled data to perform well , making data curation a crucial but costly aspect of deployment . Thus , researchers have studied dataset properties such as distributional shift ( Miller et al. , 2020 ) and the bias in crowd-sourced computer vision datasets ( Tsipras et al. , 2020 ) among others . Often , the evaluation criteria in such studies is downstream prediction accuracy . However , neural networks are increasingly deployed in sensitive real-world applications such as medicine ( Caruana et al. , 2015 ) , self-driving cars ( Bojarski et al. , 2016 ) , and scientific analysis ( Attia et al. , 2020 ) , where not only accuracy matters but also calibration . Calibration is the extent to which model certainty reflects the actual correctness likelihood . Calibration can be important when costs of false positives and false negatives are asymmetric , e.g. , for a deadly disease with cheap treatment , doctors might initiate treatment when the probability of being sick exceeds 10 % . Beyond simple classification , calibration can be important for beam-search in NLP ( Ott et al. , 2018 ) and algorithmic fairness ( Pleiss et al. , 2017 ) . Calibration in machine learning has been studied by e.g Zadrozny & Elkan ( 2001 ) ; Naeini et al . ( 2015 ) . Niculescu-Mizil & Caruana ( 2005 ) have shown that small scale neural networks can yield well-calibrated predictions . However , it has recently been observed by ( Guo et al. , 2017 ) that modern neural networks are ill-calibrated , whereas the now primitive Lenet ( LeCun et al. , 1998 ) achieves good calibration . In this work , we take a complementary approach ; instead of focusing on network architecture , we study how calibration is influenced by dataset properties . We primarily focus on computer vision and perform extensive experiments across common benchmarks and more exotic datasets such as satellite images ( the eurosat dataset ( Helber et al. , 2019 ) ) and species detection ( the iNaturalist dataset ( Van Horn et al. , 2018 ) ) . We consistently find that dataset properties can significantly affect calibration , causing effects comparable to network architecture . For example , we consider the ubiquitous problem of class imbalanced datasets , a common issue in practice ( Van Horn et al. , 2018 ; Krishna et al. , 2017 ; Thomee et al. , 2016 ) . For such datasets , the miscalibration is not uniform but instead varies across the different classes . This problem persists even when common strategies to mitigate class imbalanced are employed . Another practical concern is generating high-quality labels via e.g . crowdsourcing ( Karger et al. , 2011 ) . We demonstrate how labeling quality affects calibration , with noisier labels resulting in worse calibration . Additionally , we show that just the size of the dataset has a strong effect on calibration . This also holds when one artificially increases the dataset size by data augmentation . We motivate our findings by considering the geometry of the cross-entropy loss and utilizing recent results on network expressivity ( Yun et al. , 2019 ) . If the dataset is sufficiently small compared to the number of parameters , we argue that the lack of minimizer for the cross-entropy loss biases the network to high confidence and poor calibration . Our results highlight an underappreciated aspect of calibration and suggest that for sensitive applications , one should measure calibration during dataset curation . 2 BACKGROUND . Calibration . Calibration has a traditional place in machine learning ( Zadrozny & Elkan , 2001 ; Naeini et al. , 2015 ) . Before the advent of modern deep learning , Niculescu-Mizil & Caruana ( 2005 ) showed that neural networks can yield well-calibrated predictions for classification . However , Guo et al . ( 2017 ) showed that modern neural networks are ill-calibrated . Modern neural networks are modeled as e.g . resnet ( He et al. , 2016 ) or densenets ( Huang et al. , 2016 ) . It is important to note that accuracy and calibration do not necessarily follow each other , but can move independently – modern neural networks are ill-calibrated , but still yield excellent accuracy . Beyond image classification , the importance of calibration in NLP has further been studied by Ott et al . ( 2018 ) and its relationship to fairness by Pleiss et al . ( 2017 ) . Metrics for Calibration . We let { xi } ∈ Rn×dx be a dataset of n datapoints with dx features and take { yi } to be the labels . Following Guo et al . ( 2017 ) , we assume that a neural network h outputs h ( xi ) = ( p̂i , ŷi ) , where ŷi is the predicted class and p̂i is the estimated probability that the prediction is correct . For evaluating calibration , we divide the interval [ 0 , 1 ] into M equally sized bins and assign predictions to bins based upon p̂ . Within each bin Bm we define the accuracy as acc ( Bm ) = 1 |Bm| ∑ i∈Bm 1 ( ŷi = yi ) . Similarly , we define the confidence as conf ( Bm ) = 1 |Bm| ∑ i∈Bm p̂i . For a well-calibrated model , we would expect the confidence and accuracy of each bin to be close to each other . Calibration error can be measured by their difference , evaluated on the test set . The resulting metric is known as the expected calibration error ( Naeini et al. , 2015 ) , often abbreviated as ece . Mathematically , it is defined as follows : ece = M∑ m=1 |Bm| n ∣∣∣∣acc ( Bm ) − conf ( Bm ) ∣∣∣∣ ( 1 ) 3 EXPERIMENTS . Experimental setup . We consider the following computer vision datasets . Cifar10 & Cifar100 ( Krizhevsky & Hinton , 2010 ) which contains 50,000 RGB images spanning ten or hundred classes respectively . Classes are balanced . Eurosat ( Helber et al. , 2019 ) , which is a dataset of satellite images over continental Europe ; there are ten balanced classes and 27,000 images in total . iNaturalist ( Van Horn et al. , 2018 ) which is a dataset for species detection . We use the FGVC6 version ( FGVC6 , 2019 ) , compromising over 260,000 images and an imbalanced hierarchical class system compromising e.g. , species and phylum . We perform classification at the ” class ” level , resulting in nine classes . Across all datasets , we use the same architecture , Resnet50 ( He et al. , 2016 ) . We use hyperparameters from the original resnet paper ( He et al. , 2016 ) : cross-entropy loss optimized with SGD using a learning rate at 0.1 and decreased by a factor 0.1 after 50 % and 75 % of the training , a batch size of 128 , a weight decay of 0.0001 , and momentum of 0.9 . For the cifar/eurosat/inaturalist , networks are trained over 62/30/331 × 103 gradient steps , corresponding to 160 epochs for each dataset . We use randomized cropping and random horizontal flipping for data augmentation , see Appendix A for data preprocessing . Experiments are repeated five times , with mean and standard deviations reported . Calibration error is measured in expected calibration error ( ece ) as in eq . ( 1 ) , using M = 15 and evaluated on the test set . Inbalanced dataset . A common problem in practice , not necessarily found in benchmark datasets , is class imbalance ( i.e. , the number of available samples varies between classes ) , see Van Horn et al . ( 2018 ) ; Krishna et al . ( 2017 ) ; Thomee et al . ( 2016 ) . Here we study how imbalanced datasets affect the calibration error for individual classes . Whereas the iNaturalist dataset is naturally imbalanced , the cifar and eurosat datasets are not . For these datasets , we simulate long-tailed class imbalance following Cao et al . ( 2019 ) . We randomly reorder the classes from = 1 to n , and only keep a αi−1 fraction of examples for class i . Given some desired ratio ρ between the class with the most and fewest samples , one picks α such that αn−1 = ρ . Following Buda et al . ( 2018 ) , we also consider a step-imbalance , where half of the classes are downsampled by a factor ρ . We consider ρ = 100 as done by e.g . Cao et al . ( 2019 ) and keep the test set balanced . For cifar/eurosat , we randomly chose what classes to subsample to eliminate class-specific properties . Since iNaturalist is already imbalanced , we keep it as it is , but note that the class-specific properties are correlated with class-specific imbalance . After this procedure , we train the DNNs as normal and give the average calibration error for individual classes . The results are shown in Figure 1 . Generally , classes with fewer examples have significantly higher ECE , showing how imbalance can have a significant effect on model calibration . For the iNaturalist dataset , we have some outlier classes which is likely due to class-specific effects , e.g. , the class with the most labels might be unusually hard to calibrate . Methods for Imbalanced Datasets . As class imbalanced is a problem of practical importance , there is ample work on mitigating this issue . One common strategy is to sample the dataset unevenly when generating mini-batches , attempting to obtain a roughly balanced dataset . One can both oversample minority classes ( Buda et al. , 2018 ) and undersample the majority class ( Japkowicz & Stephen , 2002 ) . Another strategy is instead to weight the objective function to give all classes approximately the same weight in the objective function . A common strategy is weight classes inversely proportional to their frequency ( Wang et al. , 2017 ) . Recently , Cui et al . ( 2019 ) has proposed to reweight based upon the ” effective ” number of samples , which is defined per a mathematical formula . We here investigate if the calibration issues of an imbalanced dataset persist when using such mitigation strategies . Thus , we consider the standard cross-entropy ( original ) , resampling inversely proportional to the frequency ( sampling ) , reweighting inversely proportional to the frequency ( weighted ) , the weighting scheme of Cui et al . ( 2019 ) ( CB ) , and the focal loss of Lin et al . ( 2017 ) ( focal ) . Additionally we consider label smoothing ( Szegedy et al. , 2016 ) ( label smooth ) . We construct imbalanced datasets as in the previous section . Due to limited computational resources , we only consider the three smallest datasets for these experiments . Calibration errors are given in Table 1 , and we give the largest and smallest calibration error among classes and the average ratio of the two . We see that issues of imbalanced calibration errors , while sometimes improving , still persist . Standard deviations are given in Table 2 in Appendix A. Label Quality . When collecting labels , for example via crowdsourcing , a common issue is label quality ( Patterson & Hays , 2012 ; Su et al. , 2012 ; Callison-Burch & Dredze , 2010 ) . For example , workers might have poor incentives to perform well or lack the necessary skills for quality labeling . To study the effects of potentially mislabeled data , we artificially inject symmetric noise into the training set . This is done by selecting a random subset of the training set corresponding to some fixed fraction , and then shuffling the labels of this set . This setup follows conventions in label noise literature ( Patrini et al. , 2017 ; Han et al. , 2018 ) . Given these noisy labels , we train the networks and evaluate them on the original test-set ( which has no noise ) . We consider five levels of label noise in increments of 0.1 , starting at 0.0 . The resulting calibration errors for various noise levels are given in Figure 2 , where we see that label noise increases the calibration error across all datasets . Additionally , we consider the effects of non-uniform noise , studied by e.g . Crammer et al . ( 2006 ) . For class i , we linearly increase a noise level pi from 0.0 to 0.5 . Classes are randomly ordered . For each image with original class i , with probability pi we assign it to a new random class . The reassignment probability to class i is proportional to pi . Results are given in Figure 3 , where noisier classes suffer from worse calibration . This underscores how label quality control in e.g . crowdsourcing ( Su et al. , 2012 ) can be not only important for accuracy , but also for calibration of downstream models . Dataset size . The perhaps most common concern for collecting data is the dataset size , and model accuracy typically grows with this size ( Hestness et al. , 2017 ) . Crowdsourcing labels and bounding boxes for images is common practice , with many researchers investigating strategies to reduce needed queries ( Su et al. , 2012 ) In practice , dataset size can be limited by costs of labeling , but also by obtaining the actual data ( Suram et al. , 2017 ) . Motivated by this , we study the effect of dataset size upon the calibration error . We simply subsample the training sets of the datasets uniformly at random and thereafter train on them , comparing different sizes of the resulting dataset . We consider subsampled sizes , measured in fractions of the original size , from 1.0 to 0.2 in increments of 0.2 , and also consider 0.1 . The test set is not subsampled . The results of these experiments are given in Figure 4 . We see that smaller datasets have substantially larger calibration errors , demonstrating the dramatic effect that dataset size can have not only on accuracy , but also on calibration error . Augmentations . Beyond actual dataset size , it is common to artificially increase the size of the dataset by augmenting it , e.g. , randomly cropping the images or slightly shifting the color balance ( Cubuk et al. , 2018 ) . We have seen that the size of the dataset influences calibration , and now consider the effect of increasing the effective dataset size via augmentations . We use both randomized cropping and horizontal flipping for our training and consider removing these components while keeping other training parameters fixed . The outcome of this experiment is shown in Figure 5 , and we see that removing data augmentations significantly increases the calibration error . Viewing data augmentation as a strategy of extending the training set , we again see how smaller training sets increase the calibration error , just as in Figure 4 . NLP . While we primarily focus on computer vision , we here consider experiments in NLP . Language models often generate text via beam-search , and it has been observed that calibration is important for this process ( Ott et al. , 2018 ) . Here we investigate the effect of dataset size on calibration in NLP . We considering translation of the IWSLT ’ 14 German to English dataset ( Cettolo et al. , 2014 ) using transformer architectures ( Vaswani et al. , 2017 ) with code and default hyperparameters from the publicly available fairseq codebase ( Ott et al. , 2019 ) . As before , we simply subsample the training set uniformly at random for variable sizes and train the transformer with all other training parameters fixed . Figure 6 shows how the mean calibration error , with standard deviations as error bars , varies during training . Again we see how the dataset size influences the calibration . It is natural to guess that there might be word-level calibration issues too , e.g. , rare words might have worse calibration .

Summary: In this work, authors demonstrate that dataset properties can significantly affect calibration and suggest that calibration should be measured during dataset curation. In the field of applied AI to real-life problem, we face all the time decision-makings on what is the most effective strategy in the pipeline (eg. sampling, noise, labeling) and this paper present some evidence for those decisions.