Venue: NAACL 2022

# A Data Cartography based MixUp for Pre-trained Language Models

Seo Yeon Park and Cornelia Caragea

Computer Science

University of Illinois Chicago

spark313@uic.edu  cornelia@uic.edu

###### Abstract

MixUp is a data augmentation strategy where additional samples are generated during training by combining _random pairs_ of training samples and their labels. However, selecting random pairs is not potentially an optimal choice. In this work, we propose TDMixUp, a novel MixUp strategy that leverages **T**raining **D**ynamics and allows more informative samples to be combined for generating new data samples. Our proposed TDMixUp first measures confidence, variability, (Swayamdipta et al., 2020), and Area Under the Margin (AUM) (Pleiss et al., 2020) to identify the characteristics of training samples (e.g., as easy-to-learn or ambiguous samples), and then interpolates these characterized samples. We empirically validate that our method not only achieves competitive performance using a smaller subset of the training data compared with strong baselines, but also yields lower expected calibration error on the pre-trained language model, BERT, on both in-domain and out-of-domain settings in a wide range of NLP tasks. We publicly release our code.1

Footnote 1: [https://github.com/seoyeon-p/TDMixUp](https://github.com/seoyeon-p/TDMixUp)

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

MixUp (Zhang et al., 2018) is a simple data augmentation strategy in which additional samples are generated during training by combining _random pairs_ of training samples and their labels. While simple to implement, MixUp has been shown to improve both predictive performance and model calibration (i.e., avoiding over-confident predictions) (Guo et al., 2017) due to its regularization effect through data augmentation (Thulasidasan et al., 2019). However, selecting _random pairs_ in MixUp might not necessarily be optimal.

Despite this, MixUp has been explored for NLP tasks with substantial success using hidden state representations (Verma et al., 2019). For instance, Sun et al. (2020) explored MixUp, which uses the hidden representation of BERT (Devlin et al., 2019) to synthesize additional samples from randomly selected pairs. Yin et al. (2021) proposed MixUp, which uses the hidden representation of RoBERTa (Liu et al., 2019) to interpolate all samples in the same mini-batch to better cover the feature space. To date, only a few prior works have focused on selecting informative samples for MixUp. For example, Chen et al. (2020) proposed semi-supervised learning, which interpolates labeled and unlabeled data based on entropy. Kong et al. (2020) explored BERT calibration with MixUp, which generates new samples by exploiting the distance between samples in the feature space.

Recently, Swayamdipta et al. (2020) introduced data maps, which allow evaluating data quality by using training dynamics (i.e., the behavior of a model as training progresses). Specifically, they consider the mean and standard deviation of the gold label probabilities, predicted for each sample across training epochs (i.e., confidence and variability), and characterize data into three different categories: (1) samples that the model predicts correctly and consistently (i.e., **easy-to-learn**); (2) samples where true class probabilities vary frequently during training (i.e., **ambiguous**); and (3) samples that are potentially mis-labeled or erroneous (i.e., **hard-to-learn**). The author revealed that the easy-to-learn samples are useful for model optimization (parameter estimation) and without such samples the training could potentially fail to converge, while the ambiguous samples are those on which the model struggles the most and push the model to become more robust, hence, these ambiguous samples are the most beneficial for learning since they are the most challenging for the model.

Inspired by these observations, we propose a novel MixUp strategy which we call TDMixUp that monitors training dynamics and interpolates easy-to-learn samples with ambiguous samples in the feature space. That is, we pair one sample from

the easy-to-learn set with another sample from the ambiguous set to allow more informative samples to be combined for MixUp. Accordingly, we generate new samples that share the characteristics of both easy-to-learn and ambiguous data samples and are hence more beneficial for learning. However, the easy-to-learn and the ambiguous sets can contain mis-labeled samples that can degrade the model performance. Consequently, we measure another training dynamic, Area Under the Margin (AUM) Pleiss et al. (2020), to filter out possibly mis-labeled samples in each set. We validate our proposed method on a wide range of natural language understanding tasks including textual entailment, paraphrase detection, and commonsense reasoning tasks. We achieve competitive accuracy and low expected calibration error Guo et al. (2017) on both in-domain and out-of-domain settings for the pre-trained language model BERT Devlin et al. (2019), without using the full training data.

## 2 Proposed Approach: TDMixUp

We introduce our proposed TDMixUp, which generates additional samples based on the characteristics of the data samples. We first reveal the characteristics of each data sample by using training dynamics, i.e., confidence, variability, and Area Under the Margin (AUM). We then describe our MixUp operation that combines training samples based on the above data characteristics that are measured during training.

### Data Samples Characterization

We first introduce **confidence** and **variability**, that are used to evaluate the characteristics of each individual sample Swayamdipta et al. (2020). These statistics are calculated for each sample \((x_{i},y_{i})\) over \(E\) training epochs.

ConfidenceWe define confidence as the mean model probability of the true label \(y_{i}\) across epochs:

\[\hat{\mu}_{i}=\frac{1}{E}\sum_{e=1}^{E}p_{\theta^{(c)}}(y_{i}|x_{i}) \tag{1}\]

where \(p_{\theta^{(c)}}\) denotes the model's probability with parameter \(\theta^{(e)}\) at the end of \(e^{th}\) epoch.

VariibilityWe define variability as the standard deviation of \(p_{\theta^{(c)}}\) across epochs \(E\):

\[\hat{\sigma}_{i}=\sqrt{\frac{\sum_{e=1}^{E}(p_{\theta^{(c)}}(y_{i}|x_{i})-\hat {\mu}_{i})^{2}}{E}} \tag{2}\]

Given these statistics per sample, we identify the top 33% easy-to-learn samples, i.e., those samples that the model predicts correctly and consistently across epochs (high-confidence, low-variability), and the top 33% ambiguous samples, i.e., those samples whose true class probabilities have a high variance during training (high-variability).

Area Under the Margin (AUM)As another measure of data quality, we monitor training dynamics using the Area Under the Margin (AUM) Pleiss et al. (2020). AUM measures how different a true label for a sample is compared to a model's belief at each epoch and is calculated as the average difference between the logit values for a sample's assigned class (gold label) and its highest non-assigned class across training epochs. Formally, given a sample \((x_{i},y_{i})\), we compute \(AMM(x_{i},y_{i})\) as the area under the margin averaged across all training epochs \(E\). Specifically, at some epoch \(e\in E\), the margin is defined as:

\[M^{e}(x_{i},y_{i})=z_{y_{i}}-max_{y_{i}!=k}(z_{k}) \tag{3}\]

where \(M^{e}(x_{i},y_{i})\) is the margin of sample \(x_{i}\) with true label \(y_{i}\), \(z_{y_{i}}\) is the logit corresponding to the true label \(y_{i}\), and \(max_{y_{i}!=k}(z_{k})\) is the largest _other_ logit corresponding to label \(k\) not equal to \(y_{i}\). The AUM of \((x_{i},y_{i})\) across all epochs is:

\[AUM(x_{i},y_{i})=\frac{1}{E}\sum_{e=1}^{E}M^{e}(x_{i},y_{i}) \tag{4}\]

Intuitively, while both AUM and confidence measure training dynamics, confidence simply measures the probability output of the gold label and how much it fluctuates over the training epochs. In contrast, AUM measures the probability output of the gold label with respect to the model's belief in what the label for a sample should be according to its generalization capability (derived by observing other similar samples during training). More precisely, AUM considers each logit value and measures how much the gold label assigned logit value differs from _the other largest_ logit value, which allows identifying mis-labeled samples.

To identify possibly mis-labeled data in each set (i.e., the set of easy-to-learn and the set of ambiguous samples that are categorized by confidence and variability as described above), we first fine-tune a model on each set, respectively, with inserting fake data (i.e., threshold samples). Data with similar or worse AUMs than threshold samples can be 

assumed to be mis-labeled (Pleiss et al., 2020). We construct threshold samples by taking a subset of the training data and re-assigning their labels randomly, including a class that does not really exist. Specifically, given \(N\) training samples that belong to \(c\) classes, we randomly select \(N/(c+1)\) samples per class and re-assign their labels to classes that are different from the original class. We then train a model on training samples including threshold samples and measure the AUMs of all training data. We identify possible mis-labeled data by computing a threshold value (i.e., the \(k\)th percentile threshold sample AUMs where \(k\) is a hyper-parameter chosen on the validation set). At last, we filter out samples that have lower AUM than the threshold value.

### MixUp

MixUp training generates vicinity training samples according to the rule introduced in Zhang et al. (2018):

\[\begin{split}\tilde{x}&=\lambda x_{i}+(1-\lambda)x _{j}\\ \tilde{y}&=\lambda y_{i}+(1-\lambda)y_{j}\end{split} \tag{5}\]

where \(x_{i}\) and \(x_{j}\) are two randomly sampled input points, \(y_{i}\) and \(y_{j}\) are their associated one-hot encoded labels, and \(\lambda\) is a mixing ratio sampled from a Beta(\(\alpha\), \(\alpha\)) distribution with a hyper-parameter \(\alpha\). In standard MixUp, training data is augmented by linearly interpolating _random_ training samples in the input space. In contrast, our proposed TDMixUp interpolates one easy-to-learn sample with one ambiguous sample after applying AUM to filter potential erroneous samples that harm performance. Our current implementation uses easy-to-learn and ambiguous data loaders respectively and then applies MixUp to a randomly sampled mini-batch of each loader. We train a model on the generated TDMixUp samples in addition to the easy-to-learn and ambiguous samples using the cross entropy-loss.

## 3 Experiments and Results

### Tasks and Datasets

We evaluate our TDMixUp on three natural language understanding tasks. We describe our in-domain and out-of-domain sets as follows.

Natural Language Inference (NLI)Stanford Natural Language Inference (SNLI) is a task to predict if the relation between a hypothesis and a premise is _entailment, contradiction,_ or _neutral_(Bowman et al., 2015). Multi-Genre Natural Language Inference (MNLI) captures NLI with diverse domains Williams et al. (2018).

Paraphrase DetectionQuora Question Pairs (QQP) is a paraphrase detection task to test if two questions are semantically equivalent (Iyer et al., 2017). TwitterPPDB (TPPDB) is a dataset built to determine whether sentence pairs from Twitter convey similar semantics when they share URLs Lan et al. (2017).

Commonsense ReasoningSituations With Adversarial Generations (SWAG) is a commonsense reasoning task to choose the most plausible continuation of a sentence among four candidates Zellers et al. (2018). HellaSWAG is a dataset built using adversarial filtering to generate challenging out-of-domain samples.

### Experimental Setup

We use BERT Devlin et al. (2019) based classification model and pass the resulting [CLS] representation through a fully connected layer and softmax to predict the label distribution. We follow the published train/validation/test datasets splits as described in Desai and Durrett (2020). To identify mis-labeled samples in the top 33% easy-to-learn samples, we set threshold values \(k\) as: the 80th/80th/50th percentile threshold sample AUMs on SNLI/QQP/SWAG, respectively. More training details and hyper-parameter settings can be found in the Appendix. We evaluate the capability of our TDMixUp strategy to improve both predictive performance and model calibration due to its regularization effect through data augmentation. Hence, we use two metrics: (1) accuracy, and (2) expected calibration error (ECE) Guo et al. (2017); Desai and Durrett (2020). We report results averaged across 5 fine-tuning runs with random restarts for all experiments.

### Baseline Methods

BertDevlin et al. (2019) is the pre-trained base BERT model fine-tuned on each downstream task.

Back Translation Data AugmentationEdunov et al. (2018) generates augmented samples by using pre-trained translation models2 which can generate diverse paraphrases while preserving the semantics 

of the original sentences. In experiments, we translate original sentences from English to German and then translate them back to English to obtain the paraphrases.

MixUp(Zhang et al., 2018) generates augmented samples by interpolating random training samples in the input space (obtained from the first layer of the BERT pre-trained language model).

Manifold MixUp (M-MixUp)(Verma et al., 2019) generates additional samples by interpolating random training samples in the feature space (obtained from the task-specific layer on top of the BERT pre-trained language model).

MixUp for Calibration(Kong et al., 2020) generates augmented samples by utilizing the cosine distance between samples in the feature space.

Note that the above baselines use 100% training data while our proposed method focuses on particular subsets of the training data.

### Results

Fine-tuning on Subsets of Training DataTo explore the effect of different subsets of the data that are characterized using training dynamics, we compare the result of BERT fine-tuned on 100% training data with the results of BERT when fine-tuned on these subsets, and show the comparison in Table 1. Note that, for each task, we train the model on in-domain training set, and evaluate on in-domain and out-of-domain test sets. We make the following observations: First, we observe that accuracy and ECE improve when we filter out possibly mis-labeled samples in the top 33% easy-to-learn samples by using AUM in all cases (for both in-domain and out-of-domain test sets). Specifically, using _24% train, Easy-to-learn with AUM3_ returns better accuracy and lower ECE than _33% train, easy-to-learn_, showing that there are some potentially erroneous samples that harm the performance in the top 33% easy-to-learn samples. We manually investigate filtered samples on the top 33% easy-to-learn samples and observe that the top 33% easy-to-learn samples indeed include mis-labeled samples. For example, in SNLI, we observe that the relation between the following pairs of sentences in the top 33% easy-to-learn samples is _contradiction_ when it should be _neutral: <Two opposing wrestlers competing to pin one another; 'Two women are shopping in a boutique.>_ and <A person dressed in a colorful costume is holding some papers.'; 'the cat jumps on the dog.>_. In contrast, we observe that in many cases accuracy and ECE worsen when we filter out possibly mis-labeled samples in the 33% ambiguous samples by using AUM, suggesting that all ambiguous samples are useful for learning and generalization (which is consistent with Swayamdipta et al. (2020)). Sec

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & \multicolumn{2}{c}{SNLI} & \multicolumn{2}{c}{QQP} & SWAG \\ \cline{2-7}  & Acc & ECE & Acc & ECE & Acc & ECE \\ \hline
100\% train & **90.04\({}_{0.3}\)** & \(2.54_{0.8}\) & **90.27\({}_{0.3}\)** & \(2.71_{0.5}\) & **79.40\({}_{0.4}\)** & \(2.49_{1.8}\) \\ \hline
33\% train, Easy-to-learn & \(82.78_{0.6}\) & \(16.22_{0.7}\) & \(63.16_{0.1}\) & \(36.88_{0.1}\) & \(75.39_{0.2}\) & \(17.51_{0.1}\) \\
24\% train, Easy-to-learn with AUM & \(83.03_{0.9}\) & \(15.05_{0.9}\) & \(64.36_{0.6}\) & \(33.93_{0.9}\) & \(75.56_{0.1}\) & \(15.81_{0.7}\) \\
33\% train, Ambiguous & \(89.71_{0.5}\) & **0.74\({}_{0.1}\)** & \(87.51_{0.5}\) & \(71.70_{4.0}\) & \(75.91_{0.6}\) & \(184_{0.7}\) \\
24\% train, Ambiguous with AUM & \(87.88_{0.7}\) & \(7.09_{0.8}\) & \(88.63_{0.5}\) & \(6.36_{0.6}\) & \(71.74_{0.4}\) & \(7.55_{1.1}\) \\
66\% train, Easy-to-learn \& Ambiguous & \(89.65_{0.2}\) & \(2.64_{0.5}\) & \(90.23_{0.7}\) & **1.35\({}_{0.4}\)** & \(78.78_{0.5}\) & \(2.51_{0.8}\) \\ \hline \hline  & \multicolumn{2}{c}{MNLI} & \multicolumn{2}{c}{TwitterPPDB} & \multicolumn{2}{c}{HellaSWAG} \\ \cline{2-7}  & Acc & ECE & Acc & ECE & Acc & ECE \\ \hline
100\% train & \(73.52_{0.3}\) & \(7.09_{2.1}\) & **87.63\({}_{0.4}\)** & \(8.51_{0.6}\) & **34.48\({}_{0.2}\)** & \(12.62_{2.8}\) \\ \hline
33\% train, Easy-to-learn & \(61.41_{0.8}\) & \(36.68_{1.9}\) & \(81.07_{0.8}\) & \(18.92_{0.7}\) & \(33.59_{1.1}\) & \(29.38_{2.1}\) \\
24\% train, Easy-to-learn with AUM & \(62.97_{1.5}\) & \(32.48_{2.9}\) & \(82.16_{0.7}\) & \(17.46_{1.0}\) & \(33.67_{1.4}\) & \(16.89_{2.6}\) \\
33\% train, Ambiguous & \(72.52_{1.2}\) & \(10.73_{1.0}\) & \(86.62_{0.6}\) & **6.01\({}_{1.1}\)** & \(34.29_{0.9}\) & \(8.40_{1.3}\) \\
24\% train, Ambiguous with AUM & \(70.87_{0.9}\) & \(17.23_{0.6}\) & \(86.59_{0.8}\) & \(7.3los\) & \(33.81_{1.0}\) & \(3.76_{2.3}\) \\
66\% train, Easy-to-learn \& Ambiguous & **73.89\({}_{0.6}\)** & **3.46\({}_{1.9}\)** & \(87.29_{0.3}\) & \(8.04_{0.7}\) & \(34.43_{0.2}\) & \(9.68_{1.1}\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: The comparison of accuracy and expected calibration error (ECE) in percentage on in-domain (top) and out-of-domain (bottom) for BERT. We compare the results of fine-tuning on subsets of train samples (i.e., easy-to-learn and ambiguous samples) and fine-tuning on the entire 100% of training samples. Lower ECE implies better calibrated models. We report the mean accuracy across five training runs with the standard deviation shown in subscript (e.g., \(90.04_{0.3}\) indicates \(90.04\pm 0.3\)).

 

ond, we observe that fine-tuning BERT on both the easy-to-learn and the ambiguous samples (_66% train, Easy-to-learn & Ambiguous_) achieves similar performance and ECE as _100% train_.

Main ResultsTable 2 shows the result of the comparison of our proposed TDMixUp and baseline methods. We observe that our proposed method generally achieves higher accuracy and lower ECE on both in-domain and out-of-domain settings compared to any baseline using the full 100% training data showing the effectiveness of our TDMixUp strategy.

### Ablation Study

To compare the impact of the MixUp operation on samples generated by _random pairing_ and on samples generated by _informative pairing_, we conduct an ablation study. Specifically, we compare the results of MixUp on 66% train set (i.e., conduct the MixUp operation between randomly selected samples on 66% train set, which is the union of the top 33% easy-to-learn and the top 33% ambiguous samples) and our proposed TDMixUp (i.e., conduct the MixUp operation between the easy-to-learn filtered by AUM and the ambiguous samples). As shown in Table 3, we observe that our proposed TDMixUp which selects informative samples to combine performs better with respect to accuracy and ECE than vanilla MixUp that selects random samples, in all cases (in-domain and out-of-domain).

## 4 Conclusion

In this work, we propose a novel MixUp that leverages training dynamics (confidence, variability, and Area Under the Margin) to allow more informative samples to be combined in generating augmented samples. We empirically validate that our method not only achieves competitive accuracy but also calibrates BERT model on various NLP tasks, both on in-domain and out-of-domain settings.

## Acknowledgements

This research is supported in part by NSF CAREER award #1802358 and NSF CRI award #1823292. Any opinions, findings, and conclusions expressed here are those of the authors and do not necessarily reflect the views of NSF. We thank AWS for computing resources. We also thank our anonymous reviewers for their constructive feedback.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & \multicolumn{2}{c}{SNLI} & \multicolumn{2}{c}{QQP} & SWAG \\ \cline{2-7}  & Acc & ECE & Acc & ECE & Acc & ECE \\ \hline
100\% train & \(90.04_{0.3}\) & \(2.54_{0.8}\) & \(90.27_{0.3}\) & \(2.71_{0.5}\) & \(79.40_{0.4}\) & \(2.49_{1.8}\) \\
100\% train, MixUp [2] & \(88.82_{0.2}\) & \(7.73_{1.1}\) & \(89.12_{0.5}\) & \(9.04_{0.8}\) & \(74.98_{2.3}\) & \(7.08_{1.0}\) \\
100\% train, M-MixUp [18] & \(89.45_{0.9}\) & \(1.51_{0.8}\) & \(89.93_{0.6}\) & \(3.02_{1.0}\) & \(78.26_{0.4}\) & \(4.12_{0.6}\) \\
100\% train, MixUp for Calibration [10] & \(89.25_{0.5}\) & \(2.16_{0.5}\) & \(90.24_{0.3}\) & \(5.22_{0.6}\) & \(79.44_{0.6}\) & **1.10**\({}_{0.4}\) \\
100\% train, Back Translation Data Augmentation [1] & \(89.20_{0.5}\) & \(1.98_{0.6}\) & \(89.18_{0.6}\) & \(5.01_{0.3}\) & \(f62a_{0.9}\) & \(1.24_{0.2}\) \\ \hline
66\% train, TDMixUp, Easy-to-learn + Ambiguous & \multirow{2}{*}{\(89.73_{0.1}\)} & \(2.39_{0.8}\) & \(89.76_{0.7}\) & \(1.89_{0.4}\) & \(78.39_{0.3}\) & \(4.21_{0.3}\) \\
57\% train, TDMixUp, Easy-to-lean with AUM + Ambiguous (Ours) & **\(\textbf{90.31}_{0.2}\)** & \(\textbf{1.22}_{0.4}\) & \(\textbf{90.42}_{0.2}\) & \(\textbf{1.83}_{0.9}\) & \(\textbf{79.59}_{0.9}\) & \(2.16_{0.4}\) \\ \hline \hline \end{tabular} 
\begin{tabular}{l l c c c c c} \hline \hline  & \multicolumn{2}{c}{MNLI} & \multicolumn{2}{c}{TwitterPPDB} & \multicolumn{2}{c}{HellaSWAG} \\ \cline{2-7}  & Acc & ECE & Acc & ECE & Acc & ECE \\ \hline
100\% train & \(73.52_{0.3}\) & \(7.09_{2.1}\) & \(87.63_{0.4}\) & \(8.51_{0.6}\) & \(34.8_{0.2}\) & \(12.62_{2.8}\) \\
100\% train, MixUp [2] & \(69.19_{0.8}\) & \(19.51_{2.1}\) & \(87.45_{0.3}\) & \(1.71_{0.6}\) & \(33.22_{0.4}\) & \(1.093_{0.2}\) \\
100\% train, M-MixUp [18] & \(73.22_{0.6}\) & \(8.06_{1.2}\) & \(87.58_{0.7}\) & \(7.68_{1.3}\) & \(34.86_{0.9}\) & \(13.56_{1.6}\) \\
100\% train, MixUp for Calibration [10] & \(64.90_{0.5}\) & \(17.75_{1.8}\) & \(74.51_{1.1}\) & \(11.83_{0.1}\) & \(32.51_{0.8}\) & \(31.61_{2.3}\) \\
100\% train, Back Translation Data Augmentation [1] & \(73.15_{0.7}\) & \(8.46_{1.3}\) & \(86.82_{0.7}\) & \(8.83_{0.6}\) & \(34.97_{0.4}\) & \(22.68_{3.3}\) \\ \hline
66\% train, TDMixUp, Easy-to-learn + Ambiguous & \(72.83_{1.1}\) & \(5.84_{1.9}\) & \(87.63_{0.2}\) & \(6.48_{0.7}\) & \(34.11_{0.1}\) & \(10.54_{1.6}\) \\
57\% train, TDMixUp, Easy-to-learn with AUM + Ambiguous (Ours) & \(\textbf{742.8}_{0.6}\) & \(\textbf{2.91}_{1.4}\) & \(\textbf{87.89}_{0.3}\) & \(\textbf{6.08}_{0.4}\) & \(\textbf{35.21}_{0.6}\) & \(\textbf{9.45}_{1.3}\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Accuracy (in percentage) and expected calibration error (ECE) on in-domain (top) and out-of-domain (bottom) for BERT when comparing our proposed TDMixUp with baseline methods. Bold text shows the best performance and calibration. For _100% train_ results, we use reported results by Desai and Durrett (2020). We report the mean accuracy across five training runs with the standard deviation shown in subscript (e.g., \(90.04_{0.3}\) indicates \(90.04\pm 0.3\)).

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & Acc & ECE & Acc & ECE & Acc & ECE \\ \hline  & SNLI & \multicolumn{2}{c}{QQP} & SWAG \\ \hline Random & 89.59 & 1.70 & 89.87 & 3.06 & 79.15 & 4.51 \\ Ours & 90.31 & 1.22 & 90.42 & 1.53 & 79.59 & 2.16 \\ \hline \hline  & MNLI & \multicolumn{2}{c}{TwitterPPDB} & \multicolumn{2}{c}{HellaSWAG} \\ \hline Random & 73.22 & 6.89 & 87.23 & 6.53 & 34.43 & 15.87 \\ Ours & 74.28 & 2.91 & 87.89 & 6.08 & 35.21 & 9.45 \\ \hline \hline \end{tabular}
\end{table}
Table 3: The results comparison of MixUp selecting random samples on the union of the top 33% easy-to-learn and the top 33% ambiguous samples (i.e., Random) and our method.

 



## References

* B. Bowman, G. Angeli, C. Potts, and C. D. Manning (2015)A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal, pp. 632-642. External Links: Link Cited by: SS1.
* J. Chen, Z. Yang, and D. Yang (2020)MixText: linguistically-informed interpolation of hidden space for semi-supervised text classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 2147-2157. External Links: Link Cited by: SS1.
* S. Desai and G. Durrett (2020)Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 295-302. External Links: Link Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 4171-4186. External Links: Link Cited by: SS1.
* S. Edunov, M. Ott, M. Auli, and D. Grangier (2018)Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 489-500. External Links: Link Cited by: SS1.
* C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger (2017)On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330. External Links: Link Cited by: SS1.
* S. Iyer, N. Dandekar, and K. Csernai (2017)First quora dataset release: question pairs. In Quora, Cited by: SS1.
* L. Kong, H. Jiang, Y. Zhuang, J. Lyu, T. Zhao, and C. Zhang (2020)Calibrated language model fine-tuning for in- and out-of-distribution data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 1326-1340. External Links: Link Cited by: SS1.
* W. Lan, S. Qiu, H. He, and W. Xu (2017)A continuously growing dataset of sentential paraphrases. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark, pp. 1224-1234. External Links: Link Cited by: SS1.
* Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019)Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. Cited by: SS1.
* G. Pleiss, T. Zhang, E. R. Elenberg, and K. Q. Weinberger (2020)Identifying mislabeled data using the area under the margin ranking. In Advances in Neural Information Processing Systems, Vol. 33. Cited by: SS1.
* L. Sun, C. Xia, W. Yin, T. Liang, P. Yu, and L. He (2020)Mixup transformer: dynamic data augmentation for NLP tasks. In Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain, pp. 3436-3440. External Links: Link Cited by: SS1.
* S. Swayamdipta, R. Schwartz, N. Lourie, Y. Wang, H. Hajishirzi, N. A. Smith, and Y. Choi (2020)Dataset cartography: mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 9275-9293. External Links: Link Cited by: SS1.
* S. Thulasidasan, G. Chennupati, J. A Bilmes, T. Bhattacharya, and S. Michalak (2019)On mixup training: improved calibration and predictive uncertainty for deep neural networks. In Advances in Neural Information Processing Systems, Vol. 32. Cited by: SS1.
* V. Verma, A. Lamb, C. Beckham, A. Najafi, I. Mitliagkas, D. Lopez-Paz, and Y. Bengio (2019)Manifold mixup: better representations by interpolating hidden states. In International Conference on Machine Learning, pp. 6438-6447. External Links: Link Cited by: SS1.
* A. Williams, N. Nangia, and S. Bowman (2018)A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-1122. External Links: Link Cited by: SS1.
* W. Yih, H. Wang, J. Qu, and C. Xiong (2021)BatchMixup: improving training by interpolating hidden states of the entire mini-batch. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, Online, pp. 4908-4912. External Links: Link Cited by: SS1.
* R. Zellers, Y. Bisk, R. Schwartz, and Y. Choi (2018)SWAG: a large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 93-104. External Links: Link Cited by: SS1.



Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz (2018)mixup: Beyond empirical risk minimization. In Proceedings of the 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada. Cited by: SS3.
* A Supplementary Materials

### Training Details

In our experiments, we use _bert-base-uncased_ classification model on top of a task-specific fully-connected layer. The model is fine-tuned with a maximum of 3 epochs, batch size of 16 for SNLI and QQP, batch size 4 for SWAG, a learning rate of 1e-5, gradient clip of 1.0, and no weight decay. We use the hyper-parameter of MixUp \(\alpha\) as 0.4. All hyper-parameters are estimated on the validation set of each task. For all results, we report averaged results across 5 fine-tuning runs with random starts. Finally, all experiments are conducted on a single NVIDIA RTX A6000 48G GPU with the total time for fine-tuning all models being under 24 hours. For each dataset, we follow the published train/validation/test split by Desai and Durrett (2020) and show the statistics of the datasets in Table 4.

### Data Maps

In this section, we provide data maps (Swayamdipta et al., 2020) of our in-domain datasets on _bert-base-uncased_ model in Figure 1. These data maps are used to identify the characteristics of each training sample (i.e., easy-to-learn, ambiguous, and hard-to-learn).

Figure 1: Data Maps of SNLI, QQP and SWAG on _bert-uncased_ model.