Venue: NAACL 2022

# CIAug: Equipping Interpolative Augmentation with Curriculum Learning

 Ramit Sawhney\({}^{\dagger}\), Ritesh Soun\({}^{\clubsuit}\); Shrey Pandit\({}^{\lx@sectionsign}\)1, Megh Thakkar\({}^{\lx@sectionsign}\)2,

Sarvagya Malaviya\({}^{\clubsuit}\)3, Yuval Pinter\({}^{\triangle}\)4

Footnote 1: We release CIaug’s code at: [https://github.com/sounritesh/CIAug-NARCL](https://github.com/sounritesh/CIAug-NARCL)

Footnote 2: footnotemark:

###### Abstract

Interpolative data augmentation has proven to be effective for NLP tasks. Despite its merits, the sample selection process in mixup is random, which might make it difficult for the model to generalize better and converge faster. We propose CIAug, a novel curriculum-based learning method that builds upon mixup. It leverages the relative position of samples in hyperbolic embedding space as a complexity measure to gradually mix up increasingly difficult and diverse samples along training. CIAug achieves state-of-the-art results over existing interpolative augmentation methods on 10 benchmark datasets across 4 languages in text classification and named-entity recognition tasks. It also converges and achieves benchmark F1 scores 3 times faster. We empirically analyze the various components of CIAug, and evaluate its robustness against adversarial attacks.

+Georgia Institute of Technology

$Sri Venkateswara College, DU

$BITS, Pilani

$Manipal University Jaipur

$AnBen-Gurion University of the Negev rsawhney31@gatech.edu, uvp@cs.bgu.ac.il

## 1 Introduction

Data augmentation is an effective tool for avoiding overfitting in model training in cases where there is an absence of sufficient training data Liu et al. (2021). Interpolative augmentation techniques, such as Mixup Zhang et al. (2018), have shown an increase in model performance across various modalities, with further improvements gained by applying Mixup at latent representation layers Chen et al. (2020a). Current implementations of Mixup select samples for interpolation at random, not leveraging the potential for adaptive selection techniques which have been shown to lead to better generalizability Chen et al. (2020b). In addition, these methods do not account for the spatial distribution of linguistic data, known to extend beyond the capacities of euclidean space Nickel and Kiela (2017).

We propose **CIAug1**, a method which addresses these challenges by offering an augmentation procedure that selects samples in an adaptive fashion and is geometrically sound. CIAug's sampling strategy follows the idea that selecting easier mixing samples first and gradually increasing sample difficulty based on relative spatial position would generate more suitable synthetic inputs, resulting in better model training Xu et al. (2021). This notion ties in with the framework of curriculum learning Krueger and Dayan (2009), where training data is presented in a similarly staggered way, increasing model capabilities Bengio et al. (2009). CIAug's selection strategy performs distance operations in hyperbolic space, applying insights about language data spatial distribution to its definition of 'similar samples' Tifrea et al. (2019). We first train CIAug by sampling pairs of sentences similar to each other for mixing, then gradually as training progresses, we sample sentences that are dissimilar, following the curriculum learning strategy.

Footnote 1: We release CIAug’s code at: [https://github.com/sounritesh/CIAug-NARCL](https://github.com/sounritesh/CIAug-NARCL)

Through experiments on 10 benchmark datasets in English, Arabic, Turkish and German, we show

Figure 1: Overview of CIAug showing curriculum-based sample selection using hyperbolic distance to perform interpolation.



that CIAug outperforms state-of-the-art models in classification and named-entity recognition tasks. We probe the effectiveness of CIAug in conjunction with different similarity measures and qualitatively evaluate it. We show that CIAug converges faster than traditional Mixup while being both generalizable across tasks and languages, as well as more resilient to adversarial classification examples.

## 2 Methodology

We illustrate CIAug's sample selection strategy in Figure 1. In this section, we first introduce Mixup (SS2.1), and follow by formulating CIAug and its relative sample distribution component (SS2.2).

### Interpolative Mixup

Given two data samples \(x_{i},x_{j}\in X\) with labels \(y_{i},y_{j}\in Y\), where \(i,j\in[1,N]\), Mixup (Zhang et al., 2018) performs a linear interpolation with ratio \(r\) between these two samples according to eq. (1), creating a new synthetic data point \(x^{\prime}\) and its label \(y^{\prime}\):

\[\begin{split} x^{\prime}&=\text{Mixup}(x_{i},x_{ j})=r\!:\!x_{i}+(1-r)\!:\!x_{j}\\ y^{\prime}&=\text{Mixup}(y_{i},y_{j})=r\!:\!y_{ i}+(1-r)\!:\!y_{j}\end{split} \tag{1}\]

Interpolative Mixup (Chen et al., 2020a) performs linear interpolation over the latent representations of models. Let \(f_{\theta}(\cdot)\) be a model with parameters \(\theta\) having \(K\) layers. \(f_{\theta,n}(\cdot)\) denotes the \(n\)-th layer of the model, \(h_{n}\) the hidden space vector at layer \(n\) for \(n\in[1,K]\), and \(h_{0}\) the input vector. Interpolative Mixup at a layer \(k\!\sim\![1,K]\) can be done by separately calculating the latent representation of the layers before the \(k\)-th layer. For input sample \(x_{i}\), we let \(h_{n}^{i}\) denote the hidden state representations at layer \(n\),

\[\begin{split} h_{n}^{i}&\!=\!f_{\theta,n}(h_{n-1}^{i }),~{}~{}~{}n\in[1,k]\\ h_{n}^{j}&\!=\!f_{\theta,n}(h_{n-1}^{j}),~{}~{}~{} n\in[1,k]\end{split} \tag{2}\]

```
M \(\leftarrow\) Learnable distance matrix initialized with hyperbolic distances \(N\gets\textsc{No.}\) of training samples \(X\leftarrow\) Training samples \(Y\leftarrow\) Training labels \(m\leftarrow\) No. of epochs \(\tau_{s}\leftarrow\) Diversity threshold \(\in(0,1)\) \(\tau_{c}\leftarrow\) Curriculum threshold \(\in(0,1)\) for sample complexity for\(k\in\{1,\dots,m\}\)do for\(i\in\{1,\dots,N\}\)do \(S_{i}\leftarrow\textsc{SST}(X,M_{i},\tau_{s},\tau_{c})\) (6)  Select \(x_{j}\in S^{i}\) \(x_{i}\leftarrow\) CIMixup\((x_{i},x_{j})\) (8) \(y_{i}\leftarrow\) CIMixup\((y_{i},y_{j})\) \(y_{out}\) = Predict\((x^{\prime}_{i})\)  Loss\((y_{out},y^{\prime}_{i})\) endfor \(\tau_{c}\leftarrow\textsc{Update}(\tau_{c})\) (7) endfor
```

**Algorithm 1**CIAug

We then perform Mixup over individual hidden state representations \(h_{k}^{i},h_{k}^{j}\) from layer \(k\) as,

\[h_{k}\!=\!\text{Mixup}(h_{k}^{i},h_{k}^{j})\!=\!r\!:\!h_{k}^{i}+(1-r)\!:\!h_{ k}^{j} \tag{3}\]

The mixed hidden representation \(h_{k}\) is used as the input for the continuing forward pass,

\[h_{n}\!=\!f_{\theta,n}(h_{n-1});~{}~{}~{}n\in[k+1,K]. \tag{4}\]

### CIAug

Although Mixup helps models generalize better, it selects samples for interpolation randomly. Deriving information from the spatial distributions of the samples to be mixed, and utilising it to introduce curriculum and diversity in sample selection, can lead to performance improvements.

We present steps involved in CIAug in Algorithm 1. CIAug encodes the relative complexity between instances in a learnable matrix \(\mathbf{M}\in\mathds{R}^{N\times N}\), which we initialize using a distance metric. Motivated by evidence that euclidean space, where models typically operate, is not able to effectively capture the complex properties of natural language data (Ganea et al., 2018), we use hyperbolic distance as our metric for modeling representations while performing interpolative operations (Sanwhney et al., 2021). The hyperbolic distance \(\mathcal{D}_{h}\) between embeddings \(e_{i}=f_{\theta}(x_{i})\) and \(e_{j}=f_{\theta}(x_{j})\) is defined as:

\[\mathcal{D}_{h}(e_{i},e_{j})=2\tan^{-1}(\|(-e_{j})\oplus e_{i}\|), \tag{5}\]

where \(\oplus\) represents Mobius addition (A.1).

In contrast to Mixup, CIAug is defined for one sample. Given a sample \(x_{i}\), we create a set \(S_{i}\) of increasingly diverse samples in the dataset relative to \(x_{i}\) using an operation labeled SST and thresholds \(\tau_{s}\) and \(\tau_{c}\):

\[S_{i}=\textsc{SST}(X,M_{i},\tau_{s},\tau_{c}):=\{x_{k}|x_{k}\in X,\tau_{s} \leq\mathbf{M}_{ik}\leq\tau_{c}\}, \tag{6}\]

where \(\tau_{s}=\mbox{T}\cdot\max(\mathbf{M}_{i})\) at each step of training, and T \(\in(0,1)\) is a hyperparameter. \(\tau_{s}\) helps in sampling the most diverse samples relative to \(x_{i}\). Although sample diversity helps the model generalize 

it may not guarantee global convergence. Exposing the model to extremely diverse samples early in the training process can be detrimental to its performance. Therefore, we introduce a distance-based learning curriculum using a second threshold, \(\tau_{c}\), which selects mixing samples which are increasingly diverse from \(x_{i}\)'s perspective. \(\tau_{c}\) is dynamically updated during training using an Update operation:

\[\texttt{Update}(\tau_{c}):=\tau_{c}+\alpha, \tag{7}\]

where \(\alpha\in(0,1)\) is a hyperparameter and \(\tau_{c}=0.1\), initially.

We then sample a random instance from \(x_{j}\in S_{i}\) to perform Mixup with \(x_{i}\).

Using **M**, we change the Mixup formulation (Equation 1) for samples \(i\) and \(j\) and define CIMixup as,

\[\texttt{CIMixup}(x_{i},x_{j})=(1-\textbf{M}_{ij})\cdot x_{i}+\textbf{M}_{i }\cdot x_{j} \tag{8}\]

Finally, we formulate CIAug as:

\[\texttt{CIAuo}(x_{i})=\texttt{CIMixup}(x_{i},x_{j}),~{}~{}~{}x_{j}\in S_{i}. \tag{9}\]

We replace the Mixup operation from Equation 3 with the CIaug operation in Equation 9 to evaluate CIAug. The final hidden state output \(h_{K}\) is passed through a multi-layer perceptron (MLP) \(g_{\phi}\) for classification. We optimize the network using KL Divergence loss between the final output \(g_{\phi}(h_{K})\) and mixed label \(y^{\prime}=\text{CIMixup}(y_{i},y_{j})\), which also trains the matrix M end-to-end.

## 3 Experimental Setup

We evaluate CIAug on classification and NER tasks in various settings across 4 languages, and on GLUE datasets Wang et al. (2018).

### Training Setup

We use BertAdam optimizer Wolf et al. (2020) with a learning rate of 2\(e\)-5, batch size of 8 and weight decay of 0.01, trained for 10 epochs. CIAug is performed over a layer randomly sampled from all model layers. For the datasets in English we use BERT Devlin et al. (2019) as our base model \(f_{\theta}\), and for other languages we use mBERT. For calculating distances between instances, we use the [CLS] token representation from the sentence embeddings. Due to lack of powerful computational resources, we train on 10,000 samples for SST-2 dataset, and keep the validation and test set unchanged.

### Evaluation

For a comprehensive evaluation, we compare CIAug with some standard baselines: word-mixup (WMix), sentence-mixup (SMix) Guo et al. (2019), HypMix(HMix) Sawhney et al. (2021). We use **F1** score as a metric for evaluating CIAug and comparing it with other baselines.

## 4 Results and Analysis

We present our main results in Table 1. We observe that CIAug outperforms the baseline \(f_{\theta}\), validating that selecting diverse samples based on similarity enhances the model performance. We further find that the hyperbolic variant of sample selection performs better than the Euclidean CIAug (Table 5 in the appendix). This validates that hyperbolic space is more capable of capturing the complex hierarchical information present in the sentence representation, leading to better comparison and sample selection.

\begin{table}
\begin{tabular}{l c c c c|c} \hline \hline  & \multicolumn{2}{c}{Non-trainable M} & \multicolumn{2}{c}{Trainable M} \\  & Euc-CIAug & CIAug & Euc-CIAug & CIAug \\ \hline SST-2 & 91.17 & 92.67 & 91.71 & **92.93*** \\ Tree-fine & 92.10 & **93.00** & 92.40 & 92.80 \\ Tree-coarse & 97.50 & 97.80 & 97.61 & **98.20*** \\ CoLA & 87.76 & 91.92 & 92.55 & **95.32*** \\ TTC & 90.87 & 91.33 & 91.00 & **91.50*** \\ AHS & 67.42 & 72.57 & 70.42 & **74.14*** \\ \hline RTE & 64.62 & **68.95** & 62.09 & 68.23 \\ MRPC & 84.55 & 85.04 & 84.31 & **87.00*** \\ \hline CoNLL-en & 85.49 & 86.77 & 86.63 & **86.85*** \\ CoNLL-de & 91.05 & 92.32 & 91.91 & **92.64*** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Ablation study of CIAug with different distance constraints. Bold shows the best result. * shows significant (p<0.01) improvement over Euc-CIAug-NT 

We also compare CIAug with its non-trainable matrix counterpart where sample selection is based on the relative position of sentences, using a constant matrix \({\bf M}\). We observe that this variant performs worse, suggesting that \({\bf M}\) is able to capture sample-specific information relative to other samples, generating more suitable sample selection and mixing ratio while performing interpolative data augmentation.

Impact of distance metricWe explore the effectiveness of CIAug with the euclidean and hyperbolic distance measures as diversity and complexity metric for the thresholds, \(\tau_{s}\) and \(\tau_{c}\). The results, presented in Table3, show that utilizing hyperbolic distance for both thresholds yields the best results, suggesting that hyperbolic space captures the hierarchical properties of textual data better, gauging the relative diversity and complexity of samples effectively.

Analysing the convergence of CIAugFor all benchmark datasets, we observe that CIAug reaches a benchmark F1 score faster than Mixup method, as shown in Figure2.2 As CIAug selects samples for Mixup based on a learning curriculum, it leads to generation of more suitable synthetic samples in a staggered manner resulting in better training Xu et al. (2021) and faster convergence.

Footnote 2: We get similar results on the other datasets.

Adversarial RobustnessAdversarial attacks confuse a model by providing specifically designed inputs. We compare the robustness of CIAug with Mixup by performing black-box adversarial attacks. Using the NLPAug library Ma (2019) we substitute up to 10% of the words in each sentence with their synonyms found in WordNet Feinerer Hornik (2020) and present the results in Table4. We observe that both CIAug-NT and CIAug are more robust compared to regular Mixup by a difference of 6.72% and 6% respectively. This robustness towards adversarial attacks could be attributed to the curriculum-learning-based interpolative technique which resulted in better training and generalizability of underlying model.

Curriculum thresholdWe perform a study on CIAug by varying \(\alpha\) in the the curriculum learning threshold \(\tau_{c}\) as in equation7, and present it in Figure3. A lower value of \(\alpha\) would result in the slow increase of \(\tau_{c}\), which can lead the underlying model to not converge properly, whereas a higher value of \(\alpha\) would result in accelerated increase of \(\tau_{c}\), losing the advantages of curriculum learning. Figure3 shows the existence of an optimal \(\alpha\) for curriculum learning.

## 5 Conclusion

We propose CIAug, a novel mixup technique that uses curriculum learning, leveraging the relative spatial positions of the samples in the embedding space as a measure of complexity to signal curriculum learning. CIAug achieves state-of-the-art results over existing interpolative data augmentation

Figure 3: Change in performance in terms of F1 with varying \(\alpha\) for CIAug

\begin{table}
\begin{tabular}{l r r} \hline \hline Method & Accuracy & Adversarial Accuracy \\ \hline Mixup & 65.56 & 55.95 \\ CIAug-NT & 68.95 & 66.06 \\ CIAug & 68.23 & 64.62 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance on adversarial examples generated using synonym substitution on RTE.

Figure 2: Convergence comparison of Mixup with CIaug-NC (No curriculum), CIAug-NT, CIAug as number of training epochs needed to reach a benchmark F1 score. (AHS benchmark score:72%).

 

methods on 10 standard and multilingual datasets in English, Arabic, Turkish and German. CIAug converges faster than the traditional Mixup technique, while being generalizable across different tasks and modalities.

## References

* N. Albadi, M. Kurdi, and S. Mishra (2018)Are they our brothers? analysis and detection of religious hate speech in the arabic twittersphere. In Proceedings of the 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, pp. 69-76. Cited by: SS3.4.
* Y. Bengio, J. Louradour, R. Collobert, and J. Weston (2009)Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, New York, NY, USA, pp. 41-48. External Links: ISBN 978-1-580-3874-4, Link, Document Cited by: SS1, SS2.
* L. Bentivogli, P. Clark, I. Dagan, and D. Giampiccolo (2009)The fifth pascal recognizing textual entailment challenge. In TAC, Cited by: SS1, SS2.
* J. Chen, Z. Yang, and D. Yang (2020a)MixText: linguistically-informed interpolation of hidden space for semi-supervised text classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 2147-2157. External Links: ISSN 0018-902X, Link, Document Cited by: SS1, SS2.
* T. Chen, S. Kornblith, M. Norouzi, and G. Hinton (2020b)A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709. Cited by: SS1, SS2.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)Bert: pre-training of deep bidirectional transformers for language understanding. Cited by: SS1, SS2.
* W. B. Dolan and C. Brockett (2005)Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), Cited by: SS1, SS2.
* H. Feinerer and K. Hornik (2020)Wordnet: wordnet interface. Note: R package version 0.1-15 Cited by: SS1, SS2.
* O. Ganea, G. Becigneul, and T. Hofmann (2018)Hyperbolic neural networks. Cited by: SS1, SS2.
* H. Guo, Y. Mao, and R. Zhang (2019)Augmenting data with mixup for sentence classification: an empirical study. arXiv preprint arXiv:1905.08941. Cited by: SS1, SS2.
* G. Jawahar, B. Sagot, and D. Seddah (2019)What does BERT learn about the structure of language?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 3651-3657. External Links: ISBN 978-1-4503-3874-4, Link, Document Cited by: SS1, SS2.
* A. Jindal, A. Ghosh Chowdhury, A. Didolkar, D. Jin, R. Sawhney, and R. Stah Shah (2020)Augmenting NLP models using latent feature interpolations. In Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain, pp. 6931-6936. External Links: ISBN 978-1-4503-3874-4, Link, Document Cited by: SS1, SS2.
* D. Kilinc, A. Ozcift, F. Bozyigit, P. Yildirim, F. Yuclar, and E. Borandag (2017)Ttc-3600: a new benchmark dataset for turkish text categorization. Journal of Information Science43, pp. 174-185. Cited by: SS1, SS2.
* K. A. Krueger and P. Dayan (2009)Flexible shaping: how learning in small steps helps. Cognition110 (3), pp. 380-394. Cited by: SS1, SS2.
* J. P. Lee-Thorp, J. Ainslie, I. Eckstein, and S. Ontanon (2021)Finet: mixing tokens with fourier transforms. ArXivabs/2105.03824. Cited by: SS1, SS2.
* X. Li and D. Roth (2002)Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics, Cited by: SS1, SS2.
* L. Liu, B. Ding, L. Bing, S. Joty, L. Si, and Chunyan Miao (2021)MulDA: a multilingual data augmentation framework for low-resource cross-lingual NER. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 5834-5846. External Links: ISBN 978-1-4503-3874-4, Link, Document Cited by: SS1, SS2.
* Edward Ma (2019)Nlp augmentation. Note: [https://github.com/makcedward/nlpaug](https://github.com/makcedward/nlpaug) Cited by: SS1, SS2.
* M. Nickel and D. Kiela (2017)Poincare embeddings for learning hierarchical representations. Advances in neural information processing systems30, pp. 6338-6347. Cited by: SS1, SS2.
* R. Sawhney, M. Thakkar, S. Agarwal, D. Jin, D. Yang, and L. Flek (2021)HypMix: hyperbolix interplay data augmentation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online, pp. 9858-9868. External Links: ISBN 978-1-4503-3874-4, Link, Document Cited by: SS1, SS2.
* R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts (2013)Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, Washington, USA, pp. 1631-1642. External Links: ISBN 978-1-4503-3874-4, Link, Document Cited by: SS1, SS2.

 

Alexandru Tifrea, Gary Becigneul, and Octavian-Eugen Ganea. 2019. Poincare glove: Hyperbolic word embeddings. In _International Conference on Learning Representations_.
* Volume 4_, CONLL '03, page 142-147, USA. Association for Computational Linguistics.
* Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_, pages 353-355, Brussels, Belgium. Association for Computational Linguistics.
* Wang et al. (2021) Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. 2021. Entailment as few-shot learner. _ArXiv_, abs/2104.14690.
* Warstadt et al. (2018) Alex Warstadt, Anampreet Singh, and Samuel R Bowman. 2018. Neural network acceptability judgments. _arXiv preprint arXiv:1805.12471_.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Fantowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lihoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Xu et al. (2021) Shusheng Xu, Xingxing Zhang, Yi Wu, and Furu Wei. 2021. Sequence level contrastive learning for text summarization. _ArXiv_, abs/2109.03481.
* Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In _NeurIPS_.
* Yoon et al. (2021) Soyoung Yoon, Gyuwan Kim, and Kyumin Park. 2021. SSMix: Saliency-based span mixup for text classification. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 3225-3234, Online. Association for Computational Linguistics.
* Zhang et al. (2018) Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. 2018. mixup: Beyond empirical risk minimization. In _International Conference on Learning Representations_.

## Appendix A Hyperbolic Geometry

### Mobius Addition

Mobius addition \(\oplus\) for a pair of points \(x,y\in{\cal B}\), defined as,

\[x\oplus y:=\frac{(1+2\langle x,y\rangle+||y|^{2})x+(1-||x||^{2})y}{1+2\langle x ,y\rangle+||x||^{2}||y||^{2}} \tag{10}\]

, \(\langle.,.\rangle\), \(||\cdot||\) are Euclidean inner product and norm.

## Appendix B Extended Analysis

We compare the performance of CIAug with some recent baselines and interpolative augmentation techniques like (Jindal et al., 2020) and (Yoon et al., 2021) on standard English and GLUE datasets.

## Appendix C Dataset Details

1. **TTC**. (Kilinc et al., 2017), Turkish Text Categorization dataset consists of 3600 Turkish documents (news/texts) classified into 6 classes.
2. **CoLA**. (Warstadt et al., 2018), abbreviation for the Corpus of Linguistic Acceptability is a part of GLUE (Wang et al., 2018) benchmark. It is a collection of English sentences from 23 linguistic publications that are annotated for their grammatical acceptability.
3. **SST-2**. (Socher et al., 2013) is a GLUE (Wang et al., 2018) benchmark dataset consisting of English sentences from movie reviews. Samples in the dataset are annotated for sentiment classification task.
4. **TREC-Coarse**. (Li and Roth, 2002), The Text REtrieval Conference-Coarse is a question classification dataset consisting of 6 classes. The data is sourced from English questions by USC, TREC 8, TREC 9, TREC 10 and manually constructed questions.

\begin{table}
\begin{tabular}{p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt}} \hline \hline
**Model** & **SST-2** & **CoLA** & **TREC-Coarse** & **TREC-Fine** \\ \hline XLNet (2019) & 97.00 & 70.20 & 94.58 & 87.49 \\ EFL (2021) & 96.90 & 86.40 & 93.36 & 80.90 \\ FNet (2021) & 94.00 & 78.00 & 96.89 & 89.97 \\ \hline SSMix (2021) & **92.95** & 86.76 & 97.60 & 90.24 \\ EMix (2020) & 91.13 & 85.21 & 97.44 & 90.04 \\
**CIAug** (Ours) & 92.93 & **95.32** & **98.20** & **92.80** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Performance comparison with additional baselines and interpolative augmentation methods.

 

5. **TREC-Fine**. Li and Roth (2002) contains the same set of questions as TREC-Coarse grouped into 47 fine-grained classes.
6. **AHS**. Albadi et al. (2018) is an Arabic hate speech classification dataset focusing mainly on Saudi Twittersphere.
7. **RTE**. Bentivogli et al. (2009) The Recognizing Textual Entailment (RTE) datasets come from a series of textual entailment challenges. Data from RTE1, RTE2, RTE3 and RTE5 is combined. Examples are constructed based on news and Wikipedia text. Metric used here is accuracy
8. **MRPC**. Dolan and Brockett (2005) The Microsoft Research Paraphrase Corpus is a corpus of sentence pairs automatically extracted from online news sources, with human annotations for whether the sentences in the pair are semantically equivalent. Metric used here is accuracy
9. **CONLL**. Tjong Kim Sang and De Meulder (2003) It is a named entity recognition dataset released as a part of CoNLL-2003 shared task: language-independent named entity recognition. The data consists of eight files covering two languages: English and German.

## Appendix D Experimental Setup

We provide a detailed explanation of our experimental setup in table 6

## Appendix E Qualitative Analysis

### Layer-wise Ablation

We compare the performance of CIAug on different set of mixup layers in table 7. TMix attains best performance on layer set 7,9,12 is used because layers 6,7,9,12 contains the most amount of syntactic and semantic information Chen et al. (2020). CIAug achieves best performance on layer set 3,4,6,7,9,12, this suggests other than syntactic and semantic information, curriculum learning based approach in CIAug helps to capture the surface-level information in layer 3 and 4 Jawahar et al. (2019).

## Appendix F Limitations

Even though CIAug converges faster and performs better than other baseline interpolative techniques, the computational power required is not so fairly available in many devices. We plan to work on improving the efficiency of the parameterized matrices involved in the computation, such as using sparse matrices.

\begin{table}
\begin{tabular}{l r} \hline \hline
**Parameter** & Value \\ \hline Optimizer & BERTAdam 2020 \\ \hline Learning Rate & 2e-5 \\ \hline Batch Size & 8 \\ \hline \(\beta_{1},\beta_{2},\epsilon\) & 0.9, 0.999, 1e-6 \\ \hline \# Epochs & 10 \\ \hline Evaluation Metric & F1 Score \\ \hline Base Model & 
\begin{tabular}{c} BERT-base-uncased, \\ BERT-base-multilingual-uncased \\ \end{tabular} \\ \hline Classifier & Linear layer \\ (over architecture) & \multicolumn{1}{r}{Linear layer} \\ \hline Hardware & Nvidia V100 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Model and training setup for CIAug.

\begin{table}
\begin{tabular}{l c c} \hline Mixup & \multirow{2}{*}{AHS} & MRPC \\ Layer Set & & \\ \hline
[3,4] & 71.12 & 82.84 \\
[1,2] & 72.21 & 85.62 \\
[6,7,9] & 72.57 & 83.08 \\
[7,9,12] & 73.77 & 84.06 \\
[3,4,6,7,9,12] & **74.14** & **87** \\ \hline \hline \end{tabular}
\end{table}
Table 7: Layer-wise ablation score (F1) when performing interpolative augmentation.