Augmentation with Projection: Towards an Effective and Efficient Data Augmentation Paradigm for Distillation
Authors: Ziqi Wang
Yuexin Wu
Frederick Liu
Daogao Liu
Le Hou
Hongkun Yu
Jing Li
Heng Ji
Venue: ICLR 2023 poster (2023)
Keywords: Knowledge Distillation
Data Augmentation
Natural Language Processing

Abstract: Knowledge distillation is one of the primary methods of transferring knowledge from large to small models. However, it requires massive task-specific data, which may not be plausible in many real-world applications. Data augmentation methods such as representation interpolation, token replacement, or augmentation with models are applied to tackle this problem. However, these data augmentation methods either potentially cause shifts in decision boundaries (representation interpolation), are not expressive enough (token replacement), or introduce too much computational overhead (augmentation with models). To this end, we propose AugPro (Augmentation with Projection), an effective and efficient data augmentation method for distillation. Our method builds on top of representation interpolation augmentation methods to maintain the diversity of expressions and converts the augmented data to tokens to avoid shifting decision boundaries. It uses simple operations that come with little computational overhead. The results on multiple GLUE tasks show that our methods can improve distillation performance by a large margin at a low time cost.

Published as a conference paper at ICLR 2023
AUGMENTATION WITH PROJECTION: TOWARDS AN EFFECTIVE AND EFFICIENT DATA AUGMENTATION PARADIGM FOR DISTILLATION
Ziqi Wang'; Yuexin Wu², Frederick Liu', Daogao Liu', Le Hou", Hongkun Yu², Jing Lie, Heng Ji
1 University of Illinois Chaaaa-Chaappiin 2 Google 3 University of Washington
fziqiw9, hengji @illinois.edu {criekwu, frederickliu, lehou, hongkuny, jingli) @ google.com dgliu @uw.edu
ABSTRACT
Knowledge distillation is one of the primary methods of transferring knowledge from large to small models. However, it requires massive task-specific data, which may not be plausible in many real-world applications. Data augmentation methods such as representation interpolation, token replacement, or augmentation with models are applied to tackle this problem. However, these data augmentation methods either potentially cause shifts in decision boundaries (representation inter- polation), are not expressive enough (token replacement), or introduce too much computational overhead (augmentation with models). To this end, we propose Aug- Pro (Augmentation with Projection), an effective and efficient data augmentation method for distillation. Our method builds on top of representation interpola- tion augmentation methods to maintain the diversity of expressions and converts the augmented data to tokens avoid shifting decision boundaries. It uses simple operations that come with little computational overhead. The results on multiple GLUE tasks show that our methods can improve distillation performance by a large margin at low time cost. Codes are available at ht https: //tpt//tt.t/t///////////////tttt com/ google research google re http////eeeee/eteeee/eeeeeaee augpro.
INTRODUCTION
Large-scale language models (Devlin et al., 2018; Raffel et al., 2020; Brown al., 2020; Zhang et al., 2022c) have achieved great success on various natural language processing (NLP) tasks, such as information extraction (Lu et al., 2021) and question answering (Kassner & Schütze, 2020). However, large-scale models have high computational overhead, which limits their deployment in edge devices and fast response scenarios (Sun al., 2020b). One widely used solution to perform knowledge distillation (Hinton et al., 2015) from large-scale models to small-scale models. This method, however, usually requires a large amount of data to guarantee the transfer quality, which may not be easily obtained in real-world applications. To this end, data augmentation methods are applied (Liang et al., 2020; Wang & Yang, 2020; Zhang et al., 2022b) to improve the distillation performance.
There are three major types of data augmentation methods: (1) Representation interpolation. For example, Liang al. (2020), Chen et al. (2020a) and Sun et (2020a) apply linear interpolation (Zhang et al., 2017) to word embeddings, hidden states between transformer layers, and encoder outputs, respectively, to augment the original dataset with virtual data points. Data points are virtual because they are not real language inputs. Instead, they representations (e.g.. embeddings). (2) Token replacement. Kobayashi (2018) replaces tokens with their synonyms. Easy Data augmentation (Wei & Zou, 2019) combines synonym replacement, random insertion, random swap, and random deletion. (3) Augmentation with models. Yoo et al. (2021) and Zhou et al. (2021) use GPT-3 (Brown al., 2020) and T5 (Raffel et al., 2020) respectively as the language model to generate new text data of similar types. (1) supports many operations such as linear interpolation (Zhang et al., 2017) and
Work was done when the first author was interning at Google. 1 "Correponndigg author

---

Published as a conference paper at ICLR 2023
small perturbation (Madry et al., 2017). It makes the methods very expressive in generating diverse range of data. However, the newly generated representations (c.g., embeddings) may sit outside of the real data distribution. For instance, word embeddings are converted from vocabulary in the text domain. Performing augmentation at this level may result in representations that do not have their counterparts in the vocabulary. As result, the augmented data may mislead the model to generate shifted decision boundary that can largely affect the qualities (Section 3). (2) can generate in-domain data easily, By using synonym replacement (Wang& & Yang, 2015), new data can be obtained low cost. Despite this good property, this stream of methods lacks the ability to generate diversified data. Subsequently, they contribute little to sampling low-resource data areas and limit the performance gains in practice. (3) generates both diversified and in-domain data using large language models such as GPT-3 (Brown al., 2020) and T5 (Raffel et al., 2020). Due to their large computational overheads, on the other hand, the final distillation quality will be highly limited to the amount of generated data, which is usually not affordable to the scale even tens of thousands of sentences in practice. Figure summarizes the advantages of each augmentation method.
Considering all the approaches above, we pro- pose AugPro, effective and efficient data aug- mentation method for the distillation scenario, which absorbs the advantages above without be- ing limited by their drawbacks. Specifically, AugPro: (1) (effectiveness) is as expressive as representation interpolation: (2) (effectiveness) does not mislead decision boundaries; (3) (effi- ciency) has low computational overhead. In dis- tillation settings, we can always use the teacher to label the hallucinated data in the knowledge distillation scenario. This suggests that we can encourage AugPro to produce diverse data as possible that are not limit to instances with only the same or flipped labels.
Represerratiin Interparation
steraed Decium Bountarie
High Quality
Exprensev
Cheap
Tokon Seplecement
Augmentation With Modelss
Quality
Figure 1: An illustration of each augmentation method's advantages.
Concretely, our method builds on top of repre-
sentation interpolation augmentation methods (property (1)), which does not constrain the generated data to be within small regions of their "parents". The key of AugPro is to convert the augmented data the format of tokens through projections (property (2)) with low-cost operations (property (3)). We conduct our experiments on GLUB (Wang et al., 2018) datasets. Results show that our method could boost the distillation performance significantly with low computational overhead.
To sum up, our contributions are:
We propose an effective and efficient data augmentation method for knowledge distillation.
We empirically evaluate the effectiveness and efficiency of AugPro and theoretically examine that AugPro satisfies three properties under certain circumstances.
2 RELATED WORK
Knowledge Distillation Knowledge distillation was first proposed by (Hinton et al., 2015). It aims to distill knowledge from one model to another by minimizing the distance between the outputs of two models on the same input. With the rise of transformers ( (Vaswani et al., 2017) and BERT (Devlin et al., 2018), more and more attention has been paid the distillation of pre-training language models. Tang et al. (2019) distill fine-tuned BERT to single-layer BiL-TMM network and makes the BiLSTM network as good as ELMo (Peters et al., 2018). Sun et al. (2019) not only distill from outputs but also distill from teacher models' hidden layers. These methods distill language models in the fine-tuning stage, whereas Sanh et al. (2019) and Sun et al. (2020b) focus on distilling language models in the pre-training stage directly make student models aask-agn. TinyBERT (Jiao et al., 2019) distill BERT from both the pre- pre-training training and one-ttnnng stages. We focus on a widely used setting. We distill knowledge in the fine-tuning stage by minimizing the distance between two models' outputs.
Data Augmentation Representation interpolation methods are popular in the computer vision research community. MixUp (Zhang al., 2017) uses linear interpolation get augmented images
2

---

Published as a conference paper at ICLR 2023
and labels. Given an image 21, 22 and their labels y1, y2. MixUp uses linear interpolation to generate a new data point x' and its label y':
x' = MixUp((1, .2) xz1 + (1 A)æ2, v' MixUU((U, U2) Ay1 + (1 A)12
(1)
FGSM (Goodfellow et al., 2014) and PGA (Madry et al., 2017) use gradients to generate adversarial examples. Given an image z, FGSM will generate new data a':
x' = x eSign(
where L. is the loss specific task and é is small value. x' and x have the same label. CutMix (Yun et al., 2019) euts images and then concatenates them together to get methods were originally designed for images, they can be adapted to NLP tasks. Liang et al. (2020) use MixUp on word embeddings to augment data for knowledge distillation. Chen al. (2020b) use MixUp on hidden states between transformer layers. Jindal et al. (2020) also use MixUp on hidden states but consider the effect of mean and variance. Zhang et (2022b) apply PGA to student models' embeddings and leave teacher models' embeddings unchanged. finding that PGA can benefit knowledge distillation. Token replacement methods mainly focus on language inputs. Synonym replacement methods (Kobayashi, 201 8) replace tokens with their synonyms. Easy data augmentation (EDA) (Wei & Zou, 2019) incorporates synonym replacement, random insertion, random swap, and random deletion. TreeMix (Zhang et al., 2022a) uses constituency parser to decide which token should be replaced. Augmentation with models is another approach to generating new data. FlipDA (Zhou al., 2021) uses T5 to generate data that has flipped labels. GPT3Mix (Yoo et al., 2021) designs prompts and uses GPT3 to generate new data. Back translation (Yu et al., 2018) uses neural networks to translate inputs another language and then translate them back. Our method (AugPro) uses representation interpolation as the backbone and utilizes projection to convert representations to tokens with low-cost operations.
one. Though these
MOTIVATING EXAMPLES
Though representation interpolation methods have the good property of generating diverse data, we find that these vision techniques cannot be directly applied to NLP tasks. This is because representation interpolation augments data in a continuous manner, under which case the new data may never exist in the discrete input space, causing the decision boundary shifts.
Take an example of a simple two vooddmensional problem with linear separability (Figure 2). Let X' = 21 be the universe of all data to learn, and y {y1 y2, 93. y1} be the corresponding labels. Suppose we know all of X, y and run the linear support vector machine (SVM) with a hard margin, i.e., ming,5 b || such that (1 x; + b) 2 for all i, we get the solution 9*, ,b" Since is hard to get all data in the real-world setting, we suppose that we only have [3) as the training dataset. If we simply use MixUp, we get #MixUpp as the augmented data, whose label 3Mixtp = sign((3*)) xMixUp + b²) = 3/3 = y/4 Now running the linear SVM with (I2, N4, xMixUp) with labels {82, y4, /MixUp) }, we get 3MixUP and bMixUp- Nevertheless, sign sign(SMisupp + bxiipp) # ys- As comparison, if we project 2MixUp to its nearest neighbor and get 2MixUp-P
(21, 02, 43, 33,,,,,,
22 whose label yMixUp-P = y2. running SVM with {ai 22 2M--Up-P and {y1, 3/, 3MixUp- can get 8MixUp-P and UMAiUP-PP, which can classify all data correctly. Appendix B shows the concret number each parameter.
To this end, augmented data should be the real data in the input space, i.c., the format of tokens, to leverage this problem. This observation leads to our method AugPro which uses projection to convert augmented representations symbolic tokens. Compared with the virtual data point generated by representation interpolation, projection can explore more real data and leads to a lower error (Section 4.2 and Appendix H).
METHODOLOGY
In this section, we first formulate the definition of knowledge distillation in NLP (Hinton et al., 2015) and then introduce method AugPro.
3

---

Published as a conference paper at ICLR 2023
(a) Ground- -ruuth
(b) MixUp SVM
(c) MixUp with Projection SVM
Figure 2: Decision boundary shifts in 2D space for discrete datasets. (a) is the gound- where
31, 31,T3 Fy are the observable training data while 22, 24 with transparent colors mean the unseen data. In (b), one gets augmented data TMixUp with label 1/MixUp -1, 1, and do SVM with { Va, MMixUp] with their labels. In (c), one projects TMixUp to its nearest neighbor I2, and do SVM with (x) I2, J.a. We see that the correction of projection in (c) brings smaller decision boundary shifts than (b).
4.1 KNOWLEDGE DISTILLATION
Knowledge distillation is method to distill knowledge from large-scale models to small-scale models. Formally speaking. considering an NLP classification task, we have a corpus D = {(2,, y1) 1 that contains N input-output pairs, where is an input sentence with tokens r; [W,, V. is the vocabulary, n, is the number of tokens in T₁. Yi is the output label for x,, We use plain texts rather than bold texts for because language inputs a sequence of tokens, which is different from images. Then we distill knowledge from a large-scale model f(; 0r) with parameter Or (i.e., a teacher model) small-scale model g(-, 0g) with parameter Os (i.e., a student model). In practice, Or has much more parameters than 0g. The distillation process can be divided into two stages:
Wi,, wy
Teacher training. Optimize Or on the dataset D. In classification problems, we use cross-entropy loss do empirical risk minimization 0r:
1 CrossEntropy((((:: 0r),
8-- arg min Ox
Σ
N
Student training. Optimize As on the dataset D with both ground-truth labels and outputs from teachers. In classification problems,
8'g arg min LKD arg min 0g
CrossEntropy(g(e,,55)) v.) d(g(z,, 0s), f(x, 67))
0g
(4)
where d(.,> -) is a distance function. In practice, d(---) could be cross-entropy or mean square error.
Empirical results from former studies (Hinton et al., 2015; Sun et al., 2020b; Sanh et al., 2019; Sun al., 2019) show that knowledge distillation will train better f''s because the student model not only learns from ground- labels but also learns the generality from the teacher model.
Note for the student training, we can combine knowledge distillation and data augmentation together:
8's arg min LKD + Laug
0s
where Lang denotes the knowledge distillation loss on augmented data which leads to different variants of methods. As one important way help the student learn more effectively, how generate new data with augmentation loss is the key and major discussion topic in the remaining sections.
4.2 AUGPRO: AUGMENTATION WITH PROJECTION
In this section, we will introduce four variants of Caug: two backbones (MixUp and FGSM) and two AugPro variants building on top of them. Figure 4 shows the concept of our proposed method.
4

---

Published as a conference paper at ICLR 2023
Algorithm 1: AugPro Algorithm
Dataset D = {(X, Y) representation interpolation function lc()), projection function p(.). the teacher model f with fine-tuned parameters Or and the student model with parameter Os that needs to be fine-tuned, vocabulary V, learning rate 7), training steps K, batch size B, sentence length L, embedding dimension H. Output: Fine-tuned 8's
Input:
k
while k < K
Sample data batch B [(1, E B from D ((X,
Get augmented representations Brep = (BB) BxxxxH (e.g.. Equation (1, 2))
Project representations to tokens B' = (B)))) E V BxL (e.g. Equation 5)
Use Equation (4) to compute LKD.
Compute loss Laug (e.g., Equation (6, 7)) based on B', and g.
0g 0g
(CRD LAug), k k
return As
AugPro builds on top of representation interpolation augmentation methods. The pipeline (Algorithm 1) can be divided into three steps: (1) We first get augmented representations (i.e. h(-)). (2) Then use projection convert representations to tokens (i.e. p(-))). (3) At last, compute CAug to update student models. The key to step (2) is projection. Concretely, language models map tokens representations, and projection aims find an inverse mapping map representations back to tokens. This way, AugPro will avoid shifting decision boundaries (Section 3). However, the inverse mapping hard to find in practice. First, popular language model architectures such as transformers (Vaswani et al., 2017) are usually complex and are hard to get the inverse mapping. Second, the input space of language discrete, making the mapping irreversible. Thus, we could only use the approximation technique to get the approximated inverse mapping. To this end, we focus on the inverse mapping on the embedding level. First, the embedding mapping's structure is much simpler than the transformer layer and is straightforward for to find the inverse mapping. Second, we can use the nearest-neighbors as the approximation method, which is a cheap approximation.
Based on the analysis above, we use the saurest-neighboss to find our projection, i.e., the function p()) in the Algorithm 1. AugPro does not rely on specific representation interpolation methods. In this paper, apply AugPro to MixUp and FGSM, i.e. the h(-) in Algorithm is Mix FGSM as in Equation (1,2). We will illustrate two variants to perform projection (step (2)) and compute Laug loss (step (3)) in the following texts.
We slightly abuse the notion of f and to illustrate AugPro better. We divide f and g into two parts: the first part is embedding function that maps tokens to embedding vectors (se and ge, e denotes embeddings), the rest is the second part (fz and gr. denotes layers). Under this definition, f = fi f fe and g gi o ge-
AugPro-Mix We get AugPro-Mix by applying the AugPro paradigm to MixUp. First, we ap- ply MixUp on word embeddings and labels. This gives us embeddings from the teacher exxupp' embeddings from the student XxxUpp= and the label YMixUp- Laug becomes
M
LMixUPP
1 MM.
(rrssFEntropy(on(@aaauu 0s), +( Os), fi f 0% ))
when we use MixUp data construct losses, where M denotes the number of augmented data. For AugPro, we use the nearest-neighhors to get AugPro-Mix tokens x Aug
Auggroo-Mis [WAngTro.Mxx,,, ... " Ang Mix.n]
(5)
where Pri max Sim(e (1), fc(w))
5

---

Published as a conference paper at ICLR 2023
Sim means the similarity function, which can be the cosine similarity. e(i) means the ith embedding vector e. Concrete examples can be found in Appendix D. Then the loss function Laug becomes:
M
1 M 1=1
LAugPro-Mix = d(g(FAugrro-Mxx,.03) (( ( 03-))
(6)
We do not use yMixup because the projection operation (the nearest-neighbors in AugPro-Mix) does not necessarily preserve the label.
AugPro-FGSM Though adversarial examples (AE) are originally aimed to improve the robustness of models and may harm the performance clean inputs (Raghunathan et al., 2019), Zhang et al. (2022b) shows that AE can benefit knowledge distillation. We get AugPro-FGSM by applying AugPro FGSM. We first apply FGSM to the student model and get augmented data ffGMM The augmented data can be used to construct Luu directly:
M Σ
LFGSM
de dg 0s), dol 0r))
M
1=1
Following Equation 5, we could get I AugPPro FGSM by changing the footnotes accordingly.
We usually in Equation (2) large in AugPro-FGSM since do not want AAngPro-PGMM to be the same as the original input, whereas e in Equation (2) is a small value in FGSM. We use the cosine similarity to implement the Sim function. The loss function Lang becomes:
M Σ
LAngPro.PGSM
Label Diversity We take two sentences from SST-2 dataset (Socher et al., 2013) as example to further explain that projection does not necessarily preserve labels but generates diverse labels. The first sentence is watch on video at home with the sentiment Neutral. The second sentence is as good with the sentiment Posit ve. Then we can get the AugPro-Mix sentence watch home. Obviously the label of AugPro-Mix sentence should be Posi iive
good vi deo rather than the linear interpolation of Pos i ve and Neut ral. This is the desired property in distillation as will use the teacher to label these newly generated data points.
Computational Overhead. If we assume the complexity of computing cosine similarity between two vectors is O(d) where is the dimension of vectors, then the complexity of the projection (the nearest-neighborg in our implementation) is ( d), where N is the sentence length and V is the vocabulary size. N is usually within hundreds. V is usually around 30, 000 in popular pre-train language models using sub-word tokens such BERT (Devlin et al., 2018) 1) and T5 (Raffel et al., 2020). As result, (NVd) brings little costs. On the other hand, the projection operation could be parallelized since the NV similarity calculations do not affect each other. In modern parallel computing architectures, such as GPUs and TPUs, projection can be calculated in a much faster manner. Compared to the major large-scale language models' complexities, this computation will take a small portion of resources. The detailed running time comparison can be found in Section 5.2.
Three Properties of AugPro. Since AugPro supports operations used in representation interpola- tion methods, AugPro is expressive (property (1)). AugPro also converts representations to tokens avoid shifting decision boundaries, leading smaller error rate (property (2)). It can be shown that AugPro-Mix has a aN lower error rate than MixUp, and AugPro-FGSM has 2V lower error rate than FGSM with certain assumptions (Appendix H). Moreover, AugPro has low computational overhead to guarantee efficiency (property (3)), as described in the previous paragraph.
5 EXPERIMENTS
Our experiments aim to answer two questions: (1) How effective is AugPro when applied to the knowledge distillation scenario? (2) Is AugPro efficient?
Datasets and Settings Following previous knowledge distillation works (Liang et al., 2020; Zhang al., 2022b), GLUE (Wang al., 2018) datasets the benchmark. We use EncT5 (Liu

---

Published as a conference paper at ICLR 2023
et al., 2021) our teacher and student models for the following reaons: (1) T5 has much better performance than BERT and is close to SOTA in many tasks. EncTS is a simplified version of T5 that uses the whole encoders of T5 but only one decoder layer. EncT5 performs similarly to T5 on classification tasks such as GLUE tasks with fewer parameters. For example, Enc'TS (small) only contains 37M parameters but can perform similarly to T5 (small), which contains 77M parameters. Using En will make the results more convincing and show that our method is still useful even with powerful models. (2) Previous methods (Liang al., 2020; Zhang et al., 2022b) distill knowledge from 12 12-layer BERT to a 6-layer BERT or a 3-layer BERT. However, the gap between the teacher and student models is marginal. Therefore, the improvement space limited, and the existence of variance will weaken the credibility of the results. To this end, we distill knowledge from E (Large, 24-layer, 354M, teacher) to EncT5 (small, 8-layer, 37M, student), as the two models have a significant performance gap.
Baselines and Training We train several baselines for comparison: (1) Fine-Tuning (FT): We directly fine-tune EncT5 on the dataset. (2) Knowledge Distillation (KD): We first fine-tune teacher model (EncT5 Large), then distill knowledge from the teacher model to the student model (EncT5 Small). (3) Knowledge Distillation + Back Translation (KD+BT): Back translation (Yu et al., 2018) first translates input to another language and then translates it back. We choose back translation representative method for the data augmentation type "augmentation with models". (4) Knowledge Distillation + K-Nearest-Neighhors (KD+KNN) KNN (Wang & Yang, 2015) first selects tokens from inputs, then replaces them with the K nearest neighbors in the embedding space. KNN can be regarded as one token replacement method. (5) KD+MixUp (6) KD+FGSM (7) KD+ TMiX (Chen et al., 2020b) MixUp on the hidden state between transformer layers. The last three methods are of the "representation interpolation" type. We train student models with 0.6M steps and 512 batch size. Due the high computation cost, we only augment data to twice as large as the original dataset size for back translation. For all other methods, we augment data to twice large as the original batch size for each batch, i.e., we augment 0.6M steps 512 batch size- = 307. data in total. More training details are in Appendix C.
5.1 EFFECTIVENESS OF AUGPRO
Table shows the results of knowledge distillation. Due the high cost, we only report back translation results on the RTE dataset. We first use the training data to train a teacher model and then distill knowledge from the teacher model to the student model on the training data. We can conclude that: (1) All data augmentation methods will benefit the distillation. (2) AugPro can significantly improve the distillation performance compared with corresponding baselines. Specifically, AugPro is extremely useful for low-resource datasets such as CoLA and RTE. AugPro-Mix achieves scores 5.97% and 9.02% higher than MixUp on CoLA and RTE, respectively. AugPro-FGSM achieves scores 10.52% and 8.31% % higher than FGSM on CoLA and RTE, respectively. For large datasets such as MNLI, AugPro-Mix and AugPro-FGSM can also improve the performance. (3) Moreover, combining AugPro- FGSM and AugPro-Mix achieves the best performance in all listed methods. Compared with vanilla knowledge distillation, combining AugPro-Min and AugPro- -GSM improves the performance from 2% to 14%.
Table 2 uses a different setting from Table 1. We only keep 10% training data labeled and assume others unlabeled. Then we use labeled training data to train teacher model and unlabeled training data to do knowledge distillation this is a more realistic setting since it is often easier to get unlabeled data than to get labeled data. The conclusions above still hold. Specifically, AugPro can improve the accuracy from 1% to 2% on average on three datasets. Compared with the vanilla distillation, AugPro can improve around 2% accuracy at most on three datasets.
5.2 EFFICIENCY OF AUGPRO
The efficiency of AugPro lies in two aspects. First, its complexity is low. Second, it can be computed in parallel To fully demonstrate these two advantages, report the real-time cost of AugPro and baselines in Table 3. KD+data augmentation is rough twice the time of vanilla KD since these methods use twice the data vanilla KD. We can also observe that augmentation with models (KD+BT) takes much more time than other kinds of baselines, which shows that this method is not efficient enough. At last, AugPro brings little computational overhead the time cost is the same as
7

---

Published as a conference paper at ICLR 2023
SST-2
CoLA Matthew
MNLI-MM/MM
QNLI
QQP
MIPC
STS-B
RTE
Acc
Acc
Acc
Acc/F1
Acc/F1
PCIC
Acc
67.3k
8.5k
392.7k
104.7k
363.8
3.7k
5.7k
2.5k
Ene F5 (354M)
97:20
63.60
91.40/91.10
95.40
92.73/90000
91.42/93.30
88.19/88.10
86.30
FF (37M)
92.89
45.81
/860//81.6
89.81
91,45/88.11
87.00 /90.9)
86.39/85.99 94
59:21
EmcT5;-KD
92.08
45.56
85.301 185.6
80.46
91,36/ 08.24
84.56 88.85
81.20/87.18
61.37
+KNN
94:27
54.00
87. 13 18//87..8 01
91.64
04.144/80.0
86.03 190 390
87. 87.14/87 277
06. 76
+TMix
93.35
42.12
86.79/86.77 /86.99
91.10
91.78/88.88
87.25 /9099
87.52/87.58
63.18
+MixUp
93:23
51.63
86.73,
91.31
91.82/ (222//8,.71
88, 18-18/91.0
87.47/87
62.82
+AngPro-Mix
94.38
57.60
8T.40/ (800/87..5 277
92.06
92.06/10 /0////0.23 23
50.46/92.35
88, 10/87.88 .87
71.84
+FGSM
46.37
85,
84. 84156
87,
PASSPPo-PGSM
94.61
36.89
887002 /80.85
91.07
922 10/80.26
88.24/91.8 1.67
87 87.64/877 (87.5)
70:40
+FGSM-MixUp
98.12
50..9
86, 188,
91.09
91.73/
87.25 (90,88
87. 15/11///// 11187.000
62.82
95.18
59.01
87.97/87.88
02.92
92.30/89.54
89.46/92.42
88.34/9A0 98.04
74.73
Table 1:
Knowledge distillation on the GLUE dataset. We first use the training data train teacher model and then distill knowledge from the teacher model to the student model on the training data. E denotes EncT5 with I, transformer layers. L = 24 and L denote the teacher model with 354M parameters and the student model with 37M parameters, respectively.
KD
+MixUp
Aug
PGSM
AAgPro-PGSMM
+PGSMI-MixUp
MNLI-MMMM
86.81/80.99
(85.53/85.8
85.07/81.55
87/85
35.70/85.05
SST-2
92.08
93.35
91.04
92.09
94.27
83.06
94.04
QNLI
80.68
00:28
90.98
80.00
91.03
90.50
91.58
Table 2:
Knowledge distillation on the GLUE dataset with different setting from Table We regard 10% of the data as labeled and the rest as unlabeled. The teacher model is first trained on labeled training data and then used for knowledge distillation on unlabeled training data.
the baselines. Results also show that KNN is much slower than other methods, which is explained in Appendix G.
5.3 ABLATION STUDY
In ablation studies, we follow settings used in Table unless otherwise stated.
Perturbation scale for in AugPro-FGSM The key hyperparameter in AugPro-FGSM is e in Equation (2). Small e will make ZAugPro-FGSM the same as the original input. Large e tends to make AugPto-FGSM hard to understand, meaningless, and out of the domain. Therefore, a proper is essential. Our experiments find that 35 is the best fit for T5 embeddings. Table 4 shows KD+AugPro-FGSM performance with different e.
Signs of gradients in AugPro-FGSM are not important The effectiveness of AugPro-FGSM comes from gradients' signs and the projection AugPro. To prove that AugPro-FGSM mainly benefits from AugPro, we implement two AugPro-FGSM variants: AugPro-FGSMID (Descent Projection) that uses the opposite signs to AugPro-FGSM and AugPro-FGSMR (Random Projection) that uses random signs. Table 6 shows the results of AugPro-FGSM and its two variants. We can observe AugPro-FGSM has a similar score to its variants in all settings. Thus AugPro-FGss FGSM mainly benefits from AugPro. We also conduct experiments that follow the setting of Table 2, and results can be found in Appendix E.
AugPro generates diverse labels We show that AugPro generates diverse labels at the end of Section 4. Here we empirically show that assuming AugPro preserving labels may harm performance. If AugPro preserves labels, AugPro-Mix and AugPro-FGSM data should have the same labels as Mi and original data, respectively. We use these augmented data together with labels to fine-tune student models directly. Results in Table 5 suggest that such augmented data and labels may harm performance. Therefore, AugPro generates diverse labels and does not necessarily preserve labels.
AugPro consistently benefits KD with different data sizes Figure 3 shows the performance of AugPro with different data sizes. It can be observed that AugPro is better than all baselines in all data sizes. Moreover, AugPro extremely useful when the data size small. For example, AugPro can
Converted time. We BT on TPU v2 and compute the equivalent time cost.
8

---

Published as a conference paper at ICLR 2023
KD
+BT
+KNN
+TMix
+MixUp
+AugPro-Mix
+FGSM
+AugPro-FGSM
Time (min)
1.68
13.15
4.57
3.48
3.48
3.48
3.24
3.24
Table 3: Time (minutes) costs every 1000 steps average of various methods on 8 TPU v3 slices. Costs contain data augmentation, the forward pass, and the backpropagation. The table divided into four parts. Each part contains specific data augmentation type (KD, augmentation with models, token replacement, representation interpolation and AugPro).
MNLI-MM/M
SST-2
QNLI
KD
MNLI-MMMM
SST-2
30
55.60/85.28
93.69
90.44
+AugPro-PGSM
87.02/86.88 86.8
94.61
35
85.87 85.87/85.82 85.82
94.27
91.03
*AugPro-FGSMD
87.08 0888/86.9
94.27
+AugPro-FGSMR
86.57/ 86.52
94.27
40
85.79/85 85.58
93.23
90.81
+MixUUptAAugPro-PGSMM
87.35/87.41
100
85.18/84.74
91.97
90.44
-NixUUp+AAugPPr-PPGGMM
87.45/87.44
94.04
-MfixUp4-AugPro-PPGMM
87.48/87.49
94.15
Table 4: KD+AugPro-FGSM performance with
-AugPPo-Mixx-AugPP...CC
87.97 /87.7
95.18
87.81/87.67
94.61
different c.
87.77/57..0
94.95
Data Size
20%
50%
100%
Finetune
80.99/80.71
83.43/62..55
84.69/84.20
Table 6: FD+A and its vari- ants performance with different signs. AugPro- FGSMD denotes FGSM with Decent Pro- jection. AugPro-FGSMD uses the opposite sign AugPro-FGSM AugPro-FGSMR de- notes FGSM with Random Projection. AugPro- FGSMR uses the random sign.
AugPro-Mix
80.62/80.44
81,24/82.60 60
84.53/81.92
AugPro-FGSM
80.74/80.32
83.16/82.44
84.37/83.61
Table 5: AugPro-Mix and AugPro-FGSM are used to fine-tune student models. MixUp la- bels are used for AugPro-Mix data. AugPro- FGSM uses the original label.
<4- AugPro-TGMM
AugFroHii
AuggrooMlix + AugPro-FESM MidUp FSSM
AuSPro-TGMM
AugProMii
AugPro-Mix - AugPra-rGOM MixUp FGSM
FGSM
HidUp
FGGM
NidUp
85
02
91
90
20
40
60
80
100
40
100
Data Size/ 1%
Data Size %
(a) SST-2
(b) MNLI-M
Figure 3: AugPro performance with different data sizes. Figure (a) and Figure (b) for SST-2 dataset and MNLI-M dataset. Blue lines (or triangle markers) are AugPro methods. Yellow lines (or diamond markers) are baseline methods. The green line (or X marker) is KD. AugPro has the same line type the corresponding baseline. For example, AugPro-FGSM and FGSM are all dashed lines.
improve the accuracy of 4% (SST-2) and 6% (MNLI-M) when the data size is 10%. We also report results on the MNLI-MM dataset in Appendix F.
6 CONCLUSIONS AND FUTURE WORK
We propose AugPro, an effective and efficient data augmentation paradigm for knowledge distillation. We use projections to tackle the problem of shifting decision boundaries caused by traditional repre- sentation interpolation methods knowledge distillation. Moreover, AugPro has low computation costs and is fast in modern computing architectures. Results on GLUE tasks prove the effectiveness and efficiency of AugPro. In the future, we will further explore the impact of AugPro on labels to make it helpful in other scenarios.

---

Published as a conference paper at ICLR 2023
REFERENCES
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Ben- jamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL httpss: pttttps:/pprrceeeeee wwwpproooednngge ing. neur ipsrips... scc/paper/20200 wtt.//iilee/Aa httpe///450dddffc/aaaaaaa 6741 Bbf 42f pdd.
Jiaao Chen, Zichao Yang, and Diyi Yang. MixText: Linguisticalll--nfforne interpolation of hid- den space for semi-supervised text classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2147-2157, Online, July 2020a. Asso- ciation for Computational Linguistics. doi: 10. 1/205// org/2020 acl- mmain. 194.
URL https:
Jiaao Chen, Zichao Yang, and Diyi Yang. Mixtext: Linguistically-innormed interpolation of hidden space for semi-supervised text classification. arXiv preprint arXiv:2004. 122399, 2020b.
Felix Chern, Blake Hechtman, Andy Davis, Ruiqi Guo, David Majnemer, and Sanjiv Kumar. Tpu-knn: K nearest neighbor search at peak flop/s. arXiv preprint 2D0:22266/42886 2022.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 2018.
David A Freedman. On tail probabilities for martingales. the Annals of Probability, pp. 100-118, 1975.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 1rXiv:1412.6572, 2014.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, al. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0255: 2(7), 2015.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint 1909. 2019.
Amit Jindal, Arijit Ghosh Chowdhury, Aniket Didolkar, Di Jin, Ramit Sawhney, and Rajiv Ratn Shah. Augmenting NLP models using latent feature interpolations. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 6931-6936, Barcelona, Spain (Online), December 2020. International Committee Computational Linguistics. doi: (0.18653/v1/2020 coling- main.61 URL https: htt//claaaaa /alaaaaaannoooyoyyooooooo org /200 col weli-inggall 611.
Nora Kassner and Hinrich Schütze. Bert-knn: Adding knn search component pretrained language models for better qa. In Findings of the Association for Computational Linguistics: EMNLP 2020,
pp. 3424 3430, 2020.
Sosuke Kobayashi. Contextual augmentation: Data augmentation by words with paradigmatic relations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Short Papers), pp. 452-457, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/11N118-20777 URL https: / acc antho wwg..orgg/ii 8////11111000 2072.
Kevin J Liang, Weituo Hao, Dinghan Shen, Yufan Zhou, Weizhu Chen, Changyou Chen, and Lawrence Carin. Mixkd: Towards efficient distillation of large-scale language models. In Interna- fional Conference on Learning Representations, 2020.
Frederick Liu, Siamak Shakeri, Hongkun Yu, and Jing Li. Enct5: Fine-tuning t5 encoder for non-autoregressive tasks. arXiv preprint arXiv:21 10.08426, 2021.
10

---

Published as a conference paper at ICLR 2023
Yaojie Lu, Hongyu Lin, Jin Xu, Xianpei Han, Jialong Tang, Annan Li, Le Sun, Meng Liao, and Shaoyi Chen. Text2event: Controllable sequence-to-strucae generation for end-to-end event extraction. arXiv preprint 200.0:22006002232, 2021.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks, arXiv preprint arXiv: 1eee::106.0... 2017.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 2227-2237, 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21 21(140):1667, 2020.
Aditi Raghunathan, Sang Michael Xie, Fanny Yang. John C Duchi, and Percy Liang. Adversarial training can hurt generalization. arXiv preprint arXiv: (006:v:990000033 2019.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.011000 2019.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https: / www. acl web. org/ httpg///nhhologyyggg DD1311110001111110.
Lichao Sun, Congying Xia, Wenpeng Yin, Tingting Liang, Philip S Yu, and Lifang He. Mixup- transformer: dynamic data augmentation for nlp tasks. arXiv preprint arXiv:2010.02394 2020a.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 4323-4332 2019.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: a compact task agnossgc bert for resource-limited devices. arXiv preprint arXiv:2004. 02984, 2020b.
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task- specific knowledge from bert into simple neural networks. arXiv preprint arXiv: 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 1..18653/v1/111 NIWW8885446. URL
/ /w18 8188--54666
William Yang Wang and Diyi Yang. That's so annoying!!!: A lexical and frame-semantic embedding based data augmentation approach to automatic categorization of annoying behaviors using# petpeeve tweets. In Proceedings of the 2015 conference on empirical methods natural language processing. pp. 2557-2563, 2015.
Zihan Wang and Bo Yang. Attention- bidirectional long short-term memory networks for relation classification using knowledge distillation from bert. In 2020 IEEE Intl Conf De- pendable, Autonomic and Secure Computing. Intl Conf on Pervasive Intelligence and Com- puting, Intl Conf on Cloud and Big Data Computing, Intl Conf Cyber Science and Tech- nology Congress CCCCCP/CCCCCCCCCCCCC/Ceeeeeeeee CAICC-PCCo-CIiee het.eommeCyyyeeeeeeeeeee9e100000
pp. 562-568, 2020. doi: 10.1109/

---

Published as a conference paper at ICLR 2023
Jason Wei and Kai Zou. Eda: Easy data augmentation techniques for boosting performance text classification tasks. arXiv preprint arXiv:1901.11 2019.
Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, and Woomyeong Park. Gpt3mix:
Leveraging large-scale language models for text augmentation. arXiv preprint arXiv:2104.0882. 2021.
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoe V Le. Qanet: Combining local convolution with global self-attention for reading comprehension. arXiv preprint 18X1v:180.00944, 2018.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference computer vision, pp.
2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint
2017.
Le Zhang, Zichao Yang, and Diyi Yang. TreeMix: Compositional constituency-based data aug- mentation for natural language understanding. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5243-5258, Seattle, United States, July 2022a. Association for Computational Linguistics. doi: 00.18653/v1/2022..og 2022. www.aaaallaaam maaacl-maing 385.
hww.://..a..all-la//.// 2at.:.2aaacmmmmi33...5.. URL https: / aaaaacaannn aac.aaaaantoooyyyy.oooooo org/
Minjia Zhang, Niranjan Uma Naresh, and Yuxiong He. Adversarial data augmentation for task- specific knowledge distillation of pre-trained transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 1165-11 11693, 2022b.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre- pre-trained transformer language models. arXiv preprint arXiv:2205.0 01068, 2022c.
Jing Zhou, Yanan Zheng, Jie Tang, Jian Li, and Zhilin Yang. Flipda: Effective and robust data augmentation for shot learning. arXiv preprint arXiv:2 2021.
12

---

Published as a conference paper at ICLR 2023
A THE CONCEPT FIGURE OF AUGPRO
Here we show a concept figure (Figure 4) to let readers better understand the difference between AugPro (e.g., AugPro-Mix) and previous works (e.g., MixUp (Zhang et al., 2017)).
Knowledge Distilation
Knowledge Distilattoon
Student Model
Student Model
Tencher Model
MixUp Input
MixUp Input
MixUp lapur
Projection
Teacher Embedding
Student Embedding
Teacher Embeoling
Mislpp
MisUp
MisUp
Inputl
Input2
Inpett
AugPro-Mai Mix
Figure 4: Left: MixUp with knowledge distillation. Right: AugPro-Mix with knowledge distillation.
B CONCRETE NUMBERS OF THE EXAMPLE IN SECTION 3
X {21 = (2.5,2), (3 (-2.5, 22, 2. (-2,2)}
)' {11 = +1,122 +1,93s -,,, 11
-4/99,9,-1/18 00
TMixtip 25 dig 12 5 BRdixUp (250/533, 200833330)33) bmixtp 112113
3M = (4/9, 0), bM 1/9
C IMPLEMENTATION DETAILS
C.I HYPERPARAMETERS
We use JAX and T5X to implement EncT5 and AugPro, and use T5 1.1 checkpoints to initialize models. The batch size 512, and the maximum sentence length is 128. Training steps are 0.6M for all experiments. We use 8 TPU v3 slices to do all experiments.
To fine-tune the teacher model, we use a dropout rate of 0.1 and a learning rate of 1e-3 for all GLUE tasks.
For knowledge distillation, we set the dropout rate to 0.1 for both the teacher and student models. We find that adding dropout to teachers will make the distillation better. We run all experiments with le-3 and le-5 learning rates and report the best results. As result, learning rate set to le-5 for all experiments the STSB dataset, En and KnDT5;-KD experiments on CoLA, MRPC and RTE datasets. All other experiments use the learning rate le-3.
The À (Equation (1)) for AugPro-Mix is 0.5. Previous works (Zhang al., 2017: Liang al., 2020) use beta distribution to sample for MixUp. We try Beta(0.4, 0.4) and X = 0.5 for MixUp and find they have similar performance in most tasks. In some tasks such CoLA, À 0.5 is better. Therefore, we use 0.5 for MixUp.
Following previous works (Zhou et al., 2021), set k 15 for the KNN baseline and randomly select 0.1 portion of tokens to replace. We use outputs of the 4th layer of the student model and the 12th layer of the teacher model, i.e., the middle layer of both models, conduct TMix experiments.
13

---

Published as a conference paper at ICLR 2023
C.2 IMPLEMENTATION DETAIL OF EQUATION 5
Algorithm shows the AugPro pipeline. Here, we show a detailed implementation of Equation 5 in Algorithm 2. We believe these algorithms can help readers reproduce our methods and results.
Algorithm 2: Projection Algorithm
Input: Augmented representations Brep E DxLx where B is the batch size, L the sentence length, and H is the embedding dimension. Vocabulary V contains Z tokens. Token embeddings
E IR xx
Output: Projected augmented data B' VVxL
Compute the similarity between each token pair Sims Brep" E E R
Find the nearest neighbor B arg min Sims E Bxxt
amis=2
Find tokens according to indices to get B'
return E
D EXAMPLES OF AUGPRO-MIX AND AUGPRO-FGSM
We use three MNLI inputs show the augmented data by AugPro-Mix and AugPro-FGSM.
Following are three MNLI inputs:
(i) hypothesis:
The judgments need to consider the broader public interest.
premise:
These judgments need to be made in a consistent manner with consideration of the broader public interest in the program or activity under review.
(ii) hypothesis: agree.
premise: yeah i i agree with that
(iii hypothesis: She's s never been to a hospital.
premise: and uh as if that wasn't bad enough the the ones that were half alive that they rushed to the hospital and she got work on and she got to see them die and and they just get all the world's worst situations very few rewarding situations
and
If we use AugPro-Mix on (i) and (ii), (ii) and (iii), we will get:
(i+ ii)
hypothesis: I judgment.
premise: yeah judgmenti needi agree made in (ii + iii) hypothesis: She agree. never been to hospital. premise: yeah uh as agreeif that wasn't t bad enough the the ones that were half alive that they rushed to the hospital and she got to work on and she got to see them die and uh and they just get all the world's worst situations very few rewarding situations uh an
If we use AugPro-FGSM on (i), (ii) and (iii,, will get:
(i) hypothesis: River judgment Mit handy to consider the broader public interest. premise:
These judgment Mit handy to be made in consistent manner with consideration- the broader public interest in the federal clädire activity under reviewed.
(ii) hypothesis: Kyle agree.
premise: yeah chambres chambres agree with that
14

---

Published as a conference paper at ICLR 2023
SST-2
CoLA Matthew 8.5k
MNLI-MM/MM
QNLI
QQP
MIRC
STS-B POISC
RTE
Acc
Ace
Acc/F11
67.3k
392.7k
104.7k
363.8k
3.7k
5.7k
2.5k
FTT (354M)
97.20
03.60
91.40/91.10
95.40
92.73/90. 90.00
91.42/93.3
88, 19/11//8800
86.30
EncT3,-FT (37M)
92.89
45.84
84.69/ $4.26
89.84
91,45/88 41
87.01 87.01/90.3) /90.91
86.39 /85.94
59.21
EncT3,-KD
92.09
45.56
85.93/85.61
89.46
91 91.36/88.21
51.56/88.88
87 87.29/87
18
61.37
+Random
92.78
45.89
86.37/ /85.55
89.35
91.42, 112/888.43
84.07 34.07/888.93 /88.90
87.72/87..8
61.37
+MixUp
93.23
61.63
56.73/ 96, 69
91.31
91. 82/88/888.. .97
88. 18, 18/91.68
87. /87.33
62.82
+AugPro-Mix
94.38
67.60
87. 97.40/S7.22
92.06
92.06/88 /89.23
89-46/92.34
88. 10/10//7:87
Table 7: Results of the random generation baseline.
(iii) hypothesis: She' Mit never been to hospital,
premise: and uh if that wasn' financing bad enough
the the ones that are half alive that they rushed to the hospital and she got to work and she got to see them die and uh and they just get all the world' Mit worst situations very few rewarding situations uh and
The above-augmented sentences are originally token lists, but not real sentences. Luckily, TS uses Sen- tencePiece construct its vocabulary, which supports the precise de-tokenization for any token list. To convert token lists to sentences, we use simple command join (tokens) rrelaace (" ").
Linguistic Analysis. Our motivation focuses on the perspective of machine learning, i.e., avoiding shifting decision boundaries by converting representations tokens. Here, we would like add brief analysis from a linguistic perspective. We observe that the above-augmented sentences may have grammatical errors, "meaningless" tokens, and may be less meaningful than original sentences. However, "meaningless" to humans does not suggest meaningless to models, AugPro indeed boosts the distillation performance. Besides, augmented sentences are not totally semantically meaningless to humans. It is hard to why augmented data is helpful from the linguistic perspective, suggesting that we should focus on analyzing these data from the machine learning conduct simple
perspective, which is exactly our motivation. To further support motivation, baseline "random generation" that randomly chooses tokens from the vocabulary and concatenates them to form an augmented sentence, Random generation can generate meaningless sentences easily, Results are shown in Table 7. We can conclude that random generation is poor augmentation method, suggesting that AugPro helpful not because of the meaningful or meaningless semanties, but because of avoiding shifting decision boundaries.
E SIGNS OF GRADIENTS IN AUGPRO-FGSM IS NOT IMPORTANT (MORE RESULTS)
Table shows the effect of signs of gradients in AugPro-FGSM with the setting of Table The conclusion is same as the conslusion concluded from Table 6.
F AUGPRO CONSISTENTLY BENEFITS KD WITH DIFFERENT DATA SIZES (MORE RESULTS)
The main texts show that AugPro consistently benefits KD with different data sizes SST-2 and MNLI-M datasets. Figure 5 shows this conclusion still holds on the MNLI-MM dataset. The overall trend is similar the MNLI-M dataset.
G KNN IS SLOWER THAN OTHER METHODS
There two reasons that KNN is slower than other methods. First, KNN has a higher computational complexity ONNV dlogk) than AugPro (O(NV)). Second, KNN is hard to implemented by XLA. A popular and fast implementation of KNN is np. mp.argpartition However, XIAA
15

---

Published as a conference paper at ICLR 2023
MNLI-MM/M
SST-2
QNLI
KD+AugPro-FGSM
85.87 85.87/85.82 885.82
94.27
91.03
KD+AugPro-FGSMD
85.87 85.87/85.52 /8.52
93.92
91.05
KD+AugPro-FCSMR
85, 85.6/85.39
93.35
90.85
KD+MMiKUje-AugPPoPGGSM
86,20/800 06.20/86.28
93.81
90.99
AD-MMIJUpP-AuggPPP-FGGSDD
86.44/80 $6.19..19
93.92
90.98
KD-MiixlpprAAggPPr-PGSSMM
86.38/86..o 86.42
94.04
86.76, /86.81
86.59 86655
93.81
91.56
86.62/866 /86.48
93.66
91.4
Table 8: KD+AugPro-FGGSS and its variants performance with different signs. AugPro-FGSMi de- notes FGSM with Decent Projection. AugPro-FGSMi uses the opposite sign to AugPro-FGSM. AugPro-FGSMR denotes FGSM with Random Projection. AugPro-FGSMR uses the random sign. This table follows the setting in Table 2.
AugPro-FGSM
AugPro-Mix Mix MixUp
AugPro-Mii Mix AugFro-FGSM
FGSM
MixUp FGSM
KD
88
Accuracy
85
40
80
100
Data Size %%
Figure 5: AugPro performance with different data sizes on MNLI-MM dataset. Blue lines (or triangle markers) AugPro methods. Yellow lines (or diamond markers) baseline methods. The green line (or X marker) KD. AugPro has the same line type as the corresponding baseline. For example, AugPro-FGSM and FGSM are all dashed lines.
16

---

Published as a conference paper at ICLR 2023
does not support peartition operations", making KNN hard be implemented with JAX TPUs. To this end, Chern et al. (2022) propose another method to run KNN on TPUs at peak FLOP/s. We use their method to implement KNN, but such implementation may not be the optimal solution for KNN.
H AUGPRO CAN ACHIEVE LESS ERROR RATES
Suppose X- (-1, 1|2 log is the universe of all data with labels Y to learn. Define a distribution P such that (a, 1) ~ d if is uniformly independently drawn from and y uiformly independently drawn from 1-1,+13 XD is the training data compositing of samples uniformly and independently drawn from X.
Let x(j) be the j-th coordinate of vector We construct AAugPro-Mix as follows: Index the clements in Ad arbitrarily, and 2; denotes the i-th element. For each positive integer i [n/2], construct augmented data 2; such that, for each coordinate j (2 log n], 2x(i) = x x2)-1 = x2; otherwise 2; (j) is uniformly chosen from {1, (1.-1).
Suppose is
Let's consider optimal algorithms AA and AAugPro-Mix A can only observe A'D (or XMixUpp- XFGSM) and their labels D'D (or VitixUp J/GSMM). For proof, we assume A can only observe A'd and their labels V'. The other two be proved similarly. AAugPro.Mix can observe X'ugPro. and their corresponding labels J'AugProMixx-
Define the generalization error error(AA PI(x.g) P.'n. [A(x, XD, * v). where 3', E (+1,-1} is the prediction of x outputted by A, and error(AAugEro-Mtx) Pr(my)) ((Paa)) Mano MAn,, MAugPro-Min Y'ugPro + y). We have the following claim:
Claim 1. One has
(A)or(A) error ArToorAAAgPoo..... 1. 1/(4n)
lim 22000
As for constructing XAngProPGMM: for each I; E Ad, we add Gaussian (N0,4) 4) each coordinate of Fú, and then project back to X get w;. Simi- larly, assume AAuSPro-FOSM can observe the dataset XAugPro-FOSM and its labels FGAM, and define the generalization error be error(AAugPro-FCMM Pr(ma)) P. (CaM
A(x, XAugP:o-PGSM )'AuPto-FGGSM) & y).
Claim 2. One has
error(A) -error(A 1.
lim
n-000
1/(2n)
H.1 PROOF OF CLAIM
We provide the following preliminary background of martingale to prove Claim for completeness. Definition (Martingale). A sequence of random variables Y1, Y2, ... is martingale with respect to another sequence X1, X2, ... for all n, E||Y,]] | < оo and E[ Xx,..., .,,. Yn, Definition (Martingale Difference). {Du}ez is a martingale difference sequence w.r.t. { if all n:
* Dn is a measurable function of X1, X,
D|] <0
E[Da+1 XI...., [X,] 0.
If Y& is martingale, then D₁ Y8 Y is martingale difference. For martingale difference, have the following theorem:
17

---

Published as a conference paper at ICLR 2023
Proposition 1 (Freedman's Inequality, Freedman (1975)). Consider real-valued martingale dif- ference sequence {X1} which is uniformly bounded, i.e. [Xe] I M almost surely for all Define the predictable quadratic variation process of the martingale W, == مرز [[X] Fj-1 all t, where {Fi} is the filtration. Then for all i 2 0 and 0²
O.
S 2eexp {
-B 1 +2+ A1/33/
Pr
3k
Σ X 2
W& S 0²
With these tools, are ready prove the claim.
Proof of Claim I. In our setting, it is evident that error(A) = Przuple (ar & xp)/2 1/2 E||Xp|]/ /2m2, where the expectation is taken over the randomness of Lo, and denotes the cardinality after removing duplicate elements.
Similarly, we have error( (AurPoAA = 1/2 EI FAugProMMxll// /2m2 suffices to prove
E KAug Xpll =1 1.
lim 12-000
n/2
First, we show for some constant ei > 0,
Pr [ 2 n - CI / logn 2 1 1/poly(n).
(8)
It is equivalent to drawing the elements in Ap one by one. Let XS denote the elements after drawing Ty. Let Y; |. and D; be the indicator that # æz for all 0 < j < i, where D1 1. Then we know (/6 = Yn. D; Y; Y,-1, and for any X--
Pr(D; = = 2 =
1
2
|
n
Let Fj be the filtration, and hence E[D; F, 1] 1 1.2
Let D. D--E.D: Fi-i] be
a martingale difference sequence. Hence [.. < almost surely, and E[D [F-1] S 1. Let
W, = E Fy-1). By Freedman's Inequality, one has
Pr(Y, < c1 n log
Pr
Dj < n C1
log n
S Pr[] Σ
Di S c v n log
==1
Pr[l Σ Dil 1 c1 v n log A W., <n)
Gng n
exxp 2n + 2c1 v log n/3
Choosing C₁ large enough proves Equation (8). Similarly we show for some constant > 0,
Pr |AugPro-Mix 2 3n/2 - 302 / n logn| 1/poly(n).
It is equivalent constructing XAugPto-Mix by iterations. For i-th iteration, draw _2:-1 and 22 i..dd uniformly and construct 2; as described before. Let X A'AuProoMix denote the data we get after constructing 12 22; and z, Let Y | XoxggrroMMix and D' be the indicator that Y Y
(i.e. all of the three data are distinct and first constructed). We know Y Yn 2 3 3n/2 n/2 1 D/. For any
18

---

Published as a conference paper at ICLR 2023
fixed vector { we know Pr(xz2-1 z] Pr(mzs x]
Pr(z, x| 1/n².
Hence for any X Xi-1 AugPro-Mix and by union bound, one has
B B =
X AugPro-Mixl 2
3 2
AngPro-Min Mix
+3)
2
9
X
2n
Let Fj be the filtration and hence E[D; Fi-1l 2 1 Let D2 = -
E[D Fi-1l be
2n
martingale difference sequence. Similarly |P|] S 1 almost surely and [E(D; Fi-il S 1. Let
W = -j F--1l. By Freedman's Inequality, one has
Pr[Y n//2 3n/22 3c2 V n logog
n/2
Pr
D/ n/2 02 V n log
n/2
S Pr[]
Σ
111 S 02 V n nlog O())
i=1
n/2
= Pr(l
D S 02 n log - 0(1) A n/2 S n
1=1
- -2 log
S 2 exxp(
n. 22gynlognn/
where cg c2 - O(). Choosing 02 large enough completes the proof of Equation (9).
Combining equations (8) and (9) proves the statement.
H.2 PROOF OF CLAIM
Proof. Similarly, we have error (roor(AAuPHo-PO(sM)) 1/2 - AA and it suffices to prove
lim PIAAlll - Xoll 1.
We already have Equation (8). It suffices to prove that for some constant C3 >> O,
Pr FaSS 2 2n C3 / n logn]
1/poly(n).
(10)
For each I; E FAugPIo-FGSM and each coordinate j E log nl, we know Pr(a,()) = w:(i)l = (1/2) 0.7, where is cumulative distribution function (CDF) of one-dimensional standard Gaussian distribution, i.e. (tt) Præ~/i(o1) $ t).
is equivalent constructing XAugPro-FGSM iteration by iteration, where in i-th iteration, we draw 2r and construct w; described before. Let AugTroo FGSM be the data get after constructing as and wi, let Y, = & Aug and let D, be the indicator that Y; Y = 2. For any fixed vector E (-1, 1, 1 2 n, we know Pr(m: x] 1/n2 = Pr(w; x and Pr(w;; x:} < 0.72 log < x
1/2 Let F; be the filtration, and for any AmgPro-FGSM by union bound, one has
2
E
X x=1
AugPro 2 1 AngPro.FGSM 1.0 > -3/n.
X
19

---

Published as a conference paper at ICLR 2023
Let D. D, - -E,,, Fi-1] be martingale difference sequence. We know |Dy] < and
E|D,1 Ft-1] almost surely. Let W -ド D Fイ-1]. By Freedman's Inequality, one has
Pr|Y, < 2n 203 V llg
Pr
D nl
Pr[l
Dil < c/Vn/nnn n noog W nn
1=1
C3 n log n
2exp(
2n 2dz logn//
where cg c3 C3 O(1). Choosing c3 large enough completes the proof.
Combing Equation (8) and (10) completes the proof.
20


Reviews:
[{"review_id": "ajh5qpeyWi", "reviewer": "Reviewer_55Yt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel and efficient data augmentation method for knowledge distillation, by building on top of representation interpolation augmentation methods. Existing data augmentation methods can be divided as three types and have their own limitations as follows: 1) representation interpolation - the newly generated samples might be out-of-distribution 2) token replacement - lack of ability to generate diversified data 3) augmentation with models - large computational overhead. To address such problems, in this work, the proposed method aims to find inverse mapping to map representations back to tokens on the embedding level. Especially, they use nearest-neighbors as the approximated inverse function. With little computational overhead, the proposed method improves the distillation performance on multiple tasks in GLUE benchmark. ", "summary_of_the_review": "This paper proposes a simple yet effective data augmentation method for KD on NLP domain overcoming the limitations of existing methods. Also, they empirically validated the performance of the method on GLUE benchmark, which is one of main benchmark in NLP domain. However, there are some unclear parts in the motivation and why this method works well. Thus, during the rebuttal period, I hope the authors address the questions that I raised.", "strength_and_weaknesses": "- Strength\n  - The method is simple yet effective and address limitations of existing three types of data augmentation methods.\n  - On the GLUE benchmark consisting of 8 downstream tasks, this method shows consistent results outperforming the baselines.\n\n- Weaknesses\n\n[Major] The major concern is that the motivation is unclear for the reviewer. Could the authors provide more detailed explanations about the below questions?\n\n1) In section 3, I am confused about the relation between decision boundary shifts problem and domain difference property between CV and NLP. I think motivation example in (b) of Figure 2 can occur in CV domain too, but the first paragraph in Section 3, the authors said that the problem is that representation interpolation in CV domain can not directly applied to NLP tasks. I think labels in vision classification tasks are also discrete, mixup  makes interpolation between two discrete labels. Why discrete space in NLP tasks (Positive / Negative) is a problems when we try to existing data augmemtation methods?\n\n2) Following the (b) and (c) in Figure 2, for the projection, it looks that we should use the unseen data (x2). Could you explain more in terms of the fair comparison and detailed description about actual unseen data used in the proposed method?\n\n3) Why do we need to combine the proposed AugPro to existing data augmentation methods? Could we use the proposed AugPro solely?\n\n4) If I read correctly, this method said that choosing the existing tokens is better than generating new tokens based on interpolation. Could the authors provide the detailed interpretation about this?\n\n[Minor] The writing need to be improved. I think that the contributions can be more well explained in the introduction.\n[Minor] Where is the results validated on WNLI which is one of datasets in GLUE benchmark?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Pwr3IHRLdtY", "reviewer": "Reviewer_FUtZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a novel method for data augmentation to be used for knowledge distillation in NLP. It has been shown that heavy augmentation is highly useful for knowledge distillation (in vision tasks), but the same methods are not applicable to NLP due to the tokens being discrete. Therefore, previous works tend to ignore the first layer and apply the interpolation at the representation layer.\nThe authors use a simple trick in order to get around this issue: they use NN matching over the vocabulary as their approximate inverse projection, which allows them to mix the representations at the embedding layer, but still create valid tokens/input samples.", "summary_of_the_review": "The problem is well-defined, and the solution is simple and to the point. The improvements also seem to be consistent across the board. The paper is also well-written and easy to follow with multiple examples. I am not an expert in this field, so I can't be confident, but I believe this paper meets the threshold for acceptance.", "strength_and_weaknesses": "Strengths:\n- A concrete and well-scoped problem with a reasonable solution\n- consistently beats baselines\n\nWeaknesses:\n- I'm not sure how much the final sentences make sense as we'll be adding token i's from sentence 1 with token i's from sentence 2. That can totally destroy any sort of meaning. Then again, in knowledge distillation, we might not care about this and only care about matching two functions point-by-point. I'm not sure how that's better than creating gibberish sentences as augmentation though.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LTh9_Zwetu", "reviewer": "Reviewer_Lqpu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In the context of data augmentation for knowledge distillation, this paper proposes a method AugPro to address one significant and fundamental problem in representation learning for NLP: how to invert continuous representation to discrete tokens. AugPro finds tokens of which representations are close to the representation of an input augmentation sample, in the semantic space, determining what tokens can approximate the representation of augmentation samples acquired with MixUp. AugPro is validated with GLUE tasks and it shows good results in these tasks. ", "summary_of_the_review": "The idea from this paper is reasonable, but there is a huge gap between the idea and real implementation. The author should show much more detail about how to get augmentation samples from the selected token list.", "strength_and_weaknesses": "Pros:\n1. The augmentation and projection method is rational intuitively and methodologically, knowledge distillation from large PLM is a good application scenario of the proposed method. Though the author did not clearly claim the motivation of projection from the perspective of linguistics, it seems to try to look for some keywords that can summarize the augmentation sample.\n2. By combining augmentation and adversarial training together, the proposed method achieves good experimental results.\n3. The effectiveness analysis is helpful in understanding the proposed method and good ablation study.\n\nCons:\n1. There is one significant point that should be clearly explained. Equation 5 outputs a token list that can project embedding to real tokens. Technically, the token list should not be a sentence. But the examples from Appendix C are “real” sentences, though it is hard to read. This is amazing and beyond my expectation. How to convert a chaotic token list to a sentence? And there are a lot of stop words like “the” in the augmentation samples. If the project objective function aims to find keywords, I think the possibility of having stop words is really low. On the other hand, equation 5 takes the token that maximizes the similarity function from the vocabulary. But it seems there is a token list, which means all tokens in the selected token list have the distinctive similarity value. It would be helpful to explain equation 5 and its associated implementation in more detail. If more details are available, I would definitely raise my score.\n2. The section 3 motivation samples are good, but I would suggest removing it since the difficulty in inverting an embedding to real tokens is familiar to most people, a short paragraph can be enough. More spaces should be provided for explaining the details of equation 5.\n3. Better results are available if combing AugPro-FGSM and AugPro-Mix, but how to combine them together? This is because AugPro-MixUp and AugPro-FGSM may generate different token lists. Given the same objective function, the gradient derived from augmentation samples and adversarial samples are orthogonal or inverse if linearly combined them together like Loss_mix + Loss_fgsm? Provided with different input samples, it is not easy to determine the best optimization direction. Or it may be iterative training in real implementation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CkvDClaju9F", "reviewer": "Reviewer_ZbR2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses data augmentation for distillation one NLP task. Existing interpolation representation method has the issues that shifts the detection boundary in NLP because NLP space exists with discrete inputs. The proposed method addresses the limitations of interpolation representation to maintain the diversity of expression and avoid the shifting decision boundary. The main idea is trying to augment the real data in input space. In particular, it augments data to get augmented representation (feature embedding at middle layers) and project the augmented representation into tokens by the inverse mapping into nearest real data in the input space. Using these nearest real data to compute the augmentation loss. The projection (called AutoProj) is the main contribution in this work. Two main augmentations (MixUp and FGSM) are considered in the work and AutoProj is built on top of it. This method is simple with little computation overhead but improve the distillation performance with a large margin. Experiments on language models show the encouraging results compared to other data augmentation methods. ", "summary_of_the_review": "Overall, the paper is well-written and the idea is interesting. All claims are well-supported with good results and ablation study. ", "strength_and_weaknesses": "Strengths: \n\n1. The approach is sensible and novel to me in term of projecting to mapping the embedding to tokens.  \n\n2. The results are encouraging and outperform the baseline augmentations on GLUE tasks. \n\n3. The paper is very well-written and easy to follows.  \n\nWeakness:  \n\n1. It is unclear to me why the overhead is small as I have to search nearest token for each iteration. Perhaps because it leverages the parallel computation in TPU? It would be good if the authors can detail a bit more how the inverse mapping is implemented?  \n\n2. Can the authors explain why AutoProj is applied suitable for distillation? As it seems to me that the method is quite general and can applied to training the teacher network alone?    ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster