Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In the context of data augmentation for knowledge distillation, this paper proposes a method AugPro to address one significant and fundamental problem in representation learning for NLP: how to invert continuous representation to discrete tokens. AugPro finds tokens of which representations are close to the representation of an input augmentation sample, in the semantic space, determining what tokens can approximate the representation of augmentation samples acquired with MixUp. AugPro is validated with GLUE tasks and it shows good results in these tasks. 
strength_and_weaknesses: Pros:
1. The augmentation and projection method is rational intuitively and methodologically, knowledge distillation from large PLM is a good application scenario of the proposed method. Though the author did not clearly claim the motivation of projection from the perspective of linguistics, it seems to try to look for some keywords that can summarize the augmentation sample.
2. By combining augmentation and adversarial training together, the proposed method achieves good experimental results.
3. The effectiveness analysis is helpful in understanding the proposed method and good ablation study.

Cons:
1. There is one significant point that should be clearly explained. Equation 5 outputs a token list that can project embedding to real tokens. Technically, the token list should not be a sentence. But the examples from Appendix C are “real” sentences, though it is hard to read. This is amazing and beyond my expectation. How to convert a chaotic token list to a sentence? And there are a lot of stop words like “the” in the augmentation samples. If the project objective function aims to find keywords, I think the possibility of having stop words is really low. On the other hand, equation 5 takes the token that maximizes the similarity function from the vocabulary. But it seems there is a token list, which means all tokens in the selected token list have the distinctive similarity value. It would be helpful to explain equation 5 and its associated implementation in more detail. If more details are available, I would definitely raise my score.
2. The section 3 motivation samples are good, but I would suggest removing it since the difficulty in inverting an embedding to real tokens is familiar to most people, a short paragraph can be enough. More spaces should be provided for explaining the details of equation 5.
3. Better results are available if combing AugPro-FGSM and AugPro-Mix, but how to combine them together? This is because AugPro-MixUp and AugPro-FGSM may generate different token lists. Given the same objective function, the gradient derived from augmentation samples and adversarial samples are orthogonal or inverse if linearly combined them together like Loss_mix + Loss_fgsm? Provided with different input samples, it is not easy to determine the best optimization direction. Or it may be iterative training in real implementation.
clarity,_quality,_novelty_and_reproducibility: The novelty is obvious, but the clarity and reproducibility are not good.
summary_of_the_review: The idea from this paper is reasonable, but there is a huge gap between the idea and real implementation. The author should show much more detail about how to get augmentation samples from the selected token list.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a novel method for data augmentation to be used for knowledge distillation in NLP. It has been shown that heavy augmentation is highly useful for knowledge distillation (in vision tasks), but the same methods are not applicable to NLP due to the tokens being discrete. Therefore, previous works tend to ignore the first layer and apply the interpolation at the representation layer.
The authors use a simple trick in order to get around this issue: they use NN matching over the vocabulary as their approximate inverse projection, which allows them to mix the representations at the embedding layer, but still create valid tokens/input samples.
strength_and_weaknesses: Strengths:
- A concrete and well-scoped problem with a reasonable solution
- consistently beats baselines

Weaknesses:
- I'm not sure how much the final sentences make sense as we'll be adding token i's from sentence 1 with token i's from sentence 2. That can totally destroy any sort of meaning. Then again, in knowledge distillation, we might not care about this and only care about matching two functions point-by-point. I'm not sure how that's better than creating gibberish sentences as augmentation though.

clarity,_quality,_novelty_and_reproducibility: Clarity: the method is pretty clear.

Quality: improvements seem to be consistent across tasks.

Novelty: AFAIK the method is novel, but not extraordinarily so.

Reproducibility: I don't believe the code is available. The authors added implementation details in Appendix B, but that's probably not sufficient to reproduce the experiments.
summary_of_the_review: The problem is well-defined, and the solution is simple and to the point. The improvements also seem to be consistent across the board. The paper is also well-written and easy to follow with multiple examples. I am not an expert in this field, so I can't be confident, but I believe this paper meets the threshold for acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper addresses data augmentation for distillation one NLP task. Existing interpolation representation method has the issues that shifts the detection boundary in NLP because NLP space exists with discrete inputs. The proposed method addresses the limitations of interpolation representation to maintain the diversity of expression and avoid the shifting decision boundary. The main idea is trying to augment the real data in input space. In particular, it augments data to get augmented representation (feature embedding at middle layers) and project the augmented representation into tokens by the inverse mapping into nearest real data in the input space. Using these nearest real data to compute the augmentation loss. The projection (called AutoProj) is the main contribution in this work. Two main augmentations (MixUp and FGSM) are considered in the work and AutoProj is built on top of it. This method is simple with little computation overhead but improve the distillation performance with a large margin. Experiments on language models show the encouraging results compared to other data augmentation methods. 
strength_and_weaknesses: Strengths: 

1. The approach is sensible and novel to me in term of projecting to mapping the embedding to tokens.  

2. The results are encouraging and outperform the baseline augmentations on GLUE tasks. 

3. The paper is very well-written and easy to follows.  

Weakness:  

1. It is unclear to me why the overhead is small as I have to search nearest token for each iteration. Perhaps because it leverages the parallel computation in TPU? It would be good if the authors can detail a bit more how the inverse mapping is implemented?  

2. Can the authors explain why AutoProj is applied suitable for distillation? As it seems to me that the method is quite general and can applied to training the teacher network alone?    
clarity,_quality,_novelty_and_reproducibility: The idea is novel and the results are encouraging. It is supported with also theoretical analysis. It is already in the good form to publish. 
summary_of_the_review: Overall, the paper is well-written and the idea is interesting. All claims are well-supported with good results and ablation study.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel and efficient data augmentation method for knowledge distillation, by building on top of representation interpolation augmentation methods. Existing data augmentation methods can be divided as three types and have their own limitations as follows: 1) representation interpolation - the newly generated samples might be out-of-distribution 2) token replacement - lack of ability to generate diversified data 3) augmentation with models - large computational overhead. To address such problems, in this work, the proposed method aims to find inverse mapping to map representations back to tokens on the embedding level. Especially, they use nearest-neighbors as the approximated inverse function. With little computational overhead, the proposed method improves the distillation performance on multiple tasks in GLUE benchmark. 
strength_and_weaknesses: - Strength
  - The method is simple yet effective and address limitations of existing three types of data augmentation methods.
  - On the GLUE benchmark consisting of 8 downstream tasks, this method shows consistent results outperforming the baselines.

- Weaknesses

[Major] The major concern is that the motivation is unclear for the reviewer. Could the authors provide more detailed explanations about the below questions?

1) In section 3, I am confused about the relation between decision boundary shifts problem and domain difference property between CV and NLP. I think motivation example in (b) of Figure 2 can occur in CV domain too, but the first paragraph in Section 3, the authors said that the problem is that representation interpolation in CV domain can not directly applied to NLP tasks. I think labels in vision classification tasks are also discrete, mixup  makes interpolation between two discrete labels. Why discrete space in NLP tasks (Positive / Negative) is a problems when we try to existing data augmemtation methods?

2) Following the (b) and (c) in Figure 2, for the projection, it looks that we should use the unseen data (x2). Could you explain more in terms of the fair comparison and detailed description about actual unseen data used in the proposed method?

3) Why do we need to combine the proposed AugPro to existing data augmentation methods? Could we use the proposed AugPro solely?

4) If I read correctly, this method said that choosing the existing tokens is better than generating new tokens based on interpolation. Could the authors provide the detailed interpretation about this?

[Minor] The writing need to be improved. I think that the contributions can be more well explained in the introduction.
[Minor] Where is the results validated on WNLI which is one of datasets in GLUE benchmark?

clarity,_quality,_novelty_and_reproducibility: I think the overall writing and motivation clarity, quality need to be improved, yet, they are not critical and could be improved during the rebuttal. Also, the method is quite novel but I am not sure the proposed method contains important technical contributions.
summary_of_the_review: This paper proposes a simple yet effective data augmentation method for KD on NLP domain overcoming the limitations of existing methods. Also, they empirically validated the performance of the method on GLUE benchmark, which is one of main benchmark in NLP domain. However, there are some unclear parts in the motivation and why this method works well. Thus, during the rebuttal period, I hope the authors address the questions that I raised.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a simple and effective data augmentation technique for the knowledge distillation of NLP models. The proposed method secures the diversity of augmentation based on interpolation and converts the interpolated data back into tokens at the embedding level to minimize the shift of the decision boundary by the interpolation-based methods.

All four reviewers agreed that the idea is novel, and the experimental results are convincing and consistent. In the discussion, there was no disagreement in accepting the paper. However, I think it needs improvement in terms of clarity. In rebuttal, the authors explained kindly and the reviewers cleared up the misunderstanding, but it seems that more kind writing is needed for the reader. In particular, the examples and figures in Section 3 are somewhat confusing, regardless of the message the authors are trying to convey (for examples, when I first saw this example, it wasn't clear what the unseen data is in your setup. or how the label changes to blue after projection, etc.) The concept figure for the proposed method seems to be helpful in directly understanding the proposed method, so it must be included in the final version.
justification_for_why_not_higher_score: The method is simple and not that novel, so I don't think it's enough as a spotlight.
justification_for_why_not_lower_score: good paper
Decision: Accept: poster