Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a unified way to augment data in the latent (embedding) space. In particular, the paper combines three existing techniques including adversarial training, triplet loss, and joint training for data augmentation. Paper explores 4 standard modality agnostic data transformations to augment data. Since augmentation is done in the latent space, the proposed technique can be applied to any modality including text, images, time-series data. Empirical results show that the proposed method works better than the standard transformations on all modalities except image datasets. 

Strengths:
- The proposed framework works for different kinds of modalities. Paper provides extensive results on various standard benchmark datasets including SST-2, CIFAR-10. 
- Paper shows that the examples for which the model is least confident are a good target for the augmentation. This finding is useful for future data augmentation work. 
- Paper also provides ablation studies to show the effectiveness of combining adversarial and triplet loss.

Weakness:

Reservations related to the usability of the proposed method: With the results presented in the paper, I am not convinced that the proposed technique will be preferred over other domain-specific data augmentation methods which leads me to question the usability of the proposed model. For example, one can simply use any pre-trained model as in [Kumar et al. 2020] or simply back-translation for text classification instead of the proposed data augmentation and can get much better results. To show the usefulness of this work, it's crucial to show when the proposed method is competitive to the existing strong DA baselines. If authors show that on atleast one datatset, I will increase my score.  

- Paper is using weak baseline methods for input space augmentations. For example, for text augmentation, one can either use back-translation which doesn't require any model training and performs better than the EDA baseline used in the paper. Also, MODALS is using a complex architecture that relies on end2end training so I don't think that using a baseline that doesn't require any kind of training is a fair comparison. 

- The claim that MODALS's joint training for augmentation is different from previous approaches is not entirely true as the joint training for augmentation has been explored in past. Please refer to [Mounsaveng et al. 2019, Hu et al. 2019] and their citations. 

-  Latent space augmentation has been studied extensively in both images and text classification literature. Paper does not provide any comparison with the available latent space methods which seems to work well for a given task. For example, [Kumar et al. 2019] shows a comparison between different latent space augmentation techniques and shows that simply subtracting two sentence embeddings and adding it to the third embedding performs well for the text classification task. 

Questions:
-  For hard examples, authors use the difference between the prob of the most likely and the second most likely labels. How does it compare to the simple model confidence baseline where we select the examples for which model is not very confident? Have you done any experiments related to that? 
- I am unable to find the implementation details of the discriminator model used in the experiments. Could you please add those details?

References:
-  Mounsaveng, S., Vazquez, D., Ayed, I. B., & Pedersoli, M. (2019). Adversarial learning of general transformations for data augmentation. arXiv preprint arXiv:1909.09801.
-  Kumar, V., Glaude, H., Lichy, C.D., & Campbell, W. (2019). A Closer Look At Feature Space Data Augmentation For Few-Shot Intent Classification. DeepLo@EMNLP-IJCNLP.
-  Hu, Z., Tan, B., Salakhutdinov, R. R., Mitchell, T. M., & Xing, E. P. (2019). Learning data manipulation for augmentation and weighting. In Advances in Neural Information Processing Systems (pp. 15764-15775).


Update after reading the response from the authors:

I would like to thanks to authors for answering my questions and revising the draft. I agree that the main strength of this work is to explore data augmentation methods which are modality agnostic. Since most of the data augmentation research is domain specific, I think this paper will increase awareness for modality agnostic data augmentation methods. I think it's a good paper, hence I am revising my score.  

review_title: Interesting work on latent space augmentation; Emphasize the need of modality agnostic data augmentation 

final_score: rating: 6: Marginally above acceptance threshold
review_content: [Summary]
This paper proposes a framework to apply automated augmentation in the latent space which is not restricted to any specific modality. The proposed method, called MODALS, follows the procedure of Population Based Augmentation approach and consists of two main parts: latent space transformation to generate harder examples and auxiliary adversarial loss &   triplet loss to improve augmentation quality. MODALS is evaluated on multiple datasets for text, tabular, time-series and image modalities and achieves higher performance than competitors except on image data.

[Pros]
+ The idea of augmenting data in latent space enables the framework to be generic for any modalities, which has a great potential in applications.
+ This paper is well organized and easy to follow.

[Cons]
- The proposed framework is interesting but somewhat incremental since most of the modules/losses are off-the-shelf.
- The experimental settings seem not very fair and sufficient.
  1. It would be better to equip all the competitors with the triplet loss and conduct the evaluation, to show how the data transformations (Eq.(1-4)) contributes to the learning.
  2. The proposed method is very related to recent hardness aware metric learning method HDML [1] which also propose to manipulate data in latent space to generate hard examples. Could the proposed latent-space automated data augmentation method outperform such metric learning scheme?
- The use of adversarial loss might cause mode collapse, which is  harmful for the data augmentation. How the proposed framework overcome this problem?

[1] Zheng W, Chen Z, Lu J, et al. Hardness-aware deep metric learning. CVPR. 2019: 72-81.
review_title: A good generic data augmentation framework, but might need to be compared with hard example generation methods

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes an automatic data augmentation method that is modality agnostic by modifying the data in a latent space (rather than in input space). They design latent space interventions that yield hard examples (which they claim should improve downstream model learning). They apply population based training on the latent-transformation-policy search (which modifies the latent representation of a classification model directly), on top of training the classification model in question. 

The “Hard example interpolation” idea, i.e.: to interpolate towards the 5% hardest examples, is interesting. As opposed to the search and latent aspects of the method, couldn’t this be the cause of any possible gains by the method? A baseline experiment to check would be to use this interpolation method in input-level interpolation augmentations such as MixUp.

The larger concern, however, is that it’s not clear what parts of the method yield good results. For instance, in order to make sure that the latent transformations are being applied in an approximately-gaussian latent, they apply adversarial loss on the latent, to enforce the gaussian prior. They also apply a triplet loss. These are motivated decisions, but it’s hard to verify which of these experimental decisions lead to improvements. The ablation study presented shows the baseline models with the losses added to them, where a more convincing study would show the final MODALS method, with each modification removed one by one. This would not be a big concern if the method yielded much higher performance than the baselines, but it seems to underperform in the image domain (Table 4).

Overall, the method presented is really interesting and seems to work well across a variety of domains. In particular, augmentation has proven to be a hard research direction in text and tabular data, so the fact that this method outperforms the baselines is exciting.

Update after rebuttal: I appreciate the authors' response and the new ablation study. I maintain my original score (marginally above acceptance threshold). I continue to think it's a strength that the method works in many domains, but this strength is slightly diminished due to the variability of domain performance (e.g.: image domain).
review_title: Good paper, but missing a proper ablation study

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
The paper presents MODALS, a new learning objective with automated data augmentation in the latent space. For data augmentation, the paper presents 4 latent-space augmentation primitives by linear interpolation/extrapolation of feature vectors from the same class. For training objective, in addition to multi-class classification objective (formulated as cross-entropy), it introduces two additional losses, such as triplet loss whose triplets are generated using ground-truth class labels and adversarial loss that regularizes feature distribution to be Gaussian. Comprehensive empirical study across multiple domains are presented. The paper demonstrates that the proposed method is particularly effective when the training data is limited.

Pros:
- Data augmentation became very important in deep learning and becomes a bottleneck when applying deep learning methods to problem domains with less prior knowledge on data. In this sense, the paper is tackling very important problem. 
- The paper demonstrates consistent performance improvement over baselines across multiple problem domains.
- The presented methods sounds reasonable, though some clarifications are needed how they are evaluated (see below).

Cons/comments:
- While domain-specific data augmentation methods, such as AutoAugment/RandAugment, are typically "class" agnostic, meaning that we don't need to know the class labels of individual data, the presented method requires class label of individual data instances for augmentation. This could be a limitation of the proposed framework, as data augmentation methods become essential for semi-supervised and self-supervised learning methods.
- Are PBA or L_{tri}/L_{adv} used for MixUp training? Since MixUp training also involves hyperparameter (mixing coefficient), it seems fair if mixing coefficient is tuned via PBA. Also, are "w/o Aug" in Table 1-3 using L_{clf} only or L_{clf} + L_{tri} + L{adv} as training objective?
- Can MixUp be used as part of domain agnostic augmentation operations?
- One missing baseline is latent MixUp.
- For PBA, how are validation sets formulated? Is there a reason for other hyperparameters, such as margin, \alpha and \beta, validated separately?
- Figure 3 reminds me of center loss (Wen et al., A Discriminative Feature Learning Approach for Deep Face Recognition) and large-margin softmax loss (Liu et al., Large-Margin Softmax Loss for Convolutional Neural Networks), which pointed out the problem of softmax loss and presented fixes. It would be instructive to discuss some potential of these losses combined with latent space augmentation proposed in this paper.

Reason for rating:
- I believe that the paper tackles an important bottleneck of ML algorithms to be useful in real-world. I tend to recommend for acceptance for initial rating.
- However, there are several concerns that need to be clarified (see Cons) in empirical study of the work.

I have read the author response and have decided to keep my initial positive rating of 6.
review_title: Tackles important problem, consistent improvement, some clarifications required.

Meta-review: This paper proposes a unified way of data augmentation using a latent embedding space --- it learns a continuous latent space for transformation, and finds effective directions to traverse in this space for data augmentation. The proposed approach combines existing approaches for data augmentation, e.g., adversarial training, triplet loss, and joint training.  The paper also identifies input examples where the model had low performance and creates harder examples that help the model improve its performance. It is evaluated on multiple corresponding to text, table, time-series and image modalities and outperforms SOTA except on image data.

The paper has responded to the reviewers' feedback to provide more detailed experiments with stronger baselines and also ablation studies to show the effectiveness of different components of the approach. The results can be further improved by thorough empirical comparison to other SOTA methods, and by using other loss functions (e.g.,center loss, large margin loss and other contrastive losses) as alternatives to the triplet loss proposed in the paper.

Some reviewers have pointed out that the paper is somewhat limited in it's novelty, since it combines existing off-the-shelf modules/losses and similar methods have been tried in the past --- the novel contributions of the paper should be clearly highlighted in the revised submission.
Decision: Accept (Poster)