Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: ##########################################################################

Summary:
 
The paper proposes a novel data augmentation method. In particular, it proposes a reweighted loss function that allows to find the optimal weighting of the augmented samples. The approach is tested on standard image and 
language tasks and compared to mulitple alternative approaches.

##########################################################################

Reasons for score: 
 

Overall, I vote for accepting. The approach is interesting, well motivated and clearly formalized. However the experiments raise some questions which I would like the authors to comment on in the rebuttal. 
 

##########################################################################

Pros: 
 
1. The paper has an interesting take on the important topic of data augmentation  

2. It proposes a clever and theoretically well motivated way to incorporate augmented samples. 

3. The base line models are chosen reasonable. 

4. The paper is clearly structured and well written.

##########################################################################

Cons: 
 
1. One major concern is the performance comparison of varying numbers of augmented samples. Table 2 shows that often a smaller amount of augmented samples perform better. However, if the reweighting is indeed optimal, adding augmented samples should not hurt the performance.  


2. It is not clear when and why to use the hard or soft loss. In addition with the varying performance dependent on the number of augmentation samples this adds additional hyperparameters. The increase in performance might not be worth the increase in training time (see point 3).


3. Are the results significant? 
 

##########################################################################

Questions during rebuttal period: 
 

Please address and clarify the cons above 
 

#########################################################################

review_title: Reweighting review

final_score: rating: 7: Good paper, accept
review_content: Summary:
This paper proposes a simple scheme for training with multiple augmentations of training data in one iteration and reweighting the instances by their relative loss. As authors note in their related works, the idea of reweighting examples based on their relative loss has been widely studied in a variety of machine learning problems. In contrast, this work proposes the reweighting only within augmentations of a single sample. They derive their particular reweighting scheme by proposing an alternative risk (Eq. 3). The new objective is a function of both model parameters and the distribution of augmentations. They propose to find the model parameters that minimize the alternative risk for the hardest distribution of augmentations that maximizes their alternative risk (Eq. 4). Then they consider the distribution of augmentations that are a function of model parameters and input and show that for fixed model parameters, the optimal distribution on a fixed finite set of augmentations is determined by the softmax on the loss of the model for each augmented input. In section 3.2, they propose two variations of their loss using the ground-truth label to evaluate the loss (hard loss) versus using the prediction of the model for the original raw input (soft loss). In section 3.3, they propose specific considerations for augmenting text data. They provide experiments on image and text data with ablations studies.

Pros:
- Proposed methods are particularly good on large models (resnet44, resnet56)
- Figure 2 is particularly interesting because the curves for proposed methods seem less noisy in multiple datasets (CoLA, SST-2 mrpc (only soft), qnli).
- The method might show more advantage if tested in low data settings or small mini-batches. Have authors tried any setting with small training data? Or smaller mini-batches?

Cons:
- The method does not always beat DA+UNI that uses multiple augmentations but weights them uniformly. Although one could argue DA+UNI is also an interesting contribution as it can be more efficient than DA+Long for small mini-batches.
- Tables: no standard deviation is given? In particular, in Table 1, 0.5 can easily be within the standard deviation.
- Table 1: DA+UNI should have the same time as MMEL but for resnet56 it takes 7h while MMEL takes 4.5h to train
- It is not clear how the hyperparameter lambda is tuned? Was it done using cross-validation? In the appendix the lambda is given as 1. Is that the best value?
- Table 3: Results on CoLA are good but others not significant? Again, not clear based on missing standard deviation.

Additional Notes:
- Can you clarify the following sentence: “Remark 3: If we ignore the KL divergence term in equation (3), due to the equivalence of minimizing cross-entropy loss and MLE loss...”
- Is Eq 7. just Eq 5 but for a single example?
- Figure 1 is not really descriptive enough. Maybe make it one figure and use color and line style to show the difference.
- DA+UNI: does the implementation use bigger batch size to perform computations in parallel?

Typos:
Section 3.1: an uniform -> a uniform
Remark 2: regularizedr -> regularizer

==============
After rebutall:
I thank the authors and appreciate addressing all my concerns. I'm glad that the additional experiments with smaller mini-batches and smaller training set size provide more supporting evidence for the method. I encourage the authors to point to these results in the main body.

One more minor suggestion is regarding the sentence in Remark 3. The way I read the sentence the "equivalence of minimizing CE loss and MLE loss" is the reason we can remove the KL divergence term. But the authors' response seems to say it is for the 3rd part of the sentence. The authors' might want to rewrite that sentence to make it clear.
review_title: Proposes an efficient example reweighting scheme that in some experiments can improve the performance but not always.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##########################################################################

Summary:

 
The paper investigate an interesting problem, improving generalization performance by leveraging augmented data.

##########################################################################

Reasons for score: 

 
Overall, my vote lies on borderline and my main concern is the novelty of the work.

 
##########################################################################
Pros: 
1. The paper studies an important problem and propose a method that is a mixture of weighted augmentation and adversarial loss.
2. The paper is written well and easy to follow.
- Figure 1 is a nice representation to help the reader to understand the loss function better.
3. The paper investigates the performance of the proposed method on two domains: computer vision and natural language processing. The authors also provide ablation studies to evaluate the role of parameters.
 
##########################################################################

Cons: 

 
1- I have two main concerns:
a) The novelty of the method is questionable for me as I have already seen a very close idea in the following papers:
[1] https://ieeexplore.ieee.org/abstract/document/8658998
[2] https://arxiv.org/pdf/1712.04621.pdf
[3]https://arxiv.org/abs/1907.12934
In [1], the min-max (adversarial) framework used and the distribution over the data provides weighting augmentation.
In [2], the effectiveness of data augmentation in object recognition has been discussed.
In [3], the min max entropy has been leveraged for weakly supervised (pixel-level) localization.
....
It is unfortunate that this papers were not even cited.
b) Generalization aspect.
The authors of the paper presents their main objective to provide generalization while they have not provided comprehensive experiments or discussion to address this concern.

2- Although the proposed method provides experiments on computer vision and natural language processing, I still suggest the authors to conduct more experiments considering more datasets from computer vision domain.

3-It would be great if the author can what would be the advantages/ weaknesses of their approach with the references. As the proposed approach has high similarity with the previous works, the minimum requirement is reporting more experiments and compare their method with exist methods. This extra study would present how their approach affect the performance.
 
##########################################################################

Questions during rebuttal period: 

 
Please address and clarify the cons above 

 
#########################################################################
After rebuttal:

Dear Authors, Thanks for providing more details. I believe more discussion and experiments are required to present the difference of you method. As you have mentioned, one difference is in considering summation rather than maximization, so it would be required to know what would be the advantages/ weaknesses of this difference. How does it make any impact on the performance? I would increase my score considering the closed-form solution as a nice contribution and requiring more experiments and analysis on the discussed references.

Thanks!


 
review_title: REWEIGHTING AUGMENTED SAMPLES BY MINIMIZING THE MAXIMAL EXPECTED LOSS

Meta-review: Dear authors,

The reviewers appreciated the insights provided by your paper and the strong results. Congratulations.
I encourage you to address the other points raised to make your final submission as complete as possible.
Decision: Accept (Poster)