Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This work proposes to change the labels for the mixed examples in mixup. My major concerns are as follows.
1.	The motivation of adopting soft label for mixup is not clear. Label smoothing is helpful for generic training but why it can benefit mixup?
2.	The proposed method is more like a combination of mixup and label smoothing. The improvement may come from label smoothing as a generic trick rather than mixup itself.
3.	The performance of proposed method is very close to mixup, where the improvement is not significant. Additional experiments on ImageNet can make the results more convincing.
review_title: Label smoothing in mixup

final_score: rating: 4: Ok but not good enough - rejection
review_content: **Main Claim:**

The authors propose to use soft labels on naive mixup as an alternative to sophisticated mixup strategy. In experiments on 5 small datasets, the proposed method can achieve better accuracy than baselines.

**Strong points:**

The authors propose to use soft labels to overcome the mislearning features in mixup images. The idea is simple and straightforward. 

Experiment results show the method works well on small datasets.


**Weak points:**

The contribution of this work is incremental.

Experiment results on ImageNet are missing. On Tiny-ImageNet, it’s also helpful to show the performance of Mixup and CutMix.

Some decisions are made without justification. Some details are not clearly explained. See questions.



**Recommendation:**

Reject.

The proposed method is incremental. The method should be evaluated on ImageNet. 

**Questions:**

In Eq(9), a sigmoid function is applied before the softmax function. So the unnormalized logits of this distribution is in [-1, 1]. Why?

Why is “target soft labels too far away” an issue for the model? why does “a mini-batch with original inputs“ prevent “LaMix from assigning target soft labels too far away”? If the trick is not applied, what will happen to the model? Will the model take more time to converge or it won't converge?

“For LaMix, the added fully connected layer is just a copy of the fully-connected layer of the original network with a Softmax function on the top.” Is the network pre-trained? Are the two matrices sharing weights?

As shown on Figure 2, the top 2 classes occupy a large portion of the probability. So I’m curious which part actually contributes to the improvements. Is it (A) the reweighting of y_i and y_j, or (2) the introduction of other labels? I.e. After computing Eq (10), keep the value for y_i and y_j, set all other dimensions to zero, (then renormalize the distribution), and use it as the training label, what will happen?  My impression is it may solve the problem of  “target soft labels too far away”.

**Comments:**

Eq (3) is confusing to me. I think authors can follow the convention in (Yun et al.), rewrite the equation as $x_\lambda^{i,j}=\Phi(x_i, x_j, lambda) * x_i + (1 - \Phi(x_i, x_j, lambda) * x_j$

has tow forms -> has two forms

**After rebuttal:**

Thanks to the author for providing additional experimental data. But without the results of imagenet, it is difficult to judge the effectiveness of this method on complicated data. So I decided to keep the original score.

review_title: An incremental work on mixup data augmentation 

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
The previous advanced Mixup methods, such as CutMix and PuzzleMix, involve input mixing. This paper suggests a new Mixup approach, called LaMix, that does not require input mixing. The solution is combining the original target label (interpolation of two one-hot targets) and generated target labels from an additional network to use it for training. The authors argue that LaMix achieves superior performance without input mixing.

Reasons for score:
The authors should explain the reason why using global soft labels is theoretically plausible before describing the method. Although empirical results look nice, I am suspicious about the experimental settings for the comparison with other methods.

Pros:
- The paper includes diverse results on probing experiments.
- The paper is clearly written.

Concerns/Questions:
- As far as I understood, neural network parameters for the global soft label (W_t) are not trained because they are only used for training labels. Then, these parameters are just randomly initialized values. Is it right? If so, isn’t the final effect sensitive to the initialization?
- Sigmoid activation is used in equation (9), different from equation (7). However, there is no explanation about the reason for using it. My guess is to make artificial labels similar to each other.
- I don’t understand why beta in LaMix is 1.0 for Section 3.2 experiments. First, I think beta = 1.0 means not using the global soft label, and it is equivalent to standard Mixup. Second, the authors mention that setting beta to 0.5 is a good choice in Section 3.1.3 (Figure 4). Moreover, they use beta as 0.5 for the experiments of Section 3.1.1 (Table 1). The settings of Table 1 and Table 3 (model architectures and datasets) are the same except for the beta value of LaMix.
- Could you provide hyperparameters used for other Mixup methods as a baseline? Are they well-tuned?
- I am curious whether the combination with the global soft label can be done after input mixing. If then, I think providing these results would be helpful to check whether the regularization effect is orthogonal to input mixing.
- To compare LaMix with label smoothing, I think the author should apply label smoothing to Mixup rather than the vanilla setting.

Minor comment:
- “Puzzle Mix” -> “PuzzleMix”
review_title: Seemingly plausible but weak support

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary

This paper simply combines mixup and self-distillation to achieve more adaptive soft label, which effectively regularize the training. In the manuscript, authors argue that the existed mixup-based approaches has two mainly efforts, may create misleading training samples or meet computation cost issue on creating samples. Motivated by this, they propose "LaMix", which can leverage the information of self-distillation, to solve those two efforts and achieve competitive performance with SOTA "Puzzle-mixup".   


Comment

1. This paper introduces a combination method between mixup and self-distillation, and simply use an additional FC layer to have adaptive soft label, which is intuitive and clear but lack of novelty. Can you give more insightful comment about how adaptive label can help mixup soft label?

2. In Figure 3, I think the provided evidence for the effect of considering adaptive soft label with mixup approach is promising.

3. For experimental results, in section 3.2, the proposed "Lamix" seems not achieve significant improvement compared to the SOTA "Puzzle-mix" on CIFAR-10 and 100.

review_title: Simple combination of mixup and self-distillation, but marginal experimental results

Meta-review: This work proposes to improve Mixup by using soft labels, removing the need for input mixup. The reviewers found the paper was clear and found the experiments promising. The reviewers raised concerns about the lack of experiments comparing this approach to Mixup+Label smoothing, which were addressed during the rebuttal by the authors. However, the reviewers did not find the empirical evidence strong enough given that this is mostly an empirical contribution. The authors do not necessarily need to train on the full Imagenet, but it would be beneficial to evaluate on more standard settings on the dataset considered to facilitate comparison to previous work.
Decision: Reject