Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper revisits the commonly used mixup technique and proposes to modify the loss function to achieve similar effect as the time-consuming search based methods. The proposed method is simple and proved effective across various tasks, including classification, semi-supervised learning on both small and large scale datasets. 
strength_and_weaknesses: ### Strength
1. The proposed method is well motivated, simple, technically sound,  of generalizability. 
2. Extensively experiments are conducted to verify the proposed method across different tasks, datasets, and other alternative choices. 
3. The paper is well presented, with professional figures, table and language use. 

### Weaknesses
1. Setting the parameter looks very important to the performance. Sensitivity analysis has been conducted in the one dataset, which shows the parameter is important to the performance. It is unclear how the parameters are set in other tasks. Considering that in many cases the. proposed method is slightly better than the naive mixup method, the value of this method would significantly reduce if it is non-trivial to tune the parameter to get results.

2. The weakness of this work has not been discussed. As a plug-and-play tool like this, it would be highly valuable if the authors can reveal the weakness so that people can decide whether to adopt it for their own projects.   
clarity,_quality,_novelty_and_reproducibility: See the comments above. 
summary_of_the_review: Overall this paper is good: it extends an existing widely-used technique with simple modification that produces good results across various tasks. There are some issues in the experiments, which I do not think undermine the value of this work. 


Post rebuttal comments: After reading the comments from other reviewers, I tend to reject this submission. While this submission indeed proposes a novel technique that generalizes well across various tasks, the technical contribution is slightly below the bar this prestigious conference. So, I would like to join other reviewers and vote for a rejection.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In  this paper,  to solve the expensive computing resources problem of current dynamic mixup method, it proposed a method trying to transfer the decoupling mechanism of dynamic methods from the data level to the objective function level and propose the general decoupled mixup loss. The experimental results on supervised and semi-supervised tasks proved the effectiveness of DM.

strength_and_weaknesses: The paper proposed a new loss function, and extends to the semi-supervised case and binary cross entropy form. the experimental results show that it’s better than previous benchmarks. 
Some Questions:
1. Some annotations are misleading.  like in formula (2),  where is l defined. For the asymmetrical strategy DM loss, the log(\phi) is a matrix or a vector?
2. What’s the meaning of “data-efficient”? Can you show the model performance under different sizes of dataset to prove the data-efficiency advantage over the baselines?
3. As the paper argued, one of the advantages is that it’s efficient computing, can you show experiments on how compute-efficient compared with the baselines?

clarity,_quality,_novelty_and_reproducibility: The paper proposed an original loss function to optimize the mixup style objective. Extensive experimental results on the supervised and semi-supervised tasks valid the effectiveness of the proposed loss function.

summary_of_the_review: In summary, the paper proposed a novel loss function and verified the effectiveness on different dataset and different tasks.  There are some minor clarity issues on the formulations. However, the paper doesn’t provide proof on how “data-efficient” of the proposed method is and there are no Quantitative results to show the computation efficiency advantage.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a variant of the Mixup method that considers the label mismatch problem by decoupling (i.e., removing) the competitor's class in the softmax which can potentially cause mismatching between the mixed label and sample. The authors formulate the decoupled mixup cross-entropy loss ($L_{DM(CE)}$) that becomes an element of the proposed decoupled mixup (DM) loss  ($L_{DM}$) with the standard mixed cross-entropy loss ($L_{MCE}$). The authors claim that using the DM loss can address the label mismatch problem and argue a similar effect as dynamic mixup methods. The authors apply the DM loss on Image classification tasks on the datasets of CIFAR-100, Tiny-ImageNet, and ImageNet-1k with diverse network architectures and baseline Mixup methods. The authors apply the proposed loss not only to the static Mixup methods but to the so-called dynamic Mixup methods such as SaliencyMix, PuzzleMix, AutoMix, and SAMix. Moving further from the conventional image classification experiments, the authors further evaluate fine-tuning classification on FGVC datasets and semi-supervised learning methods. All the experimental results consistently back the effectiveness of the proposed loss.
strength_and_weaknesses: Pros)
+ The idea looks simple yet practical for the practitioners using Mixup in training.
+ This paper provides extensive empirical studies to support the effectiveness of the proposed method.
+ All the experimental results consistently improve when adopting the proposed loss upon the baseline Mixup methods.

Cons)
- There is no clear evidence that decoupling the class can reduce the label mismatch. It hardly agrees that only performance improvements cannot support the claim. Instead, the authors should provide any empirical or theoretical backups of whether this is actually addressed through the proposed DM loss.
- Some notations are not clearly presented:
  - $p(a,b)$ seems to be the probability of the mixed samples; $p_a$ and $p_b$ must be scalars. I could understand it only after reading the manuscript carefully.
  - l, i, and j in eq.(2)
  - M in Section 2.2
  - $y^i_a$, $y^j_b$ in proposition 2

Comments)
- The authors claim that the proposed method can adaptively focus on discriminative features as the dynamic Mixup methods, but they seem to be conceptually different. If not, there should exist any backups for this. Presumably, the performance improvements upon the dynamic Mixup methods in Table 1 seem to support my concern.
- How does "the discriminatory of instances" in p.2 be achieved through the proposed method?
- Please explain in more detail why the regressing lambda is the problem.
- Vanishing $\lambda$ in eq.(2) release the gradient magnitude not to become having a dependency on $\lambda$, but still unclear that 1) it affects the prediction; 2) it is actually connected with the label mismatching. Please clarify this.

clarity,_quality,_novelty_and_reproducibility: This paper is somewhat clearly presented, but the entire notations should be refined for a better presentation. This method seems to provide some novelty over the conventional Mixup-based methods. The reproducibility would not be a matter with the simple modification of the loss like this method.

summary_of_the_review: This work provides a novel Mixup-based loss for consistent performance improvements on several datasets with many models. I like the constant improvements over diverse setups and scenarios. However, what I'm most concerned about is that there is actually novelty in the sense of a new loss of performance gains, but there doesn't seem to be enough evidence to support the author's claim. I recommend the authors revise the paper by providing more materials that show the link between the decoupling by the proposed DM loss and the label mismatch problem; consider more on the gap between embellishing the proposed method as a dynamic aspect and the ones categorized into dynamic methods.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new objective function for mixed sample data augmentation (MSDA). Technically, the proposed Decoupled Mixup (DM) loss is equivalent to the summation of the original mixup loss and the additional losses that only consider one mixed target label while the other mixed target labels are ignored. For example, assuming a mixed label [0.7, 0.3, 0, 0] and a prediction probability [0.5, 0.45, 0.05, 0], then the DM loss is equivalent to `MixedCrossEntropy([0.7, 0.3, 0, 0], [0.5, 0.45, 0.05, 0])` (the original mixup loss) + `CrossEntropy([1, 0, 0], [0.91, 0.09, 0])` + `CrossEntropy([1, 0, 0], [0.9, 0.1, 0])` (called as "decoupled" mixup loss). The motivation of the proposed method is that the objective function of MSDA is highly correlated to the mixing ratio $\lambda$: by ignoring the other mixed targets, the DM loss is "decoupled" with the mixing ratio $\lambda$. The extensive study on image classification tasks (supervised and semi-supervised) shows that the proposed method is effective.
strength_and_weaknesses: ## Strength

- Because the proposed method is invariant to the choice of the mixing strategy, the proposed method is easy to be adopted to any MSDA method.
- The strongest strength of this paper is the expensive experimental results and consistent improvements.
    - The extensive study shows that the proposed method is effective to the existing MSDA methods. Especially, this paper studies the effect of the proposed objective function on various benchmarks with state-of-the-art optimization techniques. For example, this paper explores not only ImageNet training with 300 epochs and cosine lr scheduling (a standard one) but also ResNet Strike Back (RSB)-ish advanced optimization settings or DeiT optimization settings.
    - In my opinion, the ImageNet-1k experimental results (Table 2, 3, 4) will be helpful to researchers who work with ImageNet-1k benchmarks; as far as the reviewer knows, there are not many works that compare the existing state-of-the-art MSDA methods (Mixup, CutMix, ManifoldMixup, FMix, ResizeMix, SaliencyMix, PuzzleMix, AutoMix, SAMix) with the most advanced optimization settings, such as RSB or DeiT settings. E.g., considering the heavy computational resources of "dynamic MSDA" methods (SaliencyMix, PuzzleMix, AutoMix, SAMix), practitioners can choose ResizeMix + DM loss.
    - This paper also explores transfer learning and semi-supervised learning scenario in state-of-the-art settings.


## Weakness

### Weak connection between label mismatching problem and the proposed method

I cannot find any connection between the "label mismatching problem" and the "decoupled Mixup loss". Assuming we use a static Mixup method (e.g., Mixup), even if we use the DM loss, we still suffer from the label mismatching problem because the loss function still contains the original loss function. Recall that the label mismatching problem means that there exists a mismatch between a mixed sample and a mixed label; it means that if we want to solve this problem, we have to make a mixed label depending on a mixed sample. However, the mixed samples and the mixed labels by the proposed method are invariant; it will still suffer from the label mismatching problem if the base method suffers from the problem.

Although this paper has a meaningful empirical contribution, in terms of academic publication, I think this paper should need more verifications for the statements and the motivation. I cannot find any connection between the proposed method and the motivation.

Also, as a minor comment, I cannot find any connection between "data-efficient" (the title) and the proposed method. I think there need to be more statements for why this method is data-efficient compared to the original mixup loss.

### Weak motivation why we should "decouple" lambda and the objective function

I presume that the motivation of this paper is removing (or reducing) the dependency of $\lambda$ and the objective function. If the main motivation of this paper is "addressing label mismatch", then there should be more explanations of how removing the dependency of $\lambda$ and the objective function ("decoupling") and label mismatch are related. More specifically, it is not clear to me how "decouple" fixes the label mismatch problem.

Moreover, the proposed method still uses the original loss function. What happens if we only use $L_{DM(CE)}$ alone without $L_{MCE}$? If using $L_{DM(CE)}$ alone is worse than using $L_{MCE}$ alone, then I think it is hard to say that the proposed $L_{DM(CE)}$ itself is effective.

Furthermore, to argue that the "decoupling" (ignoring the other mixing targets) is effective, then I think this paper should include the following baseline:

- `CrossEntropy(target_1, prediction)` + `CrossEntropy(target_2, prediction)` => removing $\lambda$ in the objective function

It is the same as "Complete-label CutMix" in Table 8 of CutMix paper, and it is known to be worse than the $\lambda$ mixing strategy (following Table 8 in the CutMix paper). I think this result can make the submission stronger.

### Minor

- It seems that the title "data-efficient" and the motivation are not well-matched. I suggest using a different term rather than data-efficient in the title and Section 3.
- $l$ of Equation (2) is not defined. Please fix the equation.
- Suggestions for a better formatting
    - Please specify the full terminology of AS (page 5). I presume AS stands for "Asymmetrical strategy".
    - I would like to recommend inserting "improvements" in the tables (e.g., as Table 1 of CutMix paper).
    - Please do not use `\vspace{-XX}` between sections (e.g., page 5 "3.2 Asymmetrical strategy")
    - Please do use `\vspace{XX}` between a caption and a table (I usually recommend adding a space `~` between a caption and a table)
    - Please make more spaces (e.g., `\hfill`) between Table 2 and 3 (as well as Table 4 and 5).
clarity,_quality,_novelty_and_reproducibility: I think the clarity of this paper is just okay, and this paper has room for improvement in terms of clarity. Especially, I think this paper has a very weak connection between motivation and the method (Please see weakness for details). I also left some comments related to the formatting. I feel some statements are not supported well (e.g., "small lambda value suppress the confidence of predictions even if the mixed features are evident" => why and how will this be harmful to the training?). I suggest revising the paper to focus on the effect of decoupling rather than the label mismatch problem.

The quality in terms of the experiments looks excellent. However, I think the technical contribution and novelty of this paper (for solving the label mismatch problem as motivation) are not very significant, considering that the proposed method looks invariant to the label mismatch problem.

I think this paper will be reproducible relatively easily, considering that the base settings of this paper are widely-used ones (e.g., 300 epoch ImageNet-1k, DeiT training).
summary_of_the_review: This paper shows great empirical improvements on various benchmarks. The extensive studies are impressive and show consistent improvements. As my comments in "Strength", this paper will be helpful for many practitioners working on ImageNet-1K.

However, I think this paper has a large room for improvement in terms of academic publication. Especially, I think this paper needs a connection between motivation and the proposed method. I cannot find any reason why the proposed method will work. In terms of the label mismatch problem, I think this method will still suffer from the label mismatch problem because the label mismatch problem can be solved only by making the mixed sample and the mixed label dependent. Furthermore, I think this paper should compare using $L_{DM(CE)}$ alone as well. If using $L_{DM(CE)}$ alone does not work well, then it means that we need a better explanation (and motivation) of the effect of the combination of $L_{MCE}$ and $L_{DM(CE)}$, not only for $L_{DM(CE)}$.

Overall, I think this paper does not reach the acceptance threshold of the ICLR conference.

----

During the reviewer-author discussion period, the authors provide additional experiments to resolve my concerns. However, as my last comment, I am still skeptical about revising my score because my concerns still remain. I will maintain my initial recommendation.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new mixup-based loss for data augmentation. Reviewers appreciated the generality of the proposed method in that it can be used with existing mixup-style methods, and the extensive experiments showing its effectiveness. The major concerns were about the weak logical link between the stated motivation of resolving "label mismatch" and the proposed method. There was an extensive discussion on this point with the authors, but reviewers remained unconvinced. Reviewers lean towards rejection overall. The AC agrees with the concerns regarding the motivation and justification of the method in spite of its strong empirical performance. The authors are encouraged to consider the reviewers' comments and strengthen the paper accordingly for a future submission.
justification_for_why_not_higher_score: - Method is not well-motivated and justified in spite of its strong empirical performance
Decision: Reject