Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a suite of Mix-based augmentation techniques that improve the performance of neural networks.
strength_and_weaknesses: Strength:
1. The notations in the paper are rigorous.
2. The writing of the paper is easy to follow.
3. The proposed techniques are empirically solid.

Weakness:
1. None of the proposed techniques is theoretically justified. In other words, it is hard to judge in advance whether an approach helps before actually trying it out. Therefore, describing some scenarios where these techniques FAIL may be helpful.
2. It looks to me that the paper is an ensemble of tricks for training neural networks --- It does not solve any problem.
3. The scale of the experiments is limited --- the largest dataset is Tiny ImageNet. While this scale may be sufficient for a theoretical paper, it is less convincing for an empirical paper.
clarity,_quality,_novelty_and_reproducibility: The paper is rigorously written and easily accessible.
summary_of_the_review: Overall, I think the proposed augmentation techniques are empirically sound, which expands the toolbox of augmentations for training neural networks. However, none of these techniques has been theoretically justified; therefore, it is hard to judge whether these techniques are applicable in a given (new) scenario.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes an extension of the mixup data augmentation method where all examples in a batch are mixed with coefficients sampled from Dirichlet distribution, and the number of mixup examples can be as many as one wishes with more Dirichlet samples. On top of this, the method further adds ensemble distillation and dense augmentation and losses, which is shown to further improve the accuracy. 

The paper argues that mixup on the top layer before logit is the best and also most efficient.

strength_and_weaknesses: strengths
* proposed an extension of mixup with better accuracy and higher efficient 

Weakness
* the contribution seems a bit incremental 
* it’s unclear how the dense mixup and self-distillation are integral part of the proposed method, can we apply the same techniques to the baseline methods?

clarity,_quality,_novelty_and_reproducibility: * Clarity: the paper is overall well written 
* quality: fair, contribution is somewhat incremental
* novelty: fair, it feels like combining multiple existing techniques together, in an orthogonal way
* reproducibility: seems good 

summary_of_the_review: This paper proposed an extension of the mixup data augmentation method combined with some other techniques such as dense augmentation/loss and self-distillation with ensemble prediction. Though it demonstrates slightly better accuracy, it’s unclear how the different parts (mixup, dense augmentation, self-distillation) interact with each other. 

questions
* in experiments, half of the time it’s using put augmentation, the other half MultiMix, unclear how this hyperparameter affect performance. Do we must have input mixup? Why?
* What prevents us from applying dense augmentation and self-distillation to other mixup baselines?
* for manifold mixup, did you also apply mixup in the last layer?
* previous papers show mixup more examples doesn't help, which is contrary to what this paper argues. But there is no ablation study to still mixup two examples, but mixup more batches. How do we know the gain is from mixing up more examples or simply because we have more tuples? 
* high-level question: penultimate layer is already linear representation (i.e., input of logistic regression), and mixup is intend to regularize by enforcing linearity of the representation, in this case, why would it still help to apply mixup? In another work, would it help to apply mixup for logistic regression? (or softmax for multi-lass)
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes MultiMix which is a variant of the Mixup data augmentation technique. While Mixup creates augmented data by only considering interpolation between two data points, MultiMix considers the interpolation between $m$ data samples. In greater detail, MultiMix creates $n$ new augmented samples from $m$ data samples by linear mixing them with random weight vectors that are drawn from a Dirichlet distribution.  Moreover, the authors extend MultiMix to Dense-Multimix by removing spatial pooling and applying the loss function densely over all tokens/patches in some special deep-learning architectures. In the case that has label information, the author uses online self-distillation for creating soft labels for augmented data. On the application side, the authors conduct experiments in image classification, robustness to adversarial attacks, object detection, and out-of-distribution detection. The proposed techniques improve the classification accuracies on CIFAR10, CiFAR100, and TI better than baselines in both clean and adversarial attacked data. The proposed techniques also yield better accuracy in objective detection on VOC07+12 (SSD model) and MS-COCO (FASTER R-CNN model). 
strength_and_weaknesses: ## Strength

* The paper is well-organized and easy to follow.
* The experiments are conducted on various datasets and applications. The proposed techniques improve the performance slightly.
* Multimix is more robust to the adversarial attack than the previous baselines. 
* The dense Multimix idea is interesting to perform mixup techniques in deep learning.

## Weaknesses

* The idea of using more samples to do mixing is not new. The authors should include those variants in the comparison. The authors have mentioned $m$-Mixup. The other variant that the paper should include is $k$-MIXUP [1] which also considers $m$ samples as MultiMix, however, $k$-mixup finds the optimal mapping between two mini-batches of size $m$ via optimal transport.

[1] k-Mixup Regularization for Deep Learning via Optimal Transport, Kristjan Greenewald, et al

* MultMix needs to sacrifice the computation time for the performance. However, I see that the improvement in performance is not very significant while the computational time is only reported in the image classification application. The authors should report the computational time for all other applications for completeness.
clarity,_quality,_novelty_and_reproducibility: * Clarity: The paper is well-written.
* Quality:  The experiments are extensive, namely, they are conducted on various applications and datasets.
* Novelty: The idea of Multi-Mix is not very original. However, the extension to Dense Mult-Mix is new to the best of my knowledge.
* Reproducibility: The code is not submitted, however, experimental settings are reported carefully.
summary_of_the_review: The paper misses some important baselines that also consider $m$ samples for creating new augmented data as mentioned in the weakness part. Moreover, the trade-off between performance and computation is not well-studied. A figure is needed with the x-axis as computational time and the y-axis as the performance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes MultiMix, a strategy to extend prior MixUp style techniques to allow mixing k samples (rather than just a pair). The idea is fairly straightforward: one generalizes the $(\lambda, 1-\lambda)$ convex weights to samples for mixing coefficients from a Dirichlet distribution (which is well-defined for any number of samples k). The authors test on several datasets (testing classification, transfer, robustness) and few ablations (in particular w and w/o distillation). Overall, the feeling of the reviewers was that while the method is well-described, and the paper is easy to read and re-implement the algorithm, the improvements gained on all the datasets considered are not particular big (especially since the training computation cost rises). 

justification_for_why_not_higher_score: The method is fairly straightforward, which on its own would not be a problem --- however, the performance gains are pretty marginal (and come at an expected marginal computational cost).
Decision: Reject