Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this work, authors provide an analysis of mutual information for MSDA and the develop a new variant of mixup. The effectiveness is demonstrated by experiments.

Strength
1.	Authors study the difference between masking MSDA and interpolative MSDA, which is helpful for understanding the power of mixup and its variants.
2.	They develop a new augmentation method and improve the performance of masking MSDA.

Weakness
1.	The proposed measurement is not helpful for designing new methods. Note that the mutual information in mixup is lower than baseline while mixup still outperforms baseline.
2.	Compared to mixup and cutmix, the improvement reported in Table 2 is marginal.
3.	The experiments on ImageNet is unconvincing. Both of mixup and cutmix are worse than baseline, which contradicts the existing results.
4.	There lacks the discussion for the saliency based mixup methods, e.g., Puzzle Mix [1]. It is closely related to fmix but equipped with a learnable strategy to obtain patches for mixing.

[1] J-H Kim, et al. Puzzle mix: Exploiting saliency and local statistics for optimal mixup
review_title: A new variant of cutmix but the improvement is marginal

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces a new mixup method that builds masks by first sampling a grey-scale mask from fourier space, which is subsequently transformed into a binary mask. This improves results against several baselines and achieves state-of-the-art on a few important vision benchmarks.

My first remark is about the masking. The procedure seems fine, but why not compare to the way masks are sampled in context encoders [1]. This seems like an important baseline masking method to compare to. In addition, one could try sampling masks from a standard segmentation model, e.g., R-CNN.

My second remark is about the MI bounds. On page 14 in the Appendix, you state that the MI between Z_A and X_hat is approximately equal to the KL divergence between the posterior and the normal distribution, but in general this wont be true as in training the Gaussian mixture p_Za wont match the normal distribution, so you have an upper bound. So you have a lower bound of an upper bound to the MI, not a lower bound.

I'm curious though why not just use one of the recent neural estimators, e.g., found in [2] or [3]. In general, using VAEs for MI estimators depends heavily on the quality of the generator, so these neural estimators might be better suited.

Other comments:
P1:
* "'post-processing cannot increase information'" if such processing is deterministic, no?

P2:
* "CutMix imposes an unnecessary limitation": what limitation? Could you clarify?

Finally, do you plan to have updated results that compare to the 1024 batch size / 300 epoch settings?

[1] Pathak, Deepak, et al. "Context encoders: Feature learning by inpainting." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
[2] Belghazi, Mohamed Ishmael, et al. "Mine: mutual information neural estimation." arXiv preprint arXiv:1801.04062 (2018).
[3] Poole, Ben, et al. "On variational bounds of mutual information." arXiv preprint arXiv:1905.06922 (2019).
review_title: Review: FMix: Enhancing Mixed Sample Data Augmentation

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes an advanced masking strategy for CutMix augmentation based on the low-pass filter. The authors provide an interesting mutual information analysis for different augmentation strategies to describe their motivation. The experiments include many vision tasks (CIFAR-10, CIFAR-100, Fashion-MNIST, Tiny-ImageNet, ImageNet, Bengali datasets) and language tasks (Toxic, IMDb, Yelp).

**Pros**

\+ The mutual information analysis provides us a new perspective to understand different data augmentations.

\+ Various experiments.

**Cons**

**[Contradictory results between mutual information and performances]**
If we believe the VAE experiments in section 3, we have another paradox: the mutual information measurement and the real performance are not related.
Table 1 shows that in terms of mutual information, MixUp < Baseline < CutMix (and < FMix with a very small gap).
However, many experiments in this paper show that baseline < mixup < cutmix in terms of the performances.
This paper cites information bottleneck theory to justify the deceases shown by Mixup, but it is still contradictory to the performances.
It makes me confused to understand the meaning of mutual information. What is good for an augmentation method if we have high or low mutual information? It is still unclear to me.

A similar comment also can be applicable to the "adversarial robustness" experiments. Aside from that mixup is hard to say "adversarial training" (what is the threat model in this scenario?), I feel that this result is irrelevant to FMix motivation.


**[Weak logical connection between the motivation and the method]**
In my opinion, the connection between the analysis in the motivation and the proposed method is too weak. This paper proposes a CutMix variant where the mask is sampled by a low-pass filter. Why the low-pass filter approach can solve the motivation, i.e., enhancing mutual information between input and augmented images? There could be other possible variants as discussed in my "related works" comment


**[Related works]**
There are a few CutMix variants that employ a non-random masking strategy. Especially, I believe these two variants, which have similar motivation, should be compared:

- Walawalkar, Devesh, et al. "Attentive Cutmix: An Enhanced Data Augmentation Approach for Deep Learning Based Image Classification." ICASSP 2020
- Kim, Jang-Hyun, Wonho Choo, and Hyun Oh Song. "Puzzle mix: Exploiting saliency and local statistics for optimal mixup." ICML 2020

where Attentive CutMix uses CAM to extract masks, and PuzzleMix employs an optimization problem to optimize masks.
If it is possible, please provide more comparison between these two papers.


**[Too small performance gap, less convincing experiments]**
In Table 2, the performance gap between FMix and CutMix is too small, usually less than 0.3%. Note that the performance gaps are almost neglectable in these tasks.

Furthermore, FMix is often worse than CutMix in many tasks (Table 2 TinyImageNet, Table 3 ImageNet-A, Table 4, CIFAR-10H Table 6). I wonder what is the advantage to use FMix comparing to CutMix if FMix shows worse performance than CutMix.

Especially, I believe Table 3 is problematic. This paper argues that "Mixup uses 1024 batch size and CutMix uses 300 epochs". However, in PuzzleMix Table 5, CutMix-trained ResNet50 (top1 err 22.92) outperforms baseline ResNet50 (top1 err 24.31) with only 100 epochs. Thus, to me, this table is not convincing enough.


**[Potential issues in VAE analysis]**
The mutual information analysis is heavily relying on the learned VAE model. I wonder the quality of the generated images by VAE, in terms of both qualitatively (please provide generated samples in the supplementary) and quantitatively (e.g., FID).
If the VAE is not optimized well, the analysis will not be convincing enough.


**Minor comments**
- Why CutMix experiments are missed in Table 5?
- I suggest avoiding using the words, "clear" and "clearly".

---

**Post-rebuttal update**

My main concerns in the initial review were three-folds:

- Potential flaws in the analyses based on VAE and adversarial attacks
- Unclear connection between the MI analysis and the proposed method
- Small performance gap, and even sometimes worse performance, compared to the baseline methods (Mixup, CutMix)

After having discussions with the authors, I will keep my initial score because:

- I am still confused about the MI-based analysis conclusion. The authors mentioned *"We make no claim that increasing or decreasing the mutual information measure will have a strong impact on performance. Instead, we contend that MixUp works by forcing the model to ignore sample specific features (thus learning compressed representations – the reason for discussing the information bottleneck theory) and that CutMix works by mimicking the real data whilst preventing example memorization."* in the rebuttal, but these two conclusions are not trivial to me (by the MI analysis).
- Even if we ignore the first part, my second concern still remains. The authors mentioned *"That is the problem FMix tries to solve by removing the horizontal and vertical edge artefacts from cutmix. Our belief is that cutmix biases models towards these edges as they are a guaranteed feature of the data and learning about them would reduce the loss since these edges can tell you how much of each source image is present in the input (a key part of the objective)."*. But if this paper assumes that the rectangle masking strategy of CutMix makes bias, then I think other CutMix variants such as AttentiveCutMix or PuzzleMix should be considered as the comparison methods. Hence, I disagree with this statement *"A comparison to masks generated using additional models (and, thus, significant additional computation) does not seem fair to us."*
- For my last concern, the small performance gap, the authors claimed that this method *"was also used by the second place team in the BengaliAI Kaggle competition"*. It is good evidence that FMix can sometimes offer benefit to real-world applications, but I think more evidence that FMix can really solve problems of previous MSDA in a certain scenario, e.g., the edge bias as pointed by the authors.
review_title: Weak logical connection between the motivation and the method, too small performance gap

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents an interesting analysis of CutMix and MixUp data augmentation techniques. It also presents an improvement to CutMix that removes the horizontal/vertical axis bias. The idea to use fourier noise to construct masks for a variant of CutMix is interesting and well-motivated.

My main concern is whether the conclusions drawn by the analyses are fully grounded. The paper performs an analysis of the effect of the augmented data on learned representations by training unsupervised models on the augmented or clean data and measuring their mutual information. This analysis has the undesirable property of not matching the supervised case in a number of ways, such as different learning objectives, model architectures, etc. Even ignoring this, if we take the result that “MixUp consistently reduces the amount of information that is learned about the original data”, what then explains the improved generalization accuracy MixUp showcases in their original paper?

Moreover, after claiming that the analysis indicates that MixUp learns different representations, the paper asks “whether these different representations learned from MixUp give rise to practical differences other than just improved generalisation.” The issue is that they do this via an adversarial attack analysis, rather than a more realistic non-worst-case robustness analysis. This leads to the conclusion “MixUp (...) does not correspond to a general increase in robustness.” But it does not answer the original question of whether “MixUp gives rise to practical differences other than just improved generalisation.” The finding that MixUp yields greater ImageNet-A robustness (presented later in the paper) also contradicts this early claim.

The finding that MixUp provides more compressed representations does not necessarily mean that masking augmentation methods are better than interpolation ones. The paper seems to acknowledge this as well, in the final paragraph of the introduction, where it describes an experiment in which combining FMix+MixUp gives the best results (presumably because their representations of data are different and therefore combining them would yield the best of both worlds). This seems to contradict the previous adversarial analysis in which MixUp was found to not yield significantly more robustness. Further, the combination experiment has the two leading combination methods (FMix+MixUp and CutMix+MixUp) yield very similar results (within the margin of error), which opens the question of whether FMix meaningfully improves over CutMix.

Overall, I find the paper very easy to read and presenting some interesting ideas and even some exciting improvements in performance. I just wish the presentation and the claims made in the analysis of MSDA methods accounted for some of the inconsistencies described above.

Update after rebuttal: I appreciate the authors' response and clarifications. I maintain my original score.

review_title: Good paper, but concerns in the conclusions from the analyses

Meta-review: The paper analyzes the space of mixed sample data augmentation approaches, and proposes a new variant, FMix, based on a new masking strategy. Reviewers point to the fact that FMix is only marginally better than previous approaches, that the experimental setup is unconvincing, and that the proposed analysis might not be grounded. This is a really borderline paper but I see the issues as more important than the benefits, so I recommend rejection.
Decision: Reject