Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: 1) Summary
- The authors proposed the negative data augmentation technique which is useful for generative adversarial networks, anomaly detection, self-supervised learning frameworks.
- The idea is simple, and the technique was proven that it is powerful for several tasks.
- They performed several experiments, and I think the experiments were enough to show the technique's superiority.

2) Strong points
- Good idea
- Strong experimental results
- Simple to use
- Easy to understand

3) Weak points
- In Figure 3, they claim that in the absence of NDA. the support of a generative model learned from samples may "over-generalize"...
- I am not sure that the sentence is true.

This paper is well written, and concrete, I recommend that this paper should be presented in ICLR 2021. 

review_title: Review for Negative Data Augmentation

final_score: rating: 7: Good paper, accept
review_content: This paper investigates how augmenting the negative examples, not just the positive examples, can improve a variety of representation learning tasks. The paper investigates a number of different augmentations, and applies them to GANs and contrastive learning with images and videos.

Strengths:
A major strength of the paper is its simplicity. The method is fairly straightforward to implement into several approaches, and it obtains strong results on each approach evaluated in the paper. The approaches evaluated on GANs and contrastive learning with images and videos. 

Although the novelty of this method is limited, the paper does a good job at establishing some theoretical results to give intuition why the method works. In contrast to the number of advances in machine learning that lack intuition into why it works, this paper does a good job at offering some explanations and motivations for the approach. 

Although this paper focuses on images and videos, the same ideas could be extended to other modalities, such as text or audio, as well.

The experiments are convincing to show the generality of this idea. The experiments are on several different datasets. The experiments are supported by theoretical results, establishing intuition into why the method works. The introduction does a good job at establishing the difference to other data augmentation methods, in particular by using negative examples. 

The paper is well written and easy to read. 

Weaknesses:
In some cases, the negative data augmentations may actually be inside the positive set. How would the approach scale with noise in the negative augmentations? 

review_title: Official Review #4

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a method that uses artificial augmentation of data as Negative (aka OOD) samples to improve various computer vision tasks, including generation, unsupervised learning on images and videos.

Prons:
- The paper is very well written.
- Experiments are comprehensive across different tasks
- The usage of data augmentation seems interesting but with some questions (see below).
- It designs losses for both GANs and contrastive representation learning.
- Code is provided.

Cons:
- Augmentation has been proven in GANs to provide benefits through consistency training  (e.g. CR-GAN, ICLR2020,  Image Augmentations for GAN Training). These samples are used as "positive" samples that should generate consistent predictions. The most famous mixup is also treated as "positive" samples for training. So the augmentation usage here is a bit counterintuitive to me, because you show the opposite conclusion. Is that because only particular augmentation can be used as negative samples, e.g. Jiasaw? The answer to this question is critical. However, the paper does not mention/ study much.
- Advanced self-supervised (contrastive) learning reply on strong augmentation, how negative samples can adapt to these methods? How do we categorize augmentation types used for general cases or NDA cases? Any insights on what kind of augmentations are useful for NDA. For example, in figure 9, the paper proposes to push samples and its jigsaw version away. However, these two pairs share strong local visual contents of objects (just like an image and its crop parts) that usually contrastive learning wants to pull them together. The proposed method tries to push them away. Any insights why it should work?
- If justifications of these questions can be sufficient, I think it can be a strong paper. 

review_title: Anonymous review

final_score: rating: 6: Marginally above acceptance threshold
review_content: -- POST REBUTTAL --

The authors addressed well most of my concerns.  I increase my rating. However, the authors need to address all comments of the reviewers and also discuss all missing related works in the updated version.

– Summary – 

The paper proposes a new method of leveraging the negative samples (out-of-distribution samples purposely generated from the training data distribution) in generative modeling and representation learning. The main idea aims to leverage the inductive bias of negative samples to constrain the learning of the model, e.g., these negative samples may tell some more information about the support of the data. The experimental results suggest that using these negative samples in GANs (studies with BigGAN model for conditional/unconditional image generation) and contrastive representation learning (study with CPC (Oord et al., 2018) on unsupervised learning on images and videos ) improves the performance of baselines. The paper also reports on improvements in image-image translation and anomaly detection. The paper also provides theorems to prove the convergence of the proposed model on GANs and CPC. 

Overall, the paper is easy to read and the idea makes sense. However, I'm a bit concerned about the theory, the significance of improvements and the fairness of the comparison. The paper also misses to discuss and compare with recent works also on data augmentation for GANs.

 -- Strength -- 

S1 – The usage of negative examples, which obtained from some prior knowledge, to provide the evidence of the learning model about the support/geometry of data distribution sounds reasonable. It has been applied in (Sung et al. (2019)) in semi-supervised learning. The proposed method applies to new applications of generative and representation learnings. 

S2 – The experimental results are quite extensive in regards to the applications, and the improvements on GANs quite significant with jigsaw augmentation.

 -- Weakness --

W1 – The paper does not provide the very detailed implementations of proposed models, which is a bit difficult to justify the correctness.

*Generative learning*

W2 – The detail of how to incorporate NDA into GAN is not clear. Also, the PDA baseline for GAN is not detailedly discussed. Does the PDA, NDA, and baseline BigGAN train with the same batch size? I guess that the PDA and NDA had more augmented samples, therefore batch size is larger than the bigGAN baseline?

W3 – The paper does not discuss important related works [a,b,c] of Data Augmentation for GAN recently published. In these papers, they show transforming only real samples (if I understand correctly, it likely is similar to PDA of the proposed baseline) only to train GAN changing the target distribution therefore the generator will learn infrequent or out-of-distribution samples. However, if both real/fake are transformed, data augmentation is helpful in training GAN. Can the authors compare the proposed NDA to at least one of them with the same GAN model?

[a] Differentiable Augmentation for Data-Efficient GAN Training

[b] On Data Augmentation for GAN Training

[c] Training Generative Adversarial Networks with Limited Data


W4 – Eq. 10 showed that $L_f(\lambda * G_{\theta} + (1 - \lambda) * \overline{P}, D_{\phi}) <= D_f(P||\lambda*Q+(1-\lambda * \overline{P}))$, but then infers the lower bound: $D_f(P||\lambda*Q+(1-\lambda * \overline{P})) >= \lambda * f(\frac{1}{\lambda} + (1 - \lambda) f(0)) = D_f(P||\lambda*P+(1-\lambda * \overline{P}))$. Therefore, theoretically I am concerned a bit about the convergence of the model. I wonder whether the authors need an upper bound instead of the lower-bound in this case?

W5 – The paper claimed: “Random Horizontal Flip is not effective as an NDA; this is because flipping does not spatially corrupt the image but is rather a semantic preserving transformation”. How about the Random Vertical Flip only? Can it improve the model since this augmentation looks very good to tell us about the boundary of the data? 

*Representation learning*

W6 – The improvements in representation learning do not look significant to me, and the improvements are not too consistent on different datasets according to the type of augmentations.

W7 – The lower bound of Eq. 18 looks just like due to adding a larger batch size for negative samples to train CPC. Can authors compare NDA to CPC with just the same batch size training as the NDA method? 

review_title: The idea makes sense but some concerns about the experiment details and theory

Meta-review: All reviewers find the proposed data augmentation approach simple, interesting and effective. They agree that paper does a good job exploring this idea with number of experiments. However the paper also suffers from some drawbacks, and reviewers raise questions about some of the conclusions of the paper - in particular how to designate an augmentation as either negative or positive is not clear apriori to training. While I agree with this criticism, I believe the paper overall explores an interesting direction and provides a good set of experiments than can be built on in  future works, and I suggest acceptance. I encourage authors to address all the reviewers concerns as per the feedback in the final version.
Decision: Accept (Poster)