Negative Data Augmentation 
Authors: Abhishek Sinha
Kumar Ayush
Jiaming Song
Burak Uzkent
Hongxia Jin
Stefano Ermon
Venue: ICLR 2021 Poster (2021)
Keywords: generative models
self-supervised learning
data augmentation
anomaly detection

Abstract: Data augmentation is often used to enlarge datasets with synthetic samples generated in accordance with the underlying data distribution. To enable a wider range of augmentations, we explore negative data augmentation strategies (NDA) that intentionally create out-of-distribution samples. We show that such negative out-of-distribution samples provide information on the support of the data distribution,  and can be leveraged for generative modeling and representation learning. We introduce a new GAN training objective where we use NDA as an additional source of synthetic data for the discriminator. We prove that under suitable conditions, optimizing the resulting objective still recovers the true data distribution but can directly bias the generator towards avoiding samples that lack the desired structure. Empirically, models trained with our method achieve improved conditional/unconditional image generation along with improved anomaly detection capabilities. Further, we incorporate the same negative data augmentation strategy in a contrastive learning framework for self-supervised representation learning on images and videos, achieving improved performance on downstream image classification, object detection, and action recognition tasks. These results suggest that prior knowledge on what does not constitute valid data is an effective form of weak supervision across a range of unsupervised learning tasks.

Published as a conference paper at ICLR 2021
NEGATIVE DATA AUGMENTATION
Abhishek Sinha 1*
Kumar Ayush
Jiaming Song!*
Burak Uzkent
Hongxia Jin?
Stefano Ermon
Department of Computer Science Stanford University {a7b23, kayush, tsong, buzkent, ermon) @ @stanford.edu
Samsung Research America²
ABSTRACT
Data augmentation is often used to enlarge datasets with synthetic samples gen- erated in accordance with the underlying data distribution. To enable wider range of augmentations, we explore negative data augmentation strategies (NDA) that intentionally create out-of-distribution samples. We show that such negative out-of- di-triiiitiiittioonn samples provide information on the support of the data distri- bution, and be leveraged for generative modeling and representation learning. We introduce a new GAN training objective where we use NDA as an additional source of synthetic data for the discriminator. We prove that under suitable con- ditions, optimizing the resulting objective still recovers the true data distribution but can directly bias the generator towards avoiding samples that lack the desired structure. Empirically, models trained with our method achieve improved con- ditional/unconditional image generation along with improved anomaly detection capabilities. Further, we incorporate the same negative data augmentation strategy in a contrastive learning framework for self-supervised representation learning on images and videos, achieving improved performance downstream image clas- sification, object detection, and action recogmition tasks. These results suggest that prior knowledge on what does constitute valid data is an effective form of weak supervision across a range of unsupervised learning tasks.
INTRODUCTION
Data augmentation strategies for synthesizing new data a way that is consistent with an underlying task extremely effective in both supervised and unsupervised learning (Oord al., 2018; Zhang et al., 2016; Noroozi & Favaro, 2016; Asano et al., 2019). Because they operate at the level of samples, they be combined with most learning algorithms. They allow for the incorporation of prior knowledge (inductive bias) about properties of typical samples from the underlying data distribution (Jaiswal al., 2018; Antoniou al., 2017), e.g., by leveraging invariances to produce additional "positive" examples of how a task should be solved.
To enable users specify an even wider range of inductive biases, we propose to leverage an alter- native and complementary source of prior knowledge that specifies how a task should nor be solved. We formalize this intuition by assuming access way of generating samples that are guaranteed to be out-of-support for the data distribution, which we call a Negutive Data Augmentation (NDA). Intuitively, negative ouito--disttibuution (OOD) samples can be leveraged as a useful inductive bias because they provide information about the support of the data distribution to be learned by the model. For example, in density estimation problem we can bias the model to avoid putting any probability mass in regions which know a-priori should have zero probability. This can be an effective prior the negative samples cover a sufficiently large area. The best NDA candidates are ones that expose common pitfalls of existing models, such prioritizing local structure over global
"Equal Contribution

---

Published as a conference paper at ICLR 2021
structure (Geirhos et al., 2018); this motivates to consider known transformations from the litera- ture that intentionally destroy the spatial coherence an image (Noroozi & Favaro, 2016; De Vries & Taylor, 2017; Yun et al., 2019), such as Jigsaw transforms.
Building on this intuition, we introduce new GAN training objective where we use NDA as an additional source of fake data for the discriminator shown in Fig. 1. Theoretically, that if the NDA assumption is valid, optimizing this objective will still recover the data distribution in the limit of infinite data. However, in the finite data regime, there is need generalize beyond the empirical distribution (Zhao et al., 2018). By explicitly providing the discriminator with samples we want avoid, we are able to bias the generator towards avoiding undesirable samples thus improving generation quality.
show
Furthermore, we propose way of leveraging NDA for unsu- pervised representation learning. We propose new contrastive predictive coding (He et al., 2019; Han et al., 2019) (CPC) ob- jective that encourages the distribution of representations cor- responding to in-support data to become disjoint from that of NDA data. Empirically, we show that applying NDA with our proposed transformations (e.g.. forcing the representation of nor- mal and jigsaw images to be disjoint) improves performance in downstream tasks.
D(T(=))
D(x)
D
With appropriately chosen NDA strategies, obtain superior empirical performance on a variety of tasks, with almost no cost in computation. For generative modeling, models trained with NDA achieve better image generation, image translation and anomaly detection performance compared with the same model trained without NDA. Similar gains are observed on representa- tion learning for images and videos over downstream tasks such as image classification, object detection and action recognition. These results suggest that NDA has much potential improve a variety of self-supervised learning techniques.
z
T
T(x)
Figure 1: Negative Data Aug- mentation for GANs.
2 NEGATIVE DATA AUGMENTATION
The input to most learning algorithms is dataset of samples from an underlying data distribution Pdota- While Patta unknown, learning algorithms always rely on prior knowledge about its prop- erties (inductive biases (Wolpert & Macready, 1997)), e.g., by using specific functional forms such as neural networks. Similarly, data augmentation strategies exploit known invariances of pactas such as the conditional label distribution being invariant to semantic-preserving transformations.
While typical data augmentation strategies exploit prior knowledge about what in support of Pdsta in this paper, propose to exploit prior knowledge about what is not in the support of Pdats. This information is often available for common data modalities (e.g., natural images and videos) and under-exploited by existing approaches. Specifically, distribution p such that its support disjoint from that of Pdata: and (2) access to a procedure to efficiently sample from p. We emphasize need not be explicitly defined (e.g.. through explicit density) it may be implicitly defined by dataset or by procedure that transforms samples from Pdata ones from p by suitably altering their structure.
assume: (1) there exists alternative
Original
Jigsaw
Stitching
Mixup
Figure 2: Negative augmentations produce -ittt--distribution samples lacking the typical structure of natural images; these negative samples be used to inform model on what it should not learn.
2

---

Published as a conference paper at ICLR 2021
Analogous typical data augmentations, NDA strategies by definition domain and task specific. In this paper. we focus on natural images and videos, and leave the application to other domains (such natural language processing) as future work. How do we select good NDA strategy? Ac- cording to the manifold hypothesis (Fefferman et al., 2016), natural images lie on low-dimensional manifolds: Pdata is supported
low-dimensional manifold of the ambient (pixel) space. This suggests that many negative data augmentation strategies exist. Indeed, sampling random noise is in most cases valid NDA. However, while this prior generic, is not very informative, and this NDA will likely be ineffective for most learning problems. Intuitively, NDA is informative if its support is close (in suitable metric) that of Pdatas while being disjoint. These negative samples will provide information on the "boundary" of the support of Pdatas which we will show is help- ful in several learning problems. In most of our tasks, the images are processed by convolutional neural networks (CNNs) that are good at processing local features but not necessarily global fea- tures (Geirhos et al., 2018). Therefore, we may consider NDA examples to be ones that preserve local features ("informative") and break global features, so that it forces the CNNs to learn global features (by realizing NDAs are different from real data).
Leveraging this intuition, show several image transformations from the literature that can be viewed as generic NDAs over natural images in Figure 2. that we will use for generative modeling and representation learning in the following sections. Details about these transformations can be found in Appendix B.
3
NDA FOR GENERATIVE ADVERSARIAL NETWORKS
Pa
CannCara
Pv
P
P2
Then delec
Figure 3: Schematic overview of our NDA framework. Left: In the absence of NDA, the support of a generative model Po (blue oval) learned from samples (green dots) may "overrgeneraaize"" and include samples from P1 or P2. Right: With NDA, the learned distribution Po becomes disjoint from NDA distributions Pi and P2. thus pushing Pa closer to the true data distribution Pdata (green oval). As long as the prior is consistent, i.e. the supports of P1 and P2 are truly disjoint from Pdotas the best fit distribution in the infinite data regime does not change.
In GANs, we are interested in learning a generative model Go from samples drawn from some data distribution pdata (Goodfellow et al., 2014). GANs use binary classifier, the so-called discriminator Ds. to distinguish real data from generated (fake) samples. The generator Go is trained via the following mini-max objective that performs variational Jensen-Shannon divergence minimization:
min max Lis (Go, D&) where Go D₂
Lis(Go, D&) = - Exe [log(Dx + En-G, [log(1 - (x)))
(2)
This is special case to the more general variational f-divergence minimization objective (Nowozin et al., 2016). The optimal Do for any Go is (paata/ (aaata/Gg)/aa (1 + Pdata/Ga), the discriminator can serve as a density ratio estimator between Pdata and Gg.
With sufficiently expressive models and infinite capacity, Go will match Pdata In practice, however, we have access to finite datasets and limited model capacity. This means that the generator needs to generalize beyond the empirical distribution, which is challenging because the number of possi- ble discrete distributions scale doubly exponentially w.r.t to the data dimension. Hence, as studied in (Zhao al., 2018), 8), the role of the inductive bias is critical. For example, Zhao al. (2018) report
3

---

Published as a conference paper at ICLR 2021
that when trained on images containing 2 objects only, GANs and other generative models can some- times "generalize" by generating images with 1 or 3 objects (which were never seen in the training set). The generalization behavior which may or may not be desirable is determined by factors such as network architectures, hyperparameters, etc., and is difficult to characterize analytically.
Here propose to bias the learning process by directly specifying what the generator should not generate through NDA. We consider an adversarial game based on the following objective:
min max Las (AGg + - P D&) GeEP(2) De
where the negative samples are generated from mixture of Go (the generator distribution) and P (the NDA distribution); the mixture weights are controlled by the hyperparameter A. Intuitively, this can help addresses the above "overrgenerall "veergeneraaization"" issue, what should not be generated and thus guide the support of Ga (see Figure 3) For instance, in the object count example above, we can empirically prevent the model from generating images with an undesired number of objects (see Appendix Section A for experimental results on this task).
can directly provide supervision on
In addition, the introduction of NDA samples will not affect the solution of the original GAN ob- jective in the limit. In the following theorem, we show that given infinite training data and infinite capacity discriminators and generators, using NDA will not affect the optimal solution the gener- ator, i.e. the generator will still recover the true data distribution.
Theorem 1. Let P E P(X) be any distribution over X with disjoint support than Pdata i.e., such that supp(paara) n supp(P) = Ø. Let Do : + be the set of all discriminators over X, f: RRz0 + R be convex, remi-continuous finnttion such that f(1) 0, f be the convex conjugate of f. its derivative, and Go be a distribution with sample space X. Then Aλ E (0, we have:
arg min max (2EP(X) D
Ly(Go, D&) = arg min max L5(AG, + +1 A),, GeEP(X) De:i --
Pdata
(4)
where Ly(Q, Do) Em-paacal Em-pinulDe(tt) - Ea-ol
is the objective for f-GAN (Nowozin et al., 2016). However, the optimal discriminators are different for the two objectives:
arg max (G D&) = f (Pdata/ /Ge
(5)
Dg:X' RR
arg max maxmaxX (AGo + (1 A)P,Ds) = (' (AGo + ((1- De:X' -R
(6)
Proof. See Appendix C.
The above theorem shows that in the limit of infinite data and computation, adding NDA changes the optimal discriminator solution but not the optimal generator. In practice, when dealing with finite data, existing regularization techniques such as weight decay and spectral normalization (Miyato al., 2018) allow potentially many solutions that achieve the same objective value. The introduc- tion of NDA samples allows us to filter out certain solutions by providing additional inductive bias through OOD samples. fact, the optimal discriminator will reflect the density ratio between Pdata and XGG + (1 A)P (see (6),, and its values will be higher for samples from pdata compared to those from P. As we will show in Section 5, discriminator trained with this objective and suitable NDA performs better than relevant baselines for other downstream tasks such as anomaly detection.
4 NDA FOR CONSTRASTIVE REPRESENTATION LEARNING
Using a classifier to estimate a density ratio is useful not only for estimating f-divergenness (as in the previous section) but also for estimating mutual information between two random variables. In representation learning, mutual information (MI) maximization is often employed to learn compact yet useful representations of the data, allowing one to perform downstream tasks efficiently (Tishby & Zaslavsky, 2015; Nguyen et al., 2008; Poole et al., 2019b; Oord et al., 2018). Here, we show that NDA samples are also beneficial for representation learning.
In contrastive representation learning (such as CPC (Oord et al., 2018)), the goal is to learn a map- ping hx)) X - P(2) that maps datapoint xr to some distribution over the representation space
4

---

Published as a conference paper at ICLR 2021
2; once the network he is learned, representations are obtained by sampling from ho(x)) CPC
maximizes the following objective:
ng0 (2e, 2)
(hoc g0) = T. Regrrpann (0 (xp). log (0(x, + Σド= go (2, 2n)
where po(=) - hz (r Pastaa (x) dd is the marginal distribution of the representations associated with Pdata- Intuitively, the CPC objective involves n-class classification problem where 9Φ at- tempts to identify a matching pair (i.e. (x, z)) sampled from the joint distribution from the (n 1) non-matching pairs (i.e. (æt, 2z)) sampled from the product of marginals distribution. Note that Jo plays the role of a discriminator//rittcc and is implicitly estimating a density ratio. As n - oc, the optimal Ie corresponds to an un- normmallzze density ratio between the joint distribution and the product of marginals, and the CPC objective matches its upper bound which is the mutual informa- tion between X and Z (Poole et al., 2019a; Song & Ermon, 2019). However, this objective is no longer able to control the representations for data that are out of support of Pdata. so there is a risk that the representations are similar between Paata samples and out- diitt--iistiiuutinn ones.
To mitigate this issue, we propose to use NDA in the CPC objective, where we additionally introduce a batch of NDA samples, for each positive sample:
(n + g) (Z, x)
(hg 9o)
EE
log
g0( 2) + プØ=1 go (r, Ei) + ===1 1 (2x, 2k.
where the expectation is taken over tribution), Σn ~ hg())x) for all & E [m]] Here, the behavior of ha(#)) when x is NDA is op- timized explicitly, allowing us to impose additional constraints to the NDA representations. This corresponds to a more challenging classification problem (compared to basic CPC) that encourages learning more informative representations. In the following theorem, we show that the proposed ob- jective encourages the representations for NDA samples to become disjoint from the representations for Pdata samples, i.e. NDA samples and Paata samples do not map to the same representation.
Palata (a))ta ho(a)) 2 po(z). Ty P (NDA dis-
Theorem 2. (Informal) The optimal solution he in the NDA-CPU objective maps the representa- tions of data samples and NDA samples to disjoint regions.
Proof. See Appendix D for a detailed statement and proof.
5 NDA-GAN EXPERIMENTS
In this section we report experiments with different types of NDA for image generation. Additional details about the network architectures and hyperparameters can be found in Appendix K.
Unconditional Image Generation. We conduct experiments on various datasets using the BigGAN architecture (Brock et al., 2018) for unconditional image generation'. We first explore various im-
1200
NDA-BigGAN
BigGAN
of Images
1000
age transformations from the literature to evaluate which ones are effective as NDA. For each transformation, we evaluate its perfor-
800
600
mance as NDA (training in Eq. 3) and as a traditional data aug- mentation strategy, where we enlarge the training set by applying the transformation to real images (denoted PDA for positive data augmentation). Table I shows the FID scores for different types of transformations as PDA/NDA. The results suggest that transforma- tions that spatially corrupt the image are strong NDA candidates. can be seen that Random Horizontal Flip is not effective as an NDA; this is because flipping does not spatially corrupt the image but rather a semantic preserving transformation, hence the NDA distribution P is not disjoint from Pdato- On the contrary, it is rea- sonable to assume that if an image is likely under Pdota. its flipped variant should also be likely. This is confirmed by the effectiveness of this strategy as PDA.
400
200
0
-10-5 5 10 15 20 30 D(natural) D(jigsaw)
Figure 4: Histogram of differ- ence in the discriminator out- put for real image and it's Jigsaw version.
We feed single label to all images make the architecture suitahle for unconditional generation.
5

---

Published as a conference paper at ICLR 2021
Table 1: FID scores over CIFAR-I0 using different transformations as PDA and NDA in BigGAN. The results indicate that some transformations yield better results when used as NDA. The common feature of such transformations they all spatially corrupt the images.
Aug.
Jigsaw
Cutout
Stitch
Mixup
Cutmix
Randow Crop
Random Flip
Gaussian
PDA
NDA
PDA
NDA
PDA
NDA
PDA
NDA
PDA
NDA
PDA
NDA
PDA
NDA
PDA
NDA
18.64
98.09
12.61
79.72
14.69
108. 69
13.97
70.64
17.29
90.81
15.01
20.02
15.05
16.65
124.32
44,41
18.72
Table 2: Comparison of FID scores of different types of NDA for unconditional image generation on various datasets. The numbers in bracket represent the corresponding image resolution in pixels. Jigsaw consistently achieves the best or second best result.
BigGAN
Jigsaw
Stitching
Mixup
Cutout
Cutmix
CR-BigGAN
CIFAR-10 (32)
18.64
12.61
13.97
17.29
14.69
15.01
14.56
CIFAR-100 (32)
22.19
19.72
20.99
22.21
22.08
20.78
CelebA (64)
38.14
37.24
37.17
37.51
37.39
37.46
STI.10 (32)
26.80
23.94
26.08
24.45
24.91
25.34
We believe spatially corrupted negatives perform well as NDA in that they push the discriminator to focus on global features instead of local ones (e.g., texture). We confirm this by plotting the histogram of differences in the discriminator output for a real image and it's Jigsaw version as shown in Fig. 4. We show that the difference is (a) centered close zero for normal BigGAN (so without NDA training. the discriminator cannot distinguish real and Jigsaw samples well), and (b) centered positive number (logit 10) for our method (NDA-BigGAN), Following our findings, in our remaining experiments we use Jigsaw, Cutout, Stitch, Mixup and Cutmix they achieve significant improvements when used as NDA for unconditional image generation on CIFAR-10.
Table 2 shows the FID scores for BigGAN when trained with five types of negative data augmenta- tion on four different benchmarks. Almost all the NDA augmentations improve the baseline across datasets. For all the datasets except CIFAR-I00, 100, X 0.25, whereas for CIFAR- 100 it is 0.5. We show the effect of À on CIFAR- performance Appendix H. We additionally performed an exper- iment using mixture of augmentation policy. The results (FID 16.24) were better than the baseline method (18.64) but not as good as using a single strategy.
Conditional Image Generation. We also investigate the benefits of NDA in conditional image generation using BigGAN. The results are shown in Table 3. this setting as well, NDA gives significant boost over the baseline model. We again use À 0.25 for CIFAR-10 and X 0.5 for CIFAR-100. For both unconditional and conditional setups we find the Jigsaw and Stitching augmentations achieve better FID score than the other augmentations.
Table 3: FID scores for conditional image generation using different NDAs? 2
BigGAN
Jigsaw
Stitching
Mixup
Cutout
Cutmix
CR-BigGAN
C-10
11.51
9.42
9.47
13.87
10.52
10.3
11.48
C-100
15.04
14.12
13.90
15.27
14.21
13.99
Image Translation. Next, we apply the NDA method to image translation. In particular, we use the Pix2Pix model (Isola et al., 2017) that can perform smage-to-image translation using GANs provided paired training data. Here, the generator is conditioned on an image I, and the discriminator takes as input the concatenation of generated/real image and I. We use Pix2Pix for semantic segmentation on Cityscapes dataset (Cordts et al., 2016) (i.e. photos -> labels). Table 4 shows the quantitative gains obtained by using Jigsaw 2DA while Figure in Appendix F highlights the qualitative im- provements. The NDA-Pix2Pix model avoids noisy segmentation on objects including buildings and trees.
We use PyTorch code for BigGAN. The number reported in Brock et al. (2018) for C-10 is 14.73.
We use the official Py implementation and show the hest results.

---

Published as a conference paper at ICLR 2021
Table Results on CityScapes, using per pixel accuracy (Pp.), per class accuracy (Pe.) and mean Intersection over Union (mIOU), We
Table 5: AUROC scores for different OOD datasets. OOD-1 contains different datasets, while OOD-2 contains the set of 19 different corruptions in CIFAR-IO-C (Hendrycks & Dietterich, 2018) (the average score reported).
compare Pix2Pix and its NDA version.
Metric
Pp.
Pc.
mIOU
Pix2Pix
BigGAN
Jigsaw
EBM
0.80
0.24
0.27
(cGAN)
DTD
0.70
0.69
0.48
NDA
SVHN
0.75
0.61
0.63
(cGAN)
0.84
0.34
0.28
Places-365
0.35
0.58
0.68
OOD-1
TinyImageNet
0.40
0.62
0.67
Pix2Pix
0.72
0.23
0.18
CIFAR- 100
0.63
0.64
0.50
(LI+CGAN)
NDA
Average
0.57
0.63
0.59
0.75
0.28
0.22
(LI+cGAN)
0OD-2 CIFAR-IO-C
0.56
0.63
0.60
Anomaly Detection. As another added benefit of NDA for GANs, we utilize the output scores of the BigGAN discriminator for anomaly detection. We experiment with 2 different types of OOD datasets. The first set consists of SVHN (Netzer et al., 201 1), DTD (Cimpoi et al., 2014), Places- 365 (Zhou et al., 2017), TinylmageNet, and CIFAR-100 as the OOD datapoints following the pro- tocol in (Du & Mordatch, 2019; Hendrycks al., 2018). We train BigGAN w/ and w/o Jigsaw NDA on the train set of CIFAR-I0 and then use the output value of discriminator to classify the test set of CIFAR-10 10 (not anomalous) and different OOD datapoints (anomalous) as anomalous or not. We use the AUROC metric as proposed in (Hendrycks & Gimpel, 2016) to evaluate the anomaly detection performance. Table compares the performance of NDA with likelihood based model (Energy Based Models (EBM (Du & Mordatch, 2019)). Results show that Jigsaw NDA performs much better than baseline BigGAN and other generative models. We did not include other NDAs as Jigsaw achieved the results.
We consider the extreme corruptions in CIFAR-IO-C (Hendrycks & Dietterich, 2018) as the sec- ond set of OOD datasets. It consists of 19 different corruptions, each having 5 different levels of severity. We only consider the corruption of highest severity for our experiment, as these constitute significant shift from the true data distribution. Averaged over all the 19 different corruptions, the AUROC score for the normal BigGAN is 0.56, whereas the BigGAN trained with Jigsaw NDA achieves 0.63. The histogram of difference in discriminator's 's output for clean and OOD samples shown in Figure 8 in the appendix. High difference values imply that the Jigsaw NDA is better at distinguishing OOD samples than the normal BigGAN.
6 REPRESENTATION LEARNING USING CONTRASTIVE LOSS AND NDA
Unsupervised Learning on Images. In this section, perform experiments three benchmarks: (a) CIFAR10 10 (C10), (b) CIFAR100 (C100), and (c) ImageNet-100 (Deng et al., 2009) to show the benefits of NDA representation learning with the contrastive loss function. In our experiments, we use the momentum contrast method (He et al., 2019), MoCo-V2, as is currently the state-of-the- art model on unsupervised learning on ImageNet. For C1O and C100, we train the MoCo-V2 model for unsupervised learning (w/ and w/o NDA) for 1000 epochs. On the other hand, for ImageNet- 100, train the MoCo-V2 model (w/ and w/o NDA) for 200 epochs. Additional hyperparameter details can be found in the appendix. To evaluate the representations, we train linear classifier on the representations on the same dataset with labels. Table shows the top-1 I accuracy of the classifier. We find that across all the three datasets, different NDA approaches outperform MoCo-V2 V2. While Cutout NDA performs the best for C10, the best performing NDA for C100 and ImageNet- Jigsaw and Mixup respectively. Figure 9 compares the cosine distance of the representations learned w/ and w/o NDA (jigsaw) and shows that jigsaw and normal images are projected far apart from each other when trained using NDA whereas with original MoCo-v2 they are projected close to each other.
Transfer Learning for Object Detection. We transfer the network pre- pre-trained over ImageNet- for the task of Pascal- VOC object detection using a Faster R-CNN detector (C4 backbone) Ren et al. (2015). We fine-tune the network Pascal VOC 2007+2012 trainval set and test it the 2007 test

---

Published as a conference paper at ICLR 2021
Table Top-1 accuracy results on image recognition w/ and w/o NDA on MoCo-V2
MoCo-V2
Jigsaw
Stitching
Cutout
Cutmix
Mixup
CIFAR-10
91.20
91.66
91.59
92.26
91.51
91.36
CIFAR-100
69.63
70.17
69.21
69.81
69.83
69.99
ImageNet-100
69,41
69.95
69.54
69.77
69.61
70.01
set. The baseline MoCo achieves 38.47 AP, 65.99 AP50, 38.81 AP75 whereas the MoCo trained with mixup NDA gets 38.77 72 AP, 66.23 AP50, 19.16 AP75 (an improvement of ~ 0.3).
Unsupervised Learning on Videos. In this section, we investigate the benefits of NDA in self- supervised learning of spatio-temporal embeddings from video, suitable for human action recogni- tion. We apply NDA to Dense Predictive Coding (Han et al., 2019), which a single stream (RGB videos. For videos,
only) method for self-supervised representation learning samples by performing the same transformation on all frames of the video (e.g. the same jigsaw permutation is applied to all the frames of a video). We evaluate the approach by first training the DPC model with NDA on large-scale dataset (UCF101), and then evaluate the representations by training supervised action classifier on UCF101 and HMDB51 datasets. As shown in Table 7, Jigsaw and Cutmix NDA improve downstream task accuracy on UCF-101 and HMDB-51, achiev- ing new ofate-of-the-ar performance among single stream (RGB only) methods for self-supervised representation learning (when pre-trained using UCF-101).
create NDA
Table 7: Top-1 accuracy results action recognition in videos w/ and w/o NDA in DPC.
DPC
Jigsaw
Stitching
Cutout
Cutmix
Mixup
UCF-101 (Pre-trained on UCF-101)
61.35
64.54 46.88
66.07 45.31
64.52 45.31
63.52 48.43
63.65
HMDB51 (Pre-trained on UCF-101)
45.31
43.75
7 RELATED WORK
In several machine learning settings, negative samples are produced from a statistical generative model. Sung al. (2019) aim to generate negative data using GANs for semi-supervised learning and novelty detection while we are concerned with efficiently creating negative data to improve generative models and self-supervised representation learning. Hanneke al. (2018) also propose an alternative theoretical framework that relies on access to an oracle which classifies a sample as valid not, but do not provide any practical implementation. Bose al. (2018) use adversarial training to generate hard negatives that fool the discriminator for NLP tasks whereas we obtain NDA data from positive data to improve image generation and representation learning. Hou et al. (2018) use a GAN to learn the negative data distribution with the aim of classifying positive-unlabeled (PU) data whereas we do not have access to mixture data but rather generate negatives by transforming the positive data.
In contrastive unsupervised learning, common negative examples are ones that are assumed to be further than the positive samples semantically. Word2Vec (Mikolov et al., 2013) considers negative samples to be ones from a different context and CPC-based methods (Oord et al., 2018) such as momentum contrast (He et al., 2019), the negative samples are data augmentations from different image. Our work considers a new aspect of "negative samples" that are neither generated from some model, nor samples from the data distribution. Instead, by applying negative data augmentation (NDA) to existing samples, we are able to incorporate useful inductive biases that might be difficult capture otherwise (Zhao et al., 2018).
CONCLUSION
We proposed negative data augmentation as a method to incorporate prior knowledge through out-of- distribution (OOD) samples. NDAs complementary traditional data augmentation strategies,
8

---

Published as a conference paper at ICLR 2021
which typically focused in-distribution samples. Using the NDA framework, interpret existing image transformations (e.g.. jigsaw) as producing OOD samples and develop new learning algorithms to leverage them. Owing to nggorous mathematical characterization of the NDA assump- tion, we are able to theoretically analyze their properties. As an example, we bias the generator of a GAN avoid the support of negative samples, improving results conditiona/uucconditonnan im- age generation tasks. Finally, we leverage NDA for unsupervised representation learning in images and videos. By integrating NDA into MoCo-v2 and DPC, we improve results image and action recognition on CIFARIO, CIFAR 100, ImageNet- 100, UCF- 101, and HMDB-51 datasets. Future work include exploring other augmentation strategies as well as NDAs for other modalities.
ACKNOWLEDGEMENT
The authors would like to thank Shengjia Zhao and Kristy Choi for reviewing earlier draft of the paper. This research was supprted by NSF (#1651565, #1522054, #1733686), ONR (N00014-19-1- 2145), AFOSR (FA9550-19-1-0024)) ARO, and Amazon AWS.
REFERENCES
Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. arXiv preprint arXhv:1711.04340 2017.
Yuki M Asano, Christian Rupprecht, and Andrea Vedaldi. A critical analysis of self-supervision, or what we can learn from single image. arXiv preprint arXiv: 1904.1133, 2019.
Avishek Joey Bose, Huan Ling, and Yanshuai Cao. Adversarial contrastive estimation. In Pro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics Linguisssss (Volume 1: Long Papers), pp. 1021-1032, 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Terrance VeVir and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint
2017.
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. arXiv preprint arXiv:1903.0866 2019.
Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Sociery, 29(4):983-104) 2016.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet- trained cnns are biased towards texture; increasing shape bias im- proves accuracy and robustness. arXiv preprint arXiv:1811.1223! 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 26722-680, 2014.

---

Published as a conference paper at ICLR 2021
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. Advances in neural information processing systems, pp- 5767-5777, 2017.
Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Steve Hanneke, Adam Tauman Kalai, Gautam Kamath, and Christos Tzamos. Actively avoiding nonsense generative models. In Conference On Learning Theory, pp. 209-227, 2018.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint 1rXiv:1911.05522..00
Dan Hendrycks and Thomas G Dietterich. Benchmarking neural network robustness to common corruptions and surface variations. arXiv preprint arXiv:1807.016697 2018.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint 1611v:1610.021366 2016.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint 18/2. 046666, 2018.
Ming Hou, Brahim Chaib-Draa, Chao Li, and Qibin Zhao. Generative adversarial positive-unlabeled learning. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp. 2255 2261, 2018
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 1 112--1134, 2017.
Ayush Jaiswal, Rex Yue Wu, Wael Abd-Almageed, and Prem Natarajan. Unsupervised adversarial invariance. In Advances in Neural Information Processing Systems, pp. 5092-5102, 2018.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations words and phrases and their compositionality. In Advances neural information pro- 2013.
cessing systems, pp.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv: TrXX::/802.0595: 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Xuanlong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. arXiv preprint arXiv:0809.0853, (11):5847- 5861, September 2008, doi: 10. 10.1109/TT.20000222685
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference Computer Vision, pp. 69-84. Springer, 2016.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems, pp. 271-279, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 107.00:807.03745 2018
Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A Alemi, and George Tucker. On varia- tional bounds of mutual information. arXiv preprint arXiv:1905.06922 2019a.
Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander Alemi, and George Tucker. On varia- tional bounds of mutual information. arXiv preprint 10550622, May 2019b.
10

---

Published as a conference paper at ICLR 2021
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. arXiv preprint arXi::1910.06222 October 2019.
Yi Lin Sung, Sung-Hsien Hsich, Soo-Chang Pei, and Chun-Shien Lu. Differenne--sekingg gener- ative adversarial network-unseen sample generation. In International Conference on Learning Representations, 2019.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. arXiv preprint arXiv: 024466, March 2015.
David H Wolpert and William G Macready, No free lunch theorems for optimization. IEEE trans- actions on evolutionary computation, 1(1):67-82, 1997.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.
Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceed- ings of the IEEE International Conference on Computer Vision, pp. 6023-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 1rXiv:1710.09411 2017.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. European confer- ence on computer vision, pp. 649-666. Springer, 2016.
Shengjia Zhao, Hongyu Ren, Arianna Yuan, Jiaming Song, Noah Goodman, and Stefano Ermon. Bias and generalization deep generative models: An empirical study. In Advances in Neural Information Processing Systems, pp. 10792-10801, 2018.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions pattern analysis and machine intelligence, 40(6); 2017.
A NUMEROSITY CONTAINMENT
Zhao al. (2018) systematically investigate generalization in deep generative models using two different datasets: (a) a toy dataset where there are & non-overlapping dots (with random color and location) in the image (see Figure 5a), and (b) the CLEVR dataset where ther are k objects (with random shape, color, location, and size) in the images (see Figure 5b). They train a GAN model (WGAN-GP Guirajani et al. (2017)) with (either) dataset and observe that the learned distribution does not produce the same number of objects as in the dataset it was trained on. The distribution of the numerosity in the generated images centered the numerosity from the dataset, with slight- bias towards over-estimation. For, example when trained on images with six dots, the generated images contain anywhere from two eight dots (see Figure 6a). The observation similar when trained on images with two CLEVR objects. The generated images contain anywhere from one to three dots (see Figure 6b).
In order to remove samples with numerosity different from the train dataset, we use such samples as negative data during training. For example, while training on images with six dots we use images with four, five and seven dots as negative data for the GAN. The resulting distribution of the nu- merosity in the generated images is constrained to six. We observe similar behaviour when training GAN with images containing two CLEVR objects positive data and images with one three objects as negative data.
B IMAGE TRANSFORMATIONS
Given an image of size H x W, the different image transformations that we used are described below.
11

---

Published as a conference paper at ICLR 2021
(a) Dots
(b) CLEVR
Figure 5: Toy Datasets used in Numerosity experiments.
renu
45
80
number greemitd
(a)
(b)
Figure 6: Left: Distribution number of dots. The arrows the number of dots the learning algorithm is trained on, and the solid line the distribution over the number of dots the model generates. Right: Distribution over number of CLEVR objects the model generates. Generating CLEVR is harder so we explore only one, but the behaviour with NDA is similar to dots.
Jigsaw-K (Noroozi & Favaro, 2016) We partition the image into grid of K K patches of size (H/K) x (W/K), indexed by [l...., K K]. Then we shuffle the image patches according to random permutation (different from the original order) produce the NDA image. Empirically, we find K = 2 to work the best for Jigsaw- NDA.
Stitching We stitch two equal-sized patches of two different images, either horizontally (H/2 x W) or vertically (H x W/2), chosen uniformly at random, to produce the NDA image.
Cutout Cutmix
We select a random patch the image with its height and width lying between one-third and one-half of the image height and width respectively. To construct NDA im- ages, this patch is replaced with the mean pixel value of the patch (like cutout (DeVries & Taylor, 2017) with the only difference that they use mroo-masking,, or the pixel values of another image at the same location (cutmix (Yun et al., 2019)).
Mixup-a NDA image is constructed from a linear interpolation between two images x and y (Zhang et al., 2017), 7E + (1 7)y; 7 Beta(a, is chosen so that the dis- tribution has high density at 0.5.
Other classes NDA images are sampled from other classes in the same dataset. See Appendix A.
C NDA FOR GANS
Theorem 1. Let P P(X) be any distribution over X with disjoint support than Paata i.e., such that supp(pdata) sUpp(P) Ø. Let Do X R be the set of all discriminators over X.
12

---

Published as a conference paper at ICLR 2021
f: IRz R be convex, semi-continuon function such that f(1) -. f be the convex conjugate
of f, its derivative, and Go be a distribution with sample space X. Then AA E (0, 11. we have:
arg min max GgEP(X) D
x)P, Ly D&) = arg min max L5(AGg + (1-x),, D&) = Pdata GoEP(Z) Do:
(4)
is the objective for f-GAN (Nowozin et al., 2016). However, the optimal discriminators are different for the two objectives:
where Lq Do) Em-paaca (Do(z)) -
arg max Ly(Go, D&) ('ppata/ Ga) X
(5)
arg max (AG + (1 Ds) = (PPastaa/ (AGo + (1-A)P) Do:K-
(6)
Proof. Let us use F(z), ((z) to denote the density functions of Pdata, P and Ge respectively (and P. P. Q for the respective distributions). First, from Lemma I in Nguyen et al. (2008), we have that
max Ly D&) ((P((IG) Da R
(9)
LyG (1 (1- D2) Dz) = (1- A)P) DEX
max
(10)
where D, refers to the f-divergence. Then, we have
Dj(PI1QQ+ + (1-A)P)
p(x)
(Aq(z) + +11 (1)=)) o (g(x))
+ (1 1)p(z))
[x M((1)) {Q(x)) (1 {)p(T)
p(x)
+11 - A)S(0)
2\ (/x ((x)) )g(x)) + - 2)p(T)
p(x)
+ +(1 -))(()
(11)
(5 / )g(x)) \a(T)) + - {)p(())
p(x)
-Aʃ
+ ()S(0)
X / (Ag(a) + +11 A)p(2)) - (1 A)p(z))
1
p(x)
=>
+ -))(()))
)g(x) + + - (11-1)p(x)
_ / ((1 ((1-A) ((1-)p(()) Ag(x)) + +1 A)p(k))
1
p(x)
-x
+ +
=>ʃ 1
+ +(1 ())(()
(12)
where we use the fact that f convex with Jensen's inequality in Eq.(11) and the fact that p(x)p(x)) = 0, Vo E X in (12) since P and P has disjoint support.
We also have
/ (Ap(z)+( + (1-1)((x)) Ap(x) (1 A)p(z)
p(x)
Dy(PI|AP + (1-11))
f (Ap(z))
p(x)
+ +(1
Ap(xx) + -))p()))))))))))))))))>))))
I (Ap(z))
p(z)
>p(2)+ +0)
+ +
(1) (1-)))((0)
Therefore, in order for the inequality in Equation 11 be equality, must have that q(x) p(x) for all x E X. Therefore. the generator distribution recovers the data distribution at the equlibrium posed by the NDA-GAN objective, which is also the case for the original GAN objective.
13

---

Published as a conference paper at ICLR 2021
Moreover, from Lemma 1 in Nguyen et al. (2008), we have that:
arg max Ly(Q, (Paata/ Q)
(13)
D
Therefore, by replacing Q with Go and (AGa (1 P)P,, we have:
arg max (G Dω) f (pdataa/ (Go)
(14)
D
arg max (AGr Dz:
A)P, Db) (V (AGo + (1 P)
(15)
which shows that the optimal discriminators are indeed different for the two objectives.
D NDA FOR CONTRASTIVE REPRESENTATION LEARNING
We describe the detailed statement of Theorem and proof as follows.
Theorem 3. For some distribution p over X' such that supp(p) n supp(pant) = Ø, and for any maximizer of the NDA-CPC objective
E arg max max Tcrc(ho) 9o)
he ge
the representations of negative samples are disjoint from that of positive samples for h; i.e., Vo E supp(Paasa), E supp(p),
supp(h(#)) supp(h(z)))
Proof. We use contradiction argument to establish the proof. For any representation mapping that maximizes the NDA-CPC objective,
arg max ICPC (ha, go)
he
9s
suppose that the positive and NDA samples share some support, i.e. Eæ e supp(paara) x supp(ø).
(sup(((z)))))) #
We always construct ' that shares the same representation with h for Paata but have disjoint rep- resentations for NDA samples; i.e., Yo E supp(pdata), æ E supp(p), the following two statements true:
1. h(x) = hi(x);
2. supp(h' (z)) supp(h((z)) Ø.
Our goal is to prove that:
max (crc(hl',ga 90) (hx
(16)
9s
96
which shows contradiction.
For ease of exposition, us allow zero values for the output g, and define 0/0 (in this case, g assigns zero to positive values, then the CPC objective becomes -o0, cannot be maximizer to the objective).
Let ğ E arg max [c go) be an optimal critic to the representation model hig We then define a following critic function:
((x,2) x)
if Ex E supp(paata) s.t. E supp(f((z))
g'(z, 2)
(17)
otherwise
14

---

Published as a conference paper at ICLR 2021
In other words, the critic assigns the same value for date-represnntatioon pairs over the support of Pdata and zero otherwise. From the assumption over h, Fa supp(paata), E supp(p), and E su)p(h())).
x supp(((()))
so (a. 2) can be sampled as positive pair and (ax, 2) > 0. Therefore,
max (hrc(h'ggo) 2 Tcrc(h', ,'))
(18)
(n -m) (xx, z)
E log
(plug definition for NDA-CPC)
(1(x, + 2 g1xx, 2n}
mm g' (x) Zk
(n m)) 2)
E
log
(existence of someğ (z, z)
(x. 2) +
j== (0 2j)
12
> m
max TCPC (1, 96) 9e
(Assumption that ý is optimal critic)
which proves the theorem via contradiction.
E WHAT DOES THE THEORY OVER GANS ENTAIL?
Our goal is to show that NDA GAN objectives are principled in the sense that with infinite compu- tation, data, and modeling capacity, NDA GAN will recover the same optimal generator as a regular GAN. In other words. under these assumptions, NDA will not bias the solution in an undesirable way. We note that the NDA GAN objective is as stable as regular GAN in practice since both meth- ods estimate lower bound to the divergence with the discriminator, and then minimize that lower bound w.r.t. the generator. The estimated divergences are slightly different, but they have the same minimizer (which is the ground truth data distribution). Intuitively, while GAN and NDA GAN will give the same solution asymptotically, NDA GAN might get there faster (with less data) because leverages a stronger prior over what the support should (not) be.
F PIX2PIX
Figure 7 highlights the qualitative improvements when apply the NDA method to Pix2Pix model (Isola et al., 2017).
Input
Ground Truth
Pix2Pix
NDA-Pix2Pix
Figure 7: Qualitative results on Cityscapes.
15

---

Published as a conference paper at ICLR 2021
G ANOMALY DETECTION
Here, we show the histogram difference in discriminator's output for clean and OOD samples in Figure 8. High difference values imply that the Jigsaw NDA is better at distinguishing OOD samples than the normal BigGAN.
gunuar
EijEAN
BiGGA
Icok
Jaza
30-
-i Dzsmpti
mi
to
16
Diserupt)
Dzamptt
(a) Gaussian Noise
(b) Speckle Noise
(c) JPEG Compression
Figure 8: Histogram of D(clean) D(corrupt) for 3 different corruptions.
H EFFECT OF HYPERPARAMETER ON UNCONDITIONAL IMAGE GENERATION
Here, we show the effect for unconditional image generation on CIFAR-10 dataset.
Table 8: Effect of the FID score for unconditional image generation CIFAR-10 using Jigsaw
as NDA.
A
1.0
0.75
0.5
0.25
0.15
FID
18.64
16.61
14.95
12.61
13.01
UNSUPERVISED LEARNING ON IMAGES
0.86
0.46
0.45
0.38
0.57
Q.43
0.41
0.36
0.54
0.41
0,50
0.57
Figure 9: Comparing the cosine distance of the representations learned with Jigsaw NDA and Moco- V2 (shaded blue), and original Moco-Vi V2 (white). With NDA, we project normal and its jigsaw image representations further away from each other than the one without NDA.
J DATASET PREPARATION FOR FID EVALUATION
For dataset preparation, we follow the the following procedures: (a) CIFAR-10 contains 60K 32x32 images with 10 labels, of which 50K used for training and 10K used for testing, (b)
16

---

Published as a conference paper at ICLR 2021
CIFAR- 100 contains 60K 32 x 32 images with 100 labels, out of which 50K are used for training and 10K are used for testing, (c) CelebA contains 162.770 train images and 19,962 test images (we resize the images to 64 64px), (d) STL-10 contains 100K (unlabeled) train images and 8K (labeled) test images (we resize the images to 32 x2x32px). In our experiments the FID calculated on the test dataset. In particular,
10K generated images vs. 10K test images for CIFAR-10, 10K 10K for CIFAR-100, 19,962 vs. 19,962 for CelebA, and 8K vs 8K for STL-10.
K HYPERPARAMETERS AND NETWORK ARCHITECTURE
Generative Modelingg We use the same network architecture in BigGAN Brock et al. (2018) for our experiments. The code used for our experiments is based over the author's PyTorch code. For CIFAR-IO, CIFAR-I00, and CelebA we train for 500 epochs whereas for STL-10 we train for 300 epochs. For all the datasets we use the following hyperparameters: batch-size 64, generator learning rate = 2e-4, discriminator learning rate 2e-4, discriminator update steps per generator update step= The best model was selected on the basis of FID scores on the test set (as explained above).
Momentum Contrastive Learning. We use the official PyTorch implementation for our experi- ments. For CIFAR-10 and CIFAR-100. we perform unsupervised pre-training for 1000 epochs and supervised training (linear classifier) for 100 epochs. For Imagenet-100) 100, we perform unsupervised pre-training for 200 epochs and supervised training (linear classifier) for 100 epochs. For CIFAR- 10 and CIF, we use the following hyperparameters during pre-training: batch-size = 256, learning-date 0.3, temperature 0.07, feature dimensionality 2048. For ImageNet-100 pre- training we have the following: batch-size = 128, learning-dan = 0.015, temperature = 0.2, feature dimensionality 128. During linear classification we use batch size of 256 for all the datasets and learning rate of 10 for CIFAR-10, CIFAR-100, whereas for ImageNet- 100 we use learning rate of 30.
Dense Predictive Coding. We use the same network architecture and Han et al. (2019) for our experiments and use the official PyTorch implementation. We perform self supervieed training JCF-101 for 200 epochs and supervised training (action classifier) for 200 epochs on both UCF-101 and HMDB51 datasets.
in DPC
L CODE
The code to reproduce our experiments is given here.
M IMPLEMENTATION DETAILS
For our experiment over GAN, we augment the batch of real samples with a negative augmentation of the same batch, and we treat the augmented images as fake images for the discriminator. Similarly, for the contrastive lcarning experiments, we consider negative augmentation of the query image batch negatives for that batch.
For all our experiments we used existing open-source models. For experiments over GAN, we use the open-source implementations of BigGAN and Pix2Pix models, and for contrastive learning, use the open-source implementation of the MoCo-v2 model and Dense Predictive Coding. Hence, did not explain in detail each of the models. Implementing NDA is quite simple to generate NDA samples from the images in a mini-batch which only takes several lines of code.
only need
N DOES THE GAIN OF NDA FOR REPRESENTATION LEARNING COME FROM THE FACT THAT MORE NEGATIVE SAMPLES ARE USED?
We perform the experiments number of negatives is around 65,536. With our approach, we use the augmented versions of images in the same batch as negative. We transform both the key and query images to create NDA samples.
MoCo-v2 which maintains queue of negative samples. The
17

---

Published as a conference paper at ICLR 2021
Thus, the number of negatives for approach is 65,536 2 (one NDA sample created using query image and other using key image), only 0.00003051666 times more than the original number of negatives samples in MoCo-v2. Thus experiments are comparable to the bascline MoCo-v2. In terms of computation, we need an additional forward pass in each batch to get the representations of the NDA samples. The normal MoCo-v2 requires 1.09 secs for entire forward computation, which includes forward pass through the network, momentum update of the key encoder and dot product between the positive and negative samples. With NDA, 1 forward computation requires 1.36 sees.
O WHAT HAPPENS WHEN NEGATIVE DATA AUGMENTATIONS ARE NOISY?
Regarding the performance of negative data augmentation, perform 2 different experiments:
a) When the noise low When using jigsaw as our NDA strategy with a 2 x 2 grid, one out of the 24 permutations will be the original image. We find that when this special permutation is not removed, or there is 4% "noise", the FID score is 12.61, but when it is removed the FID score is 12.59. So, we find that when the noise is low, the performance our approach is not greatly affected and is robust in such scenarios.
b) When the noise is large- We use random vertical flipping as our NDA strategy, where with 50% probability the image is vertically flipped during NDA. In this case, the "noise" is large, as 50% of the time, the negative sample is actually the original image. We contrast this with the "noise- free" NDA strategy where the NDA image is always vertically flipped, We find that for the random vertical flipping NDA, the FID score of BigGAN is 15.84, whereas, with vertical flipping NDA, the FID score of BigGAN is 14.74. So performance degrades with larger amounts of noise.
18


Reviews:
[{"review_id": "k-TfifrIaM9", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "-- POST REBUTTAL --\n\nThe authors addressed well most of my concerns.  I increase my rating. However, the authors need to address all comments of the reviewers and also discuss all missing related works in the updated version.\n\n– Summary – \n\nThe paper proposes a new method of leveraging the negative samples (out-of-distribution samples purposely generated from the training data distribution) in generative modeling and representation learning. The main idea aims to leverage the inductive bias of negative samples to constrain the learning of the model, e.g., these negative samples may tell some more information about the support of the data. The experimental results suggest that using these negative samples in GANs (studies with BigGAN model for conditional/unconditional image generation) and contrastive representation learning (study with CPC (Oord et al., 2018) on unsupervised learning on images and videos ) improves the performance of baselines. The paper also reports on improvements in image-image translation and anomaly detection. The paper also provides theorems to prove the convergence of the proposed model on GANs and CPC. \n\nOverall, the paper is easy to read and the idea makes sense. However, I'm a bit concerned about the theory, the significance of improvements and the fairness of the comparison. The paper also misses to discuss and compare with recent works also on data augmentation for GANs.\n\n -- Strength -- \n\nS1 – The usage of negative examples, which obtained from some prior knowledge, to provide the evidence of the learning model about the support/geometry of data distribution sounds reasonable. It has been applied in (Sung et al. (2019)) in semi-supervised learning. The proposed method applies to new applications of generative and representation learnings. \n\nS2 – The experimental results are quite extensive in regards to the applications, and the improvements on GANs quite significant with jigsaw augmentation.\n\n -- Weakness --\n\nW1 – The paper does not provide the very detailed implementations of proposed models, which is a bit difficult to justify the correctness.\n\n*Generative learning*\n\nW2 – The detail of how to incorporate NDA into GAN is not clear. Also, the PDA baseline for GAN is not detailedly discussed. Does the PDA, NDA, and baseline BigGAN train with the same batch size? I guess that the PDA and NDA had more augmented samples, therefore batch size is larger than the bigGAN baseline?\n\nW3 – The paper does not discuss important related works [a,b,c] of Data Augmentation for GAN recently published. In these papers, they show transforming only real samples (if I understand correctly, it likely is similar to PDA of the proposed baseline) only to train GAN changing the target distribution therefore the generator will learn infrequent or out-of-distribution samples. However, if both real/fake are transformed, data augmentation is helpful in training GAN. Can the authors compare the proposed NDA to at least one of them with the same GAN model?\n\n[a] Differentiable Augmentation for Data-Efficient GAN Training\n\n[b] On Data Augmentation for GAN Training\n\n[c] Training Generative Adversarial Networks with Limited Data\n\n\nW4 – Eq. 10 showed that $L_f(\\lambda * G_{\\theta} + (1 - \\lambda) * \\overline{P}, D_{\\phi}) <= D_f(P||\\lambda*Q+(1-\\lambda * \\overline{P}))$, but then infers the lower bound: $D_f(P||\\lambda*Q+(1-\\lambda * \\overline{P})) >= \\lambda * f(\\frac{1}{\\lambda} + (1 - \\lambda) f(0)) = D_f(P||\\lambda*P+(1-\\lambda * \\overline{P}))$. Therefore, theoretically I am concerned a bit about the convergence of the model. I wonder whether the authors need an upper bound instead of the lower-bound in this case?\n\nW5 – The paper claimed: “Random Horizontal Flip is not effective as an NDA; this is because flipping does not spatially corrupt the image but is rather a semantic preserving transformation”. How about the Random Vertical Flip only? Can it improve the model since this augmentation looks very good to tell us about the boundary of the data? \n\n*Representation learning*\n\nW6 – The improvements in representation learning do not look significant to me, and the improvements are not too consistent on different datasets according to the type of augmentations.\n\nW7 – The lower bound of Eq. 18 looks just like due to adding a larger batch size for negative samples to train CPC. Can authors compare NDA to CPC with just the same batch size training as the NDA method? \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VIqLBw4ck5J", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method that uses artificial augmentation of data as Negative (aka OOD) samples to improve various computer vision tasks, including generation, unsupervised learning on images and videos.\n\nProns:\n- The paper is very well written.\n- Experiments are comprehensive across different tasks\n- The usage of data augmentation seems interesting but with some questions (see below).\n- It designs losses for both GANs and contrastive representation learning.\n- Code is provided.\n\nCons:\n- Augmentation has been proven in GANs to provide benefits through consistency training  (e.g. CR-GAN, ICLR2020,  Image Augmentations for GAN Training). These samples are used as \"positive\" samples that should generate consistent predictions. The most famous mixup is also treated as \"positive\" samples for training. So the augmentation usage here is a bit counterintuitive to me, because you show the opposite conclusion. Is that because only particular augmentation can be used as negative samples, e.g. Jiasaw? The answer to this question is critical. However, the paper does not mention/ study much.\n- Advanced self-supervised (contrastive) learning reply on strong augmentation, how negative samples can adapt to these methods? How do we categorize augmentation types used for general cases or NDA cases? Any insights on what kind of augmentations are useful for NDA. For example, in figure 9, the paper proposes to push samples and its jigsaw version away. However, these two pairs share strong local visual contents of objects (just like an image and its crop parts) that usually contrastive learning wants to pull them together. The proposed method tries to push them away. Any insights why it should work?\n- If justifications of these questions can be sufficient, I think it can be a strong paper. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TU9CP2fh5Ao", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates how augmenting the negative examples, not just the positive examples, can improve a variety of representation learning tasks. The paper investigates a number of different augmentations, and applies them to GANs and contrastive learning with images and videos.\n\nStrengths:\nA major strength of the paper is its simplicity. The method is fairly straightforward to implement into several approaches, and it obtains strong results on each approach evaluated in the paper. The approaches evaluated on GANs and contrastive learning with images and videos. \n\nAlthough the novelty of this method is limited, the paper does a good job at establishing some theoretical results to give intuition why the method works. In contrast to the number of advances in machine learning that lack intuition into why it works, this paper does a good job at offering some explanations and motivations for the approach. \n\nAlthough this paper focuses on images and videos, the same ideas could be extended to other modalities, such as text or audio, as well.\n\nThe experiments are convincing to show the generality of this idea. The experiments are on several different datasets. The experiments are supported by theoretical results, establishing intuition into why the method works. The introduction does a good job at establishing the difference to other data augmentation methods, in particular by using negative examples. \n\nThe paper is well written and easy to read. \n\nWeaknesses:\nIn some cases, the negative data augmentations may actually be inside the positive set. How would the approach scale with noise in the negative augmentations? \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0bTLiUex3Ha", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1) Summary\n- The authors proposed the negative data augmentation technique which is useful for generative adversarial networks, anomaly detection, self-supervised learning frameworks.\n- The idea is simple, and the technique was proven that it is powerful for several tasks.\n- They performed several experiments, and I think the experiments were enough to show the technique's superiority.\n\n2) Strong points\n- Good idea\n- Strong experimental results\n- Simple to use\n- Easy to understand\n\n3) Weak points\n- In Figure 3, they claim that in the absence of NDA. the support of a generative model learned from samples may \"over-generalize\"...\n- I am not sure that the sentence is true.\n\nThis paper is well written, and concrete, I recommend that this paper should be presented in ICLR 2021. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers find the proposed data augmentation approach simple, interesting and effective. They agree that paper does a good job exploring this idea with number of experiments. However the paper also suffers from some drawbacks, and reviewers raise questions about some of the conclusions of the paper - in particular how to designate an augmentation as either negative or positive is not clear apriori to training. While I agree with this criticism, I believe the paper overall explores an interesting direction and provides a good set of experiments than can be built on in  future works, and I suggest acceptance. I encourage authors to address all the reviewers concerns as per the feedback in the final version.
Decision: Accept (Poster)