Difference-Seeking Generative Adversarial Network--Unseen Sample Generation
Authors: Yi Lin Sung
Sung-Hsien Hsieh
Soo-Chang Pei
Chun-Shien Lu
Venue:  (2020)
Keywords: generative adversarial network
semi-supervised learning
novelty detection

Abstract: 
Unseen data, which are not samples from the distribution of training data and are difficult to collect, have exhibited importance in numerous applications, ({\em e.g.,} novelty detection, semi-supervised learning, and adversarial training).  In this paper, we introduce a general framework called  \textbf{d}ifference-\textbf{s}eeking \textbf{g}enerative \textbf{a}dversarial \textbf{n}etwork (DSGAN), to generate various types of unseen data. Its novelty is the consideration of the probability density of the unseen data distribution as the difference between two distributions $p_{\bar{d}}$ and $p_{d}$ whose samples are relatively easy to collect.

The DSGAN can learn the target distribution, $p_{t}$, (or the unseen data distribution)  from only the samples from the two distributions, $p_{d}$ and $p_{\bar{d}}$. In our scenario, $p_d$ is the distribution of the seen data, and $p_{\bar{d}}$ can be obtained from $p_{d}$ via simple operations, so that  we only need the samples of $p_{d}$ during the training. 
Two key applications, semi-supervised learning and novelty detection, are taken as case studies to illustrate that the DSGAN enables the production of various unseen data. We also provide theoretical analyses about the convergence of the DSGAN.



Published as a conference paper at ICLR 2020
DIFFERENCE-SEEKING GENERATIVE ADVERSARIAL NETWORK-UNSEEN SAMPLE GENERATION
Yi-Lin Sung
Sung-Hsien Hsieh
Graduate Institute of Communication Engineering National Taiwan University, Taiwan, ROC Institute of Information Science, Academia Sinica r069 02674207666 68ntu.edu.....
Institute of Information Science and Research Center for Information Technology Innovation, Academia Sinica, Taiwan, ROC parvaty wecettetttt thootmaall.......m.mmmm com
Soo-Chang Pei
Chun-Shien Lu
Graduate Institute of Communication Engineering National Taiwan University, Taiwan, ROC pei diiccentur .dd..t
Institute of Information Science and Research Center for Information Technology Innovation, Academia Sinica, Taiwan, ROC ccsêi is sinica cew.iicaeeddd.ee. .ed..t
ABSTRACT
Unseen data, which are not samples from the distribution of training data and are difficult to collect, have exhibited importance in numerous applications, (e.g., novelty detection, semi-supervised learning, and adversarial training). In this paper, we introduce a general framework called difference-seekingg generative adversarial network (DSGAN), to generate various types of unseen data. Its novelty is the consideration of the probability density of the unseen data distribution as the difference between two distributions py and Pa whose samples relatively easy to collect. The DSGAN can learn the target distribution, pt, (or the unseen data distribution) from only the samples from the distributions, Pa and Pa- In scenario, is the distribution of the seen data, and Pa can be obtained from via simple operations, so that we only need the samples of pa during the training. Two key applications, semi-supervised learning and novelty detection, are taken as case studies to illustrate that the DSGAN enables the production of various unseen data. We also provide theoretical analyses about the convergence of the DSGAN.
INTRODUCTION
Unseen data are not samples from the distribution of the training data and are difficult to collect. It has been demonstrated that unseen samples can be applied to several applications. [Dai et al. (2017)
proposed how to create complement data, and theoretically showed that complement data, considered as unseen data, could improve suni-supervised learning. In novelty detection, Yu et al. (2017)
proposed a method to generate unseen data and used them to train anomaly detector. Another related area adversarial training Goodfellow et al. (2013), where classifiers are trained to resist adversarial examples, which are unseen during the training phase. However, the aforementioned methods only focus on producing specific types of unseen data, instead of enabling the generation of general types of unseen data.
In this paper, we propose a general framework called difference seeernncee-eeking generative adversarial network (DSGAN), to generate variety of unseen data. The DSGAN is generative approach. Traditionally, generative approaches, which are usually conducted in an unsupervised learning manner, developed for learning the data distribution from its samples, from which subsequently. they produce novel and high-dimensional samples, such as the synthesized image [Saito et al. (2018) A state-of-the-art approach is the so-called generative adversarial network (GAN) Goodfellow al. (2014). GAN produces sharp images based on game-theoretic framework, but it can be difficult and unstable to train owing multiple interaction losses. Specifically, GAN consists of two functions: generator and discriminator. Both functions are represented parameterized neural networks. The discriminator network trained to determine whether the inputs belong to the real dataset or fake dataset created by the generator. The generator learns to map sample from latent space some distribution to increase the classification errors of the discriminator.
In traditional machine learning scenarios, "unseen" data corresponds data that is not used or seen during the training stage but rather the testing stage. The distribution of "unseen" " data could be as different.

---

Published as a conference paper at ICLR 2020
Nevertheless, if generator can learn to create unseen data, then traditional GAN requires numerous training samples of unseen classes for training, leading to a contradiction with the definition of the unseen data. This fact motivates to present the DSGAN, which can generate unseen data by adopting seen data as training samples (see Fig. 9 which illustrates the difference between GAN and the DSGAN, in Appendix [A]. The key concept to consider the distribution of the unseen data as the difference between two distributions that are relatively easy to obtain. For example, the out-of-distriutioon examples in the MNIST dataset, from another perspective, are found belong to the differences between the sets of examples in MNIST and the universal set. It should be noted that in traditional GAN, the target distribution is identical the training data distribution; however, the DSGAN these two distributions, are considered to be different.
This paper makes the following contributions:
We propose the DSGAN to generate any unseen data only if the density of the target (unseen data) distribution is the difference between those of any two distributions, Pa and pd-
(2) We show that the DSGAN possesses the flexibility to learn different target (unseen data) distributions in two key applications, semi-supervised learning and novelty detection. Specif- ically, for novelty detection, the DSGAN can produce boundary points around the seen data because this type of unseen data is easily misclassified. For semi-supervised learning, the unseen data are linear combinations of any labeled data and unlabeled data, excluding the labeled and unlabeled data themselve
(3) The DSGAN yields results comparable to semi-supervised learning but with a short training time and low memory consumption. In novelty detection, combining both the DSGAN and variational auto-encoder (VAE, Kingma & Welling (2014b)) methods achieve the state-of-ther the-art art results.
PROPOSED METHOD-DSGAN
2.1 FORMULATION
We denote the generator distribution as Py and training data distribution as Pa, both in an N- dimensional space. Let pa be the distribution decided by the user. For example, pa can be the convolution of pa and normal distribution. Let p. be the target distribution that the user is interested in, and it can be expressed
()=())((2) + aps(x) pa(z))
(1)
where a [0, 1]. Our method, the DSGAN, aims to learn Py such that Pg = Pt- Note that if the
support set of Pa belongs to that of Pa- then there exists at least an a such that the equality in (T)
holds. However, even if the equality does not hold, intuitively, the DSGAN attempts learn Pg such
(x( opa(I)
that (x)
with the constraint, Py 2 0. Specifically, the generator will output
samples located in the high-density areas of pa - opa- Furthermore, show that the DSGAN can leam Py, whose support set is the difference between those of pa and pa in Theorem [
First, we formulate the generator and discriminator in GANs. The inputs, 2, of the generator drawn from P= (2) in an M-dimensional space. The generator function, G(z; 02) RM - R& represents a mapping to the data space, where G is differentiable function with parameter 0g- The discriminator is defined as D (æ;; fa)) RNN > Jo, 1), which outputs single scalar. D (x)) can be considered as the probability that x belongs to class of the real data.
Similar to traditional GAN, we train D to distinguish the real data from the fake data sampled from G. Concurrently, G is trained to produce realistic data that mislead D. However, in the DSGAN, the definitions of "real data" and "fake data" are different from those in traditional GAN. The samples from Pa are considered as real, but those from the mixture distribution between Pa and P, are considered as fake. The objective function is defined as follows:
from the "seen" data, according applications. In this paper, focus the scenario that the two distributions different.
'The linear combination of any labeled data and unlabeled data probably belongs to the set of seen data (labeled data and unlabeled data), which contradicts the definition of unseen data. Thus, the samples generated by the DSGAN should not include the seen data themselves.
2

---

Published as a conference paper at ICLR 2020
V
[log [x=)] (1 a)) [log (1 (G) + alBe-py(s) [log (1 D
We optimize ( by min-max game between G and D. i.e.,
min max V (G, D)
G D
During the training procedure, iterative approach, like traditional GAN, is to alternate between steps of training and one step of training via backpropagation is used to update 0g and 0g. Thus, for each Pg. Pd, and Pa- samples required for computing the gradients, where m the number of samples in a minibatch. The training procedure is illustrated in Algorithm in Appendix [ The DSGAN suffers from the same drawbacks as traditional GAN, (e.g., mode collapse, overfitting, and strong discriminator) so that the generator [Arovs:Ayjyvvsy
practice, minibatch stochastic gradient descent
gradient vanishes. There are literature [Salimans al. (2016); (Arovvsyy & Bottou (2017): MMyyato et al.
(2018) focusing on dealing with the above problems, and such concepts can be readily combined with the DSGAN.
Li et al. 2017) and Reed et al. (2016) proposed an objective function similar to ( Their goal was learn the conditional distribution training data. However, we aim learn the target distribution, Ps. in Eq. (. and not the training data distribution.
2.2
CASE STUDY ON VARIOUS UNSEEN DATA GENERATION
To achieve more intuitive understanding about the DSGAN, conduct several studies two-dimensional (2D) synthetic datasets and MNIST. In Eq. (T) a 0.8 is used.
3ra
Figure 1: Complement points (in Green) between two circles (in Orange).
Figure 2: Boundary points (in Green) between four circles (in Orange).
Nocnal dx
7
7
7
7 T 7 9
7
7
7
7
7
7
7
9
onytr)
(1-olry (X)
7
7
7
7
3
7 7
7 7
7
7
7
7
7
7 7
7
7
7
7 7
7
7
7
7
7
7
7
7
7
7
7
7
7
7 7
Figure 4: Illustraioon of the difference-set seek- ing in MNIST.
7
7
7
7
17
Figure 3: Illustration of the generation the un- seen data in the boundary around the training data. First, the convolution of and normal distribu- tion ensure the density on the boundary longer zero. Second, we seek Pg such that Eq. (I) holds, where the support set Pg is approxi- mated by the difference of those between Pæ and pd-
7
7
7
7
7
7
9
7
7
7
7
7
7
Figure 5: DSGAN learns the difference be- tween two sets.
no
Complement samples generation Fig. illustrates that the DSGAN can generate complement samples between circles. Denoting the density function of the two circles as Pd, we assign the samples drawn from P linear combinations of the two circles. Then, by applying the DSGAN, we achieve our goal of generating complement samples. In fact, this type of unseen data is used in semi-supervised learning.
3

---

Published as a conference paper at ICLR 2020
Boundary samples generation Fig. illustrates that the DSGAN generates boundary points be- tween four circles. This type of unseen data is used in novelty detection. In this case, we assign Pd and Pa "the density function of four circles" and "the convolution of Pa and normal distribution." respectively. The basis of our concept is also illustrated by one-dimensional (ID) example Fig. B
Difference-set generation We also validate the DSGAN on a high-dimensional dataset such as MNIST. In this example, define Pa the distribution of digit "I" and Pa the distribution containing two digits "1" and "7" Because the density, pa(x), is high when x is digit "1," the generator prone to output digit "7" with high probability. More sample qualities of DSGAN on CelebA can be refer to Appendix G
From the above results, we can observe two properties of the generator distribution, pg: i) the higher the density pa(x)), the lower the density of py (x)); ii) Pg prefers to output samples from the high-density areas of Pa(x) opp(2).
2.3 DESIGNING Pa
Thus far, we have demonstrated how the DSGAN can produce various types of unseen data by choosing specific Pa- In this section, we introduce a standard procedure to design Pa. and illustrate each step with pictures.
Step 1. First, the training data, Pa, are collected (Fig. (a)).
Step 2. Second, based on the applications, the desired unseen data distribution is defined (e.g., complement samples for semi-superee cuni-supereised learning) (Fig. (b)).
Step 3. Third, Pa is defined a mixed distribution of (1 a)p, + (a)pa (Fig. (c)).
Step 4. Finally, suitable mapping function that transform Pa to Pa is designed (e.g, linear combination of any two samples of pa)
noi
conorate
Desize
(a)
(b)
(c)
Figure 6: Illustration for designing Pa-
In the above procedure, the most important step is determine which types of unseen data suitable for specific problem (Step 2). In this paper. we show two types of unseen data, which are useful in semi sumi-supereiee learning and novelty detection. However, determining all types of unseen data for all applications is beyond the scope of this study, and we leave this for future work.
Furthermore, we provide method (see Appendix B supplementary materials) by reformulating the objective function (2), so that it is more stable to train the DSGAN.
3 THEORETICAL RESULTS
In this section, we show that by choosing an appropriate a, the support set of Py belongs to the difference set between Pa and Ps. that the samples from Py unseen from the ps perspective.
We start our proofs from two assumptions. First, in non-parametric setting, we assume that both the generator and discriminator have infinite capacities. Second, Py is defined as the distribution of the samples drawn from G(z) under 2 ~ p.-
In the following, show that the support set of Pg is contained within the differences in the support sets of р and pa while achieving the global minimum such that we can generate the desired P. by designing appropriate Pa-
4

---

Published as a conference paper at ICLR 2020
Theorem 1. Suppose a<p((x) Pa(z) for all x Supp(pa) and all density functions Pa(z), and Pa(x) and Py (z) are continuous. If the global minimum of (G) is achieved, then
Supp (Py) _ Supp (pa) - Supp(pa),
where
C(G) = max
max V(G,
(r(
(1 - + +
= =Bxmpz(x)
log
+ (x)
log
pā(z) + (1 a)); app((t)
pa(œ) + (1 a)p; apa(x)
Proof. See Appendix C for the details.
Summarizing, the generator prone to output samples that are located in the high-density areas of Opd-
Pa
4 APPLICATIONS
The DSGAN was applied to two problems: semi-supervised learning and novelty detection. In the semi-supervised learning, the DSGAN acts "bad generator," which creates complement samples (unseen data) in the feature space of the training data. For the novelty detection, the DSGAN generates the samples (unseen data) as boundary points around the training data.
4.1 SEMI-SUPERVISED LEARNING
Semi-supervised learning (SSL) is a type of learning model that uses a few labeled data and numerous unlabeled data. The existing SSL methods based generative model, (e.g.. VAE Kingma et al.) (2014) and GAN Salimans al. (2016). yield good empirical results. Dai al. (2017) theoretically showed that a good semr-superviser learning required a bad GAN with the following objective function:
max Fa.y~e log PD (y x, S K) Easpa(t) log PD (y K x)) (a log PD (K + 11)), (3) D
where (v. y) denotes a pair of data, and its corresponding label, 21,2,.... K} denotes the label space for the classification, and L [(x, y)} is the label dataset. Moreover, under the semi-supervised settings, Pa in (3) is the distribution of the unlabeled data. Note that the discriminator, D, in GAN also plays the role of classifier. If the generator distribution exactly matches the real data distribution (i.e., Pg = Pa). then the classifier trained by the objective function (3) with the unlabeled data cannot have a better performance than that trained by the supervised learning with the objective function. Specifically,
max Eu log PD (y x,yS K). D Contrastingly, the generator is preferred to generate complement samples, which lic the low- density area of pa- Under some mild assumptions, these complement samples help D to learn the correct decision boundaries in the low-density area because the probabilities of the true classes forced to be low in the out-of-distribution areas.
The complement samples in Dai al. (201)) are complex produce. In Sec. 5.2 we will demonstrate that with the DSGAN, complement samples can be easily generated.
4.2 NOVELTY DETECTION
Novelty detection determines if query example belongs to a seen class. If the samples of one seen class are considered as positive data, then this difficulty the absence of negative data in the training phase, so that the supervised learning cannot function.
Recently, novelty detection has made significant progress with the advent of deep leaning. Pidhorskyi et al.) Sakuraa & Yairi (2014) focused on learning representative latent space for a seen class. When testing, the query image was projected onto the learned latent space. Then, the difference between the query image and its inverse image (reconstruction) was measured. Thus, only an encoder was nceded to be trained for the projection and decoder for the reconstruction. Under the circumstance, an autocncoder (AE) is generally is adopted to learn both the encoder and decoder Pidhorskyi et al. 20018 Perera et al. (2019). Let Enc(-) be the encoder and Dec(-) be the decoder. The loss function of the AE is defined
min Ene,
Ex-ppar [||T Dec(Enc(z))113)
(5)
5

---

Published as a conference paper at ICLR 2020
where Ppos is the distribution of seen class. After the training, query example, Itests is classified as the seen class if
@Feest Dec((nc((test)) )) T.
(6)
where 7 E IR+ plays the trade-off between the true positive rate and false positive rate. However, (6) is based on two assumptions: (1) the positive samples from one seen class should have a small reconstruction error; (2) the AE (or latent space) cannot well describe the negative examples from the unseen classes, leading to a relatively large reconstruction error. In general, the first assumption inherently holds when both the testing and training data originate from the same seen class. However, Pidhorskyi et al. (2018 Perera et al. (2019) observed that assumption (2) does not hold at all times because the loss function in (5) does not include loss term to enforce the negative data to have large reconstruction error.
For assumption (2) to hold, given positive data the training inputs, we propose using the DSGAN to generate negative examples in the latent space, as discussed Sec. S.3. Then, the loss function of the AE is modified to enforce the negative data to have large reconstruction error.
EXPERIMENTS
Our experiments are divided into three parts. The first one examines how the hyperparameter, a, influences the learned generator distribution, Pg. In the second and third experiments, we obtain empirical results about supervipe learning and novelty detection, which are presented in Sec. 5.2 and Sec. 5.3) respectively. Note that the training procedure of the DSGAN can be improved by other extensions of GANs such as WGAN Arjovsky et al. (2117), WGAN-GP Guuraaan et al. (2017), EBGAN /Zhao et al. (2017), and LSGAN Mao et al. (2017). In our method, the WGAN-GP was adopted for the stability of the DSGAN in training and reduction in the mode collapse.
5.1 DSGAN WITH DIFFERENT a
The impacts of different a values on the DSGAN are illustrated in Fig. 7 In this example, the support of pa is the area bounded by a red dotted line, and the orange points are the samples from Pa. Concurrently, we shift to the right by I unit and create the distribution, Pa. whose support is bounded by blue dotted lines. The overlapping area between Pa and pa is 0.5 unit (assuming the area of is 1 unit). Based on our theoretical results, = 0.5 is the smallest selected value allowing Pa to be disjoint to Pa- Therefore, we can see that some generated samples, as presented in Fig. [a). still belong to the support set of pa- Fig. (b) shows that there is perfect agreement between our 0.5. When a 0.8, there is remarkable gap between
theoretical and experiment results with the generated (green) points and yellow points, as shown in Fig. (... In theory, the result obtained at a = 0.8 should be the same as that obtained at a = 0.5. This is because the discriminator should assign the entire area, which is the intersection of the complement of support set of pa and support set of Pa. to the same score, under the assumption that the discriminator has an infinite capacity. However, in practice, the capacity of the discriminator is limited. Therefore, the score of the area near pa is lower than that far from it, when is large. Therefore, Pg tends to repel pa to achieve a high score (to deceive the discriminator).
5.2 DSGAN IN SEMI-SUPERVISED LEARNING
We first introduce how the DSGAN generates the complement samples in the feature space. Dai al. (2017) proved that if the complement samples generated by G could satisfy the following two assumptions in 77 and 8 i.e.
Vx Pg (z)
W f(x and Vx pa(x), < max w f(x), 1SiSK
max
(7)
1sisk
where / is the feature extractor and w; is the linear classifier for the 7th class, and
Vzi ~ L, 12 ~ pa(x), Br, ~ (x(x) s.t. f(x))= = (f(x)) + (1 (x)())) with 3 E [0, 1],
(8)
then all the unlabeled data would be correctly classified by the objective function (3). Specifically. (7) ensures that the classifiers can discriminate the generated data from the unlabeled data, and causes the decision boundary to be located in the low-density areas of pa-

---

Published as a conference paper at ICLR 2020
0.30
(b) 0.50
(c) a 0.80
Figure 7: Influence of a on the synthetic dataset. We observe that the samples of Pa (green points) move farther away from Pa as a increases; however, they are still bounded by the support of pa- When is 0.5, the support set of Pg is disjoint to that of Pa. satisying the theoretical results. When 0.8, Py generates the rightmost points of P3- The level curves from the discriminator show that the generator is more prone to producing samples in region with higher score than in that with lower score. Note that the outputs of the discriminator are not restricted in [0, 1], because we use the formulation of the WGAN in this experiment.
The assumption in (8) implies that the complement samples must be in the space created by the linear combination of the labeled and unlabeled data. In addition. they cannot fall into the real data
distribution, Pa. owing to the assumption (7). To allow the DSGAN generate such samples, we let
Pa(x) -
the samples of Pa be linear combinations of those from L and Pa- Since Po (x) R
1 a
Pg will tend to match Pą. whereas the term, -Oppa ensures that the samples from Pa do not belong to
Pa- Thus, Py satisfies the assumption in (8) Moreover, (7) is also satisfied by training the classifier
with 33 based on substituting the generator distribution
into the learned Pg-
Following the previous works, we apply the proposed DSGAN to semi- cemiisupervised learning on three benchmark datasets: LNNIST et al. (1998), SVHN Netzer et al. (20IT). and CIFAR-10
Krizhevsky 2009). The details of the experiments can be found in Appendix D
5.2.1 SIMULATION RESULTS
First, the selected hyperparameters are listed in Table in Appendix D.I from the DSGAN and prate-of-the-att methods on the three benchmark datasets are summarized in Table T It can be observed that our method can compete with the state-of-the-ard methods on the three datasets. Note that report the results of badGAN not only from the original papers in the literature but also by reproducing them using the released codes of the authors. The reason of presenting both the results is that cannot reproduce parts of the results. The experiments in Li et al. (2019) also showed similar problem. In comparison with Dai et al. (201)). our methods do not need rely on additional density estimation network, PixeICNN++ (Salimans et al. (017). Although PixeICNN+ is one of the best density estimation networks, learning such a deep architecture requires large computation and high memory consumption. In Table we list the training time and memory consumption for our method and badGAN. Compared to badGAN, our method consumes 15.8% less training time and saves about 9000 MB during the training.
[.T] Second, the results obtained
Moreover, it can also be observed from Table [ that our results are comparable to the best record of badGAN and CAGAN. are better than those of other approaches on the MNIST and SVHN datasets. On CIFAR- 10, our method is only inferior to the CT-GAN. However, this might not be reasonable comparison because the CT-GAN uses extra techniques, including temporal ensembling and data augmentation, which the other methods do not use.
5.3 DSGAN IN NOVELTY DETECTION
In this section, we study how to use the DSGAN for assisting novelty detection. As mentioned in Sec. 4.2 we need to train the auto-encoder (AE) such that (i) the positive samples from one class have a small reconstruction error; (ii) negative samples from the unseen classes incur relatively higher reconstruction emors.
7

---

Published as a conference paper at ICLR 2020
Table Comparison of the semi-supervised learning in our DSGAN and the state-of-the-ar methods. For reasonable comparison, we only consider GAN-based methods. denotes the use of the same architecture of the classifier. i denotes larger architecture the classifier. & denotes the use of data augmentation (in (00. The results for MNIST are recorded as the number of errors, whereas for the others are as percentage of the error.
Methods
MNIST
SVHN
CIFAR-10
FM" Salimans et al.
2016
93 6.5
8.11 1.3
18.63 + 1.32
TripleGAN Li ct al.
2017
+1
5.77 0.17
16,99 + 0.36
badGAN Dai al. 2017)
7.85
4.25
14,41 + 0.30
CAGAN
Ni al.
2018)
81.91
4.83 0.09
12.61 = 0.12
CT-GAN Wei ct al.
2018)
89 + 13
9.98 + 0.21
badGAN-ree N-rprouuuee
86.2 + 13.2
4.48
0.16
16.25 + 0.33
Our method'
82.7
4.6
4.38
0.10
14.52 + 0.14
Table 2: Training times of our method and badGAN. We only report the training time on MNIST, on which the authors badGAN applied PixeICNN++. The experiments run on a NVIDIA 1080 Ti.
Methods
Training time
Memory Consumption
badGAN
38 s/ //epcch
9763 MB 711 MB
Our method"
32 /ppoch
The fundamental concept is to use the DSGAN to generate negative samples, which originally do exist under the scenario of novelty detection. Next, we add a new loss term to penalize the small reconstruction errors of the negative samples (see the third stage below). Three stages are required to train our model (AF):
1. The encoder, E.n(-)), and decoder, Dec(-), trained using the loss function
2. Given Ppos. Enc(x) are collected as the samples drawn from Pa- Pa is the convolution pa having normal distribution with a zero mean and variance a. Then, we train the DSGAN generate negative samples, which are drawn from Pá(z)) Pa(z) and the boundary points around the positive samples in the latent space. Note that there are some variations in the DSGAN: the input of the generator, G, is Enc(x), instead of random vector in the latent space. We also add [En((r) - ((Enc(x)))I which will be explained in the next step, train the generator.
3. Fixing the encoder, retrain the decoder by the modified loss function.
min [m=ppo<(x) [|| Dec(Enc(z)) + w max (0, m - || Dec((((En((z))))) Dec
where wo is the trade-off between the reconstruction errors of positive samples Enc(a) and negative samples G ((En(zz)). Note that in the previous step, we add Enc(x) G G(Enc(z))11 to ensure that the outputs of the generator are around the input. Thus, the second term charges even though the negative samples are close the corresponding positive sample, and they still exhibit high reconstruction error, which is bounded by (Zhao et al. (2017).
The above algorithm, called VAE+DSGAN, can be used to strengthen the existing AE-based methods by using them in the first stage. In the simulation, we used a variational autoencoder (VAE) Kingma & Welling 22014a because performs better than the AE in the novelty detection.
5.3.1 SIMULATION RESULTS
In this section, following Perera et al. (019)), the performance was evaluated using the area under the curve (AUC) of the receiver operating characteristics (ROC) curve. Given dataset, class was chosen as the seen class for training, and all the classes were used for testing. There exist several testing benchmarks for novelty detection, such MNIST, COIL et al. (1996) and CIFAR- 10. The state-of ofate-of-thh-nn art method Perera et al. (2019) achieves high performance in AUC on MNIST and COIL100 (AUC is larger than 0.97). However, for CIFAR-10 Perera al. 2019) only
8

---

Published as a conference paper at ICLR 2020
Original images
VAE
Ours (VAE DSGAN)
Figure 8: Comparison of the reconstructed results of the VAE and our method. The seen class, which at the bottom of the images, is a car. Other rows are images from the unseen classes. Our method exhibits a relatively larger gap, terms of the reconstruction error between the seen data and unseen data, than the VAE.
Table Comparison of our method (VAE+DSGAN) and the state-of-the-art methods: VAE Kingma & Welling (2014a), AND Abati et al. (2019), DSVDD Ruff al. (2018), and CCCGANPeereti et al. (2019). The results for CIFAR-10 are recorded terms the AUC value. The number in the top row denotes the seen class, where O: Plain, 1: Car, 2: Bird, 3: Cat, 4: Deer, 5: Dog, 6: Frog,7: Horse, 8:
Ship, 9: Truck.
0
2
3
4
5
6
7
8
MEAN
VAE
.700
.386
.679
.535
.748
.523
.687
.493
.696
.386
.583
AND
.735
.580
.690
.542
.761
.546
.751
.535
.717
.548
.641
DSVDD
.617
.659
.508
.591
.609
.657
.677
.673
.759
.731
.648
OCGAN
.757
.531
.640
.620
.723
.620
.723
.575
.820
.554
.657
Our method
.737
.614
.676
.644
.759
.562
.660
.646
.769
.633
.670
achieves 0.656. Thus, we chose the challenging dataset, CIFAR-10, as the benchmark to evaluate our method. The detailed network architecture can be found in Appendix E
Because VAE+DSGAN can be considered as fine tuning VAE Kingma & Welling (014a), we first illustrate the key difference between the VAE and VAE+DSGAN, as shown in Fig. g The seen class, which is at the bottom of the images, is a car. Other rows are the images from the unseen classes. One can see that the reconstructed images are reasonably good even for the unseen class in the VAE. By contrast, our method enforces the reconstructed images of the unseen classes to be blurred while still preserving the reconstruction quality the seen class. Thus, our method achieves a relatively larger gap, in terms of the reconstruction error between the seen data and unseen data, than the VAE.
In Table 3 we compare the proposed method with several methods, including the VAF Kingma Welling (2014a). AND Abati et al. (2019), DSVDD Ruff et al. (2018), and OCGAN Perera et al. (2019). in terms of the AUC value. One can see that in most cases, our method almost outperforms the VAE. Furthermore, the mean of the AUC values of our method also is larger than those of the state-of-the-art methods. is worth mentioning that in addition to the VAE, the DSGAN has potential of being combined with other AE-based methods.
RELATED WORKS ABOUT UNSEEN DATA GENERATION
Yu et al. (2017) proposed method to generate samples of unseen classes in unsupervised manner via an adversarial learning strategy. However, requires solving an optimization problem for each sample, which certainly leads to high computation cost. By contrast, the DSGAN has the capability to create infinite diverse unseen samples. Hou et al. (018)) presented a new GAN architecture that could leamn two distributions of unseen data from part of seen data and the unlabeled data. However, the unlabeled data must be a mixture of seen and unseen samples; the DSGAN does not require any unseen data. Kliger & Fleishman (2018) also applied GAN in novelty detection. Their objective

---

Published as a conference paper at ICLR 2020
was to learn generator whose distribution is mixture of novelty data distribution and training data distribution. To this end, they used feature matching (FM) to train the generator and expected Py to learn the mixture of distributions. However, the ultimate goal of FM is still to learn Py Pas therefore, their method might fail when GAN learns well.
Dai al. (017)) aimed generate complementary samples (or -iittii-ittiittiion samples), but assumed that the in-distribution could be estimated by a pre-trained model, such as PixelCNN++, which might be difficult and expensive train. [ee al. (2018) used simple classifier to replace the role of PixeICNN+ in Dai et al. 2017) so that the training was comparatively much easier and more suitable. Nevertheless, their method only focused on generating unseen data surrounding the low-density area of seen data. In comparison, the DSGAN has more flexibility to generate different types of unseen data (e.g,, a linear combination of seen data, as described in Sec. (.2).. In addition, their method needs the label information of the data, whereas our method is fully unsupervised.
7 CONCLUSIONS
We propose the DSGAN, which can produce any unseen data based the assumption that the density of the unseen data distribution is the difference between the densities of any two distributions. The DSGAN is useful an environment when the samples from the unseen data distribution are more difficult to collect than those from the two known distributions. Empirical and theoretical results are provided to validate the effectiveness of the DSGAN. Finally, because the DSGAN developed based on GAN, it is easy to apply any improved versions of GAN to the DSGAN.
8 ACKNOWLEDGEMENT
This work was partially supported by grants MOST 107-221-E-001001--MY2 and MOST 108-2634- F-007-010 from Ministry of Science and Technology, Taiwan, ROC.
REFERENCES
D. Abati, A. Porrello, S. Calderara, and R. Cucchiara. And: Autoregressive novelty detectors. In IEEE CVPR, 2019.
M. Arjovsky and L.. Bottou. Towards principled methods for training generative adversarial networks. In ICLR. 2017.
M. Arjovsky, S. Chintala, and L.. Bottou. Wasserstein generative adversarial networks. In ICML, volume 70. pp. 214-223, 2017.
Zihang Dai, Zhilin Yang, Fan Yang, William W Cohen, and Ruslan R Salakhutdinov. Good semi- supervised learning that requires a bad gan. In NIPS, pp. 6510-6520. 2017.
I. Goodfellow, Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, pp. 2672-2680. 2014.
Ishaan Gulrajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville. Improved training of wasserstein gans. In NIPS, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Berhaard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NIPS, 2017.
M. Hou, B. Chaib-draa, C. Li, and Q. Zhao. Generative adversarial positive-unlabelled learning. In 2018.
IJCAI, pp.
D. Kingma and Max Welling, Auto-encoding variational bayes. In ICLR. 2014a.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. ICLR, abs/1312.6114, 2014b.
Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In NIPS, pp. 3581-3589, 2014.
10

---

Published as a conference paper at ICLR 2020
Mark Kliger and Shachar Fleishman. Novelty detection with gan. ArXiv, abs/1802. 2018.
A. Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Y. LeCun, C. Cortes, and C. J. C. Burges. The mnist database of handwritten digits. 1998.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrated classifiers for detecting out-of-distribution samples. ICLR, 2018.
Chongxuan Li, Kun Xu, Jun Zhu, and Bo Zhang. Triple generative adversarial nets. In NIPS, 2017.
Wenyuan Li, Zichen Wang, Jiayun Li, Jennifer S Polson, William Speier, and Corey Conkling Arnold Semi-supervised learning based on generative adversarial network: comparison between good gan and bad gan approach. ArXiv, 0bs/1905.066e 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
X. Mao, Q. Li, H. Xie, R. Y. K. Lau, Z. Wang, and S. P. Smolley. Least squares generative adversarial networks. In IEEE ICCV, pp. 2813-2821, 2017.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. ICLR. 2018
Sameer A. Nene, Shree K. Nayar, and Hiroshi Murase, Columbia object image library (coil-20). 1996.
Y. Netzer, Т. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop, 2011.
Yao Ni, Dandan Song, Zhang, Hank Wu, and Lejian Liao. Cagan: Consistent adversarial training enhanced gans. In IJCAI, 2018.
Pramuditha Perera, Ramesh Nallapati, and Bing Xiang. OCGAN: one-class novelty detection using gans with constrained latent representations. In JEEE CVPR, 2019.
Stanislav Pidhorskyi, Ranya Almohsen, Donald A. Adjeroh, and Gianfranco Doretto. Generative probabilistic novelty detection with adversarial autoencoders. In NIPS, pp. 6823-6833 2018.
Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi-supervised learning with ladder networks. NIPS, 2015.
Scott E. Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text image synthesis. In ICML, 2016.
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In ICML, pp. 4393- 4402, 2018
Y. Saito, S. Takamichi, and H. Saruwatari. Statistical parametric speech synthesis incorporating generative adversarial networks. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(1):84 96, 2018.
Mayu Sakurada and Takehisa Yairi. Anomaly detection using autoencoders with nonlinear dimen- sionality reduction. In MLSDA, pp. 4-11, 2014.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. In NIPS, pp. 2234-2242. 2016.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. Pixelcnn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. In ICLR, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision, 2015.
11

---

Published as a conference paper at ICLR 2020
Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel Gonzalez-Garciaa and Bogdan Raducanu. Transferring gans: generating images from limited data. CoRR, abs/1805.01677, 2018.
URL http::/atxii.....mmm//// org. abs/ 1805 01677
Xiang Wei, Boqing Gong, Zixia Liu, Wei Lu, and Liqiang Wang. Improving the improved training of wasserstein gans: A consistency term and its dual effect. In ICLR, 2018.
Y. Yu, W.YY Qu, N. Li, and Z. Guo. Open-category classification by adversarial sample generation. In IJCAI, pp. 3357-3363, 2017.
J. Zhao, M. Mathicu, and Y. LeCun, Energy-based generative adversarial network. In ICLR, 2017.
12

---

10. end for Published as a conference paper at ICLR 2020
APPENDIX
A FLOWCHART AND ALGORITHM OF DSGAN
Soon data
Traditional GAN
Generator
seen data!
True Data
False
Data
Soon example
Discriminttor
Generated esampll (belong to Set 1)
Set I
via some operatione
DSGAN
Generator
Unseen data!
Troe Dota
False Data
Intermeditte exouuple
Discriminator
Geuerated exomplle (belong the difference of Set 2 and Set 1)
Set
Figure 9: Illustration of the differences between traditional GAN and DSGAN.
Algorithm The training procedure of DSGAN using minibatch stochastic gradient descent. i the number of steps applied to discriminator. is the ratio between Pg and in the mixture distribution. used & = and a 0.8 experiments.
01. for number of training iterations do for l: steps do Sample minibatch of m noise samples 2(1)
02.
03.
..... (m) from Py(2).
04.
Sample minibatch of m samples x. (1) (m)
d
from pa(x).
Sample minibatch of m samples x (1) x m)) from pz(x).
05.
06.
Update the discriminator by ascending its stochastic gradient:
Vo.
log
D
(1) Idd
+ log (1-D( (G ())))) + log (1- D
(1-D( (z. (2((()))
1=1
07.
end for
08.
Sample minibatch m noise samples 2(1) ..., (m) from (2).
09.
Update the generator by descending its stochastie gradient:
77
log 11 D ( (2(0)
8
m
13

---

Published as a conference paper at ICLR 2020
B TRICKS FOR STABLE TRAINING
We provide trick to stabilize the training procedure by reformulating the objective function. Specifi- cally, V(G, D) in reformulated as:
V (G) = Pa(z) log (D (x))
+ ((1 - (2) + app())) log (1 D (x:) dz
Exw((()) [log D(x)
+ (0 [log (1 - D (x))).
Instead of sampling a mini-batch of m samples from P= and Pa in Algorithm 1 (1 - a.mm and om samples from both distributions are required, respectively. The computation cost in training can be reduced due to fewer samples. Furthermore, although (9 is equivalent to (2) in theory, we find that the training using (9) achieves better performance than using (2) via empirical validation Table 4 We conjecture that the equivalence between ( and is based the linearity of expectation, but mini-batch stochastic gradient descent in practical training may lead to the different outcomes.
Table 4: Semi-supervised learning results on MNIST with and without the use sampling tricks.
Methods
MNIST (# errors)
Our method w/o tricks
91.0 + 7.0
Our method w/ tricks
82.7 + 4.6
C PROOF OF THEOREM 1
In this section, we show Theorem
This proof includes two parts: the first part shows that the objective function is equivalent to minimizing the Jensen-Shannon divergence in the mixture distribution (pa and p2) and Pa if G and D are assigned sufficient capacity: the second part shows that by choosing an appropriate a, the support set of Py belongs the difference between pa and pa, so that the samples from Py are unseen from the perspective.
For the first part, we show the optimal discriminator given G, and then show that minimizing V (G, D) via G, given the optimal discriminator, is equivalent to minimizing the Jensen- Shannon divergence between (1 a'),, + apa and På-
Proposition 1. If G is fixed, the optimal discriminator, D, is
Po(x)
(x) pa(x) + (1 a) (pp(xx)
14

---

Published as a conference paper at ICLR 2020
Proof. Given any generator G, the training criterion for the discriminator D is to maximize the quantity D):
/ Pa(a)) log (D de
V(G, D)
+ +1 +11-a) 00 p000ppp())))) (2) (1 dz
+ / pa(x))og (1 D (I)) de
Pa(z) log (x)) do
+11 (1--a) a) / pp(x) (1 D (x:) dz
+ / po(x)log (1 - D (x)) de
/ Pa(a) log (D
log (1 (o2t)
+ (1--a)pg(z) +
D dz.
For any (a, RO, 0}. the function a log (y) + b blog y) achieves its maximum in [0, 1] at a+6- The discriminator only needs to be defined within Supp(pa) Supp(Paa U (Ppp((gg) We complete this proof.
D can be considered to discriminate between samples from Pa and ((1- - a)P, (x) + apa(lr)). By replacing the optimal discriminator in D), we trivially
Moreover,
obtain
C(G) max D)
pa( Pa(x) + (1 (x))())))))))))))))))))))))))) + a<p(ax)
+ Beng (a) log
(1 ())p(()) + apa(+) Pa(x)) + (1 ()) + opa(l)
Bumpy(a) log
(10)
Actually, the results thus far yield the optimal solution of given G is fixed T Now, the next step is to determine the optimal G with Di fixed.
The global minimum of ((G) is achieved if and only if (1 - (x) + oop(ar) = Pā(x) for all Then, C(G) achieves the value, log 4.
Theorem 2.
Proof. We start from
I
1og(4)
Pa(x) + 2pa(x)
+ Exppp(x)
log
+ apd(
2 - a opa(z))
+ (x)
log
pa(x) + (1 - a) py (x) + ap)(
Pa Pa+ (1 a) Py + app
= =-log(4) KL
Pa
+ KL. (1 (1- (1) (I) + apd
Pj P + (1 a) + opd
log 2JSD (pa || (1 a0pp + apa).
where p²() = (1- (x) + (=). KL is the Kullback-Leid Leibler divergence and JSD is the Jensen-Shannon divergence. The JSD returns the minimal value, which is 0, iff both distributions are the same, namely Pa = (1 a)),, + ap.. Because (2))'s 's are always non-negative. should be noted both distributions are the same only if opa(æ) S pa(x)) for all w's. We complete this proof. [T
15

---

Published as a conference paper at ICLR 2020
Note that (1 -1-aa ())p(x)) + app(a)) Pd(x) may not hold if cap((a) Pa(a)- However, the DSGAN still works based two facts: i) given D, V(G, D) is convex function in Pa and ii) because
| Po (r)dd = 1, the set collecting all the feasible solutions of Pg is convex. Thus, there always exists a global minimum of V(G, D) given D, but it may not (4)(
We show that the support set of Pg is contained within the differences in the support sets of Pa and Pa while achieving the global minimum such that we can generate the desired P, by designing appropriate På.
Now, we back to prove Theorem 1
Proof. Recall that
C(G)= [ Pa(x)) log
Pa(œ)
Pa(x) + (1 a ag)))()))))))))))))))))))))))a + ape(a))
(1 a)), (x) + apa(a)
+ v² (x) log
dr
pa(x) + (1 (2) + opa(x))
S(pgi r)
(esSupp(ea)
#ESupp(pa) ((pgia)drr.
S(py x) used to simplify the notations inside the integral. For any x. S(pg; x)) in p, is non- increasing and S(pgi x) I 0 always holds. Specifically, S(pgix) is decreasing along the increase of (x) if pz(x) > 0: Spy z) attains the maximum value, zero, for any (x) pa(z) = Since
(g]ddd 1, the solution attaining the /
DSGAN aims minimize C(G) with the constraint
global minima must satisfy (x(x) = 0 if pa(x) = O; otherwise, there exists another solution with smaller value of C(G). Thus, Supp (p,) C. Supp (pz).
Furthermore, T(p;; x) @S(pgiz)
(1 a)p, (z) + apa(x)
log Pa(x) + (1 a)) + apa(x)) which is expected
8p, (x)
to be small possible minimize C(G). is increasing Pa (z) and converges 0. Then, show that T(p;;)) x) for x E Supp(pa) [Supp(ps) is always larger than that for x E Supp(ra) Supp(ra) for all Py- Specifically.
1. When E Supp(pa) Supp(pa). T(p35x) log & always holds due to the assumption of opa(x) 2 Pa(x).
2. When x E Supp(ra) - Supp(pa), T(pgia) < log for all (2(x''s satisfying (1 v))p,(r) S Pa(x).
Thus, the minimizer prefers (2) > O for x. E Supp(pa) Supp(pa) and (1 - a)),(x) S
Pā(a): We check whether there exists solution Py such that (1 a)p)(r) < Pā(a)) and
(d(d)dz= 1, implying Px(x) 0 for e Supp(pa) SSppp(pu). Based
#ESuppp(pa) SSppp(n)))
16

---

Published as a conference paper at ICLR 2020
on the following expression,
/
Pa(x)dr +
(xeSuppine) pa(x)dx= =1
EESupp(ps) -Suppo(aa)-Supppp)
pa(z)dr
(acppp((a))Suppppe
1
apa(x)dr
>#SSpp((Pa)
Pa(x)der 2 1
(ESupp(ea))
LESSupira)-Supptee Pa(x)do
SESupp(o;) (1- (z))),
the last inequality implies that there must exist a feasible solution. We complete this proof.
Another concern is the convergence of Algorithm 1
Proposition 2. The discriminator reaches its optimal value given G in Algorithm) 7 and Py is updated
by minimizing
Ez>pa(s) [log Dī(()] (x)
(a) [log (1 -1--10 D2 (x))).
If G and D have sufficient capacities, then Py converges to argmin (pa (1 a)p, + apa))
Proof. Consider VGG. D) Upg, D) as function of Pg. By the proof idea of Theorem 2 in Goodfellow et al. (2014). if f(x) = suP.c fa(x) and (x) is convex in x for every a, then Ø]a(z) OJ it argsup.e. (z). In other words, if supp V(G, D) is convex in Py, the subderivatives of supp V(G, D) includes the derivative of the function at the point, where the maximum is attained, implying the convergence with sufficiently small updates of Pg- We complete this proof.
D EXPERIMENTAL DETAILS FOR SEMI-SUPERVISED LEARNING
D.O.1 DATASETS: MNIST, SVHN, CIFAR-10
For evaluating the semi-supervised learning task, we used 60000/ 73257/ 50000 samples and 10000/ 26032/ 10000 samples from the MNIST/ SVHN/ CIFAR-10 datasets for the training and testing, respectively. Under the semi-supervised setting, we randomly chose 100/ 1000/ 4000 samples from the training samples, which the MNIST/ SVHN/ CIFAR-10 labeled datasets, and the amounts of the labeled data for all the classes are equal. Furthermore, our criterion to determine the hyperparameters is introduced in Appendix D.I] and the network architectures Appendix D.2 We performed testing with 10/ 5/ 5 runs on MNIST/ SVHN/ CIFAR- 10 based on the selected hyperparameters, and randomly selected the labeled dataset. The results were recorded as the mean and standard deviation of the number of errors from each run.
described in
D.1 HYPERPARAMETERS
The hyperparameters were chosen to make our generated samples consistent with the assumptions in (7) and (8). However, in practice, if we make all the samples produced by the generator following the assumption in 8 then the generated distribution is not close to the true distribution, even a large margin between them exists, which is not what we desire. So, our experiments, we make concession that the percentage of generated samples, which accords with the assumption, around 90%. To meet this objective, we tune the hyperparameters. Table shows our setting of hyperparameters, where 3 is defined in
17

---

Published as a conference paper at ICLR 2020
Table 5: Hyperparameters semi-supervised learning.
Hyperparameters MNIST
SVHN
CIFAR-10
0.8
0.8
0.5
0.3
0.1
0.1
D.2 ARCHITECTURE
In order to fairly compare with other methods, our generators and classifiers for MNIST, SVHN, and CIFAR-10 10 are same as in Salimans et al. (2016) Dnd et al. (2017). However, different from previous works that have only generator and discriminator, in the feature space, and its architecture is similar across all datasets with only the difference in the input dimensions. Following Dai et al. (2017), also define the feature space the input space of the output layer of discriminators.
design additional discriminator
Compared SVHN and CIFAR-10, MNIST is simple dataset it is only composed of fully connected layers. Batch normalization (BN) or weight normalization (WN) is used in every layer to stable training. Moreover, Gaussian noise is added before each layer in the classifier, as proposed in Rasmus et al. (201)). We find that the added Gaussian noise exhibits a positive effect for semi- supervised learning. The architecture is shown T
Table 7 and Table are models for SVHN and CIFAR-10, 10, respectively, and these models are almost the same except for some implicit differences, e.g., the number of convolutional filters and types of dropout. In these tables, given dropping rate, "Dropout" denotes normal dropout in that the elements of input tensor are randomly set to zero while Dropout2d a dropout only applied on the channels to randomly zero all the elements.
Table 6: Network architectures for semi-supervised learning on MNIST. (GN: Gaussian noise)
Generator G
Discriminator 1)
Classifier C
Input: E 100 from unif(0,) 1)
Input: 250 dimension feature
Input: 28 x 28 gray image
100 x 500 FC layer with BN
250 x 400 FC layer
GN,
Softplus
ReLU
784 x 1000 FC layer with WN ReLU
500 x 500 FC layer with BN
400 x 200 FC layer
GN, std = 0.5
Softplus
ReLU
1000 x 500 FC layer with WN, ReLU
500 x 784 FC layer with WN
200 100 FC layer
GN, std 0.55
Sigmoid
ReLU
FC layer with WN ReLU
100 x FC layer
GN, std = 0.5
250 250 FC layer with WN, ReLU
GN, std = 0.5
250 250 layer with WN, ReLU
250 x 10 FC layer with WN
Furthermore, the training procedure alternates between steps of optimizing and one step of optimizing G. We find that l in Algorithm [ is key role in the problem of mode collapse for different applications. For semi-supervised learning, we set for all datasets.
E EXPERIMENTAL DETAILS FOR NOVELTY DETECTION
The architecture GAN and VAE are depicted Table 9 and 10
respectively.
In the experiment, first trained the VAE for 500 epochs and then trained DSGAN for 500 epochs with m 1.3 and w 0.5. Third, we fixed the encoder and tuned the decoder with both positive and negative samples (generated by DSGAN) for 600 epochs.
18

---

Published as a conference paper at ICLR 2020
Table 7: The architectures of generator and discriminator for semi-supervised learning on SVHN and CIFAR-10. N was set to 128 and 192 for SVHN and CIFAR-10, respectively.
Generator G
Diseriminator D
Input: E R100 from unif(0, 1)
Input: N dimension feature
100 x 8192 FC layer with BN, ReLU
N x 400 FC layer, ReLU
Reshape to 4 x 4 x 512
400 x 200 FC layer, ReLU
x 5 conv. transpose 256 stride 2 with BN, ReLU
200 X 100 FC layer, ReL.U
x conv. transpose 128 stride 2 with BN, ReLU
100 FC layer
x conv. transpose stride = with WN, Tanh
Table 8: The architecture of classifiers for semi-supervised learning on SVHN and CIFAR-10. (GN: Gaussian noise; IReLU(lcak rate): LeakyReLU(laak rate))
Classifier SVHN
Classifier C CIFAR-10
Input: 32 x 32 RGB image
Input: 32 x 32 RGB image
GN, std = 0.05 Dropout2d, dropping rate 0.15
GN, std = 0.05
Dropout2d, dropping rate = 0.2
3 x 3 conv. 64 stridc= 1 with WN, IReLU(0.2)
3 3 cony. 96 stride with WN, IReLU(0.2)
3 x conv. 64 stride 1 with WN, IReLU(0.2)
3 3 conv. 96 stride with WN, IReLU((.2)
3 x conv. 64 stride 2 with WN, (ReLU(0.2)
3 x 3 conv. 96 stride with WN, IReLU(0.2)
Dropout2d, dropping rate 0.5
Dropout, dropping rate 0.5
3 x 3 conv. 128 stride = 1 with WN, IReLU(0.2)
3 x 3 cony. 192 stride 1 with WN IReLU(0.2)
3 x conv. 128 stride with WN (ReLU(0.2)
3 3 conv. 192 stride 1 with WN, IReLU(0.2)
x conv. 128 stride with WN, (0..)(0.2)
3 x conv. 192 stride with WN, (0e2) Dropout, dropping rate 0.5
Dropout2d, dropping rate 0.5
3 x 3 conv. 128 stride = 1 with WN, IRel.U(0.2)
3 x 3 conv. 192 stride = 1 with WN, IReL.((0.))
1 X 1 conv. 128 stride 1 with WN, IRel.U(0.2)
1 X 1 conv. 192 stride 1 with WN, IRel.((.2)
1 conv. 128 stride with WN, (Re2.
1 conv. 192 stride 1 with WN (0..)
Global average Pooling
Global average Pooling
128 x 10 FC layer with WN
192 x 10 FC layer with WN
Table 9: The architectures of generator and discriminator in DSGAN for novelty detection.
Generator G
Discriminator D
Input: 128 dimension feature
Input: 128 dimension feature
128 x 1024 FC layer with BN. ReLU
128 x 400 FC layer, ReLU
1024 x 512 FC layer with BN, ReLU
400 x 200 FC layer, ReLU
512 x 256 FC layer with BN, ReLU
200 X 100 FC layer, ReLU
256 x 128 FC layer
100 1 FC layer
F ABLATION STUDY ON DIFFERENT & VALUES FOR SEMI-SUPERVISED LEARNING
Fig. 7 shows how different a values influence DSGAN. The optimal a for DSGAN to generate "unseen" data depends Pa and pa- According Fig. 7 we can figure out that DSGAN is prone to generating unseen data under a larger a. Recall that Theorem illustrates a should be expected to be as large possible if both network G and D have infinite capacity. Though the networks never have the infinite capacity in real applications, a general rule is to pick large a and force the complement data to be far from Pa, which is similar the results in Sec. 5.1
19

---

Published as a conference paper at ICLR 2020
Table 10: The architectures of VAE for novelty detection.
Encoder
Decoder
5 x conv. 32 stride 2, with BN, (0..)
conv. transpose 128 stride with BN, (ReL
5 x 5 conv. 64 stride = 2, with BN, (0.2)
5 x 5 conv. transpose 64 stride 2 with BN, IReLU(0.2)
5 x 5 conv. 128 stride = 2, with BN, (Rel.
5 x 5 conv. transpose 32 stride 2 with BN, (Rel.U(0.2))
(For mean)
5 x 5 conv. transpose 3 stride = 2, Tanh
4 conv. 128 stride
(For std)
4 x 4 conv. 128 stride 1
Here, we conduct the experiments on different a under semi-supervised learning settings. From Sec. 4.1 and 5.2 badGAN already shows that, if the desired unseen data can be generated, then the classifier will put the correct decision boundary in the low-density area.
In Table [1 demonstrate the classification results
0.5 and 0.8, respectively. We can observe that the results generated at a = 0.8 is better than those generated at a = 0.5, meeting the above discussion. From empirical observations, DSGAN is prone to generating unseen data
= 0.8, leading better classifier.
Table 11: Ablation study of different a values for DSGAN in semi-supervised learning, where the result for MNIST is represented in terms of number of errors and the percentage of errors was used for other datasets.
Methods
MNIST
SVHN
CIFAR-10
DSGAN (a = 0.5)
91.5 1 5.6
4.59 1 0.15
14.52 1 0.14
DSGAN (a = 0.8)
82.7 = 4.6
4.38 + 0.10
14.47 = 0.15
G SAMPLE QUALITY OF DSGAN ON CELEBA
We show one more experiment CelebA (Liu et al. 2013)) ) to demonstrate DSGAN work well even for complicated images. In this experiment, we generate the color images of size 64 x 64. Similar our 1/7 experiments on the MNIST dataset, we let pa be the distribution of face images with glasses and without glasses and let pa be the distribution of images without glasses. We validate DSGAN with a 0.5 and a 0.8, respectively. For 0.5, we sample 10000 images with glasses and 10000 images without glasses from CelebA. When a is 0.8, we sample 40000 instead of 10000 images without glasses.
We also train GAN to verify the generated image quality of DSGAN. For fair comparison, GAN trained under two kinds of settings. The first one is that GAN is only trained with the images with glasses. Second, it is pretrained with all images, and is finetuned with the images with glasses, namely transferring GANs in Wang et al. (20T8). It should noted that transferring GAN uses the same amount of training data as DSGAN and serves stronger baseline than GAN under the first setting.
Fréchet Inception Distance (FID) (Heusel et al. (2017) is used to evaluate the quality of generated images. FID calculates the Wasserstein-2 distance between generated images and real images (images with glasses) in the feature space of Inception-v3 network (Szegedy et al. (2015)). We train both networks for 600 epochs, and use WGAN-GP as the backbone for both GAN and DSGAN. In addition, transferring GANs pretrained for 500 epochs, then being finetuned for 600 epochs.
Fig. 10 and Table 12 show generated images and FID) for all methods, respectively. We can see that our DSGAN can generate images with glasses from the given pa and pæ. and the FID of DSGAN are comparable to that of GAN, The experiment validates that DSGAN still works well to create complement data for complicate images.
20

---

Published as a conference paper at ICLR 2020
(a) Transferring GANs (pretrained with 20000 images and finetuned with 10000 samples with glasses)
(b) DSGAN (a 0.5)
(c) Transferring GANs (pretrained with 50000 images and finetuned with 10000 samples with glasses) Figure 10: Sampled generated images of GAN and DSGAN on CelebA.
(d) DSGAN (a 0.8)
Table 12: FIDs of GAN and DSGAN on CelebA. Smaller FID means that the generated distribution is closer to the distribution of images with glasses.
10000 samples
20000 samples
50000 samples
GAN
transferring GAN DSGAN 0.5)
transferring GAN DSGAN 0.8)
FID)
22.37
18.34
18.05
16.45
15.39
21


Reviews:
[{"review_id": "S1gKDV-0Fr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed DSGAN which learns to generate unseen data from seen data distribution p_d and its somehow “broad” version p_{\\hat d} (E.g., p_d convolved with Gaussian). The “unseen data” is the one that appears in p_{\\hat d} but not in p_d. DSGAN is trained to generate such data. In particular, it uses samples from p_d as fake data and samples from p_{\\hat d} as the real one. \n\nAlthough the idea seems to be interesting, the paper seems to be a bit incremental and is a simple application of existing GAN techniques. The paper shows two applications (semi-supervised learning and novelty detection) and it is not clear that the proposed method outperforms existing GAN methods in the classification accuracy in MNIST/SVHN/CIFAR10 (Table 1) and existing sampling methods (Table. 3). It seems that the sampled reconstruction results (Fig. 8) are not as good as VAE on CIFAR10. I would also expect more ablation studies about how to pick p_{\\had d}, which seems to be the key of this approach, in MNIST and CIFAR10. \n\nIn terms of writing, the paper is a bit confusing in terms of motivations and notations. \n\nOverall, the method looks incremental and experimental results are mixed on small datasets so I vote for rejection. Note that I am not an expert on GAN/VAE so I put low confidence here. \t", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJe5Adneor", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed the DSGAN model to generate unseen data. The intuition based on standard GAN is straightforward and makes sense. The paper is well written, especially the case studies illustrate the idea clearly. The designing of p_{\\bar{d}} also presented the limitation of this method. Two main discussed applications, semi-supervised learning and novelty detection are important in machine learning. In general, this is an interesting paper.\n\nHowever, my concern is about the experiments. As a generative model for unseen data, I would like to see the generated results, which is more convincing. Only the 1/7 examples of MNIST dataset are provided in case studies. I am wondering for more complicated images, how is the performance?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BygFHVOeqB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper provides an interesting application of GAN which can generate the outlier distribution of training data which forces generator to learn the distribution of the low probability density area of given data. To show the effectiveness of the method, the author intuitively shows how it works on 2-D points data as well as the reconstructed Mnist dataset. Additionally, this approach reaches a comparable performance on semi-supervised learning and novelty detection task.\n\nPaper Strengths\n\n1. The idea of this paper is novel, and the implementation of this method is easily interacted with any GAN model. Also, due to its concise structure compared to the existing method, it saves more computational memory and is time efficiency.\n\nPaper Weaknesses\n\n1. Experimental settings are clear, however, what makes me confused is that the construction for p_{\\bar{d}} is straightforward for simple distribution like 2D points dataset, however, it might be intractable for complex high dimensional data such as images. \n2. The model seems to be sensitive to the hyper-parameter \\alpha, is this parameter always fixed at 0.5 or needed to fine-tune for different datasets?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a way to generate unseen examples in GANs by learning the difference of two distributions for which we have access. The majority of reviewers agree on the originality and practicality of the idea.
Decision: Accept (Poster)