Title: DIFFERENCE-SEEKING GENERATIVE ADVERSARIAL NETWORK-UNSEEN SAMPLE GENERATION

Abstract: Unseen data, which are not samples from the distribution of training data and are difficult to collect, have exhibited importance in numerous applications, (e.g., novelty detection, semi-supervised learning, and adversarial training). In this paper, we introduce a general framework called difference-seeking generative adversarial network (DSGAN), to generate various types of unseen data. Its novelty is the consideration of the probability density of the unseen data distribution as the difference between two distributions p d and p d whose samples are relatively easy to collect. The DSGAN can learn the target distribution, p t , (or the unseen data distribution) from only the samples from the two distributions, p d and p d. In our scenario, p d is the distribution of the seen data, and p d can be obtained from p d via simple operations, so that we only need the samples of p d during the training. Two key applications, semi-supervised learning and novelty detection, are taken as case studies to illustrate that the DSGAN enables the production of various unseen data. We also provide theoretical analyses about the convergence of the DSGAN.

1. INTRODUCTION:

Unseen data1 are not samples from the distribution of the training data and are difficult to collect. It has been demonstrated that unseen samples can be applied to several applications. Dai et al. (2017) proposed how to create complement data, and theoretically showed that complement data, considered as unseen data, could improve semi-supervised learning. In novelty detection, Yu et al. (2017) proposed a method to generate unseen data and used them to train an anomaly detector. Another related area is adversarial training Goodfellow et al. (2015) , where classifiers are trained to resist adversarial examples, which are unseen during the training phase. However, the aforementioned methods only focus on producing specific types of unseen data, instead of enabling the generation of general types of unseen data.

In this paper, we propose a general framework called difference-seeking generative adversarial network (DSGAN), to generate a variety of unseen data. The DSGAN is a generative approach. Traditionally, generative approaches, which are usually conducted in an unsupervised learning manner, are developed for learning the data distribution from its samples, from which subsequently, they produce novel and high-dimensional samples, such as the synthesized image Saito et al. (2018) . A state-of-the-art approach is the so-called generative adversarial network (GAN) Goodfellow et al. (2014) . GAN produces sharp images based on a game-theoretic framework, but it can be difficult and unstable to train owing to multiple interaction losses. Specifically, GAN consists of two functions: generator and discriminator. Both functions are represented as parameterized neural networks. The discriminator network is trained to determine whether the inputs belong to the real dataset or fake dataset created by the generator. The generator learns to map a sample from a latent space to some distribution to increase the classification errors of the discriminator.

Nevertheless, if a generator can learn to create unseen data, then a traditional GAN requires numerous training samples of unseen classes for training, leading to a contradiction with the definition of the unseen data. This fact motivates us to present the DSGAN, which can generate unseen data by adopting seen data as training samples (see Fig. 9 , which illustrates the difference between GAN and the DSGAN, in Appendix A). The key concept is to consider the distribution of the unseen data as the difference between two distributions that are relatively easy to obtain. For example, the out-of-distribution examples in the MNIST dataset, from another perspective, are found to belong to the differences between the sets of examples in MNIST and the universal set. It should be noted that in traditional GAN, the target distribution is identical to the training data distribution; however, in the DSGAN these two distributions, are considered to be different. This paper makes the following contributions:

(1) We propose the DSGAN to generate any unseen data only if the density of the target (unseen data) distribution is the difference between those of any two distributions, p d and p d .

(2) We show that the DSGAN possesses the flexibility to learn different target (unseen data) distributions in two key applications, semi-supervised learning and novelty detection. Specifically, for novelty detection, the DSGAN can produce boundary points around the seen data because this type of unseen data is easily misclassified. For semi-supervised learning, the unseen data are linear combinations of any labeled data and unlabeled data, excluding the labeled and unlabeled data themselves 2 .

(3) The DSGAN yields results comparable to a semi-supervised learning but with a short training time and low memory consumption. In novelty detection, combining both the DSGAN and variational auto-encoder (VAE, Kingma & Welling (2014b) ) methods achieve the state-of-the-art results.

2. PROPOSED METHOD-DSGAN:

2.1 FORMULATION We denote the generator distribution as p g and training data distribution as p d , both in an Ndimensional space. Let p d be the distribution decided by the user. For example, p d can be the convolution of p d and normal distribution. Let p t be the target distribution that the user is interested in, and it can be expressed as

EQUATION

where α ∈ [0, 1]. Our method, the DSGAN, aims to learn p g such that p g = p t . Note that if the support set of p d belongs to that of p d, then there exists at least an α such that the equality in (1) holds. However, even if the equality does not hold, intuitively, the DSGAN attempts to learn p g such that p g (x) ∼ p d(x) -αp d (x) 1 -α with the constraint, p g (x) ≥ 0. Specifically, the generator will output samples located in the high-density areas of p d -αp d . Furthermore, we show that the DSGAN can learn p g , whose support set is the difference between those of p d and p d in Theorem 1.

First, we formulate the generator and discriminator in GANs. The inputs, z, of the generator are drawn from p z (z) in an M -dimensional space. The generator function, G(z; θ g ) : R M → R N , represents a mapping to the data space, where G is a differentiable function with parameter θ g . The discriminator is defined as D (x; θ d ) : R N → [0, 1], which outputs a single scalar. D (x) can be considered as the probability that x belongs to a class of the real data.

Similar to traditional GAN, we train D to distinguish the real data from the fake data sampled from G. Concurrently, G is trained to produce realistic data that can mislead D. However, in the DSGAN, the definitions of "real data" and "fake data" are different from those in traditional GAN. The samples from p d are considered as real, but those from the mixture distribution between p d and p g are considered as fake. The objective function is defined as follows:

from the "seen" data, according to applications. In this paper, we focus on the scenario that the two distributions are different. 2 The linear combination of any labeled data and unlabeled data probably belongs to the set of seen data (labeled data and unlabeled data), which contradicts the definition of unseen data. Thus, the samples generated by the DSGAN should not include the seen data themselves.

V (G, D) := E x∼p d (x) [log D(x)] + (1 -α)E z∼pz (z) [log (1 -D (G (z)))] + αE x∼p d (x) [log (1 -D(x))] .

(2)

We optimize ( 2 

2.2. CASE STUDY ON VARIOUS UNSEEN DATA GENERATION:

To achieve a more intuitive understanding about the DSGAN, we conduct several case studies on two-dimensional (2D) synthetic datasets and MNIST. In Eq. ( 1), α = 0.8 is used. First, the convolution of p d and normal distribution ensure the density on the boundary is no longer zero. Second, we seek p g such that Eq.

(1) holds, where the support set of p g is approximated by the difference of those between p d and p d . Complement samples generation Fig. 1 illustrates that the DSGAN can generate complement samples between 2 circles. Denoting the density function of the two circles as p d , we assign the samples drawn from p d as linear combinations of the two circles. Then, by applying the DSGAN, we achieve our goal of generating complement samples. In fact, this type of unseen data is used in semi-supervised learning.

Boundary samples generation Fig. 2 illustrates that the DSGAN generates boundary points between four circles. This type of unseen data is used in novelty detection. In this case, we assign p d and p d as "the density function of four circles" and "the convolution of p d and normal distribution," respectively. The basis of our concept is also illustrated by a one-dimensional (1D) example in Fig. 3 . Difference-set generation We also validate the DSGAN on a high-dimensional dataset such as MNIST. In this example, we define p d as the distribution of digit "1" and p d as the distribution containing two digits "1" and "7". Because the density, p d (x), is high when x is digit "1," the generator is prone to output digit "7" with a high probability. More sample qualities of DSGAN on CelebA can be refer to Appendix G.

From the above results, we can observe two properties of the generator distribution, p g : i) the higher the density of p d (x), the lower the density of p g (x); ii) p g prefers to output samples from the high-density areas of p d(x) -αp d (x).

2.3. DESIGNING p d:

Thus far, we have demonstrated how the DSGAN can produce various types of unseen data by choosing a specific p d. In this section, we introduce a standard procedure to design p d, and illustrate each step with pictures.

Step 1. First, the training data, p d , are collected (Fig. 6 (a)).

Step 2. Second, based on the applications, the desired unseen data distribution is defined (e.g., complement samples for semi-supervised learning) (Fig. 6 (b)).

Step 3. Third, p d is defined as a mixed distribution of (1 -α)p g + (α)p d (Fig. 6 (c)).

Step 4. In the above procedure, the most important step is to determine which types of unseen data are suitable for a specific problem (Step 2). In this paper, we show two types of unseen data, which are useful in semi-supervised learning and novelty detection. However, determining all types of unseen data for all applications is beyond the scope of this study, and we leave this for future work.

Furthermore, we provide a method (see Appendix B in supplementary materials) by reformulating the objective function (2), so that it is more stable to train the DSGAN.

3. THEORETICAL RESULTS:

In this section, we show that by choosing an appropriate α, the support set of p g belongs to the difference set between p d and p d , so that the samples from p g are unseen from the p d perspective.

We start our proofs from two assumptions. First, in a non-parametric setting, we assume that both the generator and discriminator have infinite capacities. Second, p g is defined as the distribution of the samples drawn from G(z) under z ∼ p z .

In the following, we show that the support set of p g is contained within the differences in the support sets of p d and p d while achieving the global minimum such that we can generate the desired p g by designing an appropriate p d. 

Supp (p g ) ⊆ Supp (p d) -Supp(p d ),

where

C(G) = max D V (G, D) = E x∼p d (x) log p d(x) p d(x) + (1 -α)pg(x) + αp d (x) + E x∼p * (x) log (1 -α)pg(x) + αp d (x) p d(x) + (1 -α)pg(x) + αp d (x)

.

Proof. See Appendix C for the details.

Summarizing, the generator is prone to output samples that are located in the high-density areas of p d -αp d .

4. APPLICATIONS:

The DSGAN was applied to two problems: semi-supervised learning and novelty detection. In the semi-supervised learning, the DSGAN acts as a "bad generator," which creates complement samples (unseen data) in the feature space of the training data. For the novelty detection, the DSGAN generates the samples (unseen data) as boundary points around the training data.

4.1. SEMI-SUPERVISED LEARNING:

Semi-supervised learning (SSL) is a type of learning model that uses a few labeled data and numerous unlabeled data. The existing SSL methods based on a generative model, (e.g., VAE Kingma et al. (2014) and GAN Salimans et al. (2016) ), yield good empirical results. Dai et al. (2017) theoretically showed that a good semi-supervised learning required a bad GAN with the following objective function:

EQUATION

where (x, y) denotes a pair of data, and its corresponding label, {1, 2, . . . , K} denotes the label space for the classification, and L = {(x, y)} is the label dataset. Moreover, under the semi-supervised settings, p d in (3) is the distribution of the unlabeled data. Note that the discriminator, D, in GAN also plays the role of a classifier. If the generator distribution exactly matches the real data distribution (i.e., p g = p d ), then the classifier trained by the objective function (3) with the unlabeled data cannot have a better performance than that trained by the supervised learning with the objective function. Specifically,

EQUATION

Contrastingly, the generator is preferred to generate complement samples, which lie on the lowdensity area of p d . Under some mild assumptions, these complement samples help D to learn the correct decision boundaries in the low-density area because the probabilities of the true classes are forced to be low in the out-of-distribution areas.

The complement samples in Dai et al. (2017) are complex to produce. In Sec. 5.2, we will demonstrate that with the DSGAN, complement samples can be easily generated.

4.2. NOVELTY DETECTION:

Novelty detection determines if a query example belongs to a seen class. If the samples of one seen class are considered as positive data, then this difficulty is the absence of negative data in the training phase, so that the supervised learning cannot function.

Recently, novelty detection has made significant progress with the advent of deep leaning. Pidhorskyi et al. (2018 )Sakurada & Yairi (2014) focused on learning a representative latent space for a seen class.

When testing, the query image was projected onto the learned latent space. Then, the difference between the query image and its inverse image (reconstruction) was measured. Thus, only an encoder was needed to be trained for the projection and a decoder for the reconstruction. Under the circumstance, an autoencoder (AE) is generally is adopted to learn both the encoder and decoder Pidhorskyi et al. (2018 )Perera et al. (2019) . Let Enc(•) be the encoder and Dec(•) be the decoder.

The loss function of the AE is defined as min

EQUATION

where p pos is the distribution of a seen class. After the training, a query example, x test , is classified as the seen class if

EQUATION

where τ ∈ R + plays the trade-off between the true positive rate and false positive rate. However, ( 6) is based on two assumptions: (1) the positive samples from one seen class should have a small reconstruction error;

(2) the AE (or latent space) cannot well describe the negative examples from the unseen classes, leading to a relatively large reconstruction error. In general, the first assumption inherently holds when both the testing and training data originate from the same seen class. However, Pidhorskyi et al. ( 2018) Perera et al. (2019) observed that assumption (2) does not hold at all times because the loss function in (5) does not include a loss term to enforce the negative data to have a large reconstruction error.

For assumption (2) to hold, given positive data as the training inputs, we propose using the DSGAN to generate negative examples in the latent space, as discussed in Sec. 5.3. Then, the loss function of the AE is modified to enforce the negative data to have a large reconstruction error.

5. EXPERIMENTS:

Our experiments are divided into three parts. The first one examines how the hyperparameter, α, influences the learned generator distribution, p g . In the second and third experiments, we obtain empirical results about semi-supervised learning and novelty detection, which are presented in Sec. 5.2 and Sec. 5.3, respectively. Note that the training procedure of the DSGAN can be improved by other extensions of GANs such as WGAN Arjovsky et al. (2017) , WGAN-GP Gulrajani et al. (2017) , EBGAN Zhao et al. (2017) , and LSGAN Mao et al. (2017) . In our method, the WGAN-GP was adopted for the stability of the DSGAN in training and reduction in the mode collapse.

5.1. DSGAN WITH DIFFERENT α:

The impacts of different α values on the DSGAN are illustrated in Fig. 7 . In this example, the support of p d is the area bounded by a red dotted line, and the orange points are the samples from p d . Concurrently, we shift p d to the right by 1 unit and create the distribution, p d, whose support is bounded by blue dotted lines. The overlapping area between p d and p d is 0.5 unit (assuming the area of p d is 1 unit). Based on our theoretical results, α = 0.5 is the smallest selected value allowing p g to be disjoint to p d . Therefore, we can see that some generated samples, as presented in Fig. 7 (a), still belong to the support set of p d . Fig. 7 (b) shows that there is a perfect agreement between our theoretical and experiment results with α = 0.5. When α = 0.8, there is a remarkable gap between the generated (green) points and yellow points, as shown in Fig. 7(c ). In theory, the result obtained at α = 0.8 should be the same as that obtained at α = 0.5. This is because the discriminator should assign the entire area, which is the intersection of the complement of support set of p d and support set of p d, to the same score, under the assumption that the discriminator has an infinite capacity. However, in practice, the capacity of the discriminator is limited. Therefore, the score of the area near p d is lower than that far from it, when α is large. Therefore, p g tends to repel p d to achieve a high score (to deceive the discriminator).

5.2. DSGAN IN SEMI-SUPERVISED LEARNING:

We first introduce how the DSGAN generates the complement samples in the feature space. Dai et al. (2017) proved that if the complement samples generated by G could satisfy the following two assumptions in ( 7) and ( 8), i.e.,

EQUATION

)

where f is the feature extractor and w i is the linear classifier for the i th class, and

EQUATION

then all the unlabeled data would be correctly classified by the objective function (3). Specifically, (7) ensures that the classifiers can discriminate the generated data from the unlabeled data, and (8) causes the decision boundary to be located in the low-density areas of p d . When α is 0.5, the support set of p g is disjoint to that of p d , satisying the theoretical results. When α is 0.8, p g generates the rightmost points of p d. The level curves from the discriminator show that the generator is more prone to producing samples in a region with higher score than in that with a lower score. Note that the outputs of the discriminator are not restricted in [0, 1], because we use the formulation of the WGAN in this experiment.

The assumption in (8) implies that the complement samples must be in the space created by the linear combination of the labeled and unlabeled data. In addition, they cannot fall into the real data distribution, p d , owing to the assumption (7). To allow the DSGAN to generate such samples, we let the samples of p d be linear combinations of those from L and p d . Since 

p g (x) ≈ p d(x) -αp d (x) 1 -α ,

5.2.1. SIMULATION RESULTS:

First, the selected hyperparameters are listed in Table 5 in Appendix D.1. Second, the results obtained from the DSGAN and state-of-the-art methods on the three benchmark datasets are summarized in Table 1 . It can be observed that our method can compete with the state-of-the-art methods on the three datasets. Note that we report the results of badGAN not only from the original papers in the literature but also by reproducing them using the released codes of the authors. Moreover, it can also be observed from Table 1 that our results are comparable to the best record of badGAN and CAGAN. are better than those of other approaches on the MNIST and SVHN datasets.

On CIFAR-10, our method is only inferior to the CT-GAN. However, this might not be a reasonable comparison because the CT-GAN uses extra techniques, including temporal ensembling and data augmentation, which the other methods do not use.

5.3. DSGAN IN NOVELTY DETECTION:

In this section, we study how to use the DSGAN for assisting novelty detection. As mentioned in Sec. 4.2, we need to train the auto-encoder (AE) such that (i) the positive samples from one seen class have a small reconstruction error; (ii) negative samples from the unseen classes incur relatively higher reconstruction errors. The fundamental concept is to use the DSGAN to generate negative samples, which originally do not exist under the scenario of novelty detection. Next, we add a new loss term to penalize the small reconstruction errors of the negative samples (see the third stage below). Three stages are required to train our model (AE):

1. The encoder, Enc(•), and decoder, Dec(•), are trained using the loss function (5). 2. Given x ∼ p pos , Enc(x) are collected as the samples drawn from p d . p d is the convolution of p d having a normal distribution with a zero mean and variance σ. Then, we train the DSGAN to generate negative samples, which are drawn from p d(x) -p d (x) and are the boundary points around the positive samples in the latent space. Note that there are some variations in the DSGAN: the input of the generator, G, is Enc(x), instead of a random vector z in the latent space. We also add Enc(x) -G(Enc(x)) 2 2 , which will be explained in the next step, to train the generator. 3. Fixing the encoder, we retrain the decoder by the modified loss function,

min Dec E x∼ppos(x) x -Dec(Enc(x)) 2 2 + w • max 0, m -x -Dec(G(Enc(x))) 2 2 ,

where w is the trade-off between the reconstruction errors of positive samples Enc(x) and negative samples G(Enc(x)). Note that in the previous step, we add Enc(x) -G(Enc(x)) 2 2 to ensure that the outputs of the generator are around the input. Thus, the second term charges even though the negative samples are close to the corresponding positive sample, and they still exhibit a high reconstruction error, which is bounded by m (Zhao et al. ( 2017)).

The above algorithm, called VAE+DSGAN, can be used to strengthen the existing AE-based methods by using them in the first stage. In the simulation, we used a variational autoencoder (VAE) Kingma & Welling (2014a) because it performs better than the AE in the novelty detection.

In this section, following Perera et al. (2019) , the performance was evaluated using the area under the curve (AUC) of the receiver operating characteristics (ROC) curve. Given a dataset, a class was chosen as the seen class for training, and all the classes were used for testing. There exist several testing benchmarks for novelty detection, such as MNIST, COIL100 Nene et al. (1996) and CIFAR-10. The state-of-the-art method Perera et al. (2019) achieves high performance in AUC on MNIST and COIL100 (AUC is larger than 0.97). However, for CIFAR-10, Perera et al. (2019) only achieves 0.656. Thus, we chose the challenging dataset, CIFAR-10, as the benchmark to evaluate our method. The detailed network architecture can be found in Appendix E.

Because VAE+DSGAN can be considered as a fine tuning VAE Kingma & Welling (2014a) , we first illustrate the key difference between the VAE and VAE+DSGAN, as shown in Fig. 8 . The seen class, which is at the bottom of the images, is a car. Other rows are the images from the unseen classes. One can see that the reconstructed images are reasonably good even for the unseen class in the VAE. By contrast, our method enforces the reconstructed images of the unseen classes to be blurred while still preserving the reconstruction quality of the seen class. Thus, our method achieves a relatively larger gap, in terms of the reconstruction error between the seen data and unseen data, than the VAE.

In Table 3 , we compare the proposed method with several methods, including the VAE Kingma & Welling (2014a) , AND Abati et al. (2019) , DSVDD Ruff et al. (2018) , and OCGAN Perera et al. (2019) , in terms of the AUC value. One can see that in most cases, our method almost outperforms the VAE. Furthermore, the mean of the AUC values of our method also is larger than those of the state-of-the-art methods. It is worth mentioning that in addition to the VAE, the DSGAN has potential of being combined with other AE-based methods.

6 RELATED WORKS ABOUT UNSEEN DATA GENERATION Yu et al. (2017) proposed a method to generate samples of unseen classes in a unsupervised manner via an adversarial learning strategy. However, it requires solving an optimization problem for each sample, which certainly leads to a high computation cost. By contrast, the DSGAN has the capability to create infinite diverse unseen samples. Hou et al. (2018) presented a new GAN architecture that could learn two distributions of unseen data from a part of seen data and the unlabeled data. However, the unlabeled data must be a mixture of seen and unseen samples; the DSGAN does not require any unseen data. Kliger & Fleishman (2018) also applied GAN in novelty detection. Their objective was to learn a generator whose distribution is a mixture of novelty data distribution and training data distribution. To this end, they used feature matching (FM) to train the generator and expected p g to learn the mixture of distributions. However, the ultimate goal of FM is still to learn p g = p d ; therefore, their method might fail when GAN learns well. Dai et al. (2017) aimed to generate complementary samples (or out-of-distribution samples), but assumed that the in-distribution could be estimated by a pre-trained model, such as PixelCNN++, which might be difficult and expensive to train. Lee et al. (2018) used a simple classifier to replace the role of PixelCNN++ in Dai et al. (2017) so that the training was comparatively much easier and more suitable. Nevertheless, their method only focused on generating unseen data surrounding the low-density area of seen data. In comparison, the DSGAN has more flexibility to generate different types of unseen data (e.g., a linear combination of seen data, as described in Sec. 5.2). In addition, their method needs the label information of the data, whereas our method is fully unsupervised.

7. CONCLUSIONS:

We propose the DSGAN, which can produce any unseen data based on the assumption that the density of the unseen data distribution is the difference between the densities of any two distributions. The DSGAN is useful in an environment when the samples from the unseen data distribution are more difficult to collect than those from the two known distributions. Empirical and theoretical results are provided to validate the effectiveness of the DSGAN. Finally, because the DSGAN is developed based on GAN, it is easy to apply any improved versions of GAN to the DSGAN.

B TRICKS FOR STABLE TRAINING:

We provide a trick to stabilize the training procedure by reformulating the objective function. Specifically, V (G, D) in ( 2) is reformulated as:

EQUATION

Instead of sampling a mini-batch of m samples from p z and p d in Algorithm 1, (1 -α)m and αm samples from both distributions are required, respectively. The computation cost in training can be reduced due to fewer samples. Furthermore, although ( 9) is equivalent to (2) in theory, we find that the training using (9) achieves better performance than using (2) via empirical validation in Table 4 . We conjecture that the equivalence between ( 9) and ( 2) is based on the linearity of expectation, but mini-batch stochastic gradient descent in practical training may lead to the different outcomes.

Table 4 : Semi-supervised learning results on MNIST with and without the use of sampling tricks.

Methods:



MNIST (# errors):

Our method w/o tricks 91.0 ± 7.0 Our method w/ tricks 82.7 ± 4.6

C PROOF OF THEOREM 1

In this section, we show Theorem 1.

This proof includes two parts: the first part shows that the objective function is equivalent to minimizing the Jensen-Shannon divergence in the mixture distribution (p d and p g ) and p d if G and D are assigned sufficient capacity; the second part shows that by choosing an appropriate α, the support set of p g belongs to the difference set between p d and p d , so that the samples from p g are unseen from the p d perspective.

For the first part, we show the optimal discriminator given G, and then show that minimizing V (G, D) via G, given the optimal discriminator, is equivalent to minimizing the Jensen-Shannon divergence between (1 -α)p g + αp d and p d.

Proposition 1. If G is fixed, the optimal discriminator, D, is

D * G (x) = p d (x) p d (x) + (1 -α)p g (x) + αp d (x)

.

Proof. Given any generator G, the training criterion for the discriminator D is to maximize the quantity V (G, D):

V (G, D) = x p d(x) log (D (x)) dx + (1 -α) z p z (z) log (1 -D (G (z))) dz + α x p d (x) log (1 -D (x)) dx = x p d(x) log (D (x)) dx + (1 -α) x p g (x) log (1 -D (x)) dz + α x p d (x) log (1 -D (x)) dx = x p d(x) log (D (x)) + ((1 -α)p g (x) + αp d (x)) log (1 -D (x)) dx.

For any (a, b) ∈ R 2 \{0, 0}, the function a log (y) + b log (1 -y) achieves its maximum in [0, 1] at y = a a+b . The discriminator only needs to be defined within Supp(p d) Supp(p d ) Supp(p g ). We complete this proof.

Moreover, D can be considered to discriminate between samples from p d and ((1 -α)p g (x) + αp d (x)). By replacing the optimal discriminator in V (G, D), we trivially obtain

EQUATION

Actually, the results thus far yield the optimal solution of D given G is fixed in (1). Now, the next step is to determine the optimal G with D * G as fixed. Theorem 2. The global minimum of C(G) is achieved if and only if (1 -α)p g (x) + αp d (x) = p d(x) for all x. Then, C(G) achieves the value, -log 4.

Proof. We start from

(1) = -log( 4)

+ E x∼p d(x) log 2p d(x) p d(x) + (1 -α)p g (x) + αp d (x) + E x∼p * (x) log 2 ((1 -α)p g (x) + αp d (x)) p d(x) + (1 -α)p g (x) + αp d (x) = -log(4) + KL p d p d + (1 -α)p g + αp d 2 + KL (1 -α)p g (x) + αp d p d + (1 -α)p g + αp d 2 = -log(4) + 2 JSD (p d (1 -α)p g + αp d ) ,

where p * (x) = (1 -α)p g (x) + αp d (x), KL is the Kullback-Leibler divergence and JSD is the Jensen-Shannon divergence. The JSD returns the minimal value, which is 0, iff both distributions are the same, namely p d = (1 -α)p g + αp d . Because p g (x)'s are always non-negative, it should be noted both distributions are the same only if αp d (x) ≤ p d(x) for all x's. We complete this proof.

Note that (1 -α)p g (x) + αp d (x) = p d(x) may not hold if αp d (x) > p d(x) . However, the DSGAN still works based on two facts: i) given D, V (G, D) is a convex function in p g and ii) because x p g (x)dx = 1, the set collecting all the feasible solutions of p g is convex. Thus, there always exists a global minimum of V (G, D) given D, but it may not be -log(4). Now, we go back to prove Theorem 1. We show that the support set of p g is contained within the differences in the support sets of p d and p d while achieving the global minimum such that we can generate the desired p g by designing an appropriate p d. Furthermore,

Proof. Recall that:

C(G) = x p d(x) log p d(x) p d(x) + (1 -α)p g (x) + αp d (x) + p * (x) log (1 -α)p g (x) + αp d (x) p d(x) + (1 -α)p g (x) + αp d (x) dx

T (p g ; x) = ∂S(p g ; x) ∂p g (x) = log (1 -α)p g (x) + αp d (x) p d(x) + (1 -α)p g (x) + αp d (x)

, which is expected to be as small as possible to minimize C(G), is increasing on p g (x) and converges to 0. Then, we show that T (p g ; x) for x ∈ Supp(p d) Supp(p d ) is always larger than that for x ∈ Supp(p d) -Supp(p d ) for all p g . Specifically,

1. When x ∈ Supp(p d) Supp(p d ), T (p g ; x) ≥ log 1 2 always holds due to the assumption of αp d (x) ≥ p d(x).

2. When x ∈ Supp(p d) -Supp(p d ), T (p g ; x) < log 1 2 for all p g (x)'s satisfying (1α)p g (x) ≤ p d(x) .

Thus, the minimizer prefers p g (x) > 0 for x ∈ Supp(p d) -Supp(p d ) and (1 -α)p g (x) ≤ p d(x) . We check whether there exists a solution p g such that (1 -α)p g (x) ≤ p (1 -α)p g (x)dx, the last inequality implies that there must exist a feasible solution. We complete this proof.

Another concern is the convergence of Algorithm 1. Proposition 2. The discriminator reaches its optimal value given G in Algorithm 1, and p g is updated by minimizing

E x∼p d (x) [log D * G (x)] + E x∼p * (x) [log (1 -D * G (x))] . If G and D have sufficient capacities, then p g converges to argmin pg JSD (p d (1 -α)p g + αp d ).

Proof. Consider V (G, D) = U (p g , D) as a function of p g . By the proof idea of Theorem 2 in Goodfellow et al. (2014) , if f (x) = sup α∈A f α (x) and f α (x) is convex in x for every α, then ∂f β (x) ∈ ∂f if β = argsup α∈A f α (x). In other words, if sup D V (G, D) is convex in p g , the subderivatives of sup D V (G, D) includes the derivative of the function at the point, where the maximum is attained, implying the convergence with sufficiently small updates of p g . We complete this proof.

D EXPERIMENTAL DETAILS FOR SEMI-SUPERVISED LEARNING D.0.1 DATASETS: MNIST, SVHN, AND CIFAR-10 For evaluating the semi-supervised learning task, we used 60000/ 73257/ 50000 samples and 10000/ 26032/ 10000 samples from the MNIST/ SVHN/ CIFAR-10 datasets for the training and testing, respectively. Under the semi-supervised setting, we randomly chose 100/ 1000/ 4000 samples from the training samples, which are the MNIST/ SVHN/ CIFAR-10 labeled datasets, and the amounts of the labeled data for all the classes are equal. Furthermore, our criterion to determine the hyperparameters is introduced in Appendix D.1, and the network architectures are described in Appendix D.2. We performed testing with 10/ 5/ 5 runs on MNIST/ SVHN/ CIFAR-10 based on the selected hyperparameters, and randomly selected the labeled dataset. The results were recorded as the mean and standard deviation of the number of errors from each run.

D.1 HYPERPARAMETERS:

The hyperparameters were chosen to make our generated samples consistent with the assumptions in ( 7) and (8). However, in practice, if we make all the samples produced by the generator following the assumption in (8), then the generated distribution is not close to the true distribution, even a large margin between them exists, which is not what we desire. So, in our experiments, we make a concession that the percentage of generated samples, which accords with the assumption, is around 90%. To meet this objective, we tune the hyperparameters. Table 5 shows our setting of hyperparameters, where β is defined in (8). Dai et al. (2017) . However, different from previous works that have only a generator and a discriminator, we design an additional discriminator in the feature space, and its architecture is similar across all datasets with only the difference in the input dimensions. Following Dai et al. (2017) , we also define the feature space as the input space of the output layer of discriminators.

Compared to SVHN and CIFAR-10, MNIST is a simple dataset as it is only composed of fully connected layers. Batch normalization (BN) or weight normalization (WN) is used in every layer to stable training. Moreover, Gaussian noise is added before each layer in the classifier, as proposed in Rasmus et al. (2015) . We find that the added Gaussian noise exhibits a positive effect for semisupervised learning. The architecture is shown in Table 6 .

Table 7 and Table 8 are models for SVHN and CIFAR-10, respectively, and these models are almost the same except for some implicit differences, e.g., the number of convolutional filters and types of dropout. In these tables, given a dropping rate, "Dropout" denotes a normal dropout in that the elements of input tensor are randomly set to zero while Dropout2d is a dropout only applied on the channels to randomly zero all the elements. Furthermore, the training procedure alternates between k steps of optimizing D and one step of optimizing G. We find that k in Algorithm 1 is a key role in the problem of mode collapse for different applications. For semi-supervised learning, we set k = 1 for all datasets.

E EXPERIMENTAL DETAILS FOR NOVELTY DETECTION:

The architecture of GAN and VAE are depicted in Table 9 and 10, respectively.

In the experiment, we first trained the VAE for 500 epochs and then we trained DSGAN for 500 epochs with m = 1.5 and w = 0.5. Third, we fixed the encoder and tuned the decoder with both positive and negative samples (generated by DSGAN) for 600 epochs. Here, we conduct the experiments on different α under semi-supervised learning settings. From Sec. 4.1 and 5.2, badGAN already shows that, if the desired unseen data can be generated, then the classifier will put the correct decision boundary in the low-density area.

In Table 11 , we demonstrate the classification results on α = 0.5 and α = 0.8, respectively. We can observe that the results generated at α = 0.8 is better than those generated at α = 0.5, meeting the above discussion. From our empirical observations, DSGAN is prone to generating unseen data at α = 0.8, leading a better classifier. We show one more experiment on CelebA (Liu et al. (2015) ) to demonstrate DSGAN can work well even for complicated images. In this experiment, we generate the color images of size 64 × 64. Similar to our 1/7 experiments on the MNIST dataset, we let p d be the distribution of face images with glasses and without glasses and let p d be the distribution of images without glasses. We validate DSGAN with α = 0.5 and α = 0.8, respectively. For α = 0.5, we sample 10000 images with glasses and 10000 images without glasses from CelebA. When α is 0.8, we sample 40000 instead of 10000 images without glasses.

We also train GAN to verify the generated image quality of DSGAN. For fair comparison, GAN is trained under two kinds of settings. The first one is that GAN is only trained with the images with glasses. Second, it is pretrained with all images, and is finetuned with the images with glasses, namely transferring GANs in Wang et al. (2018) . It should be noted that transferring GAN uses the same amount of training data as DSGAN and serves as a stronger baseline than GAN under the first setting.

Fréchet Inception Distance (FID) (Heusel et al. (2017) ) is used to evaluate the quality of generated images. FID calculates the Wasserstein-2 distance between generated images and real images (images with glasses) in the feature space of Inception-v3 network (Szegedy et al. (2015) ). We train both networks for 600 epochs, and use WGAN-GP as the backbone for both GAN and DSGAN. In addition, transferring GANs are pretrained for 500 epochs, then being finetuned for 600 epochs.

Fig. 10 and Table 12 show generated images and FID for all methods, respectively. We can see that our DSGAN can generate images with glasses from the given p d and p d, and the FID of DSGAN are comparable to that of GAN. The experiment validates that DSGAN still works well to create complement data for complicate images. 

:

In traditional machine learning scenarios, "unseen" data corresponds to data that is not used or seen during the training stage but rather the testing stage. The distribution of "unseen" data could be same as or different

Figures and Tables:

) by a min-max game between G and D, i.e., During the training procedure, an iterative approach, like traditional GAN, is to alternate between k steps of training D and one step of training G. In practice, minibatch stochastic gradient descent via backpropagation is used to update θ d and θ g . Thus, for each p g , p d , and p d, m samples are required for computing the gradients, where m is the number of samples in a minibatch. The training procedure is illustrated in Algorithm 1 in Appendix A. The DSGAN suffers from the same drawbacks as traditional GAN, (e.g., mode collapse, overfitting, and strong discriminator) so that the generator gradient vanishes. There are literature Salimans et al. (2016); Arjovsky & Bottou (2017); Miyato et al. (2018) focusing on dealing with the above problems, and such concepts can be readily combined with the DSGAN. Li et al. (2017) and Reed et al. (2016) proposed an objective function similar to (2). Their goal was to learn the conditional distribution of training data. However, we aim to learn the target distribution, p t , in Eq. (1), and not the training data distribution.

Figure 1: Complement points (in Green) between two circles (in Orange).

Figure 2: Boundary points (in Green) between four circles (in Orange).

Figure 3: Illustration of the generation of the unseen data in the boundary around the training data. First, the convolution of p d and normal distribution ensure the density on the boundary is no longer zero. Second, we seek p g such that Eq.(1) holds, where the support set of p g is approximated by the difference of those between p d and p d .

Figure 4: Illustration of the difference-set seeking in MNIST.Figure5: DSGAN learns the difference between two sets.

Figure 4: Illustration of the difference-set seeking in MNIST.Figure5: DSGAN learns the difference between two sets.

Figure 6: Illustration for designing p d.

Figure 7: Influence of α on the synthetic dataset. We observe that the samples of p g (green points) move farther away from p d as α increases; however, they are still bounded by the support of p d.When α is 0.5, the support set of p g is disjoint to that of p d , satisying the theoretical results. When α is 0.8, p g generates the rightmost points of p d. The level curves from the discriminator show that the generator is more prone to producing samples in a region with higher score than in that with a lower score. Note that the outputs of the discriminator are not restricted in [0, 1], because we use the formulation of the WGAN in this experiment.

g will tend to match p d, whereas the term, -αp d , ensures that the samples from p g do not belong to p d . Thus, p g satisfies the assumption in (8). Moreover, (7) is also satisfied by training the classifier with (3) based on substituting the generator distribution in (3) into the learned p g . Following the previous works, we apply the proposed DSGAN to semi-supervised learning on three benchmark datasets: MNISTLeCun et al. (1998),SVHN Netzer et al. (2011), and CIFAR-10  Krizhevsky (2009). The details of the experiments can be found in Appendix D.

Figure8: Comparison of the reconstructed results of the VAE and our method. The seen class, which is at the bottom of the images, is a car. Other rows are images from the unseen classes. Our method exhibits a relatively larger gap, in terms of the reconstruction error between the seen data and unseen data, than the VAE.

g ; x)dx = x∈Supp(p d )-Supp(p d ) S(p g ; x)dx + x∈Supp(p d ) S(p g ; x)dx.S(p g ; x) is used to simplify the notations inside the integral. For any x, S(p g ; x) in p g (x) is nonincreasing and S(p g ; x) ≤ 0 always holds. Specifically, S(p g ; x) is decreasing along the increase of p g (x) if p d(x) > 0; S(p g ; x) attains the maximum value, zero, for any p g (x) if p d(x) = 0. Since DSGAN aims to minimize C(G) with the constraint x p g (d)dx = 1, the solution attaining the global minima must satisfy p g (x) = 0 if p d(x) = 0; otherwise, there exists another solution with smaller value of C(G). Thus, Supp (p g ) ⊆ Supp (p d).

d(x) and x∈Supp(p d )-Supp(p d ) p g (d)dx = 1, implying p g (x) = 0 for x ∈ Supp(p d) Supp(p d ). Based on the following expression, x∈Supp(p d )-Supp(p d )p d(x)dx + x∈Supp(p d ) p d(x)dx = 1 ⇒ x∈Supp(p d )-Supp(p d ) p d(x)dx ≥ 1 -x∈Supp(p d ) αp d (x)dx ⇒ x∈Supp(p d )-Supp(p d ) p d(x)dx ≥ 1 -α ⇒ x∈Supp(p d )-Supp(p d ) p d(x)dx ≥ x∈Supp(p d )-Supp(p d )

Figure 10: Sampled generated images of GAN and DSGAN on CelebA.

Theorem 1. Suppose αp d (x) ≥ p d(x) for all x ∈ Supp(p d ) and all density functions p d (x), and p d(x) and p g (x) are continuous. If the global minimum of C(G) is achieved, then

Table content: <table/>

The reason of presenting both the results is that we cannot reproduce parts of the results. The experiments inLi et al. (2019) also showed a similar problem. In comparison withDai et al. (2017), our methods do not need to rely on an additional density estimation network, PixelCNN++Salimans et al. (2017). Although PixelCNN++ is one of the best density estimation networks, learning such a deep architecture requires large computation and high memory consumption. In Table2, we list the training time and memory consumption for our method and badGAN. Compared to badGAN, our method consumes 15.8% less training time and saves about 9000 MB during the training.

Table content: <table/>

Comparison of the semi-supervised learning in our DSGAN and the state-of-the-art methods. For a reasonable comparison, we only consider GAN-based methods. * denotes the use of the same architecture of the classifier. † denotes a larger architecture of the classifier. ‡ denotes the use of data augmentation (in CIFAR-10). The results for MNIST are recorded as the number of errors, whereas for the others are as percentage of the error.

Table content: <table><tr><td>Methods</td><td>MNIST</td><td>SVHN</td><td>CIFAR-10</td></tr><tr><td>FM  *  Salimans et al. (2016)</td><td>93 ± 6.5</td><td>8.11 ± 1.3</td><td>18.63 ± 1.32</td></tr><tr><td>TripleGAN  † Li et al. (2017)</td><td>91 ± 58</td><td colspan="2">5.77 ± 0.17 16.99 ± 0.36</td></tr><tr><td>badGAN  *  Dai et al. (2017)</td><td>79.5 ± 9.8</td><td colspan="2">4.25 ± 0.03 14.41 ± 0.30</td></tr><tr><td>CAGAN  *  Ni et al. (2018)</td><td>81.9± 4.5</td><td colspan="2">4.83 ± 0.09 12.61 ± 0.12</td></tr><tr><td>CT-GAN  ‡ Wei et al. (2018)</td><td>89 ± 13</td><td>-</td><td>9.98 ± 0.21</td></tr><tr><td>badGAN-reproduce  *</td><td colspan="3">86.2 ± 13.2 4.48 ± 0.16 16.25 ± 0.33</td></tr><tr><td>Our method  *</td><td>82.7 ± 4.6</td><td colspan="2">4.38 ± 0.10 14.52 ± 0.14</td></tr></table>

Training times of our method and badGAN. We only report the training time on MNIST, on which the authors of badGAN applied PixelCNN++. The experiments run on a NVIDIA 1080 Ti.

Table content: <table><tr><td>Methods</td><td colspan="2">Training time Memory Consumption</td></tr><tr><td>badGAN</td><td>38 s / epoch</td><td>9763 MB</td></tr><tr><td>Our method  *</td><td>32 s / epoch</td><td>711 MB</td></tr></table>



Table content: <table><tr><td/><td>0</td><td>1</td><td>2</td><td>3</td><td>4</td><td>5</td><td>6</td><td>7</td><td>8</td><td>9</td><td>MEAN</td></tr><tr><td>VAE</td><td colspan="10">.700 .386 .679 .535 .748 .523 .687 .493 .696 .386</td><td>.583</td></tr><tr><td>AND</td><td colspan="10">.735 .580 .690 .542 .761 .546 .751 .535 .717 .548</td><td>.641</td></tr><tr><td>DSVDD</td><td colspan="10">.617 .659 .508 .591 .609 .657 .677 .673 .759 .731</td><td>.648</td></tr><tr><td>OCGAN</td><td colspan="10">..757 .531 .640 .620 .723 .620 .723 .575 .820 .554</td><td>.657</td></tr><tr><td colspan="11">Our method .737 .614 .676 .644 .759 .562 .660 .646 .769 .633</td><td>.670</td></tr></table>

Hyperparameters in semi-supervised learning.

Table content: <table><tr><td colspan="4">Hyperparameters MNIST SVHN CIFAR-10</td></tr><tr><td>α</td><td>0.8</td><td>0.8</td><td>0.5</td></tr><tr><td>β</td><td>0.3</td><td>0.1</td><td>0.1</td></tr><tr><td>D.2 ARCHITECTURE</td><td/><td/><td/></tr><tr><td colspan="4">In order to fairly compare with other methods, our generators and classifiers for MNIST, SVHN,</td></tr><tr><td colspan="2">and CIFAR-10 are same as in Salimans et al. (2016) and</td><td/><td/></tr></table>

Network architectures for semi-supervised learning on MNIST. (GN: Gaussian noise)

Table content: <table><tr><td>Generator G</td><td>Discriminator D</td><td>Classifier C</td></tr><tr><td colspan="3">Input: z ∈ R 100 from unif(0, 1) Input: 250 dimension feature Input: 28 × 28 gray image</td></tr><tr><td>100 × 500 FC layer with BN</td><td>250 × 400 FC layer</td><td>GN, std = 0.3</td></tr><tr><td>Softplus</td><td>ReLU</td><td>784 × 1000 FC layer with WN ,ReLU</td></tr><tr><td>500 × 500 FC layer with BN</td><td>400 × 200 FC layer</td><td>GN, std = 0.5</td></tr><tr><td>Softplus</td><td>ReLU</td><td>1000 × 500 FC layer with WN, ReLU</td></tr><tr><td>500 × 784 FC layer with WN</td><td>200 × 100 FC layer</td><td>GN, std = 0.5</td></tr><tr><td>Sigmoid</td><td>ReLU</td><td>500 × 250 FC layer with WN, ReLU</td></tr><tr><td/><td>100 × 1 FC layer</td><td>GN, std = 0.5</td></tr><tr><td/><td/><td>250 × 250 FC layer with WN, ReLU</td></tr><tr><td/><td/><td>GN, std = 0.5</td></tr><tr><td/><td/><td>250 × 250 FC layer with WN, ReLU</td></tr><tr><td/><td/><td>250 × 10 FC layer with WN</td></tr></table>

The architectures of VAE for novelty detection. × 5 conv. 32 stride = 2, with BN, lReLU(0.2) 5 × 5 conv. 64 stride = 2, with BN, lReLU(0.2) 5 × 5 conv. 128 stride = 2, with BN, lReLU(0.2) (For mean) 4 × 4 conv. 128 stride = 1 (For std) 4 × 4 conv. 128 stride = 1 5 × 5 conv. transpose 128 stride = 2 with BN, lReLU(0.2) 5 × 5 conv. transpose 64 stride = 2 with BN, lReLU(0.2) 5 × 5 conv. transpose 32 stride = 2 with BN, lReLU(0.2) 5 × 5 conv. transpose 3 stride = 2, Tanh

Table content: <table><tr><td>Encoder</td><td>Decoder</td></tr><tr><td>5</td><td/></tr></table>

Ablation study of different α values for DSGAN in semi-supervised learning, where the result for MNIST is represented in terms of number of errors and the percentage of errors was used for other datasets. DSGAN (α = 0.5) 91.5 ± 5.6 4.59 ± 0.15 14.52 ± 0.14 DSGAN (α = 0.8) 82.7 ± 4.6 4.38 ± 0.10 14.47 ± 0.15 G SAMPLE QUALITY OF DSGAN ON CELEBA

Table content: <table><tr><td>Methods</td><td>MNIST</td><td>SVHN</td><td>CIFAR-10</td></tr></table>

FIDs of GAN and DSGAN on CelebA. Smaller FID means that the generated distribution is closer to the distribution of images with glasses.

Table content: <table><tr><td/><td>10000 samples</td><td>20000 samples</td><td/><td>50000 samples</td><td/></tr><tr><td/><td>GAN</td><td colspan="4">transferring GAN DSGAN (α = 0.5) transferring GAN DSGAN (α = 0.8)</td></tr><tr><td>FID</td><td>22.37</td><td>18.34</td><td>18.05</td><td>16.45</td><td>15.39</td></tr></table>

Peer review (iclr2020):
This paper proposed the DSGAN model to generate unseen data. The intuition based on standard GAN is straightforward and makes sense. The paper is well written, especially the case studies illustrate the idea clearly. The designing of p_{_x0008_ar{d}} also presented the limitation of this method. Two main discussed applications, semi-supervised learning and novelty detection are important in machine learning. In general, this is an interesting paper. However, my concern is about the experiments. As a generative model for unseen data, I would like to see the generated results, which is more convincing. Only the 1/7 examples of MNIST dataset are provided in case studies. I am wondering for more complicated images, how is the performance?