DIFFERENCE-SEEKING GENERATIVE ADVERSARIAL NETWORK–UNSEEN SAMPLE GENERATION
Year: 2020

Abstract:
Unseen data, which are not samples from the distribution of training data and are difficult to collect, have exhibited importance in numerous applications, (e.g., novelty detection, semi-supervised learning, and adversarial training). In this paper, we introduce a general framework called difference-seeking generative adversarial network (DSGAN), to generate various types of unseen data. Its novelty is the consideration of the probability density of the unseen data distribution as the difference between two distributions pd̄ and pd whose samples are relatively easy to collect. The DSGAN can learn the target distribution, pt, (or the unseen data distribution) from only the samples from the two distributions, pd and pd̄. In our scenario, pd is the distribution of the seen data, and pd̄ can be obtained from pd via simple operations, so that we only need the samples of pd during the training. Two key applications, semi-supervised learning and novelty detection, are taken as case studies to illustrate that the DSGAN enables the production of various unseen data. We also provide theoretical analyses about the convergence of the DSGAN.

heading: 1 INTRODUCTION
text: Unseen data1are not samples from the distribution of the training data and are difficult to collect. It has been demonstrated that unseen samples can be applied to several applications. Dai et al. (2017) proposed how to create complement data, and theoretically showed that complement data, considered as unseen data, could improve semi-supervised learning. In novelty detection, Yu et al. (2017) proposed a method to generate unseen data and used them to train an anomaly detector. Another related area is adversarial training Goodfellow et al. (2015), where classifiers are trained to resist adversarial examples, which are unseen during the training phase. However, the aforementioned methods only focus on producing specific types of unseen data, instead of enabling the generation of general types of unseen data.
In this paper, we propose a general framework called difference-seeking generative adversarial network (DSGAN), to generate a variety of unseen data. The DSGAN is a generative approach. Traditionally, generative approaches, which are usually conducted in an unsupervised learning manner, are developed for learning the data distribution from its samples, from which subsequently, they produce novel and high-dimensional samples, such as the synthesized image Saito et al. (2018). A state-of-the-art approach is the so-called generative adversarial network (GAN) Goodfellow et al. (2014). GAN produces sharp images based on a game-theoretic framework, but it can be difficult and unstable to train owing to multiple interaction losses. Specifically, GAN consists of two functions: generator and discriminator. Both functions are represented as parameterized neural networks. The discriminator network is trained to determine whether the inputs belong to the real dataset or fake dataset created by the generator. The generator learns to map a sample from a latent space to some distribution to increase the classification errors of the discriminator.
1In traditional machine learning scenarios, "unseen" data corresponds to data that is not used or seen during the training stage but rather the testing stage. The distribution of "unseen" data could be same as or different
Nevertheless, if a generator can learn to create unseen data, then a traditional GAN requires numerous training samples of unseen classes for training, leading to a contradiction with the definition of the unseen data. This fact motivates us to present the DSGAN, which can generate unseen data by adopting seen data as training samples (see Fig. 9, which illustrates the difference between GAN and the DSGAN, in Appendix A). The key concept is to consider the distribution of the unseen data as the difference between two distributions that are relatively easy to obtain. For example, the out-of-distribution examples in the MNIST dataset, from another perspective, are found to belong to the differences between the sets of examples in MNIST and the universal set. It should be noted that in traditional GAN, the target distribution is identical to the training data distribution; however, in the DSGAN these two distributions, are considered to be different.
This paper makes the following contributions:
(1) We propose the DSGAN to generate any unseen data only if the density of the target (unseen data) distribution is the difference between those of any two distributions, pd̄ and pd.
(2) We show that the DSGAN possesses the flexibility to learn different target (unseen data) distributions in two key applications, semi-supervised learning and novelty detection. Specifically, for novelty detection, the DSGAN can produce boundary points around the seen data because this type of unseen data is easily misclassified. For semi-supervised learning, the unseen data are linear combinations of any labeled data and unlabeled data, excluding the labeled and unlabeled data themselves2.
(3) The DSGAN yields results comparable to a semi-supervised learning but with a short training time and low memory consumption. In novelty detection, combining both the DSGAN and variational auto-encoder (VAE, Kingma & Welling (2014b)) methods achieve the state-of-the-art results.

heading: 2 PROPOSED METHOD-DSGAN

heading: 2.1 FORMULATION
text: We denote the generator distribution as pg and training data distribution as pd, both in an N - dimensional space. Let pd̄ be the distribution decided by the user. For example, pd̄ can be the convolution of pd and normal distribution. Let pt be the target distribution that the user is interested in, and it can be expressed as
(1− α)pt(x) + αpd(x) = pd̄(x), (1)
where α ∈ [0, 1]. Our method, the DSGAN, aims to learn pg such that pg = pt. Note that if the support set of pd belongs to that of pd̄, then there exists at least an α such that the equality in (1) holds. However, even if the equality does not hold, intuitively, the DSGAN attempts to learn pg such that pg(x) ∼ pd̄(x)− αpd(x)
1− α with the constraint, pg(x) ≥ 0. Specifically, the generator will output
samples located in the high-density areas of pd̄ − αpd. Furthermore, we show that the DSGAN can learn pg , whose support set is the difference between those of pd̄ and pd in Theorem 1.
First, we formulate the generator and discriminator in GANs. The inputs, z, of the generator are drawn from pz (z) in an M -dimensional space. The generator function, G(z; θg) : RM → RN , represents a mapping to the data space, where G is a differentiable function with parameter θg . The discriminator is defined as D (x; θd) : RN → [0, 1], which outputs a single scalar. D (x) can be considered as the probability that x belongs to a class of the real data.
Similar to traditional GAN, we train D to distinguish the real data from the fake data sampled from G. Concurrently, G is trained to produce realistic data that can mislead D. However, in the DSGAN, the definitions of “real data” and “fake data” are different from those in traditional GAN. The samples from pd̄ are considered as real, but those from the mixture distribution between pd and pg are considered as fake. The objective function is defined as follows:
from the "seen" data, according to applications. In this paper, we focus on the scenario that the two distributions are different.
2The linear combination of any labeled data and unlabeled data probably belongs to the set of seen data (labeled data and unlabeled data), which contradicts the definition of unseen data. Thus, the samples generated by the DSGAN should not include the seen data themselves.
V (G,D) := Ex∼pd̄(x) [logD(x)] + (1− α)Ez∼pz(z) [log (1−D (G (z)))] + αEx∼pd(x) [log (1−D(x))] . (2) We optimize (2) by a min–max game between G and D, i.e.,
min G max D V (G,D) .
During the training procedure, an iterative approach, like traditional GAN, is to alternate between k steps of training D and one step of training G. In practice, minibatch stochastic gradient descent via backpropagation is used to update θd and θg. Thus, for each pg, pd, and pd̄, m samples are required for computing the gradients, where m is the number of samples in a minibatch. The training procedure is illustrated in Algorithm 1 in Appendix A. The DSGAN suffers from the same drawbacks as traditional GAN, (e.g., mode collapse, overfitting, and strong discriminator) so that the generator gradient vanishes. There are literature Salimans et al. (2016); Arjovsky & Bottou (2017); Miyato et al. (2018) focusing on dealing with the above problems, and such concepts can be readily combined with the DSGAN.
Li et al. (2017) and Reed et al. (2016) proposed an objective function similar to (2). Their goal was to learn the conditional distribution of training data. However, we aim to learn the target distribution, pt, in Eq. (1), and not the training data distribution.

heading: 2.2 CASE STUDY ON VARIOUS UNSEEN DATA GENERATION
text: To achieve a more intuitive understanding about the DSGAN, we conduct several case studies on two-dimensional (2D) synthetic datasets and MNIST. In Eq. (1), α = 0.8 is used.
1
7
1
7
Figure 4: Illustration of the difference-set seeking in MNIST.
Figure 5: DSGAN learns the difference between two sets.
Complement samples generation Fig. 1 illustrates that the DSGAN can generate complement samples between 2 circles. Denoting the density function of the two circles as pd, we assign the samples drawn from pd̄ as linear combinations of the two circles. Then, by applying the DSGAN, we achieve our goal of generating complement samples. In fact, this type of unseen data is used in semi-supervised learning.
Boundary samples generation Fig. 2 illustrates that the DSGAN generates boundary points between four circles. This type of unseen data is used in novelty detection. In this case, we assign pd and pd̄ as “the density function of four circles” and “the convolution of pd and normal distribution,” respectively. The basis of our concept is also illustrated by a one-dimensional (1D) example in Fig. 3.
Difference-set generation We also validate the DSGAN on a high-dimensional dataset such as MNIST. In this example, we define pd as the distribution of digit “1” and pd̄ as the distribution containing two digits “1” and “7”. Because the density, pd(x), is high when x is digit “1,” the generator is prone to output digit “7” with a high probability. More sample qualities of DSGAN on CelebA can be refer to Appendix G.
From the above results, we can observe two properties of the generator distribution, pg: i) the higher the density of pd(x), the lower the density of pg(x); ii) pg prefers to output samples from the high-density areas of pd̄(x)− αpd(x).
2.3 DESIGNING pd̄
Thus far, we have demonstrated how the DSGAN can produce various types of unseen data by choosing a specific pd̄. In this section, we introduce a standard procedure to design pd̄, and illustrate each step with pictures.
Step 1. First, the training data, pd, are collected (Fig. 6 (a)).
Step 2. Second, based on the applications, the desired unseen data distribution is defined (e.g., complement samples for semi-supervised learning) (Fig. 6 (b)).
In the above procedure, the most important step is to determine which types of unseen data are suitable for a specific problem (Step 2). In this paper, we show two types of unseen data, which are useful in semi-supervised learning and novelty detection. However, determining all types of unseen data for all applications is beyond the scope of this study, and we leave this for future work.
Furthermore, we provide a method (see Appendix B in supplementary materials) by reformulating the objective function (2), so that it is more stable to train the DSGAN.

heading: 3 THEORETICAL RESULTS
text: In this section, we show that by choosing an appropriate α, the support set of pg belongs to the difference set between pd̄ and pd, so that the samples from pg are unseen from the pd perspective.
We start our proofs from two assumptions. First, in a non-parametric setting, we assume that both the generator and discriminator have infinite capacities. Second, pg is defined as the distribution of the samples drawn from G(z) under z ∼ pz . In the following, we show that the support set of pg is contained within the differences in the support sets of pd̄ and pd while achieving the global minimum such that we can generate the desired pg by designing an appropriate pd̄.
Theorem 1. Suppose αpd(x) ≥ pd̄(x) for all x ∈ Supp(pd) and all density functions pd(x), and pd̄(x) and pg(x) are continuous. If the global minimum of C(G) is achieved, then
Supp (pg) ⊆ Supp (pd̄)− Supp(pd), where C(G) = max
D V (G,D) = Ex∼pd̄(x) [ log
pd̄(x)
pd̄(x) + (1− α)pg(x) + αpd(x)
] + Ex∼p∗(x) [ log
(1− α)pg(x) + αpd(x) pd̄(x) + (1− α)pg(x) + αpd(x)
] .
Proof. See Appendix C for the details.
Summarizing, the generator is prone to output samples that are located in the high-density areas of pd̄ − αpd.

heading: 4 APPLICATIONS
text: The DSGAN was applied to two problems: semi-supervised learning and novelty detection. In the semi-supervised learning, the DSGAN acts as a “bad generator,” which creates complement samples (unseen data) in the feature space of the training data. For the novelty detection, the DSGAN generates the samples (unseen data) as boundary points around the training data.

heading: 4.1 SEMI-SUPERVISED LEARNING
text: Semi-supervised learning (SSL) is a type of learning model that uses a few labeled data and numerous unlabeled data. The existing SSL methods based on a generative model, (e.g., VAE Kingma et al. (2014) and GAN Salimans et al. (2016)), yield good empirical results. Dai et al. (2017) theoretically showed that a good semi-supervised learning required a bad GAN with the following objective function: max D Ex,y∼L logPD (y | x, y ≤ K) + Ex∼pd(x) logPD (y ≤ K | x) + Ex∼pg(x) logPD (K + 1 | x) , (3)
where (x, y) denotes a pair of data, and its corresponding label, {1, 2, . . . ,K} denotes the label space for the classification, and L = {(x, y)} is the label dataset. Moreover, under the semi-supervised settings, pd in (3) is the distribution of the unlabeled data. Note that the discriminator, D, in GAN also plays the role of a classifier. If the generator distribution exactly matches the real data distribution (i.e., pg = pd), then the classifier trained by the objective function (3) with the unlabeled data cannot have a better performance than that trained by the supervised learning with the objective function. Specifically,
max D Ex,y∼L logPD (y | x, y ≤ K) . (4) Contrastingly, the generator is preferred to generate complement samples, which lie on the lowdensity area of pd. Under some mild assumptions, these complement samples help D to learn the correct decision boundaries in the low-density area because the probabilities of the true classes are forced to be low in the out-of-distribution areas.
The complement samples in Dai et al. (2017) are complex to produce. In Sec. 5.2, we will demonstrate that with the DSGAN, complement samples can be easily generated.

heading: 4.2 NOVELTY DETECTION
text: Novelty detection determines if a query example belongs to a seen class. If the samples of one seen class are considered as positive data, then this difficulty is the absence of negative data in the training phase, so that the supervised learning cannot function.
Recently, novelty detection has made significant progress with the advent of deep leaning. Pidhorskyi et al. (2018)Sakurada & Yairi (2014) focused on learning a representative latent space for a seen class. When testing, the query image was projected onto the learned latent space. Then, the difference between the query image and its inverse image (reconstruction) was measured. Thus, only an encoder was needed to be trained for the projection and a decoder for the reconstruction. Under the circumstance, an autoencoder (AE) is generally is adopted to learn both the encoder and decoder Pidhorskyi et al. (2018)Perera et al. (2019). Let Enc(·) be the encoder and Dec(·) be the decoder. The loss function of the AE is defined as
min Enc,Dec
Ex∼ppos(x) [ ‖x−Dec(Enc(x))‖22 ] , (5)
where ppos is the distribution of a seen class. After the training, a query example, xtest, is classified as the seen class if ‖xtest −Dec(Enc(xtest))‖22 ≤ τ, (6) where τ ∈ R+ plays the trade-off between the true positive rate and false positive rate. However, (6) is based on two assumptions: (1) the positive samples from one seen class should have a small reconstruction error; (2) the AE (or latent space) cannot well describe the negative examples from the unseen classes, leading to a relatively large reconstruction error. In general, the first assumption inherently holds when both the testing and training data originate from the same seen class. However, Pidhorskyi et al. (2018)Perera et al. (2019) observed that assumption (2) does not hold at all times because the loss function in (5) does not include a loss term to enforce the negative data to have a large reconstruction error.
For assumption (2) to hold, given positive data as the training inputs, we propose using the DSGAN to generate negative examples in the latent space, as discussed in Sec. 5.3. Then, the loss function of the AE is modified to enforce the negative data to have a large reconstruction error.

heading: 5 EXPERIMENTS
text: Our experiments are divided into three parts. The first one examines how the hyperparameter, α, influences the learned generator distribution, pg. In the second and third experiments, we obtain empirical results about semi-supervised learning and novelty detection, which are presented in Sec. 5.2 and Sec. 5.3, respectively. Note that the training procedure of the DSGAN can be improved by other extensions of GANs such as WGAN Arjovsky et al. (2017), WGAN-GP Gulrajani et al. (2017), EBGAN Zhao et al. (2017), and LSGAN Mao et al. (2017). In our method, the WGAN-GP was adopted for the stability of the DSGAN in training and reduction in the mode collapse.
5.1 DSGAN WITH DIFFERENT α
The impacts of different α values on the DSGAN are illustrated in Fig. 7. In this example, the support of pd is the area bounded by a red dotted line, and the orange points are the samples from pd. Concurrently, we shift pd to the right by 1 unit and create the distribution, pd̄, whose support is bounded by blue dotted lines. The overlapping area between pd̄ and pd is 0.5 unit (assuming the area of pd is 1 unit). Based on our theoretical results, α = 0.5 is the smallest selected value allowing pg to be disjoint to pd. Therefore, we can see that some generated samples, as presented in Fig. 7(a), still belong to the support set of pd. Fig. 7(b) shows that there is a perfect agreement between our theoretical and experiment results with α = 0.5. When α = 0.8, there is a remarkable gap between the generated (green) points and yellow points, as shown in Fig. 7(c). In theory, the result obtained at α = 0.8 should be the same as that obtained at α = 0.5. This is because the discriminator should assign the entire area, which is the intersection of the complement of support set of pd and support set of pd̄, to the same score, under the assumption that the discriminator has an infinite capacity. However, in practice, the capacity of the discriminator is limited. Therefore, the score of the area near pd is lower than that far from it, when α is large. Therefore, pg tends to repel pd to achieve a high score (to deceive the discriminator). 5.2 DSGAN IN SEMI-SUPERVISED LEARNING We first introduce how the DSGAN generates the complement samples in the feature space. Dai et al. (2017) proved that if the complement samples generated by G could satisfy the following two assumptions in (7) and (8), i.e.,
∀x ∼ pg(x), 0 > max 1≤i≤K wTi f(x) and ∀x ∼ pd(x), 0 < max 1≤i≤K wTi f(x), (7)
where f is the feature extractor and wi is the linear classifier for the ith class, and
∀x1 ∼ L, x2 ∼ pd(x),∃xg ∼ pg(x) s.t. f(xg) = βf(x1) + (1− β)f(x2) with β ∈ [0, 1],
(8)
then all the unlabeled data would be correctly classified by the objective function (3). Specifically, (7) ensures that the classifiers can discriminate the generated data from the unlabeled data, and (8) causes the decision boundary to be located in the low-density areas of pd.
The assumption in (8) implies that the complement samples must be in the space created by the linear combination of the labeled and unlabeled data. In addition, they cannot fall into the real data distribution, pd, owing to the assumption (7). To allow the DSGAN to generate such samples, we let the samples of pd̄ be linear combinations of those from L and pd. Since pg(x) ≈ pd̄(x)− αpd(x)
1− α ,
pg will tend to match pd̄, whereas the term, −αpd, ensures that the samples from pg do not belong to pd. Thus, pg satisfies the assumption in (8). Moreover, (7) is also satisfied by training the classifier with (3) based on substituting the generator distribution in (3) into the learned pg .
Following the previous works, we apply the proposed DSGAN to semi-supervised learning on three benchmark datasets: MNIST LeCun et al. (1998), SVHN Netzer et al. (2011), and CIFAR-10 Krizhevsky (2009). The details of the experiments can be found in Appendix D.

heading: 5.2.1 SIMULATION RESULTS
text: First, the selected hyperparameters are listed in Table 5 in Appendix D.1. Second, the results obtained from the DSGAN and state-of-the-art methods on the three benchmark datasets are summarized in Table 1. It can be observed that our method can compete with the state-of-the-art methods on the three datasets. Note that we report the results of badGAN not only from the original papers in the literature but also by reproducing them using the released codes of the authors. The reason of presenting both the results is that we cannot reproduce parts of the results. The experiments in Li et al. (2019) also showed a similar problem. In comparison with Dai et al. (2017), our methods do not need to rely on an additional density estimation network, PixelCNN++ Salimans et al. (2017). Although PixelCNN++ is one of the best density estimation networks, learning such a deep architecture requires large computation and high memory consumption. In Table 2, we list the training time and memory consumption for our method and badGAN. Compared to badGAN, our method consumes 15.8% less training time and saves about 9000 MB during the training.
Moreover, it can also be observed from Table 1 that our results are comparable to the best record of badGAN and CAGAN. are better than those of other approaches on the MNIST and SVHN datasets. On CIFAR-10, our method is only inferior to the CT-GAN. However, this might not be a reasonable comparison because the CT-GAN uses extra techniques, including temporal ensembling and data augmentation, which the other methods do not use.

heading: 5.3 DSGAN IN NOVELTY DETECTION
text: In this section, we study how to use the DSGAN for assisting novelty detection. As mentioned in Sec. 4.2, we need to train the auto-encoder (AE) such that (i) the positive samples from one seen class have a small reconstruction error; (ii) negative samples from the unseen classes incur relatively higher reconstruction errors.
The fundamental concept is to use the DSGAN to generate negative samples, which originally do not exist under the scenario of novelty detection. Next, we add a new loss term to penalize the small reconstruction errors of the negative samples (see the third stage below). Three stages are required to train our model (AE):
1. The encoder, Enc(·), and decoder, Dec(·), are trained using the loss function (5). 2. Given x ∼ ppos, Enc(x) are collected as the samples drawn from pd. pd̄ is the convolution
of pd having a normal distribution with a zero mean and variance σ. Then, we train the DSGAN to generate negative samples, which are drawn from pd̄(x) − pd(x) and are the boundary points around the positive samples in the latent space. Note that there are some variations in the DSGAN: the input of the generator, G, is Enc(x), instead of a random vector z in the latent space. We also add ‖Enc(x)−G(Enc(x))‖22, which will be explained in the next step, to train the generator.
3. Fixing the encoder, we retrain the decoder by the modified loss function,
min Dec
Ex∼ppos(x) [ ‖x−Dec(Enc(x))‖22 + w ·max ( 0,m− ‖x−Dec(G(Enc(x)))‖22 ]) ,
where w is the trade-off between the reconstruction errors of positive samples Enc(x) and negative samples G(Enc(x)). Note that in the previous step, we add ‖Enc(x) − G(Enc(x))‖22 to ensure that the outputs of the generator are around the input. Thus, the second term charges even though the negative samples are close to the corresponding positive sample, and they still exhibit a high reconstruction error, which is bounded by m (Zhao et al. (2017)).
The above algorithm, called VAE+DSGAN, can be used to strengthen the existing AE-based methods by using them in the first stage. In the simulation, we used a variational autoencoder (VAE) Kingma & Welling (2014a) because it performs better than the AE in the novelty detection.

heading: 5.3.1 SIMULATION RESULTS
text: In this section, following Perera et al. (2019), the performance was evaluated using the area under the curve (AUC) of the receiver operating characteristics (ROC) curve. Given a dataset, a class was chosen as the seen class for training, and all the classes were used for testing. There exist several testing benchmarks for novelty detection, such as MNIST, COIL100 Nene et al. (1996) and CIFAR-10. The state-of-the-art method Perera et al. (2019) achieves high performance in AUC on MNIST and COIL100 (AUC is larger than 0.97). However, for CIFAR-10, Perera et al. (2019) only
achieves 0.656. Thus, we chose the challenging dataset, CIFAR-10, as the benchmark to evaluate our method. The detailed network architecture can be found in Appendix E.
Because VAE+DSGAN can be considered as a fine tuning VAE Kingma & Welling (2014a), we first illustrate the key difference between the VAE and VAE+DSGAN, as shown in Fig. 8. The seen class, which is at the bottom of the images, is a car. Other rows are the images from the unseen classes. One can see that the reconstructed images are reasonably good even for the unseen class in the VAE. By contrast, our method enforces the reconstructed images of the unseen classes to be blurred while still preserving the reconstruction quality of the seen class. Thus, our method achieves a relatively larger gap, in terms of the reconstruction error between the seen data and unseen data, than the VAE.
In Table 3, we compare the proposed method with several methods, including the VAE Kingma & Welling (2014a), AND Abati et al. (2019), DSVDD Ruff et al. (2018), and OCGAN Perera et al. (2019), in terms of the AUC value. One can see that in most cases, our method almost outperforms the VAE. Furthermore, the mean of the AUC values of our method also is larger than those of the state-of-the-art methods. It is worth mentioning that in addition to the VAE, the DSGAN has potential of being combined with other AE-based methods.

heading: 6 RELATED WORKS ABOUT UNSEEN DATA GENERATION
text: Yu et al. (2017) proposed a method to generate samples of unseen classes in a unsupervised manner via an adversarial learning strategy. However, it requires solving an optimization problem for each sample, which certainly leads to a high computation cost. By contrast, the DSGAN has the capability to create infinite diverse unseen samples. Hou et al. (2018) presented a new GAN architecture that could learn two distributions of unseen data from a part of seen data and the unlabeled data. However, the unlabeled data must be a mixture of seen and unseen samples; the DSGAN does not require any unseen data. Kliger & Fleishman (2018) also applied GAN in novelty detection. Their objective
was to learn a generator whose distribution is a mixture of novelty data distribution and training data distribution. To this end, they used feature matching (FM) to train the generator and expected pg to learn the mixture of distributions. However, the ultimate goal of FM is still to learn pg = pd; therefore, their method might fail when GAN learns well.
Dai et al. (2017) aimed to generate complementary samples (or out-of-distribution samples), but assumed that the in-distribution could be estimated by a pre-trained model, such as PixelCNN++, which might be difficult and expensive to train. Lee et al. (2018) used a simple classifier to replace the role of PixelCNN++ in Dai et al. (2017) so that the training was comparatively much easier and more suitable. Nevertheless, their method only focused on generating unseen data surrounding the low-density area of seen data. In comparison, the DSGAN has more flexibility to generate different types of unseen data (e.g., a linear combination of seen data, as described in Sec. 5.2). In addition, their method needs the label information of the data, whereas our method is fully unsupervised.

heading: 7 CONCLUSIONS
text: We propose the DSGAN, which can produce any unseen data based on the assumption that the density of the unseen data distribution is the difference between the densities of any two distributions. The DSGAN is useful in an environment when the samples from the unseen data distribution are more difficult to collect than those from the two known distributions. Empirical and theoretical results are provided to validate the effectiveness of the DSGAN. Finally, because the DSGAN is developed based on GAN, it is easy to apply any improved versions of GAN to the DSGAN.

heading: 8 ACKNOWLEDGEMENT
text: This work was partially supported by grants MOST 107-2221-E-001-015-MY2 and MOST 108-2634- F-007-010 from Ministry of Science and Technology, Taiwan, ROC.

heading: A FLOWCHART AND ALGORITHM OF DSGAN
text: Algorithm 1 The training procedure of DSGAN using minibatch stochastic gradient descent. k is the number of steps applied to discriminator. α is the ratio between pg and pd in the mixture distribution. We used k = 1 and α = 0.8 in experiments.
01. for number of training iterations do 02. for k steps do 03. Sample minibatch of m noise samples z(1), ..., z(m) from pg(z). 04. Sample minibatch of m samples x(1)d , ..., x (m) d from pd(x). 05. Sample minibatch of m samples x(1) d̄ , ..., x (m) d̄ from pd̄(x). 06. Update the discriminator by ascending its stochastic gradient:
∇θd
[ 1
m m∑ i=1 logD ( x (i) d ) + log ( 1−D ( G ( z(i) ))) + log ( 1−D ( x (i) d̄ ))] 07. end for 08. Sample minibatch of m noise samples z(1), ..., z(m) from pg(z). 09. Update the generator by descending its stochastic gradient:
∇θg 1
m m∑ i=1 [ log ( 1−D ( G ( z(i) )))]
10. end for

heading: B TRICKS FOR STABLE TRAINING
text: We provide a trick to stabilize the training procedure by reformulating the objective function. Specifically, V (G,D) in (2) is reformulated as:
V (G,D) = ∫ x pd̄(x) log (D (x))
+ ((1− α)pg(x) + αpd(x)) log (1−D (x)) dx = Ex∼pd̄(x) [logD(x)] + Ex∼(1−α)pg(x)+α∼pd(x) [log (1−D (x))] .
(9)
Instead of sampling a mini-batch of m samples from pz and pd in Algorithm 1, (1− α)m and αm samples from both distributions are required, respectively. The computation cost in training can be reduced due to fewer samples. Furthermore, although (9) is equivalent to (2) in theory, we find that the training using (9) achieves better performance than using (2) via empirical validation in Table 4. We conjecture that the equivalence between (9) and (2) is based on the linearity of expectation, but mini-batch stochastic gradient descent in practical training may lead to the different outcomes.

heading: C PROOF OF THEOREM 1
text: In this section, we show Theorem 1.
This proof includes two parts: the first part shows that the objective function is equivalent to minimizing the Jensen–Shannon divergence in the mixture distribution (pd and pg) and pd̄ if G and D are assigned sufficient capacity; the second part shows that by choosing an appropriate α, the support set of pg belongs to the difference set between pd̄ and pd, so that the samples from pg are unseen from the pd perspective.
For the first part, we show the optimal discriminator givenG, and then show that minimizing V (G,D) via G, given the optimal discriminator, is equivalent to minimizing the Jensen–Shannon divergence between (1− α)pg + αpd and pd̄.
Proposition 1. If G is fixed, the optimal discriminator, D, is
D∗G(x) = pd(x)
pd(x) + (1− α)pg(x) + αpd(x) .
Proof. Given any generator G, the training criterion for the discriminator D is to maximize the quantity V (G,D):
V (G,D) = ∫ x pd̄(x) log (D (x)) dx
+ (1− α) ∫ z pz(z) log (1−D (G (z))) dz
+ α ∫ x pd(x) log (1−D (x)) dx
= ∫ x pd̄(x) log (D (x)) dx
+ (1− α) ∫ x pg(x) log (1−D (x)) dz
+ α ∫ x pd(x) log (1−D (x)) dx
= ∫ x pd̄(x) log (D (x))
+ ((1− α)pg(x) + αpd(x)) log (1−D (x)) dx.
For any (a, b) ∈ R2\{0, 0}, the function a log (y) + b log (1− y) achieves its maximum in [0, 1] at y = aa+b . The discriminator only needs to be defined within Supp(pd̄) ⋃ Supp(pd) ⋃ Supp(pg). We complete this proof.
Moreover, D can be considered to discriminate between samples from pd̄ and ((1− α)pg(x) + αpd(x)). By replacing the optimal discriminator in V (G,D), we trivially obtain
C(G) = max D V (G,D) = Ex∼pd̄(x) [ log
pd̄(x)
pd̄(x) + (1− α)pg(x) + αpd(x)
] + Ex∼p∗(x) [ log
(1− α)pg(x) + αpd(x) pd̄(x) + (1− α)pg(x) + αpd(x)
] .
(10)
Actually, the results thus far yield the optimal solution of D given G is fixed in (1). Now, the next step is to determine the optimal G with D∗G as fixed.
Theorem 2. The global minimum of C(G) is achieved if and only if (1−α)pg(x)+αpd(x) = pd̄(x) for all x. Then, C(G) achieves the value, − log 4.
Proof. We start from
(1) = − log(4) + Ex∼pd̄(x) [ log
2pd̄(x)
pd̄(x) + (1− α)pg(x) + αpd(x) ] + Ex∼p∗(x) [ log
2 ((1− α)pg(x) + αpd(x)) pd̄(x) + (1− α)pg(x) + αpd(x) ] = − log(4) + KL ( pd̄ ∥∥∥∥ pd̄ + (1− α)pg + αpd2 )
+KL ( (1− α)pg(x) + αpd ∥∥∥∥ pd̄ + (1− α)pg + αpd2 )
= − log(4) + 2 JSD (pd̄ ‖ (1− α)pg + αpd) ,
where p∗(x) = (1 − α)pg(x) + αpd(x), KL is the Kullback-Leibler divergence and JSD is the Jensen-Shannon divergence. The JSD returns the minimal value, which is 0, iff both distributions are the same, namely pd̄ = (1− α)pg + αpd. Because pg(x)’s are always non-negative, it should be noted both distributions are the same only if αpd(x) ≤ pd̄(x) for all x’s. We complete this proof.
Note that (1− α)pg(x) + αpd(x) = pd̄(x) may not hold if αpd(x) > pd̄(x). However, the DSGAN still works based on two facts: i) given D, V (G,D) is a convex function in pg and ii) because∫ x pg(x)dx = 1, the set collecting all the feasible solutions of pg is convex. Thus, there always exists a global minimum of V (G,D) given D, but it may not be − log(4). Now, we go back to prove Theorem 1. We show that the support set of pg is contained within the differences in the support sets of pd̄ and pd while achieving the global minimum such that we can generate the desired pg by designing an appropriate pd̄.
Proof. Recall that
C(G) = ∫ x pd̄(x) log ( pd̄(x) pd̄(x) + (1− α)pg(x) + αpd(x) ) + p∗(x) log ( (1− α)pg(x) + αpd(x)
pd̄(x) + (1− α)pg(x) + αpd(x)
) dx
= ∫ x S(pg;x)dx
= ∫ x∈Supp(pd̄)−Supp(pd) S(pg;x)dx
+ ∫ x∈Supp(pd) S(pg;x)dx.
S(pg;x) is used to simplify the notations inside the integral. For any x, S(pg;x) in pg(x) is nonincreasing and S(pg;x) ≤ 0 always holds. Specifically, S(pg;x) is decreasing along the increase of pg(x) if pd̄(x) > 0; S(pg;x) attains the maximum value, zero, for any pg(x) if pd̄(x) = 0. Since
DSGAN aims to minimize C(G) with the constraint ∫ x pg(d)dx = 1, the solution attaining the global minima must satisfy pg(x) = 0 if pd̄(x) = 0; otherwise, there exists another solution with smaller value of C(G). Thus, Supp (pg) ⊆ Supp (pd̄).
Furthermore, T (pg;x) = ∂S(pg;x)
∂pg(x) = log
( (1− α)pg(x) + αpd(x)
pd̄(x) + (1− α)pg(x) + αpd(x)
) , which is expected
to be as small as possible to minimizeC(G), is increasing on pg(x) and converges to 0. Then, we show that T (pg;x) for x ∈ Supp(pd̄) ⋂ Supp(pd) is always larger than that for x ∈ Supp(pd̄)−Supp(pd) for all pg . Specifically,
1. When x ∈ Supp(pd̄) ⋂ Supp(pd), T (pg;x) ≥ log 12 always holds due to the assumption of
αpd(x) ≥ pd̄(x).
2. When x ∈ Supp(pd̄) − Supp(pd), T (pg;x) < log 12 for all pg(x)’s satisfying (1 − α)pg(x) ≤ pd̄(x).
Thus, the minimizer prefers pg(x) > 0 for x ∈ Supp(pd̄) − Supp(pd) and (1 − α)pg(x) ≤ pd̄(x). We check whether there exists a solution pg such that (1 − α)pg(x) ≤ pd̄(x) and∫ x∈Supp(pd̄)−Supp(pd) pg(d)dx = 1, implying pg(x) = 0 for x ∈ Supp(pd̄) ⋂ Supp(pd). Based
on the following expression,∫ x∈Supp(pd̄)−Supp(pd) pd̄(x)dx + ∫ x∈Supp(pd) pd̄(x)dx = 1
⇒ ∫ x∈Supp(pd̄)−Supp(pd) pd̄(x)dx
≥ 1− ∫ x∈Supp(pd) αpd(x)dx
⇒ ∫ x∈Supp(pd̄)−Supp(pd) pd̄(x)dx ≥ 1− α
⇒ ∫ x∈Supp(pd̄)−Supp(pd) pd̄(x)dx
≥ ∫ x∈Supp(pd̄)−Supp(pd) (1− α)pg(x)dx,
the last inequality implies that there must exist a feasible solution. We complete this proof.
Another concern is the convergence of Algorithm 1.
Proposition 2. The discriminator reaches its optimal value givenG in Algorithm 1, and pg is updated by minimizing
Ex∼pd̄(x) [logD ∗ G(x)] + Ex∼p∗(x) [log (1−D∗G (x))] .
If G and D have sufficient capacities, then pg converges to argmin pg JSD (pd̄ ‖ (1− α)pg + αpd).
Proof. Consider V (G,D) = U(pg, D) as a function of pg. By the proof idea of Theorem 2 in Goodfellow et al. (2014), if f(x) = supα∈A fα(x) and fα(x) is convex in x for every α, then ∂fβ(x) ∈ ∂f if β = argsupα∈A fα(x). In other words, if supD V (G,D) is convex in pg, the subderivatives of supD V (G,D) includes the derivative of the function at the point, where the maximum is attained, implying the convergence with sufficiently small updates of pg . We complete this proof.

heading: D EXPERIMENTAL DETAILS FOR SEMI-SUPERVISED LEARNING
text: D.0.1 DATASETS: MNIST, SVHN, AND CIFAR-10
For evaluating the semi-supervised learning task, we used 60000/ 73257/ 50000 samples and 10000/ 26032/ 10000 samples from the MNIST/ SVHN/ CIFAR-10 datasets for the training and testing, respectively. Under the semi-supervised setting, we randomly chose 100/ 1000/ 4000 samples from the training samples, which are the MNIST/ SVHN/ CIFAR-10 labeled datasets, and the amounts of the labeled data for all the classes are equal. Furthermore, our criterion to determine the hyperparameters is introduced in Appendix D.1, and the network architectures are described in Appendix D.2. We performed testing with 10/ 5/ 5 runs on MNIST/ SVHN/ CIFAR-10 based on the selected hyperparameters, and randomly selected the labeled dataset. The results were recorded as the mean and standard deviation of the number of errors from each run.
D.1 HYPERPARAMETERS
The hyperparameters were chosen to make our generated samples consistent with the assumptions in (7) and (8). However, in practice, if we make all the samples produced by the generator following the assumption in (8), then the generated distribution is not close to the true distribution, even a large margin between them exists, which is not what we desire. So, in our experiments, we make a concession that the percentage of generated samples, which accords with the assumption, is around 90%. To meet this objective, we tune the hyperparameters. Table 5 shows our setting of hyperparameters, where β is defined in (8).
In order to fairly compare with other methods, our generators and classifiers for MNIST, SVHN, and CIFAR-10 are same as in Salimans et al. (2016) and Dai et al. (2017). However, different from previous works that have only a generator and a discriminator, we design an additional discriminator in the feature space, and its architecture is similar across all datasets with only the difference in the input dimensions. Following Dai et al. (2017), we also define the feature space as the input space of the output layer of discriminators.
Compared to SVHN and CIFAR-10, MNIST is a simple dataset as it is only composed of fully connected layers. Batch normalization (BN) or weight normalization (WN) is used in every layer to stable training. Moreover, Gaussian noise is added before each layer in the classifier, as proposed in Rasmus et al. (2015). We find that the added Gaussian noise exhibits a positive effect for semisupervised learning. The architecture is shown in Table 6.
Table 7 and Table 8 are models for SVHN and CIFAR-10, respectively, and these models are almost the same except for some implicit differences, e.g., the number of convolutional filters and types of dropout. In these tables, given a dropping rate, “Dropout” denotes a normal dropout in that the elements of input tensor are randomly set to zero while Dropout2d is a dropout only applied on the channels to randomly zero all the elements.
Furthermore, the training procedure alternates between k steps of optimizing D and one step of optimizing G. We find that k in Algorithm 1 is a key role in the problem of mode collapse for different applications. For semi-supervised learning, we set k = 1 for all datasets.

heading: E EXPERIMENTAL DETAILS FOR NOVELTY DETECTION
text: The architecture of GAN and VAE are depicted in Table 9 and 10, respectively.
In the experiment, we first trained the VAE for 500 epochs and then we trained DSGAN for 500 epochs with m = 1.5 and w = 0.5. Third, we fixed the encoder and tuned the decoder with both positive and negative samples (generated by DSGAN) for 600 epochs.
F ABLATION STUDY ON DIFFERENT α VALUES FOR SEMI-SUPERVISED LEARNING
Fig. 7 shows how different α values influence DSGAN. The optimal α for DSGAN to generate “unseen” data depends on pd̄ and pd. According to Fig. 7, we can figure out that DSGAN is prone to generating unseen data under a larger α. Recall that Theorem 1 illustrates α should be expected to be as large as possible if both network G and D have infinite capacity. Though the networks never have the infinite capacity in real applications, a general rule is to pick a large α and force the complement data to be far from pd, which is similar to the results in Sec. 5.1.
Table 10: The architectures of VAE for novelty detection.
Encoder Decoder
5× 5 conv. 32 stride = 2, with BN, lReLU(0.2) 5× 5 conv. 64 stride = 2, with BN, lReLU(0.2) 5× 5 conv. 128 stride = 2, with BN, lReLU(0.2) (For mean) 4× 4 conv. 128 stride = 1 (For std) 4× 4 conv. 128 stride = 1 5× 5 conv. transpose 128 stride = 2 with BN, lReLU(0.2) 5× 5 conv. transpose 64 stride = 2 with BN, lReLU(0.2) 5× 5 conv. transpose 32 stride = 2 with BN, lReLU(0.2) 5× 5 conv. transpose 3 stride = 2, Tanh
Here, we conduct the experiments on different α under semi-supervised learning settings. From Sec. 4.1 and 5.2, badGAN already shows that, if the desired unseen data can be generated, then the classifier will put the correct decision boundary in the low-density area.
In Table 11, we demonstrate the classification results on α = 0.5 and α = 0.8, respectively. We can observe that the results generated at α = 0.8 is better than those generated at α = 0.5, meeting the above discussion. From our empirical observations, DSGAN is prone to generating unseen data at α = 0.8, leading a better classifier.

heading: G SAMPLE QUALITY OF DSGAN ON CELEBA
text: We show one more experiment on CelebA (Liu et al. (2015)) to demonstrate DSGAN can work well even for complicated images. In this experiment, we generate the color images of size 64 × 64. Similar to our 1/7 experiments on the MNIST dataset, we let pd̄ be the distribution of face images with glasses and without glasses and let pd be the distribution of images without glasses. We validate DSGAN with α = 0.5 and α = 0.8, respectively. For α = 0.5, we sample 10000 images with glasses and 10000 images without glasses from CelebA. When α is 0.8, we sample 40000 instead of 10000 images without glasses.
We also train GAN to verify the generated image quality of DSGAN. For fair comparison, GAN is trained under two kinds of settings. The first one is that GAN is only trained with the images with glasses. Second, it is pretrained with all images, and is finetuned with the images with glasses, namely transferring GANs in Wang et al. (2018). It should be noted that transferring GAN uses the same amount of training data as DSGAN and serves as a stronger baseline than GAN under the first setting.
Fréchet Inception Distance (FID) (Heusel et al. (2017)) is used to evaluate the quality of generated images. FID calculates the Wasserstein-2 distance between generated images and real images (images with glasses) in the feature space of Inception-v3 network (Szegedy et al. (2015)). We train both networks for 600 epochs, and use WGAN-GP as the backbone for both GAN and DSGAN. In addition, transferring GANs are pretrained for 500 epochs, then being finetuned for 600 epochs.
Fig. 10 and Table 12 show generated images and FID for all methods, respectively. We can see that our DSGAN can generate images with glasses from the given pd and pd̄, and the FID of DSGAN are comparable to that of GAN. The experiment validates that DSGAN still works well to create complement data for complicate images.

Review summaries:
This paper proposed DSGAN which learns to generate unseen data from seen data distribution p_d and its somehow “broad” version p_{\hat d} (E.g., p_d convolved with Gaussian). The “unseen data” is the one that appears in p_{\hat d} but not in p_d. DSGAN is trained to generate such data. In particular, it uses samples from p_d as fake data and samples from p_{\hat d} as the real one. 

This paper provides an interesting application of GAN which can generate the outlier distribution of training data which forces generator to learn the distribution of the low probability density area of given data. To show the effectiveness of the method, the author intuitively shows how it works on 2-D points data as well as the reconstructed Mnist dataset. Additionally, this approach reaches a comparable performance on semi-supervised learning and novelty detection task.