Differential-Critic GAN: Generating What You Want by a Cue of Preferences
Authors: Yinghua Yao
Yuangang Pan
Ivor Tsang
Xin Yao
Venue:  (2021)
Keywords: GAN
user-desired data distribution
user preference
critic

Abstract: This paper proposes Differential-Critic Generative Adversarial Network (DiCGAN) to learn the distribution of user-desired data when only partial instead of the entire dataset possesses the desired properties. Existing approaches select the desired samples first and train regular GANs on the selected samples to derive the user-desired data distribution. DiCGAN introduces a differential critic that can learn the preference direction from the pairwise preferences over the entire dataset. The resultant critic would guide the generation of the desired data instead of the whole data. Specifically, apart from the Wasserstein GAN loss, a ranking loss of the pairwise preferences is defined over the critic. It endows the difference of critic values between each pair of samples with the pairwise preference relation. The higher critic value indicates that the sample is preferred by the user. Thus training the generative model for higher critic values would encourage generating the user-preferred samples. Extensive experiments show that our DiCGAN can learn the user-desired data distributions.

Under review as a conference paper at ICLR 2021
DIFFERENTIAL-CRITE GAN: GENERATING WHAT YOU WANT BY A CUE OF PREFERENCES
Anonymous authors Paper under double-blind review
ABSTRACT
This paper proposes Differential- Generative Adversarial Network (DiCGAN) to learn the distribution of user-desired data when only partial instead of the entire dataset possesses the desired properties. Existing approaches select the desired samples first and train regular GANs on the selected samples to derive the user- desired data distribution. DiCGAN introduces differential critic that can learn the preference direction from the pairwise preferences over the entire dataset. The resultant critic guides the generation of the desired data instead of the whole data. Specifically, apart from the Wasserstein GAN loss, a ranking loss of the pairwise preferences is defined over the critic. It endows the difference of critic values between each pair of samples with the pairwise preference relation. The higher critic value indicates that the sample is preferred by the user. Thus training the generative model for higher critic values encourages the generation of user- preferred samples. Extensive experiments show that our DiCGAN can learn the user-desired data distributions.
INTRODUCTION
Learning good generative model for high-dimensional natural signals, such images (Zhu et al., 2017), video (Vondrick et al., 2016) and audio (Fedus et al.] (018)) has long been one of the key milestones of machine learning. Powered by the learning capabilities of deep neural networks, generative adversarial networks (GANs) (Goodfellow et al. 2014) have brought the field closer to attaining this goal. Currently, GANs are applied a setting where the whole training dataset is of user interest. Therefore, regular GANs no longer meet our requirement when only partial instead of the entire training dataset possesses the desired properties (Killoran et al. 2017). is more challenging when the given dataset has a small number of desired data.
Adapting vanilla GAN to this setting, naive way to first select the samples possessing the desired properties and then perform regular GAN training only on the selected samples to derive the desired distribution. However, vanilla GAN fails when the desired samples are limited. FBGAN overcomes the limited data problem by iteratively introducing desired samples from the generation into the training data. Specifically, FBGAN is pretrained with all training data using the vanilla GAN. In each training epoch, the generator first generates certain amounts of samples. The generated samples possessing the desired properties are selected by an expert selector and used to replace the old training data. Then, regular WGAN is trained with the updated training data. Since the ratio of the desired samples gradually increases in the training data, all training data will be replaced with the desired samples. Finally, FBGAN would derive the desired distribution when convergence. However, bluntly eliminating undesired samples may lead to a biased representation of the real desired data distribution. Because the undesired samples can also reveal useful clues about what is not desired. Suppose we want to generate old face images, however the training data contains only a few old face images whereas it has many young face images.
this case, the young face images can be used as negative sampling (Mikolov et al., 2013) to learn the subtle aging features (e.g. wrinkles, pigmented skin, etc.), which guides the generation of the desired old face images. The conditional variants of GAN, such as CGAN (Mirza and Osindero 2014) and ACGAN (Odena et al. (017) can be also applied in this setting by introducing condition variables to model the conditional desired data distribution. However, the generation performance of condition-bast the respective conditions with sufficient training observations. When the desired data is limited, the conditional modeling is dominated by the major classes, i.e., undesired data, resulting in failure
GAN is governed by

---

Under review as a conference paper at ICLR 2021
(a)
(b)
Real data
boundary
WGAN critic
direction
boundury
DiCGAN critic
0
direction
Undesired data
Preference direction
Fake data
Figure 1: Illustration of why DiCGAN can learn the user-desired data distribution. (a) DiCGAN's critic pushes fake data towards the real desired data while WGAN's critic pushes fake data towards the entire real data. (b) The change of DiCGAN's critic direction is driven by the preference direction. Note that the preference direction learned from all pairwise preferences.
to capture the desired distribution. All the literature methods require user-defined criteria to select the desired data in order to learn the distribution of the desired data, which may not exist in real applications.
Instead of soliciting ready-to-use criteria, consider guided towards the distribution user-desired data by the user preference. In particular, pairwise preferences the most popular form of user preference due to their simplicity and easy accessi- bility (Lu and Boutilier 2011). Therefore, our target is to incorporate pairwise preferences into the learning process of GAN, so guide the generation of the desired data.
general setting where GAN can be
Relativistic GAN (RGAN) Molicoeur-Martinee 2019) is a variant of regular GAN and is proposed learn the whole data distribution. It considers the critic value the indicator of sample quality and defines the discriminator using the difference in the critic values. The critic value in RGAN is similar to the ranking score, but it is used to describe sample quality. Motivated by this, we consider taking the critic value as the ranking score and define the ranking loss for pairwise preferences based on the critic value directly. In particular, the difference in critic values for each pair of samples reflects the user's preference over the samples. This is why we call our critic the differential critic, and we propose Differential-Critto GAN (DiCGAN) for learning the user-desired data distribution. As shown in Fig. [ the differential critic incorporates the preference direction, which pushes the original critic direction towards the real desired data region instead of the entire real data region. The main contributions are summarized follows:
We propose DiCGAN learn the distributions of the desired data from the entire data using pairwise preferences. To the best of our knowledge, this is the first work to promote the ratio of the desired data by incorporating user preferences directly into the data generation.
We introduce the differential critic by defining an additional pairwise ranking loss on the WGAN's critic. It endows the difference in the critic values between each pair of samples with user preferences.
The empirical study shows that DiCGAN learns the distribution of user-desired data and the differential critic can derive the preference direction even from limited umber of preferences.
2 GENERATIVE ADVERSARIAL NETWORKS
Generative Adversarial Network (GAN) (Goodfellow et al. 2014) performs generative modeling by learning a map from low-dimensional input space Z to data space X' i.e., Go 2 - X, given samples from the training data distribution, namely, x ~ Pr (x). The goal
find 0 which achieves Ge (2). Let p(z) be the input noise
pz(x) pr (z), where pe (z) is the fake data distribution distribution and G indicate Gg. GAN defines a discriminator that is trained to discriminate real data from fake data guide the learning of G.
Wasserstein GAN (WGAN) (Arjovsky et al. 2017) proposes to use the Wasserstein metric as a critic, which measures the quality of fake data in terms of the distance between the real data distribution and the fake data distribution. The Wasserstein distance (W-distance) is approximated by the difference in the average critic values between the real data and the fake data. The empirical experiments show
2

---

Under review as a conference paper at ICLR 2021
that the W-distance between two distributions corresponds well to the quality of the generated data. WGAN's objective function is defined as follows:
min max (s) [D(x)] (p) (x)),
where D is the critic and satisfies 1-Lipschitz.
3 DICGAN FOR USER- DESSIDE DISTRIBUTION
No longer learning the distribution of the whole dataset, GAN is applied in a new scenario, where the distribution of the partial dataset is what we desire. User-desired data may refer to some certain class of data among multiple class datasets, or observations with/without some particular attributes. Such data can be induced from the user preference, which can be represented as an ordering relation between two or more samples in terms of the desired properties. We propose differential-crite GAN learn the desired data distribution from the user preferences along with the whole dataset.
3.1 LEARNING THE DISTRIBUTION OF USER-DESIRED DATA
Following the score-based ranking literature, we suppose that there exists numeric score associated with each sample, reflecting the user's s preference for the sample. A higher score indicates that its corresponding sample is preferred by the user. In detail, let f denote a score function that maps sample x to score f(x)). Then, if sample x is desired by the user, its score f(x) exceeds predefined
threshold c, namely. [=) > c) 1. I is sign function, which equals if its condition is true and 0 otherwise. For the sake of explanation, we use p; (z), Pa Pa();, (x to denote the distribution of the whole data, the user-desired data and the undesired data, respectively.
FBGAN Gupta and Zou 2019) was proposed to learn the distribution of the desired data pa pa(x). FB- GAN executes alternatively between two steps: (1) construct the desired dataset X, {>> [()(() > <) =, x pr (æ)}; (2) train GAN on Xa to derive (r)).. However, the assumption that the score function f is predefined in FBGAN may be too restrictive for real applications, where no universal and explicit criteria exists. Further, the definitions of the desired/undesired samples are highly dependent on the choice of the threshold e. The removal of the so-called undesired samples may result in biased representation of real desired data distribution.
Instead of relying on predefined score function, we propose learn the desired data distribution in straightforward manner from the user preference. Here, we consider a general auxiliary informa- tion, i.e., the pairwise preferences, to represent the user preference, due to its simplicity and easy accessibility. For any two samples 1 22 pr (2r), let x > 22 denote that 21 is preferred over 22 according to the user-defined criteria. Let X be the training samples, i.e., X {ar ~ px))). A collection of pairwise preferences S is obtained by:
(2) Definition (Problem Setting). Given the training samples and the pairwise preferences S, the target is to learn a generative model pa (z) that is identical to the distribution of the desired data pa(x), i.e., (x(x) = Pa()).
S = { (I1. [2)](2) > F2, 31, D2 E X).
3.2 DIFFERENTIAL CRITIC GAN
Instead of scANtccritic for quality assessment, we present the differential critic for modelling pairwise preferences. The differential critic can guide the generation of the user-desired data.
3.2. PAIRWISE PREFERENCE
In this section, we consider incorporating the pairwise preference into the training of GAN.
The score-based ranking model (Zhou et al., 2008) is used to model the pairwise preferences. It learns the score function f. of which the score value, called ranking score in the model, is the indicator of the user preference. Further, the difference of ranking scores can indicate the pairwise preference relation. That is, for any pair of samples I1, 22, if 21 > 22 then f(x1) f(x2) > 0 and vice versa. For any pairwise preference s : σ1 > 22, the ranking loss we consider is as follows:
h(s)) max (0, (f (x1) (x2)) + m),
(3)
3

---

Under review as a conference paper at ICLR 2021
where is the ranking margin. For other forms of ranking losses, the reader can refer to (Zhou et al. 2008).
Instead of learning the score function independent of GAN's training, we consider incorporating into GAN's training, guiding GAN towards the generation of the desired data. The critic in RGAN (Jolicoeur-Me Martineau 2019) is similar to the score function, where the critic values used to describe the sample quality. We are motivated to take the critic value as the ranking score and define the ranking loss on the critic value directly. In particular, the difference in the critic values for each pair of samples reflects the user's preference over the samples.
Remark (Pairwise regularization to the generator). It is possible to consider a pairwise regular- ization the generator. As the larget to learn the desired distribution, the regularization the generator can be used to make the critic values of the generated samples larger than those of the undesired samples. We construct the regularization with the principle similar as FBGAN. Specifically, a selector is first applied to give full ranking for the training data and then bottom K samples are picked up as the undesired samples. The pairwise preferences are then defined over the generated samples and the undesired samples.
3.2.2 LOss FUNCTION
We build DiCGAN based on WGAN and the pairwise ranking loss is defined over the WGAN's critic. The loss function for DiCGAN is defined as:
min max (x)(.) [D(x)] Ep (x) [D -
1 IS
Σ (b)] .ES
(4)
G
where hss) is the pairwise ranking loss (equation 3.. À is a balance factor, which will be discussed
further in section 3.3 Similar to WGAN, we formulate the objective for the differential critic Ln and the generator LG as:
Lo 1 =-=1 D(x') D(G(()))
-
1 The En(s))
h(s)),
1 LG - Σ D(G()))),
(5)
The
where b is the batch size. This is the same for the fake samples. ng is the number of pairs sampling from S.
The advantages of DiCGAN are twofold: (1) The introduced ranking loss in DiCGAN is defined on the critic value. Apart from WGAN, it can be casily applied to other GAN variants developed based on the critic, e.g., RGAN. (2) DiCGAN leverages the entire dataset. The pairwise preferences are constructed on the whole dataset. Thus the undesired samples are also utilized during the training.
In the following, we argue that the differential critic in DiCGAN can guide the generator to learn the user-desired distribution from two aspects. (1) As shown in Fig. D the differential critic in DiCGAN provides the direction towards the real desired data. We denote the critic direction as the moving direction of fake data, which is orthogonal to the decision boundary of the critic. Referring to equation 4 DiCGAN's critic loss consists of two terms: the vanilla WGAN loss and the ranking loss. The vanilla WGAN loss imposes the critic direction from the fake data to the real data. Mcanwhile, the ranking loss induces a user preference direction, which points from the undesired data to the desired data. Combining these two effects, the critic direction of DiCGAN targets for the region of the real desired data only. (2) DiCGAN assigns high critic values for user-desired data and promotes the generation of samples with high critic values. The vanilla WGAN loss encourages the critic to assign high critic values for real data and low critic values for fake data. Meanwhile, the ranking loss encourages high critic values to be assigned to the real desired data while low critic values are assigned to real undesired data. Therefore, the real desired data achieves high critic values from the critic. Similar to WGAN, the training paradigm of DiCGAN promotes the generation of samples with high critic values, which is equivalent to encouraging the generation of dseirde data in DiCGAN.
3.3 REFORMULATING DICGAN TO ENSURE DATA QUALITY
Let us revisit the objective of DICGAN (equation 4. The first two terms of equation can be considered the WGAN regularisation, which ensures the generated data distribution is close the
4

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Training algorithm of DiCGAN
1: input: training data X. pairwise preferences S
2: initilization; balance factor X, #generated samples ng. #pairs nh, batchsize b, #iterations per epoch ni
#critic iterations per generator iteration Meritic
3: Pretrain D and G
4: repeat
5:
% Shift to the user- user-pre preferred distribution
6: Generate samples using equation
7: Replace old samples in X with Xg using equation g
8:
Obtain pairwise preferences using equation
9:
% Training of D and G at an epoch
10:
for i 1, .......... do
11:
for 1, 1....... Periiii do
Sample (a [='):=1 from X. [=" ~ p(z)]i=1
13:
Sample (s) from S.
14:
Train the differential critic D using Ln in equation
16:
Train the generator G using La in equation 5
17:
end for
18:
until converge
whole real data distribution, i.e., Po R Pr. The third term serves as correction of WGAN, which makes WGAN slightly biased to our target of learning the desired data distribution, i.e., Po Pa-
Therefore, the WGAN regularisation serves the cornerstone of our DiCGAN. Particularly. if the desired data distribution close to the entire data distribution, the rank loss easily corrects the WGAN to achieve the desired data distribution. Otherwise, the satisfactory performance of DiCGAN may require the online hyperparameter tuning of during the training process.
According the above analysis, consider reformulating the objective of DiCGAN, i.., equation into an equivalent objective with a hard WGAN constraint:
min max-
[h (s)], s.t. d(pr, pe) (x) [D(x)] Epa(x) [x)]] S.
(6)
where E > O. Note that we impose an explicit non-negative constraint on d(pr. to highlight that it is distance metric. is still equivalent WGAN loss from its definition. Therefore, equation is the Lagrangian function. Since equation [6 imposes a hard constraint on the WGAN loss, it is more difficult optimize compared equation 4 However, more efficient solutions of DiCGAN can be explored by analyzing equation 6 regarding the hard constraint on d(pr. Pe).
In terms of minor correction situation, this means the desired data distribution pa close the real data distribution Pr- Therefore, the hard constraint dominates the training goal of DiCGAN. By assigning a proper À ensure the constraint is satisfied, equation can learn the distribution of the user-desired data while ensuring data quality.
In terms major correction situation, this means the desired data distribution pa is quite diverse from the real data distribution Pr- Therefore, DiCGAN needs to achieve an equilibrium between the correction, imposed by the ranking loss, and the hard constraint, imposed by the WGAN loss. However, a large correction may not ensure the quality of the generated data, since the WGAN loss, used to guarantce the image quality, is defined between the generated data and the whole real data. To avoid the major correction, we propose to break the major correction into sequence of minor corrections to ensure data quality. Namely, at each epoch, we first use the generator G to generate ng samples, denoted X
X +
(²**))) {at~ ~ p(=))>
(7)
where e the epoch index. Then we replace the old training samples with the generated samples:
x X X U Xg.
(8)
where XÇ the old (least-recently added) ny samples in X
Due to the ranking loss, the generated data distribution b is closer to the desired data distribution Pa- compared to the constructed of each epoch. Therefore, the iterative replacement (equation g
5

---

Under review as a conference paper at ICLR 2021
rcal (desired) (undesred)
15.
15
15
velue
0.5
35
-0.
-25
-55
-15
0.5
-0.1
0.0
0.1
-15
-15 45
(a) Data
(h) WGAN
(c) DiCGAN
(d) WGAN
(c)
DICGAN
Figure 2: (a-c) Comparison of the critic in DiCGAN and WGAN. DiCGAN's critic can assign higher critic values for real desired data than real undesired data while WGAN's critic cannot. (d-e) The visualization of the generated samples from DiCGAN and WGAN. Learning the desired distribution means that the fake data should overlap with the real desired data only.
can gradually shift the real data distribution pr towards the desired data distribution pa. Namely, d(pe. Pa) > ... > d(pf.Pal > d(pf+1, .Pa) > .... So that only a minor correction needs to be imposed on Pá through optimizing equation at each epoch. Iteratively, the generated distribution Po shifts towards Pa- The training algorithm i8 summarized in Algorithm [ For the sake of easy optimization, we pretrain the differential critic D) and the generator G using WGAN
4 CASE STUDY ON SYNTHETIC DATA
To gain an intuitive understanding of the difference between our DiCGAN and WGAN regarding the critic and the generator, we conduct a case study on the synthetic dataset.
The synthetic dataset consists of two concentric circles by adding Gaussian noise with standard deviation of 0.05 (See Fig. 2a). The samples located on the inner circle are considered to be the desired data, while the samples on the outer circle are defined as the undesired data. By labeling the desired data as y = 1 and the undesired data as y = O, we can construct the pairwise preference for two samples 2 and xa based on their labels. Namely T1 > 22 if 1/1 A //2 0, and vice versa. The pairs are constructed within each mini-batch. Our target is to learn the distribution of the desired data (i.e., samples the inner circle), using the whole data along with the constructed pairwise preferences.
4.1 WGAN vs DICGAN ON CRITIC
Experiment setting: we fix the generator and simulate the fake data as the 2D Gaussian blob with a standard deviation of 0.05 (green pluses). We first train the critic to converge. Then, we project the output the second last layer of the critic into ID space using kernel principal components analysis (Schölkopf et al.] 1997). to derive the projected features. To explore the difference between the critics of WGAN and DiCGAN, features for WGAN and DICGAN, respectively (Fig. 2b 2ch.
draw the curve of the critic values versus the projected
2c we can see: (1) in terms of the real data and the fake data, the critic of both WGAN and DiCGAN can achieve perfect discrinination. Meanwhile, the projected features of the real data and those of the fake data are also completely separated; (2) in terms of the real desired data and the real undesired data, the critic of DiCGAN assigns higher values to the desired samples, compared to the undesired samples. This is because our ranking loss expects higher ranking score (i.e., critic values) for the corresponding desired data. (3) In contrast, the critic of WGAN assigns lower values the desired data, since the desired data is closer to the fake data compared to the undesired data.
From Fig. 2b
4.2 WGAN VS DICGAN ON GENERATOR
Experiment setting: train the critic and the generator following the regular GANs' training procedure. The generation results of WGAN and DiCGAN shown in Fig. 2d 2e
DICGAN (shown in Fig. 2eh only generates the user-desired data, i.e., generated data covering the inner circles, while WGAN (shown in Fig. 2d generates all data, i.e., generated data covering the inner and outer circles. As the critic in DiCGAN can guide the fake data towards the real data region and away from the undesired data region, the generator thus produces data which is similar to the real desired data. Because the critic in WGAN pushes the fake data the real data region only, the generator finally produces whole real-alike data.

---

Under review as a conference paper at ICLR 2021
/
2
3
7
7
4
0
8
3
0
3
6
3
X
7
6
4
0
2
5
2
1
0
0 0
4
0
9
4
1
7
9
1
1
0
1
0
0
0
/
3
0
6
4
5
2
5
8
3
3
8
7
7
/
4
7
6
3
6
0
6
0
0
4
J
0
(a) Epoch 0
(b) Epoch 1
(c) Epoch 2
(d) Epoch 5
(e) Epoch 10
Figure 3: Generated digits of DiCGAN on MNIST. DiCGAN learns the distribution of small digits.
5 EXPERIMENTAL STUDY
DiCGAN is applied to learn the distribution of the desired data on MNIST (Lecun et al. 1998) and CelebA-HO (Karras et al. 2018) datasets. Due to the limited space, we present more experiment results in the appendix.
Networks& & Hyperparameters The balance factor À and the ranking margin m is to We adopt the same network structures as in (Gulrajani et al. 2017). See the appendix for other settings.
Evaluation Metric To evaluate the performance of learning the desired data distribution, we calculate the percentage of user-desired data in GAN's generation, i.e., the ratio of the desired generated data to the whole generated data (D/W). We use inception score (IS) (Salimans et al. 2016) and multiscale structural similarity (MS-SSIM) (Odena et al. 2017) to evaluate the quality and intra-class diversity for GANs' generation, respectively.
Baselines We compare DiCGAN with WGAN (Gulrajani et al. 2017). CWGAN (Mirza and Osindero 2014) and FBGAN (Gupta and Zou 2019). WGAN is only trained with the desired data to derive the desired distribution. CWGAN is the extension of GAN with conditional label c. To train CWGAN, we split the training data into the desired class (c = 1) and the undesired class based on a predefined user criteria (c 0). Then p(z/c 1) is the desired data distribution. FBGAN adopts an iterative training paradigm to derive the desired data distribution. At each training epoch, FBGAN resorts to an extra selector to select the desired samples from the generated samples and performs regular GAN training using the selected desired samples. ACGAN (Odena et al. 2017) also shows poor results like CWGAN when the desired data is limited, so we do not report here.
5.1 LEARNING THE DISTRIBUTION OF DESIRED DIGITS
We design the experiment to learn the distribution of small digits in MNIST. We use 50K 28 x 28 images as the training data. Zero is the smallest digit in MNIST, thus as the desired data,
As for WGAN and CWGAN, zero digits in the training data regarded as the desired samples (c= 1), whose size is 4, 950. The other digits are labeled as the undesired samples, whose size is 45, 050 (c = 0). WGAN is only trained with the constructed desired data. CWGAN conditions on e to model a conditional data distribution p(z|c) for MNIST dataset. FBGAN and our DiCGAN resort to classifier, pre-trained for digit classification, to obtain the labels for the generated samples. At every training epoch, FBGAN generates 50000 samples and requests the classifier to label them. Then the images are ranked using the predicted label, with the smaller digit ranked higher. The generated images with digits ranked in the top 50%, i.e., small digits, are selected the desired data to replace old training data. As for DiCGAN, the pairwise comparison can be obtained for two images 21 and 22 according to their predicated label y/1 and y/2 namely T1 > 22 if y/1 < y/, and vice versa. At each iteration, we construct 32 pairwise preferences for each mini-batch with 64 training samples.
Fig. presents the generated MNIST images randomly sampled from the generator of DICGAN. It shows that the generated MNIST digits gradually shift to smaller digits during the training, and converge to the digit zero We sample 50K samples from the generators of various GANs and respectively calculate the percentage of digit zero and digits zero four among the generated digits for a quantitative evaluation. In T only small digits are generated by DiCGAN and FBGAN; WGAN and CWGAN can also learn the distribution of the desired digit since the dataset is simple and has relatively sufficient data for the desired digit. However, WGAN and CWGAN do not exhibit a smooth convergence to digit zero like FBGAN and DiCGAN (See Fig. T0 in the appendix). In addition, when the dataset is complex and the desired data is insufficient, WGAN and CWGAN fail, which is described in the next section.
7

---

Under review as a conference paper at ICLR 2021
5.1.1 COMPARISON OF DICGAN AND FBGAN
Though FBGAN achieves good performance in learning the desired data distribution, it requires lot of supervision information from the selector. We calculate the number of effective pairs (#EP) used in DiCGAN and FBGAN, respectively. #EP in DiCGAN denotes the total number of explicitly constructed pairs during the training, i.c., #EP = samples from the generated samples. #EP can be induced by the implicit pairs implied by the desired samples versus the undesired samples, i.e., #EP Σx= !gdd X Tggu. where ne is the number of training epochs. where ngd and 7ggu denote the number of desired samples and undesired samples in the generation, respectively.
Li=1 Zi=1 72.. FBGAN selects the desired
Fig. 4a shows that (1) the #EP used in DiCGAN is much smaller than that in FBGAN at each training epoch; (2) the total #EP used in DiCGAN is significantly less than that in FBGAN, which can be reflected from the shadow area. In total, DiCGAN used 9.53e4 effective pairs while FBGAN used 2.02e8 effective pairs. Our DiCGAN is scalable to the large training dataset, e-g. MNIST. #EP in DiCGAN is linearly correlated to the training size. In contrast, #EP in FBGAN is determined by Vgd and Zgg.- which are both linearly correlated to the training size. Thus #EP in FBGAN thus is quadratically correlated to the training size.
5.1.2 COMPARING DICGAN AND FBGAN GIVEN THE LIMITED SUPERVISION
We compare DiCGAN and FBGAN on MNIST dataset given the limited supervision. Specifically, the query amount of resorting to the pre-trained classifier to obtain the prediction of the generated samples is restricted to 5, 000 for both FBGAN and DICGAN.
Fig. 41 shows that DiCGAN can learn the desired distribution while FBGAN fails, only generating 10.3% digit zero, which is consistent with the visual results in Fig. 4p and Fig. 4e. This supports the claim that the negative samples are beneficial to learn the user-desired distribution. Particularly, the preference direction can be captured by our differential critic even the supervision is limited, which guides the generation towards the desired data.
5.2 LEARNING THE DISTRIBUTION OF FACE WITH THE DESIRED ATTRIBUTE
We consider old face images as the desired data and design the experiment to learn the distribution of old face images on CelebA-HQ. CelebA-HQ dataset is the high quality version of subset from Celeb Faces Attributes (CelebA) dataset, which consists of 30K images face images of celebrities, annotated with 40 binary attributes such as age. We resize the images to 64 x 64. The training setting for each GAN is similar to those mentioned above. (See the appendix for more details).
In Fig. 5 we visualize the generated face im- ages randomly sampled from the generator of each model. (1) WGAN has poor generation since its training data, i.e., the desired subset is insufficient. (2) CWGAN has good quality of generation but fails to capture the desired distri- bution. There is only two old face image out of 9 randomly sampled images. (3) All sampled images from FBGAN and our DiCGAN are the desired old faces. We sample 10K samples from the generator and calculate the percentage of old face images among the generated samples for images generated by DiCGAN and FBGAN are old face images. WGAN and CWGAN both fail to
MNIST D/W
CelebA-HQ
Method
D/W
IS MS-SSIM
tup 1
tap
Original WGAN (subsel)
999
31.1
22.1
2.81
0.52
97.3
98.2
73.8
1.77
0.60
CWGAN
95.0
96.4
16.4
2.12
0.65
FEGAN
100.0
100.0
98.2
2.05
0.60
DiCGAN
100.0
100.0
99.5
1.93
0.60
Table 1: Results on the MNIST and CelebA-HQ dataset. Top I refers to digit zero. Top 5 refers to digits zero to four.
quantitative evaluation. In Table D almost all the
FIGH DIGAN
5 A
1
5 4
9
7
2
6
5
2
5
/
4
0
4
9
MNIST
(
5
4
3
0
8
2
D
Method
Top
top 5
10/
9
7
5
6
3
0
6
2
4
6
0
FBGAN
10.3
52.6
30
23
Eooch
20
40
9
5
0
3
7
0
9
4
5
DiCGAN
99.7 99.9
(a)
(b)
(c)
(d)
Figure 4: (a) Comparison between DiCGAN and FBGAN w.I.t #EP per epoch MNIST in
Section 5.1 (b) The generated results of FBGAN in Section 5.1.2 (c) The generated results of
FBGAN Section 5.1.2 (d) % of desired digits (D/W) of FBGAN and DiCGAN in Section 5.1.2
8

---

Under review as a conference paper at ICLR 2021
(a) WGAN
(b) CWGAN
(c) FBGAN
(d) DICGAN
Figure 5: Generated images by WGAN, CWGAN, FBGAN and DiCGAN. The green boxes refer to the images which are classified as old images by trained classifier.
capture the distribution of old face images. As for the quality and the diversity, (1) GAN shows the lowest IS, meaning poor-quality of generation. The generations of other GANs except for WGAN present similar IS and MS- SSIM. (2) We calculate the IS of the old faces in the training data (denoted as "original" Table [ The IS and MS-SSIM of the generations do not exhibit big difference from the "original",, which means that the quality and the diversity of the generations are relatively good.
DISCUSSIONS
Current GAN (Goodfellow et al., 2014, Mirza and Osindero 2014 Odena et al. 2017 Gupta and Zou 2019 based methods require user-defined criteria to select the desired data in order to derive the distribution of the desired data. There are two limitations for these methods: (1) the criteria are not
always accessible in real applications; (2) eliminating the handerafted undesired samples loses useful information about what is not desired. Other works (Gómez-Bonne Bombbrelli et al., 2018 Engel et al. 2018) proposed find the subset of the latent space corresponding to the desired data and generates data from the latent subset. However, they also rely on user-defined criteria for labeling the desired data or a ready-to-use score function find the subset of the latent space with high scores.
This paper proposes DiCGAN to learn the distributions of the user-desired data from the entire data using the pairwise preferences. We empirically demonstrate the efficacy of DiCGAN in terms of promoting samples with user-desired properties on MNIST and CelebA-HQ datasets, respectively, One promising future direction for DiCGAN could be the minority population promotion for imbal- anced data tasks, such as imbalanced classification problems, few-shot learning, one-shot learning or even an open-set problem. Another interesting direction of DiCGAN could be the desired policy generation in imitation learning if the pairwise comparison between the policies can be properly designed.
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pages 214-221 214-223, 2017.
Jesse Engel, Matthew Hoffman, and Adam Roberts. Latent constraints: Learning to generate conditionally from unconditional generative models. In International Conference on Learning Representations, 2018
William Fedus, Ian J Goodfellow, and Andrew M Dai. MaskGAN: Better Text Generation via Filling in the In International Conference on Learning Representations, 2018.
Rafael
Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato Benjamín Sânchez-LLeneeling Dennis Sheberla, Jorge Aguilera-Iparraguirre, Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules. ACS Central Science, 4(2):268-276) 276, 2018. doi: 10. 1021/ acscentsci.7600577 URL htwph:tttsssttt /ttps:://doo.. http:///ooooooooo//////oo htwp:////accceescccooooooo.
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C Courville, and Yoshua Bengio. Generative Adversarial Nets. In Advances neural information processing systems, pages 2672-2680, 2014.

---

Under review as a conference paper at ICLR 2021
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In Advances in neural information processing systems, pages 5767-5777, 2017.
Anvita Gupta and James Zou. Feedback gan for dna optimizes protein functions. Nature Machine Intelligence, 1(2):105, 2019.
Alexia Jolicoeur-Martineau. The relativistic discriminator: key element missing from standard GAN. In International Conference on Learning Reprexentations, 2019.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In 6th International Conference on Learning Representa- tions, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
Nathan Killoran, Leo J Lee, Andrew Delong, David Duvenaud, and Brendan J Frey. Generating and designing dna with deep generative models. arXiv preprint arXiv:1712.061-44 2017.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 22224, Nov 1998.
Tyler Lu and Craig Boutilier. Learning mallows models with pairwise preferences. In International Conference Machine Learning, pages 145-152, 2011.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg $ Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 31 11-11119, 2013.
Mchdi Mirza and Simon Osindero. Conditional generative adversarial nets. ar1iv: 2014.
arXiv preprint
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In International conference machine learning, pages 2642-26222222 2651, 2017.
Tim Salimans, Ian Goodfellow, Wojeiech 7aremba, Vicki Cheung, Alec Radford, and Chen. Improved techniques for training gans. In Advances in neural information processing systems, pages 2234-2242, 2016.
Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller. Kernel principal component analysis. In International conference on artificial neural networks, pages 583-588. Springer, 1997.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In Advances in neural information processing systems, pages 613-621 2016.
Ke Zhou, Gui-Rong Xue, Hongyuan Zha, and Yong Yu. Learning to rank with ties. Proceedings of the 31 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '08, page 275-282, New York, NY, USA, 2008 Association for Computing Machinery.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei Efros. Unpaired image-to-ine to-inage translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.
A COMPARISON OF DICGAN AND FBGAN
We plot the ratio generated zero digit to the whole generated data (D/W) of DiCGAN and FBGAN during the training process in Fig. 6a It shows that DiCGAN converges faster than FBGAN.
We explore the gap in the performance between DiCGAN and FBGAN evolves as the number of supervision increases on MNIST. Specifically.the query amount of resorting to the pre-trained classifier to obtain the prediction of the generated samples is restricted to 5K, 50K, 100K, 150K, 180K, 200 500K for both FBGAN and DiCGAN.
10

---

Under review as a conference paper at ICLR 2021
1.0
1.0
0.8
0.8
D/W
0.6
D/W
0.6
0.4
0.4
FBGAN DiCGAN
FBGAN DICGAN
0.2
0.2
5K
50K 100K 150K 180K 200K 500K
Epoch
Number of Supervision
(a)
(b)
Figure 6: Comparison of DiCGAN and FBGAN. (a) plots the ratio of generated zero digit to the whole generated data (D/W) versus the training epoch for FBGAN and DiCGAN, respectively. (b) plots the ratio of generated zero digit to the whole generated data (D/W) versus the number of supervision for FBGAN and DiCGAN, respectively.
Fig- 6 plots D/W versus the number of supervision for FBGAN and DiCGAN, respectively. It shows that (I) DiCGAN always learns the desired data distribution even given the limited supervision: (2)
when given the limited supervision, FBGAN fails to learn the desired distribution, i.c., achieving a
small D/W: (3) FBGAN performs better and achieves higher D/W, narrowing the performance gap with DiCGAN as the number of supervision increases.
B ABLATION STUDY
The objective in our DiCGAN (equation consists two components, i.e. the WGAN loss. which serves as the cornerstone of DiCGAN, and the ranking loss, which serves as the correction for WGAN. Meanwhile, we introduce the operation of replacement (equation ( during the model training.
1.00
DICGAN(Á
DICGANI,, =0)
0.75
Method
MNIST
DNW
DICGAN
top
0.50
0.50
Original
9.9
51.1
DICGANIA-O)
WGAN
50.9
0.25
025
DICGANIA O0)
DICGAN (A = 0)
9.1
54.6
0.00
20
Epoch
40
20
Epoch
DICGAN (ng 0)
DICGAN
100.0
24.0
100.0
87.8
(a)
(b)
Figure 7: (a-b) The percentage of desired samples (D/W) versus epoch in DiCGAN (A = 0), DICGAN
(ng = 0) and DiCGAN. (a) plots the D/W of the digit zero. (b) plots the D/W of the digit zero to four. (c) The percentage of desired samples (D/W)from the orginial datasets, WGAN, DiCGAN (A = 0), DiCGAN (ng 0) and DiCGAN.
To analyze the effects of the correction for WGAN (the third term in equation 5) and the replacement operation, we plot the percentage of desired samples (D/W) versus the training epoch for DiCGAN (A 0), DiCGAN (ng = 0) and DiCGAN in Fig. 7h. 7pp. Meanwhile, the the converged percentage of desired samples (D/W) are reported in Fig. 7b. It can be seen that
1. Without the correction term (A 0), DiCGAN cannot learn the desired data distribution. The percentage of desired samples (D/W) from DiCGAN (A 0) remains constant during training on MNIST (Fig.) 7h, 7D) compared with the original datasets (Fig- (Fig-Z-). This is because that the remaining WGAN term in DiCGAN(A = 0) focuses on learning the training data distribution.
2. Without the replacement (ng = 0), DiCGAN makes a minor correction to the generated distribution. In Fig. 7h, 7b, the D/W of DiCGAN (ng 0) slightly increases compared with
11

---

Under review as a conference paper at ICLR 2021
4 5
3
4
2
7
7
6
8
6
7
7
2
7
4
4
2
4
9
8
6
2
6
2
6
6 2 8
8
2
R
2
5
7
<
7
0
2
9
8
2
O
2
9
2
Y
7
2
7
9
8
3
(a)
PRG-1
3
3
6 /
4 /
2
8
7
1 4
7
/
0
0
5
2
1
/
4
7
3
3
U
6
0
6
0
W
0
0
0
7
7
0
/
2
/
0
0
3
5
6
D
5 3
0
O
O
0
24
(b) PRG-2
Figure 8: Generated digits of PRG-1 1 and PRG-2 MNIST during the training process.
the original datasets. This is consistent with our analysis that the correction term would drive the generation towards the desired distribution.
3. DiCGAN learns the desired data distribution with a sequential minor correction. The D/W of DiCGAN grows with training and reaches almost 100% when convergence. The correction term drives DiCGAN's generation towards the desired data slightly at each epoch. With the iterative replacement, the minor correction sequentially accumulates and finally the generated distribution shifts to the desired data distribution.
C PAIRWISE REGULARIZATION ON THE GENERATOR
As discussed in Remark T the pairwise regularization is possibly added to the generator. We consider two cases of adding the regularization the generator. First, we only add the pairwise regularization to the generator (PRG-1). Second, we add the regularization to the generator together with the regularization on the critic (PRG-2).
The objective for PRG-1 is as follows:
Lp = Ep)(a) [D(x)]
(a)) [ (x)),
(9)
LG = Epa(a) [D()) - A S
Σ SES'
[4 (s)
(10)
where h(s)) is equation 3 S' is the pairwise preferences constructed between the generated data and the undesired data, i.e., = {s (x1, x2) > F2, 21 ~ (a()), 42 ~ p. Now the generator
consists of two terms, the original WGAN loss the generator aims achieve Ep (z) [D (x)] >
Ep,(a) [D(z),, while the regularization aims to achieve Epa(a) [D [x)) > Ep,(,) [D(x)]. Since the undesired data is subset of the real data, i.e., {æl/æ ~ Pu(x)) } C {a|æ ~ (.)).. the WGAN loss always dominates the training of the generator. Therefore, PRG-1 degenerates to WGAN.
The objective for PRG-2 is as follows:
LD = (x)()) [D(x]] - E po (x) 1D (x) - A Σ [s)),
(11)
S
SES
LG = (x) (x))
Xg
(s)),
(12)
SS'
ees'
where S is constructed based on equation 2 Although the generator consists of two terms, same to analysis about PRG-1, the extra pairwise regularization the generator is invalid. Mcanwhile, the extra pairwise regularization on the critic works like that in DiCGAN. Therefore, the whole framework degenerates DiCGAN.
We conducted the experiments on MNIST to show the effectiveness of these two methods. À and Ag both set to 1. Fig, shows the generated digits of PRG-1 and PRG-2 during the training process. PRG-1 failed to learning the desired distribution. PRG-2 can learn the desired distribution. The quantitative results consistent with the visual results, with 13.9% and 99.4% D/W, respectively.
12

---

Under review as a conference paper at ICLR 2021
(a) Epech 0
(b) Epoch
Epoch 20
(d) Epoch 50
(e) Epech 100
Figure 9: Generated images of DiCGAN on CIFAR. DiCGAN aims to learn the distribution of car
images of CIFAR. The training dataset is composed of car and plane images in CIFAR-10.
D LEARNING THE DISTRIBUTION OF DESIRED OBJECTS
We consider cars the desired objects and design the experiment to learn the distribution of cars in CIFAR.
Sample selection in FBGAN: A classifier, pretrained for classifying cars and planes, is adopted for selection. The generated objects, classified to car, are selected to replace the old training data.
Pairwise preferences construction in DICGAN: Denoting label of CIFAR image y. the pairwise preference between two images 21 and 22 are Iι > 32 when y1 = "car", V/2 "planc", and vice versa. At each iteration, construct 32 pairs by random sampling pairs from the mini-batch 64 samples.
In Fig. 9 visualize the generated CIFAR images ran- domly sampled from the generator of DiCGAN. It shows that DiCGAN gradually generates cars, as we desired.
CIFAR (IS)
Method
D/W
IS
MS-SSIM
Meanwhile, we sample 10K samples from the generator and calculate the percentage of car images among the generated samples for quantitative evaluation. In Table (1) almost all images generated by DiCGAN and FBGAN are car images; (2) the percentage of car images generated by WGAN similar to the training dataset.
Dataset
50.0
4.96
0.45
FBGAN
93.9
3.78
0.52
DiCGAN 95.4
3.67
0.49
Table 2: Results on CIFAR dataset.
E EXPERIMENT SETTINGS
Hyperparameter The batch size b is set to for MNIST, 64 for CIFAR and CelebA-HQ datasets. The #generated samples rg is set to 50K for MNIST, 1 K for CIFAR and 3, 000 for CelebA-HQ. respectively. Other hyperparameters are adopted the same as in Gulrajani et al. (017)).
We construct pairwise preferences using the minibatch samples at each iteration based the classification labels. We construct the pairs by randomly selecting two samples from the minibatch samples, respectively. The pairs, in which two samples belong to the same class, i.c., same digits or same objects, are removed.
CelebA-HO training setting WGAN is only trained with the constructed desired dataset. CWGAN conditions on c to model a conditional data distribution p(#lc). There are 6, 632 samples labeled desired and 23, 368 samples labeled as undesired in the training data. A classifier, pre-trained for classifying young faces and old faces, is adopted for predicting the labels for the generated face images. At every training epoch, FBGAN generates 000 images and those classified the old selected to replace the old training data. As for DiCGAN, the generated face image classified with the old attribute is preferred over the face image classified with the young attribute. each iteration. we construct 32 pairs by random sampling pairs from the bani-batch 64 samples.
F MORE VISUAL RESULTS
13

---

Under review as a conference paper at ICLR 2021
WGAN (subset)
i
Z
0
6
=
O
0
0 C
0
3
(b) CWGAN
8
8
7
4 9
3
4
3
2
0
O 2
/
2
6 1
6
7
6
9 9
/
2
/
2 1 3 /
7
4
O
0
5
4
5
2
0
5
0
0
0
3
/ 0
0
7
8
9
2
4
J 0
1
O
0
8 3
0
3
7
3
5
9
2
7
3
2
0
4 /
(c) FBGAN
/
2
9
$
3
3 7
8
/ 7
4
3
/
0
/
3
0
3
63
X 7
4
0
4
5
2
1
0
0
0
4
9
4
7
0
/
/
0
4
0
0
0
b 7
4
2
8
3
3
2
7
0
7
/
4
6
8
6
5
0
8
0
0
4
5
(d) DiCGAN
Figure 10: Generated digits of DiCGAN on MNIST during the training process.
(a) Epoch 0
(b) Epoch
(c) Epoch
(d) Epoch
Epoch
Figure 1I: Generated images of DiCGAN on CelebA-HQ. DiCGAN learns the distribution of old faces. DiCGAN gradually generates more old face images.
14


Reviews:
[{"review_id": "QNWevyBzpf", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Overview**\nThe paper proposed differential critic GAN, which proposes a differential critic that learns the preference direction from pairwise preferences. A corresponding loss function is added to the WGAN loss to ensure that the model learns samples that have high rank. Empirical results show that the method is able to reflect the ranking from the user.\n\n**Strengths**\n- The paper presents a decent solution to the problem where additional rank-based supervision is provided. \n- The paper have superior performance than FBGAN and WGAN on specific settings concerning MNIST and CelebA.\n\n**Weaknesses and Questions**\n- Why would we consider pairwise preferences over explicitly labeling what the users consider to be good data or not?\n- Do the preferences define a partial order? Can user preferences be cyclic?\n- The problem statement seems to be flawed; when we say x1 ranks higher than x2, does that mean we want x1 to appear in the distribution and x2 to not appear in distribution (I suppose not)? If the answer is no, then of what percentile (assuming total order, that is) do we consider to have low probability (rank does not indicate weights which is eventually what we want from a well defined user-based data distribution)?\n- Evaluation is a bit limited, especially in terms of sample quality. CelebA-HQ inception score does not count, because IS is not a good metric for image quality on human faces (with a untrained model you might get 3 and it gets lower as you train). The CelebA image quality is very far from that trained with a decent GAN (like smaller BigGANs, which you can train with 1 GPU in about a day).\n- Why not use a CWGAN like objective where you only train on datapoints with c = 1? If your goal is to make use of the larger dataset for better generation, then there are some other work, then there are some other importance-weighted work to your interest: Choi et al. 2020, Fair Generative Modeling via Weak Supervision.", "extra_scores": {}, "extra_text": {}}, {"review_id": "IglYO6UxTpv", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The motivation of this study is to estimate the distribution of desired data from the entire data distribution. And the proposed solution extends existing GAN solutions by introducing an additional pairwise loss on the discriminator, e.g., its scores on the desired instances should be higher than the undesired ones. The idea is natural and neat, and it is also proved to be effective in the reported experiments. \n\nFirst of all, why not add such pairwise regularization to the generator? I did not find any discussion on this alternative and the advantage of adding regularization on the discriminator side. At least, it is important and interesting to empirically compare these two choices to understand their difference. On a related note, as the proposed DiCGAN is very similar to RGAN, it is important to compare these two solutions to better understand how their design difference translates to empirical performance difference. \n\nI do not fully follow the discussion in Section 3.3, especially the part about the major correction aspect. The claim is when the desired data distribution is far away from the entire data distribution, the generator might have difficulty in satisfying the WGAN loss. And then the proposed solution is to gradually generate new instances to replace the old one. But I am not sure why this would lead to convergence, although empirical results provided in Appendix shows this strategy. For example, in the first round, the quality of generator is bad, such that the pairs constructed from the generated instances do not reflect the preference direction. Hence the only pairs useful there are those from the initial training instances, which might still require major correction on the generator? And I also did not find any discussion about how the number of pairwise samples or the way to sample such pairs affects the model’s performance. As the pairs are constructed from individual instances, transitivity among the pairs would introduce additional complexity in model training, as the pairs are no longer independent. \n\n**Acknowledgement of author responses**\nThe authors' responses were helpful to clarify the settings and basic ideas of the proposed solution. I highly appreciate their effort. However, the limitations of the proposed solution and limited novelty still outweigh the merit; and therefore, I will keep my original evaluation. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "ErVMYc36gg", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors introduce DiCGAN, an algorithm to learn a generative model that comes up with samples whose likelihood is based on a real dataset but adjusted given user preferences. They train the critic to assign high values to samples with higher preference values and thus the generator tends to move its samples towards these points. The idea is nice and reasonably novel in my opinion, but the paper has quite a few problems.\n\nThe first problem is that the writing of the paper is awful. Already in the abstract there are several syntactical and semantical errors. It is actually fairly hard to read this paper, likely the idea is simple and one can understand it from the equations, but everything that's written pretty much makes the paper harder to read. The authors also write too strong claims for a scientific paper: many times they write that DiCGAN learns the user desired data distribution (learning a high dimensional data distribution with finite data is not a possible not an interesting goal, the users should use \"approximates\" instead of \"learns\" and describe *how* it approximates it and what is lost), and things like \"the superiority of DiCGAN is twofold\". There's an entire section titled \"Superiority of DiCGAN over FBGAN\"!\nThe paper is also quite imprecise / uses the wrong mathematical terms at points. For instance, \"equation 4 is the Lagrange function of equation 6\".\n\nFinally, while the experiments are interesting, they're all on MNIST or aligned celeb-A in 64x64, and the samples are terrible. It is hard to believe that this method could be scaled up as is, or at least there is little evidence to that regard.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4i0tzCBh1Ia", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- Overview:\n\tThe paper addresses the problem of training a GAN to match the distribution of part of the dataset called the 'desired data distribution', instead of the whole dataset as usually done in the context of GANs. This problem can be of interest when for instance the 'desired training data'  is limited and one would like to leverage additional 'undesired' data for training while still ensuring the model generates samples similar to the desired data.\n\tThe address this problem, the authors propose to use an additional set S of pairwise samples  (X_1,X_2) called the set of preferences to guide the training of a WGAN. This set specifies that the user prefers sample X_1 over X_2.  X_1 is typically  a sample from the desired data and X_2 is an undesired sample. This provides an additional information during training to favor samples that are similar to X_1.\n\tThe authors then propose to encourage the critic to give a higher score to sample X_1 and a lower score to sample X_2. This is achieved by adding a penalty to the WGAN loss. This  penalty is simply given by the margin loss between the scores of X_1 and X_2.    \n\tThe critic loss is then a tradeoff between two terms: the standard WGAN term which encourages 'data quality' and the preference term which favors samples similar to the preferred data. \n\t The authors then identify two major training situations which determine how easily the tradeoff can be achieved: 'minor correction' and 'major correction'.\n\t- The minor correction is when the 'desired data' is very similar to the whole dataset. Thus the tradeoff is easily achieved using the proposed loss of eq 4.\n\t- The major correction is the harder situation where the 'desired' data are different from the whole dataset. Thus using the proposed method directly could either learn to preference at the expense of sample quality or maintain good quality but without learning the preference. To address this, the authors propose a modification of the algorithm that consist in replacing part of the training samples by generated ones at every epoch. The authors then claim that this addresses the problem since the modified training data should get closer to the desired data at every epoch.\n\tThe authors then demonstrate empirically the ability of the proposed methods to successfully learn the desired distribution while using shared structure with the undesired data. This is first illustrated in a simple synthetic example in section 4. then in section 5, the author compare the proposed method with other baselines on MNIST and CelebA datasets. \n\n\tThe authors conclude that the proposed method has two advantages compared to existing methods.\n\t- The method doesn't need a pre-defined score function to distinguish between desired and undesired data. That is because they instead rely on the pairwise preference set S. \n\t- While other methods do not use undesired data for training, the proposed method can effectively exploit those data to yield better performance when the desired dataset is scarce.\n\n- Strength of the paper:\n\tThe author show experimentally that the proposed method is able to take advantage of negative samples to learn the user-desired distribution. This allows to use fewer supervision than the method FBGAN which only relies on positive samples. This is illustrated in the experiment in 5.1.2. where the number of supervision is limited in purpose.\n\n- Weakness of the paper:\n\tThe main weakness is the soundness of the approach and the clarity of the paper.\t\n\t- In definition 1 (problem setting), the  authors first claim in the that proposed method allows to learn the desired distribution. But it remains unclear from the structure of the loss whether this is the case. It seems that the loss achieves only a tradeoff between learning the desired data and being close the whole training set. This is in particular apparent from the discussion about the minor and major correction scenarios. The statement in definition 1 should be adapted to reflect this limitation of the method and avoid confusion.\n\t- In the major correction scenario, the authors propose to introduce fresh generated samples as part of the training set at every epoch. By doing so, the authors claim that the modified training set is getting closer and closer to the desired distribution thus leading to a decreasing sequence of distances to the desired data. I honestly don't see how this solves the problem: \n\t\t- Why would one obtain a decreasing sequence as claimed?\n\t\t- How does it prevent the quality of the generated data from being degraded? It doesn't seem like using bad generated samples as part of the training set could help prevent the degradation of sample quality.\n\t\t- It would be good to have either theoretical or empirical evidence of why this approach is relevant.\n\t\n\t- The authors present a limitation of FBGAN as needing a predefined score function, while the proposed method relies on a set of pairwise preferences. Yet in practice, it seems that the set S is construction using a classifier or from the labels of the data. Isn't it equivalent to having  a pre-defined score function?\n\n\t- In 5.1.1, the authors compare the number \"effective pairs\" used during training in the case of FBGAN and the proposed method. For the proposed method, this is simply the number of elements in the set S. But for the competing method FBGAN which doesn't rely on pairs, I don't see what it means. The authors say that is corresponds to the product of desired and undesired samples. Is that a relevant quantity to consider for FBGAN? The authors then compare those numbers and conclude that the proposed method requires way fewer effective pairs. But I'm not convinced this is a fair comparison. Perhaps what could be more relevant is to simply compare the performance as a function of the number of epochs:  FID score, percentage of desired samples every epoch.\n\t\n- Clarity:\n\tWhile the experiment sections are clear  overall, I found section 3 to be very confusing:\n\t- My current understanding is that, the score loss encourages the critic to take into account a preference of the user. Such preference is supposed to be encoded by the set of 'pairwise preferences' S. The procedure makes sense to me if the set $S$ is pre-determined by the user.\n\t- However, section 3.2.1 is confusing in that it suggests that the set S is constructed using the critic itself. More precisely, what was confusing is that the ranking between samples (which is used to construct the set S in equation 2) is defined by means of the score function $f$ (the last two sentences before eq 3.). Then the author say that the score $f$ is chosen to be the critic $D$ (which is learned during training). This clearly wouldn't makes sense, since no information about the user's preference is included.  Later in the experiments sections, the authors clarify how the set S is pre-defined, but still this makes the reading experience very confusing.\n\t- The background section do not contain relevant concepts and work that are key to understand the paper: The authors take inspiration from Relative GAN to designing the critic, but we don't know exactly how. This should be discussed in more details before presenting the method. The same is valid for FBGAN, a competing method, which is partially introduced in the method section. Only later, in the experiment section, that the reader discovers some subtleties in how FBGAN are trained. In particular, the authors explain in section 5 that the generated samples need also to be scored during training in the case of FBGAN, however, this is never mentioned before and seems to be critical to understand the experiments of section 5.1.2 (limited supervision scenario). \n\n\t - In 5.1, the authors say that they use a pre-trained classifier to distinguish between desired and undesired generated data. This suggests that the generated samples are also used to construct the set S? However, isn't S an input to the algorithm and thus independent of the generator? Are those samples only used to evaluate to what extend the generator learns the preferred distribution?\n\n\n- Questions and suggestions:\n\t- The training algorithm should be in the main paper.\n\t- How does the gap in performance between the two methods evolve as the number of supervision increases?\n\t- Is this behavior consistent across datasets? For instance on CelebA? It seems like on CelebA both methods perform equally well. However, showing that the method still requires fewer supervision in the case of datasets such as CelebA could be strengthen the results.\n\t- Why do the critic and generators need to be pre-trained? What happens when training from scratch?\n\t- 'As MNIST contains digits from 0 to 9, 0 is the most desirable data'  This sentence doesn't really make sense, or does it?\n\nUpdated review: Thank you  for your response and  for your efforts in addressing those points. I raised my score to 5 for the clarifications made to the document. However, I still think there are unjustified claims about the method, especially those related to the strong correction scenario and how introducing fresh generated samples is an effective procedure (conceptually / theoretically).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a method to regularize the discriminator in  GAN training with a ranking loss based on the user preference for a desired set within a larger dataset. The tradeoff between GAN loss and preference loss dependence on the distance of the set to the full dataset and the authors consider two regimes : "small and major correction". A major correction is needed when the targeted set is very different from the whole density, authors propose in this scenario to replace samples from the data by samples from the generator. The setting in the paper is interesting and can be useful in practice. 

There was a lengthy discussions between the authors and the reviewers, the discussion pinpointed issues , some of them were addressed in the rebuttal . Some issues remain unanswered regarding the clarity and some claims in the paper.

The clarity of the paper needs further improvement and  1)  clarify section 3  the setup and the background section   2)  justify claims about the method, in  the strong correction scenario  when fresh generated samples are introduced how  is this an effective procedure? (conceptually / theoretically). 

Decision: Reject