Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors introduce DiCGAN, an algorithm to learn a generative model that comes up with samples whose likelihood is based on a real dataset but adjusted given user preferences. They train the critic to assign high values to samples with higher preference values and thus the generator tends to move its samples towards these points. The idea is nice and reasonably novel in my opinion, but the paper has quite a few problems.

The first problem is that the writing of the paper is awful. Already in the abstract there are several syntactical and semantical errors. It is actually fairly hard to read this paper, likely the idea is simple and one can understand it from the equations, but everything that's written pretty much makes the paper harder to read. The authors also write too strong claims for a scientific paper: many times they write that DiCGAN learns the user desired data distribution (learning a high dimensional data distribution with finite data is not a possible not an interesting goal, the users should use "approximates" instead of "learns" and describe *how* it approximates it and what is lost), and things like "the superiority of DiCGAN is twofold". There's an entire section titled "Superiority of DiCGAN over FBGAN"!
The paper is also quite imprecise / uses the wrong mathematical terms at points. For instance, "equation 4 is the Lagrange function of equation 6".

Finally, while the experiments are interesting, they're all on MNIST or aligned celeb-A in 64x64, and the samples are terrible. It is hard to believe that this method could be scaled up as is, or at least there is little evidence to that regard.
review_title: Review for "Differential-Critic GAN: Generating What You Want by a Cue of Preferences"

final_score: rating: 5: Marginally below acceptance threshold
review_content: The motivation of this study is to estimate the distribution of desired data from the entire data distribution. And the proposed solution extends existing GAN solutions by introducing an additional pairwise loss on the discriminator, e.g., its scores on the desired instances should be higher than the undesired ones. The idea is natural and neat, and it is also proved to be effective in the reported experiments. 

First of all, why not add such pairwise regularization to the generator? I did not find any discussion on this alternative and the advantage of adding regularization on the discriminator side. At least, it is important and interesting to empirically compare these two choices to understand their difference. On a related note, as the proposed DiCGAN is very similar to RGAN, it is important to compare these two solutions to better understand how their design difference translates to empirical performance difference. 

I do not fully follow the discussion in Section 3.3, especially the part about the major correction aspect. The claim is when the desired data distribution is far away from the entire data distribution, the generator might have difficulty in satisfying the WGAN loss. And then the proposed solution is to gradually generate new instances to replace the old one. But I am not sure why this would lead to convergence, although empirical results provided in Appendix shows this strategy. For example, in the first round, the quality of generator is bad, such that the pairs constructed from the generated instances do not reflect the preference direction. Hence the only pairs useful there are those from the initial training instances, which might still require major correction on the generator? And I also did not find any discussion about how the number of pairwise samples or the way to sample such pairs affects the model’s performance. As the pairs are constructed from individual instances, transitivity among the pairs would introduce additional complexity in model training, as the pairs are no longer independent. 

**Acknowledgement of author responses**
The authors' responses were helpful to clarify the settings and basic ideas of the proposed solution. I highly appreciate their effort. However, the limitations of the proposed solution and limited novelty still outweigh the merit; and therefore, I will keep my original evaluation. 
review_title: Extending GAN with pairwise loss/regularization on the discriminator

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Overview**
The paper proposed differential critic GAN, which proposes a differential critic that learns the preference direction from pairwise preferences. A corresponding loss function is added to the WGAN loss to ensure that the model learns samples that have high rank. Empirical results show that the method is able to reflect the ranking from the user.

**Strengths**
- The paper presents a decent solution to the problem where additional rank-based supervision is provided. 
- The paper have superior performance than FBGAN and WGAN on specific settings concerning MNIST and CelebA.

**Weaknesses and Questions**
- Why would we consider pairwise preferences over explicitly labeling what the users consider to be good data or not?
- Do the preferences define a partial order? Can user preferences be cyclic?
- The problem statement seems to be flawed; when we say x1 ranks higher than x2, does that mean we want x1 to appear in the distribution and x2 to not appear in distribution (I suppose not)? If the answer is no, then of what percentile (assuming total order, that is) do we consider to have low probability (rank does not indicate weights which is eventually what we want from a well defined user-based data distribution)?
- Evaluation is a bit limited, especially in terms of sample quality. CelebA-HQ inception score does not count, because IS is not a good metric for image quality on human faces (with a untrained model you might get 3 and it gets lower as you train). The CelebA image quality is very far from that trained with a decent GAN (like smaller BigGANs, which you can train with 1 GPU in about a day).
- Why not use a CWGAN like objective where you only train on datapoints with c = 1? If your goal is to make use of the larger dataset for better generation, then there are some other work, then there are some other importance-weighted work to your interest: Choi et al. 2020, Fair Generative Modeling via Weak Supervision.
review_title: GAN + Score based ranking preferences

final_score: rating: 5: Marginally below acceptance threshold
review_content: - Overview:
	The paper addresses the problem of training a GAN to match the distribution of part of the dataset called the 'desired data distribution', instead of the whole dataset as usually done in the context of GANs. This problem can be of interest when for instance the 'desired training data'  is limited and one would like to leverage additional 'undesired' data for training while still ensuring the model generates samples similar to the desired data.
	The address this problem, the authors propose to use an additional set S of pairwise samples  (X_1,X_2) called the set of preferences to guide the training of a WGAN. This set specifies that the user prefers sample X_1 over X_2.  X_1 is typically  a sample from the desired data and X_2 is an undesired sample. This provides an additional information during training to favor samples that are similar to X_1.
	The authors then propose to encourage the critic to give a higher score to sample X_1 and a lower score to sample X_2. This is achieved by adding a penalty to the WGAN loss. This  penalty is simply given by the margin loss between the scores of X_1 and X_2.    
	The critic loss is then a tradeoff between two terms: the standard WGAN term which encourages 'data quality' and the preference term which favors samples similar to the preferred data. 
	 The authors then identify two major training situations which determine how easily the tradeoff can be achieved: 'minor correction' and 'major correction'.
	- The minor correction is when the 'desired data' is very similar to the whole dataset. Thus the tradeoff is easily achieved using the proposed loss of eq 4.
	- The major correction is the harder situation where the 'desired' data are different from the whole dataset. Thus using the proposed method directly could either learn to preference at the expense of sample quality or maintain good quality but without learning the preference. To address this, the authors propose a modification of the algorithm that consist in replacing part of the training samples by generated ones at every epoch. The authors then claim that this addresses the problem since the modified training data should get closer to the desired data at every epoch.
	The authors then demonstrate empirically the ability of the proposed methods to successfully learn the desired distribution while using shared structure with the undesired data. This is first illustrated in a simple synthetic example in section 4. then in section 5, the author compare the proposed method with other baselines on MNIST and CelebA datasets. 

	The authors conclude that the proposed method has two advantages compared to existing methods.
	- The method doesn't need a pre-defined score function to distinguish between desired and undesired data. That is because they instead rely on the pairwise preference set S. 
	- While other methods do not use undesired data for training, the proposed method can effectively exploit those data to yield better performance when the desired dataset is scarce.

- Strength of the paper:
	The author show experimentally that the proposed method is able to take advantage of negative samples to learn the user-desired distribution. This allows to use fewer supervision than the method FBGAN which only relies on positive samples. This is illustrated in the experiment in 5.1.2. where the number of supervision is limited in purpose.

- Weakness of the paper:
	The main weakness is the soundness of the approach and the clarity of the paper.	
	- In definition 1 (problem setting), the  authors first claim in the that proposed method allows to learn the desired distribution. But it remains unclear from the structure of the loss whether this is the case. It seems that the loss achieves only a tradeoff between learning the desired data and being close the whole training set. This is in particular apparent from the discussion about the minor and major correction scenarios. The statement in definition 1 should be adapted to reflect this limitation of the method and avoid confusion.
	- In the major correction scenario, the authors propose to introduce fresh generated samples as part of the training set at every epoch. By doing so, the authors claim that the modified training set is getting closer and closer to the desired distribution thus leading to a decreasing sequence of distances to the desired data. I honestly don't see how this solves the problem: 
		- Why would one obtain a decreasing sequence as claimed?
		- How does it prevent the quality of the generated data from being degraded? It doesn't seem like using bad generated samples as part of the training set could help prevent the degradation of sample quality.
		- It would be good to have either theoretical or empirical evidence of why this approach is relevant.
	
	- The authors present a limitation of FBGAN as needing a predefined score function, while the proposed method relies on a set of pairwise preferences. Yet in practice, it seems that the set S is construction using a classifier or from the labels of the data. Isn't it equivalent to having  a pre-defined score function?

	- In 5.1.1, the authors compare the number "effective pairs" used during training in the case of FBGAN and the proposed method. For the proposed method, this is simply the number of elements in the set S. But for the competing method FBGAN which doesn't rely on pairs, I don't see what it means. The authors say that is corresponds to the product of desired and undesired samples. Is that a relevant quantity to consider for FBGAN? The authors then compare those numbers and conclude that the proposed method requires way fewer effective pairs. But I'm not convinced this is a fair comparison. Perhaps what could be more relevant is to simply compare the performance as a function of the number of epochs:  FID score, percentage of desired samples every epoch.
	
- Clarity:
	While the experiment sections are clear  overall, I found section 3 to be very confusing:
	- My current understanding is that, the score loss encourages the critic to take into account a preference of the user. Such preference is supposed to be encoded by the set of 'pairwise preferences' S. The procedure makes sense to me if the set $S$ is pre-determined by the user.
	- However, section 3.2.1 is confusing in that it suggests that the set S is constructed using the critic itself. More precisely, what was confusing is that the ranking between samples (which is used to construct the set S in equation 2) is defined by means of the score function $f$ (the last two sentences before eq 3.). Then the author say that the score $f$ is chosen to be the critic $D$ (which is learned during training). This clearly wouldn't makes sense, since no information about the user's preference is included.  Later in the experiments sections, the authors clarify how the set S is pre-defined, but still this makes the reading experience very confusing.
	- The background section do not contain relevant concepts and work that are key to understand the paper: The authors take inspiration from Relative GAN to designing the critic, but we don't know exactly how. This should be discussed in more details before presenting the method. The same is valid for FBGAN, a competing method, which is partially introduced in the method section. Only later, in the experiment section, that the reader discovers some subtleties in how FBGAN are trained. In particular, the authors explain in section 5 that the generated samples need also to be scored during training in the case of FBGAN, however, this is never mentioned before and seems to be critical to understand the experiments of section 5.1.2 (limited supervision scenario). 

	 - In 5.1, the authors say that they use a pre-trained classifier to distinguish between desired and undesired generated data. This suggests that the generated samples are also used to construct the set S? However, isn't S an input to the algorithm and thus independent of the generator? Are those samples only used to evaluate to what extend the generator learns the preferred distribution?


- Questions and suggestions:
	- The training algorithm should be in the main paper.
	- How does the gap in performance between the two methods evolve as the number of supervision increases?
	- Is this behavior consistent across datasets? For instance on CelebA? It seems like on CelebA both methods perform equally well. However, showing that the method still requires fewer supervision in the case of datasets such as CelebA could be strengthen the results.
	- Why do the critic and generators need to be pre-trained? What happens when training from scratch?
	- 'As MNIST contains digits from 0 to 9, 0 is the most desirable data'  This sentence doesn't really make sense, or does it?

Updated review: Thank you  for your response and  for your efforts in addressing those points. I raised my score to 5 for the clarifications made to the document. However, I still think there are unjustified claims about the method, especially those related to the strong correction scenario and how introducing fresh generated samples is an effective procedure (conceptually / theoretically).

review_title: Interesting results, but lacks clarity in the presentation and the soundness of the approach needs more evidence

Meta-review: The paper presents a method to regularize the discriminator in  GAN training with a ranking loss based on the user preference for a desired set within a larger dataset. The tradeoff between GAN loss and preference loss dependence on the distance of the set to the full dataset and the authors consider two regimes : "small and major correction". A major correction is needed when the targeted set is very different from the whole density, authors propose in this scenario to replace samples from the data by samples from the generator. The setting in the paper is interesting and can be useful in practice. 

There was a lengthy discussions between the authors and the reviewers, the discussion pinpointed issues , some of them were addressed in the rebuttal . Some issues remain unanswered regarding the clarity and some claims in the paper.

The clarity of the paper needs further improvement and  1)  clarify section 3  the setup and the background section   2)  justify claims about the method, in  the strong correction scenario  when fresh generated samples are introduced how  is this an effective procedure? (conceptually / theoretically). 

Decision: Reject