Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes an alternative GAN formulation that replaces the standard binary classification task in the discriminator with a autoregressive model that attempts to capture discriminative feature dependencies on the true data samples.

Summary assessment:
The paper presents a novel perspective on GANs and an interesting conjecture regarding the failure of GANs to capture global consistency. However the experiments do not directly support this conjecture. In addition, both qualitative and quantitative results to not provide significant evidence of the value of this technique over and above the establish methods in the literature.

The central motivation of the method proposed in the paper, is a conjecture that the lack of global consistency in GAN-generated samples is due to the binary classification formulation of the discriminator. While this is an interesting conjecture, I am somewhat unconvinced that this is indeed the cause of the problem. First, I would argue that other high-performing auto-regressive models such as PixelRNN and PixelCNN also seem to lack global consistency. This observation would seem to violate this conjecture. More importantly, the paper does not show any direct empirical evidence in support of this conjecture. 

The authors make a very interesting observation in their description of the proposed approach. In discussing an initial variant of the model (Eqns. (5) and (6) and text immediately below), the authors state that attempting to maximize the negative log likelihood of the auto-regressive modelling the generated samples results in unstable training. I would like to see more discussion of this point as it could have some bearing on the difficulty of GAN to model sequential data in general. Does the failure occurs because the auto-regressive discriminator is able to "overfit" the generated samples?

As a result of the observed failure of the formulation given in Eqns. (5) and (6), the authors propose an alternative formulation that explicitly removes the negative likelihood maximization for generated samples. As a result the only objective for the auto-regressive model is an attempt to maximize the log-likelihood of the true data. The authors suggest that this should be sufficient to provide a reliable training signal for the generator. It would be useful if the authors showed a representation of these features (perhaps via T-SNE) for both true data and generated samples. 

Empirical results:
The authors experiments show samples (qualitative comparison) and inception scores (quantitative comparison) for 3 variants of the proposed model and compare these to methods in the literature. The comparisons show the proposed model preforms well, but does not exceed the performance of many of the existing methods in the literature.

Also, I fail to observe significantly more global consistency for these samples compared to samples of other SOTA GAN models in the literature. Again, there is no attempt made by the authors to make this direct comparison of global consistency either qualitatively or quantitatively. 

Minor comment:
I did not see where PARGAN was defined. Was this the combination of the Auto-regressive GAN with Patch-GAN?

review_title: Interesting idea. Insufficient empirical support.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a new GAN model whereby the discriminator (rather than being a binary classifier) consists of an encoding network followed by an autoregressive model on the encoded features. The discriminator is trained to maximize the probability of the true data and minimize the probability of the generated samples.  The authors also propose a version that combines this autoregressive discriminator with a patchGAN discriminator. The authors train this model on cifar10 and stl10 and show reasonable generations and inception scores, comparing the latter with existing approaches. 

Pros: This discriminator architecture is well motivated, intuitive and novel. the samples are good (though not better than existing approaches as far as I can tell). The paper is also well written and easy to read.

Cons: As is commonly the case with GAN models, it is difficult to assess the advantage of this approach over exiting techniques. The samples generated form this model look fine, but not better than existing samples. The inception scores are ok, but seem to be outperformed by other models (though this shouldn't necessarily be taken as a critique of the approach presented here as inception scores are an approximation to what we care about and we should not be trying to tune models for better inception scores). 

Detailed comments:
- In terms of experiments, I think think paper is missing the following: (1) An additional dataset -- cifar and stl10 are very similar, a face dataset for example would be good to see and is commonly used in GAN papers. (2) the authors claim their method is stable, so it would be good to see quantitative results backing this claim, i.e. sweeps over hyper-parameters / encoding/generator architectures with evaluations for different settings. 
- the idea of having some form of  recurrent (either over channels of spatially)  processing in the discriminator seems more general that the specific proposal given here. Could the authors say a bit more about what they think the effects of adding recurrence in the discriminator vs optimizing the likelihood of the features under the autoregressive model?

Ultimately, the approach is interesting but there is not enough empirical evaluations.

review_title: interesting gan architecture, evaluations limited

final_score: rating: 3: Clear rejection
review_content: This work attempts to improve the global consistency of samples generated by generative adversarial networks by replacing the discriminator with an autoregressive model in an encoded feature space. The log likelihood of the classification model is then replaced with the log likelihood of the feature space autoregressive model. It's not clear what can be said with respect to the convergence properties of this class of models, and this is not discussed.

The method is quite similar in spirit to Denoising Feature Matching of Warde-Farley & Bengio (2017), as both estimate a density model in feature space -- this method via a constrained autoregressive model and DFM via an estimator of the score function, although DFM was used in conjunction with the standard criterion whereas this method replaces it. This is certainly worth mentioning and discussing. In particular the section in Warde-Farley & Bengio regarding the feature space transformation of the data density seems quite relevant in this work.

Unfortunately the only quantitative measurements reporter are Inception scores, which is known to be a poor measure (and the scores presented are not particularly high, either); Frechet Inception distance or log likelihood estimates via AIS on some dataset would be more convincing. On the plus side, the authors report an average over Inception scores for multiple runs. On the other hand, it sounds as though the stopping criterion was still qualitative.
review_title: Autoregression in feature space

Meta-review: The reviewers (all experts in this area) appreciated the novelty of the idea, though they felt that the experimental results (samples and Inception scores) did not provide convincing evidence value of this method over already established techniques. The authors responded to the concerns but were not able to address the issue of evaluation due to time constraints. The idea is likely sound but evaluation does not meet the bar, it may make a good contribution as a workshop paper.
Decision: Invite to Workshop Track