Title: Adversarial Feature Learning
Authors: Jeff Donahue
Philipp Krähenbühl
Trevor Darrell
Venue: ICLR 2017

Abstract: The ability of the Generative Adversarial Networks (GANs) framework to learn generative models mapping from simple latent distributions to arbitrarily complex data distributions has been demonstrated empirically, with compelling results showing generators learn to "linearize semantics" in the latent space of such models. Intuitively, such latent spaces may serve as useful feature representations for auxiliary problems where semantics are relevant. However, in their existing form, GANs have no means of learning the inverse mapping -- projecting data back into the latent space. We propose Bidirectional Generative Adversarial Networks (BiGANs) as a means of learning this inverse mapping, and demonstrate that the resulting learned feature representation is useful for auxiliary supervised discrimination tasks, competitive with contemporary approaches to unsupervised and self-supervised feature learning.


Review discussion:
Given that you have the encoder from the pixel space to the latent space, I wonder whether it is possible to calculate log-likelihoods of the data under the model. As shown by Theis et al (https://arxiv.org/abs/1511.01844),  evaluating only on samples as indication of a good model fit is very problematic. I would be very interested to see how the model compares to other models like Pixel-RNN in terms of likelihood.
An additional question regarding Table 2: For the two rows about your model, I understand that you retraing the deeper layers supervised on ImageNet.
However I am not sure how to interpret the rows above. Which networks exactly did you freeze and retrain here? If it is other networks than your own, do they have the same architecture (e.g. conv1-conv5)?
> Given that you have the encoder from the pixel space to the latent space, I wonder whether it is possible to calculate log-likelihoods of the data under the model.
The idea of using the encoder to compute a log-likelihood is intriguing. One difficulty could be that Pixel-RNN uses a discrete distribution, while GAN based models rely on a continuous parametrization. It might be possible to compare the two by integrating over the probability density P_{EX}.  We will consider this further.
> An additional question regarding Table 2: For the two rows about your model, I understand that you retrain the deeper layers supervised on ImageNet.
However I am not sure how to interpret the rows above. Which networks exactly did you freeze and retrain here? If it is other networks than your own, do they have the same architecture (e.g. conv1-conv5)?
All Table 2 results other than our own are from Noroozi & Favaro (2016).  All networks included in the table are minor variants on AlexNet, pre-trained for the self-supervised/unsupervised task described in the cited paper.  The Noroozi & Favaro architecture might have a slight advantage here as their model produces larger intermediate feature maps -- with stride 2 (rather than 4) in the conv1 layer. (We will make a note of this difference in the results table.) All others use the same convolutional and pooling layer settings (kernel size, stride, padding) used in AlexNet.
For BiGAN,  G(E(x)) seems produces reasonable output similar to x. Do you observe improvement of diversity of G by using Z?
Also, Z is usually a uniform distribution, but seems E(x) should be very different to Z. What is relation of E(G(Z)) to Z?
> Do you observe improvement of diversity of G by using Z?
We haven’t figured out a principled way to measure the diversity of generator outputs.  Anecdotally, the BiGAN generator outputs appear to be similar to standard GAN generator outputs.
> Also, Z is usually a uniform distribution, but seems E(x) should be very different to Z.
We show in Theorem 1 that P_{EX} = P_{GZ} at the global optimum.  As a necessary condition for this, the distribution over E(x) (with x ~ p_x) must be uniform as well. We understand that this is counter-intuitive as this is not the typical latent feature distribution produced by convnets. We tried other distributions, such as Gaussians, with similar results.
> What is relation of E(G(Z)) to Z?
Theorem 2 shows that for all z in the support of P_Z, we have E(G(z)) = z (symmetric to the G(E(x)) = x result).
In practice, we don’t train D or G/E to convergence (doing so may be difficult or impossible), so these properties will not hold exactly, just as G(E(x)) = x doesn’t hold exactly as shown in Figs. 2 and 4.
The authors extend GANs by an inference path from the data space to the latent space and a discriminator that operates on the joint latend/data space. They show that the theoretical properties of GANs still hold for BiGAN and evaluate the features learned unsupervised in the inference path with respect to performance on supervised tasks after retraining deeper layers.
I see one structural issue with this paper: Given that, as stated in the abstract, the main purpose of the paper is to learn unsupervised features (and not to improve GANs), the paper might spent too much space on detailing the relationship to GANs and all the theoretical properties. It is not clear whether they actually would help with the goal of learning good features.
While reading the paper, I actually totally forgot about the unsupervised features until they reappeared on page 6. I think it would be helpful if the text of the paper would be more aligned with this main story.
Still, the BiGAN framework is an elegant and compelling extension to GANs.
However, it is not obvious how much the theoretical properties help us as the model is clearly not fully converged. To me, especially Figure 4 seems to suggest that G(E(x)) might be doing not much more than some kind of nearest neighbour retrival (and indeed one criticism for GANs has always been that they might just memorize some samples). By the way, it would be very interesting to know how well the discriminator actually performs after training.
Coming back to the goal of learning powerful features: The method does not reach state-of-the-art performance on most evaluated tasks (Table 2 and 3) but performs competitive and it would be interesting to see how much this improves if the BiGAN training (and the convolutional architecture used) would be improved.
The paper is very well written and provides most necessary details, although some more details on the training (learning rates, initialization) would be helpful for reproducing the results.
Overall I think the paper provides a very interesting framework for further research, even though the results presented here are not too impressive both with respect to the feature evaluation (and the GAN learning).
Minor: It might be helpful to highlight the best performance numbers in Tables 2 and 3.
This is a parallel work with ALI.  The idea is using auto encoder to provide extra information for discriminator. This approach seems is promising from reported result. For feature learning part of BiGAN, there still is a lot of space to improve, compare to standard supervised convnet.
This paper provides an interesting idea, which extends GAN by taking into account bidirectional network. Totally, the paper is well-written, and easy to follow what is contribution of this paper. From the theoretical parts, the proposed method, BiGAN, inherits similar properties in GAN. The experimental results show that BiGAN is competitive with other methods. A drawback would a non-convex optimization problem in BiGAN, this paper is still suitable to be accepted in my opinion.
We thank all reviewers for their thoughtful comments and suggestions. We’ve uploaded a revision with an expanded introduction which clarifies the motivation for BiGAN and the theory in Section 3 in the context of feature learning.
Note that our theoretical arguments go beyond simply showing that BiGAN maintains the properties of the original GAN framework.
In particular, we show that the BiGAN objective is equivalent to that of an “L0 autoencoder”, encouraging the encoder and generator to invert one another (Theorem 3). If fully optimized, the encoder and generator are inverses of one another almost everywhere (Theorem 2).  We believe that these insights are quite important to understanding BiGAN’s behavior and its use as a model for feature learning.  We hope that the revised introduction explains this better.
With respect to the ImageNet visual feature learning evaluation, our results show BiGAN is state-of-the-art among purely unsupervised learning methods.  The methods that outperform BiGAN (e.g., Doersch et al.) are all based on “self-supervision,” requiring the design of a domain-specific supervised prediction task, and some (Agrawal et al. and Wang & Gupta) are “weakly supervised,” requiring auxiliary information (video, egomotion, or tracking) unavailable in images alone.  We’ve reorganized Table 3 to emphasize this point, and added discussion to Section 4.4 on the benefits of purely unsupervised approaches vs. self-supervised approaches.  (Following your suggestion, we’ve also bolded the best results of each group in Tables 2 and 3.)  We do agree, however, that our results are somewhat preliminary and likely to improve significantly with further model architecture search and improvements to the optimization.
Most optimization details (optimization algorithm, learning rate / step size, etc.) were included in Appendix C.  In the updated version, network initialization details are now included in Appendix C as well.  We’ll also be releasing the training code.
Note: the latest version includes significantly improved ImageNet classification results in Table 2 (up ~2% across the board compared to the previous version).  This is due to the fact that in previous versions, we evaluated using the predictions for a single (center) crop at test time, rather than averaging over 10 crops as we learned by correspondence was how the previous results from Noroozi & Favaro (2016) were obtained.  (Thanks to Mehdi for bringing this up!) There are also very slightly improved VOC classification results (Table 3) due to an unrelated minor bug.

Meta-review:
role: PC
data: comment: All reviewers unanimously praised the novelty and quality of the paper. Minor revisions, following the reviewers' suggestions, will make the paper even better. 

Decision: Accept (Poster)