Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Hi, if I am reading the paper correctly, it seems from each Langevin dynamics, only one sample is used.  This appears to be case for the descriptor network where \tilde{n} parallel dynamics are run and then expectation is computed using one sample each from those.  This is perhaps ok if \tilde{n} is large enough, however why not take more samples from each dynamics?
For the case of generator network an expectation is needed for each training sample, and it appears that this is approximated by only one sample. 
I may be misreading things or there may be a lack in my understanding of how you are applying Langevin dynamics.  Would appreciate a clarification.
review_title: Question about Langevin dynamics used to compute expectations

final_score: rating: null
review_content: Could the authors comment on the lack of train/test split for the results of Fig. 2, 3 ? Previous work on texture modeling has split individual textures in two (training and testing halves) and featured a numerical evaluation based on the Textual Similarity Score. This is particularly worrisome as there are clear signs of overfitting (not surprising given size of training set). As for Table 1, why did the authors not include more recent baselines such Variational Auto-Encoders ?

[R1] Heess, N., Williams, C. K. I., and Hinton, G. E. (2009). Learning generative texture models with extended fields of-experts.
review_title: Train / test split

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposed a new joint training scheme for two probabilistic models of signals (e.g. images) which are both deep neural network based and are termed generator and descriptor networks.  In the new scheme, termed cooperative training, the two networks train together and assist each other: the generator network provides samples that work as initial samples for the descriptor network, and the descriptor network updates those samples to help guide training of the generator network.

This is an interesting approach for coupling the training of these two models.  The paper however is quite weak on the empirical studies.  In particular:
- The training datasets are tiny, from sets of 1 image to 5-6.  What is the reason for not using larger sets?  I think the small datasets are leading to over training and are really masking the true value of the proposed cooperative training approach.
- For most of the experiments presented in the paper it is hard to assess the specific value brought by the proposed cooperative training approach because baseline results are missing.  There are comparisons provided for face completion experiments - but even there comparisons with descriptor or generator network trained separately or with other deep auto-encoders are missing.  Thus it is hard to conclude if and how much gain is obtained by cooperative training over say individually training the descriptor and generator networks.

Another comment is that in the “related work” section, I think relation with variational auto encoders (Kingma and Welling 2013) should be included.

Despite limitations mentioned above, I think the ideas presented in the paper are intuitively appealing and worth discussing at ICLR.  Paper would be considerably strengthened by adding more relevant baselines and addressing the training data size issues.


review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors proposes an interesting idea of connecting the energy-based model (descriptor) and 
the generator network to help each other. The samples from the generator are used as the initialization 
of the descriptor inference. And the revised samples from the descriptor is in turn used to update
the generator as the target image. 

The proposed idea is interesting. However, I think the main flaw is that the advantages of having that 
architecture are not convincingly demonstrated in the experiments. For example, readers will expect 
quantative analysis on how initializing with the samples from the generator helps? Also, the only 
quantative experiment on the reconstruction is also compared to quite old models. Considering that 
the model is quite close to the model of Kim & Bengio 2016, readers would also expect a comparison 
to that model. 

** Minor
- I'm wondering if the analysis on the convergence is sound when considering the fact that samples 
from SGLD are biased samples (with fixed step size). 
- Can you explain a bit more on how you get Eqn 8? when p(x|y) is also dependent on W_G?

review_title: Official Review

final_score: rating: 3: Clear rejection
review_content: This paper introduces CoopNets, an algorithm which trains a Deep-Energy Model (DEM, the “descriptor”) with the help of an auxiliary directed bayes net, e.g. “the generator”. The descriptor is trained via standard maximum likelihood, with Langevin MCMC for sampling. The generator is trained to generate likely samples under the DEM in a single, feed-forward ancestral sampling step. It can thus be used to shortcut expensive MCMC sampling, hence the reference to “cooperative training”.

The above idea is interesting and novel, but unfortunately is not sufficiently validated by the experimental results. First and foremost, two out of the three experiments do not feature a train /test split, and ignore standard training and evaluation protocols for texture generation (see [R1]). Datasets are also much too small. As such these experiments only seem to confirm the ability of the model to overfit. On the third in-painting tasks, baselines are almost non-existent: no VAEs, RBMs, DEM, etc which makes it difficult to evaluate the benefits of the proposed approach.

In a future revision, I would also encourage the authors to answer the following questions experimentally. What is the impact of the missing rejection step in Langevin MCMC (train with, without ?). What is the impact of the generator on the burn-in process of the Markov chain (show sample auto-correlation). How bad is approximation of training the generator from ({\tilde{Y}, \hat{X}) instead of ({\tilde{Y}, \tilde{X}) ? Run comparative experiments.

The paper would also greatly benefit from a rewrite focusing on clarity, instead of hyperbole (“pioneering work” in reference to closely related, but non-peer reviewed work) and prose (“tale of two nets”). For example, the authors fail to specify the exact form of the energy function: this seems like a glaring omission.

PROS:
+ Interesting and novel idea
CONS:
- Improper experimental protocols
- Missing baselines
- Missing diagnostic experiments

[R1] Heess, N., Williams, C. K. I., and Hinton, G. E. (2009). Learning generative texture models with extended fields of-experts.
review_title: Interesting idea, but improperly evaluated

final_score: rating: null
review_content: -
review_title: -

Meta-review: While the paper may have an interesting theoretical contribution, it seems to greatly suffer from problems in the presentation: the basic motivation is of the system is hardly mentioned in the introduction, and the conclusion does not explain much either. I think the paper should be rewritten, and, as some of the reviewers point out, the experiments strengthened before it can be accepted for publication.
 
 (I appreciate the last-minute revisions by the authors, but I think it really came too late, 14th/21st/23rd Jan, to be taken seriously into account in the review process.)
Decision: Reject