Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper incorporated hierarchical representation of complex, reichly-structured data to extend the Adversarially Learned Inference (Dumoulin et al. 2016) to achieve hierarchical generative model. The hierarchical ALI (HALI) learns a hierarchy of latent variables with a simple Markovian structure in both the generator and inference. The work fits into the general trend of hybrid approaches to generative modeling that combine aspects of VAEs and GANs. 

The authors showed that within a purely adversarial training paradigm, and by exploiting the model’s hierarchical structure, one can modulate the perceptual fidelity of the reconstructions. We provide theoretical arguments for why HALI’s adversarial game should be sufficient to minimize the reconstruction cost and show empirical evidence supporting this perspective.

The performance of HALI were evaluated on four datasets, CIFAR10, SVHN, ImageNet 128x128 and CelebA. The usefulness of the learned hierarchical representations were demonstrated on a semi-supervised task on MNIST and an attribution prediction task on the CelebA dataset. The authors also noted that the introduction of a hierarchy of latent variables can add to the difficulties in the training. 

Summary:
——
In summary, the paper discusses a very interesting topic and presents an elegant approach for modeling complex, richly-structured data using hierarchical representation. The numerical experiments are thorough and HALI is shown to generate better results than ALI. Overall, the paper is well written. However, it would provide significantly more value to a reader if the authors could provide more details and clarify a few points. See comments below for details and other points.

Comments:
——
1.	Could the authors comment on the training time for HALI? How does the training time scale with the levels of the hierarchical structure?

2.	How is the number of hierarchical levels $L$ determined? Can it be learned from the data? Are the results sensitive to the choice of $L$?

3.	It seems that in the experimental results, $L$ is at most 2. Is it because of the data or because of the lack of efficient training procedures for the hierarchical structure?



review_title: review

final_score: rating: 5: Marginally below acceptance threshold
review_content: _________________________________________________________________________________________________________

I raise my rating on the condition that the authors will also address the minor concerns in the final version, please see details below.
_________________________________________________________________________________________________________

This paper proposes to perform Adversarially Learned Inference (ALI) in a layer-wise manner. The idea is interesting, and the authors did a good job to describe high-level idea, and demonstrate one advantage of hierarchy: providing different levels reconstructions. However, the advantage of better reconstruction could be better demonstrated.  Some major concerns should be clarified before publishing:

(1) How did the authors implement p(x|z) and q(z|x), or p(z_l | z_{l+1}) and q(z_{l+1} | z_l )? Please provide the details, as this is key to the reconstruction issues of ALI.

(2) Could the authors provide the pseudocode procedure of the proposed algorithm? In the current form of the writing, it is not clear what the HALI procedure is, whether (1) one discriminator is used to distinguish the concatenation of (x, z_1, ..., z_L), or (2) L discriminators are used to distinguish the concatenation of (z_l, z_{l+1}) at each layer, respectively?

The above two points are important. If not correctly constructed, it might reveal potential flaws of the proposed technique.

Since one of the major claims for HALI is to provide better reconstruction with higher fidelity than ALI. Could the authors provide quantitative results on MNIST and CIFAR to demonstrate this? The reconstruction issues have first been highlighted and theoretically analyzed in ALICE [*], and some remedy has been proposed to alleviate the issue.  Quantitative comparison on MNIST and CIFAR are also conducted. Could the authors report numbers to compare with them (ALI and ALICE)? 

The 3rd paragraph in Introduction should be adjusted to correctly clarify details of algorithms, and reflect up-to-date literature. "One interesting feature highlighted in the original ALI work (Dumoulin et al., 2016) is that ... never explicitly trained to perform reconstruction, this can nevertheless be easily done...". Note that ALI can only perform reconstruction when the deterministic mapping is used, while ALI itself adopted the stochastic mapping. Further, the deterministic mapping is the major difference of BiGAN from ALI. Therefore, more rigorous way to phrase is that "the original ALI work with deterministic mappings", or "BiGAN" never explicitly trained to perform reconstruction, this can nevertheless be easily done... This tiny difference between deterministic/stochastic mappings makes major difference for the quality of reconstruction, as theoretically analyzed and experimentally compared in ALICE. In ALICE, the authors confirmed further source of poor reconstructions of ALI in practice. It would be better to reflect the non-identifiability issues raised by ALICE in Introduction, rather than hiding it in Future Work as "Although recent work designed to improve the stability of training in ALI does show some promise (Chunyuan Li, 2017), more work is needed on this front."

Also, please fix the typo in reference as:
[*] Chunyuan Li, Hao Liu, Changyou Chen, Yunchen Pu, Liqun Chen, Ricardo Henao and Lawrence Carin. ALICE: Towards understanding adversarial learning for joint distribution matching. In Advances in Neural Information Processing Systems (NIPS), 2017.


 
review_title: An interesting idea, but important details are missing

final_score: rating: 5: Marginally below acceptance threshold
review_content: ******
Please note the adjusted review score after revisions and clarifications of the authors. 
The paper was improved significantly but still lacks novelty. For context, multi-layer VAEs also were not published unmodified as follow-up papers since the objective is identical. Also, I would suggest the authors study the modified prior with marginal statistics and other means to understand not just 'that' their model performs better with the extra degree of freedom but also 'how' exactly it does it. The only evaluation is sampling from z1 and z2 for reconstruction which shows that some structure is learned in z2 and the attribute classification task. However, more statistical understanding of the distributions of the extra layers/capacity of the model would be interesting.
******

The authors propose a hierarchical GAN setup, called HALI, where they can learn multiple sets of latent variables.
They utilize this in a deep generative model for image generation and manage to generate good-looking images, faithful reconstructions and good inpainting results.

At the heart of the technique lies the stacking of GANS and the authors claim to be proposing a novel model here.
First, Emily Denton et. al proposed a stacked version of GANs in "Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks", which goes uncited here and should be discussed as it was the first work stacking GANs, even if it did so with layer-wise pretraining.
Furthermore, the differences to another very similar work to that of the authors (StackGan by Huan et al) are unclear and not well motivated.
And third, the authors fail to cite 'Adversarial Message Passing' by Karaletsos 2016, which has first introduced joint training of generative models with structure by hierarchical GANs and generalizes the theory to a particular form of inference for structured models with GANs in the loop. 
This cannot be called concurrent work as it has been around for a year and has been seen and discussed at length in the community, but the authors fail to acknowledge that their basic idea of a joint generative model and inference procedure is subsumed there. In addition, the authors also do not offer any novel technical insights compared to that paper and actually fall short in positioning their paper in the broader context of approximate inference for generative models.

Given these failings, this paper has very little novelty and does not perform accurate attribution of credit to the community.
Also, the authors propose particular one-off models and do not generalize this technique to an inference principle that could be reusable.

As to its merits, the authors manage to get a particularly simple instance of a 'deep gan' working for image generation and show the empirical benefits in terms of image generation tasks. 
In addition, they test their method on a semi-supervised task and show good performance, but with a lack of details.

In conclusion, this paper needs to flesh out its contributions on the empirical side and position its exact contributions accordingly and improve the attribution.
review_title: The authors propose a hierarchical GAN variant of ALI, but offer little novelty or insights.

Meta-review: Pros:
- The paper proposes to use a hierarchical structure to address reconstruction issues with ALI model.
- Obtaining multiple latent representations that individually achieve a different level of reconstructions is interesting.  
- Paper is well written and the authors made a reasonable attempt to improve the paper during the rebuttal period. 

Cons:
- Reviewers agree that the approach lacks novelty as similar hierarchical approaches have been proposed before. 
- The main goal of the paper to achieve better reconstruction in comparison to ALI without changing the latter's objective seems narrow. More analysis is needed to demonstrate that the approach out-performs other approaches that directly tackle this problem in ALI.
- The paper does not provide strong arguments as to why hierarchy works (limited to 2 levels in the empirical analysis presented in the paper). 
- Semi-supervised learning as a down-stream task is impressive but limited to MNSIT.  
Decision: Reject