Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper makes the following contribution:
- using the AdaIn architecture proposed by Karras et al., 2019 with the autoencoding architecture of AGE/PIONEER;
- a cyclic loss to enforce disentangling between different layers;
- a method to enforce invariances at specific layers.

The adaptation of the AdaIn architecture in an autoencoding fashion (a la AGE/PIONEER) is sensible and well motivated, combining state-of-the-art generator while allowing inference in a compact setting (i.e. not requiring an additional discriminator). 

The other contribution are harder to read and the writing should be improved.
The cyclic loss should be better described. The notation of the KL divergence is confusing if you are using the KL divergence defined by AGE/PIONEER and will need to be explained. I will also assume that d_cos is the cosine loss as defined by the PIONEER paper. This should be mentioned as well.
The method to enforce invariance is also not clear to me. While the authors introduce F as a "known invariance", it is unclear what role it plays in the cost function. Is F an invariant on which we measure this reconstruction loss d? What is d? Explaining that might shed light on the result Figure 5, e.g. why the images become blurry when doing this rotation. 

The experiment demonstrates the sampling quality of the model and the transfer of features at different level (coarse-medium-fine) Figure 4. It is unclear what was the contribution of the layer specific loss metric to allow that feature transfer. It seems from Figure 5 the invariance objective has been roughly satisfied but at the cost of a significant drop in image quality.

The clearest contribution from this paper is definitely the AGE/PIONEER approach to train the AdaIn architecture. The two other contributions are unclear, both in their explanation and in what they contribute: the layer specific loss not compared to an architecture just trained in an AGE way, and the enforcing of invariance, although filling its objective, might deteriorate other desirable properties of the model (e.g. sample quality). 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Deep Automodulators introduces a generative autoencoder architecture that replaces the canonical encoder decoder autoencoder architecture with one inspired by StyleGAN. The encoder interacts with the decoder by modulating layer statistics via Adaptive Instance Normalization (AdaIN) conditioned on the latent. The paper trains this architecture with the loss framework of the Adversarial Generator–Encoder (AGE) and utilizes the progressive growing trick originally introduced in Progressive GAN which is also adapted by the Pioneer models, recent followups to AGE.

The use of AdaIN conditioning across multiple layers and multiple scales (like StyleGAN) and the ability to directly compute latent codes via the encoder allows the authors introduce a disentanglement objective L_j and also an invariance objective L_inv to help encourage these properties in the models via consistency objectives 

The paper shows results demonstrating StyleGAN style coarse/fine visual transfer on two high quality face datasets (importantly this is demonstrated on real inputs rather than samples as in StyleGAN) as well as respectable sample quality on LSUN Bedrooms and the LSUN Cars dataset.

My decision is weak reject. Overall, I think the paper is promising and shows a nice combination of efficient latent inference and controllable generation but the authors do not include ablations to validate some of their core contributions such as the L_j objective. Additionally, the improved controllability of the approach seems to unfortunately result in lower reconstruction quality than direct prior work such as Balanced Pioneer and this potential tradeoff is not investigated/discussed.

To expand a bit, there are three changes from that prior work that that stood out to me. 1) The StyleGAN inspired architecture 2) the disentangling objective L_j and 3) using the loss function dρ of Barron 2019. Successful ablations to demonstrate the importance of 2) to the presented results as well as better motivating / demonstrating the impact of including 3) would raise my score to an weak acceptance.

My other concern is that the reconstruction quality seems noticeably lower than that of the proceeding work, Balanced Pioneer. This is reflected in its 10% reduction in LPIPS compared to the Automodulator’s paper. In general there also seems to be noticeable grid artifacts in the samples across all datasets samples/reconstructions, which don’t seem as prominent in Balanced Pioneer. It is not immediately clear why this is the case and additional investigation of this, such as checking whether this is due to the introduction of the disentanglement objective, or the inclusion of the Barron 2019 loss function would be informative.

Additional Comments:

Each subsection of 3 could be improved by providing a brief introduction to the motivation for and aim of each contribution before launching directly into how it is implemented / achieved. Without that bit of context on the goals of each subsection, it was more difficult to follow along with what was being done and why.

The presentation of L_j with lots of inlined equations intermixed with text gets a bit difficult to read / follow.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The submission proposes an autoencoder architecture which combines two recent GAN-based architectural innovations, namely the progressive growing of the decoder architecture (as well as the encoder architecture in this case) and the use of the encoded representation to modulate the decoder via a feature-wise transformation mechanism.

I think the overall idea behind the paper is valuable, but I don’t think the submission meets the acceptance bar from a clarity point of view. I also have concerns with its characterization of the literature.

Clarity-related comments:

- “This allows the layers to work independently of each other.” This is an imprecise use of the term “independently”. How can two layers work independently if one’s input is the output of the other?
- “The reconstructed samples could be re-introduced to the encoder, repeating the process, and requiring consistency between passes.” The rest of the paragraph is built on the premise that this is a desirable property, but I’m not sure I understand why this is a desirable property in the first place.
- How to define “disentanglement” in the context of representation learning is in itself an unsettled question as far as I’m aware, but the submission uses the term without an intuitive or formal definition. What do the authors mean by “disentangled representation”? What is measured by perceptual path length (PPL), and in which ways does PPL relate to the author’s definition of “disentangled representation”?
- “[...] a new autoencoder-like model with powerful properties not found in regular autoencoders, including style transfer.” The term “style transfer” is overloaded; what do the authors mean?
- The submission defines AdaIn as a way to combine “content” and “style”, and defines the style “y” in terms of mean and variance. In the context of the AdaIn paper, this makes sense: the instance normalization shifting and scaling coefficients are heuristically defined as the channel-wise means and standard deviations of a “style” stack of feature maps. However, in the context of this submission I’m not sure this definition makes as much sense: the instance normalization shifting and scaling coefficients are the result of a linear projection of the latent representation and do not involve the channel-wise means and standard deviations of an external stack of feature maps; is this correct?
- “This setup follows the same logic as that of Karras et al. (2019), but we do not require an ad-hoc disentanglement stack.” Can the authors clarify what they mean by an “ad-hoc disentanglement stack”?
- Section 3.2 uses some notation for the encoder without introducing it first. I believe the only way to understand that \phi(x) refers to the encoder network is to look at Figure 2.
- Section 3.2 as a whole is hard to follow, in part due to the use of imprecise language (“mutually independent”, “representation of those levels disentangled in z”). At some point probability distributions are introduced (up until now the reader is operating under the assumption that the model is an autoencoder with no probabilistic interpretation), and mutual information is mentioned to justify an L2 reconstruction loss in z-space (which I would argue is an instance of mathiness that does not serve the reader’s comprehension). Can the authors explain in plain language how layer-specific losses are defined and how the complete loss is obtained?
- The submission presents model samples, but as far as I can tell the procedure for sampling is not provided. Unlike VAEs, autoencoders do not explicitly model the empirical distribution -- although reconstruction in denoising autoencoders is related to the score of the empirical distribution (Alain & Bengio, 2014). How are samples obtained from the trained model?

Literature-related comments:

- The use of normalization layers to implement feature-wise transformation mechanisms is fairly widespread nowadays, but for instance normalization specifically the work of Dumoulin et al. (2017) pre-dates that of Huang & Belongie (2017). Both are cited by Karras et al. (2019) in relation to AdaIn (which is termed “conditional instance normalization” in Dumoulin et al. (2017)).
- I disagree with the characterization of ALI/BiGAN as “hybrid models that combine the properties of VAEs and GANs”: unlike AAE and AVB, which minimize KL-divergence terms in the VAE loss adversarially, the objective for ALI/BiGAN is purely adversarial. I would also include IAE (Makhzani et al., 2018) and BigBiGAN (Donahue et al., 2019) in the list of GAN variants that incorporate an inference mechanism.
- The submission repeatedly asserts that GANs lack an inference mechanism: “Unlike GANs, autoencoder models can directly operate on input samples.”; “To work on new input images, GANs either need to be extended with a separate encoder, or inverted [...].”; “[...] GANs show good image quality, but have no built-in encoding mechanism [...]”; “[...] the problem with GANs is that they lack the encoder [...]”. This is false: see for example ALI, BiGAN, and BigBiGAN. The problem in my opinion is elsewhere: the kinds of reconstructions these models yield are not suited to the downstream applications investigated in this submission, because they oftentimes fail to preserve low-level details.

References:

- Alain, G., & Bengio, Y. (2014). What regularized auto-encoders learn from the data-generating distribution. The Journal of Machine Learning Research, 15(1), 3563-3593.
- Dumoulin, V., Shlens, J., & Kudlur, M. (2017). A learned representation for artistic style. In Proceedings of the International Conference on Learning Representations.
- Makhzani, A. (2018). Implicit autoencoders. arXiv:1805.09804.
- Donahue, J., & Simonyan, K. (2019). Large scale adversarial representation learning. arXiv:1907.02544.
review_title: Official Blind Review #4

Meta-review: The manuscript proposes an autoencoder architecture incorporating two recent architectural innovations from the GAN literature (progressive growing & feature-wise modulation), trained with the adversarial generator-encoder paradigm with a novel cyclic loss meant to encourage disentangling, and procedure for enforcing layerwise invariances. The authors demonstrate coarse/fine visual transfer on generative modeling of face images, as well as generative modeling results on several Large Scale Scene Understanding (LSUN) datasets. 

Reviewers generally found the results somewhat compelling and the ideas valuable and well-motivated, but criticized the presentation clarity, lack of ablation studies, and that the claims made were not sufficiently supported by the empirical evidence. The authors revised, and while it was agreed that clarity was improved, some reviewers were still not satisfied with the level of clarity (the revision appeared at the very end of the discussion period, unfortunately not allowing for any further refinement). Ablation studies were added in the revised manuscript, which were appreciated, but seemed to suggest that the proposed loss function was of mixed utility: while style-mixing quantitatively improved, overall sample quality appeared to suffer.

As the reviewers remain unconvinced as to the significance of the contribution and the clarity of its presentation, I recommend rejection at this time, while encouraging the authors to further refine the presentation of their ideas for a future resubmission.
Decision: Reject