Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper considers the problem of training VAEs which are robust to adversarial attacks. It shows that learning disentangled representations improves the robustness of VAE. However, this hurts the reconstruction accuracy. The paper then shows that using hierarchical VAEs can ensure robustness without sacrificing reconstruction.

Strengths:

1. The problem considered is interesting and relevant. There is very little work on training robust VAEs---though they have been found to not be robust, and are also used for training robust classifiers downstream.
2. The paper uncovers some interesting phenomenon about VAEs such as links between disentangled representations and robustness, and tradeoffs between disentanglement and reconstruction accuracy.
3. The method also provides some protection for downstream classification tasks.

Weaknesses:

1. There is a lack of baselines to compare against, and the paper has not really stress-tested the algorithm to ensure it is in fact robust. Most of the literature on adversarial examples does significantly more extensive testing to attempt to break the model. 
2. I found the story and the experimental evaluation a bit incomplete. Though the experiments demonstrate that the method achieves success in practice, the paper does not seem to sufficiently explore "why" it seems to work. I think this is a bit important here considering the previous point regarding baselines and testing. At the moment, it seems like the paper combines these two ideas of learning disentangled representation and hierarchical VAEs in a somewhat ad-hoc manner which ends up providing some robustness, but it is a bit opaque why this is supposed to work. Some experiments or ablation studies which establish a more direct link between disentangled representations and robustness will be helpful.

Overall, I think this is a decent paper but I don't feel that I can advocate strongly for it since the scientific contribution seems a bit limited. 

------Updates after author response------

I thank the authors for the response and the new experiments. In light of the clarifications and additional evaluations, I have increased my score to 7.
review_title: Suggests connections between robustness and disentangled representations, but isn't explored fully

final_score: rating: 6: Marginally above acceptance threshold
review_content: --Summary:
They proposed a robust method for the adversarial attack on VAE using a hierarchical version of $\beta$-TCVAE and conduct analysis on the relationship between disentanglement and robustness to support their choice of approach. The experimental results demonstrate the effectiveness of the proposed defense method.

--Strongness:
1. The paper is well organized.
2. They provide extensive analysis of their approach and provide proof.
3. They demonstrate that the proposed method is more robust to other VAE baselines for the attacks.

--Weakness:
1. The qualitative results are not quite convincing. See below question 1.


--Questions:
1. Question on qualitative results:
- For Figure 1, the adversarial examples for the three approaches are largely different. For the baselines -- VAE and $\beta$-TCVAE, the provided inputs look not like just applying a small noise/distortion (which is the setting of adversarial attack) but have a huge difference from the original input. Therefore, the reconstructions can be expected that it won't look similar to the original reconstruction.
- In Figure 6(c), are you using similar adversarial examples / similar amount of distortions to generate the reconstruction images for $\beta$-TCVAE and your approach?

I'm not an expert in the adversarial attack domain, but shouldn't the adversarial examples be similar across different baselines?

2. Figure 4(b) top-right adversarial example seems not to be distorted (almost the same as the input). Is it the adversarial example derived by applying distortion? Since the distortion is large, I don't know why the adversarial looks just the same as the input.

--Recommendation
In sum, the paper is well organized and consists of extensive theoretical proofs and experiments. While I'm not very convinced by the results. I vote for a more neutral score for now. The authors are couraged to address these issues.
review_title: Paper is well organized, but there are issues in experimental results.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper considers the regularization of latent space toward achieving adversarial robustness against latent space attack. The paper demonstrates the applicability of disentanglement promoting VAEs for achieving adversarial robustness and further enhancing such VAEs by considering their hierarchical counterparts. The paper demonstrates their results in the benchmark datasets considered in the disentanglement and computer vision literature. The overall research direction pursued by this paper is exciting. However, I have some concerns, which include:

1. The paper attempts to establish the connection between disentanglement and robustness. The linkage, however, is not clear. In section 3, the paper argues for the smoothness of the encoder mapping and the decoder mapping. Toward this, the paper postulates additional regularization to enforce "simplicity" or "noiseness". First of all, it is unclear how disentangled latent space helps achieve "simplicity" in the "encode-decode process". Secondly, regarding "noiseness", it is not explained what extra would disentangled version of VAEs (e.g., TCVAE) provide compared to the standard setup. 

2. In section 3.2, the paper empirically demonstrates the connection between disentanglement and adversarial robustness. However, the evaluation carried out are not explicit. Firstly, to demonstrate the connection, the paper uses the attacker's achieved loss \delta (from Eqn 1) as the metric. Although the \delta is shown across different \beta values, it is still unclear if disentanglement is directly related to robustness. Can the authors point out some disentanglement metrics (e.g., MIG) for each beta and compare MIG vs. \delta? Also, the curves are combined for all the d_z. What is the motivation behind doing that? Because it has been known that disentanglement behavior is related to the dimension of latent space. Also, authors could consider decomposing the first term of \delta for all the latent space dimensions and analyze if the disentangled dimensions are robust compared to the entangled dimension. This could be more helpful to establish the linkage. Secondly, authors have picked TCVAE considering "reconstruction quality" compared to \beta-VAE, but in Fig 2 (right), ELBO is compared. Can the authors compare the reconstruction error? Also, for fig 3, I think it is natural to see the comparison with \beta-VAE. Why is such a comparison not included? 

3. In section 4, for the motivation for the hierarchical TC-penalised VAEs, the paper states that "TC-penalisation in single layer VAEs comes at the expense of model reconstruction quality". However, this directly contradicts the use of TC-VAE in the previous section. Although the results presented afterward support the authors' statement, the motivation must be clear and well written. The same comments for section 3.2 apply here too. 

4. The experimental results demonstrating protection against downstream tasks is performed using a simple 2-lear MLPs. This is different from the regular CNN network commonly considered for these datasets. Although this was meant to demonstrate the proposed model's efficacy, it would be more clear if the experimental setup is consistent with the current literature setting. Also, can the authors point out the initial results for the models before the attack? 


Minor comments:

- There are a lot of grammatical errors and hard-to-follow sentences. Some examples:
    - ".. are not only even more .."
    - ".. attack the models using methods outlined ..". But Eq (1) refers to only one method, right?
    - "… then \delta too is small .." 

(Update): The score has been updated after a rebuttal from the authors. 
review_title: Hierarchical disentangled VAE for adversarial robustness. 

final_score: rating: 7: Good paper, accept
review_content: Summary: This work builds on the vulnerability of VAEs to adversarial attacks to propose investigate how training with alternative losses may alleviate this problem, with a specific focus on disentanglement. In particular it is found that disentanglement constraints may improve the robustness to adversarial attacks, to the detriment of the performance. In order to get the best of both, the author(s) propose a more flexible (hierarchical) model, trained with the beta-TC penalization on the ELBO. The algorithm, named Seatbelt-VAE, shows improvement over the beta-TC VAE in terms of reconstruction, as well as in term of adversarial robustness for several datasets (Chairs, 3D Faces, dSprites). 

Comments: 
1. The paper is well-written and the underlying reasoning is easy to follow.
2. The experiments are sound and well documented (results are reported across latent space dimensions, and adversarial attack parameters)

Questions:
1. I am wondering how the bias of estimating the TC term on Z^L in Eq (8) scales with L and the minibatch size, compared to the more simple TC estimator from Chen et al. (2018) and if the author(s) had any evidence from the experiments that it might be problematic? Does the algorithm require even larger batch sizes?
2. Should this approach be compared as well to weight decay or other simple regularization on the weights?

Minor questions:
3. I wish the paper would make a stronger connection between disentanglement and robustness. The beta-TC VAE is only one choice among other possible to constrain the variational network. Did the authors ever try anything else? 
4. Is it possible that the TC-VAE is effective at providing a defense against adversarial attacks in this manuscript because of the nature of the attack used in this manuscript (Eq. (1))? If the attack was not based on the Kullback-Leibler divergence, but based on another geometry, maybe another disentanglement constraints would be more performant?
review_title: Official review #2

Meta-review: This paper presents a hierarchical version of β-TCVAE that promotes disentanglement in the latent space and improves the robustness of VAEs over adversarial attacks, without (much) degeneration on the quality of reconstructions. The analysis on the relationship between disentanglement and adversarial robustness is valuable and the method is new. The results look promising. The comments were properly addressed.
Decision: Accept (Poster)