Disentangling Improves VAEs' Robustness to Adversarial Attacks
Authors: Matthew Willetts
Alexander Camuto
Stephen Roberts
Chris Holmes
Venue:  (2020)

Abstract: This paper is concerned with the robustness of VAEs to adversarial attacks. We highlight that conventional VAEs are brittle under attack but that methods recently introduced for disentanglement such as β-TCVAE (Chen et al., 2018) improve robustness, as demonstrated through a variety of previously proposed adversarial attacks (Tabacof et al. (2016); Gondim-Ribeiro et al. (2018); Kos et al.(2018)). This motivated us to develop Seatbelt-VAE, a new hierarchical disentangled VAE that is designed to be significantly more robust to adversarial attacks than existing approaches, while retaining high quality reconstructions.

Under review as a conference paper at ICLR 2020
DISENTANGLING IMPROVES VAEs* ROBUSTNESS TO ADVERSARIAL ATTACKS
Anonymous authors Paper under double-blind review
ABSTRACT
This paper is concerned with the robustness of VAES to adversarial attacks. We highlight that conventional VAEs are brittle under attack but that methods recently introduced for disentanglement such as 3-TCVAE (Chen et al., 2018) improve robustness, demonstrated through variety of previously proposed adversarial attacks (Tabacof et al. (2016); Bondii--Rbbiir et al. (2018); Kos et al.(2018) 1)). This motivated to develop Seatbelt- Seatbelt-VAE, VAE, new hierarchical disentangled VAE that is designed to be significantly more robust to adversarial attacks than existing approaches, while retaining high quality reconstructions.
INTRODUCTION
Unsupervised learning of disentangled latent variables in generative models remains an open research problem, as is an exact mathematical definition of disentangling (Higgins al. 2018). Intuitively, disentangled generative model has one-to-one correspondence between each input dimension of the generator and some interpretable aspect of the data generated.
For VAF-derived models (Kingma & Welling 2013 Rezende et al. 2014) this is often based around rewarding independence between latent variables. Factor VAE (Kim & Mnih (018), 3-TCVAE (Chen et al., 2018) and HFVAE (Esmaeili et al.) (019) have shown that the evidence lower bound can be decomposed to obtain a term capturing the degree independence between latent variables of the model, the total correlation. By up-eeighting this term, can obtain better disentangled representations under various metrics compared to 3-VAEs (Higgins et al. 2017a).
Disentangled representations, much like PCA factor analysis, not only human- camman-interrretaai interpretable but also offer more informative and robust latent space representations. In addition, information theoretic interpretations of deep learning show that having disentangled hidden layer within discriminative deep learning model increases robustness to adversarial attack (Alemi et al. (017).
Adversarial attacks on deep generative models, more difficult than those on discriminative models (Tabacof et al.) 2016 Gondim-Ribeiro Ribeiro et al. 2018 Kos et al.] 2018). attempt to fool model into reconstructing chosen target image by adding distortions the original input image. Generally, the most effective attack mode involves making the satent-space representation of the distorted input match that of the target image (Gondim-Ribeir particularly relevant to applications where the encoder's s output is used downstream.
al. [0018] Kos al. 2018). This kind of attack is
Projections of data from VAEs, disentangled not, are used for tasks such as: text classification (Xu et al. 2017); discrete optimisation (Kusner et al.) (2017); image compression (Theis et al.) 2017] Townsend et al. 2019); and as the perceptual part of a reinforcement learning algorithm (Ha & Schmidhuber 2018 [Hggins et al. (0177)) the latter of which uses disentangled VAE's encoder to improve the robustness of the agent to domain shift.
Here we demonstrate that 3-TCVAEs are significantly more robust 'latent-space' attack than standard VAEs, and are generally more robust to attacks that act to maximise the evidence lower bound for the adversarial input. The robustness of these disentangled models is highly relevant because of the use-cases for VAEs highlighted above.
However, imposing additional disentangling constraints quality of resulting drawn or reconstructed images (Higgins et al. 2017a| Chen et al. 2018). We sought whether more powerful, expressive models, can help ameliorate this and in doing so built
VAE training objective degrades the

---

Under review as a conference paper at ICLR 2020
Orgina
Orginal
Advananta
Origiall
Orginal
Advemata
Orgiaa
orginamnc
Adwedd
Orgita
Orgitainc
Adveraaaa
Vanilla VAE
(b) 3-TCVAE
(c) 3-TCDLGM, L Seatbelt-VAre
Tugge:
Detuttion
Torper
Tarye
Adverunt
Togge
(e) Vanilla VAE
(1) B-TCVAE
(g) 3-TCDLGM, L =
(h) Seatbelt-Var VAE, L
Ccto
Acvemartaa
brighaa
Original
Adu martt
Orteinal
Acvemanta
Larpo:
Adveranalres
1g5
Dalation
Tarpss
Alveranal
Vanilla VAE
(i) 8-TCVAE
(k) 8-TCDLGM,
(I) Seatbelt-VAE.
Latent-space adversarial attacks Chairs, 3D Faces and CelebA for different models, including our proposed Seathelt- VAE. = 10 for 3-TCVAE, 3-TCDLGM and Seatbelt- -AE. L is the number of stochastic layers. Clockwise within each plot show the initial input, its reconstruction, the adversarial input, the adversarial distortion added to make it (shown normalised), the adversarial al.
Figure 1:
input's reconstruction, and the target image. Following Tabacof al. (2016); (Gondim-Ribein (2018) we attack with different degrees of penalisation on the magnitude of the adversarial distortion; choosing the distortion to show, we pick the one with the penalisation that resulted in the value of the attack objective just better than the mean. See Section 5 for more details.
hierarchical disentangled VAE, Seatbelt- VAF, drawing works like Ladder VAEs (Sønderby ct al. 2016) and BIVA (Maalee et al. 2019). We demonstrate that Seatbelt- VAEs are more robust to adversarial attacks than B-TCVAES and 3-TCDLGMs (the latter a simple generalisation make of 3-TC penalisation to hierarchical VAEs). See Figure for a demonstration.
Rather than being concerned with human- interprettte controlled generation by our models, which has been the focus of much research into disentangling, instead we are interested in the robustness afforded by disentangled representations.
Thus our key contributions are:
A demonstration that 3-TCVAES are significantly more robust to adversarial attacks via
their latents than vanilla VAEs.
The introduction of Seatbelt- VAE, hierarchical version of the 3-TCVAE, designed to further increase robustness to various types of adversarial attack, while also giving better perceptual quality of reconstructions even when regularised.
VARIATIONAL AUTOENCODERS
Variational autoencoders (VAEs) are a deep extension of factor analysis suitable for data like images (Kingma & Welling 2013 Rezende al. 2014). They have joint distribution
2

---

Under review as a conference paper at ICLR 2020
po(x)2)p(2) where p(z) (0, and pe (2]22) is an appropriate distribution given the form of the data, the parameters of which are represented by deep nets with parameters 0. As exact inference is intractable for this model, in VAE we perform amortised stochastic variational inference. By introducing an approximate posterior distribution (s(2|=) (pp Eg (z)),,
over data and latent variables po(x, 2)
perform gradient ascent the evidence lower bound (ELBO) 2 log p(z) w.r.t.both 0 and jointly, using the reparameterisation trick take gradients through Monte Carlo
[(x) = - -K (1)()))) ( (P)((, z)) = (=lar) log pa( (x] DxL DKL
samples from ( (2)x).
2.1 DISENTANGLING VAES
In B-VAE (Higgins et al. 201 a free parameter / multiplies the DEL term in ((x: above. This objective [z( remains lower bound on the evidence.
Decompositions ((x) shed light on its meaning. As shown in Hoffman Johnson (2016);
Makhzani et al. (2016); Kim & Mnih (2018); [Cen et al. (2018); Esmaeili et (2019), one can define the evidence lower bound not per data-point, but instead write over a dataset of size N, have [(0, 6, D).
D (z") },
Esmaeili et al. (2019) gives a decomposition of this dataset-level evidence lower bound:
C(0, Φ. D) DKL x))po, =))
(1)
log Po (a))
log (2(2(x) Lo (=)
= (x,a)
DKL (g(r)))op - Dxr (=) (=)l(())
where under the assumption that p(=) factorises further decompose i
(x(2)
(kL(qo())llp(2) = -g)()) logg
+
Σ DKL DKL (lo()))))))
(3)
II 4s (2,)
where j indexes over coordinates in z. (o x) = go (x(2)))((x) and ((x) = 1 EN' 8(x - x")
is the empirical data distribution. (2) v Dim Us (2(xx=)) is called the average encoding
distribution following Hoffman & Johnson 2016).
A is the total correlation (TC) for q2(2), generalisation of mutual information to multiple variables (Watanabe 1960). With this mean-field p(2), Factor and 3-TCVAEs upweight this term, so we have an objective:
L CTC D) =
Chen et al. 2018) gives a differentiable, stochastic approximation to (zy(() log (=)),, rendering this decomposition simple use training objective using stochastic gradient descent. We also note that ()) the total correlation, is also the objective in Independent Component Analysis (Bell & Sejnowski 1995 Roberts Everson, 200T.
2.2 HIERARCHICAL VAES
We now have set of L layers of variables: z= 22,,22,... 2 The evidence lower bound for models of this form is:
C CLGM (0, d. D) (z. log po(x,2) (e, (og 2 (z(2. x)
(5)
The simplest VAE with a hierarchy of conditional stochastic variables in the generative model the Deep Latent Gaussian Model (DLGM) of Rezende et al. (2014). The forward model factorises as chain:
T-
po(x, 2) = 2=(x po 22
(6)
3

---

Under review as a conference paper at ICLR 2020
Each (=* is Gaussian distribution with mean and variance parameterised by deep nets. p(=") is a unit isotropic Gaussian.
We can understand this additional expressive power as coming from having richer family of distributions for the likelihood over data æ marginalising out all intermediate layers: po(x = M dz (z z) is non- -ausssas,,n, highly flexible, distribution.
To perform amortised variational inference one introduces a recognition network, which can be any directed acyclic graph where each node, each distribution over each z², is Gaussian conditioned on its parents. This could be chain, as in Rezende al. (2014):
L-1 II (2")0"1)""
g0
Again, marginalising out intermediate z layers, we see ga(=410) = I dz {o(2|x) is non-Gaussian, highly flexible, distribution.
However, training DLGMs is challenging: the latent variables furthest from the data can fail to learn anything informative (Sønderby et al. 2016 Zhao al. 2017)) Due the factorisation of go (z)z) and (x, z) in DL.MM, is possible for single-layer VA train in isolation within a hierarchical model: each Po (2 (=) distribution can become fixed distribution not depending on 2++1 such that each DKL divergence present in the objective between corresponding 2² layers can still be driven to local minima. Zhao et al. 2017 gives proof of this separation for the case where the model perfectly trained, i.c. DKL. go
(I, 2)) 0.
This is the hierarchical version the collapse of units in a single-layer VAE (Burda al. 2016). but now the collapse is entire layers of. It is part of the motivation for the Ladder VAE (Sonderby
ct al. 2016) and BIVA (Maaløe et al. 2019).
3 SEATBELT- VEAT: HIERARCHICAL 3-TCVAE WITH SKIP CONNECTIONS
We propose novel hierarchical disentangled VAEs where we aim to disentangle only in the top-most latent variables 24, Following the Factor and 3-TCVAEs we upweight the term of the form of A for z Empirically we find models of this type are unable to converge when disentangling at the bottom most layer, or when disentangling at each layer. Intuitively, we want to capture high-level disentangled information at the top, but leave lower layers free to learn rich entangled representations. If pe (z|z) = Po(x) 2 we obtain the generalisation of B-TC penalisation to DLGM and call it 3-TCDLGM. suffers from the problems of collapse described above.
(a) Generative Model
(b) Approx. Posterior
Inspired by BIVA (Maalee et al. 2019), we choose instead to condition our likelihood all z¹ layers:
Figure 2: = MaabelltVAAE Shaded lines indicate 3-TC fac- torisation given node.
L-1
(o(2, z) Po (z(z))
II po(2*) (2 (p(
(8)
Combining Eqs 7 and applying 8-TC penalisation the DKL. term over zh:
4

---

Under review as a conference paper at ICLR 2020
CB (0, D)
(2)p2)))) (z,,)) log
Bg((s) ) log g(x) 2y(x, (Ki (as (2=")==))))) (=))
1--1
Eg (= ( (ga( 1=" 11110(1" (2 12=
m=2
DRI.(uo(=* (laoa( - 3KI (=00(=))))))
II {o(x)))
j=1
Σ Decllo(f5)lll(
(9)
(x,,)) log po (x/z)
(10)
where j is indexing over the coordinates in z²_ See Appendix for the derivation. We call this model Seatbelt-VAL as with the extra conditional dependencies and nodes we increase the safety of our model to adversarial attacks, to noise, and to decreases in perceptual quality 3 increases. We find that using free-bits regularisation (Kingma et al. 2016) greatly ameliorates the optimisation challenges associated with DLGMs. For L this reduces to a 3-TCCVA,, and for L > 1, = 1 it produces a DLGM with our augmented likelihood function.
For completeness, note that for 3-TCDLGMM
@ (0, D, = E (x)(xx)) logpo(x)2")
(1I)
3.1 MINIBATCH TRAINING
VABs and derived models are commonly trained using stochastic gradient ascent on the ELBO, on minibatches of the training data. With the ELBO in Eq (9), this would be challenging because of the presence of average encoding distributions, which depend on the entire dataset.
To avoid having to handle large mixture distributions in our objective functions, we derive minibatch estimators that are simple generalisation to disentangled hierarchical VAEs of the Minibatch Weighted Sampling estimator proposed in Chen et al. 2018) in the context of 8-TCVAES. See Appendix for further details.
4 ROBUSTNESS OF VAES TO ADVERSARIAL ATTACKS
Most adversarial attack research has focused discriminative models (Akhtar & Mian 2018 Gilmer et al. 2018) and recently VAEs have found use in protecting discriminative models against attack (Schott et al. 2019 Ghosh al. (2019). Currently, two adversarial modes have been proposed for attacking VAES (Tabacof et al. 2016:Gonndmm
et al. 2018 Kos et al. 2018). In both attack close to a target image x², when given
modes the adversary wants draws from the model distorted image x = z + d as input. When attacking discriminative model the aim is to manipulate comparatively low-dimensional output layer of the network, commonly aiming with the attack to diminish or increase only handful of the output units. However, for a generative model, the attacker is aiming to change large number of pixel values in the output, changing the content the reconstruction. Intuitively this is harder task, and the attacks proposed in the above papers do not always result in adversarial examples that are very close to the initial image in appearance.
The first mode of attack, which call the output attack, aims to reward draws from the decoder conditioned on z ~ (2(22) that are close to x² via the ELBO.
For vanilla VAE, this attack's 's adversarial objective is:
Aoutput (z. d X) (x log (0o [2)] DKL(go(2) d)llp(>)) + Ibll|
(12)
The second mode of attack, the latent attack, aims to find x + d such that go (2(x=)) (2(2))))
under some similarity measure r(;,), :), which implicitly means that the likelihood po (o<x122) [z) is high when conditioned draws from the posterior of the adversarial example. This attack is important if
5

---

Under review as a conference paper at ICLR 2020
one is concerned with using the encoder network of VAE part of downstream task. For single stochastic layer VAE, the latent-space adversarial objective is:
llatent (I, d, x'; X) = r r(gs + d), (=(x)=)) + |||l|K
(13)
Note that both modes of attack penalise the L2 norm of d, prioritising smaller distortions. We denote samples from qo(2 (2| + d) as 2.
For Tabacof al. 2016); Gondim-Riberie et al. (2018) r(::-) is DKL (go(a + (z|z)) and for Kos et al. (2018) it IS the Lg distance || 2 12 ~ go(2|æ x + d), 2- ~ (x(2))) between draws [)m))(2) [a + d)ll2 between their means. We follow the former papers and use the DKL formulation. All three papers find that the latent attack mode is as or more effective than the output attack for single layer VAEs both under perceptual evaluation and various proposed metrics Tabacof et al. 2016; Gondim-RRbeed et al. 2018 Kos et al. 2018).
from the corresponding posteriors
For latent attacks, the choice of which layers attack depends on model architecture. For DLGMS and B- 3-TCDLGME the attacker only needs to match at the bottom latent layer as po (z]z) Pe (z(22)), see Eq See Appendix for plots showing how effective this attack regardless of / and
Even though the decoder is conditioned on all latent layers, one could choose to attack individual layers for Seatbelt- VAE. For example, one could attack just the first layer 21. If one were able to find perfect latent-space attack in z²_ DKL( (2 [x: + d) d))lq>(='<x")) = O, then the variational posteriors in higher layers would also be well matched. Attacks that do not perfectly match the target 2² may have their mismatch with the target posterior amplified in higher layers. In Seatbelt-VAR the likelihood over data is conditioned on all z layers, being off off-target these higher layers matters, In the Appendix we show that targeting the top or base layers individually is not as effective attacking all layers. Hence:
latent (xr, d, x²; A) = =DKI do) (x')) + Alld|
(14)
L
A latent (I, d, x1;
Σ DKL(go(z'1z +c), + Iblla
(15)
5 EXPERIMENTS
Here we perform four tranches of experiments. Firstly, we demonstrate that the reconstructions given by Seatbelt- VAEs (and 3-TCDLGMs) degrade much less strongly as B is increased than in 3-TCVAFS. Secondly, we perform variety of adversarial attacks on all models. We demonstrate that increasing 3 makes 3-TCVAFs more robust to adversarial attacks than vanilla VAEs, and that Scatbelt-VAEs are more robust still. Thirdly, show that these disentangled models are most robust than vanilla VAES to unstructured noise distorting their inputs, with Seatbelt- VABs again the most robust. Finally, we study the effect of disentangling on the sparsity of model weights.
We perform these experiments on Chairs (Aubry et al., 2014), 3D faces (Paysan et al. 2009), and CelebA (Liu et al. 2015). Additional results for dSprites (Higgins et al. 201 7a) can be found in the Appendix. We used the same encoder and decoder architectures as Chen et al. (2018) for each dataset. For the details of neural network architectures and training, see Appendix and accompanying code. To show the degree to which our models are disentangling, the Appendix also contains the Mutual Information Gap (MIG) (Chen et al. 2018 at the top layer of each model. Though our models obtain high MIG at = this does not imply that decoding from latent traversals in z" will result in the generation of images with human-ine inmmannnterrrettbble factors of variation. This is made abundantly clear in the latent space traversal plots, also shown in the Appendix. As such, we do not believe existing disentangling metrics directly apply to hierarchical models.
5.1 ELBO AND RECONSTRUCTION QUALITY: B- TCVAES TO SEATBELT- VAES
We trained 3-TCVAES. 3-TCLLGMs, and Seatbelt VAEs for range of 3 penalisations. In Figure 3 we plot the final ELBO of our trained models, but calculated without the additional 3 penalisation that was applied during training. The ELBO for 3-TCVAE [Eg 4 declines with 3 much more quickly than Seatbelt VAEs [Eq [10)] or 3-TCDLGMs [Eq [TT)]. In the Appendix we also show that increasing reduces DRL. collapse. This is interesting, it shows that we can increase the

---

Under review as a conference paper at ICLR 2020
penalisation for Seatbelt- VAEs, without large degradation in the quality of the model as measured by the ELBO.
In Figure 4 we the effect of depth and disentangling reconstructions of CelebA. The bottom row, showing the reconstructions from a Seatbelt- VAE with L = and 20 clearly maintains facial identity better than those from B-TCVAE in the middle row. The effect is clearest for the 3rd 4th and 7th columns, where many of the individuals' finer facial features are lost by the 3-TCVAE but maintained by the Seat This fits with the results Figure and shows that resistance the quality of the reconstructions of Seatbelt to increasing 3 is visually perceptible as well as measurable.
-400
-1500
-1600
600
-1800
B-TC
P-TC
-800
BTCLLGM
B-TODLGM
1900
(a) Chairs ELBO
(b) 3D Faces ELBO
Figure 3: Plots showing the effect of varying 3 under various datasets the ELBO of 3-TCVAES, 9-TCDLGMs and Seatbelt-VAIs [Eqs (4), (II) and (IO) respectively]. Shading corresponds to the 95% CI over variation due variation of || and L.
Figure 4: Top row shows CelebA input data. Below are reconstructions from 3-TCVAE, 20 and then Seatbelt VAE, L = 4, 3 20.
Labont
162
Ourput
Latort
103
Output
7.5
10
Adversafal Loga
102
7.0
Adversafial Losa
10"
6.5
1.60
10"
10
6.0
1.55
16
(a) Chairs Losses
(b) 3D Faces Losses
Figure 5: A Iltent/outputt for (a) Chairs (b) 31D Faces, for 3-TCVAE for different 33 values. Shading corresponds to the 95% CI over variation due to our stable of images and our values of ||||| and X.
5.2 ADVERSARIAL ATTACK
We apply attacks minimising each of Aoutput and Atateut vanilla VAEs, 3-TCVAEs, --- TCDLGMs and Seatbelt- VAEs; trained on: Chairs (Aubry et al. (014)), 3D faces (Paysan et al. 2009) and CelebA (Liu et al. 2015); for range of (3 T. and À values.
7

---

Under review as a conference paper at ICLR 2020
We randomly sampled 10 input-target pairs for each dataset. As in Tabacof et al. (2016); Gondim Ribeiro et al. (2118), for each pair of images used takes 50 geometrically- distrribbedd values from 2-20 20 to 2 T Thus each model undergoes 500 attacks for each attack mode. Like Tabacof et al. (2016);
Gonnii--ibbiiro et al. (018), we used L-BBGGSBB for gradient descent (Byrd et al. 1995).
We prefer to avoid classifier based metrics (Kos et al. 2018) as in general we think that such analysis can be hard to interpret given the many available choices of classifier. Instead, we evaluate the effectiveness of adversarial attacks from the values reached by - -log po by the attack objectíves (Aoueput: Alatent) and by visually appraising the adversarial input (x + d) and the adversarial reconstruction. Note that higher values of log Pog Doupput: lattent indicate less effective attacks.
Figure T shows latent space attacks and demonstrates that they are less effective on disentangled models. As in Chondim-Ribeire et al. (2018), we are showing the attack for the À that gives us an attack objective just better than the average objective over all attacks tried. Note that for Seatbelt- VAEs, for high values of 3 and L latent attacks often result in the outputs from adversarial attack resembling the original inputs. See Appendix for more examples of the attacks for [Aiatent: Aoutput. } for the models trained on dSprites (a toy dataset for disentangling), Chairs. 3D Faces and CelebA; each over a range of values for 3, L. and X. Note that we rarely observe perceptually effective output attacks regardless of model or settings, though vanilla VAEs are the most susceptible.
One might expect that adversarial attacks targeting a single factor of the data would be casier for the attacker. However, we find that disentangled models protect effectively against these attacks well. See the Appendix for plots showing an attacker attempting to rotate a dSprites heart.
Figure Saaute/ssuasn quantitaaivvl shows that 3-TCVAEs become harder to attack 3 increases. The values of Antent for 3-TCVAES are R 103 times higher than for a standard VAE on Chairs, and still greater than factor of 10 for 31) faces. Doutput attack is also less effective, by smaller factor 1.2.
Figure6]sshows - log po Siatentt/ for 3-TCDLGMs and Seatbelt- varying L and Larger values of these metries correspond to less successful adversarial attacks. Generally, 3-TCDLGMs are very sensitive to latent attack, as we expect. Like 3-TCVAES, Seatbelt- VAEs offer significant protection latent attacks, and somewhat increased protection to output attacks compared to vanilla VAEs. For Seatbelt- VAEs, as we go to the largest values of /3 and L for both Chairs and 3D Faces, Aattent. grows by factor of z 10".
and Figure 7 shows | over a range of datasets
The bottom rows of Figures 6 & (c) (d) have L = 1, and thus correspond to 8-TCVAES. They contain relatively low values of the adversarial objectives compared to L > Similarly the first column, corresponding to 3=11 models, contains relatively low values. These results tell us that depth and disentangling together offer the most effective protection from the adversarial attacks studied.
In the Appendix we also calculate the L2 distance between target images and adversarial outputs and show that the loss of effectiveness of adversarial attacks is not due to the degradation of reconstruction quality from increasing B. By these metrics too Seatbelt- Beatbelt-VAEs outperform other models.
5.3 ROBUSTNESS TO NOISE
In addition to studying the robustness of these models to highly structured distortion, we can also consider robustness to random noise. We add ~ (0, I) the datasets, which are scaled to -1 S x < 1, and then evaluate [ys(=|=-e) Pe (z]==)) where = corresponds to the encoder embedding of x + and x is the original (non-noisy) data. See Figure g for smoothed histogram plots of this for different models for different degrees of B. Both 3-TC and Seatbelt-VAL are effectively denoising autoencoders. They become more robust to noise with increasing 3, while 3-TCDI get worse. See Appendix for plots showing the robustness of these models to smaller magnitude noise.
Some of the robustness of disentangled models to adversarial attacks may be conferred by their robustness to random perturbations of their inputs.
5.4 TOTAL CORRELATION PENALISATION AS REGULARISATION
In the auto-encoder view of these models, the DKL. terms in L(0, Φ, D) are associated with form of regularisation of the model (Doersch) 2016). Recent work shows that for linear autoencoders,
8

---

Under review as a conference paper at ICLR 2020
attent
Output
Latent
110
Output
1.80
1.80
1.65
1.85
0.6
1.50
1.35
1.35
0.2
2
10
1,20
10
1.20
2
10
0.0
10
0.0
B
P
(a) 3D Faces
(b) Chairs
Latent
x10
Output
x10
Latent
x10
1.0
Oulput
x10
1.0
1.95
1.95
0.8
0.8
1.80
1.80
1.65
1.85
0.6
0.8
0.4
04
1.35
0.2
0.2
1,20
1.20
0.0
0.0
2
4
10
2
4
6
8
10
10
(c) 3D Faces
(d) Chairs
Figure 6: - log po (x*1=) for (a) (b) 3-TCDLGMs and (c) (d) Seatbelt-VAEs for Chairs and 3D Faces; over and L (total number of stochastic layers) values and for latent and outpur attacks. Larger values of log pa (z* ( correspond to less successful adversarial attacks.
Latent
Ted
Output
10
Letent
1e5
Oulput
x010
10
1.6
4.0
6,8
1.6
6,5
0
1.6
6.3
0.8
1.6
04
1.5
0.4
0.8
6.0
1.5
10
(a) 3D Faces
(b) Chairs
Lalent
Outpul
Latent
Oulpul
x10
7.5
1.0
6.5
1.6
4.5
0.6
1.5
0.4
6.2
2
N
1.5
1.5
0.2
6.0
(c) 3D Faces
(d) Chairs
Figure 7: (Alatent Aoutput } for (a) (b) 3- 3-TCDLGMs and (c) (d) Seatbelt- VAEs for Chairs and 3D Faces; under varying 3 and (total number of stochastic layers) values.

---

Under review as a conference paper at ICLR 2020
0000
2.003
5,000
0004
L,05
RIVES
03005
5.00
-000
2p3022 3-2
200
-200 243
0pte218-33
-008
opteal p-2
2000
-200
N914/18-15
200
3,000
-206
0pte//pp1
2000
80000
-2000
19000
(a) 3-TCVAE
(b) 8-TCDLGM
(c) Seatbelt-VAE
Figure 8: Robustness of (o) to Gaussian noise (N, 1) scaled by different magnitudes and added to on CelebA; for B-TCVAE, 3-TCDLGM, Seatbelt- VAE: B 0, 10 Best viewed digitally.
Table 1: Relative change of the L2 of Encoders and Decoders by dataset for 3-TCVAE and Seatbelt-
VAE (L- 4) when increasing 3 from 10.
Chairs
3D Faces
CelebA
3:1 + 10
B:1 + 10
8: +
10
B-TCVAE
+5.0%
+19.5%
+73.7%
Encoder
MaatelltVAAE,
=
+1.0%
+2.7%
+40.2%
B-TCVAE
-1944%
-15.0%
-6.8%
Decoder
Seatbelt-VAE,
-7.6%
-6.0%
-11.4%
Le regularisation of the weights corresponds to orthogonality of the latent projections (Kunin et al. 2019). For deep models we expect that disentangling associated with regularised decoders and more complex encoders. The decoder receives simpler representation, but building this representation requires more calculation. Here we measure the L2 norm of the weights of our networks as a function of B, shown in Table 1. See Appendix for results for 3-TCDLGM.
As we increase for 3-TCVAEs and Scatbelt- VAEs for Chairs, 3D Faces, and CelebA the L2 norm increases for the encoder and decreases for the decoder. A more complex encoder is more difficult to match the latent space and regularised decoders may be contributing to the denoising properties seen in Figure 8 That the changes generally greater for 3-TCVAE than Seatbelt-VAr makes sense, as the encoder and decoder of the former interact directly with the disentangled representation. For the latter the decoder receives inputs from all 21, of varying degrees of disentanglement.
CONCLUSION
We have presented the increases in robustness adversarial attack afforded by 3-TCVAEs. This increase in robustness is strongest for attacks via the latent space. While disentangled models are often motivated by their ability to provide interpretable conditional generation, many use cases for VAEs centre on the learnt latent representation of data. Given the use of these representations as inputs for other tasks, the latent attack mode is the most important protect against.
Recent work by Shamir et al. (2019) gives constructive proof for the existence of adversarial inputs for deep neural network classifiers with small Hamming distances. The proof holds with deterministic defence procedures that work additional deterministic layers of the networks, and in the presence
of adversarial training (Szegedy et al. 2014; Ganin et al., 2016 Tramèr et al. [0088) 8 Shaham et al.) 2018). Shamir et al. (2019) thus give theoretical grounding for using stochastic methods to defend against adversarial inputs. As VAES are already used to defend deep net classifiers (Schott et al.
2019
Ghosh et al. 2019), more robust VAEs, like 3-TCVAES, could find use in this area.
We introduce Seatbelt- VAE, a particular hierarchical VAE disentangled on the top-most layer with skip connections down the decoder. This model further increases robustness adversarial attacks, while also increasing the quality of reconstructions. The performance of our model under adversarial attack to robustness is mirrored in robustness to uncorrelated noise: these models are effective denoising autoencoders as well. We hope this work stimulates further interest in defending and attacking VAEs.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Naveed Akhtar and Ajmal Mian. Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey.
IEEE Access, 6:14410-14430, 2018. ISSN 21693536.
Alexander A Alemi. lan Fischer, Joshua V Dillon, and Kevin Murphy. Deep Variational Information Bottleneck. In ICLR, 2017. ISBN 1612.00410v5. URL httt:/ss//aaxx...orrr///totee
00410. P0410.ppdftttp://aaaaa httpaaawwww.rr.g.aaaaoooe
00410
Mathieu Aubry, Daniel Maturana, Alexei A Efros, Bryan Russell, and Josef Sivic. Seeing 3D chairs: Exemplar part-based 21)-3I alignment using a large dataset of CAD models. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3762-3769, 2014, ISBN 9781479951178. 1of10..109//VPRR200114 2014444 URL pistps://wiee di ens Er
pdf
Anthony J Bell and Terrence J Sejnowski. An information-maxxiiistion approach to blind separation and blind deconvolution. Neural Computation, 71000000o00000700000000000000030 1995.. URL http::/ww:: inf
2016. http://.....o////o//////////ee/a
Fu fu-berling deeee/deeleeeee
pddf
Yuri Burda, Roger Grosse, and Ruslan Salakhutdiino Iww.aiiiiiwnmmmmmmmmottotaoo Importance Weighted Autoencoders, In ICLR,
Richard H Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. A Limited Memory Algorithm for Bound Constrained Optimization, SIAM J. Sci. Comput., 16(5); 1208, 1995. ISSN 1064-8275. doi 10. 10.1137/091606) URL //xp. doi/ org 10.
Ricky T Q Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating Sources of Disentan- glement in Variational Autoencoders. In NeurIPS, 2018. URL
1802. 04942.9422.0 21000.099...cctttp////aaaarroa..
04942
Carl Doersch. Tutorial on Variational Autoencoders. Technical report, Carnegie Mellon University, 05908 http999..dd..ttpp//a/vv..oo.. http:://wha.....o/////////////ap
2016. Ut..tttpss//aaxt..orr//// 1606. 05908
Babak Esmaeili, Hao Wu, Sarthak Jain, Alican Bozkurt, N Siddharth, Brooks Paige, Dana H Brooks, Jennifer Dy, and Jan- anWWillem van de Meent. Structured Disentangled Representations, AISTATS. 2019. htt aawiv.iiivvv..... httgg/df/1844 02086. 1804.02080
/bs/
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, Victor Lempitsky, Urun Dogan, Marius Kloft, Francesco Orabona, and Tatiana Tommasi. Domain-Adversarial Training of Neural Networks. Journal of Machine Learning Research, 17:1-35, 2016. (tttsttps: /ttp:////aavvv... org/ http:/df///// 110.. 07818 pdf
Partha Ghosh, Arpan Losalka, and Michael Black, Resisting Adversarial Attacks Using Gaussian Mixture Variational Autoencoders. In AAAI, 2019. URL aaai org
Justin Gilmer, Ryan P Adams, Ian Goodfellow, David Andersen, and George E Dahl. Motivating the Rules of the Game for Adversarial Example Research. CoRR, 2018. URL http:::ttps.............tt pesttss//arxe //arxiv org/pdE/1807.c 06732 www.ttp:////ppmc http://00000......pttpp..//p:: http://ppp//aa.....g//aa///////a
06732
George Gondim-RRbeiro, Pedro Tabacof, and Eduardo Valle. Adversarial Attacks on Variational Autoencoders. CoRR, 2018. /ttpa///../aaaa//////o//////////
David Ha and Jürgen Schmidhuber. World Models. In NewrlPS, 2018 2018. doi 1t..2000..20ooo//oo 120763
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick,
Shakir Mohamed, and Alexander Lerchner. 3-VAE: Learning Basic Visual Concepts with
Constrained Variational Framework. In ICRL, 2017a. 1770100017771777807700032201 URL
http:///ournn http://ourrnalsssspppuu.....
www.sug.bbcc... com wttppppp...ccoooooo///.00/000000

---

Under review as a conference paper at ICLR 2020
Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. DARLA: Improving Zero-Shot Transfer in Reinforcement Learning. In ICML, 2017b. URL
pdf
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner Deepmind. Towards Definition of Disentangled Representations. CoRR, 2018. /
02230. pdf]
Matthew D Hoffman and Matthew J Johnson. ELBO surgery: yet another way to carve up the varia- tional evidence lower bound. In NewrIPS, 2016. URL http:t/:/aappoootttttteeeeeeeeee
pdf
Hyunjik Kim and Andriy Mnih. Disentangling by Factorising, In NeurIPS, 2018. URL https:
//ax/iv.rrg//af/1002. 05983. pdf
Diederik P Kingma and Jimmy Lei Ba. Adam: A Method for Stochastic Optimisation. In ICLR, 6980. pdE.
2015. /
Diederik Kingma and Max Welling, Auto-EEncoding Variational Bayes. In NeurIPS, 2013. ISBN
1312.6114v00. db//10..05/0000-66666///11733333 URL http:
/1312
6114
Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improving Variational Inference with Inverse Autoregressive Flow. In NeurIPS, 2016. ISBN 9781611970685.
J Kos, I Fischer, and D Song. Adversarial Examples for Generative Models. In IEEE Security and Privacy Workshops, pp. 36-42, 2018. doi I0. [0.1109//SW.2211.00011
Tejas D Kulkarni, Will Whitney, Pushmeet Kohli, and Joshua B Tenenbaum. Deep Convolutional
Inverse Graphics Network. NeurlPS, 2015. (o01T0..06331449144400 URL httttss//a/xxii 03167
org/pdf/ 1503. 03167 http::/www..cp...a.a.ooooo//////
Daniel Kunin, Jonathan M Bloom, Aleksandrina Goeva, and Cotton Seed. Loss Landscapes of Regularized Linear Autoencoders. In ICML, 2019. URL UR//ttpss///arii..or///df// 081 281668.. .df
Matt Kusner, Brooks Paige, and José Miguel Hemández- Grammar Variational Autoencoder. In ICML, 2017. URL htttp://openee les
wwwpsmmlllssloo..eeeoeeeeeooeeee wttpessrrr//ppppppeoooeee/e/eeee les 2-grammar
html
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep Learning Face Attributes in the Wild. In Proceedings of International Conference on Computer Vision (ICCV), 2015.
Lars Maalóe, Marco Fraccaro, Valentin Liévin, and Ole Winther, BIVA: Very Deep Hierarchy of Latent Variables for Generative Modeling. CoRR, 2019. URL https::/arxxvv.rrgg///
1902. 02100.11222.22 pdf.
Alireza Makhzani, Jonathon Shlens. Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial Autoencoders. In ICLR, 2016. ISBN 4928-4931. doi [0010016fj.eeee22122.0..0 URL https: 7/arxiv. wetxxiv.org/dff//e 111.41111.544...0 pdf
Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani. and Thomas Vetter. A 3D face model for pose and illumination invariant face recognition. In 6th IEEE International Conference on Advanced Video and Signal Based Surveillance, AVSS 2009, pp. 296-301, 2009. ISBN URL htttt:tftfcccesss cs un anibas.cii
9780769537184. doi TO. 1099//VSS.2009.55
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. ICML, 2014. ISBN 9781634393373. 400..4082.0000
12

---

Under review as a conference paper at ICLR 2020
S Roberts and R Everson. Independent Component Analysis: Principles and Practice. Cambridge University Press, 2001. ISBN 9780521792981.
Lukas Schotr, Jonas Rauber, Matthias Bethge, and Wieland Brendel. Toward the First Adversarially Robust Neural Network Model on MNIST. In ICLR, 2019, URL httpps///ar...oor/////
1805. 091099111 19990.ppp. pdf
Uri Shaham, Yutaro Yamada, and Sahand Negahban. Understanding adversarial training: Increasing local stability of supervised models through robust optimization. Neurocomputingg 307:195-204, 100..o0006...uucmm2011133333 pdff / 111./1111..1 05432. pdf
2018. ISSN 18728286.
URL
Adi Shamir, Itay Safran, Eyal Ronen, and Orr Dunkelman. A Simple Explanation for the Existence of Adversarial Examples with Small Hamming Distance. CoRR, 2019. URL gradients.
com/ wwwnshhhhhaaaaaamaaaaaaaaaaaaaaa httphalllooooffuctoeeeee
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Spren Kaae Sønderby, and Ole Winther, Lad- der Variational Autoencoders. In NeurlPS, 2016. URL
02282.pdE
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. CoRR, 2014. Utthttps:: Etq/pdf/1312 6199.19..60.. pdf
Pedro Tabacof, Julia Tavares, and Eduardo Valle. Adversarial Images for Variational Autoencoders.
In NIPS Workshop on Adversarial Training, 2016. BRLLhttss:// http:://arriv.oor//odft..
00155. pwtp:td:ttt://tttttttt /ttp://f.p.e//aaaa..o.oo/a/aa//p w1tp://.wwwr/aa//........ 00155
I Lucas Theis, Wenzhe Shi, Andrew Cunningham&, and Ferenc Huszár. Lossy Image Compression with Compressive Autoencoders. In ICLR, 2017. URL
00395. pdf
James Townsend, Tom Bird, and David Barber. Practical Lossless Compression with Latent Vari- ables using Bits Back Coding. ICLR, 2019. URL
coombbits/.....ooc.......ooe.ott back/
bits- back, http:: org/ abs/1901. 04866
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble Adversarial Training: Attacks and Defenses. In ICLR, 2018. URL. https! 175. 07204
//a //arxiv.orr//ddF/100. 07204, http:///....ffatpp//aaaooooo/o//
Satosi Watanabe. Information Theoretical Analysis of Multivariate Correlation. IBM Journal of Research and Development, 4(1):66-82 82, 1960. ISSN 0018-8644 011.1..114//rd.4..0000
Weidi Xu, Haoze Sun, Chao Deng, and Ying Tan. Variational Autoencoder for Semi-supervised
Text Classification. In AAAI, pp. 3358-3364, 2017. ISBN 978 1784450329569 100511005000004 6361/201527329
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning Hierarchical Features from Generative Models. In ICML, 2017. (tLhhtt:: //tpx//aaavv///a///a/eee/eaaa wrtppdf//7022222222//
htttes://aaii..orr/////////22/0 08396.pdl.
13

---

Under review as conference paper ICLR 2020
SUPPLEMENTARY MATERIAL FOR: DISENTANGLING IMPROVES VAES' ROBUSTNESS TO ADVERSARIAL AT- TACKS
Anonymous authors Paper under double-blind review
CONTENTS
Derivation of ELBO for Seathelt- Fs
Minibatch Weighted Sampling for 2'
B.1 MWS for (=(="); 3-TCVAES
B.2 Minibatch Weighted Sampling for (=(="), 1: 8-TCGLGMs and Seatbelt- VAEs
Implementation Details
Encoder and Decoder Architectures
D La norm of 3-TCDLGM
Robustness to Noise
F Activation of
3-TCVAES
Seatbelt- VAES
12
Aggregate Analysis of Adversarial Attack
15
G.I
3-TCVAE
15
G.2
3-TCDLGMs
16
G.3
Beathelt-VAFz
17
G.4
Seatbelt- VAE layerwise attacks
18
Adversarial Attack Plots
19
dSprites Adversarial Attack on a Single Factor
19
H.2
dSprites Adversarial Attack
20
H.2.1
B-TCVAES
20
H.2.2
B-TCDLGMs
22
H.2.3
Seatbelt- VAES VAESs
24
3
Chairs Adversarial Attack
26
H.3.1
3-TCVARS
26

---

Under review as conference paper ICLR 2020
H.3.2
B-TCDLGMs
28
H.3.3
Seatbelt- VAES
30
H.4
3D Faces Adversarial Attack
32
H.4.1
3-TCVAES
32
H.4.2
B-TCDLGMs
34
H.4.3
Seathelt- VAES
36
H.5
CelebA Adversarial Attack
38
H.5.1
3-TCVAES
38
H.5.2
B-TCDLGMs
38
H.5.3
Seatbelt-VAEs
38
Data Generation from Models
39
Mutual Information Gap
41
2

---

Under review as a conference paper at ICLR 2020
A
DERIVATION OF ELBO FOR SEATBELT- VAES
Start with Eq (5) cf. Eq in the main paper. The likelihood is conditioned on all a layers: (z(z)))
C Φ, D) = (x))))) log po z) log (z, (x. (ogaa)oooggp porl))))))) Bx(
(A.1)
= =g log (x(lz))) By(=) log ((x) + Be(e,s) log [z
(A.2)
((2)2z) xr)
= -Bg(n,)) log po (z/z)) + Hl((()))
(A.3)
+ / dx d2 (2 [d= (=))(()))))))))) log pl=²) ITk= po (
20 (z) IIm go ( |
p(2²)
W
dr ds (dz)) (z())))))))) log
90(24121-1)
k= 10(21124t1)
dæ
[d (zol)) log
(A.4)
4a ( (z) (x)
Ihm 9o (
B (KL(
(A.5)
(dz) (00(2))) log po Po(21127 dx
R
(=(21 (z)
i=1
po m=+1)
Σ
dx II (d22)) (z))))(()) log ao mm
(A.6)
m=
Rs
R₂
R (o a)) DKL. (gsl= (=o
(A.7)
L-1
L-1
po (2 m2 m
R
=
dr II (dz') (1)))(())))) (z)((z)
II Arm I, Jegfor
log qs (2 / 2m
m=2
(A.8)
1-1
L-
Σ
dx II (dz (a (='
II k.ppf
(10(2*+112)))
mm-1)|(pp (=m²) m
m=2
(A.9)
L-1 Σ
Ey
(A.10)
m=
3

---

Under review as conference paper at ICLR 2020
Now have:
C(O, D) By(,,)) logpo(>>z)) H(g(x))
R +
R
(A.11)
(A.12)
Apply BTC decomposition to T as Chen et al. (2018). indexes over units in 2
T
[log do (2 12 p(z) log 90(2") (2 (1 B
loog (s
log II [0(2)) - log II [00(2)
(A.13)
B 90(2" Dog 20(24124~1)
8o(24)
EE go Dlg II go(o)))
Boo(2")
90(24)
II, (0()))
pl=
(A.14)
-o 1) flog 20(=4 اا )
E 2 (log go(24)
20(2)) )
Σ
W8 (2 flog
qa (= yas (2
II, 9s (2
pl(t)
(A.15)
=-DKu(go(3" + (=)))===-1) (KI
go (2))))
Σ
DKL (1p
(A.16)
Where have used p(2²) p(zf)) for our chosen generative model. As in Chen et al. (2018),
choose to weight To the total correlation for qo (=*". by prefactor B.
SB CS Φ. D) = Ey((,.)) po(l))) + H(g(z)) +
R₂
+
R
+
T
T
(A.17)
Giving us the ELBO for Seatbelt- VAEs, Eq (10)
4

---

Under review as a conference paper at ICLR 2020
B MINIBATCH WEIGHTED SAMPLING FOR 2"
As in Chen et al. (2018), applying 3-TC decomposition requires to calculate terms of the form:
Byo(et) log q (z')
(B.1)
The = 1 case is covered in the appendix of Chen et al. (2018). First we will repeat the argument for i made in Chen et al. (2018), but in notation, and then cover the case >> 1 for models with factorisation of 40 (z) as in Eq 7 in the main paper.
B.I MWS FOR (="=):: B-TCVAES
Introduce BM = {I1, T2., RM}. a minibatch of datapoints drawn uniformly iid from g(x) = 1/N E 5(x z.)) For for any minibatch we have p(Bar) N Chen et al. (2018) introduce r(B r(Bu(z), the probability of a sampled minibatch given that one member is x and the remaining M points are sampled iid from q(x), r (z)
M-1
(x:) log (=") go . [log By(z) (2 [+]]]
(B.2)
M Σ m=1
Eg .r) [log Bp(Bar)
90(2 [xm)]]
(B.3)
M
B(BM) 1
(x .n) loo Br (n)
Σ go(2" [m)]]]
(B.4)
rBB (x) M
m==
1 NM
go . log By(Bxr (w)
(0(2' [5))
(B.5)
(B.6)
So then during training, one samples minibatch {œ1 22, .... Fash and can estimate (xy(= log (2()))
as:
1 M
E (x+) log (=")
=
Σ log Σ go (o - log NM]
(B.7)
M
and =t is sample from (0(
B.2 MINIBATCH WEIGHTED SAMPLING FOR go(2"),i> 1: 8-TCGLGMS AND SEATBELT-VE VAES
Here we have that g(z, .)) (="""" datapoints, we have minibatch of draws of ='11; B {=f 22 ,.... --.)),((1) }. Each member of
[z) Now instead of having a minibatch of
which is the result of sequentially sampling along chain, starting with some particular datapoint Im ~ ((x).
For > 2, members of B are drawn:
2) g0(= (1ff2)
(B.8)
and for i = 2:
2) as(22 (z,))
(B.9)
Thus each member of this batch B the descendant of a particular datapoint that was sampled in an iid minibatch B& as defined above. We similarly define rB:i ( as the probability of selecting a particular minibatch B of these values out from our set { (of cardinality N) given that we have selected into our minibatch one particular zt-1 from these N values. Like above, r(B(T) (
5

---

Under review as conference paper at ICLR 2020
Now consider (zt( log go(2") for i 1:
Bygls) log go(=²) = E ( [log Beg(st- (a (a (=11))))1))
(B.10)
E go [log E p(B ")
1
Σ (
(B.11)
M
(BBM)
M
E 2t flg E ((1377 |
Σ 90(2 |mm
20 [
(B.12)
r(B (x)
M
=E (o Dlog E ((B77
1 | 1)
(=
(B.13)
NM
m==1
Where we have followed the same steps as in the previous subsection.
During training, one samples a minibatch { ,22 _--1 2 sampling ancestrally. Then one can estimate (x)() log g0 as:
}. where each constructed by
M
E Bottt) log (=()")
log
go (o(2)l=))1)
- log NM
(B.14)
M
1=
and of is sample from g0(=11z(-1) In our model we only need terms of this form for i = T., so we have:
M
M
E (x log (2a
log
- log NM
(B.15)
M
and # is a sample from go (=

---

Under review as conference paper ICLR 2020
C
IMPLEMENTATION DETAILS
All runs were done on the Azure cloud system on NC6 GPU machines.
C.I
ENCODER AND DECODER ARCHITECTURES
We
used
the
same
architectures as Chen et al.
2018. and
See
file
src/stochastic wtwphhhstttslaasssreeeeeeaee sereers/enoderrs. py in the accompanying repository.
ptC/stochattic__aarrseeeeee
For 8-TCVAE the range of | values used was (4, 16, 32, 64, 128}. For 3-TCDLGMs and Seatbelt- VAEs the number of units each layer z' decreases sequentially, There a list n izes for each dataset, and for model of L layers that the last L entries to give ||a ||, L).
[I dSprites -996, 48, 24, 12, 6}
(C.1)
{ [|]+|]] Chairs - 48, 24, 12, 6}
(C.2)
{ [||=||] (96,, 48, 24, 12, 6}
(C.3)
[12211] ColebA ={26,, 128, 64, 32)
(C.4)
(C.5)
For 3-TCDLGGMM and Seatbelt-VARR we also have the mappings To (2*+ and (1). These
are amortised as MLPs with hidden layers with batchnorm and Leaky-ReL.U activation. The dimensionality of the hidden layers also decreases as a function of layer index i:
( (=*+1(=() = -haizes [1]
(C.6)
[[b]] (='112=== (=))2 = =hsines [i]
(C.7)
hsizes [1024, 512, 256, 128, 64]
(C.8)
To train the model used ADAM (Kingma & Ba, 2015) with default parameters and a learning rate 0.001. All data was preprocessed to fall on the interval to 1. CelebA and Chairs were both downsampled and cropped as in (Chen al., 2018) and (Kulkarni al., 2015) respectively.
D L2 NORM OF 3- 3-TCDLGM
Table D.1: L2 of Encoders and Decoders by dataset for 3-TCDLGM (L = 4) showing the proportional change from increasing from to 10.
Chairs
3D Faces
CelebA
8: + 10
/: + 10
8:1 + 10
Encoder B 3CTCLLMM
+3.1%
+5.7%
+1.22%
Decoder 8- 3TTCLGM
-4.8%
-3.5%
+7.3%
7

---

Under review as a conference paper ICLR 2020
E
ROBUSTNESS TO NOISE
6.4
0.000
$000
2.000
690
8:00
809
0:01
0:31
0301
2008
2.00
0300
6400
0.001
0.50
6:000
3400
0.85
C004
0.50
5.06
800
6002
100
(a) Chairs 3-TCVAE
(b) Chairs 3-TCDLGM
(c) Chairs Seatbelt- VAE
6005
3500
0:00
600
3004
0.00
Saat
0000
6.00
Tarks
800
601
800
k00
0.07
6101
600
201
030
6.57
4.08
0.20
6.00
300
8.08
0.00
300
8000
630
600
350
0.0
600
602
300
620
670
600
200
130
200
000
020
0301
300
200
3008
2000
1000
300
-00
-50
2800
300
1100
300
2005
1003
130
-003
1003
-120
130
-200
20.5
-000
1200
-220
730
(d) 3D Faces 3-TCVAE
(e) 3D Faces 8-TCDLGM
(f) 3D Faces Seathelt-Vas VAE
0.0025
0.0025
0.0025
0.0025
celobA CelebA + D.1c
3.0008
CeloBA
calibAA
0.0020
0.0020
0.0020
0.0020
0.0006
3.0006
0.0015
0.0015
0,0015
0.0015
0.0010
6.0030
0.0002
3.0022
0.0000
-2000
2000
3.0000
2000
2000
0.0000
2000
0.0000
2000
0.8000
-2000
2000
0.0000
-2000
(090122)
p=1
(ogtxR2)) p-10
Aoppet211 B=1
loglx12") p=10
logixt2" p=1
1og p=10
(g) CclebA 8-TCVAE
(h) CelebA B- 3-DDLLMM
(i) CclebA Seatbelt VAE
Figure E. Robustness of log (x) to Gaussian noise A0, and added to x: on Chairs, 3D Faces and CelebA by row; for 3-TCVAE 3-TCDLGM, Seatbelt-VAn VAE by column. Within each plot range of 3 values shown. Best viewed digitally
(0. 1) scaled by different magnitudes
8

---

Under review as a conference paper at ICLR 2020
F ACTIVATION OF
F.1 3-TCVAES
For this subsection. within each subplot we order the units of by the values of their DKL divergence. dSprites
5
5
5
5
0
B=1.||z||=4
0
B-4,1|Z|1-4
6-8.1Z21144
0
6 B=10,1|Z|
5
5
5
0
B-1.1ZZ|-8
B-4.1
B 8
0
8 6 1
5
5
-=1. |
0
B-4.
|
8 2
0
B-10. 116
166
5
5
5
0
0
0
/=1.11Z| 332
B=4
2 32
=3
B-10,12|=32 =32
5
5
5
5
B=1.||Z||64
0
B=4.
1//4||2|/66
--
0
B=10.|
5
5
5
0
B=1/|2/|=128
B=
B 12|8
8-
0
|
B-10,
128
Figure F22 Egg()) DKL (c0(2)lz) over dSprites for 8-TCVAE over values of ||=|| and 3. Best viewed digitally.

---

Under review as a conference paper at ICLR 2020
Chairs
5
5
5
B=1.1lz|=44
B=4.1|z|1=4
B-8.1ZZ|14
β 1-10, 1//11/2|==
5
5
5
0
0
0
8-1.11Z|F8
B 1=4.11Z|188
β 8-8.12
G 1=00,122 1
5
5
5
0
|=1.1|2|=16
0
B-8.12
0
B=10,|Z|==1
5
5
5
0
/21
3 |
--8.12
0
B=10,|1z|=22
5
5
5
|=1.1|z||64
3-4.12Z|-66
/-8.1z
0
B=10,|zz|=6
5
5
5
0
0
0
B=1,|lz|=128
6=4
/z/,/22/=128
1-1,121
|=10,||z||=11
Figure F.3: (.) DKL (2)(l())) over Chairs for 3-TCVAE over values of ||=|| and 3. Best viewed digitally.
10

---

Under review as conference paper at ICLR 2020
3D Faces
10
10
10
10
B=1,1|z|F=
B= B=4.1lZ|F4
β 1=1.11Z11=4
G= 30 1z
10
10
10
10
0
B-1,1|Z|=8
B-4.1|z|
0
β 8
0
B=
10. 10,1
10
10
10
10
0
B=1 1=1.||Z|116
6-4. |=
0
8-8 8-8,112 /
6= 12.10
10.1
10
10
10
10
B-1, 6-1,1|z||=32
/=4,1|z|=32
0
8-8 /=8/1/Z|1=32
B=10.||z|=32
10
10
10
10
|||
0
0
B=1. //11/|//|=64
|z4,|12|1-44
B= |=|
B=10.||z|==4
10
10
10
10
0
0
0
B=1, /=//|/2/==128
1
|=10,||z|=122
Figure F.4: By(2)) DEL (c0(2)/))))(((()) over 3D Faces for 3-TCVAE over values of ||||| and 3. Best viewed digitally.

---

Under review as conference paper ICLR 2020
F.2
SEATBELT- VAES
Here the DKL. divergences are calculated per layer. dSprites
30
30
30
30
20
20
20
20
10
10
10
10
0
0
G=1,L=2
B=4,L=2
B=8,L=2
B=10,L=2
30
30
30
30
20
20
20
20
10
10
10
10
0
0
0
B=1,L=3
B=4,L=3
B=8,L=3
B=10,L=3
30
30
30
30
20
20
20
20
10
10
10
10
0
0
0
6=1,L=4
B=4,L=4
6-8,L=4
B=10,L=4
30
30
30
30
20
20
20
20
10
10
10
10
0
B=1,L=5
B=4,L=5
B=8,L=5
B=10,L=5
Figure F.5: E (2 . DKL. (2. ||(2*12
where x 20 and p(2 = p(=))
go
over dSprites for Seatbelt- VAES over values of L and B. Best viewed digitally.
12

---

Under review as conference paper ICLR 2020
Chairs
30
30
30
30
20
20
20
20
10
10
10
10
0
0
0
B=1,L=2
B=4,L=2
B=8,L=2
B=10,L=2
30
30
30
30
20
20
20
20
10
10
10
10
0
0
0
6=1,L=3
B=4,L=3
6=8,L=3
B=10,L=3
30
30
30
30
20
20
20
20
10
10
10
0
B=1,L=4
B=4,L=4
B=8,L=4
B=10,L=4
30
30
30
30
20
20
20
20
10
10
10
10
0
0
0
B=1,L=5
B=4,L=5
B=8,L=5
B=10,L=5
Figure F.6: (g Liilly (x
||
||("||+(+1)
where 20 and p( (22112++1 p(2)),
over Chairs for Seatbelt- VAEs over values of and B. Best viewed digitally.
13

---

Under review as conference paper ICLR 2020
3D Faces
40
40
40
40
20
20
20
20
0
0
0
B=1,L=2
B=4,L=2
B=8,L=2
B=10,L=2
40
40
40
40
20
20
20
20
0
0
0
B=1,L=3
B=4,L=3
6=8,L=3
B=10,L=3
40
40
40
40
20
20
20
0
B=1,L=4
B=4,L=4
B=8,L=4
B=10,L=4
40
40
40
40
20
20
20
20
0
0
0
B=1,L=5
B=4,L=5
B=8,L=5
B=10,L=5
Figure F.7: E (s' 1iyl DRL DKL(go<="|<<11) (+=21=+(+1
where 20 and p(24|24+1) p(2)),
over 3D Faces for Seatbelt- VAEs over values of L and B. Best viewed digitally.
14

---

Under review as conference paper at ICLR 2020
G AGGREGATE ANALYSIS OF ADVERSARIAL ATTACK
G.1 3-TCVAE
.Latent Space Altack
Outpul Attack
Lafent
1e3
Oulput
Parge--Recon Distance
6.0
Adversarial-1 Soantal-Targjee Tanget Distance
16
10
1.9
5.5
14
5.0
Adversartal
10"
12
4.5
10
1.5
4.0
3.5
5
B
B
10
B
(a) dSprites Distances
(b) dSprites Losses
Latent Space Altack
Oulput Atack
1e2
Output
7.5
7.0
Adversarial- Target Distance
16
10
Tange-RRecon Distance
6.5
Adversaftal
10
7.0
12
10
6.0
10
5.5
8
10"
5.0
5
2
6
10
2
6
10
B
(c) Chairs Distances
(d) Chairs Losses
Latent Space F Ollpkl
Latent
Oulput
3.4
Adversarial- Target Distance
Targel- Targel-Recon ReRcon Distance
3.2
10"
1.65
Adversartal
2.8
10a
1.60
2.6
1.55
1.2
6
10
2
6
10
8
B
(e) 3D Faces Distances
Faces Losses
Figure G.8: Plots showing the effect of varying 3 in 3-TCVAE for dSprites and 3D Faces on: (d) the L2 distance from zt to its reconstruction when given as input and the L2 distance from the adversarial input x and its reconstruction; (b).(e) the adversarial objectives ootpet/output
15

---

Under review as conference paper at ICLR 2020
G.2 B- 3-TDLLGMs
[|Adversaraa- Targed
Latent Space
Latent
x10
Output
x10
Oulput Attack
1.5
Latent Space Altack
[[Targe:
50000
4,8
4000
Output Altack
4.0
Jeg(pixtel
3000
2
3
4
2
3
4
depth
0.9
3.2
4000
0.6
2000
2.4
0.3
1000
2000
2
3
4
2
3 depth
4
5
2
4
6
8
10
2
4
6
8
10
depth
depth
dSprites Distances
dSprites Losses
(c) dSprites Negative Log Likelihood
||1 [TTarget-Raoo Rect-R
|AAvarsatial-Tan Saaesaiill-Tagellll Targell]
200000
Latent Space Atlack
Output Attack
Latent
Oulput
1,590
800
Latent Space Allack
Outpul Altack
1.8
100000
700
1.575
log(p(xtz)
depth
1.560
0.8
860
1.545
1.530
800
5
2
1
10
depth
depth
depth
(d) Chairs Distances
(c) Chairs Losses
(f) Chairs Negative Log Likelihood
160)
Latent Space Allack
Latent
x10
Output
x10
Oulput Altack
Latent Sacee Allack
Target RecoR Distancs
2.40
Adversarial-Tai Distance
1400
1.590
1575
Outpul Altack
1.6
2.38
1.575
2.36
1000
1.2
1625
depth
800
1.560
2.34
0.8
1475
1.545
0.4
200
1.530
1425
10
depth
depth
B
B
depth
(g) 3D Faces Distances
(h) 3D Faces Losses
(i) 3D Faces Negative Log Likelihood
Figure G.9: Plots showing the effect of varying L on 3-TCDLGMs for dSprites 3D Faces and Chairs, on: (g) the L2 distance between x² and its reconstruction when given input and the same between the adversarial input x" and its reconstruction; (b), the adversarial objectives Anuput/imager (f). log (=(1222), g0(2(=") and the MIG.
For DLGM (Rezende et al., 2014) with 2-5 = layers, with (z(z)z) factorised as in Eq 7 (x, z) factorised as in Eq (6). and BTC penalisation applied the top layer, we find that latent attacks targeted =² are highly effective and remain L and 3 each increase. These models slightly more robust to output attacks and this attack becomes less effective as B increases, but more effective L increases.
however,
The ease of attacking via 2 consistent with its separation out from the rest of the model.
16

---

Under review as conference paper at ICLR 2020
G.3
SEATBELT- VAES
[|Adversanall Targelll
Latent Space Atlack
Latent
x10
Outpul
x10
-100
Latent Space Atlack
[Target-Recon
Oulput Allack
Output Altack
6
-1500
4.8
-20000
lagip(xttl
4.0
de petth
400000
2500
4000
1.0
2.1
-60000
3000
0.5
2000
1.8
-80000
0.0
depth
depth
depth
(a) dSprites Distances
(b) dSprites Losses
(c) dSprites Negative Log Likelihood
[Taage:-Racnn)) Rec-Rl
[|Adversaraal Target]]
1:
Latent Space
Output
Latent
x10
Output
x10
1.0
6.8
4
0.8
2
4
4
depth
0.6
142
143
193
650
0.4
183
625
162
163
0.0
2
4
10
2
4
6
8
10
depth
depth
(d) Chairs Distances
(e) Chairs Losses
(i) Chairs Negative Log Likelihood
|lAdversania-11 argatil
610
Space
x10
Output
x10
Space
Parget-Reconl
3.5
Output Allack
7.5
2800
1520
Output Allack
3.0
6.0
1510
2.5
2400
2
2
3
depth
4.5
2200
1500
3.5
1490
3.0
1560
1800
1.5
2.5
1540
1600
0.0
4
2
3
2 4
10
1
2
8
10
2
3
4
5
depth
depth
B
depth
(g) 3D Faces Distances
(h) 3D Faces Losses
(i) 3D Faces Negative Log Likelihood
Figure G.10: Plots showing the effect of varying L, 3 on Seatbelt- trained on dSprites, 3D Faces and Chairs on: (a).(d)(g) the L2 distance between xt and its reconstruction when given as input and the same between the adversarial input x" and its reconstruction; (b), the adversarial objectives Aoutput/image- (f) log pg(x"12), (2(x)²))
17

---

Under review as conference paper at ICLR 2020
G.4
SEATBELT- VAE LAYERWISE ATTACKS
X10
Z1
x10"
x10"
7
5
6
6
6
5
5
3
3
3
3
2
2
4 B
8
1
4
8
4 B
8
B
(a) 3D Faces
x10
x10
Z
x10
3.0
2.5
2.5
2.5
2.0
2.0
2.0
3
1.5
1.0
1.0
1.0
0.5
4
8
1
4
8
B
(b) Chairs
Figure G.11: - log (=())) for Seatbelt-VAEs for (a) 3D Faces and (b) Chairs; over 3 and values for latent attacks. We attack the bottom layer (z'), the top layer (z²), and finally show the effect when attacking all layers (2). Larger values of log po (x*12) correspond to less successful adversarial attacks.
18

---

Under review as conference paper ICLR 2020
H ADVERSARIAL ATTACK PLOTS
H.1 DSPRITES ADVERSARIAL ATTACK ON A SINGLE FACTOR
Latent Attack
Original
Original rec.
Adversriial
Original
Original rec.
Advemsrial
Target
Adversarial rec.
Distortion
Target
Adversarial
Distortion
S-TCVAE, 8
I|]|
8-TCVAE, ||=|| 64
Original
Original
Advereariil
Adversarial
Target
Adversarial rec.
Distortion
Target
Advemsrial rec
Distortion
A-TCDLGM, B
B-TCDLGM. 2,
Original
Original rec.
Adversarial
Original
Original rec.
Advensarial
Tamett
Advemarial
Distortion
Advemariall
Distortion
8
Seatbele-AAE
Figure H.12: Latent space attacks rotation only of a heart-shaped dSprite for 3-TCVAEs, 8- TCDLGMs and Seatbelt- VAEs for 3 = (1,2).
19

---

Under review as conference paper at ICLR 2020
H.2
DSPRITES ADVERSARIAL ATTACK
H.2.1
8-TCVAES
Output Attack
2|1111
Mill
2|111
8illll
10:1l11
2|111
10.1=11
Milll
MIll
10.1::1
Milll
M|=l
1||ls|
== 2||l|]
4.|/=|| 64
6.112 64
8 8=|=]]]|
10||=||
Figure H.13: Output attacks dSprites 8-TCVAES for 3 =(1,2, 4, 6,8, 10) and |||||
(4,6, 6, 8, 16, 32, 64).
20

---

Under review as conference paper ICLR 2020
Latent Attack
2|1==1|
10/|=11
2|111
10.
|=||
1|||=|1
|=||
16
0
4|1=11 16
16
8|1|=1
16
10:11=11
16
1||=|| 32
|||||| 32
M|=11 32
32
6:1|11 || 32
10 32
4= 1|/11| 64
P 2|||=|| 64
4.||=|1 = 64
e 6.11= = 64
A A= 8.1|-1 = 64
10/|==1 64
Figure H.14: Latent attacks on dSprites on 3-TCVAEs for 3 = (1,2,4,6, 10} and || (4. 6, 8, 16, 32, 64).
21

---

Under review as conference paper ICLR 2020
H.2.2 3-TCDLGMS
Output Attack
Doginal
3 2,
B
4,
6,
L
B = 10, L
Orcpet
Doppettnn
Drope
Langt
DISCOE
Taye:
TAye
Targe
Adwiti
Targe
1,
2, L = 2
8 = 4, L
8 6, T
=
2
2
8 = 10, T 1
Tirget
Terget
L 3
B = 2, L 3
B = 4, 3
8 = 6, L
8 = 8, L = 3
8 = 10, L =
Tayel
Advesataa
Tagel
Taye
DrSdton
Targ
Adwrts
Dilution
Targe
Abwatan
T,
4
4
i
4, T
B
6, T
4
4
8 10,
Adwessn
Oopnss
Adwrsre
Ture
1, 5
= 2, L
A = 4,
8 = 6, L =
= 8, L
8 = 10, L
Figure H.15: Output attacks on dSprites on 8-TCDLGMs for = (1,2, and L = {1, 2, 4,5).
22

---

Under review as conference paper ICLR 2020
Latent Attack
Ortene
Oniinal
Adtemment
Onjinel
Orignn
Ademsne
Orane
Orgwn
Adementt
Oroine
Organiee
Adwmsnn
Oroine
Orgnns loc
Orginel
Orginernc
Lurget
Iong
8=1,==1
= 2
8 == -=1
B = 6
B -=8
8 = 10, L=1
Torpet
Tumpst
Tomes
Tompe
Dilerdn
Torer
Torer
B 1, T. 2
2, I.
B = 4,
= 2
B 6, T
T 2
8 10, T.
Terpet
Dal-hin
1, 3
3
8
4,
8
6,
L
8, L 3
8 10, L
Timet
4
4
8 4,
L
8
6,
L
4
8 10, 4
Tayw:
Drodto
8=LL=
8 4,
6, L
8 10,
Figure H.16: Latent attacks on dSprites on 3-TCDLGMs for 8 (1,2, 2, 8, 10) and L (1, 2, 5).
23

---

Under review as conference paper ICLR 2020
H.2.3 SEATBELT- VAES
Output Attack
Dogina
Orgalte
3 = 2,
B
4,
6,
L
8,
B = 10,
Orypet
Doppeten
Dropee
Taye:
ACMMERAINNG
TAye
Dridoo
Ad
Targe
1=1,=2
8 2, 2
8 = 4,
8 = 6, T = 2
T 2
8 = 10, T. =
Torget
Terget
= L 3
8 = 2, L = 3
B = 4, 3
8 = 6, L
8 = 8, L = 3
8 = 10, L = 3
Tagel
Advesataa
Tagel
Taye
Diidton
Tagar
Targ
Aderts
Drlutton
Targe
Adeata
Eilotton
T
4
4
i
4,
8
6,
T
4
8 10,
Adwrssr
Oopns
Orginenic
DuAnbin
1, 5
= 2, L
i = 4,
L
8 = 6, L = = 8, L
8 10, L
Figure H.17: Output attacks on dSprites on Seatbelt- VAEts for B = (1,2,4, 6, 10) and = (1, 2,3, 4,5).
24

---

Under review as conference paper ICLR 2020
Latent Attack
Ortener
Oniynnll
Anhemment
Oniinel
Orignn
Adement
Orane
Oroan
Ademant
Oroine
Organiee
Ademsun
Oroinel
Oriins loc
Orginel
Orginrnc
Lurget
Tong
8=1,==1
0=2L=1
B=4L==1
B = 6=6.L=1 ==1
8=
8 == 10, L=1
Torpet
Tomps
Tome
Tompe
1, T. = 2
== T. =
B 8=4,1=2
B = 6, T. = 2
= 8, T = 2
8 10, T. 2
Terpet
1, L =3
== 2, L =
8 4,
8 = 6, L
8, L 3
8 10, L
Timel
L 4
2, L 4
8 4, L
8 6, L
8 10,
Taye
Drodto
8=1L=5
0 4,
= 6, L
8, 5
8 10,
Figure H.18: Latent attacks on dSprites for Seatbelt-VAN for B (1, 4, 6,8, 10) and I. (1, 2, 4, 5).
25

---

Under review as conference paper at ICLR 2020
H.3 CHAIRS ADVERSARIAL ATTACK
H.3.1 3-TCVAES
Output
2||l|||
10.
|=ll
2|||||
6.||=11
8.1ll|
|=ll
2||ll|
G||l||
8.1|=|
|0|||:||
21||
4||lll
6.11
8.1|=11
10.11
M||ll
|||||||
4||=||
6.11=
8|||=||
10.|1=||
1||=|| 64
|||||||
4.||=|| 64
6.112 64
8.1|=|| 64
10
Figure H.19: Output attacks (4, 6, 8, 16, 32, 64).
Chairs for 3-TCVAES for [1,2, 4, 6, 8, 10) and |||||
26

---

Under review as conference paper ICLR 2020
Latent
2||ll|
G||lll
10/||=11
2||1=||
101/|=11
2||==|
Gi|lll
8.1|+11
|0.|==||
1|/|=|| 16
||||||| 16
6 4.||=|| 16
G 16
8|||=11 16
f 1il||=|| 16
1||=I| 32
A ||||||| 32
A 4||==|| = 32
# 6.11= = 32
= 8|||=|| = 32
8 10 = 32
1|/+|| 64
P |||=|| 64
A 4|||=| = 64
e 6.11= I = 64
A 8|||-1 = 64
10 64
Figure H.20: Latent attacks on Chairs for 3-TCVAEs for B = 4, 8, 10} and || (4. 6, 8, 16, 32, 64).
27

---

Under review as conference paper ICLR 2020
H.3.2 3-TCDLGMS
Output Attack
Orgiaa
Originll
Ooginal
Origial
Oegral
Degine
Orgaaing
Oegina
Orgaann
Tappe
L
1
B 2
==4.
B = 6, ==
== 1
B 10, =1
Original
Orypod
Depie
Diget
Tagel
Tage:
Tage
Drietion
Taree
Targe
= 1, L = 2
== 2
== 4 ==
B = 6, T 2
== 8, T=
B 10, T T=2
Torpel
Terget
Topge
Momanion
Tapge
I2 7'
= 1, L 3
B 3= 2, L=
1=4L==3
B = 6, L =
B = 8, L 3
B = 10, L =
Oniginl
Oropertee
Tagel
Advesata IG
Brocot
Tage:
Drodd
Tage
Dridtn
Tager
Alwaran
Target
Drlation
Targe
Adwaaa
Divcton
1, L 4
B 2 2,
B 4
B = 6, I = 4
= L
8 = 10, T. =
Tupel
Mimumito
1, L
3=2,L=
8=4,
3= 6, =
8= 8,8,
== 10, L
Output attacks on Chairs for 3-TCDLGMs for 3 = (1,2,4,6,68 10) and (1,2,8.4,5).
28

---

Under review as conference paper ICLR 2020
Latent Attack
Orghnl
Orignnl
Adaement
Orginel
Orignl
Oege
Orginn
Orginel
Organinc
Orginal
Orgnnl NC
Orgnn
Orgaennc
Eagget
Terget
Togge
Toggl
==1L=1
1
2
8=4L=1
B
8=6,L=1
8=&
10 L=1
Capprll
Depwe
Toppe
Denene
Timee
Detrte
Tome
Detrteo
Topp
Mareratre
Daledon
Teree
Deledon
Torpr
= 1, I = 2
== =
B 1=4T==2
B = 6, T. = 2
= 8, T 2
8 = 10, T. = 2
Ademsan
Adwrssna
Terpd
Momunion
Delnte
Momunio
Dhleder
Dakrbe
Dulrbon
1, L 3
3
β 4, L
8 = 6, L
8, L 3
B 10, L 3
Aevematd
Timeet
Anvinete
4
B 2, 4
8 = 4,
8 = L
6,
8, 4
3 10, 4
Adveaate
Anvertateec
Drodto
Adwranmmc
Drordion
1,
8 4,
6, L
B=
3 10, 5
Figure H.21: Latent attacks on Chairs for 8-TCDLGMs for 21, 4, 6, 101 and L = (1, 2, 4,5). 5).
29

---

Under review as conference paper ICLR 2020
H.3.3 SEATBELT- VAES
Output Attack
Orgiaar
Original
Orginal
Orggaal
Oignal
Orygal
Oogina
Orgaains
Deginal
Orgaain
Tappet
L
B = =1
8=4L==
B = 6, =1
L
B = 10, =1
Origonal
Orypoal
Depee
Depasten
Tagel
Tage:
Tage
Tape
Alwaraiicc
Dristo
Tage
Bradton
Targe
= 1, L = 2
B == 2, L = 2
8=4,L=3
B = 6, T =
== 8 T 2
B = 10, T. T=
Torpel
Terge
Topgl
Tupge
1, L = 3
B == 2, L = 3
B = 4, =
B = 6, =
= 8, L = 3
B = 10, L =
Tagel
Advesta IG
Tage:
Tage
Dridton
Tager
Adwarann
Ditstion
Targe
Drlston
Targe
Adwaan
Diboton
1, L 4
4
B ==4,1 4,
B 6, I 4
= 8, T T
8 = 10, T =
Terp
Dalnt
Tupel
Mimumit
Dulaker
L
8 L
8 0
8 = 6, L =
= 8, L 5
8 10, L
Figure H.22: Output attacks on Chairs for Seatbelt-VAEs for = = 21,2,4, 4,4,6688 6,8, 10) and L = {1, 2,3, 4,5).
30

---

Under review as conference paper ICLR 2020
Latent Attack
Orghnl
Orignnll
Advementt
Orignal
Oege
Organ
Orginel
Organin
Orginal
Organi NC
Orgn
Orgaannc
Eagget
Target
Togge
Torge
=
β 2=2,
8=4L=1
6=6
8=&
== 10, L=1
Ceppell
Torpet
Denone
Times
Detrte
Time
Tepe
Toree
Tome
= 1, I =
== 2= =
8=4, T=
B = 6, T. = 2
= 8, T 2
B == 10, T. =
Ademsan
Delete
Duidker
1, L 3
3
β 8= 4,
B 6, L
8, L
B 10, L =
4
B 2, L 4
8 4,
8 6, L
8=
8 10, 4
Advoaate
Drodtr
Adwannmcc
1, 5
8 4=4.-5 5
6, L
8=8,L=5
3 10,
Figure H.23: Latent attacks on Chairs for Seatbelt- VAEs for B = 21,2,4,6, 10) and L {1, 2, 5).
31

---

Under review as a conference paper at ICLR 2020
H.4 3D FACES ADVERSARIAL ATTACK
H.4.1 8-TCVAES
Output Attack
I|||=||
2||l||1
4.||=|1
8|||=||
10]]|=||
Milll
2||1|| =
4.||=|1 =
6.11a|
8.11=11
10.11=11
2|/1/|1
4|||=|1
6.11=|
8.11=11
10.]|=||
2.1111
4.11al
6.1|
8.l|ll
10.1
M||=||
2|||=||
4.||=|
61||=|1
B 8.|| 32
10.11=11
1||=||
8 2|||a|| 64
4.11a|
8 8.]|=|| 64
B 10.|1=||
Figure H.24: Output attacks on 3D Faces for 3-TCVAES for -1 4, 6, 10) and |||||
(4. 6, 8, 16, 32, 64).
32

---

Under review as conference paper at ICLR 2020
Latent Attack
x|||=||
I||||||
4|||=||
8|||=||
10.]|=11
2||||||
L||=||=
8.1[2|
8|||=||
10.]|=||
2||=1|
4,||=||
A 6.1|=|
8//|=1|
10.]|=||
I|||=|| 16
P 2|||=|| 16
4.||=| 16
6,|/=||
0 8|||=|| 16
B 10///=|| 16
1||=1| 32
# = 2|||=|| = 32
= 4.||=| = 32
6|||=|1 32
B == 8.||=|| = 32
8 10.1 = 32
== 1|//=|1 = 64
A = 2|||=|| = 64
= 4.1|=| = 64
6/|/|| 64
A 8.||=|1 = 64
A 10.1 64
Figure H.25: Latent attacks on 3D Faces for 8-TCVAEs for 3 = (1 4, 810} and ||*|| (4. 6, 8, 16, 32, 64).
33

---

Under review as conference paper ICLR 2020
H.4.2 3-TCDLGMS
Output Attack
Originl
Oeggal
Origna
Oegrat
Oryant
Oeggna
Organiie
Deginal
Orgintin
Orggaric
Tarpel
Tarpet
Meranitec
= 1, =
B == 2, L
=
B = 6, L 8, L =
8 = 10, L = 1
Origin
Anterment
Orypoct
Oroppit
Orppertin
Deppsten
Dripuat
Tagget
Aduetata
Dromaa
Tage:
Tage
DIDTOO
Targe
Targe
= 1, 2
B = 2, L =
B = 4.
8 = 6, T = 2
= 8, L 2
B = 10, T. ==2
Onig
Orignd
Ohignnal
Tergel
Terget
Momuniioo
B = 1, = 3
B 3= 2, L =
B 4
== 6, L = 3
B = 8, L = 3
8 = 10, L =
Tagge
Advesstaa INC.
Tagel
Advenitanic
Tage
Tage
DIESSOO
Tage:
Aderaca
Tagg
1, T 4
L = 4
B 4, T
8 =
6,
I 4
= 8, To = 4
8 10, T 4
Momunito
Tumpel
Mimunioo
f = 1, L. =
8 2, L
β 8= 4,
8 = =
8 8, L 5
8 10, L
Figure H.26: Output attacks on 3D Faces for 3-TCDLGMs for = (1,2,4, 10) and L = {1, 2,3, 4,5).
34

---

Under review as conference paper ICLR 2020
Latent Attack
Originl mel
Adderment
Oriine
Origna
Orone
Organ NO
OrOie
OrganinG
Orona
Orgnni HC
Orgna
Orggnnnn HY
Eagget
Target
Momunate
Tirgel
Momustt
Taggl
Wine
=
==
β == 2=
==4,L=1
B = 6, ==1
β == 8, ==1
B = 10, ==1
Droped
Drppuct
Tirpet
Tereet
Timee
Thepe
Deteroo
Turm!
Menert
8 = 1, T. 2
= T. =
8= T =
B = 6, T,
= 8, T. 2
B 10, T. =
Oniginll
Terpe
Momonioo
1, L 3
3
β = 4, L
8 6, L
8, L 3
8 10, L
Advomantt
Target
Aevomatt
Mereet
L 4
= 2, L 4
B 4, L
8 6, L
8,
3 10, 4
Drodor
Diortoon
8=LL==
8 4,
6, L
==
8 10,
Figure H.27: Latent attacks on 3D Faces for 3-TCDLGMs for B = = (1,2,4,8, 10) and L {1, 2, 5).
35

---

Under review as conference paper ICLR 2020
H.4.3 SEATBELT- VAES
Output Attack
Originll
Oeggal
Oegnal
Oryant
Degina
Organtie
Deginal
Orgintin
Orgaaric
Tarpet
Advemane
B == 2,
=
8 = 6, L =
== 1
8 10, L =
Orypoit
Oropoit
Deppette
Dripucto
Tagg
Adverata
Duprat
Tagel
Tage
Tage
Targe
Adwaraa
Targe
B = 1, L
B == 2, T= =
B = 4,
8 = 6, T = 2
= 8, T 2
B = 10, T. =
Origan
Origod
Tergel
Turget
Momanion
Tapge
Seat
B = 1, 3
B = 2, L =3
B = 6, L =
B = 8, L = 3
8 = 10, L =
Onlg
Tagge
Advesata IG
DESCOR
Tage:
Tage
Tage
DIESSOO
Tage
Targe
1, T 4
B == 2, T 4
B = 4, T == 4
8 = 6, I 4
8, T
8 10, T
Momoniton
Tumpel
1, L= =
8 2, L
8 L
8 = L =
8 = 8, L
8 10, L
Figure H.28: Output attacks on 3D Faces for Seatbelt- Seatbelt-VAEs VAEs for 3 = (1,2,4, 6, 8, 10) and = {1, 2,2,3,4,4 4,5).
36

---

Under review as conference paper ICLR 2020
Latent Attack
Oniginl nel
Addemmennll
Oriinel
Orignn
OroNe
Organ NC
Ademann
OHOPE
OrgaAiNG
Orona
Orgnnl NC
Orgna
Organnncc
Eagget
Target
Targel
Momundtes
Targe
β ==
==
= =
#=4,L=1
B = 6, ==1
B -=1
B = 10, -=1
Dropectoo
Tireet
Tecoree
Terest
Time
Thee
Maranatie
Term
Addemenna
B = =
== T =
8=4.1=2
B = 6, T. =
= 8, T. = 2
B 10, T.
Turpe
1, L 3
B ==
3
β = 4, L
8 = 6, L
= 8, L
3= 10, L =
Advomant
L 4
== 2, L 4
B 4, L
8 6,
L
8, L 4
8 10, 4
31
=
8 8
= 6, L
8= 5
8 10,
Figure H.29: Latent attacks on 3D Faces for Seatbelt-VAN VAEs for B {1, 4, 6, 8, 10) and {1, 2, 4, 5).
37

---

Under review as conference paper at ICLR 2020
H.5 CELEBA ADVERSARIAL ATTACK
H.5.1 8-TCVAES
Output and Latent Attacks
Torget
Advenatal
Tarnet
Datarte
Corasion
Tango
Eoberton
Ourpus,
(b) Outpor,
(c) Latent, B
Latent, =
Figure H.30: Output (a) (b) and Latent (c) (d) attacks on CelebA on 3-TCVAES for 3 {1, 103 and ||||| = 32.
H.5.2 3-TCDLGMS Output and Latent Attack
Geggnal gaal
org
Orgaa
Adamnts
Doggt
(a) Outpus,
(b) Outpur, 5 10
(c) Latent, 8
(d) Latent, # = 10
Figure H.31: Output (a) (b) and Latent (c) (d) attacks on CelebA on L = 4 3-TCDLLGMs for {1, 103.
H.5.3 SEATBELT- VAES Output and Latent Attack
Geppal Oremial
Drged
Tecsst
Torest
O Daltte
Texge
Telerios
(a) Outpur, (b) Outpus,
(c) Latent, 0
(d) Latent.
Figure H.32: Output (a) (b) and Latent (c) (d) attacks CelebA L Seatbelt-Var for = 10}.
38

---

Under review as a conference paper at ICLR 2020
1 DATA GENERATION FROM MODELS
Ancestral Sampling in CelebA
(a) L=
(b) == 10
(c) L=
(d) L = 10
Figure 1.33: Means of the decoder from ancestral sampling in z, for Seatbelt- VAES with L (1,4).
{1, 10). Note that there reduction in diversity of the samples for L (ie a 3-TC VAE),
3 10, which is not the case for the samples from the 3 = 10 L = 4 Seatbelt-VAE.
39

---

Under review as conference paper at ICLR 2020
Latent Traversals for dSprites
(a) 3-TC DLGM
Seatbelt-VAL
Figure I.34: Latent traversals in the |2| = top layer of L 2, B for 3-TC DLGM and Seatbelt-Var VAE trained on dSprites. Note that the traversals do capture the ground-truth factors of variation.
40

---

Under review as conference paper at ICLR 2020
J MUTUAL INFORMATION GAP
The Mutual Information Gap (Chen et al., 2018) average over ground truth factors variation of the entropy-normaliie difference between the greatest mutual information between the any of the units in z and a given ground-truth factor of variation and the second-greatest such mutual information:
K
MIG
[2z vx) max (2jjitt PR)]
(I.1)
H(vk)
where 2j arg max, 1(zj, vk).
Table J.2: MIG in sh: for L 2; for 3-TC DLGMs and Scatbelt-VAEs. for range of B values; for dSprites, 3D Faces and Chairs.
dSprites
3D Faces
Chairs
3
3-TC DLGM
Seatbelt-VAF VAE
3-TC DLGM
Seatbelt-VAE
3-TC DLGM
Seatbelt-VAI
1
0.0411
0.0475
0.0211
0.0300
0.0381
0.0134
2
0.3294
0.3589
0.4038
0.2200
0.2641
0,5366
4
0.2751
0.3235
0.2904
0.2349
0.7660
0.4963
6
0.3213
0.3258
0.1806
0.1890
0.1929
0,3111
8
0.3076
0.3182
0.2046
0.2301
0.2526
0.3329
10
0.3547
0.3415
0.1440
0.1053
0.1625
0.2802
41

---

Under review as conference paper ICLR 2020
noon
0.35
0.33
0.25
0.25
0.15
0.10
0.06
10
(a) dSprites
0.22
0.12
0.05
10
(b) 3D Faces
02
10'
(c) Chairs
Figure J.35: MIG for 3-TC VAEs function of |2] for different values of B. Note that MIG decreases as we increase |z], indicating that we get degenerate latent representations. that is different units in end up with similar mutual information the ground truth factors. The red line in a) at |z| = 6, the number of ground-truth factors of variation for dSprites.
42


Reviews:
[{"review_id": "r1eYj0gftS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I am a bit confused about the model. While i understand the generative model in Fig 2a, i am a bit puzzled about the choice of proposal distribution eq(7)/Fig 2b for the Seatbelt-VAE. The key claim of the paper is that an attacker has to attack all layers at the same time to attack the reconstruction in eq (12). However, Figure 2b and eq(7) claim that all deeper layers only depend on the previous layer in the approximate posterior. Since in (12) we rely on the posterior for the attack, a successful attack on layer m < L should immediately generate the correct values from q_phi(z_{i+1}|z_i) i=m,...L-1.  So from that point of view it is not a seatbelt, as since in Fig 2b if the attacker manages to control z^1, he has immediate control of z_2 and therefore the now  attacker controlled z_2 will directly feed into the generated target. What might be is that the optimization problem (12) becomes harder to solve because of the increased model-complexity. \n\nI am not too impressed by the attacks presented in Fig1 as well as the appendix. One of the key points of the old adversarial attacks was that the attack-image was indistinguishable from the true image by a human.  However, the adversarial inputs, even for VAEs are clearly not part of the distribution and the errors reported for eq (12) are very large to the point where attacking via the target image would probably be harder to spot. If we for example look at page 24, second row: there is no way, that the adversarial image has a likelihood similar to the target. This looks more like the algorithm did not manage to find a suitable direction.\n\nI am therefore not sure whether the evaluation is correct: if we did not manage to find an attack image, does it proof there is none? And is it meaningful to report their error values if we did not manage an attack?\nBtw: did the optimization of (12) begin with d=x_t-x or d=0? maybe starting with d=x_t-x would be more meaningful because it would make it easiest for an attacker to ensure the correct reconstruction.\n\nGiven the quality of the attack images, the error of eq(12) should be reported when choosing d=x_t-x as a baseline in Fig 5. It would also be good to see the actual VAE values.\n\nUnfortunately, the reconstructions on dsprites are bad. But an important experiment could be to check whether you can attack the orientation of an object. Orientation is difficult to regularize via TC, since the parameterisation is inherently circular. Thus TC might make it difficult to encode orientation in higher layers and it should be easier to attack.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SJxmteaQtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors of this paper propose a new VAE model called seatbelt-VAE and investigate its robustness to output and latent adversarial attacks. Inspired by beta-TCVAE, DLGM, and BIVA, seatbelt-VAE allows multiple latent layers, enforce disentanglement via weighted total correlation on the top latent layer, and conditions the likelihood on all latent layers. Robustness to adversarial attacks is the focus in experiments. Visual and quantitative comparisons show that seatbelt-VAE is more robust for latent attack than benchmarks. Specifically I have the following three concerns:\n\n1. Defining ELBO using samples from the entire dataset may bring in some benefits, but it complicates the calculation of ELBO and related distributions when minibatch or single sample are used in learning and inference. Please explicitly discuss this issue.\n\n2. I would like to see how seatbelt-VAE performs in sampling and generating new samples, instead of just reconstruction. \n\n3. Similarly, it would be beneficial to investigate disentanglement, that is the interpretation of the top latent factors in seatbelt-VAE. \n\nMinors: \nFactor analysis -> factor analysis\nsection X -> Section X\nfigure X -> Figure X\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BkelA7lkqr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper examines adversarial attacks to a VAE. It is known that by small norm perturbations on the conditioning input x of a VAE can dramatically change the generated output. This paper\nempirically illustrates that alternative objectives can improve robustness, in the sense of previously proposed adversarial attacks such as (Tabacof et al. (2016); Gondim-Ribeiro et al. (2018); Kos et al.(2018)). This paper is concerned with the stability of reconstructions and their quality, and proposes Seatbelt-VAE to remedy some of the shortcomings in the original VAE objective. \t\t\n\nThe key idea of the Seatbelt-VAE is introducing a conditionally Gaussian chains (of length L) in the encoder and decoder distributions of a VAE. This is a plausible and sensible idea. Then the authors evaluate the robustness of reconstructions under various output attacks.\n\nThe methodological part of the paper is quite well written and easy to follow, despite the fact that it is somewhat overloaded with too many abbreviations. The experimental section is harder to read as the motivations and its organization is not clearly stated. Overall, this section feels as if it is too hastily written, many results put into appendix without much discussion. The organization can be much more improved.\n\nThe disentanglement achieved by this novel representation is characterized only anecdotally and by contrasting the resulting objectives to a beta-VAE. It would have been much more informative to illustrate and discuss further the representations learned by such a conditionally Gaussian architecture. Figure 6 and 7 partially try to achieve this by showing the interplay of depth L and the inverse-dispersion parameter beta but I found it hard to interpret this results, for which an entire page is devoted.\n\nIn the experiments, the ELBO is reported for various methods. I would argue that the ELBO is not a very representative proxy for robustness. For example VAE ELBO and beta-VAE ELBO are both lower bounds of the true marginal likelihood and it is possible that beta-VAE is much lower while attaining a higher robustness in the sense of being resilient to suitably defined attacks.\n\nThe authors claim that there are no clear classification tasks for the datasets -- but this is not accurate as both celeb-a has clear classification tasks in the form of predicting attributes. It would have been really quite informative if adversarial accuracy on downstream tasks would have been reported. Relying on qualitative results in Figure 1 is only providing partial evidence about the approach.\n\nRobustness to independent noise, as the authors have, is a good experiment to have -- however typical adversarial examples may be quite structured and such a randomized strategy may not give an accurate indication about the nature of the representation.\n\nOverall, the paper is quite promising but I feel that one more iteration maybe needed.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work a "Seatbelt-VAE" algorithm to improve the robustness of VAE against adversarial attacks. The proposed method is promising but the paper appears to be hastily written and leave many places to improve and clarify. This paper can be turned into an excellent paper with another round of throughout modification. 



Decision: Reject