Year: 2021

Abstract:
Variational autoencoders (VAEs) have recently been shown to be vulnerable to adversarial attacks, wherein they are fooled into reconstructing a chosen target image. However, how to defend against such attacks remains an open problem. We make significant advances in addressing this issue by introducing methods for producing adversarially robust VAEs. Namely, we first demonstrate that methods proposed to obtain disentangled latent representations produce VAEs that are more robust to these attacks. However, this robustness comes at the cost of reducing the quality of the reconstructions. We ameliorate this by applying disentangling methods to hierarchical VAEs. The resulting models produce high–fidelity autoencoders that are also adversarially robust. We confirm their capabilities on several different datasets and with current state–of–the–art VAE adversarial attacks, and also show that they increase the robustness of downstream tasks to attack.

heading: 1 INTRODUCTION
text: Variational autoencoders (VAEs) are a powerful approach to learning deep generative models and probabilistic autoencoders (Kingma & Welling, 2014; Rezende et al., 2014). However, previous work has shown that they are vulnerable to adversarial attacks (Tabacof et al., 2016; Gondim-Ribeiro et al., 2018; Kos et al., 2018): an adversary attempts to fool the VAE to produce reconstructions similar to a chosen target by adding distortions to the original input, as shown in Fig 1. This kind of attack can be harmful when the encoder’s output is used downstream, as in Xu et al. (2017); Kusner et al. (2017); Theis et al. (2017); Townsend et al. (2019); Ha & Schmidhuber (2018); Higgins et al. (2017b). As VAEs are often themselves used to protect classifiers from adversarial attack (Schott et al., 2019; Ghosh et al., 2019), ensuring VAEs are robust to adversarial attack is an important endeavour.
Despite these vulnerabilities, little progress has been made in the literature on how to defend VAEs from such attacks. The aim of this paper is to investigate and introduce possible strategies for defence. We seek to defend VAEs in a manner that maintains reconstruction performance. Further, we are also interested in whether methods for defence increase the robustness of downstream tasks using VAEs.
Our first contribution is to show that regularising the variational objective during training can lead to more robust VAEs. Specifically, we leverage ideas from the disentanglement literature (Mathieu et al., 2019) to improve VAEs’ robustness by learning smoother, more stochastic representations that are less vulnerable to attack. In particular, we show that the total correlation (TC) term used to encourage independence between latents of the learned representations (Kim & Mnih, 2018; Chen et al., 2018; Esmaeili et al., 2019) also serves as an effective regulariser for learning robust VAEs.
Though a clear improvement over the standard VAE, a severe drawback of this approach is that the gains in robustness are coupled with drops in the reconstruction performance, due to the increased regularisation. Furthermore, we find that the achievable robustness with this approach can be limited (see Fig 1) and thus potentially insufficient for particularly sensitive tasks. To address this, we apply TC–regularisation to hierarchical VAEs. By using a richer latent space representation than a standard VAE, the resulting models are not only more robust still to adversarial attacks than single-layer models with TC regularisation, but can also provide reconstructions which are comparable to, and often even better than, the standard (unregularised, single-layer) VAE.
∗Equal Contribution. Contact at: mwilletts@turing.ac.uk; acamuto@turing.ac.uk
Published as a conference paper at ICLR 2021
Adversary Target 055 056 057 058 059 060 061 062 063 064 065 066 067 068 069 070 071 072 073 074 075 076 077 078 079 080 081 082 083 084 085 086 087 088 089 090 091 092 093 094 095 096 097 098 099 100 101 102 103 104 105 106 107 108 109 Improving VAEs’ Robustness to Adversarial Attack correlation penalty as being regularised VAEs. Leveraging the insight that regularised VAEs are robust to adversarial attacks, we develop a class of hierarchical VAEs that are more resilient still. Our model, Seatbelt-VAE, provides both robustness to adversarial attack, and higher quality reconstructions, relative to single layer regularised VAEs. We show that regularised hierarchical VAEs, without our proposed extensions, are not robust to adversarial attack. See Figure 1 for a demonstration of how adversarial attacks are highly effective on vanilla VAEs, less effective on regularised VAEs and close to ineffective on our proposed Seatbelt-VAE. Thus our key contributions are: • A demonstration that regularised VAEs, trained with an up-weighted total correlation, are significantly more robust to adversarial attacks than vanilla VAEs. • We introduce a hierarchical VAE, the Seatbelt-VAE, that provides further robustness to adversarial attack. • New connections between robustness, disentangling and adversarial attack, linked through regularisation. 2. Background 2.1. Variational Autoencoders Variational autoencoders (VAEs) are a deep extension of factor analysis suitable for high-dimensional data like images (Kingma & Welling, 2013; Rezende et al., 2014). They have a joint distribution over data x and latent variables z: p✓(x, z) = p✓(x|z)p(z) where p(z) = N (0, I) and p✓(x|z) is an appropriate distribution given the form of the data, the parameters of which are represented by deep nets with parameters ✓. As exact inference is intractable for this model, in a VAE we perform amortised stochastic variational inference. By introducing a variational posterior distribution over the latent variables q (z|x) = N (µ (x),⌃ (x)), we can perform gradient ascent on the evidence lower bound (ELBO) L(x) = DKL(q (z|x)||p✓(x, z)) = Eq (z|x) log p✓(x|z) DKL(q (z|x)||p(z))  log p(x) w.r.t.both ✓ and jointly, using the reparameterisation trick to take gradients through Monte Carlo samples from q (z|x). 2.2. Attacks on VAEs In an adversarial attack an agent is trying to manipulate the behaviour of some machine learning model towards a goal of their choosing. Commonly in deep learning this would be fooling a classifier to misclassify an image through adding a small perturbation (Akhtar & Mian, 2018; Gilmer et al., 2018). Very small changes in input, of little importance to the human eye, can produce large changes in the model’s (a) (b) (c)
Figure 1. Latent-space adversarial attacks on CelebA for different
models: a) Vanilla VAE b) -TCVAE c) our proposed Seatbelt-
VAE. Clockwise within each plot we show the initial input, its
reconstruction, the best adversarial input the adversary could pro-
duce, the adversarial distortion that was added to make the adver-
sarial input, the adversarial input’s reconstruction, and the target
image. We are trying to make the initial input (Hugh Jackman)
look like the target (Anna Wintour). You can see that the ad-
versarial reconstruction for the Vanilla VAE looks substantially like Wintour, indicating a successful attack. The -TCVAE adv. reconstruction does not look like Wintour, so the attack has not been successful, but it is not Jackman either. Our proposed model, Seatbelt-VAE, is sufficiently hard to attack that the output under attack still looks like Jackman, not Wintour. output. Attacks on VAEs have been proposed in Tabacof et al. (2016); Gondim-Ribeiro et al. (2018); Kos et al. (2018). The adversary wants draws from the model to be close to a target image when given a distorted image as input. See Figure 1.a) for an example of a successful attack on a vanilla VAE. Here we are trying to turn Hugh Jackman (Original, top left) into Anna Wintour (Target, bottom left). We can see that, by adding a well-chosen distortion (Distortion, bottom right), the reconstruction of Jackman goes from looking like a somewhat blurry version of the input (Original rec., top middle) to a somewhat blurry version of Wintour (Adversarial rec., bottom middle). The adversary has achieved their goal. The current most effective mode of attack on VAEs, the latent space attack (Tabacof et al., 2016; Gondim-Ribeiro
055 056 057 058 059 060 061 062 063 064 065 066 067 068 069 070 071 072 073 074 075 076 077 078 079 080 081 082 083 084 085 086 087 088 089 090 091 092 093 094 095 096 097 098 099 100 101 102 103 104 105 106 107 108 109
Improving VAEs’ Robustness to Adversarial Attack
correlation penalty as being regularised VAEs.
Leveraging the insight that regularised VAEs are robust to adversarial attacks, we develop a class of hierarchical VAEs that are more resilient still. Our model, Seatbelt-VAE, provides both robustness to adversarial attack, and higher quality reconstructions, relative to single layer regularised VAEs. We show that regularised hierarchical VAEs, without our proposed extensions, are not robust to adversarial attack.
See Figure 1 for a demonstration of how adversarial attacks are highly effective on vanilla VAEs, less effective on regularised VAEs and close to ineffective on our proposed Seatbelt-VAE.
Thus our key contributions are:
• A demonstration that regularised VAEs, trained with an up-weighted total correlation, are significantly more robust to adversarial attacks than vanilla VAEs. • We introduce a hierarchical VAE, the Seatbelt-VAE, that provides further robustness to adversarial attack. • New connections between ro stness, disentangling and adversarial attack, linked through regularisation.
2. Background 2.1. Variational Autoencoders
Variational autoencoders (VAEs) are a deep extension of factor analysis suitable for high-dimensional data like images (Kingma & Welling, 2013; Rezende et al., 2014). They have a joint distribution over data x and latent variables z: p✓(x, z) = p✓(x|z)p(z) where p(z) = N (0, I) and p✓(x|z) is an appropriate distribution given the form of the data, the parameters of which are represented by deep nets with parameters ✓. As exact inference is intractable for this model, in a VAE we perform amortised stochastic variational inference. By introducing a variational posterior distribution over the latent variables q (z|x) = N (µ (x),⌃ (x)), we can perform gradient ascent on the evidence lower bound (ELBO) L(x) = DKL(q (z|x)||p✓(x, z)) = Eq (z|x) log p✓(x|z) DKL(q (z|x)||p(z))  log p(x) w.r.t.both ✓ and jointly, using the reparameterisation trick to take gradients through Monte Carlo samples from q (z|x).
2.2. Attacks on VAEs
In an adversarial attack an agent is trying to manipulate the behaviour of some machine learning model towards a goal of their choosing. Commonly in deep learning this would be fooling a classifier to misclassify an image through adding a small perturbation (Akhtar & Mian, 2018; Gilmer et al., 2018). Very small changes in input, of little importance to the human eye, can produce large changes in the model’s
(a)
(b)
(c)
Figure 1. Latent-space adversarial attacks on CelebA for different models: a) Vanilla VAE b) -TCVAE c) our proposed SeatbeltVAE. Clockwise within each plot we show he initial input, its reconstruction, the best adversarial input the adversary could produce, the adversarial distorti n that was added to make the adversarial input, the adversarial input’s reconstruction, and the target image. We are trying to make the initial input (Hugh Jackman) look like the target (Anna Wintour). You can se that the adversarial reconstruction for the Vanilla VAE looks substantially like Wintour, indicating a successful attack. The -TCVAE adv. reconstruction does not look like Wintour, so the attack has not been successful, but it is not Jackman either. Our proposed model, Seatbelt-VAE, is sufficiently hard to attack that the output under attack still looks like Jackman, not Wint ur.
output.
Attacks on VAEs have been proposed in Tabacof et al. (2016); Gondim-Ribeiro et al. (2018); Kos et al. (2018). The adversary wants draws from the model to be close to a target image when given a distorted image as input.
See Figure 1.a) for an example of a successful attack on a vanilla VAE. Here we are trying to turn Hugh Jackman (Original, top left) into Anna Wintour (Target, bottom left). We can see that, by adding a well-chosen distortion (Distortion, bottom right), the reconstruction of Jackman goes from looking like a somewhat blurry version of the input (Original rec., top middle) to a somewhat blurry version of Wintour (Adversarial rec., bottom middle). The adversary has achieved their goal.
The current most effective mode of attack on VAEs, the latent space attack (Tabacof et al., 2016; Gondim-Ribeiro
055 056 057 058 059 060 061 062 063 064 065 066 067 068 069 070 071 072 073 074 075 076 077 078 079 080 081 082 083 084 085 086 087 088 089 090 091 092 093 094 095 096 097 098 099 100 101 102 103 104 105 106 107 108 109
Improvi g VAEs’ Robustness t A i l Attack correlation penalty as being regularised VAEs.
Leveraging the insight that regularised VAEs are robust to adversarial attacks, we develop a class of hierarchical VAEs that re more r silient still. Our model, Seatbelt-VAE, provides both robustness to adversarial attack, and higher quality reconstructions, relative to single layer regula ised VAEs. We show that regularised hierarch cal VAEs, without our proposed extensions, are not robust to adversarial att .
See Figure 1 for d monstration of how adversarial attacks are highly effective on vanilla s, less effective on regularised VAEs a d cl se to ineff ctive on our proposed Seatbelt-VAE.
Thus our key contributions are:
• A demonstration that regularised VAEs, trained with an up-weighted total correlation, are significa tly more r bust to adversarial attacks than vanilla VAEs. • We introduce a hierarchical VAE, the Seatbelt-VAE, that provides further robustness to adversarial attack. • New connections between robustness, disentangling and adversarial attack, linked through regularisation.
2. Background 2.1. Variational Autoencoders
Variational autoencoders (VAEs) are a deep extensi n of factor analysis suitable for high-dimensional data like images (Kingma & Welling, 2013; Rezende et al., 2014). They have a joint distribution over data x and latent variables z: p✓(x, z) = p✓(x|z)p(z) where p(z) = N (0, I) and p✓(x|z) is an appropriate distribution given the form of the data, the parameters of which are represented by deep nets with parameters ✓. As exact inference is intractable for this model, in a VAE we perform amortised stochastic variational inf rence. By introducing a variational posterior distribution over the latent variables q (z|x) = N (µ (x),⌃ (x)), we can perform gradient ascent on the evidence lower bound (ELBO) L(x) = DKL(q (z|x)||p✓(x, z)) = Eq (z|x) log p✓(x|z) DKL(q (z|x)||p(z))  log p(x) w.r.t.both ✓ and jointly, using the reparameterisation trick to take gradients through Monte Carlo samples from q (z|x).
2.2. Attacks on VAEs
In an adversarial attack an agent is trying to manipulate the behaviour of some machine learning model towards a goal of their choosing. Commonly in deep learning this would be fooling a classifier to misclassify an image through adding a small perturbation (Akhtar & Mian, 2018; Gilmer et al., 2018). Very small changes in input, of little importance to the human eye, can produce large changes in the model’s
(a)
(b)
(c)
Figure 1. Latent-space adversarial attacks on CelebA for different models: a) Vanilla VAE b) -TCVAE c) our proposed SeatbeltVAE. Clockwise within each plot we show the initial input, its reconstruction, the best adversarial input the adversary could produce, the adversarial distortion that was added to make the adversarial input, the adversarial input’s reconstruction, and the target image. We are trying to make the initial input (Hugh Jackman) look like the target (Anna Wintour). You can see that the adversarial reconstruction for the Vanilla VAE looks substantially like Wintour, indicating a successful attack. The -TCVAE adv. reconstruction does not look like Wintour, so the attack has not been successful, but t is not Jackman either. Our proposed model, Seatbelt-VAE, is sufficiently hard to attack that the output under attack still looks like Jackman, not Wintour.
output.
Attacks on VAEs have been proposed in Tabacof et al. (2016); Gondim-Ribeiro et al. (2018); Kos et al. (2018). The adversary wants draws from the model to be close to a target image when given a distorted image as input.
See Figure 1.a) for an example of a successful attack on a vanill VAE. Here we are trying to turn Hugh Jackman (Original, top left) int Anna Wintour (Target, bottom left). We can see that, by adding a well-chosen distortion (Distortion, bottom right), the reconstruction of Jackman goes from looking like a somewhat blurry version of the input (Original rec., top middle) to a somewhat blurry version of Wintour (Adversarial rec., bottom middle). The adversary has achieved their goal.
The current most effectiv mode of attack VAEs, the latent space attack (Tabacof et al., 2016; Gondim-Ribeiro 055 056 057 058 059 060 061 062 063 064 065 066 067 068 069 070 071 072 073 074 075 076 077 078 079 080 081 082 083 084 085 086 087 088 089 090 091 092 093 094 095 096 097 098 099 100 101 102 103 104 105 106 107 108 109
Improving VAEs’ Robustness to Adversarial Attack correlation penalty as being regularised VAEs.
Leveraging the i sight that regulari ed VAEs are robust to adversarial attacks, we dev lop a lass of hierarchical VAEs tha are more resili nt still. Our model, Seatbelt-VAE, provides both robustness to adversarial attack, and higher quality reconstructions, relativ to sin le lay r regularised VAEs. We how th t r gularised hierarchical VAEs, without our proposed exten ions, ar not robust to adversaria ttack.
See Figure 1 for a d monstration of how adversarial attacks ar hig ly effective on vanilla VAEs, less effective on regulari d VAEs and cl se to ineffective on our proposed Seatbelt-VAE.
Th s ur key contributions are:
• A demonst ation that regularised VAEs, trained with an up-weighted tot correlation, are significantly more robust to adversarial attacks than vanilla VAEs. • We introduce a hierarchical VAE, the Seatbelt-VAE, that provid further robustness to adversarial attack. • N w co nections between robustness, disentangling and adversarial attack, linked through regularisation.
2. Backgr und 2.1. Variational Autoencoders
Variational autoencoders (VAEs) are a deep extension of factor analysis suitable for high-dimensional data like images (Kingma & Welling, 2013; Rezende t al., 2 1 ). They have a joint distribution over data x and latent variables z: p✓(x, = p✓(x|z)p(z) where p(z) = N (0, I) and ✓(x|z) is an appropriate distribution given the form of the data, the parameters of which are repres nted by deep n t with parameters ✓. As exac inference is intractable for this model, in a VAE we perf rm amortised stochastic variational inference. By n r ducing a va iational p sterior distrib tion over the latent variables q (z|x) = N (µ (x),⌃ (x)), we can perform gradi nt ascent on the evidence lower bound (ELBO) L(x) = DKL(q (z|x)||p✓(x, z)) = Eq (z|x) log p✓ x z DKL(q (z|x)||p(z))  log p(x) w.r.t.both ✓ a d jointly, using the reparameterisation trick o take gradients thr ugh Monte Carlo samples from q (z|x).
2.2. Attacks on VAEs
In an dversarial attack an agent is trying to manipulate the beh viour of some achine le ning model towards a goal of their choosing. Commonly in deep learni g this would be fo ling a classifier to misclassify an image thr ugh adding a small pe turbatio (Akhtar & Mian, 2018; Gilmer t al., 2018). Very small changes in input, of little importance o the human eye, can produce large changes in the model’s
(a)
(b)
(c)
Figure 1. Latent-space adversarial attacks on CelebA for different m dels: a) Vanilla VAE b) -TCVAE c) our proposed SeatbeltVAE. Clockwise within each plot we show the initial input, its r construction, the best adversarial input the adversary could produce, the adversarial distorti n that was added to make the adversari l input, the adver arial input’s reconstruction, and the target image. W ar trying to make the initial input (Hugh Jackman) look like the ta get (Ann Wintour). You can see that the adversarial reconstruction for the Vanilla VAE looks substantially like Wintour, indica ing a successful attack. The -TCVAE adv. reconstruction does not l ok like Wintour, so the attack has not bee successful, but it is not Jackman either. Our proposed model, Seatbelt-VAE, is sufficiently ard to a tack that the output under att still lo ks l ke Jackman, not Wintour.
output.
Att cks on VAEs have been pr posed in Tabacof et al. (2016); Gondim-Ribeiro et al. (2018); Kos et al. (2018). The adversary wants draws from the model to be close to a target image when given d stor ed image as input.
See Figure 1.a) or an example of a successful attack on a vanill VAE. Here we ar trying to turn Hugh Jackman (Original, top l ft) i to Anna Wintour (Target, bottom left). We can see that, by adding a well-chosen distortion (Distortion, bott m right), the reconstruction of Jackman goes from looking like a somewhat blurry version of the input (Original rec., top middle) to a somewhat blurry version of Wintour (Adversarial r c , bottom middle). The adversary has achieved their goal.
The urr nt most effective m de of attack on VAEs, the latent space attack (Tabacof et al., 2016; Gondim-Ribeiro
5 56 57 58 59 0 1 2 3 4 5 66 67 68 69 0 1 2 3 4 5 76 77 78 79 0 1 2 3 4 5 86 87 88 89 0 1 2 3 4 5 096 097 098 099
0 1 2 3 4 5 106 107 108 109
Improving VAEs’ Robustness to Adve sarial Attack
correlation penalty as being regularised VAEs.
Leveraging the insight that regularised VAEs are robust to adversarial at acks, we develop lass of hierarchical VAEs that are more resilient still. O r m del, Se tbelt-VAE, provides both robustness to adversarial attack, and higher quality reconstructions, relative to single layer regularised VAEs. We show that regularised hierarchical VAEs, without our proposed extensions, are not robust to dv rsarial attack.
See Figure 1 for a demonstration of how adversarial attacks are highly effective on vanilla VAEs, less effective on regularis d VAEs and close to ineffective n our proposed Seatbelt-VAE.
Thus our key contributions re:
• A dem nstration that regularised VAEs, t ained with an up-weighted total corr lati n, are significa ly more robust to advers ri l attac s than vanilla VAEs. • We introduce a hierarchical V E, the Seatbelt-VAE, that provides further robustness to adversarial attack. • New connections between robustness, disentangling and adversarial attack, linked through regularisation.
2. Background 2.1. Variational Autoencoders
Variational autoencoders (VAEs) are a deep extension of factor analysis suitable for high-dimensional data like images (Kingma & Welling, 2013; Rezend et al., 2014). They hav a joint distribution over data x and latent variables z: p✓(x, z) = p✓(x|z)p(z) where p(z) = N (0, I) nd p✓(x|z) is an app opriate distri bution given the form of the d ta, th parameters f which are r presented by deep nets with parameters ✓. As exact inference is intractable for this model, in a VAE we perform amortised stochastic variation l inference. By introducing a variational posterior distribution over the latent variables q (z|x) = N (µ (x),⌃ (x)), we can perfo m gradient ascent on the evidenc lower bound (ELBO) L(x) = DKL(q (z|x)||p✓(x, z)) = Eq (z|x) log p✓(x|z) DKL(q (z|x)||p(z))  log p(x) w.r.t.both ✓ and jointly, using the reparameterisation trick to take gradients through Monte Carlo samples from q (z|x).
2.2. Attacks on VAEs
In an adv sa ial attack an agent is trying to manipulate the behaviour of some m chine learning model towards a goal of their choosing. Commonly in d ep learning this would be fooling a classifier to misclassify an image through adding a small perturbation (Akhtar & Mian, 2018; Gilmer et al., 2018). Very small changes in input, of little importance to the human eye, can produce large changes in the model’s
(a)
(b)
(c)
Figure 1. Lat nt-space adversarial attacks Ce ebA for diff rent models: a) Vanilla VAE b) -TCVAE c) our proposed S atbeltVAE. Clockwise wi hi each pl t we sh w th i itial input, its reco structio , the be t adversarial input the adversa y could pr - duce, the adversarial distortion that was d ed to make he adversarial input, the adversaria input’s reconstruction, nd the target image. We are trying to make the initial input (Hugh Jackman) look like the target (Anna Wintour). You can see that the adversarial reconstruction for the Vanilla VAE looks substantially like Wintour, indicating a successful attack. The -TCVAE adv. reconstruction does not look like Wintour, so the attack has not been successful but it is not Jackman either. Our proposed model, Seatbelt-VAE, is sufficiently hard to attack that the output under attack still looks like Jackman, not Wintour.
output.
Attacks on VAEs have been proposed in Tabacof et al. (2016); Gondim-Ribeiro et al. (2018); Kos et al. (2018). The adversary wants draws from the model to be close to a target image when given a distorted image as input.
See Figure 1.a) for an example of a successful attack on a vanilla VAE. Here we are trying to turn Hugh Jackman
, top left) into Anna Wintour (Target, bottom left). e can see that, by adding a well-chosen distortion (Distortion, bottom right), the reconstruction of Jackman goes from looking like a somewhat blurry version of the input (Original rec., top middle) to a somewhat blurry version of Wintour (Adversarial rec., bottom middle). The adversary has achieved their goal.
The current most effective mode of attack on VAEs, the latent space attack (Tabacof et al., 2016; Gondim-Ribeiro
055 056 057 058 059 060 061 062 063 064 065 066 067 068 069 070 071 072 073 074 075 076 077 078 079 080 081 082 083 084 085 086 087 088 089 090 091 092 093 094 095 096 097 098 099 100 101 102 103 104 105 106 107 108 109
Improving VAEs’ Robustness to Adversarial Attack
correlation penalty as being regularised VAEs.
Leveraging the insight that regularised VAEs are robust to adversarial attacks, we develop a class of hierarchical VAEs that are more resilient till. Our model, Seatbelt-VAE, provides both robustness to adversarial attack, and higher quality reconstructions, relative to single layer regularised VAEs. We show that regularised hierarchical VAEs, without our proposed extensions, are not robust to adversarial attack.
S e Figure 1 for a demon tration of how adv rsarial attacks are ighly effective on vanill VAEs, less effective on regularised VAEs nd close to i effective on our proposed Seatbelt-VAE.
Thus our key contributions are:
• A demonstration at eg larised VAEs, tra ned with an up-weighted total corr lation, are significa tly more robust to adversarial att cks than v nilla VAEs. • We introduce a hierarchical VAE, the Seatbelt-VAE, that provides further robustness to adversarial attack. • New connections between robustness, disentangling and adversarial attack, linked through regularisation.
2. Background 2.1. Variational Autoencoders
Variatio al autoencoders (VAEs) are a deep extension of factor analysis suitable for high-dimensional data like images (Kingma & Welling, 2013; Rezende et al., 2014). They have a joint distribution ov r data x and latent variables z: p✓(x, z) = p✓(x|z)p(z) wh re p(z) = N (0, I) and p✓(x|z) is an appropriate distribution given the form of the data, the pa ameters of which are represented y dee nets with parameters ✓. As xact inference is intr c ble for this model, in VAE we perform mortised stochastic varia al inference. By i roducing a variational o terio distribution ov the latent vari bles q (z|x) = N (µ (x),⌃ (x)), we can perform gradient ascent on he evidence lower bou d (ELBO) L(x) = DKL(q (z|x)||p✓( , z)) = Eq (z|x) log p✓(x|z) DKL(q (z|x)||p(z))  log p(x) w.r.t.both ✓ nd jointly, using the reparameterisation trick to take gradients through Monte Carlo samples from q (z|x).
2.2. Attacks on VAEs
In an adversarial attack an agent is trying to manipulate the behaviour of so e machine learning model towards a goal of their choosing. Commonly in deep learning this would be fooling a classifier to misclassify an image through adding a small perturbation (Akhtar & Mian, 2018; Gilmer et al., 2018). Very small changes in input, of little importance to the human eye, can produce large changes in the model’s
(a)
(b)
(c)
Figure 1. Late t-space adversarial attacks on CelebA for different m dels: a) Vanilla VAE b) -TCVAE c) our proposed SeatbeltVAE. Clockwise within each plot we show the initial input, its reconstructio , the best adversarial input the adversary could produce, the advers rial distortion that was added to make the adversarial input, the adversarial input’s reconstructio , and the target image. We are trying t make th initial input (Hugh Jackman) look like the targe (Anna Wintour). You can see that the adversarial reconstruction or the Vanilla VAE looks substantially like Wintour, indicat ng a successful attack. The -TCVAE adv. r c nstruct on d es not look like Wintour, so the attack has not been successful, but it is n t Jackman either. Our proposed model, Seatbelt-VAE, is sufficiently hard to attack that the output under attack still looks like Jackman, not Wintour.
output.
Attacks on VAEs h ve be n pr p sed in Tabac f et al. (2016); Gon im-Ribeiro et al. (2018); K et l. (2018). Th adv rsary w nts draws from the model to e close to target image when given a distorted image as input.
See Figure 1.a) for an example of a successful attack on a v nill VAE. Here we are tryin to turn Hugh Jackman (Original, top left) into Anna Wi tour (Target, bottom left). We can see that, by adding a well-chosen distortion (Distortion, bottom right), the reconstruction of Jackman goes from looking like a somewhat blurry version of the input (Original rec., op middle) to a somewh t blurry vers on of Wintour (Adversarial rec., bottom middle). The adversary has achieved their goal.
The current most effective mode of attack on VAEs, the latent space attack (Tabacof et al., 2016; Gondim-Ribeiro
5 56 57 58 59 0 1 2 3 4 5 66 67 68 69 0 1 2 3 4 5 76 77 78 79 0 1 2 3 4 5 86 87 88 89 0 1 2 3 4 5 096 097 098 099
0 1 2 3 4 5 106 107 108 109
Improving VAEs’ Robustness to Adversarial Attack
correlation penalty as being regularised VAEs.
Leveraging the i si ht that regulari ed VAEs are robust to adversarial at acks, we evelop l ss of hierarchi l VAEs that are more resilient till. O r m del, Se tbe t- AE, provides both robustness to adv sarial ttack, and higher quality reconstructions, relative to single layer regularised VAEs. We show that regularised hierarchical VAEs, without our proposed extensions, are not robust to adversarial attack.
Se Figure 1 for a demonstration of how adversarial attacks are high y effective on vanilla VAEs, less effective on regularis d VAEs and close to ineffective on our proposed Seatbelt-VAE.
Thus our k y contributions are:
• A demonstration that regularised VAEs, t ained with an up-weighted otal corr lati n, a significa ly more robust to a ve s ri l attac than vanilla VAEs. • We introduce a hierarchical VAE, the Seatbelt-VAE, that provides further robustness to adversarial attack. • Ne connectio s betwee robustness, disentangling and adversarial attack, linked through regularisation.
2. Background 2.1. Variational Autoencoders
Variational autoencoders (VAEs) ar a deep extension of factor analy is suitable for high-dimensional data like images (Kingma & Welling, 2013; Rezend et al., 2 14). Th y hav a joint distribution over data x and latent variables z: p✓(x, z) = p✓(x|z)p(z) where p(z) = N (0, I) nd p✓(x|z) is an app opriate distri bution given the form of the d ta, th parameters of which are r presented by deep nets with parameters ✓. As exact inference is intractable for this model, in a VAE we perform amortised stochastic variational inference. By introducing a variational posterior distrib tion over the latent variables q (z|x) = N (µ (x),⌃ (x)), we can perfo m gradient ascent on the evide l wer bound (ELBO) L(x) = DKL(q (z|x)||p✓(x, z)) = Eq (z|x) log p✓(x|z) DKL(q (z|x)||p(z))  log p(x) w.r.t.both ✓ a d jointly, using the reparameterisation trick to take gradients through Monte Carlo samples from q (z|x).
2.2. Attacks on VAEs
In an adv sa ial attack an agent is trying to man pulat he be viour of some m chine learning model towards a goal of their choosing. Commonly in d ep l arning this would be fooling a classifier to misclassify an image thr ugh adding a small perturbation (Akhtar & Mian, 2018; Gilmer t al., 2018). Very small changes in input, of little importance to the human eye, can produce large changes in the model’s
(a)
(b)
(c)
Figure 1. Latent-space adversarial attacks on CelebA for different models: a) Vanilla VAE b) -TCVAE c) our proposed SeatbeltVAE. Clockwise within each plot we show the initial input, its reconstruction, the best adversarial input the adver ary could produce, the adversarial distortion that was added to make the adversarial input, the adversarial input’s reconstruction, and the target image. We are trying to make the initial input (Hugh Jackman) look like the target (Anna Wintour). You can see that the adversarial reconstruction for the Vanill VAE looks substantially like Wintour, indicating a succ ssful ttack. The -TCVAE adv. reconstruction does n t look l ke Wi tour, so he attack has n t been successful, but it is not Jackman either. Our proposed model, Seatbelt-VAE, is sufficiently hard to attack that the output under attack still looks like Jackman, not Wintour.
output.
Attacks on VAEs have been proposed in Tabacof et al. (2016); Gondim-Ribeiro et al. (2018); Kos et al. (2018). The adversary wants draws from the model to be close to a target image when given a distorted image as input.
See Figur 1.a) for an example of a successful attack on a vanilla VAE. Here we are trying to t rn Hugh Jackman
, top left) i to Anna Wintour (Target, bottom left). e can see hat, by adding a well-chosen distortion (Distortion, bottom right), the reconstruction of Jackman goes from looking like a somewhat blurry version of the input (Original rec., top middle) to a somewhat blurry version of Wintour (Adversarial rec., bottom middle). The adversary has achieved their goal.
The current most effective m de of attack on VAEs, the latent space attack (Tabacof et al., 2016; Gondim-Ribeiro
5 56 57 58 59 0 1 2 3 4 5 66 67 68 69 0 1 2 3 4 5 76 77 78 79 0 1 2 3 4 5 86 87 88 89 0 1 2 3 4 5 096 097 098 099
0 1 2 3 4 5 106 107 108
Impr ving VAEs’ Robustnes to Adversarial Attack
correlation penalty as being regularised VAEs.
Leveraging the insight that regularised VAEs are robust to adversarial attacks, we develop lass of hierarchical VAEs that are more resilient still. O r m del, Se tbelt-VAE, provides both robustness to adversarial attack, and higher quality reconstructions, rel tive to single layer regul rised VAEs. We show that reg larised hierarchical VAEs, without our proposed extensi ns, are not robust to adversarial att ck.
See Fig re 1 for a demonstration of how dv rsari l attacks are highly effective on vani la VAEs, less eff c ive on regularis d VAE and close to i eff ctive on our propose Seatbelt-VAE.
Thus our key contributions are:
• A demonstration that regulari ed VAEs, t ained with an up-weighted total corr lati n, are significa tly more robust to advers ri l attac s than vanilla VAE . • We introduce a hierarchical VAE, the Seatbelt-VAE, hat provides further robustness to adversarial attack. • New connections between obustness, di ent ngling and adversarial ttack, linked through regularisation.
2. Background 2.1. Variational Autoencoders
Variatio al uto ncod rs (VAEs) ar d e extension of factor an lysis suitable for high-dimens onal data like images (Kingma & Welling, 2013; Rezende et al., 2014). They hav a joint distribution over data x and latent variabl s z: p✓(x, z) = p✓(x|z)p(z) where p(z) = N (0, I) and p✓(x|z) is an app opriate distri bution given the form of the d ta, the parameters of
hich are r presented b deep nets with parameters ✓. As exact inference is intra table for this model, in a VAE we perform amortised stochastic variational inference. By introducing a variational posterior distribution over the latent variables q (z|x) = N (µ (x),⌃ (x)), we can perfo m gradient ascent on the evidenc lower bound (ELBO) L(x) = DKL(q (z|x)||p✓(x, z)) = Eq (z|x) log p✓(x|z) DKL(q (z|x)||p(z))  log p(x) w.r.t.both ✓ and jointly, using the reparameterisation trick to take gradients through Monte Carlo samples from q (z|x).
2.2. Attacks on VAEs
In an adv sa ial attack an gent is trying to manipulate the behaviour of some m chine learning model towards a goal of their choosing. Commonly in d ep learning this would be fooling a classifier to misclassify an image through adding a small perturbation (Akhtar & Mian, 2018; Gilmer et al., 2018). Very small changes in input, of little importance to the human eye, can produce large changes in the model’s
(a)
(b)
(c)
Figure 1. Latent-space adversarial attacks CelebA for differe t models: a) Vanilla VAE b) -TCVAE c) our proposed SeatbeltVAE. Clockwise within each pl t we sh w th i itial input, its reconstruction, the best adversarial input the adversary could produce, the adversarial distortion that was added to make he adversarial input, the adversarial input’s reconstruct on, nd the target image. We are trying to m ke the initial input (Hugh Jackman) look like the targe (Anna Wintour). You can see that the adversarial reconstruction for the Vanilla VAE looks substantially like Wint ur, indicating a successful attack. The -TCVAE adv. econstruction does ot lo k like Wintour, so the att k has not been succe sful, but it is not Jackman either. Our proposed model, Seatbelt-VAE, is sufficiently hard to attack that the output under attack still looks like Jackman, not Wintour.
output.
Attacks on VAEs have been proposed in Tabacof et al. (2016); Gondim-Ribeiro et al. (2018); Kos et al. (2018). The adversary wants draws from the model to be close to a target image when given a distorted image as input.
See Figur 1.a) for an example of a successful attack on a vanilla VAE. Here we are trying to turn Hugh Jackman
, top left) into Anna Wintour (Target, bottom left). e can see that, by adding a well-chosen distortion (Distortion, bottom right), the reconstruction of Jackman goes from looking like a somewhat blurry version of the input (Original rec., top middle) to a somewhat blurry version of Wintour (Adversarial rec., bottom middle). The adversary has achieved their goal.
The current most effective mode of attack on VAEs, the latent space attack (Tabacof et al., 2016; Gondim-Ribeiro
55 56 57 58 59 0 1 62 63 64 65 66 67 68 69 0 1 72 73 74 75 76 77 78 79 0 1 82 83 84 85 86 87 88 89 0 1 092 093 094 095 096 097 098 099
0 1 102 103 104 105 106 107 108 109
Improving VAEs’ Robustness to Adversarial Attack
correlation penalty as b ing regularised VAEs.
Leveraging the insight that regularised VAEs are robust to adversarial att cks, we develop class of hierarc i al VAEs that re more r ilient still. Our model, Seatbelt-VAE, provides both robustness to advers rial attack, nd higher qualit reconstructions, rel tive to single layer regularised VAEs. We show tha regularised hierarchical VAEs, without our proposed extensions, are not robust to adversarial attack.
See Fig e 1 for d m n ration of how dversarial attacks are highly effectiv on vani la s, less eff c ive on regularised VAE a d cl s t ineffective on o r proposed Seatbelt-VAE.
Thus our key contributions are:
• A demonstration t t regularised VAEs, trai d with an up-weighted total correlation, are significantly more r bust to adversarial attacks than vanilla VAEs. • We introduce a hierarchical VAE, the Seatbelt-VAE, that provides further robustness t adversarial att ck. • New con ections between obustness, di ent ngling and adversarial ttack, linked through regu arisation.
2. Background 2.1. V ri tion l Autoencoders
Variational autoencoders (VAEs) are a deep ext nsi n of factor analysis suitable for high-dim nsional dat like images (Kingma & W lling, 2013; Rezende et al., 2014). They h e a joint di tribution ov r dat x and latent variables z: p✓(x, z) = p✓(x|z)p(z) where p(z) = N (0, I) and p✓(x|z) is an appro riate distribution given the form of the ata, the parameters of
hich are represented b deep nets with parameters ✓. As exact inferenc is intractable for t is model, in a VAE we perform amortised stochastic variational inf rence. By introducing a variational posterior distribution over the latent variables q (z|x) = N (µ (x),⌃ (x)), we can perform gradient ascent on the evidence lower bound (ELBO) L(x) = DKL(q (z|x)||p✓(x, z)) = Eq (z|x) log p✓(x|z) DKL(q (z|x)||p(z))  log p(x) w.r.t.both ✓ and jointly, using the reparameterisation trick to take gradients through Monte Carlo samples from q (z|x).
2.2. Attacks on VAEs
In an adversarial attack an agent is trying to manipulate the behaviour of some machin learning model towards a goal of their choosing. Commonly in deep learning this would be fooling a classifier to misclassify an image through adding a small perturbation (Akhtar & Mian, 2018; Gilmer et al., 2018). Very small changes in input, of little importance to the human eye, can produce large changes in the model’s
(a)
(b)
(c)
Figure 1. Lat nt-space adv rsarial attacks o CelebA for different mod ls: a) Vanill VAE b) -TCVAE c) o r proposed S atbeltVAE. Clockwise within each plot we how t initial input, its r tr ti , th b st dversarial input the advers ry could pro d ce, the dversarial dist rtion th t was added t m ke the adversarial put, the adversarial inp t’ reconstru tion, and the target image. We a e trying to make the initial input (Hugh Jackman) look like the target (Anna Wintour). You can see that the adversarial reconstruction for t e Va illa VAE looks s bstantially like Wint ur, indicating a uccessful attack. The -TCVAE dv. reconstruction does not lo k like Wintour, so the att ck has not been succe sful, but it is not Jackman either. Our proposed model, Seatbelt-VAE, is sufficiently hard to attack that the output under attack still looks like Jackman, not Wintour.
output.
Attacks n VAEs have been proposed in Tabacof et al. (2016); Gondim-Ribeiro et al. (2018); Ko et al. (2018). The adversary wa ts draws from th m del to be close to a target image when given a distorte image as input.
See Figur 1.a) for n example of a succ ssful attack a vanill VAE. He e we are trying to turn Hugh Jackman (Original, top left) int Anna Wintour (Target, bottom left). We can see that, by adding a well-chosen distortion (Distortion, bottom right), t e reconstruction of Jackman goes from l oking like a s mewh blurry version of the input (Original rec., top middle) to a somewhat blurry version of Wintour (Adversarial rec., bottom middle). The adversary has achieved their goal.
The current most effectiv mode of attack VAEs, the latent sp ce attack (Tabacof et al., 2016; Gondim-Ribeiro
5 56 57 58 59 0 1 2 3 4 5 66 67 68 69 0 1 2 3 4 5 76 77 78 79 0 1 2 3 4 5 86 87 88 89 0 1 2 3 4 5 096 097 098 099
0 1 2 3 4 5 106 107 108 109
Improving VAEs’ Robustness to Adversarial Attack
correlation penalty as being regularised VAEs.
Leveraging the i sight that r l VAEs are robust to adversarial att cks, we dev lop lass of hierarchical VAEs tha are more resili nt till. O r m del, Se tbelt-VAE, provides bot robustne s to adversarial attack, and higher quality reconstructions, relativ to single layer regularised VAEs. W how th t regularised hierarchical VAEs, without our proposed extensions, are not robust to adversarial attack.
Se Fig re 1 for a demonstration of how dversarial attacks a e hig ly effectiv on vani a s, less eff c ive on regulari d VAE and cl e to i ffe tive on our proposed Sea belt-VAE.
Th s ur key contributions are:
• A dem nst ation that regularised VAEs, t ained with an up-weighted total corr lati n, are significa tly more robust to advers ri l attac s than vanilla VAEs. • We introduce a hierarchical VAE, the Se tbelt-VAE, that provid further robustness to adversari l attack. • N w co nect ons between obustness, di ent ngling and adversarial attack, linked through regularisation.
2. Background 2.1. Variatio al Autoencod rs
Variational auto ncoders (VAEs) are a deep extension of factor analysis suitable for high-dimensional data like images (Kingma & Welling, 2013; Reze d t al., 2014). They h v a joint distribution over data x and l tent variables z: p✓(x, = p✓(x|z)p(z) where p(z) N (0, I) nd p✓(x|z) is an app o riate distri bution given the form of the d ta, th parameters f
hich are r pres nted b de p n ts with parameters ✓. As exac inferenc is intr ctable f r this model, in a VAE we perf rm amortised stochastic variational inference. By n r ducing a va iational p sterior distrib tion over the latent variables q (z|x) = N (µ (x),⌃ (x)), we can perfo m gradi nt ascent on the evide c lower bound (ELBO) L(x) = DKL(q (z|x)||p✓(x, z)) = Eq (z|x) log p✓ x z DKL(q (z|x)||p(z))  log p(x) w.r.t.both ✓ a d jointly, using the reparameterisation trick o take gradients thr ugh Monte Carlo samples from q (z|x).
2.2. Attacks on VAEs
In an adv sa ial attack an agent is trying to manipulat the beh viour of some chine le ning model towards a goal of their choosing. Commonly in d ep learning this would be fo ling a classifier to misclassify an image thr ugh adding a small pe turbat o (Akhtar & Mian, 2018; Gilmer et al., 2018). Very small changes in input, of little importance o the human eye, can produce large changes in the model’s
(a)
(b)
(c)
Figure 1. Latent-space adversari l at acks on CelebA for different mod ls: a) Vanilla VAE b) -TCVAE c) ur propose Seat ltVAE. Clockwise withi e ch plot we show the initial input, its r construction, the best adver arial input the adversary could produce, the adversari l distorti n that was added to make the adversari l input, the adv r arial input’s reconstruction, and the target image. W a trying to mak th initi l input (Hugh Jackman) look like th t g t (Ann Wi tour). Y u can see that the adversarial reconstruction for the Vanilla VAE looks substantially like Wint ur, indica a succe sful attack. The -TCVAE adv. reconstruc ion does not l k like Wintour, so the att ck has not been succe sful, but i is not Jackman either. Our proposed model, Seatbelt-VAE, is sufficiently ard to a tack that the output under att still lo ks l ke Jackman, not Wintour.
output.
Att cks on VAE have been pr posed in Tabacof et al. (2016); G ndim-Ribeiro et a . (2018); Kos et al. (2018). The adversa wants draws from the odel to be close to a target image when given d st r ed image as input.
See Figure 1.a) or an example of a successful attack on a vanill VAE. Here we are trying to turn Hugh Jackman
i i l, top left) i to Anna Wintour (Target, bottom left). e can see that, by adding a well-chosen distortion (Distortion, bott m right), the reconstruction of Jackman goes from looking like a somewhat blurry version of the input (Original rec., top middle) to a somewhat blurry version of Wintour (Adversarial r c , bottom middle). The adversary has achieved their goal.
The urrent most effective m de of attack on VAEs, the latent space attack (Tabacof et al., 2016; Gondim-Ribeiro
a a es
a
r e o st a e t o
r s l
e
av a
p
h
. n e a . he ,
econs c , u , a
, u c , .
.
, s . . ,
, . , ,
, .
e
a a r
o
o t
on a
VA E
βTC
VA E
Se at
be lt-
VA E
Adversarial Input
Figure 1: Adversarial attacks on CelebA for different models. Here we start with the image of Hugh Jackman and introduce an adversary that tries to produce reconstructions that look like Anna Wintour. This is done by applying a distortion (third column) to the original image to produce an adversarial input (second column). We can see that th adversarial reconstruction for the Vanilla VAE looks s bstantially like Wintour, indicating a successful attack. Adding a regularisation term using the β-TCVAE produc s an adversarial reconstruction that does not look like Wintour, but it is also far from a successful reconstruction. The hierarchical version of a β-TCVAE (which we call Seatbelt-VAE) is sufficiently hard to attack that the output under attack s ill looks like Jackman, not Wintour.
To summarise: We provide insights into what makes VAEs vulnerable to attack and how we might go about defending them. We unearth novel connections between disentanglement and adversarial robustness. We demonstrate that regularised VAEs, trained with an up-weighted total correlation, are much more robust to attacks than vanilla VAEs. Building on this we develop regularised hierarchical VAEs that are more robustness still and offer improved reconstructions. Finally, we show that robustness to adversarial attack also confers increased robustness to downstream tasks.

heading: 2 BACKGROUND: ATTACKING VAES
text: In adversarial attacks an agent is trying to manipulate the ehaviour of some model towards a goal of their choosing (Akhtar & Mian, 2018; Gilmer et al., 2018). For many deep learning models, very small changes in the input can produce large changes in output. Attacks on VAEs have been proposed where the adv rsary looks to apply small input distortions that produce reconstructions close to a target adversarial image (Tabacof et al., 2016; Gondi -Ribeiro et al., 2018; Kos et al., 2018). An exa ple is hown in Fig 1: a standard VAE is successfully attacked, turning Jackman into Wintour.
Unlike mor e t blished adversarial settings, only a small number of such VAE attacks have been u gested in the lite ature. The current k own most effective mode of attack is a latent space attack (Tabac f et al., 2016; Gondim-Ribeiro et al., 2018; Kos et al., 2018). This aims to find a distorted imag x∗ = x + d such tha its po terior qφ(z|x∗) i close to that f the agent’s chosen target image qφ(z|xt) under s me metric. This then implies that the likelihood pθ(xt|z) is high when given draws from the post rior of the adv rsarial example. It is particularly important to be robust to this attack if o is conce ned with using the encoder netwo k f a VAE s part of a downstream task. For a VAE with a s ngle stochastic layer, the latent-space adversarial objective is
∆r(x,d,x t;λ) = r(qφ(z|x + d), qφ(z|xt)) + λ||d||2, (1)
where r(·, ·) is some divergence or distance, commonly a DKL(Tabacof et al., 2016; Gondim-Ribeiro et al., 2018). e are pen lising the L2 norm of d too, so as to aim for attacks that change the image less. We can hen simply optimise to find a good distortion d.
Alte ativ ly, we can aim to directly increase the ELBO for the target datapoint (Kos et al., 2018):
∆output(x,d,x t;λ) =Eqφ(z|x+d)
[ log(xt|z) ] −DKL(qφ(z|x + d)||p(z)) + λ||d||2. (2)

heading: 3 DEFENDING VAES
text: This pr ble was not con id red by prior works1. To address it, we first need to consider what makes VAEs vulnerable to adversarial attacks. We argue that two key factors dictate whether we c n rform a successful ttack on a VAE: a) whether we can induce significant changes in the encoding distribution qφ(z|x) through only small changes in the data x, and b) whether we can induce signific nt changes i the reco structed imag s through only small changes to the latents z. The first of the e relates to th smooth ess of the encoder mapping, the latter to the smoothness of the decoder mapping.
1We note that the earliest version of this work appeared in June 2019 (Willetts et al., 2019), here extended. Since then other works, eg Camuto et al. (2020); Cemgil et al. (2020); Barrett et al. (2021), have built of our own to consider this problem of VAE robustness, including investigating it from a more theoretical standpoint.
2
Consider, for the sake of argument, the case where the encoder–decoder process is almost completely noiseless. Here successful reconstruction places no direct pressure for similar encodings to correspond to similar images: given sufficiently powerful networks, very small changes to embeddings z can imply very large changes to the reconstructed image; there is no ambiguity in the “correct” encoding of a particular datapoint. In essence, we can have lookup–table style behaviour – nearby realisations of z do not necessarily relate to each other and very different images can have very similar encodings.
This will now be very vulnerable to adversarial attacks: small input changes can lead to large changes in the encoding, and small encoding changes can lead to large changes in the reconstruction. It will also tend to overfit and have gaps in the aggregate posterior, qφ(z) = 1N ∑N n=1 qφ(z|xn), as each qφ(z|xn) will be sharply peaked. These gaps can then be exploited by an adversary. There are two mechanisms by which we can reduce this lookup-table behaviour, thereby reducing gaps in the aggregate posterior. First, we can try to regulate the level of noise in the per-datapoint posterior covariance, to then obtain smoothness in the overall embeddings. Having a stochastic encoding creates uncertainty in the latent that gives rise to a particular image, forcing similar latents to correspond to similar images. Adding noise forces the VAE to smooth the encode-decode process in that similar images will lead to similar embeddings in the latent space, ensuring that small changes in the input result in small changes in the latent space and result in small changes in the decoded outputs. This proportional input-output change is what we refer to as a ‘simple’ encode-decode process, which is the second mechanism that can reduce look-up table behaviour.
The fact that the VAE is vulnerable to adversarial attack suggests that its standard setup does not obtain sufficiently smooth and simple representations to provide an adequate defence. Introducing additional regularisation to enforce simplicity or increased posterior covariance thus provides a prospect for defending VAEs. We could attempt to obtain this by direct regularisation of the networks (e.g. weight decay). Here, however, we focus on macro-level regularisation approaches as discussed in the next section. The reason for this is that controlling the macroscopic behaviour of the networks through low-level regularisations can be difficult to control and, in particular, difficult to calibrate. Further, as the most effective attack on VAEs currently attack the latent space, it is reasonable that regularisation methods that directly act on the properties of the latent space form a good place to start.

heading: 3.1 DISENTANGLING METHODS AND ROBUSTNESS
text: Recent research into disentangling VAEs (Higgins et al., 2017a; Siddharth et al., 2017; Kim & Mnih, 2018; Chen et al., 2018; Esmaeili et al., 2019; Mathieu et al., 2019) and the information bottleneck (Alemi et al., 2017; 2018) has looked to regularise the ELBO with the hope of providing more interpretable embeddings. These regularisers also have influences on the smoothness and stochasticity of the embeddings learned.
Of particular relevance, Mathieu et al. (2019) introduce the notion of overlap in the embedding of a VAE: the level of overlap between per-datapoint posteriors as they combine to form the aggregate posterior. Controlling this is critical to achieving a smoothly varying latent embedding. Overlap encapsulates both the level of uncertainty in the encoding process and also a locality of this uncertainty. To learn a smooth representation we not only need our encoder distribution to have an appropriate entropy, we also want the different possible encodings to be similar to each other. Critically, Mathieu et al. (2019) show that many methods proposed for disentangling, and in particular the β-VAE (Higgins et al., 2017a; Alemi et al., 2017), provide a mechanism for directly controlling this overlap.
Going back to our previous arguments, we see that controlling this overlap may also provide a mechanism for improving VAEs’ robustness. This observation now hints at an interesting question: can we use methods initially proposed to encourage disentanglement to encourage robustness?
It is important to note here that disentangling can be difficult to achieve in practice, typically requiring precise choices in the hyperparameters of the model and the weighting of the added regularisation term, and often also a fair degree of luck (Locatello et al., 2019; Mathieu et al., 2019; Rolinek et al., 2019). As such, we are not suggesting to induce disentangled representations to induces robustness, or indeed that disentangled representations should be any more robust. Rather, as highlighted above, we are interested in whether the regularisers traditionally used to encourage disentanglement reliably lead to adversarially robust VAEs. Indeed, we will find that though our approaches—based on these regularisers—provide reliable and significant improvements in robustness, these improvements are not generally due to any noticeable improvements in disentanglement itself (see Appendix E.1).
Regularising for Robustness There are a number of different disentanglement methods that one might consider using to train robust VAEs. Perhaps the simplest would be to use a β-VAE (Higgins et al., 2017a), wherein we up-weight the DKL term in the VAE’s ELBO by a factor β ≥ 1. However, as mentioned previously the β-VAE only increases overlap at the expense of substantial reductions in reconstruction quality as the data likelihood term has, in effect, been down-weighted (Kim & Mnih, 2018; Chen et al., 2018; Mathieu et al., 2019).
Because of these shortfalls, we instead propose to regularise through penalisation of a total correlation (TC) term (Kim & Mnih, 2018; Chen et al., 2018). As discussed in Section A.1, this looks to directly force independence across the different latent dimensions in aggregate posterior qφ(z), such that the aggregate posterior factorises across dimensions. This approach has been shown to have a smaller deleterious effect on reconstruction quality than found in β-VAEs (Chen et al., 2018). As seen in Fig 2 this method also gives greater overlap by increasing posterior variance. To summarise, the greater overlap and the lesser degradation of reconstruction quality induced by β-TCVAE make them highly suitable for our purposes.

heading: 3.2 ADVERSARIAL ATTACKS ON TC-PENALISED VAES
text: We now consider attacking these TC-penalised VAEs and demonstrate one of the key contributions of the paper: that empirically this form of regularisation makes adversarial attacks on VAEs harder to carry out. To do this, we first train them under the β-TCVAE objective (i.e. Eq (15)), jointly optimising θ, φ for a given β. Once trained, we then attack the models using the latent-space attack method outlined in Section 2, finding an input distortion d that minimises the latent attack loss ∆ as per Eq (1) with r(·, ·) = DKL(·||·).
One possible metric for how successful such attacks have been is the achieved value reached of the attack loss ∆KL. If the latent space distributions for the original input and for the distorted input match closely for a small distortion, then ∆KL is small and the model has been successfully fooled – reconstructions from samples from the attacked posterior would be indistinguishable from those from the target posterior. Meanwhile, the larger the converged value of the attack loss the less similar these distributions are and the more different the reconstructed image is to the adversarial target image.
We carry our these attacks for dSprites (Matthey et al., 2017), Chairs (Aubry et al., 2014) and 3D faces (Paysan et al., 2009), for a range of β and λ values. We pick values of λ following standard methodology (Tabacof et al., 2016; Gondim-Ribeiro et al., 2018), and use L-BFGS-B for gradient descent (Byrd et al., 1995). We also varied the dimensionality of the latent space of the model, dz, but found it had little effect on the effectiveness of the attack.
In Fig 3 we show the effect on the attack loss ∆KL for varying β, averaged over different original input-target pairs and values of dz. Note that the plot is logarithmic in the loss. We see a clear pattern for each dataset that the loss values reached by the adversary increases as we increase β from the standard VAE (i.e. β = 1). This analysis is also borne out by visual inspection of the effectiveness of these attacks, for example as shown in Fig 1. We will return to give further experimental results in Section 5. An interesting aspect of Fig 3 is that in many cases the adversarial loss starts to decrease if β is too large: as β increases there is less pressure in the objective to produce good reconstructions.

heading: 4 HIERARCHICAL TC–PENALISED VAES
text: We are now armed with the fact that penalising the TC in the ELBO induces robustness in VAEs. However, TC-penalisation in single layer VAEs comes at the expense of model reconstruction quality (Chen et al., 2018), albeit less than that in β-VAEs. Our aim is to develop a model that is robust to adversarial attack while mitigating this trade-off between robustness and sample quality. To achieve this, we now consider instead using hierarchical VAEs (Rezende et al., 2014; Sønderby et al., 2016; Kingma et al., 2016; Zhao et al., 2017; Maaløe et al., 2019; Vahdat & Kautz, 2020; Child, 2021). These are known for their superior modelling capabilities and more accurate reconstructions. As these gains stem from using more complex hierarchical latent spaces, rather than less noisy encoders, this suggests they may be able to produce better reconstructions and generative capabilities, while also remaining robust to adversarial attacks when appropriately regularised.
The simplest hierarchical extension of conditional stochastic variables in the generative model is the Deep Latent Gaussian Model (DLGM) of Rezende et al. (2014). Here the forward model factorises as a chain, pθ(x,~z) = pθ(x|z1) ∏L−1 i=1 pθ(z
i|zi+1)p(zL), where each pθ(zi|zi+1) is a Gaussian distribution with mean and variance parameterised by deep nets, while p(zL) is an isotropic Gaussian. Unfortunately, we found that naively applying TC-correlation penalisation to DLGM-style VAEs did not confer the improved robustness we observed in single layer VAEs. We postulate that this observed weakness is inherent to the structure of chain factorisation in the generative model. This means that the data-likelihood depends solely on z1, the bottom-most latent variable, and attackers only need to manipulate z1 to produce a successful attack.
To account for this, we instead use a generative model in which the likelihood pθ(x|~z) depends on all the latent variables in the chain ~z, rather than just the bottom layer z1, as has been done in Kingma et al. (2016); Maaløe et al. (2019). This leads to the following factorisation of the generative structure:
pθ(x,~z) = pθ(x|~z) ∏L−1
i=1 pθ(z
i|zi+1)p(zL). (3)
To construct the ELBO, we must further introduce an inference network qφ(~z|x). On the basis of simplicity and that it produces effective empirical performance, we use the factorisation:
qφ(~z|x) = qφ(z1|x) ∏L−1
i=1 qφ(z
i+1|zi,x), (4)
where each conditional distribution qφ(zi+1|zi,x) takes the form of a Gaussian. Again, marginalising out intermediate zi layers, qφ(zL|x) is a non-Gaussian, highly flexible distribution. To defend this model against adversarial attack, we apply TC regularisation term as per the last section. We refer to the resulting models as Seatbelt-VAEs. We obtain a decomposition of the ELBO for this model, revealing the existence of a TC term for the top-most layer (see Appendix B for proof).
Theorem 1. The Evidence Lower Bound, for a hierarchical VAE with forward model as in Eq (3) and amortised variational posterior as in Eq (4), can be decomposed to reveal the total correlation (see Definition A.1), of the aggregate posterior of the top-most layer of latent variables:
L(θ, φ;D) = Eq(~z,x) log pθ(x|~z) + R + Sa + Sb −DKL ( q(zL)|| ∏ j q(zLj ) ) , (5)
where the last term is the required TC term, and, using j to index over the coordinates in zL, R = ∫ dx ∏L
i=1 (dzi)qφ(~z|x)q(x) log
∏L−1 k=1 pθ(z
k|zk+1) qφ(z1|x) ∏L−2 m=1 qφ(z m+1|zm,x) (6)
Sa =− Eqφ(zL−1))DKL(qφ(zL,x|zL−1)||qφ(zL)q(x)) (7) Sb =− ∑
j DKL(qφ(z
L j )||p(zLj )). (8)
In other words, following the Factor and β-TCVAEs, we up-weight the TC term for zL. We can upweight this term then recombine the decomposed parts of the ELBO, to give us the following compact form of this objective.
Definition 1. A Seatbelt-VAE is a hierarchical VAE with forward model as in Eq (3) and amortised variational posterior as in Eq (4), trained wrt its parameters θ, φ to maximise the objective:
LSeatbelt(θ, φ;β,D) :=Eqφ(~z,x) [ log pθ(x,~z)
qφ(~z|x)
] − (β − 1)DKL ( q(zL)|| ∏ j q(zLj ) ) . (9)
We see that, when L = 1, a Seatbelt-VAE reduces to a β-TCVAE. We use the β = 1 case as a baseline in our experiments as it corresponds to a Vanilla VAE for L = 1 and for L > 1, β = 1 it produces a hierarchical model with a likelihood function conditioned on all latents.
As with the β-TCVAE, training LSeatbeltθ,φ;β,D using stochastic gradient ascent with minibatches of the data is complicated by the presence of aggregate posteriors qφ(z) which depend on the entire dataset. To deal with this, Appendix C we derive a minibatch estimator for TC-penalised hierarchical VAEs, building off that used for β-TCVAEs (Chen et al., 2018). We note that, as in Chen et al. (2018), large batch sizes are generally required to provide accurate TC estimates.
Attacking Hierarchical TC–Penalised VAEs In the above hierarchical model the likelihood over data is conditioned on all layers, so manipulations to any layer have the potential to be significant. We focus on simultaneously attacking all layers, noting that, as shown in Appendix D, this is more effective that just targeting the top or base layers individually. Hence our adversarial objective for latent-space attacks on Seatbelt-VAEs is the following generalisation of that introduced in Tabacof et al. (2016); Gondim-Ribeiro et al. (2018); Kos et al. (2018), to attack all the layers at the same time:
∆Seatbeltr (x,d,x t;λ) = λ||d||2 + ∑L i=1 r(qφ(z i|x + d), qφ(zi|xt)). (10)

heading: 5 EXPERIMENTS
text: Expanding on the brief experiments in Section 3.2, we perform a battery of adversarial attacks on each of the introduced models. We do this for three different adversarial attacks: first (as in Section 3.2) a latent attack, Eqs (1,10) using the DKL divergence between attacked and target posteriors; secondly, we attack via the model’s output, aiming to make the target maximally likely under the attacked model as in Eq (2); finally, a new latent attack method as per Eqs (1,10) where we use r(·, ·) = W2(·, ·), the 2-Wasserstein distance between attacked and target posteriors. We then evaluate the effectiveness of these attacks in three ways. First, like Fig 1, we can plot the attacks themselves, to see how effective these attacks are in fooling us. Secondly, we can measure the adversary’s loss under the attack objective. Thirdly, we give the negative adversarial likelihood of the target image xt given an attacked latent representation z∗. Larger, more positive, values of − log pθ(xt|z∗) correspond to less successful attacks as they correspond to large distances between the target and the adversarial reconstruction. Lower values correspond to successful attacks as they correspond to a small distance between the adversarial target and the reconstruction. We also measure
reconstruction quality of these models, as a function of degree of regularisation. Finally, we also measure how downstream tasks that use output of these models perform under attack. We train classifiers, on the reconstructions and on the latent representations, and see how robust performance is when the upstream VAE is attacked.
We demonstrate that hierarchical TC–Penalised VAEs (Seatbelt-VAEs) confer superior robustness to β-TCVAEs and standard VAEs, while preserving the ability to reconstruct inputs effectively. Through this, we demonstrate that they are a powerful tool for learning robust deep generative models.
Following previous work (Tabacof et al., 2016; Gondim-Ribeiro et al., 2018) we randomly sample 10 input-target pairs for each dataset and for each image pair we consider 50 different values of λ geometrically-distributed from 2−20 to 220. Thus each individual trained model undergoes 500 attacks for each attack mode. As before, we used L-BFGS-B for gradient descent (Byrd et al., 1995). We perform these experiments on Chairs (Aubry et al., 2014), 3D faces (Paysan et al., 2009), and CelebA (Liu et al., 2015). Details of neural architectures and training are given in Appendix G.

heading: 5.1 VISUAL APPRAISAL OF ATTACKS
text: We first visually appraise the effectiveness of attacks that use the DKL divergence on vanilla VAEs, β-TCVAEs, and Seatbelt-VAEs. As mentioned in Section 1, Fig 1 shows the results of latent space attacks on three models trained on CelebA. It is apparent that the β-TCVAE provides additional resilience to the attacks compared with the standard VAE. Furthermore, this figure shows that SeatbeltVAEs are sufficiently robust to almost completely thwart the adversary: its adversarial construction still resembles the original input. Moreover, this was achieved while also producing a clearer non– adversarial reconstruction. One might expect attacks targeting a single generative factor underpinning the data to be easier. However, we find that these models protect effectively against this as well. For example, see Fig 4 for plots showing an attacker attempting to rotate a dSprites heart.
In both figures we follow the method of Gondim-Ribeiro et al. (2018) to plot attacks. Those shown are representative of the adversarial inputs the attacker was able to find over the 50 different values of λ. The Seatbelt-VAE input only undergoes a small perturbation because it is sufficiently robust that the attacker is not able to make the reconstruction look more like the target image in any meaningful way, such that the optimiser never drifts far from the initial input. Note that the β-TCVAE is also robust here. The attacker is unable to induce the desired adversarial reconstruction, even though the attack may be of large magnitude. In contrast, attacks on vanilla-VAEs are able to move through the latent space and find a perturbation that reconstructs to the adversary’s target image.

heading: 5.2 QUANTITATIVE ANALYSIS OF ROBUSTNESS
text: Having ascertained perceptually that Seatbelt-VAEs offer the strongest protection to adversarial attack, we now demonstrate this quantitatively. Fig 5 shows − log pθ(xt|z∗) and ∆ over a range of datasets and βs for Seatbelt-VAEs (L = 4) and β-TCVAEs for our three different attacks. It demonstrates that the combination of depth and high TC-penalisation offers the best protection to
adversarial attacks and that the hierarchical extension confers much greater protection to adversarial attack than a single layer β-TCVAE. As we go to the largest values of β for both Chairs and 3D Faces, adversarial loss ∆KL grows by a factor of≈ 107 and− log pθ(xt|z∗) for those attacks doubles for Seatbelt-VAE. For all attacks, TC-penalised models outperformed standard VAEs (β=1) and Seatbelt-VAEs outperform single-layer VAEs. β-TCVAEs do not experience such a large uptick in adversarial loss and negative adversarial likelihood. These results show that the hierarchical approach can offer very strong protection from the adversarial attacks studied.
In Appendix D we provide plots detailing these metrics for a range of L values. In Appendix E we also calculate the L2 distance between target images and adversarial outputs and show that the loss of effectiveness of adversarial attacks is not due to the degradation of reconstruction quality from increasing β. We also test VAE robustness to random noise. We noise the inputs and evaluate the model’s ability to reconstruct the original input. Through this we are evaluating their ability to denoise. See Appendix F for an illustration of this for TC-penalised models. It is plausible that the ability of these models’ to denoise is linked to their robustness to attacks.
ELBO and Reconstructions Though Seatbelt-VAEs offer better protection to adversarial attack than β-TCVAEs, we also motivate their utility by way of their reconstruction quality. In Fig 6 we plot the ELBO of the two TC-penalised models, calculated without the β penalisation that was applied during training. We further show the effect of depth and TC-penalisation on CelebA reconstructions. These plots show that Seatbelt-VAEs’ reconstructions are more resilient to increasing β than β-TCVAEs’.

heading: 5.3 PROTECTION TO DOWNSTREAM TASKS
text: Finally, we consider the protection that Seatbelt-VAEs might provide to downstream tasks, noting that VAEs are often used as subcomponents in larger ML systems (Higgins et al., 2017b), or as a mechanism to protect another model from attack (Schott et al., 2019; Ghosh et al., 2019). Table 1 shows results for classification tasks using 2-layer MLPs and fully-convolutional nets trained on the reconstructions or on the embeddings. It shows the drop in accuracy caused by an adversary that picks a target with a different label and attacks the VAEs’ embedding using the attack objective with λ = 1. We see that Seatbelt-VAEs produced significantly better accuracies under these attacks.

heading: 6 CONCLUSION
text: We have shown that VAEs can be rendered more robust to adversarial attacks by regularising the evidence lower bound. This increase in robustness can be strengthened by extending these regularisation methods to hierarchical VAEs, forming Seatbelt-VAEs, which uses a generative structure where the likelihood makes use of all the latent variables. Designing robust VAEs is becoming pressing as they are increasingly deployed as subcomponents in larger pipelines. As we have shown, methods typically used for disentangling, motivated by their ability to provide interpretable representations, also confer robustness. Studying the beneficial effects of these methods is starting to come to the fore of VAE research.

heading: ACKNOWLEDGEMENTS
text: This research was directly funded by the Alan Turing Institute under Engineering and Physical Sciences Research Council (EPSRC) grant EP/N510129/1. MW was supported by EPSRC grant EP/G03706X/1. AC was supported by an EPSRC Studentship. SR gratefully acknowledges support from the UK Royal Academy of Engineering and the Oxford-Man Institute. CH was supported by the Medical Research Council, the Engineering and Physical Sciences Research Council, Health Data Research UK, and the Li Ka Shing Foundation
We thank Tomas Lazauskas, Jim Madge and Oscar Giles from the Alan Turing Institute’s Research Engineering team for their help and support.

heading: B TOTAL-CORRELATION DECOMPOSITION OF ELBO
text: Proof of Theorem 1
Here we prove that the ELBO for a hierarchical VAE with forward model as in Eq (3) and amortised variational posterior as in Eq (4) can be decomposed to reveal a total-correlation in the top-most latent variable.
Specifically, now considering the ELBO for the whole dataset and using q(x) to indicate the empirical data distribution, we will obtain, denoting z0 = x:
L (θ, φ;D) = Eqφ(~z,x) [log pθ(x|~z)]− Eqφ(~z|x)q(x) [ L−1∑ i=1 DKL(qφ(z i|zi−1,x)||pθ(zi|zi+1)) ] − Eqφ(zL−1)DKL(qφ(zL,x|zL−1)||qφ(zL)q(x))
− ∑
j DKL(qφ(z
L j )||p(zLj ))− βDKL ( qφ(z L)|| ∏
j qφ(z
L j ) )
(16)
We start with the forms of p and q given in Theorem 1. The likelihood is conditioned on all z layers: pθ(x|~z).
L(θ, φ;D) =Eqφ(~z,x) log pθ(x,~z)
qφ(~z,x) (17)
=Eqφ(~z,x) [ log pθ(x|~z)]− Eq(x) [DKL(qφ(~z,x)||pθ(~z))] (18)
=Eq(~z,x) log pθ(x|~z)− Eq(x) log q(x) + Eq(~z,x) log pθ(~z)
q(~z|x) (19)
=Eq(~z,x) log pθ(x|~z) +H(q(x)) (20)
+ ∫ dx dz1 L∏ i=2 (dziqφ(z i|zi−1,x))qφ(z1|x)q(x) log
p(zL) ∏L−1 k=1 pθ(z
k|zk+1) qφ(z1|x) ∏L−1 m=1 qφ(z
m+1|zm,x)︸ ︷︷ ︸ W
So here we have three terms: an expectation over the data likelihood, the entropy of the empirical data distribution (a constant) and W . We now can expand W to a term involving the prior for the latent zL and a term involving the conditional distributions from the generative model for the remaining components of ~z:
W = ∫ dx L∏ i=1 (dzi)qφ(~z|x)q(x) log ∏L−1 k=1 pθ(z k|zk+1) qφ(z1|x) ∏L−2 m=1 qφ(z
m+1|zm,x)︸ ︷︷ ︸ R
+ ∫ dx L∏ i=1 (dzi)qφ(~z|x)q(x) log p(zL)
qφ(zL|zL−1,x)︸ ︷︷ ︸ S
(21)
The first part R , it that part of W not involving the prior for ‘top-most’ latent variable zL, is the first subject of our attention. We split out the part of R involving the generative and posterior terms for the latent variable closest to the data, z1 and the rest:
R = ∫ dx L∏ i=1 (dzi)qφ(~z|x)q(x) log pθ(z 1|z2) qφ(z1|x)︸ ︷︷ ︸
Ra
+ L−1∑ m=2 ∫ dx L∏ i=1 (dzi)qφ(~z|x)q(x) log pθ(z m|zm+1) qφ(zm|zm−1,x)︸ ︷︷ ︸
Rb
.
The first of these terms Ra is an expectation over a DKL:
Ra = −Eqφ(z2,x)DKL(qφ(z1|x)||pθ(z1|z2)). (22)
And the rest, Rb , provides the DKL divergences in the ELBO for all latent variables other than zL
and z1. It reduces to a sum of expectations over DKL divergences, one per latent variable.
Rb = L−1∑ m=2 ∫ dx L∏ i=1 (dzi)qφ(z 1|x)q(x) L−1∏ k=1,6=m (qφ(z k+1|zk,x))qφ(zm|zm−1,x) log pθ(z m|zm+1)
qφ(zm|zm−1,x) (23)
=− L−1∑ m=2 ∫ dx L∏ i=1 (dzi)qφ(z 1|x)q(x) L−1∏ k=1,6=m (qφ(z k+1|zk,x))DKL(qφ(zm|zm−1,x)||pθ(zm|zm+1))
(24)
=− L−1∑ m=2 Eqφ(zm+1,zm−1,x)DKL(qφ(z m|zm−1,x)||pθ(zm|zm+1)). (25)
Now we have:
L(θ, φ;D) =Eq(~z,x) log pθ(x|~z) +H(q(x)) + Ra + Rb + S (26)
We wish to apply TC decomposition to the top-most latent variable zL. S is an expectation over the DKL divergence between qφ(zL|zL−1,x) and p(zL)
S = −Eqφ(zL−1,x)DKL(qφ(zL|zL−1,x)||p(zL)) (27)
Applying the decomposition, with j indexes over units in zL. S =− Eqφ(zL,zL−1,x) [ log qφ(z L|zL−1,x)− log p(zL) + log qφ(zL)
− log qφ(zL) + log ∏ j qφ(z L j )− log ∏ j qφ(z L j ) ]
=− Eqφ(zL,zL−1,x) [ log
qφ(z L|zL−1,x) qφ(zL)
] − Eqφ(zL) [ log qφ(z L)∏
j qφ(z L j )
]
− Eqφ(zL) [ log ∏ j qφ(z L j )
p(zL)
]
=− Eqφ(zL,zL−1,x) [ log
qφ(z L|zL−1,x)q(x) qφ(zL)q(x)
] − Eqφ(zL) [ log qφ(z L)∏
j qφ(z L j )
]
− ∑ j Eqφ(zL)
[ log qφ(z L j )
p(zLj ) ] =−Eqφ(zL−1))DKL(qφ(zL,x|zL−1)||qφ(zL)q(x))︸ ︷︷ ︸
Sa − ∑ j
DKL(qφ(z L j )||p(zLj ))︸ ︷︷ ︸
Sb
−DKL(qφ(zL)|| ∏ j
qφ(z L j ))︸ ︷︷ ︸
Sc Where we have used p(zL) = ∏ j p(z L j ) for our chosen generative model, a product of independent unit-variance Gaussian distributions.
L(θ, φ;D) = Eq(~z,x) log pθ(x|~z) +H(q(x)) + Ra + Rb + Sa + Sb + Sc (28)
Giving us a decomposition of the evidence lower bound that reveals the TC-term in zL, as required. Multiplying this with a chosen pre-factor β gives us the required form.

heading: C MINIBATCH WEIGHTED SAMPLING
text: As in Chen et al. (2018), applying β-TC decomposition requires us to calculate terms of the form:
Eqφ(zi) log qφ(z i) (29)
The i = 1 case is covered in the appendix of Chen et al. (2018). First we will repeat the argument for i = 1 as made in Chen et al. (2018), but in our notation, and then we cover the case i > 1 for models with factorisation of qφ(~z|x) of Seatbelt VAEs.
C.1 MWS FOR β-TCVAES
We denote BM = {x1,x2, ...,xM}, a minibatch of datapoints drawn uniformly iid from q(x) = 1/N ∑N n=1 δ(x − xn). For any minibatch we have p(BM ) = 1N
M . Chen et al. (2018) introduce r(BM |x), the probability of a sampled minibatch given that one member is x and the remaining M − 1 points are sampled iid from q(x), so r(BM |x) = 1N M−1.
Eqφ(z1) log qφ(z 1) =Eqφ(z1,x) [ logEq(x) [qφ(z 1|x)]] (30)
=Eqφ(z1,x) [ logEp(BM ) [ 1
M M∑ m=1 qφ(z 1|xm)]] (31)
≥Eqφ(z1,x) [ logEr(BM |x) [ p(BM ) r(BM |x) 1 M M∑ m=1 qφ(z 1|xm)]] (32)
=Eqφ(z1,x) [ logEr(BM |x) [ 1
NM M∑ m=1 qφ(z 1|xm)]] (33)
(34)
So then during training, one samples a minibatch {x1,x2, ...,xM} and can estimate Eqφ(z1) log qφ(z1) as:
Eqφ(z1) log qφ(z 1) ≈ 1
M M∑ i=1 [ log M∑ j=1 qφ(z 1 i |xj)− logNM ] (35)
and z1i is a sample from qφ(z 1|xi).
C.2 MINIBATCH WEIGHTED SAMPLING FOR SEATBELT-VAES Here we have that q(~z,x) = ∏L l=2 [qφ(z
l|zl−1,x)]qφ(z1|x)q(x). Now instead of having a minibatch of datapoints, we have a minibatch of draws of zi−1: Bi−1M = {zi−11 , zi−12 , ..., zi−1M }. Each member of which is the result of sequentially sampling along a chain, starting with some particular datapoint xm ∼ q(x).
For i > 2, members of Bi−1M are drawn:
zi−1j ∼ qφ(zi−1|zi−2j ,xj) (36)
and for i = 2:
z1j ∼ qφ(z1|xj) (37)
Thus each member of this batch Bi−1M is the descendant of a particular datapoint that was sampled in an iid minibatch BM as defined above. We similarly define r(Bi−1M |zi−1,x) as the probability of selecting a particular minibatch Bi−1M of these values out from our set {(xn, zi−1n )} (of cardinality N ) given that we have selected into our minibatch one particular pair of values (x, zi−1) from these N values. Like above, r(Bi−1M |zi−1,x) = 1N M−1
Now we can consider Eqφ(zi) log qφ(zi) for i > 1:
Eqφ(zi) log qφ(z i) =Eqφ(zi,zi−1,x) [ logEqφ(zi−1,x) [qφ(z
i|zi−1,x)]] (38)
=Eqφ(zi,zi−1,x) [ logEp(Bi−1M ) [ 1
M M∑ m=1 qφ(z i|zi−1m ,xm)]] (39)
≥Eqφ(zi,zi−1,x) [ logEr(Bi−1M |zi−1,x) [ p(Bi−1M )
r(Bi−1M |zi−1,x) 1 M M∑ m=1 qφ(z i|zi−1m ,xm)]]
(40)
=Eqφ(zi,zi−1,x) [ logEr(Bi−1M |zi−1,x) [ 1
NM M∑ m=1 qφ(z i|zi−1m ,xm)]] (41)
Where we have followed the same steps as in the previous subsection.
During training, one samples a minibatch {zi−11 , zi−12 , ..., zi−1M }, where each is constructed by sampling ancestrally. Then one can estimate Eqφ(zi) log qφ(zi) as:
Eqφ(zi) log qφ(z i) ≈ 1
M M∑ k=1 [ log M∑ j=1 qφ(z i k|zi−1j ,xj)− logNM ] (42)
and zik is a sample from qφ(z i|zi−1k ,xk). In our approach we only need terms of this form for i = L, so we have:
Eqφ(zL) log qφ(z L) ≈ 1
M M∑ k=1 [ log M∑ j=1 qφ(z L k |zL−1j ,xj)− logNM ] (43)
and zLk is a sample from qφ(z L|zL−1k ,xk).

heading: D SEATBELT-VAE RESULTS
text: D.1 SEATBELT-VAE LAYERWISE ATTACKS
D.2 SEATBELT-VAE ATTACKS BY MODEL DEPTH AND β
E AGGREGATE ANALYSIS OF ADVERSARIAL ATTACK
5 10
3.5
4.0
4.5
5.0
5.5
6.0 6.5 Ta rg et -R ec on D is ta nc e
5 10
8
10
12
14
16
18
A dv
er sa
ria l-T
ar ge
t D is
ta nc
e
Latent Space Attack Output Attack
(a) dSprites Distances
1 2 4 6 8 10
10 4
10 5
10 6
10 7
A dv
er sa
ria l L
os s
Latent
1 2 4 6 8 10
1.3
1.5
1.7
1.9
1e3 Output
(b) dSprites Losses
5 10 5.0
5.5
6.0
6.5
7.0
Ta rg
et -R
ec on
D is
ta nc
e
5 10 6
8
10
12
14
16
A dv
er sa
ria l-T
ar ge
t D is
ta nc
e
Latent Space Attack Output Attack
(c) Chairs Distances
1 2 4 6 8 10
10 3
10 4
10 5
10 6
10 7
A dv
er sa
ria l L
os s
Latent
1 2 4 6 8 10
6.0
6.5
7.0
7.5 1e2 Output
(d) Chairs Losses
2.5 5.0 7.5 10.0
2.6
2.8
3.0
3.2
3.4
Ta rg
et -R
ec on
D is
ta nc
e
2.5 5.0 7.5 10.0
4
5
6
7
A dv
er sa
ria l-T
ar ge
t D is
ta nc
e
Latent Space Attack Output Attack
(e) 3D Faces Distances
1 2 4 6 8 10
10 3
10 4
A dv
er sa
ria l L
os s
Latent
1 2 4 6 8 10
1.55
1.60
1.65
1.70
1e3 Output
(f) 3D Faces Losses
Figure E.10: Plots showing the effect of varying β in a β-TCVAE trained on dSprites (a,b), Chairs (c,d), and 3D Faces (d,e) on: the L2 distance from the adversarial target xt to its reconstruction when given as input (target-recon distance) and the L2 distance between the adversarial input x∗ and xt (adversarial-target distance); and the adversarial objectives ∆. We also include these metrics for “output” attacks Gondim-Ribeiro et al. (2018), which we find to be generally less effective. In such attacks the attacker directly tries to reduce the L2 distance between the reconstructed output and the target image. For latent attacks the adversarial-target L2 distance grows more rapidly than the target-recon distance (i.e the degradation of reconstruction quality) as we increase β. This effect is much less clear for output attacks. This makes it apparent that the robustness we see in β-TCVAE to latent space adversarial attacks is not due the degradation in reconstruction quality we see as β increases. It is also apparent that increasing β increases the adversarial loss for latent attacks and output attacks.
E.1 DISENTANGLING AND ROBUSTNESS? Although we are using regularisation methods that were initially proposed to encourage disentangled representations, we are interested here in their effect on robustness not whether the representations we learn are in fact disentangled. This is not least due to the questions that have arisen about the hyperparameter tuning required for disentangled representations Locatello et al. (2019); Rolinek et al. (2019). For us the β pre-factor is just the degree of regularisation imposed.
However, it may be of interest to see what relationship, if any, exists between the ease of attacking of a model and how disentangled it is. Here we show the MIG score (Chen et al., 2018) against the achieved adversarial loss on the Faces data for β-TCVAEs. MIG measures the degree to which representations are disentangled and larger adversarial losses correspond to a less successful attack. Shading is over the range of β and dz values. There does not seem to be any simple correspondence between increased MIG and increases in adversarial loss, indicative of a less successful attack.
0.05 0.10 0.15 0.20 MIG
2000
3000
4000
5000
6000
Ad ve
rs ar
ia l L
os s
(a) Faces
0.05 0.10 0.15 0.20 MIG
101 102 103 104 105 106 107 Ad ve rs ar ia l L os s
(b) Chairs
Figure E.11: Adversarial attack loss reached vs MIG score for β-TCVAEs trained on Faces and Chairs presented for a range of β = {1, 2, 4, 6, 8, 10} and dz = {8, 32} values.

heading: F ROBUSTNESS TO NOISE
text: 4000 2000 0 =1
0.000
0.002 0.004 chairs chairs + 0.01 chairs + 0.1 chairs +
4000 2000 0 =2
0.000
0.002
0.004
4000 2000 0 =4
0.000
0.002
0.004
4000 2000 0 =6
0.000
0.002
0.004
4000 2000 0 =8
0.000
0.002
0.004
4000 2000 0 =10
0.000
0.002
0.004
(a) Chairs β-TCVAE log pθ(x|z)
4000 3000 2000 1000 0 =1
0.000
0.002 0.004 chairs chairs + 0.01 chairs + 0.1 chairs +
4000 3000 2000 1000 0 =2
0.000
0.002
0.004
4000 3000 2000 1000 0 =4
0.000
0.002
0.004
4000 3000 2000 1000 0 =6
0.000
0.002
0.004
4000 3000 2000 1000 0 =8
0.000
0.002
0.004
4000 3000 2000 1000 0 =10
0.000
0.002
0.004
(b) Chairs Seatbelt-VAE log pθ(x|z)
Figure F.12: Here we measure the robustness of both β-TCVAE and Seatbelt-VAE when Gaussian noise is added to Chairs. Within each plot a range of β values are shown. We evaluate each model’s ability to decode a noisy embedding to the original non-noised data x by measuring the distribution of log pθ(x|z) when z ∼ qφ(z|x + a ) (a some scaling factor taking values in {0.1, 0.5, 1} and ∼ N (0, 1)) for which higher values indicate better denoising. We show these likelihood values as density plots for the β-TCVAE in (a) and for the Seatbelt-VAE with L = 4 in (b), taking β ∈ {1, 2, 4, 6, 8, 10}. Note the axis scalings are different for each subplot. We see that for both models using β > 1 produces autoencoders that are better at denoising their inputs. Namely, the mean of the density, i.e. Eqφ(z|x+ ) [ log pθ(x|z)], shifts dramatically to higher values for β > 1 relative to β = 1. In other words, for both these models, the likelihood of the dataset in the noisy setting is much closer to the non-noisy dataset when β > 1 across all noise scales (0.1 , 0.5 , ).
G IMPLEMENTATION DETAILS All runs were done on the Azure cloud system on NC6 GPU machines.
G.1 ENCODER AND DECODER ARCHITECTURES We used the same convolutional network architectures as Chen et al. (2018). For the encoders of all our models (q(·|x)) we used purely convolutional networks with 5 convolutional layers. When training on single-channel (binary/greyscale) datasets such as dSprites, 3D Faces, or Chairs the 5 layers took the following number of filters in order: {32, 32, 64, 64, 512}. For more complex RGB datasets, such as CelebA, the layers had the following number of filters in order: {64, 64, 128, 128, 512}. The mean and variance of the amortised posteriors are the output of dense layers acting on the output of the purely convolutional network, where the number of neurons in these layers is equal to the dimensionality of the latent space Z . Similarly, for the decoders (p(x|z)) of all our models we also used purely convolutional networks with 6 deconvolutional layers. When training on single-channel (binary/greyscale) datasets, dSprites, 3D Faces, or Chairs, the 6 layers took the following number of filters in order: {512, 64, 64, 32, 32, 1}. For CelebA the layers had the following number of filters in order: {512, 128, 128, 64, 64, 3}. The mean of the likelihood p(x|·) was directly encoded by the final de-convolutional layer. The variance of the decoder, σ, was fixed to 0.1.
For β-TCVAE the range of dz values used was {4, 6, 8, 16, 32, 64, 128}. For Seatbelt-VAEs the number of units in each layer zi decreases sequentially. There is a list z sizes for each dataset, and for a model of L layers that the last L entries to give dz,i, i ∈ {1, ..., L}.
{dz}dSprites ={96, 48, 24, 12, 6} (44) {dz}Chairs ={96, 48, 24, 12, 6} (45) {dz}3DFaces ={96, 48, 24, 12, 6} (46) {dz}CelebA ={256, 128, 64, 32} (47)
For Seatbelt-VAEs we also have the mappings qφ(zi+1|zi,x) and pθ(zi|zi+1). These are amortised as MLPs with 2 hidden layers with batchnorm and Leaky-ReLU activation. The dimensionality of the hidden layers also decreases as a function of layer index i:
dh(qφ(z i+1|zi,x)) = hsizes[i] (48)
dh(pθ(z i|zi+1)) = hsizes[i] (49)
hsizes = [1024, 512, 256, 128, 64] (50)
To train the model we used ADAM Kingma & Lei Ba (2015) with default parameters, a cosine decaying learning rate of 0.001, and a batch size of 1024. All data was pre-processed to fall on the interval -1 to 1. CelebA and Chairs were both downsampled and cropped as in Chen et al. (2018) and Kulkarni et al. (2015) respectively. We find that using free-bits regularisation (Kingma et al., 2016) greatly ameliorates the optimisation challenges associated with DLGMs.

Review summaries:
This work builds on the vulnerability of VAEs to adversarial attacks to propose investigate how training with alternative losses may alleviate this problem, with a specific focus on disentanglement. In particular it is found that disentanglement constraints may improve the robustness to adversarial attacks, to the detriment of the performance. In order to get the best of both, the author(s) propose a more flexible (hierarchical) model, trained with the beta-TC penalization on the ELBO. The algorithm, named Seatbelt-VAE, shows improvement over the beta-TC VAE in terms of reconstruction, as well as in term of adversarial robustness for several datasets (Chairs, 3D Faces, dSprites).