The Variational InfoMax AutoEncoder
Authors: Vinenzo Crescimanna
Bruce Graham
Venue:  (2020)
Keywords: autoencoder
information theory
infomax
vae

Abstract: We propose the Variational InfoMax AutoEncoder (VIMAE), an autoencoder based on a new learning principle for unsupervised models: the Capacity-Constrained InfoMax, which allows the learning of a disentangled representation while maintaining optimal generative performance. The variational capacity of an autoencoder is defined and we investigate its role. We associate the two main properties of a Variational AutoEncoder (VAE), generation quality and disentangled representation, to two different information concepts, respectively Mutual Information and network capacity. We deduce that a small capacity autoencoder tends to learn a more robust and disentangled representation than a high capacity one. This observation is confirmed by the computational experiments.

Under review as a conference paper at ICLR 2020
THE VARIATIONAL INFOMAX AUTOENCODER
Anonymous authors
Paper under double-blind review
ABSTRACT
The Variational AutoEncoder (VAE) can learn simultancously a representation and generative model, but often, in order to generate sharper samples, useless representation is learnt. Starting from the observation that a meaningful represen- tation is an informative one, in this work we analyse the VAE from an information theoretic perspective. We define the concept of capacity for the variational net- work, and we associate the two tasks of VAE: generation and representation qual- ity. to two information properties of the network: information of the generative model and capacity of the network. Then we suggest that an optimal generative model is the one optimising the Capacity-Constraiin retical objective learning the maximal informative generative model while main- taining bounded the network capacity. The theoretical assumptions are confirmed by the computational experiments, where between the different families of VAE, the one optimising variational lower bound of the CCIM generates sharper sam- ples and learns a clustered and robust representation. We call this the Variational InfoMax AutoEncoder (VIMAE).
InfoMax (CCIM), theo-
INTRODUCTION
A common assumption in machine learning is that any visible data x E X' is completely described by some generative factor o, living in smaller hidden space O i.e. x g(o) with g a (possibly stochastic) generative function. The aim of unsupervised representation learning research is to find a representation 2 of the generative factor o living in a known space describing, as well as o, the visible data x. This is particularly relevant because the learnt small representation z is task agnostic and, in principle, can be used as input for networks performing different tasks, leading to faster and more robust learning (generalisation property). (Rifai al. 2011).
Many models fo X + Z trying to learn such representations have been proposed (Dinh et al, 2016 Hinton et al. 2006 Maddison et al. 2017: Radford et al. 2015), but recently in order to solve this problem it was proposed to consider a dual problem: define a priori 2 and find a generator map go. such that for any 2, go g0(2) is an element of X. In particular, two families of probabilistic generative models have become dominant: Variational AutoEncoder (VAE) (Kingma & Welling 2013 Rezende et al., 2014) and Generative Adversarial Network (GAN) (Goodfellow al. 2014). The common idea of the two approaches is that good generator pe (z|22 is the one able to generate the data that as close as possible to the visible one, i.c. that with respect certain metric D, the distance between the marginal Po (xx = (Po (z)] and the visible distribution PD is minimal.
In this manuscript we restrict our attention the VAE model, since by its architecture, it is the only one where the learnt representation, possibly from different datasets, can be used as input for networks performing different tasks, (Achille al. 2018 Ramapuram al., 2017). Although VAE, by its training robustness and general good generative performance is the most popular model for representation learning, in particular cases it suffers from the uninformative representation issue: the representation is entangled and the generative model tends to be independent of z, i.c. pe(x]2) se po(x). As highlighted in the next section such behaviour is intrinsic in the variational loss, the Evidence Lower BOund (ELBO), encouraging a less informative representation.
Following the direction suggested in (Alemi et al. 2017 Zhao et al. 2017) we describe the VAE from an information theoretic perspective. Such description lead us to two observations: the WAE (Tolstikhin et al.) (017) and InfoVAE (Zhao et al. (017)) models are actually maximising a lower bound of the mutual information associated generator po px(x))) belonging in certain family

---

Under review as a conference paper at ICLR 2020
P. the Variational InfoMax (VIM); and given that the capacity of the network is function of the entropy of the prior p(z), the VIM is the variational expression of the Capacity-Constern Constraindd InfoMax (CCIM). Following the analogies between the CCIM and the Information Bottleneck (IB), the theoretical principle associated with the ELBO, we deduce that the representation quality is associated to the capacity term.
The theoretical arguments are confirmed by the performed experiments where we observe that, dif- ferently from what was argued in previous works (Higgins al.) 2017 Burgess et al., 2018). is possible to train a model that is able to learn good (able to generalise) representations while main- taining optimal generative performance. The main contributions of the paper are summarised in the following points:
derivation of a variational lower bound for the maximal mutual information of generaive model belonging in a certain family, see equation]] 7
definition and bounds estimation for the network capacity for variational autoencoder,
equation 9
association of the two main properties of VAE, generation quality and good representa- tion, to two different information concepts, respectively Mutual Information and network capacity:
proposal of new learning principle for unsupervised models: the Capacity- Conetty-oonttrained InfoMax, see equation T0 that allows both to learn a good representation while maintaining optimal generative performance.
The work is divided follows: in the second section describe briefly the VAE and its variants; in the third and fourth sections we describe the variational infomax method and related work. We conclude the paper with the experimental results and the final observations.
BACKGROUND
The aim this section is to describe VAE, understand principal issues of the ELBO objective and describe the three most relevant approaches overcome such issues.
2.1 NOTATION AND PRELIMINARY DEFINITIONS
We use calligraphic letters (i.e. X) for sets, capital letters (i.e. X) for random variables, and lower case letters (i.e. .) for their samples. With abuse of notation we denote both the probability and the corresponding density with the lower case letters (i.e. p(x)).
KL divergence Given two random distributions p(x) and q(x), the Kullback-Leibler (KL) diver- gence
((3) ((v)
DkL DrI(p(=))l(())) - log p(y)dy
is an (intuitive) measure of the distance between the distributions and g.
Mutual Information and Capacity Given channel + X with and Z random variables, jointly distributed according to p(z, , z) and with marginals p(z) and p(z). The mutual information
[(x, Z) (xI(p(a, =))))))))())
measure of the reduction of uncertainty in X due to the knowledge of Z, and the capacity
C(X, = sup I(X,2)
p(=)EP
the maximal information that can be shared for fixed generator p(x(z).
2

---

Under review as a conference paper at ICLR 2020
2.2 VARIATIONAL AUTOENCODER
From now on let us assume that the unknown distribution of the data p(æ) coincides with the em- pirical one (2r), and that the distribution of the latent representation p(=) ) known. In this text the VAE is a model solving the following optimisation problem: find the generative model pa(x,2) 2) Pa. specified by the parameters of the associated neural network, maximising the
ELBO objective
ELBOa = (oool--DD((o + ploc)))(
(go
a lower bound of the unfeasible-to -oocooptll marginal likelihood Eptw) [lo po())) The ELBO ob- jective is optimised by regularised autoencoder, with encoder and decoder parametetrising, spectively, the inference and generative distributions, (s(2/z) and po(x)zz, with ø E Φ, 0 E θ and regulariser defined by the rate term D required to encode samples from the encoder using the optimal code designed for p(z).
measuring the excess number of bits
2.3 UNINFORMATIVE REPRESENTATION ISSUE
As underlined in the introduction, the main issue of VAE that the representations are not really informative of the input data and in the worst case, it is learned Z-independent generative model po(x|2) (o(x). Such issues are intrinsic in the ELBO objective, equation 2] that can reach the optimum when Dxr.(us (cs()))))lp = (Zhao et al., 2017/. The latter case means that the representation completely uninformative, indeed the rate term, which can be rewritten as
D (xz(g<<<|>>(lp<))) (x(X 2) D
is a penalty on the encoding information, and is zero when 1 Z) = O, with (o(2]²) go(2) = p(z), i.e. when ds does not encode any information about the input x.
We now describe the three most relevant models that try to overcome the uninformative representa- tion issue.
InfoVAE In (Zhao et al., (017) the Info VAE family of models was proposed, a generalisation of the VAE model optimising the objective
ax,(X, (X, Z) AD (1s()l)))) + Ep(a) Pocee
with a and À two real positive hyper-parameters.
The main advantage of this definition is that it is possible consider separately the two components of the rate term. In particular, in (Zhao et al. (017)) it was observed that by eliminating the infor- mation penalty (a 0), the generative performance of the model improves and the representation results are more informative.
starting from the observation that the optimal case rare, but most of the learned features by VAE are not disentangled, it is proposed opposite approach: put high penalty to the rate term, in order to constrain the model to learn the most informative property of the data, and then have disentangled representation of the data. The 3-VAE family is particular case of Info VAE where a = X > 1. This idea, that at first sight looks counter-intuitive is based on the observation that by the additive property of the KL-divergence
3-VAE In (Higgins et al. 2017 7
dim(2) 2)
DKL (el)((()))) = Σ (xL(go(z(<))llde
pushing the penalty associated with the rate is equivalent to penalising the informativeness of most features, leaving few features containing the relevant information. Starting from bits-back coding argument, a similar conclusion was derived in (Chen et al. 2016).
Minimal rate bound In order avoid the null rate issue, and maintain semantic informative representation, AAlemi et al. 2017) was suggested to optimise the following variant of ELBO:
Ex((a) (E)() [oggpp [(]==]]] B Ro D
3

---

Under review as a conference paper at ICLR 2020
where Rg 0. defines both the lower bound of the rate and the upper bound of the encoding
information (X, ((,,2) 2). If, as we will see below, the objective in equation 4 is a generalisation of both VAE (Ro maxg [x, Z)) and 3-VAE (Rg 0), the correct Ry is unknown and its optimisitation is not an easy task.
3
THE MODEL
3.1 THE VARIATIONAL INFOMAX
Assuming the distribution associated to the two random variables p(x) and p(z) is known, the InfoMax objective is defined as: find the joint distribution Po(x; z) E Po == {po(r,
Eptz)]pe [=]=)] p(z), [p(z)[pe [=]] p(z)} maximising the mutual information (x(X,
DEL (xo(Po 2) (p)))))(()), Le. find 0r O s.t. Ig. Ig for any 8 E O.
Since the definition via KL divergence is computationally intractable, it is necessary to re-write the mutual information
Tx(X, 2) = ho(X) (X),,
(5)
where he (X)) (x) [log (=() is the entropy of X, and he (x(x2) = (x.(z..)) llg pe (x|²)] is the conditional entropy hig (X Z). Since po( z) Po the entropy ho = -(X) constant, and in order to maximise the mutual information is sufficient to minimise the conditional entropy.
Excluding some special cases (Bell & Sejnowski 1997), minimising the conditional entropy is un- feasible, so it is necessary to consider an associated variational problem: for any go (z(2)) such that (=(=) p(z) and Φ E + O, learn the generative model pe (=]lz) minimising the reconstruction accuracy term Ep(z) (= (loo(po
Indeed, the following variational objective:
To [X, Z) = h(X) + (ott)) po [=][²]
s.t. gx(2) = p(2)
(6)
lower bound of To- (X, Z) and is maximal when g(=1x)) po(2| po- (=|z²), (see the Ap- pendix).
Unfortunately, the formulation in equation 6 still unfeasible to compute, because requires that (s(=) p(z), but by the butterfly architecture of the autoencoder, (=(z) tends to be uniformly distributed on the space Z. For this reason, the model is trained maximising the following relaxed form:
VIMo.s = Ep(n) (=]))) A=)((())llp()))
where introduced term D(gs (z) (=)))( encouraging the empirical distribution (x to be close, according to the metric D. to p(2). In the following assume D DEL, and in order to avoid any confusion the variational autoencoder trained maximising equation 7 will be dubbed VIMAE.
Encoding channel In VAE we observed that an uninformative representation was caused by the non-informativeness of the encoding map (2 Since from equation i not clear how (=(x)) behaves, we consider an equivalent representation, (see the Appendix):
VIMg, = D (p(x))lpo(z)) - (A
+ IX, Z).
From equation & we see that the infomax objective, equation 7 can be read as composition of three sub-objectives: find a generative model pe (=(2)), with marginal resembling the visible distribution p(æ) (first term); maximise the (unbounded) variational mutual information (third term); and learn an inferred distribution (2, x) close to the generative model (ar, 2). Then the optimum obtained by qa (z. 2) Po (z, 2) such that To (X, Z) is maximal, confirming the validity of the approximation made above.
3.2 CHANNEL CAPACITY
In channel with variational mutual information lo as defined in equation o the (variational) capacity Co (X, Z), is defined as
(x.
sup (x,( 0.P.p(Z)P
(9)
4

---

Under review as a conference paper at ICLR 2020
If P is the space of all distributions on 2, the capacity of the network coincides with the variational mutual information of the model trained minimising only the reconstruction loss. In the latter case, as observed above, it is not guaranteed to learn the generator with Po(2) = p(z). This is because,
given two equalty informative generative models po (2rl2) and po (z][2,, with encoder respectively (=(=))) and qør (z]az), with h())(=)) < h(]ar(=)), then To.$ < Tor From such observation
deduce that bounding the entropy of the representation Z is the way to bound the capacity without penalising the encoding information, and that the penalty introduced in equation 7 is actually bound to the capacity itself. So, the VIM objective can be defined as a variational approximation of the Capacity-Constrriiee Constrained InfoMax:
max (X, XC(X,
(10)
ZEZ
which given set of equally informative generators. learns the one having the minimal capacity. The idea of the CCIM is similar to the idea of the IB, from which was derived the 8-VAE: constrain the capacity of the network in order to learn only the relevant features of the input data. The difference with the IB,
max (x, (X, ZEZ
(11)
lies in the second term: the network capacity instead of the encoding information. This choice allows the network to learn a good representation (small capacity) while maintaining good generative performance (high mutual information). Indeed, as shown in equation 8 the generative performance is associated to the informativeness of (=|xr) and po (x/lz)
In order to test the assumption that it is sufficient to bound the entropy of Z, instead of the encoding mutual information, to learn a good representation, in the experiments (see below) we consider the cases Z is Normal (VIMAE-n) or Logistic (VIMAE-I) distributed. We choose to compare the popular Normal distribution with the Logistic one for two reasons: the Logistic has less entropy than Gaussian distribution and because it is a common assumption in natural science to suppose that the hidden factors of the visible data are logistically distributed (Hyvärinen et al. 2009).
Entropy vs Rate According to what written above the choice of the distribution p(z) is a way to bound the maximal information lo of the generative model as well as Rg in equation indeed, if we choose Rs = C(X, Z) the theoretical solutions of equation[[and equation 7 coincide. Although the two approaches are similar, both defines target Ip for the generator Pa (z]22, they differ in one fundamental aspect: the ratio between the information Is and network capacity C. For any given choice of the prior p(=) the generative model maximising VIM is the one with Io = C, instead for any Ro < C' the information I9 associated to the generative model learned, by equation 4 coincides with Ro. i.c. Le Ro C. That implies an high sensitivity to the hyper-parameter Ro for the model in equation 4 In fact, as suggested in (Mathieu et al. 2018)), ratio To/C" ~ 1, guarantees an appropriate level of overlap in the latent space between latent encoding, and then a better representation quality.
4 RELATED WORK
Autoencoder literature Autoencoder models are one of the most used family of neural net-
works to extract features in an unsupervised way (Bengio et al. 2013), and their relationship with Information Theory is well-established from the first unregularised autoencoders (Baldi & Hornik 1989. The classical unregularised autoencoders, minimising the reconstruction loss Ep(a) | [ log =)]], are maximising an unbounded information, i.e. they are looking for a solution in the space Po = {Po (x)) p(x)). In general, a solution in this wide space is good only for reconstruction performance because Z contains all the possible information that can be stored in the space Z. and is not robust to input noise (Vincent et al., 2008); but, as observed in (Grover & Ermon 2018). if (2(xx) ~ ((((x), (z)) the model po x (2) is robust to noise and is Gaussian generator. In this context the uninformative issue is avoided, but the price to pay is the impossibility to sample directly from prior p(z) that is not defined; indeed, the model described in
(Grovrr& & Ermon 2018) requires running relatively expensive Markov Chain to obtain samples.
Many regularised models have been proposed, but the most well known is VAE, that minimises the expected code length of communicating ær As we observed in the previous sections, it is not
5

---

Under review as a conference paper at ICLR 2020
guaranteed that the method finds useful representation. Such issue be solved both controlling the information of the model, or considering a more flexible prior p()),(Rezend) & Mohamed 2015
Kingma et al., 2016 Dinh et al.) 2016). In this manuscript because by the model structure, it requires quite expensive computation (autoregressive models are necessary) and because the learned representation is not suitable for task different than the generation such as lifelong context, for which the standard VAE is used with success (Achille et al. 2018).
do not consider the latter approach,
The objective in equation was firstly derived in (Tolstikhin et al. 2017) and (Zhao et al. 2017) Particularly relevant is the derivation in (Tolstikhin et al. (017)) because it allows us to describe an informative model po (z, 2) as the one minimising the transport cost between the original and generated data.
Finally, we underline that in case we wish to consider a Sensen-Shannor divergence in equation it is necessary to consider an adversarial network model, discriminating the true samples ~ p(2) from the fake sampled by qa (z) (Goodfellow et al. 2014). In the latter case the obtained model is equivalent to the Adversarial AutoEncoder (Makhzani et al. 2015). We conclude by remarking that in all the cases cited above the Infomax objective was never maximised using a prior p(=) different from Gaussian.
Information theoretic literature Information theory is strongly related with neural networks, and not only with autoencoders. Originally the InfoMax objective was applied self-organised system with a single hidden layer, (Bell & Sejnowski 1997, Linsker 1989) where the bound in the capacity was given by the numbers of hidden neurons. More recently, the (naive) InfoMax has given way to a new information-theerrtie principle: the cntemaation-ootlenneen
(Tishby et al. 2000). The idea of this principle is that a feed-forward neural network trained for task 7' tends learn minimal sufficient representation of the data, maximising the following objective:
max - 2).
(12)
Although it was shown that in the general case this principle does not hold true (Saxe et al. 2018 the principle was used as a regularisation technique with success both in unsupervised (Alemi et al. 2017 Higgins et al. 2017) and supervised (Alemi al.) 2016) settings. We observe that the VIM, equation and IB, equation 12, coincide in the case of a deterministic encoder, where the encoding information is the entropy of Z.
EXPERIMENTS
Here empirically evaluate the VIMAE. The section is divided into three parts: in the first part we compare the ability of the described models to infer the representation, 2 ~ N/(0, (0, 7). Such an experiment is to evaluate the entropy of Z ((Z) and then, as observed in section 3.2, an indirect way to estimate capacity of the network (C(X, 2) & h(2)). In the second part we evaluate the reconstruction and generative performance of the models. Indeed, the combination the two tasks is estimation of the mutual information of the generative model [X, Z), see equation g
In the third part we evaluate the robustness to noise and generalisation property of the learnt repre- sentation, observing that an informative model with small capacity is the best one for these tasks.
In all the described experiments, the divergence DKL L (g(=)]lp(=)) in equation is approximated via the Maximum Mean Discrepancy (Zhao et al. (017) defined as:
MMD(a(2),
sup f
Ep(=)[(()) Ef())((2)
(13)
where Hy is the Reproducing Kernel Hilbert Space associated to positive definite kernel k(-,))
Z x + R., Moreover, by difficulties to compute the objective equation 4 as suggested in (Alemi
et al. 2017) we decided to optimise 3- 3-VAE, denoted 3 VAE, with 3 < 1: in order to avoid any
confusion, the original version proposed in (Higgins et al. 2017) with $ >> 1 will be renamed VAE.

---

Under review as a conference paper at ICLR 2020
p(z) 1))
VAE, 0.70
9n- VAE, 0.53
-AE, 0.76
VIMAE, 0.08
Figure 1: 2-d learned representations. Under each plot the model name is followed by the respective MMD value.
5.1 THE ENTROPY OF Z
Experiments in this part are performed with an autoencoder trained with the MNIST data-set, a col- lection of 70k monocromatic handwritten digits. where both the inference and generative distribution are modelled by 3-layer deep neural nets with 256 hidden units in each layer and 2 = IR2
we observe that the ELBO-baaed models are not able to learn a distribution g(z) that is close to the prior p(z); and, as we will see in the following experiments, 35-VAE, the only ELBO-model having comparing results to the VIMAE, has to penalise drastically the decoding information.
From the 2d hidden distribution learnt by the different methods figure 1
Thanks to this experiment we see that, in general, the ELBO-based models are not seeking for small capacity network, since h(g(=)) >> h(p(=)), and that the divergence penalty introduced in equation 7 is bound for the entropy h(g())), and then the channel capacity C.,
5.2 THE MODEL INFORMATION
The experiments in these final sections were performed with the same settings and autoencoder models used in (Tolstikhin et al. 2017), an architecture similar to the DCGAN (Radford et al. 2013) with batch normalization (loffe Szegedy 2015) (more details given in the Appendix). We consider four data-sets: MNIST and CIFARIO. two standard data-sets with ground-truth labels; Omniglot, data-set of 1623 characters from 50 alphabets, training and evaluation, where each character appears 80 times, to evaluate the informativeness of the model and the quality of the learned representation: in the Appendix, we also consider CelebA (Liu et al. (015), consisting of roughly of 203k faces of 64 x 64 resolution, in order to compare the generative quality of the pictures. After considering many parameters for 3H, A and X, we choose, in accordance with what was suggested in (Tolstikhin al. 2017), 3h
10, and BA 0.2 for MNIST and Omniglot and 3 = X 100 and AA = 0.4 for CelebA and CIFARIO experiments.
The goal of this section is to evaluate the informativeness of the learnt generative model po z)) From what was described above, the reconstruction loss or the generation quality alone are not reli- able metrics, because the reconstruction loss is an estimation of the variational mutual information (X, 2), and the generation quality is an estimation of Dx (p)(2))lp(z)) that, as observed in section 2.3, it possible to minimise with an uninformative generator. But, according to equation the combination of the two task performances is a good empirical estimation of [X(,, Z); indeed, by generative experiments, we require that q(=) p(=), ) so the KL divergence term equation comes for free.
Reconstruction and generative performances According to what was asserted above, we ob- serve from figure 2 (top) and figure 3 that an high penalty on the rate term (small encoding infor- mation) the EL based model coincides with poor reconstruction. Indeed, the VAF,, the with worst reconstruction, and instead VAE is the model that behaves similarly to VIMAE, that is optimising an encoding information- free objective. Such behaviour is consistent with the rate term associated each model, see caption of each figure, where observe that VIMAE, have the highest rate and in particular this is twice the one associated to 3 confirming that VIMAE learns maximal informative decoder, and that VAE is only theoretically analogous to VIMAE.
The models that we are considering are defined as generative models, so giving a sample 2 ~ p(2) they should be able to generate data similar the original As observe from the

---

Under review as a conference paper at ICLR 2020
C
Y
F
0
C
m
C
X
m
6
E
8
3
8
0
X
2
9 E
B
3
3
3
R
1
2
5
kn
#
M
4 2
? T
7 2
8 8
E
4
6
1
3
2
9
9
0
P
VAE, (0.75, 48.7)
3и-VAR (0.9, 18.8)
BA-VAE (0.74, 67)
VIMAE-n, (0.75, 125)
VIMAE-I, (0.76, 120)
Figure 2: Test reconstruction (top) and random generative samples (bottom) of the different methods with Omniglot (right). In test reconstructions, the odd rows are the original data. Each model is denoted by its name and the reconstruction ||2 and rate term.
VAE, (8.29, 251)
8y-VAF, (9.8, 28)
8A-VAF, (5.7, 423)
VIMAF-n, (4.74, 445)
VIMAF-I, (4.85, 443)
Figure 3: Test reconstruction, CIFAR 10. Odd rows the original data. Each model is denoted by its name and the reconstruction ||2 and rate term.
generated samples of Omniglot in figure (bottom) and Negative LogLikelihood (NLL) listed in table ! both VAE and 3h-VAB do not generate good samples, confirming that sharper samples are associated to informative generators. Such qualities Omniglot are confirmed by the experiments with the CIFAR 10 and CelebA data-set (Liu et al. 2015), see the Appendix, where it observed that if the generative difference between the two VIMA is small, the difference with the VAE counter- parts is high. Such behaviour is in agreement with what was observed until now: the ELBO based model does learn good generative network, and the good reconstruction is simply associated to large entropy of Z, (encoding information) instead of an informative generative model py (x) 2))
Similar experiments were conducted with the MNIST dataset and are dis- cussed in the appendix.
Table 1: NLL for generated samples on Omniglot (smaler is better)
5.3 GENERALISATION PROPERTY
Method
NLL
We defined a good representation the one containing the relevant prop- erties of the visible data and able to generalise from the task for which was trained. In order to evaluate such quality, following the approach proposed in (Rifai et al. 20IT), we evaluate the accuracy of an SVM directly trained on the learned features of the data. Proceeding as in (Zhao et al. 2017) we train the M1+TSVM (Kingma et al., 20114) and use the semi-supervised per- formance over 1000 (100 for Omniglot) samples as an approximate metric verify the relevance and the quality of the learned representation. In order to evaluate the robustness of the learned features, we performed the same algo- rithm the representation associated to corrupted data, i.e. q(2|z + v), considering two types of noise: Gaussian and mask. In the Gaussian case, add to each pixel value sampled from A o²) with E {0.2, 0.3, 0.4), and in the masking
VAE
1224
BH-VAE
1254
8A-VAE
1228
VIMAE-n
1190
VIMAE-I
1223
8

---

Under review as a conference paper at ICLR 2020
Table 2: Semi-supervised classification CIFARIO.
accuracy (%)
Method
== [0 0.3%)
((022)
VAE
25
16
VAE
29
26
19
8A- VAE
31
18
VIMAE-n
29
28
23
VIMAE-I
32
34
23
Table Semi-supervised classification, Omniglot (random sampling: 20%).
Table 3: Semi-supervixe classification, MNIST.
accuracy (%)
accuracy (%)
Method
0
N(0 o²)
B(y)
V
0
NO o²)
B(p)
0.2
0.4
0.2
0.5
0.2
0.4
0.2
0.5
VAE
80
77
70
72
52
22
22
17
22
16
92
86
82
91
84
21
22
19
17
BA-VAE
93
66
13
85
65
22
22
21
21
24
VIMAE-n
93
92
86
92
86
22
23
24
22
22
VIMAE-I
93
92
88
92
87
24
23
20
23
22
fraction of the elements forced to be O: each pixel is masked according Bernoulli distribution B E {0.2, 0.5). Higher classification performance suggests that the learned repre- sentation contains the relevant information and, in case of corrupted input data, that it is robust. In the Omniglot case by the challenge of the task (the test alphabet was never seen in the training) we consider a 5-character data-set, split into 300 (60 5) for training and 100 for evaluation.
From the classification scores listed in tables 2 4 we see that the ELBO-based based model learnt good representations for clean data, but not when corrupted data given as input. This particularly clear in the Bernoulli case, that is noise different from the one seen in the training. Particularly relevant is the behaviour of the two VIMAES: they are comparable in the cases of clean data and small noise, but the one with big capacity, VIMAE-1, suffers in large noise, while the one with small capacity, VIMAE-I, is the most robust and in some challenging cases, see table 2 the noise helps to improve the model accuracy. Such result is consistent with the idea that small capacity network learning the relevant factors of the input data, that are the only ones robust to the input noise.
CONCLUSION
We observe, via information theoretic description of VAE, that it is possible to learn good gener- ative model while maintaining a meaningful hidden representation, and that goal can be reached by optimising the CCIM, an objective that separates the two properties of network: the generative information and its capacity. We compare both theoretically and computationally the CCIM objec- tive with the IB and its variation (Alemi et al.) 2017). observing that the CCIM optimal solution, in particular cases, coincides with the IB one, but that in the practical case the difference is substantial.
Future work includes the extension of the VIMAE model to a lifelong task, a scenario where is necessary to learn both an informative representation while maintaining bounded the capacity of the network, in order to avoid the catastrophic forgetting issue (Achille et al. 2018).
REFERENCES
Alessandro Achille, Tom Eccles, Loic Matthey, Chris Burgess, Nicholas Watters, Alexander Lerch- ner, and Irina Higgins. Life-long disentangled representation learning with cross-dar domain latent homologies. In Advances in Neural Information Processing Systems, pp. 9873-9883, 2018.

---

Under review as a conference paper at ICLR 2020
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410 2016.
Alexander A Alemi, Ben Poole, lan Fischer, Joshua V Dillon, Rif A Saurous, and Kevin Murphy. Fixing broken elbo. arXiv preprint arXiv:1711.00466 2017.
Pierre Baldi and Kurt Horiik, Neural networks and principal component analysis: Learning from examples without local minima. Neural networks, 2(1):53-58, 1989.
Anthony J Bell and Terrence J Sejnowski. The independent components of natural scenes edge filters. Vision research, 37(23):3327-33333 1997.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1988-1828, 2013.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerehner. Understanding disentangling in ß-vae. arXiv preprint arXiv:1804.03550 033599 2018.
Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. arXiv preprint 2016.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv 2016.
preprint
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Aditya Grover and Stefano Ermon. Uncertainty autoencoders: Learning compressed representations via variational information maximization. arXiv preprint /8/2. 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vac: Learning basic visual concepts with a constrained variational framework. In International Conference Learning Representations, volume 3, 2017.
Geoffrey E Hinton, Simon Osindero, and Yey-Whye Teh. A fast learning algorithm for deep belief nets. Neural computation, 18(7):15271154,, 2006.
Aapo Hyvärinen, Jarmo Hurri, and Patrick O Hoyer. Natural image statistics: A probabilistic ap- proach early computational vision., volume 39. Springer Science & Business Media, 2009.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6958 69800, 2014.
arXiv preprint Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114,0 2013.
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pp. 3581-3589, 2014.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. In Advances in neural information processing systems, pp. 4743-4751, 2016.
Ralph Linsker. An application of the principle of maximum information preservation to linear sys- tems. In Advances in neural information processing systems, pp. 186-194, 1989.
10

---

Under review as a conference paper at ICLR 2020
Ziwei Liu, Ping Luo, Xiaogang Wang. and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 7730-3738, 2015.
Chris Maddison, John Lawson, George Tucker, Nicolas Heess, Mohammad Norouzi, Andriy Mnih, Arnaud Doucet, and Yee Teh. Filtering variational objectives. In Advances in Neural Information Processing Systems, pp. 6573-6583, 2017.
Alireza Makhzani, Jonathon Shiens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint arXiv: 1.05644, 2015.
Emile Mathicu, Tom Rainforth, Siddharth Narayanaswamy, and Yee Whye Tch. Disentangling disentanglement. arXiv preprint arXiv: wrxv::8//2.02833, 2018.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, arXiv preprint arXiv:1511.06434, 2015.
Jason Ramapuram, Magda Gregorova, and Alexandros Kalousis. Lifelong generative modeling.
arXiv preprint urXiv: moo:::7000000000 09847, 2017.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint 1xXvx:1500005700 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4000 1401.40042 2014.
Salah Rifai, Grégoire Mesnil, Pascal Vincent, Xavier Muller, Yoshua Bengio, Yann Dauphin, and Xavier Glorot. Higher order contractive auto-encoder. In Joint European Conference on Machine Learning and Knowledge Discovery in Datahases, pp. 645-660. Springer, 2011.
Andrew Michael Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Bren- dan Daniel Tracey, and David Daniel Cox. On the information bottleneck theory of deep learning. 2018.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physicc/00044057, 2000.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. encoders. arXiv preprint 1rXiv: 2017.
Wasserstein auto-
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Picrre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings the 25th international conference on Machine learning, pp. 1096-1103. ACM, 2008.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning.
D. D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 3630-3638. 3638. Curran Associates, Inc., 2016. URL http:://pppppe http:://apers.nnpsscc//ppppp wwwp.cccpp..c......// hag.-aattccingg--toooo------- for wwwe-ooo---ottt--aaaaaa---------
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Infovae: Information maximizing variational autoencoders, arXiv preprint 770.:22706.02262, 2017.

---

Under review as a conference paper at ICLR 2020
RELATIONSHIP BETWEEN ENCODING, DECODING AND VARIATIONAL INFORMATION
Defined g(2, := (=(2)))))) the encoding distribution and po(x, z) == po (=())))p(2) the decoding
one. The encoding (X, 2). decoding (x( 2) and the variational La (X, Z) information are defined respectively:
1x(X, 2) =
= - (1,)) (x(2)
Ix(X, Z) Drr.(po(2..))l((())))))
M(X) ha(XX))
To. (X, 2) = -h(X)= [= logg
Assuming 0" E @ is the parameter associated the maximal decoding information, Io(X, Z) s I-- (X, Z) for any 0 E O, it follows that for any g0(I,2) Pe. i.c. for any qo(z) Ep(x) (x]])]] p(z) and O.
la (X, 2 (X,
Then lower bound of T is lower bound of Tg.. By property of KL-divergence we have that for any pa (x|2) the following relationship holds:
E Rg(t log po (ar =)] = (X + By(ss) (Dx)) (Xaee//ae((ellllllooo0aaaa
(14)
From equation 14 and the definition of the variational information Io.s we deduce that:
1g- (X, 2) > (X 2) 2 Le. (X,Z) Z)
We conclude observing that if @ Φ, optimum the three information terms above equal, and then (0 2) Po 2).
Observation: If + O, the variational mutual information can be at most equal to Igr, the maximal lq. but it is not guaranteed that 4-- = l8".
DERIVATION OF EQUATION
In Zhao al. 2017), is observed that equation can be written follows:
VIMa. = (xI,(p(=)llo(<)) - (eye (poo=))))))))))))
(A (x))))x (X,
From equation above, to verify that equation is correct, is sufficient to show that
1a (X, Z) = L2 Z) (x
(15)
The equation
15 follows by the property of the autoencoder and equation 15
More precisely, by equation 14 we have that
Ie.5 (X, 2) I, (X,
and, by AutoEncoder architecture, observed in section p2(2) Ep(z) (2|2)) q(2). Then the following equation holds
By(z) D (kaa) = (xr
Indeed,
By(s) B = g(2) / a(x)=) log g(xl=)
dadz
po (x)2))
= Epp (0(x)22(2) dxdz = / p(a)) ((<)2)) log
= f (()))(l)))) log (((22)(())
q(>(x) drdz
po(2|x)
12

---

Under review as a conference paper at ICLR 2020
FURTHER DETAILS ON EXPERIMENTS
In all the experiments in section 5.2 considered the latent space Z = RR for all the models choose the prior p(z) to be a Gaussian with zero mean and identity covariance, only VIMAE-I we choose the prior (2) to be a logistic with mean zero and identity variance. We choose pe (1(22) to be similar to DCGAN with batch normalization and qo [e(2(2)) to be a convolutional deep neural network. The entire models are trained end to end by Adam (Kingma 0.5, B2 0.999. We considered a deterministic decoder and we approximate the reconstruction loss with the Lg loss, i.c. pp(x)) [[]((]=>]] - log pe p=[22)]] = || - Tg ll.. with Ia indicating the generated datum. In VIMAE case, while training we were adding pixel-wise Gaussian noise truncated at 0.01 to all the images before feeding them to encoder, in order to make the encoder random. In VAE and B VAE case, instead we used the standard reparameterization trick (Kingma & Welling 2013).
Ba, 2014) with a 10-3, 1
In the following we describe the data-sets considered and the associated neural networks, we follow the same description given in (Tolstikhin et al. 2017) since used the same neural nets.
MNIST AND OMNIGLOT
MNIST is a data-set containing 70k grey-scale handwritten digits and associated labels of resolution 28 x 28, subdivided in three subsets: train (50k), validation (10k) and test (10k). Omniglot is data-set containing 1623 different handwritten characters of resolution 28 x 28 from 50 different alphabets. Each of the 1623 characters
drawn online via Amazon's 's Mechanical Turk by 20 different people. In the same fashion as done (Vinyals et al.) (016) we considered an augmented version where each character is rotated respectively by 90, 180, 270 degrees, in this way each character appears 80 times. The Omniglot data-set although has the same resolution of the MNIST, for this reason we use the same network, it is more challenging because, more entropic then MNIST, in fact the classes move from 10 to 1623 and the test classes are never seen in the training.
We choose = R and A 10, we used bini-batches of size 100 and trained the model for 80 epochs. Both encoder and decoder used fully convolutional architectures with 4 x 4 convolutional filters. Encoder:
V. E R Ip28x25
Conv + BN + ReLu
Conv256 BN ReLu
Conv512 - BN + ReLu
Conv 1024 BN ReLu
Decoder:
R
FC7x7x1021 FSConV512 + BN + ReLu
FSConV256 256 + BN ReLu + FSConV1
Where Conv, stands for a convolution with & filters, FSConv; for the fractional strided convolution with & filters, BN for batch normalization, ReLU for the rectified linear units, and FC₁ for the fully connected layer mapping to R All the convolutions in the encoder used vertical and horizontal strides and SAME padding.
CELEBA AND CIFAR10
CelebA a data-set with 202 599 faces images. We preprocessed the images by first taking a 140 140 center crops and then resizing to the 64 x 64 resolution and we consider the last 20k images as test subset.
CIFAR10 10 is dataset consisting of of 60k 32 32 colour images in 10 classes, with 6k images per class. There are 50k training images and 10k test images.
13

---

Under review as a conference paper at ICLR 2020
For these data-sets choose the same network with the same nyper-parameterr X 100 and 2 = R44 We used mini-batches batches size 100 and trained the model for 60 epochs. Both encoder and decoder used fully convolutional architectures with 5 x 5 convolutioanal filters.
Encoder:
R 16xx61x3
+ Convi2s
BN + ReLu
+ ComV256
BN + ReLu
+ Convs12
BN - ReLu
+ ConV1024 + BN - ReLu
Decoder:
2 E Ro1
+ FC8x8x1021
FSConvs12 BN + ReLu
+ FSConV256 + BN + ReLu
FSConV12s BN + ReLu FSConv1
CELEBA AND OMNIGLOT EXPERIMENTS
In section 5.2 we evaluate the generative performance on relatively simple grey-scale data-set. In or- der consider a more challenging data-set and quantitatively compare the generative performances of the trained models we evaluate the Frechet Inception Distance (FID) on CelebA and CIFARIO based on 10 samples. From table 5 we observe, in agreement on what observed in section 5.2 and figures 4 and [ that the difference between the two VIMAE models is minimal, instead it is big difference with the ELBO based models (3n-VAE is not listed in table 5 because does not converge).
Table 5: FII scores for generated samples on CelebA (smaller better)
Method
FID
CelebA
CIFARIO
VAE
82
168
BH- VAB
262
BA- VAE
89
174
VIMAE-I
56
103
VIMAE-n
55
104
MNIST EXPERIMENTS
The MNIST dataset classical benchmark but quite simple, indeed as we see from figure 6 the models considered are able to reconstruct without big differences, although as see from the L2 reconstruction loss in the VIMAE models smaller than the ELBO-based counterpart. That results combined with the high value of the rate term, suggest that the VIMAE are actually learning a maximal informative decoder. We conclude observing that, according to what seen until now, the high rate value for 3A- VAE is associated to high entropy h(g(z)), this is well visible by the generated samples from p(2) in figure where the samples associated to 8 VAE seem to belong the same cluster.
14

---

Under review as a conference paper at ICLR 2020
VIMAE-n
VIMAE-I
Figure 4: Test reconstruction (top) and random samples (bottom) of the two VIMAE models,
A 10.
VAE
8 -AE
VAE
VIMAE-n
VIMAE-I
Figure 5: random samples generated by the VAEs trained on CIFARI0
15

---

Under review as a conference paper at ICLR 2020
6
7 7
F 2
30 /
3
5
3
3
O
P 7 O
5 64 3
0
5 C 73
b
O 2 A
52 7 7077111755 55
8
6 2 700772122
P
8
2 2 B 10
31 17
2 2
11 17
7 1 4 1
7 8/3668700
VAE (0.51, 54.88)
VAE, (0.62, 25.92)
BA- VAE.
(0.5 70,56)
661 4
7
7 2.5
3.0
F 3 3
2 5
2
3
6
5
D
0 S 1 b
04
5 T 0 S
/
2 >
267 2
0
0 8 11188660200
3
B C 60 5
9 L 22 7
1
L
67 5 L
V 7 2
4
1
6 1
G
3
1 2
17 A
3 4 33
5
668
6
36637
VIMAE-n, (0.47, 101.02)
VIMAF-I, (0.48, 109.76)
Figure Test reconstruction by the VAEs trained on MNIST
4
V
7
7
9
3
2
7 7
0 6
3
5
5
3 5
5
7
5 3 1
L - 007
8
O
1940 1111111122
2
VAE
-и-VAE
9A-VAE
B
2
1 34 7 88 22
47
08
6 6 / 50
r3
6
1
1
D
9 5 12100
D 0
3 3
> 24
1
Y /
R 230
666666722 667
15605
23 / 4 101111000/00 /13
13 1 7770775747 750413
7 1 199
3 1 53153332200
o 2 7
SEEY330945222100
3 7 05556600/460
5 2 2 23055+645533
7
VIMAE-n
VIMAE-I
Figure 7: random samples generated by the VAEs trained on MNIST
16


Reviews:
[{"review_id": "rkgKQMmd9r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I went over this work multiple times and had a really hard time judging the novelty of this work. The paper seems to be a summary of existing work reinterpreting variational autoencoding objectives from an information theoretic standpoint. In particular, the paper seems to follow the same analysis as in Wasserstein Autoencoders (Tolstikhin et al., 2017) and InfoVAE (Zhao et al., 2017).  It is unfair to say that the objectives were \"derived independently\" since these works are from a couple of years ago. \n\nThe paper also lacks discussion on two crucial works in this space:\n1. https://arxiv.org/abs/1711.00464 shows how to trade off rate-distortion in VAEs.\n2. https://arxiv.org/abs/1812.10539 shows how to learn informative representations by eliminating the KL divergence term and at the same time specifying an implicit generative model (Theorem 1).\n\nre: disentanglement. Unsupervised disentanglement has been shown to be theoretically impossible and several key challenges have been highlight w.r.t. prior work. Again, the relevant paper: https://arxiv.org/abs/1811.12359 has not even been cited. More importantly, the claims around \"more disentangled representation\" are imprecise in light of this work.\n\nA proper discussion on the contributions of this work as well as discussion on the above related works would be desirable on the author's end.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "r1lWnRs7YH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper develops an information-theoretic training scheme for Variational Auto-Encoders (VAEs). This scheme is tailored for addressing the well-known disentanglement problem of VAEs where an over-capacity encoder sometimes manages to both maximize data fit and shrink the KL divergence between the approximate posterior and prior to zero. Consequently, the latent representations of the observations become independent, making them unusable for any downstream task.\n\nThe method developed in Section 3 and proposed explicitly in Eqs 7 to 11 is novel per se, though not groundbreaking.\n\nFigs 2 and 3 are only few examples manually chosen from rather simple tasks. In the absence of a quantitative evaluation metric, they are not informative. In the outcomes of the same runs, there might exist counterexamples where the vanilla VAE generate perceptively more appealing reconstructions than VIMAE.\n\nAs a minor comment, I think the paper unnecessarily complicates the presentation of the idea. The development of notions such as f-Divergence, links to MMD, the constrained optimization setting in Eq 5 etc., do not serve to the main story line, but only distracts a mind. I would prefer a brief intro on VAEs and directly jumping into the proposed method. Last but not least, the statement of technical novelty in Section 3 is extremely encrypted. What exactly is \"absolutely\" novel there and what is prior art? Can the authors give a to-the-point answer to this question during the rebuttal?\n\nThe paper has two fundamental weaknesses:\ni) The paper misses a key reference, which addresses the same representation disentanglement problem using the same information-theoretic approach, only with some minor technical divergences:\n\nAlemi et al., Fixing a Broken ELBO, ICML, 2018. \n\nThis paper is a must-cite, plus a key baseline. This submission cannot be treated as a contribution without showing an improvement on top of this extremely closely related work. The problem is the same, the solution is almost the same, and the theoretical implications of the solution are also the same.\n\nii) The paper builds the entire story line around the statement that the representation disentanglement problem is caused by the properties of the ELBO formula and attempts to fix it by developing a new inference technique. This is attitude largely overlooks simpler explanations. For instance, the vanilla VAE assumes a mean field q(z|x) across the z dimensions. This makes q(z|x) fall largely apart from p(x) which definitely does not factorize that way. Earlier work has shown substantial improvements on the quality of q(z|x) when structured variational inference techniques are used, such as normalizing flows. Furthermore, there has also emerged techniques that can very tightly approximate p(x) in closed-form using the change of variables formula without performing any variational inference at all. No need to say, a tight approximation on p(x) means the same for p(z|x) simply using the Bayes rule. This leaves no room for further improvements by information-theoretic inference alternatives. For instance see:\n\nDinh et al., Density Estimation Using Real NVP, ICLR, 2017\n\nThe presence of such two strong alternative remedies to the problem addressed by this work makes its fundamentals shaky. The only way to get over this situation is to provide a thorough comparison against these methods, which is obviously missing.\n\n---\nPost-rebuttal: Thanks to authors for all the effort they put on the updated version. I appreciate that the paper now provides a quantitative comparison. However, my point (ii) remains fully unaddressed and I fully share the theoretical concerns raised by Reviewer 1. All in all, I keep my position on the below the threshold side with a weak reject.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SkxE4Bxr9r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview: This paper describes the Variational InfoMax AutoEncoder (VIMAE), which is based on the learning principle of the Capacity Constrained InfoMax. The core idea behind VIMAE is that the encoding information is not bounded while network capacity is. The issue that VIMAE can handle, and where VAE fails, is that representations are not informative of input data, due to the information bottleneck idea that VAE is built upon. The authors describe InfoVAE and β-VAE, which both attempt to solve this problem. The theory behind VIMAE is then described and tested against VAE and β-VAE, in their abilities to evaluate the entropy of Z, in reconstruction and generative performance, and in robustness to noise and generalization. \n\nContributions: The authors clearly state the contributions of the paper themselves, but in a summary: a derivation of a variational lower bound for the max mutual info of a generative model, definitions and bounds estimation for a VAE, associations for generative quality and disentanglement representation to mutual information and network capacity, and finally proposing the Capacity-Constrained InfoMax.\n\nComments:\nPage 1: “Our derivation allows us to define…” -> this sentence is a bit long and took me a few reads to understand, reword this please.\n“Derivation of a variational lower bound for ...a geneaive* model belonging..” -> “generative”\nPage 2: “We conclude the paper with experimental results and conclusions.” You already said conclude twice in this sentence, I feel like this could be better worded to avoid that.\nPage 5: “In order to test the assumption that it is sufficient..” -> This wording is also hard to wrap my head around. Too many commas.\nPage 6: “In figure 1 are plotted the 2d…” -> This sentence could be reworded\n\nMy main complaint with the paper is that there are quite a few places that could be reworded better in addition to these. Please fix this.\n\nYou mention both InfoVAE and β-VAE yet only test your models against a β-VAE model. What was your reasoning for this?\n\nThe semi-supervised learning experiment is interesting (the one based on Zhao et al.) VMAE models are able to classify better regardless of noise or sampling procedure, especially the smaller capacity model VMAE-1. I’d like to hear a discussion in the paper about future work and how the authors believe this could be applied elsewhere.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper describes a new generative model based on the information theoretic principles for better representation learning. The approach is theoretically related to the InfoVAE and beta-VAE work, and is contrasted to vanilla VAEs. The reviewers have expressed strong concerns about the novelty of this work. Some of the very closely related baselines (e.g. Zhao et al., Chen et al., Alemi et a) are not compared against, and the contributions of this work over the baselines are not clearly discussed. Furthermore, the experimental section could be made stronger with more quantitative metrics. For these reasons I recommend rejection.
Decision: Reject