Title: NON-SYN VARIATIONAL AUTOENCODERS

Abstract: Learning disentangling representations of the independent factors of variations that explain the data in an unsupervised setting is still a major challenge. In the following paper we address the task of disentanglement and introduce a new state-ofthe-art approach called Non-synergistic variational Autoencoder (Non-Syn VAE). Our model draws inspiration from population coding, where the notion of synergy arises when we describe the encoded information by neurons in the form of responses from the stimuli. If those responses convey more information together than separate as independent sources of encoding information,they are acting synergetically. By penalizing the synergistic mutual information within the latents we encourage information independence and by doing that disentangle the latent factors. Notably, our approach could be added to the VAE framework easily, where the new ELBO function is still a lower bound on the log likelihood p x . In addition, we qualitatively compare our model with Factor VAE and show that this one implicitly minimises the synergy of the latents.

1. INTRODUCTION:

Our world is hierarchical and compositional, humans can generalise better since we use primitive concepts that allow us to create complex representations (Higgins et al. (2016) ). Towards the creation of truly intelligent systems, they should learn in a similar way resulting in an increase of their performance since they would capture the underlying factors of variation of the data ( Bengio et al. (2013) ; Hassabis et al. (2017) ; Botvinick et al. (2017) ). In addition, good representations improve the performance for tasks involving transfer learning and multi-task learning; since it will capture the explanatory factors.

According to Lake et al. (2016) , a compositional representation should create new elements from the combination of primitive concepts resulting in a infinite number of new representations. For example if our model is trained with images of white wall and then is presented a boy with a white shirt, it should identify the color white as a primitive element. Intuitively, our model will be able to construct different and multiple representations from the primitives. Furthermore, a disentangled representation has been interpreted in different ways, for instance Bengio et al. (2013) define it as one where single latent variables are sensitive to changes in generative factors, while being invariant to changes in other factors. In addition, we agree with Higgins et al. (2017a) , which mentions that a disentangle representation should be factorised and interpretable. Intuitevely, the model could learn generative factors such as position, scale or colour; if it is disentangle it should be able to traverse along the position variable without changing the scale or the colour. It's worth noting that disentangled representations have been useful for a variety of downstream tasks such as domain adaptation by training a Reinforcement Learning agent that uses a disentangled representation of its environment Higgins et al. (2017b) ; or for learning disentangled primitives grounded in the visual domain discovered in an unsupervised manner Higgins et al. (2017c) .

2. RELATED WORK:

The original Variational autoencoder (VAE) framework (Kingma & Welling (2013) ; Rezende et al. ( 2014)) has been used extensively for the task of disentanglement by modifying the original ELBO formulation; for instance β-VAE is presented in Higgins et al. (2017a) which increases the latent capacity by penalising the KL divergence term with a β hyperparameter. In addition, Kim & Mnih (2018) achieved a more robust disentangled representation by using the model called Factor VAE which penalises the total correlation of the latent variables encouraging the independence of the latents; a similar approach is shown in Chen et al. (2018) , where they present a clever ELBO decomposition based on Hoffman & Johnson (2016) . Other approaches rely on information bottleneck presented in Tishby et al. (1999) to model frameworks for this task such as Alemi et al. (2016) ; Achille & Soatto (2016) . Furthermore, Chen et al. (2016) describe a model based on Generative Adversarial Networks (Goodfellow et al. (2014) ) by encouraging the mutual information between the latents and the output of the generator. Notably, in Higgins et al. (2016) , they describe comprehensively the β-VAE model using a neuroscience and information theory approaches; they suggest that by encouraging redundancy reduction the model achieves statistical independence within the latents. This model inspired us to look into different fields for new ways to enforce disentanglement of the latents.

3. SYNERGY:

To understand our model, we need first to describe Synergy (Gat & Tishby (1998) ; Schneidman et al. (2003) ) being a popular notion of it as how much the whole is greater than the sum of its parts. It's common to describe it with the XOR gate, since we need two independent variables to fully specified the value of the output. Following, we describe the synergy from two related fields.

3.1. INFORMATION THEORY APPROACH:

Computing the multivariate synergistic information is an ongoing topic of research Schneidman et al. (2003) ; Williams & Beer (2010) ; Bertschinger et al. (2012) ; Griffith & Koch (2012) . Most of the current research in this topic uses the Partial information diagram described by Williams & Beer (2010) . In order to understand the importance of the Synergy information in our framework it's essential to describe the relations with the Unique and Redundant Information. Introducing the notation from Williams & Beer (2010) , let's consider the random variable S and a random vector R = {R 1 , R 2 , .., R n }, being our goal to decompose the information that the variable R provides about S; the contribution of these partial information could come from one element from R 1 or from subsets of R (ie. R 1 , R 2 ). Considering the case with two variables, {R 1 , R 2 }, we could separate the partial mutual information in unique (U nq(S; R 1 ) and U nq(S; R 2 ) ), the information that only R 1 or R 2 provides about S is redundant (Rdn(S; R 1 , R 2 )), which it could be provided by R 1 or R 2 ; and synergistic (Syn(S; R 1 , R 2 )), which is only provided by the combination of R 1 and R 2 . The figure 1 depicts the decomposition; this diagram is also called PI-diagram (Partial information) . Additional notation is used for a better visualisation in the case of more variables: Unique {1},{2}; Redundant {1}{2} and Synergistic {12}. For the case of 2 variables (X 1 , X 2 ), we expect four contributions to the mutual information as described in Bertschinger et al. (2012) ; Olbrich et al. (2015) :

I(S; R 1 , R 2 ) = SI(S; R 1 , R 2 ) Redundant + U nq(S; R 1 \ R 2 ) Unique + U nq(S; R 2 \ R 1 ) Unique + Syn(S; R 1 , R 2 ) Synergistic (1)

It's easy to see that the number of terms increases exponentially as the number of sources increases. The best measure for synergy is an ongoing topic of research. In the subsection we are going to talk about the synergy metrics.

3.2. POPULATION CODING APPROACH:

For neural codes there are three types of independence when it comes to the relation between stimuli and responses; which are the activity independence, the conditional independence and the information independence. One of the first measures of synergy for sets of sources of information came from this notion of independence. In Williams & Beer (2010) it is stated that if the responses come from different features of the stimulus, the information encoded in those responses should be added to estimate the mutual information they provide about the stimulus. Formally:

I(S; R 1 , R 2 ) = I(S; R 1 ) + I(S; R 2 ) (2)

However, we just saw in the previous sections that the I(S; R 1 ) and I(S; R 2 ) could be decomposed in their unique and redundant and synergistic terms. Intuitively, this formulation only holds if there is no redundant or synergistic information present; which means in the context of population coding that the responses encoded different parts of the stimulus. If the responses R 1 and R 2 convey more information together than separate, we can say we have synergistic information; if the information is less, we have redundant information. That's the reason why in Gat & Tishby (1998) , the synergy is considered as measure of information independence:

Syn(R 1 , R 2 ) = I(S; R 1 , R 2 ) -I(S; R 1 ) -I(S; R 2 ) (3)

3.3. SYNERGY METRIC:

First we need to state the notation (the same as Griffith & Koch (2012) ):

• n: Number of individual predictors X i • A i : subset of individual predictors (ie. A i = {X 1 , X 3 }) • X: Joint random variable of all individual predictors X 1 X 2 ..X n • {X 1 , X 2 , ..., X n }: Set of all the individual predictors • Y: Random variable to be predicted • y: A particular outcome of Y.

The intuition behind this metric is that synergy should be defined as the "whole beyond the maximum of its parts". The whole is described as the mutual information between the joint X and the outcome Y; whereas the maximum of all the possible subsets is interpreted as the maximum information that any of the sources A i provided about each outcome. Formally, this is stated as:

EQUATION

This metric derives from Williams & Beer (2010) and Griffith & Koch (2012) , however one of the differences with this metric with the one presented in Griffith & Koch (2012) is that in this one we are considering the specific mutual information I max in a group of latents A i , whereas in the paper mentioned it considers only an individual latent. Notably the I max can be expressed in terms of the KL divergence.

EQUATION

Putting together the equation 7 and 5, we have the following:

EQUATION

In the following section we are going to use the intuition provided by the above equation in the VAE framework for the task of disentanglement.

4. MODEL DERIVATION:

The motivation of our contribution is inspired in this concept and driven by the belief that synergy is not desirable for the task of disentanglement, since we want the latents to be independently informative as possible about the data, instead of needing many latents to specify the data. Therefore, we argue that by penalising the synergistic information within the latents and the data, we would encourage the disentanglement of the underlying factors of variation. This hypothesis is also inspired in the information presented in Griffith & Koch (2012) ; Gat & Tishby (1998) , where it is stated that the synergy is a measure of independence information in the responses of the stimuli.

First, we need to change the notation to match the VAE framework notation (Z are the latents and X is the observations). Also, A i is a subset of the latents, such that A i ∈ {Z 1 , Z 2 , ..., Z n } and Z is the joint of the latents. Formally: Z = d i Z i , where d is the number of dimensions of the latent variables. Besides, from the VAE standard framework, we know that the distribution p(A i |x) is intractable which is why we need to use an approximate distribution q φ (A i |x) parametrised by φ parameters. It's important to notice that this KL divergence could be computed in the same way as in the VAE framework; the only difference is the number of dimensions used for the random variable z. In the original VAE framework, we compute the KL divergence considering the joint Z = d i Z i ; whereas for the Synergy metric we don't use the joint but a subset of the latents. For instance, if A i = Z 2 Z 5 Z 8 , we have the following expression:

EQUATION

Taking in account these considerations, we express the equation 8 as follows:

S max ({Z 1 , Z 2 , ..., Z d }; X) = I(Z; X) - x∈X p(X = x) max i D KL q φ (A i | x) p(A i ) (10)

We start with the original ELBO formulation (Rezende et al. (2014) ; Kingma & Welling ( 2013)) and add the penalised term corresponding to the synergy, where α is a hyperparameter:

EQUATION

Expanding the S max term, we have the llowing:

L new (θ, φ, x) = L elbo (θ, φ, x) -α(I(z; x) - x∈X p(X = x) max i D KL q φ (A i | x) p(A i ) ) (13)

From Hoffman & Johnson (2016), we know that the KL term in the ELBO loss is decomposed in D KL q φ (z n ) p(z n ) + I(x n ; z) when we use the aggregate posterior and define the loss over the empirical distribution of the data p data (x). Taking in account that, we can express the equation 12 as follows:

EQUATION

If we penalise the synergy (see Eq. 12), we will be penalising the mutual information term which is not desirable for this task Kim & Mnih (2018) ; we can see this effect explicitly in Eq. 14. Therefore, we use only the second term to perform the optimisation which means maximising the subset of latents with the most amount of MI per outcome.

EQUATION

It's easy to see in Eq. 15 that it's not a guaranteed lower bound on the log likelihood p x anymore, which is why we decided to penalise the subset of latents with the minimum specific mutual information (ie. A w ). In practice we found that computing the maximum subset A i for each outcome of x is too computational intensive, which is why we decided to use a mini-batch approximation, which is the version we show in the pseudo-code using a two step optimisation in the next section.

The final version we are going to use is the one below, where Imax is the KL term of the synergy term.

EQUATION

Algorithm 1 Non Syn VAE Input: Observations (x (i) ) N i=1 , batch size m, latent dimension d, weight of synergy loss α, discount factor ω, optimiser optim, reparametrisation function g φ . θ, φ ← Initialise VAE parameters repeat 3:

x (i) ← Random minibatch B of size m, i ∈ B z i ← g φ ( , x (i) ) Sample z i ∼ q φ (z|x) 6: φ, θ ← optim(∇ θ,φ L elbo (θ, φ; x)) : Gradients of ELBO minibatch x (i) ← Random minibatch B' of size m, i ∈ B mu, logvar ← Encoder(x (i) , φ) 9:

worst index ← get index greedy(mu, logvar, ω) L syn ← α * Imax(mu, logvar, worst index) See Eq.16 for Imax function φ ← optim(∇ φ L syn (φ; x (i) ))

: Gradients of Syn loss minibatch 12: until convergence of objective In the algorithm shown above we see in practice that we get better results when we sample the values of mu and logvar from the encoder for the step 2 of the optimisation. We use a greedy approximation of the best latents by following a greedy policy (See Appendix).

5. EXPERIMENTS:



5.1. LATENT TRAVERSALS:

For disentanglement, the dataset most commonly used is the dsprites dataset Matthey et al. (2017) , which consists on 2D shapes generated from independent latent factors. We used the same architecture and optimizer as Factor VAE Kim & Mnih (2018) for training our model. In order to test qualitatively the Non-Syn VAE model, we decided to tranverse the latents and plot the mean activations of the latents.

In Figure 2 (left), we see clearly that our model disentangles the factors of variation. Likewise, on the right we see the mean activation of each active latent averaged across shapes, rotations and scales. After looking at the figure above we can state that our model achieves state-of-the-art results using a qualitatively benchmark. Interestingly, both models perform quite similar in this test.

5.2. SYNERGY IN FACTOR VAE:

Also, we decided to compute the same synergy term from the Non-Syn VAE in Factor VAE (just compute it, we didn't use it for training). The hypothesis was that if Factor VAE achieves disentanglement, it should minimise the synergy as well. We train Factor VAE using the same parameters and architecture described in Kim & Mnih (2018) . We show the first 4000 steps for the Synergy term (i.e. α * Imax), since most of the interaction happens in the first steps. In this paper we presented the intuition and derivation of the lower bound of a model that uses a novel approach inspired by the information theory and Neuroscience fields to achieve the disentanglement of the underlying factor of variations in the data. After looking at the results,we can state that our model achieved state-of-the-art results, with a performance close to FactorVAE Kim & Mnih (2018) . This is not the first time that a model draws ideas from information theory. Many models Tishby et al. (1999) ; Alemi et al. (2016) ; Achille & Soatto (2016) used the information bottleneck presented in Tishby et al. (1999) using the VAE framework. Therefore, we truly believe that we should keep looking at the neuroscience and information theory fields for inspiration. In general, we don't need to replicate or simulate biological models; however we should analyse the intuition about the known main mechanisms of our brain and adapt those to our models.

7. APPENDIX:



7.1. DETAILS OF TRAINING:

We trained the Non-Syn VAE model for 150,000 steps using the d-sprites data set (unsupervised setting) obtaining the best result using the following parameters:

• Optimiser: Adam

• Learning rate: 1e-4

• Beta 1 (Adam): 0.9

• Beta 2 (Adam): 0.999

• Batch size: 64

• Discount ω of greedy policy: 0.9 return worst index

:



Figures and Tables:

Figure 1: Structure of total information of two variables about S

Figure 2: Left: Traverse of latents (110k steps). Right: Mean activations (110k steps)

Figure 3: Synergy loss for Factor VAE -4k steps As a comparison, we also show the synergy for the Non-Syn VAE for the same number of steps in Fig 4. Surprisingly, Factor VAE minimises the Synergy implicitly by penalising the Total correlation term.

Figure 4: Synergy loss for Non-Syn VAE -4k steps

Weight of the Synergy loss α: 5.0 7.2 ARCHITECTURE OF NON-SYN VAE -DSPRITES Algorithm 4 get index greedy Input: mu µ, log std. deviation σ, discount factor ω Output: list of indices of the latents with the lowest Specific Mutual Information function GET INDEX GREEDY(mu, logvar, ω)

Table content: <table><tr><td colspan="2">Algorithm 3 generate candidate</td></tr><tr><td colspan="2">Input: dimension of Z, index</td></tr><tr><td colspan="2">Output: List of indices</td></tr><tr><td colspan="2">1: function GENERATE CANDIDATE(d, index)</td></tr><tr><td>2:</td><td>if len(d) = 0 then</td></tr><tr><td>3:</td><td>candidates ← [1 : 10]</td></tr><tr><td>4:</td><td>else</td></tr><tr><td>5:</td><td>candidates ← [1 : 10] not in index</td></tr><tr><td>6:</td><td>return candidates</td></tr><tr><td/><td>Encoder</td><td>Decoder</td></tr><tr><td/><td>Input 64 x 64 binary image</td><td>Input</td></tr><tr><td/><td>4 x 4 conv.32 ReLU, stride 2</td><td>FC. 128 ReLU</td></tr><tr><td/><td>4 x 4 conv.32 ReLU, stride 2</td><td>FC. 4 x 4 x 64 ReLU.</td></tr><tr><td/><td colspan="2">4 x 4 conv.64 ReLU, stride 2 4 x 4 upconv.64 ReLU, stride 2</td></tr><tr><td/><td colspan="2">4 x 4 conv.64 ReLU, stride 2 4 x 4 upconv.32 ReLU, stride 2</td></tr><tr><td/><td>FC. 128</td><td>4 x 4 upconv.32 ReLU, stride 2</td></tr><tr><td/><td>FC. 2x10</td><td>4 x 4 upconv.1, stride 2</td></tr><tr><td colspan="2">7.3 DETAILS OF ALGORITHM</td></tr><tr><td colspan="2">Algorithm 2 Imax</td></tr><tr><td colspan="2">Input: Compute specific mutual information</td></tr><tr><td colspan="2">Output:</td></tr><tr><td>2:</td><td>mu syn ← mu[:, index]</td></tr><tr><td>3:</td><td>logvar syn ← logvar[:, index]</td></tr><tr><td>4: 6:</td><td>index ← best index + [c]</td></tr><tr><td>5:</td><td colspan="2">Imax ← compute KL(mu syn, logvar syn) Imax new ← Imax(µ, σ, index)</td></tr><tr><td>6: 8:</td><td/></tr><tr><td>7:</td><td colspan="2">return Imax if Imax new  *  ω &gt; I max best then</td></tr><tr><td>10:</td><td>Imax best ← Imax new</td></tr><tr><td/><td>best index ← index</td></tr><tr><td>12:</td><td/></tr><tr><td/><td colspan="2">worst index ← [1 : 10] not in best index</td></tr><tr><td>14:</td><td/></tr></table>

Peer review (iclr2019):
 This paper proposes a new approach to enforcing disentanglement in VAEs using a term that penalizes the synergistic mutual information between the latent variables, encouraging representations where any given piece of information about a datapoint can be garnered from a single latent. In other words, representations where there is no information conveyed by combinations of latents that is not conveyed by considering each latent in isolation. As the resultant target is intractable to evaluate, a number of approximations are employed for practical training. The high-level idea is quite interesting, but the paper itself is quite a long way of convincing me that this is actually a good approach. Moreover, the paper is a long way of the level of completeness, rigor, clarity, and polish that is required to seriously consider it for publication. In short, the work is still at a relatively early stage and a lot more would need to be done for it to attain various minimum standards for acceptance. A non-exhaustive list of specific examples of its shortfalls are given below. 1. The paper is over a page and a half under length, despite wasting large amounts of space (e.g. figures 3 and 4 should be two lines on the same plot) 2. The experimental evaluation is woefully inadequate. The only quantitative assessment is to compare to a single different approach on a single toy dataset, and even then the metric being used is the one the new method uses to train for making it somewhat meaningless. 3. The introduction is completely generic and says nothing about the method itself, just providing a (not especially compelling) motivation for disentanglement in general. In fact, the motivation of the introduction is somewhat at odds with the work -- correctly talking about the need for hierarchical representations which the approach actually actively discourages. 4. There are insufficient details on the algorithm itself in terms of the approximations that are made to estimate the synergistic mutual information. These are mostly glossed over with only a very short explanation in the paragraph after equation 15. Yes there are algorithm blocks, but these are pretty incomprehensible and lack accompanying text. In particular, I cannot understand what A_w is supposed to be. This is very important as I suspect the behavior of the approximation is very different to the true target. Similarly, it would be good to provide more insight into the desired target (i.e. Eq 15). For example, I suspect that it will encourage a mismatch between the aggregate posterior and prior by encouraging higher entropy on the former, in turn causing samples from the generative model to provide a poor match to the data. 5. The repeated claims of the approach and results being "state-of-the-art" are cringe-worthy bordering on amusing. Writing like this serves no purpose even when it justified, and it certainly is not here. 6. There are a lot of typos throughout and the production values are rather poor. For example, the algorithm blocks which are extremely messy to the point where they are difficult to follow, citep/citet mistakes occur almost every other citation, there is a sign error in Equation 16. This is a piece of work in an exciting research area that, with substantial extra work, could potentially result in a decent paper due to fact that the core idea is simple and original. However, it is a long way short of this in its current state. Along with addressing the specific issues above and improving the clarity of the work more generally, one thing in particular that would need to address in a resubmission is a more careful motivation for the method (ideally in the form of a proper introduction). Though I appreciate this is a somewhat subjective opinion, for me, penalizing the synergistic information is probably actually a bad thing to do when taking a more long-term view on disentanglement. Forcing simplistic representations where no information is conveyed through the composition of latents beyond that they provide in isolation is all well and good for highly artificial and simplistic datasets like dsprites, but is clearly not a generalizable approach for larger datasets where no such simplistic representation exists. As you say in the first line of your own introduction, hierarchy and composition are key parts of learning effective and interpretable representations and this is exactly what you are discouraging. A lot of the issue here is one of the disentanglement literature at large rather than this paper (though I do find it to be a particularly egregious offender) and it is fine to have different opinions. However, it is necessary to at least make a sensible case for why your approach is actually useful. Namely, is there actually any real applications where such a simplistic disentanglement is actually useful? Is there are anyway the current works helps in the longer vision of achieving interpretable representations? When and why is the synergistic information a better regularizer than, for example, the total correlation? The experiments you have do not make any inroads to answering these questions and there are no written arguments of note to address them. I am not trying to argue here that there isn*t a good case to be made for the suggested approach in the context of these questions (though I am suspicious), just that if the work is going to have any lasting impact on the community then it needs to at least consider them.