Title: NON-SYN VARIATIONAL AUTOENCODERS

Abstract: Learning disentangling representations of the independent factors of variations that explain the data in an unsupervised setting is still a major challenge. In the following paper we address the task of disentanglement and introduce a new state-ofthe-art approach called Non-synergistic variational Autoencoder (Non-Syn VAE). Our model draws inspiration from population coding, where the notion of synergy arises when we describe the encoded information by neurons in the form of responses from the stimuli. If those responses convey more information together than separate as independent sources of encoding information,they are acting synergetically. By penalizing the synergistic mutual information within the latents we encourage information independence and by doing that disentangle the latent factors. Notably, our approach could be added to the VAE framework easily, where the new ELBO function is still a lower bound on the log likelihood p x . In addition, we qualitatively compare our model with Factor VAE and show that this one implicitly minimises the synergy of the latents.

1. INTRODUCTION:

Our world is hierarchical and compositional, humans can generalise better since we use primitive concepts that allow us to create complex representations (Higgins et al. (2016) ). Towards the creation of truly intelligent systems, they should learn in a similar way resulting in an increase of their performance since they would capture the underlying factors of variation of the data ( Bengio et al. (2013) ; Hassabis et al. (2017) ; Botvinick et al. (2017) ). In addition, good representations improve the performance for tasks involving transfer learning and multi-task learning; since it will capture the explanatory factors.

According to Lake et al. (2016) , a compositional representation should create new elements from the combination of primitive concepts resulting in a infinite number of new representations. For example if our model is trained with images of white wall and then is presented a boy with a white shirt, it should identify the color white as a primitive element. Intuitively, our model will be able to construct different and multiple representations from the primitives. Furthermore, a disentangled representation has been interpreted in different ways, for instance Bengio et al. (2013) define it as one where single latent variables are sensitive to changes in generative factors, while being invariant to changes in other factors. In addition, we agree with Higgins et al. (2017a) , which mentions that a disentangle representation should be factorised and interpretable. Intuitevely, the model could learn generative factors such as position, scale or colour; if it is disentangle it should be able to traverse along the position variable without changing the scale or the colour. It's worth noting that disentangled representations have been useful for a variety of downstream tasks such as domain adaptation by training a Reinforcement Learning agent that uses a disentangled representation of its environment Higgins et al. (2017b) ; or for learning disentangled primitives grounded in the visual domain discovered in an unsupervised manner Higgins et al. (2017c) .

2. RELATED WORK:

The original Variational autoencoder (VAE) framework (Kingma & Welling (2013) ; Rezende et al. ( 2014)) has been used extensively for the task of disentanglement by modifying the original ELBO formulation; for instance β-VAE is presented in Higgins et al. (2017a) which increases the latent capacity by penalising the KL divergence term with a β hyperparameter. In addition, Kim & Mnih (2018) achieved a more robust disentangled representation by using the model called Factor VAE which penalises the total correlation of the latent variables encouraging the independence of the latents; a similar approach is shown in Chen et al. (2018) , where they present a clever ELBO decomposition based on Hoffman & Johnson (2016) . Other approaches rely on information bottleneck presented in Tishby et al. (1999) to model frameworks for this task such as Alemi et al. (2016) ; Achille & Soatto (2016) . Furthermore, Chen et al. (2016) describe a model based on Generative Adversarial Networks (Goodfellow et al. (2014) ) by encouraging the mutual information between the latents and the output of the generator. Notably, in Higgins et al. (2016) , they describe comprehensively the β-VAE model using a neuroscience and information theory approaches; they suggest that by encouraging redundancy reduction the model achieves statistical independence within the latents. This model inspired us to look into different fields for new ways to enforce disentanglement of the latents.

3. SYNERGY:

To understand our model, we need first to describe Synergy (Gat & Tishby (1998) ; Schneidman et al. (2003) ) being a popular notion of it as how much the whole is greater than the sum of its parts. It's common to describe it with the XOR gate, since we need two independent variables to fully specified the value of the output. Following, we describe the synergy from two related fields.

3.1. INFORMATION THEORY APPROACH:

Computing the multivariate synergistic information is an ongoing topic of research Schneidman et al. (2003) ; Williams & Beer (2010) ; Bertschinger et al. (2012) ; Griffith & Koch (2012) . Most of the current research in this topic uses the Partial information diagram described by Williams & Beer (2010) . In order to understand the importance of the Synergy information in our framework it's essential to describe the relations with the Unique and Redundant Information. Introducing the notation from Williams & Beer (2010) , let's consider the random variable S and a random vector R = {R 1 , R 2 , .., R n }, being our goal to decompose the information that the variable R provides about S; the contribution of these partial information could come from one element from R 1 or from subsets of R (ie. R 1 , R 2 ). Considering the case with two variables, {R 1 , R 2 }, we could separate the partial mutual information in unique (U nq(S; R 1 ) and U nq(S; R 2 ) ), the information that only R 1 or R 2 provides about S is redundant (Rdn(S; R 1 , R 2 )), which it could be provided by R 1 or R 2 ; and synergistic (Syn(S; R 1 , R 2 )), which is only provided by the combination of R 1 and R 2 . The figure 1 depicts the decomposition; this diagram is also called PI-diagram (Partial information) . Additional notation is used for a better visualisation in the case of more variables: Unique {1},{2}; Redundant {1}{2} and Synergistic {12}. For the case of 2 variables (X 1 , X 2 ), we expect four contributions to the mutual information as described in Bertschinger et al. (2012) ; Olbrich et al. (2015) :

I(S; R 1 , R 2 ) = SI(S; R 1 , R 2 ) Redundant + U nq(S; R 1 \ R 2 ) Unique + U nq(S; R 2 \ R 1 ) Unique + Syn(S; R 1 , R 2 ) Synergistic (1)

It's easy to see that the number of terms increases exponentially as the number of sources increases. The best measure for synergy is an ongoing topic of research. In the subsection we are going to talk about the synergy metrics.

3.2. POPULATION CODING APPROACH:

For neural codes there are three types of independence when it comes to the relation between stimuli and responses; which are the activity independence, the conditional independence and the information independence. One of the first measures of synergy for sets of sources of information came from this notion of independence. In Williams & Beer (2010) it is stated that if the responses come from different features of the stimulus, the information encoded in those responses should be added to estimate the mutual information they provide about the stimulus. Formally:

I(S; R 1 , R 2 ) = I(S; R 1 ) + I(S; R 2 ) (2)

However, we just saw in the previous sections that the I(S; R 1 ) and I(S; R 2 ) could be decomposed in their unique and redundant and synergistic terms. Intuitively, this formulation only holds if there is no redundant or synergistic information present; which means in the context of population coding that the responses encoded different parts of the stimulus. If the responses R 1 and R 2 convey more information together than separate, we can say we have synergistic information; if the information is less, we have redundant information. That's the reason why in Gat & Tishby (1998) , the synergy is considered as measure of information independence:

Syn(R 1 , R 2 ) = I(S; R 1 , R 2 ) -I(S; R 1 ) -I(S; R 2 ) (3)

3.3. SYNERGY METRIC:

First we need to state the notation (the same as Griffith & Koch (2012) ):

• n: Number of individual predictors X i • A i : subset of individual predictors (ie. A i = {X 1 , X 3 }) • X: Joint random variable of all individual predictors X 1 X 2 ..X n • {X 1 , X 2 , ..., X n }: Set of all the individual predictors • Y: Random variable to be predicted • y: A particular outcome of Y.

The intuition behind this metric is that synergy should be defined as the "whole beyond the maximum of its parts". The whole is described as the mutual information between the joint X and the outcome Y; whereas the maximum of all the possible subsets is interpreted as the maximum information that any of the sources A i provided about each outcome. Formally, this is stated as:

EQUATION

This metric derives from Williams & Beer (2010) and Griffith & Koch (2012) , however one of the differences with this metric with the one presented in Griffith & Koch (2012) is that in this one we are considering the specific mutual information I max in a group of latents A i , whereas in the paper mentioned it considers only an individual latent. Notably the I max can be expressed in terms of the KL divergence.

EQUATION

Putting together the equation 7 and 5, we have the following:

EQUATION

In the following section we are going to use the intuition provided by the above equation in the VAE framework for the task of disentanglement.

4. MODEL DERIVATION:

The motivation of our contribution is inspired in this concept and driven by the belief that synergy is not desirable for the task of disentanglement, since we want the latents to be independently informative as possible about the data, instead of needing many latents to specify the data. Therefore, we argue that by penalising the synergistic information within the latents and the data, we would encourage the disentanglement of the underlying factors of variation. This hypothesis is also inspired in the information presented in Griffith & Koch (2012) ; Gat & Tishby (1998) , where it is stated that the synergy is a measure of independence information in the responses of the stimuli.

First, we need to change the notation to match the VAE framework notation (Z are the latents and X is the observations). Also, A i is a subset of the latents, such that A i ∈ {Z 1 , Z 2 , ..., Z n } and Z is the joint of the latents. Formally: Z = d i Z i , where d is the number of dimensions of the latent variables. Besides, from the VAE standard framework, we know that the distribution p(A i |x) is intractable which is why we need to use an approximate distribution q φ (A i |x) parametrised by φ parameters. It's important to notice that this KL divergence could be computed in the same way as in the VAE framework; the only difference is the number of dimensions used for the random variable z. In the original VAE framework, we compute the KL divergence considering the joint Z = d i Z i ; whereas for the Synergy metric we don't use the joint but a subset of the latents. For instance, if A i = Z 2 Z 5 Z 8 , we have the following expression:

EQUATION

Taking in account these considerations, we express the equation 8 as follows:

S max ({Z 1 , Z 2 , ..., Z d }; X) = I(Z; X) - x∈X p(X = x) max i D KL q φ (A i | x) p(A i ) (10)

We start with the original ELBO formulation (Rezende et al. (2014) ; Kingma & Welling ( 2013)) and add the penalised term corresponding to the synergy, where α is a hyperparameter:

EQUATION

Expanding the S max term, we have the llowing:

L new (θ, φ, x) = L elbo (θ, φ, x) -α(I(z; x) - x∈X p(X = x) max i D KL q φ (A i | x) p(A i ) ) (13)

From Hoffman & Johnson (2016), we know that the KL term in the ELBO loss is decomposed in D KL q φ (z n ) p(z n ) + I(x n ; z) when we use the aggregate posterior and define the loss over the empirical distribution of the data p data (x). Taking in account that, we can express the equation 12 as follows:

EQUATION

If we penalise the synergy (see Eq. 12), we will be penalising the mutual information term which is not desirable for this task Kim & Mnih (2018) ; we can see this effect explicitly in Eq. 14. Therefore, we use only the second term to perform the optimisation which means maximising the subset of latents with the most amount of MI per outcome.

EQUATION

It's easy to see in Eq. 15 that it's not a guaranteed lower bound on the log likelihood p x anymore, which is why we decided to penalise the subset of latents with the minimum specific mutual information (ie. A w ). In practice we found that computing the maximum subset A i for each outcome of x is too computational intensive, which is why we decided to use a mini-batch approximation, which is the version we show in the pseudo-code using a two step optimisation in the next section.

The final version we are going to use is the one below, where Imax is the KL term of the synergy term.

EQUATION

Algorithm 1 Non Syn VAE Input: Observations (x (i) ) N i=1 , batch size m, latent dimension d, weight of synergy loss α, discount factor ω, optimiser optim, reparametrisation function g φ . θ, φ ← Initialise VAE parameters repeat 3:

x (i) ← Random minibatch B of size m, i ∈ B z i ← g φ ( , x (i) ) Sample z i ∼ q φ (z|x) 6: φ, θ ← optim(∇ θ,φ L elbo (θ, φ; x)) : Gradients of ELBO minibatch x (i) ← Random minibatch B' of size m, i ∈ B mu, logvar ← Encoder(x (i) , φ) 9:

worst index ← get index greedy(mu, logvar, ω) L syn ← α * Imax(mu, logvar, worst index) See Eq.16 for Imax function φ ← optim(∇ φ L syn (φ; x (i) ))

: Gradients of Syn loss minibatch 12: until convergence of objective In the algorithm shown above we see in practice that we get better results when we sample the values of mu and logvar from the encoder for the step 2 of the optimisation. We use a greedy approximation of the best latents by following a greedy policy (See Appendix).

5. EXPERIMENTS:



5.1. LATENT TRAVERSALS:

For disentanglement, the dataset most commonly used is the dsprites dataset Matthey et al. (2017) , which consists on 2D shapes generated from independent latent factors. We used the same architecture and optimizer as Factor VAE Kim & Mnih (2018) for training our model. In order to test qualitatively the Non-Syn VAE model, we decided to tranverse the latents and plot the mean activations of the latents.

In Figure 2 (left), we see clearly that our model disentangles the factors of variation. Likewise, on the right we see the mean activation of each active latent averaged across shapes, rotations and scales. After looking at the figure above we can state that our model achieves state-of-the-art results using a qualitatively benchmark. Interestingly, both models perform quite similar in this test.

5.2. SYNERGY IN FACTOR VAE:

Also, we decided to compute the same synergy term from the Non-Syn VAE in Factor VAE (just compute it, we didn't use it for training). The hypothesis was that if Factor VAE achieves disentanglement, it should minimise the synergy as well. We train Factor VAE using the same parameters and architecture described in Kim & Mnih (2018) . We show the first 4000 steps for the Synergy term (i.e. α * Imax), since most of the interaction happens in the first steps. In this paper we presented the intuition and derivation of the lower bound of a model that uses a novel approach inspired by the information theory and Neuroscience fields to achieve the disentanglement of the underlying factor of variations in the data. After looking at the results,we can state that our model achieved state-of-the-art results, with a performance close to FactorVAE Kim & Mnih (2018) . This is not the first time that a model draws ideas from information theory. Many models Tishby et al. (1999) ; Alemi et al. (2016) ; Achille & Soatto (2016) used the information bottleneck presented in Tishby et al. (1999) using the VAE framework. Therefore, we truly believe that we should keep looking at the neuroscience and information theory fields for inspiration. In general, we don't need to replicate or simulate biological models; however we should analyse the intuition about the known main mechanisms of our brain and adapt those to our models.

7. APPENDIX:



7.1. DETAILS OF TRAINING:

We trained the Non-Syn VAE model for 150,000 steps using the d-sprites data set (unsupervised setting) obtaining the best result using the following parameters:

• Optimiser: Adam

• Learning rate: 1e-4

• Beta 1 (Adam): 0.9

• Beta 2 (Adam): 0.999

• Batch size: 64

• Discount ω of greedy policy: 0.9 return worst index

:



Figures and Tables:

Figure 1: Structure of total information of two variables about S

Figure 2: Left: Traverse of latents (110k steps). Right: Mean activations (110k steps)

Figure 3: Synergy loss for Factor VAE -4k steps As a comparison, we also show the synergy for the Non-Syn VAE for the same number of steps in Fig 4. Surprisingly, Factor VAE minimises the Synergy implicitly by penalising the Total correlation term.

Figure 4: Synergy loss for Non-Syn VAE -4k steps

Weight of the Synergy loss α: 5.0 7.2 ARCHITECTURE OF NON-SYN VAE -DSPRITES Algorithm 4 get index greedy Input: mu µ, log std. deviation σ, discount factor ω Output: list of indices of the latents with the lowest Specific Mutual Information function GET INDEX GREEDY(mu, logvar, ω)

Table content: <table><tr><td colspan="2">Algorithm 3 generate candidate</td></tr><tr><td colspan="2">Input: dimension of Z, index</td></tr><tr><td colspan="2">Output: List of indices</td></tr><tr><td colspan="2">1: function GENERATE CANDIDATE(d, index)</td></tr><tr><td>2:</td><td>if len(d) = 0 then</td></tr><tr><td>3:</td><td>candidates ← [1 : 10]</td></tr><tr><td>4:</td><td>else</td></tr><tr><td>5:</td><td>candidates ← [1 : 10] not in index</td></tr><tr><td>6:</td><td>return candidates</td></tr><tr><td/><td>Encoder</td><td>Decoder</td></tr><tr><td/><td>Input 64 x 64 binary image</td><td>Input</td></tr><tr><td/><td>4 x 4 conv.32 ReLU, stride 2</td><td>FC. 128 ReLU</td></tr><tr><td/><td>4 x 4 conv.32 ReLU, stride 2</td><td>FC. 4 x 4 x 64 ReLU.</td></tr><tr><td/><td colspan="2">4 x 4 conv.64 ReLU, stride 2 4 x 4 upconv.64 ReLU, stride 2</td></tr><tr><td/><td colspan="2">4 x 4 conv.64 ReLU, stride 2 4 x 4 upconv.32 ReLU, stride 2</td></tr><tr><td/><td>FC. 128</td><td>4 x 4 upconv.32 ReLU, stride 2</td></tr><tr><td/><td>FC. 2x10</td><td>4 x 4 upconv.1, stride 2</td></tr><tr><td colspan="2">7.3 DETAILS OF ALGORITHM</td></tr><tr><td colspan="2">Algorithm 2 Imax</td></tr><tr><td colspan="2">Input: Compute specific mutual information</td></tr><tr><td colspan="2">Output:</td></tr><tr><td>2:</td><td>mu syn ← mu[:, index]</td></tr><tr><td>3:</td><td>logvar syn ← logvar[:, index]</td></tr><tr><td>4: 6:</td><td>index ← best index + [c]</td></tr><tr><td>5:</td><td colspan="2">Imax ← compute KL(mu syn, logvar syn) Imax new ← Imax(µ, σ, index)</td></tr><tr><td>6: 8:</td><td/></tr><tr><td>7:</td><td colspan="2">return Imax if Imax new  *  ω &gt; I max best then</td></tr><tr><td>10:</td><td>Imax best ← Imax new</td></tr><tr><td/><td>best index ← index</td></tr><tr><td>12:</td><td/></tr><tr><td/><td colspan="2">worst index ← [1 : 10] not in best index</td></tr><tr><td>14:</td><td/></tr></table>

Peer review (iclr2019):
 The authors aim at training a VAE that has disentangled latent representations in a "synergistically" maximal way. For this they use one (of several possible) versions of synergy defintions and create a straight forward penalization term for a VAE objective (roughly the whole mutual information minus the maximum mutual information of its parts). They train this VAE on one dataset, namely dsprites, and compare it to a VAE with total correlation penalization. The paper is well written and readable. The idea of using synergy is an important step forward in understanding complex models. The concept of synergy has great potential in machine learning and is highly relevant. The main concepts of synergy are not developed in this paper and the used penalization term is straight forward. The number of experiments conducted and comparisons done is quite limited. Also the potential of synergy is not really demonstrated, e.g. for representation learning, causality, etc., and appears here ad hoc. Also why one should use the authors* suggested penalization term instead of total correlation is not discussed, nor demonstrated as they perform similarly on both disentanglement and synergy loss. I hope the authors find more relevant applications or data sets in the future to demonstrate the importance of synergy.