Topographic VAEs learn Equivariant Capsules
Authors: T. Anderson Keller
Max Welling
Venue: NeurIPS 2021 Poster (2021)
Keywords: Topographic Organization
Unsupervised
Equivariance
Variational Inference
Deep Generative Model
Disentanglement

Abstract: In this work we seek to bridge the concepts of topographic organization and equivariance in neural networks. To accomplish this, we introduce the Topographic VAE: a novel method for efficiently training deep generative models with topographically organized latent variables. We show that such a model indeed learns to organize its activations according to salient characteristics such as digit class, width, and style on MNIST. Furthermore, through topographic organization over time (i.e. temporal coherence), we demonstrate how predefined latent space transformation operators can be encouraged for observed transformed input sequences -- a primitive form of unsupervised learned equivariance. We demonstrate that this model successfully learns sets of approximately equivariant features (i.e. "capsules") directly from sequences and achieves higher likelihood on correspondingly transforming test sequences. Equivariance is verified quantitatively by measuring the approximate commutativity of the inference network and the sequence transformations. Finally, we demonstrate approximate equivariance to complex transformations, expanding upon the capabilities of existing group equivariant neural networks. 

Topographic Es learn Equivariant Capsules
T. Anderson Keller UvA-Bosch Delta Lab University of Amsterdam anderson, keller@gmail.com
Max Welling UvA-Bosch Delta Lab University of Amsterdam m. m.welligguuv.
Abstract
In this work we seek to bridge the concepts of topographic organization and equiv- ariance in neural networks. accomplish this, we introduce the Topographic VAE: novel method for efficiently training deep generative models with topographically organized latent variables. We show that such model indeed learns to organize its activations according salient characteristics such digit class, width, and style on MNIST. Furthermore, through topographic organization over time (i.c. temporal coherence), we demonstrate how predefined latent space transformation operators can be encouraged for observed transformed input sequences a primitive form unsupervised learned equivariance. We demonstrate that this model successfully learns sets of approximately equivariant features (i.e. "capsules") directly from sequences and achieves higher likelihood correspondingly transforming sequences. Equivariance is verified quantitatively by measuring the approximate commutativity of the inference network and the sequence transformations. Finally, we demonstrate approximate equivariance to complex transformations, expanding upon the capabilities of existing group equivariant neural networks.
Introduction
Many parts of the brain organized topographically. Famous examples the ocular dominance maps and the orientation maps in V1. What is the advantage of such organization and what can we learn from develop better inductive biases for deep neural network architectures?
7
7
Tg
Br (Uglx))))
Pa(X1o (10)
Rollio
-1-----11------1111
Figure Overview of the Topographic VAE with shifting temporal coherence. The combined color/rotation transformation in input space Tg becomes encoded as Roll within the capsule dimension. The model is thus able decode unseen sequence elements by encoding partial sequence and Rolling activations within the capsules. We see this resembles commutative diagram.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

One potential explanation for the emergence of topographic organization is provided by the principle of redundancy reduction M. In the language of Information Theory, redundancy wastes channel capacity, and thus to represent information as efficiently as possible, the brain may strive to transform the input to a neural code where the activations are statistically maximally independent. In the machine learning literature, this idea resulted in Independent Component Analysis (ICA) which linearly transforms the input to a new basis where the activities are independent and sparse 13 29 44). was soon realized that there are remaining higher order dependencies (such correlation between absolute values) that can not be transformed away by a linear transformation. For example, along edges of an image, linear-ICA components (e.g. gabor filters) still activate in clusters even though the sign of their activity is unpredictable [48] 56). This led to new algorithms that explicitly model these remaining dependencies through topographic organization of feature activations [27 45 46 59].
Such topographic features were reminiscent of pinwheel structures observed in V1, encouraging multiple comparisons with topographic organization in the biological visual system 28 30 42]
second, almost independent body of literature developed the idea of "equivariance" of neural network feature maps under symmetry transformations. The idea of equivariance is that symmetry transformations define equivalence classes as the orbits of their transformations, and we wish to maintain this structure in the deeper layers of a neural network. For instance, for images, asserting rotated image contains the same object for all rotations, the transformation of rotation then defines an orbit where the elements of that orbit can be interpreted pose angular orientation. When image is processed by a neural network, we want features at different orientations to be able to be combined to form new features, but want to ensure the relative pose information between the features is preserved for all orientations. This has the advantage that the equivalence class of rotations for the complex composite features is guaranteed to be maintained, allowing for the extraction of invariant features, a unified pose, and increased data efficiency. Such ideas are reminiscent of the capsule networks of Hinton al. [21 22 51), and indeed formal connections to equivariance have been made [39]. Interestingly, by explicitly building neural networks to be equivariant, we additionally see geometric organization of activations into these equivalence classes, and further, the elements within an equivalence class are seen to exhibit higher-ordet non- Gaussian dependencies [40 41 [66 57). The insight of this connection between topographic organization and equivariance hints at a possibility to encourage approximate equivariance from an induced topology in feature space.
To build a model, we need to ask what mechanisms could induce topographic organization of observed transformations specifically? We have argued that removing dependencies between latent variables is possible mechanism; however, to obtain the more structured organisation of equivariant capsule representations, the usual approach is to hard-code this structure into the network, or to encourage it through regularization terms to incorporate another key inductive bias: "temporal coherence" [8]
4 [5 To achieve this same structure unsupervised, we propose
60] The principle of temporal coherence, or "slowness". asserts than when processing correlated sequences, we wish for our representations to change smoothly and slowly over space and time. Thinking of time sequences as symmetry transformations on the input, we desire features undergoing such transformations to be grouped into equivariant capsules. We therefore suggest that encouraging slow feature transformations to take place within a capsule could induce such grouping from sequences alone.
24
52
In the following sections we will explain the details of our Topographic Variational Autoencoder which lies at the intersection of topographic organization, equivariance, and temporal coherence, thereby learning approximately equivariant capsules from sequence data completely unsupervised.
2 Related Work
The history of statistical models upon which this work builds is vast, including sparse coding [44].
Independant Component Analysis (ICA) 22 13 29), Slow Feature Analysis (SFA) [54] 60]. and Gaussian scale mixtures 4T 48
56
57 Most related to this work are topographic generative
models including Generative Topographic Maps 6). Bubbles 25), Topographic ICA (27), and
the Topographic Product of Student's- s-t [66 559. Prior work on learning equivariant and invariant representations is similarly vast and also has a deep relationship with these generative models. Specifically, Independant Subspace Analysis 26 53), models involving temporal coherence [88 24 [52mg) 60) and Adaptive Subspace Self Organizing Maps 35 have all demonstrated the ability to learn invariant feature subspaces and even 'disentangle" space and time [19 (53. Our work assumes similar generative model to these works while additionally allowing for efficient estimation of
2

---

Although our work is not the first to combine Student's-t s-t distributions and variational inference 77 is the first to provide an efficient method to do so for Topographic Student's s-t distributions.
the model through variational inference 133 50].
Another line of work has focused on constructing neural networks with equivariant representations separate from the framework of generative modeling. Analytically equivariant networks such as
Group Equivariant Neural Networks [II]. and other extensions [ 16
17
55
49
58
61
62 propose
to explicitly enforce symmetry to group transformations in neural networks through structured weight sharing. Alternatively, others propose supervised and self supervise methods for learning
equivariance or invariance directly from the data itself [4 [4 15]. One related example in this category uses a group sparsity regularization term to similarly learn topographic features for the purpose of modeling invariance [31]. We believe the Topographic Variational Autoencoder presented in this paper is another promising step in the direction of learning approximate equivariance, and may even hint at how such structure could be learned in biological neural networks.
Furthermore, the idea of disentangled representations [3] has also been been connected to cquivariance and representation theory in multiple recent papers 12 10 200. Our work shares a fundamental connection to this distributed operator definition of disentanglement where the slow roll of capsule activations can be seen as the latent operator. Recently, the authors of [34] demonstrated that incorporating the principle of 'slowness' in variational autoencoder ( (VAE) yields the ability to learn disentangled representations from natural sequences. While similar in motivation, the generative model proposed in 33 unrelated to topographic organization and equivariance, and is more aligned with traditional notions of disentanglement.
Finally, and importantly, in the neuroscience literature, another popular explanation for topographic organization arises as the solution to the 'wiring length' minimization problem [36). Recently, models which attempt to incorporate wiring length constraints have been shown to yield topographic orga- nization of higher level features, ultimately resembling the 'face patches' found in primates [32] 38) Interestingly, the model presented in this paper organizes activity based on the same statistical prop- erty (local correlation) as the wiring length proxies developed in [38], but from a generative modeling perspective, demonstrating a computationally principled explanation for the same phenomenon.
3 Background
The model in this paper is a first attempt at bridging two yet disjoint classes of models: Topographic Generative Models, and Equivariant Neural Networks. In this section, we will provide a brief background on these two frameworks.
3.1 Topographic Generative models
Inspired by Topographic ICA, the class of Topographic Generative models can be understood as gener- ative models where the joint distribution over latent variables does not factorize into entirely indepen- dent factors, as is commonly done in ICA or VAEs, but instead has more complex 'local' correlation structure. The locality is defined by arranging the latent variables into an n-dimensional lattice or grid, and organizing variables such that those which are closer together on this grid have greater correlation of activities than those which are further apart. In the related literature, activations which are nearby in this grid are defined to have higher-order correlation, e.g. correlations of squared activations (aka 'energy"), ) asserting that all first order correlations are removed by the initial ICA de-mixing matrix.
Such generative models can be seen as hierarchical generative models where there exist higher level independent 'variance generating" variables V which are combined locally to generate the variances o of (V of the lower level topographic variables T ~ NO, o²)), for an appropriate non-linearity c. The variables T are thus independent conditioned on a. Other related models which can be described under this umbrella include Independent Subspace Analysis (ISA) 26 where all variables within a predefined subspace (or 'capsule') share common variance, and 'temporally coherent' models [24] where the energy of a given variable between time steps is correlated by extending the topographic neighborhoods over the time dimension (25). The topographic latent variable T can additionally be described as an instance of a Gaussian scale mixture (GSM). GSMs have previously been used to model the observed non-Gaussian dependencies between coefficients of stecrable wavelet pyramids (interestingly also equivariant to translation & rotation) [88 56 57).
3

---

3.2 Group Equivariant Neural Networks
Equivariance is the mathematical notion of symmetry for functions. A function is said to be an equivariant map if the the result of transforming the imput and then computing the function is the same as first computing the function and then transforming the output. In other words, the function and the transformation commute. Formally. [(]x[x]) = T [x)],
where and I denote the (potentially different) operators on the domain and co-domain respectively, but are indexed by the same element p.
is well known that convolutional maps in neural networks are translation equivariant, i.e., given translation r (applied to each feature map separately) and a convolutional map f(),, we have [(]x[x]) (x)]. This can be extended to other transformations (e.g. rotation or mirroring) using Group convolutions (G-convolutions) M.. As result of the design of G-convolutions, feature maps that are related to each other by a rotation of the filter/input are grouped together. Moreover, a rotation of the imput results in a transformation (i.e. permutation and rotation) on the activations of each of these groups in the output. Hence, we can think of these equivalence class groups capsules where vransformations of the input only cause structured transformations within a capsule. As we will demonstrate later, this is indeed analogous to the structure of the representation learned by the Topographic VAE with temporal coherence transformation of the input yields cyclic permutation of activations within each capsule. However, due to the approximate learned nature of the equivariant representation, the Topographic VAE does not require the transformations Tp to constitute a group.
The Generative Model
The generative model proposed in this paper is based on the Topographic Product of Student's s-t (TPoT) model as developed in [46 [59]. In the following, we will show how a TPoT random variable can be constructed from set of independent univariate standard normal random variables, enabling efficient training through variational inference. Subsequently, we will construct a new model where topographic neighborhoods are extended over time, introducing temporal coherence and encouraging the unsupervised learning of approximately equivariant subspaces we call 'capsules'.
4.1 The Product of Student's-4 s-t Model
We assume that that our observed data is generated by latent variable model where the joint distribution over observed and latent variables x and t factorizes into the product of the conditional and the prior. The prior distribution Pr(t) is assumed to be Topographic Product of Student's s-t (TPoT) distribution, and we parameterize the conditional distribution with a flexible function approximator:
(x, t) (x(r(x(t)pr(())
(x|](x|t) Po (t)
Pr(t) TPoT(t;
(1)
The goal of training is thus to learn the parameters 0 such that the marginal distribution of the model (x) matches that of the observed data. Unfortunately, the marginal likelihood generally intractable except for all but the simplest choices of go and pr (45). Prior work has therefore resorted to techniques such as contrastive divergence with Gibbs sampling (59] to train TPoT models as energy based models. In the following section, we instead demonstrate how TPoT variables can be constructed as a deterministic function of Gaussian random variables, enabling the use of variational inference and efficient maximization of the likelihood through the evidence lower bound (ELBO).
4.2 Constructing the Product of Student's-t Distribution
First, note a univariate Student's s-t random variable T with v degrees of freedom can be defined as:
7
=
with Z.U, N(0, Vi
I U
Where Z and (U:}_1 are independent standard normal random variables. If is multidimmensional Student's: s-t random variable, composed of independent Z; and U, then T ~ PoT(r), i.::
Z1
Zz
Zn
T=
PoT(u)
(3)
Σ U?
1
1=v41
U?
in=(n (n 1)ut1 U2

---

Note that the Student's s-t variable T is large when most of the {Ut]: in its set are small. We can therefore think the {U;): as constraint violations rather then pattern matches: if the input matches all constraints U, zs 0, the corresponding 7' variables will activate (see 23 for further discussion).
4.3 Introducing Topography
To make the PoT distribution topographic, strive correlate the scales of T, which are 'nearby' in our topographic layout. One way to accomplish this is by sharing some U,-arriables between neigh- boring T s. Formally, we define overlapping neighborhoods N(j) for each variable Т₃ and write:
Z1
Z2
Zn
T
TPoT(u) (4)
V E U?
V LieNa)) U
V
U?
With some abuse of notation, we define W to be the adjacency matrix which defines our neighborhood structure, U and Z to be the vectors of random variables U/ and Zj. we can write the above succinetly as:
Z1
Z2
Zn
Z
T
TP)T())
(5)
V
W.222
V U2
WU2
Due to non-linearities such as ReLUs which may alter input distributions, is beneficial to allow the Z variables to model the mean and scale. We found this can be achieved with the following parameterization: T 2 In practice, we found that o vv often works well, finally yielding:
"/1//w07
Z r WU
Given this construction, observe that the TPoT generative model can instead be viewed as latent variable model where all random variables are Gaussian and the construction of T in Equation o is the first layer of the generative 'decoder': ge ge(t) go z). In Section we then leverage this interpretation to show how an approximate posterior for the latent variables Z and U can be trained through variational inference.
4.4 Capsules as Disjoint Topologies
U²
One setting of neighborhood structure W which is of par- ticular interest is when there exist multiple sets of disjoint neighborhoods. Statistically, the variables of two disjoint topologies are completely independent. An example of capsule neighborhood structure shown Figure The idea of independant subspaces has previously been shown learn invariant feature subspaces in the linear setting and is present in early work on Independent Subspace Analysis (26) and Adaptive Subspace Self Organizing Maps (AS- SOM) [35]. It is also very reminiscent of the transformed sets of features present in a group equivariant convolu- tional neural network. In the next section, will show how temporal coherence can be leveraged to induce the encoding of observed transformations into the internal di- mensions of such capsules thereby yielding unsupervised approximately equivariant capsules.
W1
Zi
T
U2
W
Zz
T
Us
U/
W3
Z3
Us
W
Z
T
U
Figure An example of neighborhood structure which induces disjoint topolo- gies (aka capsules). Lines between vari- ables T; indicate that sharing of U; and thus correlation.
4.5 Temporal Coherence and Learned Equivariance
We now describe how the induced topographic organization can be leveraged learn basis of ap- proximately equivariant capsules for observed transformation sequences. The resulting representation composed large set of 'capsules' where the dimensions inside the capsule are topographically structured, but between the capsules there is independence. To benefit from sequences of input, we encourage topographic structure over time between sequentially permuted activations within capsule, property refer to shifting temporal coherence.
5

---

4.5.1 Temporal Coherence
Temporal Coherence can be measured as the correlation of squared activation between time steps. One way we can achieve this our model is by having 7, share U; between time steps. Formally, the generative model is identical to Equation I factorizing over timesteps denoted by subscript 1, i.e. PX,,T, (Xi, bz) px, (T, (p,)))))t,t))))))))) However, T2 is now a function of a sequence {U1+s)
Z 1
T
W 10 UR-L
Where ( -7-L] denotes vertical concatenation of the column vectors U2, and 2L can be seen as the window size. We see that the choice of W now defines correlation structure over time. In prior work on temporal coherence (denoted 'Bubbles' (25)), the grouping over time such that given variable Tı, has correlated energy with the same spatial location (i) at previous time step (1 (i.e. cov T(,,s) > 0). This can be implemented as:
w [U UE-1] Σ W.U
Where Wa defines the topography for a single timestep, and is typically the same for all timesteps.
4.5.2 Learned Equivariance with Shifting Temporal Coherence
In our model, instead of requiring single location have correlated energies over sequence, would like variables at sequentially permuted locations within a capsule to have correlated energy between timesteps (cov 77 T2-1.--1 0). Similarly, this can be implemented as:
W [ : U Σ
WaRolls
T.
Where R denotes cyclic permutation of 5 steps along the capsule dimension. The exact implementation Roll can be found in Section [A.I]] As we will show Scction 6.33 TVAE models with such a topographic structure learn to encode observed sequence transformations as Rolls within the capsule dimension, analogous to group equivariant neural network where Tp and Roll can be seen as the action of the transformation on the input and output spaces respectively.
Topographic VAE
To train the parameters of the generative model use the above formulation to parameterize an approximate posterior for in terms of a deterministic transformation of approximate posteriors over simpler Gaussian latent variables u and z. Explicitly:
(x)) = (zz no (xi), (x(()
(x = po (x: go (Zi, {wiH)
(10)
(uixi) = (x),;
2
(11)
We denote this model the Topographic VAE (TVAE) and optimize the parameters 6, 0. (and through the ELBO, summed over the sequence length S:
μ))
(a,. (llo po(x -
-Dxlle(zz/x))llzzleee - -aclilDDxLlgggg (xx (12)
1
where (1)xxi (2 (z(x)))) (14+s)x44) and { denotes over time.
Experiments
In the following experiments, we demonstrate the viability of the Topographic VAE as a novel method for training deep topographic generative models. Additionally, quantitatively verify that shifting

---

temporal coherence yields approximately equivariant capsules by computing an "equivariance loss' and correlation metric inspired by the disentanglement literature. We show that equivariant capsule models yield higher likelihood than baselines test sequences, and qualitatively support these results with visualizations of sequences reconstructed purely from Rolled capsule activations.
6.1 Evaluation Methods
As depicted in Figure D we make use of capsule traversals to qualitatively visualize the transforma- tions learned by our network. Simply, these are constructed by encoding partial sequence into a to variable, and decoding sequentially Rolled copies of this variable. Explicitly, in the top row we show the data sequence (xi),. and in the bottom we show the decoded sequence: (ge (Roll; (t
To measure equivariance quantitatively, we measure an equivariance error similar to [1]]. The equivariance eπrоr can be seen the difference between traversing the two distinct paths of the commutative diagram, and provides some measure of how precisely the function and the transform commute. Formally, for sequence of length S, and i = t/ the error is defined as:
8-18-1
((t))
ΣΣ
[Roll &
(13)
Additionally, inspired by existing disentanglement metrics, we measure degree which observed transformations in capsule space are correlated with input transformations by introducing new metric we call CapCorr,, Simply, this metric computes the correlation between the amount of observed Roll of capsule's activation at two timesteps ( and ( ++ 6, and the shift of the ground truth generative factors y/ in that same time. Formally, for a correlation coefficient Corr:
CapCorr(t,, t145, Vt, 31+55 Corr (argmax [t: t(+5] ly M++6l)
(14)
Where * is discrete periodic cross- coss-coorelation across the capsule dimension, and the correlation coefficient is computed across the entire dataset. We see the argmaax the cross-correlation an estimate of the degree to which capsule activation has shifted from time / to + 5. To extend this to multiple capsules, we can replace the argmax function with the mode of the argmax computed for all capsules. We provide additional details and extensions of this metric in Section A.10 For measuring mupsule-metrics on bascline models which do not naturally have capsules, we simply arbitrarily divide the latent space into a fixed set of corresponding capsules and capsule dimensions, and provide such results as equivalent to 'random baselines' for these metrics.
6.2 Topographic VAE without Temporal Coherence
To validate the TVAE is capable of learning topographi- cally organized representations with deep neural networks, we first perform experiments on Topographic VAE with- out Temporal Coherence. The model is constructed as in Equations T0 and 11 with I. 0, and is trained to maxi- mize Equation T2 We fix W such that globally the latent variables are arranged in grid single capsule), and locally W sums over 5x5 2D groups of variables. In this setting, W can be easily implemented as 2D convolution with a 5x5 kernel of 1's, stride 1, and eyclic padding. We see that training the model with 3- layer MLP's for the encoders and decoder indeed yields 2D topographic organization of higher level features. In Figure we show the maximum activating image for each flattened
13
23 2
2
22 2
6
2
6 6622 162220
6622
2-dimensional torus (a
6622
6622/4 6622/229
6 62///8
0
6
P
D
4
3
7
7
9
9
7
final layer neuron of the capsule, plotted torus. We see that the neurons become arranged according class, orientation, width, and other learned features.
Figure 3: Maximum activating images for Topographic VAE trained with 2D torus topography on MNIST.
6.3 Learning Equivariant Capsules
In the remaining experiments, we provide evidence that the Topographic VAE can be leveraged to learn equivariant capsules by incorporating shifting temporal coherence into ID baseline topographic
7

---

model. We compare against two baselines: standard normal VAEs and models that have non-shifting
"stationary" temporal coherence as defined in Equation 8 (denoted 'BubbleVAE (25]).
In all experiments we use a 3-layer MLP with ReI.U activations for both encoders and the decoder.
We arrange the latent space into 15 circular capsules each of 15-dimensions for dSprites [3] and 18 circular capsules each of 18-dimensions for MNIST [37]. Example sequences {x1 [xt]i= are formed by taking a random initial example, and sequentially transforming it according to one of the available transformations: (X-Pos, Y-Pos, Orientation, Scale) for dSprites, and (Color, Scale, Orientation) for MNIST. All transformation sequences are cyclic such that when the maximum transformation parameter is reached, the subsequent value returns to the minimum. We denote the length of full transformation sequence by S, and the time-exten of the induced temporal coherence (i.e. the length of the input sequence) by 2L. For simplicity. both datasets are constructed such that the sequence length equals the capsule dimension (for dSprites this involves taking a subset of the full dataset and looping the scale 3-times for scale-sequence). Exact details are in Sections A.8 & A.9
In Figure 4 we show the capsule traversals for TVAE models with = J. We see that despite the to variable encoding only 3 of the sequence, the remainder of the transformation sequence can be decoded nearly perfectly by permuting the activation through the full capsule implying the model has learned to approximately equivariant to full sequences while only observing partial sequences per training point. Furthermore, we see that the model is able to successfully learn all transformations simultancously for the respective datasets.
9
9
S
9
9
9
9
9
6
9
2
Figure 4: Capsule Traversals for TVAE models dSprites and MNIST. The top rows show the encoded sequences (with greyed-out images held-out), and the bottom rows show the images generated by decoding sequentially Rolled copies of the initial activation to (indicated by grey border).
Capsule traversals for the non-equivariant baselines, well TVAEs with smaller values of L. (which only learn approximate equivariance to partial sequences) are shown in Section [ We note that the capsule traversal plotted in Figure demonstrates transformation where color and rotation change simultaneously, differing from how the models this section are trained. However, as we describe in more detail in Section B.4 observe that TVAEs trained with individual transformations isolation (as in this section) are able to generalize, generating sequences of combined transformations when presented with such partial input sequences at test time. We believe this generalization capability to be promising for data efficiency, but leave further exploration to future work. Additional capsule traversals with such unseen combined transformations are shown in Section B 4 and further complex learned transformations (such as perspective transforms) are shown at the end of Section D
For quantitative evaluation, in Table 1 measure the equivariance error and log-likelihood (reported in nats) of the test data under our trained MNIST models as estimated by importance sampling with 10 samples. We observe that models which incorporate temporal coherence (Bubble- VAE and TVAE with L >> 0) achieve low equivariance error, while the TVAE models with shifting temporal coherence achieve the highest likelihood and the lowest equivariance error simultaneously.
Table 1: Log Likelihood and Equivariance Error MNIST for different settings of temporal coherence length relative to sequence length S. Mean std. over random initalizations.
Model
TVAE L 1
TVAE 5 -=
TVAE
BubbleVAE
VAE
L
L L=0 0
L 4
L 0
logp(x) f
-186.8 0.1
-186.0 + 0.7
-218.5+ 0.9
-191.4 + 0.5
-189.0 + 0.8
Eeg
574 +
3247 + 3
3217 + 105
3370 + 12
13274 +
8

---

Table 2: Equivariance error (Eeg 4) and correlation of observed capsule roll with ground truth factor shift (CapCorr 1) for the dSprites dataset. Mean + standard deviation over 3 random initalizations.
Model
TVAE
TVAE L= -
TVAE
TVAE L= 0
Bubble
VAE
L.
L= -
I. L= S
L S
L= 0
CapCorrx x
1.0 +0
1.0
0.67 0.02
0.17
0.03
0.13 + 0.01
0.18 + 0.01
CapCorry
1.0
1.0
0.66
0.21 0.02
0.02
0.16 0.12
CapCorro f
1.0
1.0
0.01
0.09 0.01
0.10
0.11
CapCorrs f
1.0
1.0
0.51 0.00
0.52 0.50
Eeg
344 5
1034 6
2549 + 38
2971 +
1951 34
6934 + 0
To further understand how capsules transform for observed input transformations, in Table 2 we measure Eeg and the CapCorr metric on the dSprites dataset for the four proposed transformations.. We see that the TVAE with L 2 S achieves perfect correlation - implying the learned representation indeed permutes cyclically within capsules for observed transformation sequences. Further, this correlation gradually decreases as L decreases, eventually reaching the same level as the baselines. We also see that, on both datasets, the equivariance losses for the TVAE with L Bubble VAE are significantly lower than the baseline VAE, while conversely, the CapCorr metric is not significantly better. We believe this to be due to the fundamental difference between the metries: Eng measures continuous LI similarity which is still low when representation is locally smooth (even if the change of the representation does not follow the observed transformation), whereas CapCorr more strictly measures the correspondence between the transformation of the input and the transformation the representation. In other words, Egg may be misleadingly low for invariant capsule representations (as with the BubbleVAE) whereas CapCorr strictly measures equivariance.
and the
7 Future Work & Limitations
The model presented in this work has number of limitations in its existing form which we believe to be interesting directions for future research. Foremost, the model is challenging to compare directly with existing disentanglement and equivariance literature since it requires an input sequence which determines the transformations reachable through the capsule roll. Related to this, we note the temporal coherence proposed in model is not 'causal' (i.e. to depends future x)). We believe these limitations could be at least partially alleviated with minor extensions detailed in Section [
We additionally note that some model developers may find a priori definition of topographic structure burdensome. While true, we know that the construction of appropriate priors is always a challenging task in latent variable models, and we observe that our proposed TVAE achieves strong performance even with improper specification. Furthermore, in future work, we believe adding learned flexibility to the parameters W may alleviate some of this burden.
Finally, we note that while this work does demonstrate improved log-likelihood and equivariance error, the study is inherently preliminary and does not examine all important benefits of topographic or approximately equivariant representations. Specifically, further study of the TVAE both with and without temporal coherence in terms of the sample complexity, semi-supervised classification accuracy, and invariance through structured topographic pooling would be enlightening.
Conclusion
In the above work we introduce the Topographic Variational Autoencoder method train deep topographic generative models, and show how topography can be leveraged to learn approximately equivariant sets of features, a.k.a. capsules, directly from sequences of data with no other supervision. Ultimately, we believe these results may shine some light on how biological systems could hard-wire themselves to more effectively learn representations with equivariant capsule structure. terms of broader impact, it is foreseeable our model could be used to generate more realistic transformations of 'deepfakes', enhancing disinformation. Given that the model learns approximate equivariance, we caution against the over-reliance on equivariant properties as these have no known formal guarantees.

---

References
[1] Horace Barlow al. Possible principles underlying the transformation of sensory messages. Sensory communication, 1(01), 1961.
[2] Anthony J. Bell and Terrence J. Sejnowski. An
Approach to Blind
Separation and Blind Deconvolution. Neural Computation, 7(6): 11 1995.
[3] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798- 1828, 2013.
[4] Gregory Benton, Mare Finzi, Pavel Izmailov, and Andrew Gordon Wilson. Learning invariances neural networks. Advances in Neural Information Processing Systems, December, 2020.
[5] Lukas Biewald. Experiment tracking with weights and biases, 2020. Software available from wandb.com.
[6] Christopher Bishop, Markus Svensen, and Christopher Williams. Gtm: The generative topo- graphic mapping. Neural Computation, 10:215-22 2244,,34, 05 1997.
[7] Benedikt Boenninghoff, Steffen Zeiler, Robert M. Nickel, and Dorothea Kolossa. Varia- tional autoencoder with embedded student-l mixture model for authorship attribution. ArXiv, abs/2005. 13930, 2020.
[8] Diane Bouchacourt, Mark Ibrahim, and Stéphane Deny. Addressing the topological defects of disentanglement via distributed operators. ArXiv, abs/2102.05623, 2021.
[9] Taco Cohen and M. Welling. Steerable cnns. ArXiv,
2017.
[10] Taco Cohen and Max Welling. Learning the irreducible representations of commutative lie groups. Proceedings of the 31 International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 1755-1763, Bejing, China, 22-24 Jun
2014. PMLR.
[11] Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pages 2990-2999, 2016.
[12] Taco S. Cohen and Max Welling. Transformation properties of learned visual representations. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[13] Pierre Comon. Independent component analysis, a new concept? Signal processing, 36(3):287-
31 314, 1994.
[14] Marissa Connor, Gregory Canal, and Christopher Rozell. Variational autoencoder with learned latent structure. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 23697 PMLR, 13-15 Apr 2021.
[15] Nichita Diaconu and Daniel Worrall. Learning to convolve: A generalized weight-tying approach. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 1586-1595. PMLR, 09-15 Jun 2019.
[16] Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3165-3176. PMLR, 13-18 Jul 2020.
[17] Marc Finzi, Max Welling, and Andrew Gordon Gordon Wilson. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedingy of Machine Learning Research, pages 3318-3328. PMLR, 18-24 Jul 2021.
[18] Peter Földiák. Learning invariance from transformation sequences. Neural Computation, 3:194-200, 06 1991.
[19] Will Grathwohl and Aaron Wilson. Disentangling space and time in video with hierarchical variational auto-encoders. CoRR, abs/1612.04440, 2016.
10

---

[20] Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. ArXiv, abs/1812.02245 2018.
[21] Geoffrey E. Hinton, Alex Krizhevsky, and Sida D. Wang. Transforming auto-encoders. In Timo Honkela, Włodzistaw Duch, Mark Girolami, and Samuel Kaski, editors, Artificial Neural Networks and Machine Learning ICANN 2011, pages 44-51, Berlin, Heidelberg, 2011.
Springer Berlin Heidelberg.
[22] Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with EM routing. In International Conference on Learning Representations, 2018.
[23] Geoffrey E. Hinton and Yee- Fhe-Whye Teh. Discovering multiple constraints that are frequently approximately satisfied. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, UATO11, page 227-234, 2001.
[24] Jarmo Hurri and Aapo Hyvärinen. Silple-Cell-Like Receptive Fields Maximize Temporal Coherence in Natural Video. Neural Computation, 15(3):663-691, 03 2003.
[25] A. Hyvärinen, J. Hurri, and Jaakko J. Väyrynen. A unifying framework for natural image statistics: spatiotemporal activity bubbles. Neurocomputing, 58-60:801-806, 2004.
[26] Aapo Hyvärinen and Patrik Hoyer. [Emergence of phase-and shift-invarinn features by decom- position of natural images into independent feature subspaces Neural computation, 12(7):1705-
1720, 2000.
[27] Aapo Hyvärinen, Patrik O Hoyer, and Mika Inki. Topographic independent component analysis Neural computation, 13(7):1527-1558, 2001.
[28] Aapo Hyvärinen, Jarmo Hurri, and Patrick Hoyer. Natural image statistics: A probabilistic approach to early computational vision., volume 39. Springer Science & Business Media, 2009,
[29] Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications Neural networks, 13(4-5):411-44 1-300, 2000.
[30] Aapo Hyvärinen and Patrik O. Hoyer. A two-layer sparse coding model learns simple and complex cell receptive fields and topography from natural images. Vision Research, 41(18):2413- 2423, 2001.
[31] Koray Kavukcuoglu, Marc' Aurelio Ranzato, Rob Fergus, and Yann LeCun. Learning invariant Features through topographic filter maps) In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 1612-1612. IEEE, 2009.
[32] T. Anderson Keller, Qinghe Gao, and Max Welling. Modeling category-selective cortical regions with topographic variational autoencoders. In SVRIM 2021 Workshop @ NeurlPS, 2021.
[33] Diederik P. Kingma and Max Welling. Auto-EEcodiig Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
[34] David A. Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglemmnnt in natural data with temporal sparse coding. In International Conference on Learning Representations, 2021.
[35] Teuvo Kohonen, Emergennc of -eeaurr detectors in the adaptive self organizing map Biological cybernetics, 75(4):281-29) 1996.
[36] Alexci A Koulakov and Dmitri B Chklovskii. Orientation preference patterns in mammalian visual cortex: wire length minimization approach. Neuron, 29(2)5519-577, 2001.
[37] Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann. lecun. com/exlb/mnnitt, 2, 2010.
[38] Hyodong Lee, Eshed Margalit, Kamila M. Jozwik, Michael A. Cohen, Nancy Kanwisher, Daniel L. K. Yamins, and James J. DiCarlo. Topographic deep artificial neural networks reproduce the hallmarks of the primate inferior temporal cortex face processing network. bioRxiv, 07/2020 2020.
[39] Jan Eric Lenssen, Matthias Fey, and Pascal Libuschewski. Group equivariant capsule networks. In NewrIPS. pages 8858,8867, 2018.

---

[40] S Lyu and E P Simoncelli. Nonlinear image representation using divisive normalization. In Proc. Computer Vision and Pattern Recognition, pages 1-8. IEEE Computer Society, Jun 23-28 2008.
[41] S Lyu and E P Simoncelli. Modeling multiscale subbands of photographic images with fields of Gaussian scale mixtures. IEEE Trans. Patt. Analysis and Machine Intelligence, 31(4):693-706 Apr 2009.
[42] Libo Ma and Liqing Zhang. Overcomplete topographic independent component analysis Neurocomputing, 71(10- 12):2217 22232222233222 2008
[43] Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentangle- ment testing sprites dataset.
2017.
[44] Bruno A Olshausen and David J Field. (Sparse coding with an overcomplete basis set: A strategy employed by VI? Vision research, 37 1997.
[45] Simon Osindero, Max Welling, and Geoffrey E. Hinton. Topographic Product Models Applied to Natural Scene Statistics. Neural Computation, 18(2):381-411 02 2006.
[46] Simon Kayode Osindero. Contrastive Topographic Models. PhD thesis, University of London, 2004.
[47] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024 8035. 2019.
[48] J Portilla, Strela, M Wainwright, and E Simoncelli. Image denoising using scale mixtures of Gaussians in the wavelet domain. IEEE Trans Image Processing, (2(1))1338-1351 Nov 2003. Recipient, IEEE Signal Processing Society Best Paper Award, 2008
[49] Siamak Ravanbakhsh. Jeff Schneider, and Barnabás Póczos. Equivariance through parameter- sharing. In Doina Precup and Yee Whye Tch, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2891-2901. PMLR, 06-11 Aug 2017.
[50] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on International Conference on Machine Learning Volume 32, 1CML page 11-1278-11-1286. JMLR.org. 2014.
[51] Sara Sabour, Nicholas Frosst, and Geoffrey E. Hinton, Dynamic routing between capsules. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, page 3859-3869, Red Hook, NY, USA, 2017. Curran Associates Inc
[52] James V. Stone. Learning Perceptually Salient Visual Parameters Using Spatiotemporal Smooth- ness Constraints. Neural Computation, (0))1463-1142. 10 1996.
[53] Jan Stühmer, Richard E. Turner, and Sebastian Nowozin. Independent subspace analysis for unsupervised learning of disentangled representations, 2019.
[54] Richard Turner and Maneesh Sahani. maximum- interpretation for slow feature analysis. Neural computation, 19:1022-38, 05 2007.
[55] Elise van der Pol, Daniel E. Worrall, Herke van Hoof, Frans A. Olichock, and Max Welling. MDP homomorphic networks: Group symmetries in reinforcement learning. CoRR, abs/2006.16908, 2020.
[56] MJ Wainwright and E Simoncelli. Scale mixtures of Gaussians and the statistics of natural images. S. A. Solla, T. K. Leen, and K.-.. Müller, editors, Adv. Neural Information Processing Systems (NIPS*99), volume 12, pages 855-861, Cambridge, MA, May 2000. MIT Press.
[57] MJ Wainwright, E P Simoncelli, and A S Willsky. Random cascades on wavelet trees and their use in analyzing and modeling natural images. Applied and Computational Harmonic Analysis, 1 11(1):89-122 123, Jul 2001.
12

---

[58] Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steerable cnns: Learning rotationally equivariant features volumetric data. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, page 10402-10413, Red Hook, NY, USA, 2018. Curran Associates Inc.
[59] Max Welling, Simon Osindero, and Geoffrey E Hinton. Learning sparse topographic represen- tations with products of student-t distributions In Advances in neural information processing
systems, pages 1383-1390, 2003.
[60] Laurenz Wiskott and Terrence J Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural computation, 14(4):715-770, 2002.
[61] Daniel Worrall and Max Welling. Deep scale-spaces: Equivariance over scale. In H. Wallach,
H. Larochelle, A. Beygelzimer, F. T'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[62] Daniel E. Worrall, Stephan Garbin, Daniyar Turmukhambetov, and Gabriel J. Brostow. Harmonic networks: Deep translation and rotation equivariance. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7168-71771 2017.
Acknowledgements
We would like to thank Jorn Peters for his invaluable contributions to this work at its earliest stages. We would additionally like to thank Patrick Forré, Emiel Hoogeboom, and Anna Khoreva for their helpful guidance throughout the project. We would like to thank the creators of Weight & Biases [5] and PoTor [47]. Without these tools our work would not have been possible. Finally, we thank the Bosch Center for Artificial Intelligence for funding, and the reviewers for their helpful comments.
13


Reviews:
[{"review_id": "fYR6fJjaJE", "reviewer": "Reviewer_P3ig", "summary": "In their work, the authors propose a novel VAE-based model named \"Topographic VAE\", which is inspired by topographic ICA and based on the Topographic Product of Student's t (TPoT) distribution. The authors show how topographic correlations following the TPoT distribution can be modelled by combining independent, normally distributed random variables across local neighbourhoods. This way, a model based on a latent TPoT distribution can be learned in a variational framework similar to standard VAEs. The authors highlight the similarity to capsules when considering multiple subspaces with independent local neighbourhood structures. Inspired by previous works on temporal coherence, the model is extended to image sequences by using spatiotemporal neighbourhood structures. When using a neighbourhood that is shifted for each time step, the authors show that transformations in the input sequences can be projected into the future by applying shifts to the encoded latent code. So the latent of the model is shown to be equivariant w.r.t. the transformations observed in the training data.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n### Originality\n- The contribution of the work is the integration of topographic modelling with VAEs, and extending temporal coherence to learn equivariant representations. To my knowledge, this has not been done before.\n- The idea of topographic models, also including temporal coherence, is not new. The authors thoroughly cite the previous literature on the topic, both in a dedicated related work section and when explaining the model components.\n\n### Quality\n- To me, the derivation of the model is well justified and related to prior work with great care. Moreover, the metrics proposed for evaluating the equivariance properties of the model are technically sound.\n- The authors evaluate several variants of the model on two datasets (MNIST variant, dSprites) to evaluate the usefulness of the novel modelling components. I have however some concerns regarding the empirical evaluation of the model. In my view, some important questions regarding the representation learned by the model are not answered in the paper:\n    - No samples from the model are shown or evaluated. The model is generative and VAE based, and the derivation of the TPoT distribution requires independent and normally distributed random variables. When training VAE based models in practice, fitting the factorized normal distribution and obtaining low reconstruction error frequently have to be traded against each other. In the paper, only the reconstruction capabilities of the model are evaluated. So there's no control whether the latent variables actually follow the intended distribution after training.\n    - The aim of the shifting temporal coherence is to learn an equivariant representation of the training sequences. Only one of the transformations is changed in every training sequence, the shifting constraint however always affects all latent variables. To me it is not clear how the constant parameters are represented by the model, especially since the encoders are independent for every image according to equations (10) and (11). An anlysis of the representation in this respect would contribute to a better understanding the model.\n    - As discussed with the limitations, the model is only able to infer the latent representation $T$ for image sequences. It would be interesting to know how the representation of the same image changes when it is part of different sequences. Moreover, the representation $T$ of a sequence could be sampled given $z$ and $u$ for a fixed image. This kind of experiments could also help to yield insights regarding the previous issue.\n    - How does the model depend on the size of the latent space, i.e. the number and dimensionality of the capsules?\n    - How is an image transformed, if the latent representation is rolled only for a single capsule? Does this correspond to meaningful transformations?\n\n\n### Clarity\n- Overall, the paper is very clearly written, well-structured and provides the necessary background for readers not familiar to topographic modelling.\n- In my view, more details about the models used for the experiments, especially on the choice of neighbourhood structures and the weight normalization, should be provided in the main paper. To obey the page limit, the background and derivation of the model could be described more compactly.\n- Figure 1: I do not understand the visualizion of the rolls in the latent space. Visualizing more latent variables per capsule, or smaller shifts would make the figure clearer.\n- Some minor issues:\n    - Figure 2: There should not be a line between the $Z$s as they are independent.\n    - L217: Missing reference / undefined BibTeX entry\n    - Equation 1 and 10: Whats the role of $g_\\theta(t)$? Can it be subsumed into $p_\\theta(x|t)$?\n\n### Significance\n- As stated by the authors, topographic organization is an important principle also found in the human brain and has been used in various forms by research related to computational neuroscience. Integrating this idea with VAEs may foster future research in that direction.\n- However, no results have been presented that show the advantage of a spatially topographic (i.e., L=0) latent representation, e.g. in terms of the sample quality or model interpretability/controllability. So this model is most likely of less interest to researchers not interested in topography itself.\n- The temporal model (L > 0) has shown appealing results regarding the completion of sequences of complex transformations. Compared to related works, e.g. on disentanglement, this has important limitations: The learned transformations have to be presented as sequences during training. More importantly, only image sequences can be encoded, and samples can only be transformed by the transformation observed in the input sequence. So the applicability of the learned equivariance of the latent space is very narrow.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\n- The authors discuss the limitations of the model honestly in Section 7, and the potential societal impact of the work in Section 9.\n- As detailed above, the limitation and effects stemming from the fact that the model can only encode the latent $T$ for sequences should be adressed in the experimental section. The discussion in the limitations section is adequate.\n- The discussion of the potential societal impact is brief, but given the rather theorectical nature of the paper I think it is okay.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "WmAiGkVxOjd", "reviewer": "Reviewer_h1aQ", "summary": "The paper introduces a VAE that can model the spatial and temporal “coherence” of latent variables. The idea is to construct student’s-t distributions from initial Gaussian latents. These student’s-t distributions are made coherent by sharing some of the Gaussian latents that are used in their construction. These variables are then divided into blocks called capsules where the variables in different blocks do not share any Gaussian latent vars. Temporal coherence is achieved by applying a similar “sharing” of Gaussians in a moving time window. However, the paper also suggests “rolling” the variables inside a capsule. The paper’s claim is that this rolling of the variables leads to an equivariant representation. Both qualitative and quantitative experiments on variations of MNIST and dSprites demonstrate the ability of this model in learning representations that are spatially and temporally coherent.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution of the paper is in proposing a mechanism for achieving coherence in VAEs. This idea is quite appealing and also supported by the experiments. However, the claim that topographic representations learned by the model is necessarily equivariant needs more work. First, note that equivariance is more strict than coherence, in the sense that each sequence should be produced through symmetry transformation. With this assumption, each sequence can be a one-dimensional submanifold of a Lie group. It is not clear how a combination of capsules each of which is equivariant to SO(2) can represent such arbitrary one-dimensional submanifolds. This undermines a key claim of the paper, which also appears in the title.\n\nLet’s see the experimental setup: the symmetry group is a product of SO(2)s, and compatible with the rolling operation of the capsule (note that this is not general). Moreover, the “sequences” used in experiments seem to have been produced by only changing one property at a time, and the same is done for testing the model. Therefore, train and test sequences can be quite similar, and the benefit of learning equivariant representation, in generalizing to new transformations from the same group, is not validated. \n\nDid I understand the model and your setup correctly? \n\nCould you please further elaborate on the capsule traversal process? Do you roll ALL the capsules at the same time? If so would this be able to produce a range of different one-dimensional submanifolds of a group that shares parts of a sequence (even the group is SO(2)xSO(2))? As an example, in a sequence, you may translate a digit on different trajectories in the training data, but suppose all these trajectories share parts of the initial sequence. How does the traversal process work for this example, when you feed parts of the sequence that is shared in these sequences?\n\nAs for other aspects of the paper, the coverage of related literature is quite extensive and it shows a broad perspective in this general area. However, the presentation can be improved especially at the beginning; I had little idea about what the paper was trying to achieve until section 4.2. (this may be because I wasn’t familiar with the meaning of the keyword topography in the context of representation learning.) Also, the champion figure is difficult to interpret partly because the rolling operation is not clear in the example. \n\nMinor:\nQ vs q in Eq(12)\nMissing reference on line 217\nMeasuring capsule metrics on baseline models doesn’t seem fair (lines 262-263 need clarification.)\nFigure 3 is produced by using inputs that maximize the activation of certain units. Is it possible to generate a similar traversal figure by “translating” the posterior for a particular input?\nCan you test the traversal procedure by “combining” transformations (e.g., rotation+change of color in Fig 4)?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes!", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "1JZEo4dXsk", "reviewer": "Reviewer_GHqG", "summary": "**Summary**: \nThis paper first introduces the topographical VAE (TVAE): which is a latent variable model with a topographical product of student-t prior, and is trained with the variational autoencoding objective. They further show that the topographic organisation in TVAE can be leveraged to learn a basis of equivariant capsules, by using a neighbourhood structure which induces disjoint topologies, and by encouraging “shifting temporal coherence”. The authors test their models on MNIST and synthetic sequential data.\n\n**Contributions**:\n- This paper shows that the existing topographical product of student’s-t model can be constructed from a set of independent standard Gaussian random variables, and can be trained efficiently through variational inference.\n- This paper discusses the disjoint neighbourhood structure in TVAE where each neighbourhood can be seen as a capsule.\n- This paper shows that, when taking sequence as inputs, encouraging shift temporal coherence could lead to unsupervised learning of equivariance in capsules.\n- In preliminary experiments, they demonstrate that TVAE learns to organise its activations according to salient characteristics on MNIST. They further show that, by incorporating topographic organisation over time (temporal coherence), they can learn equivariant capsules from synthetic sequential data. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: \nAlthough topographical product of student’s-t model is already known, it is previously trained with techniques such as contrastive divergence. This paper shows that the model can be constructed from a set of independent standard Gaussian random variables, and can be trained efficiently through variational inference, therefore proposing topographical VAE (TVAE).\n\nThis paper also discussed topographical organisation over time (temporal coherence), and how to incorporate it into TVAE to learn approximately equivariant capsules from sequential data in an unsupervised manner.\n\n**Quality**:\nThis paper contains many interesting ideas that are worthwhile to investigate further, including VAE training of topographical product of student’s-t model, the topographical organisation over time, unsupervised learning of equivariant capsules. However, because the authors want to put too many things in one work, the discussions or experiments for each part are inadequate in my opinion. For example, what’s the merits of having VAE training rather than contrastive divergence for TVAE? Why would people want to use TVAE rather than standard VAE for various applications? I can only get rough ideas of the properties of TVAE from the preliminary experiments on MNIST, but then the authors move on to discuss learning equivariant capsules. However, for learning equivariant capsules, the discussion is also inadequate. What would happen if the underlying transformations at each time step are different? Moreover, by rolling variables at each time step, the authors are using a permutation representation of group actions. But why can’t the authors use a simpler matrix representation, so that the dimensions of the learned representations can be greatly reduced?\n\nI would like to also highlight my biggest concern of this paper below: If the roll operator is predefined and it rolls variables in each capsule by $1$ unit at each time step, is it suggesting that the input sequences should also have an identical transformation at each time step? Could the model still learn equivariance, if the sequential data is a video of a dropping ball (so not constant speed, but with acceleration)? If it is true that this model requires a constant underlying transformation for each sequence, how can the model potentially be applied to real-world problems in the future when such sequential data is almost never available unless you have control over the data generating process. (I might be mistaken about this part, so I would appreciate it if the authors could explain.)\n\n**Clarity**:\nThe paper is well written and the authors clearly have already put some effort into presenting their ideas. Below are some minor suggestions:\n- In Figure 1, because some variables are omitted, it is not so obvious that this is actually rolling. Maybe draw fewer variables but don’t use “…”?\n- Line 120-122, Using TPoT as an example before introducing it does not help.\n- Section 4.5.2, when talking about equivariance, maybe the authors should specify group actions in both the input and output space.\n\n**Significance**:\nThis paper presents several interesting ideas including VAE training of topographical product of student’s-t model, the topographical organisation over time, unsupervised learning of equivariant capsules. These ideas can potentially inspire other works in this area. However, I would also like to point out that there are inadequate demonstrations of properties and merits of these approaches, and there should be more ablation study for complexities in the proposed models, before other people can confidently build on top of them.\n\nIn general, this paper presents several interesting ideas, even though I still have the concerns listed above. I think it is a borderline case, so I will be ready to adjust my score and will carefully read the authors’ responses.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "**Limitations**:\nBelow are a few of my suggestions:\n- This paper should include more empirical and theoretical analysis of TVAEs (without temporal coherence) on still images.\n- This paper should probably explore how to learn equivariant capsules from sequential data where the underlying transformations can change over time (or being continuous). If this is hard, what is the authors’ opinion on how we can potentially apply this approach to real-world sequential data?\n\n**Societal Impact**:\nI think the authors already have a good discussion of potential societal impact.\n", "review_text": "", "extra_scores": {"ethical_concerns": "I have no ethical concerns regarding this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Solid work making an original contribution on the link between topographic representations (such as those found in the brain)  to equivariance learning.
Decision: Accept (Poster)