Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduces a VAE that can model the spatial and temporal “coherence” of latent variables. The idea is to construct student’s-t distributions from initial Gaussian latents. These student’s-t distributions are made coherent by sharing some of the Gaussian latents that are used in their construction. These variables are then divided into blocks called capsules where the variables in different blocks do not share any Gaussian latent vars. Temporal coherence is achieved by applying a similar “sharing” of Gaussians in a moving time window. However, the paper also suggests “rolling” the variables inside a capsule. The paper’s claim is that this rolling of the variables leads to an equivariant representation. Both qualitative and quantitative experiments on variations of MNIST and dSprites demonstrate the ability of this model in learning representations that are spatially and temporally coherent.

main_review: The main contribution of the paper is in proposing a mechanism for achieving coherence in VAEs. This idea is quite appealing and also supported by the experiments. However, the claim that topographic representations learned by the model is necessarily equivariant needs more work. First, note that equivariance is more strict than coherence, in the sense that each sequence should be produced through symmetry transformation. With this assumption, each sequence can be a one-dimensional submanifold of a Lie group. It is not clear how a combination of capsules each of which is equivariant to SO(2) can represent such arbitrary one-dimensional submanifolds. This undermines a key claim of the paper, which also appears in the title.

Let’s see the experimental setup: the symmetry group is a product of SO(2)s, and compatible with the rolling operation of the capsule (note that this is not general). Moreover, the “sequences” used in experiments seem to have been produced by only changing one property at a time, and the same is done for testing the model. Therefore, train and test sequences can be quite similar, and the benefit of learning equivariant representation, in generalizing to new transformations from the same group, is not validated. 

Did I understand the model and your setup correctly? 

Could you please further elaborate on the capsule traversal process? Do you roll ALL the capsules at the same time? If so would this be able to produce a range of different one-dimensional submanifolds of a group that shares parts of a sequence (even the group is SO(2)xSO(2))? As an example, in a sequence, you may translate a digit on different trajectories in the training data, but suppose all these trajectories share parts of the initial sequence. How does the traversal process work for this example, when you feed parts of the sequence that is shared in these sequences?

As for other aspects of the paper, the coverage of related literature is quite extensive and it shows a broad perspective in this general area. However, the presentation can be improved especially at the beginning; I had little idea about what the paper was trying to achieve until section 4.2. (this may be because I wasn’t familiar with the meaning of the keyword topography in the context of representation learning.) Also, the champion figure is difficult to interpret partly because the rolling operation is not clear in the example. 

Minor:
Q vs q in Eq(12)
Missing reference on line 217
Measuring capsule metrics on baseline models doesn’t seem fair (lines 262-263 need clarification.)
Figure 3 is produced by using inputs that maximize the activation of certain units. Is it possible to generate a similar traversal figure by “translating” the posterior for a particular input?
Can you test the traversal procedure by “combining” transformations (e.g., rotation+change of color in Fig 4)?

limitations_and_societal_impact: yes!
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: **Summary**: 
This paper first introduces the topographical VAE (TVAE): which is a latent variable model with a topographical product of student-t prior, and is trained with the variational autoencoding objective. They further show that the topographic organisation in TVAE can be leveraged to learn a basis of equivariant capsules, by using a neighbourhood structure which induces disjoint topologies, and by encouraging “shifting temporal coherence”. The authors test their models on MNIST and synthetic sequential data.

**Contributions**:
- This paper shows that the existing topographical product of student’s-t model can be constructed from a set of independent standard Gaussian random variables, and can be trained efficiently through variational inference.
- This paper discusses the disjoint neighbourhood structure in TVAE where each neighbourhood can be seen as a capsule.
- This paper shows that, when taking sequence as inputs, encouraging shift temporal coherence could lead to unsupervised learning of equivariance in capsules.
- In preliminary experiments, they demonstrate that TVAE learns to organise its activations according to salient characteristics on MNIST. They further show that, by incorporating topographic organisation over time (temporal coherence), they can learn equivariant capsules from synthetic sequential data. 

main_review: **Originality**: 
Although topographical product of student’s-t model is already known, it is previously trained with techniques such as contrastive divergence. This paper shows that the model can be constructed from a set of independent standard Gaussian random variables, and can be trained efficiently through variational inference, therefore proposing topographical VAE (TVAE).

This paper also discussed topographical organisation over time (temporal coherence), and how to incorporate it into TVAE to learn approximately equivariant capsules from sequential data in an unsupervised manner.

**Quality**:
This paper contains many interesting ideas that are worthwhile to investigate further, including VAE training of topographical product of student’s-t model, the topographical organisation over time, unsupervised learning of equivariant capsules. However, because the authors want to put too many things in one work, the discussions or experiments for each part are inadequate in my opinion. For example, what’s the merits of having VAE training rather than contrastive divergence for TVAE? Why would people want to use TVAE rather than standard VAE for various applications? I can only get rough ideas of the properties of TVAE from the preliminary experiments on MNIST, but then the authors move on to discuss learning equivariant capsules. However, for learning equivariant capsules, the discussion is also inadequate. What would happen if the underlying transformations at each time step are different? Moreover, by rolling variables at each time step, the authors are using a permutation representation of group actions. But why can’t the authors use a simpler matrix representation, so that the dimensions of the learned representations can be greatly reduced?

I would like to also highlight my biggest concern of this paper below: If the roll operator is predefined and it rolls variables in each capsule by $1$ unit at each time step, is it suggesting that the input sequences should also have an identical transformation at each time step? Could the model still learn equivariance, if the sequential data is a video of a dropping ball (so not constant speed, but with acceleration)? If it is true that this model requires a constant underlying transformation for each sequence, how can the model potentially be applied to real-world problems in the future when such sequential data is almost never available unless you have control over the data generating process. (I might be mistaken about this part, so I would appreciate it if the authors could explain.)

**Clarity**:
The paper is well written and the authors clearly have already put some effort into presenting their ideas. Below are some minor suggestions:
- In Figure 1, because some variables are omitted, it is not so obvious that this is actually rolling. Maybe draw fewer variables but don’t use “…”?
- Line 120-122, Using TPoT as an example before introducing it does not help.
- Section 4.5.2, when talking about equivariance, maybe the authors should specify group actions in both the input and output space.

**Significance**:
This paper presents several interesting ideas including VAE training of topographical product of student’s-t model, the topographical organisation over time, unsupervised learning of equivariant capsules. These ideas can potentially inspire other works in this area. However, I would also like to point out that there are inadequate demonstrations of properties and merits of these approaches, and there should be more ablation study for complexities in the proposed models, before other people can confidently build on top of them.

In general, this paper presents several interesting ideas, even though I still have the concerns listed above. I think it is a borderline case, so I will be ready to adjust my score and will carefully read the authors’ responses.

limitations_and_societal_impact: **Limitations**:
Below are a few of my suggestions:
- This paper should include more empirical and theoretical analysis of TVAEs (without temporal coherence) on still images.
- This paper should probably explore how to learn equivariant capsules from sequential data where the underlying transformations can change over time (or being continuous). If this is hard, what is the authors’ opinion on how we can potentially apply this approach to real-world sequential data?

**Societal Impact**:
I think the authors already have a good discussion of potential societal impact.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: In their work, the authors propose a novel VAE-based model named "Topographic VAE", which is inspired by topographic ICA and based on the Topographic Product of Student's t (TPoT) distribution. The authors show how topographic correlations following the TPoT distribution can be modelled by combining independent, normally distributed random variables across local neighbourhoods. This way, a model based on a latent TPoT distribution can be learned in a variational framework similar to standard VAEs. The authors highlight the similarity to capsules when considering multiple subspaces with independent local neighbourhood structures. Inspired by previous works on temporal coherence, the model is extended to image sequences by using spatiotemporal neighbourhood structures. When using a neighbourhood that is shifted for each time step, the authors show that transformations in the input sequences can be projected into the future by applying shifts to the encoded latent code. So the latent of the model is shown to be equivariant w.r.t. the transformations observed in the training data.
main_review: 
### Originality
- The contribution of the work is the integration of topographic modelling with VAEs, and extending temporal coherence to learn equivariant representations. To my knowledge, this has not been done before.
- The idea of topographic models, also including temporal coherence, is not new. The authors thoroughly cite the previous literature on the topic, both in a dedicated related work section and when explaining the model components.

### Quality
- To me, the derivation of the model is well justified and related to prior work with great care. Moreover, the metrics proposed for evaluating the equivariance properties of the model are technically sound.
- The authors evaluate several variants of the model on two datasets (MNIST variant, dSprites) to evaluate the usefulness of the novel modelling components. I have however some concerns regarding the empirical evaluation of the model. In my view, some important questions regarding the representation learned by the model are not answered in the paper:
    - No samples from the model are shown or evaluated. The model is generative and VAE based, and the derivation of the TPoT distribution requires independent and normally distributed random variables. When training VAE based models in practice, fitting the factorized normal distribution and obtaining low reconstruction error frequently have to be traded against each other. In the paper, only the reconstruction capabilities of the model are evaluated. So there's no control whether the latent variables actually follow the intended distribution after training.
    - The aim of the shifting temporal coherence is to learn an equivariant representation of the training sequences. Only one of the transformations is changed in every training sequence, the shifting constraint however always affects all latent variables. To me it is not clear how the constant parameters are represented by the model, especially since the encoders are independent for every image according to equations (10) and (11). An anlysis of the representation in this respect would contribute to a better understanding the model.
    - As discussed with the limitations, the model is only able to infer the latent representation $T$ for image sequences. It would be interesting to know how the representation of the same image changes when it is part of different sequences. Moreover, the representation $T$ of a sequence could be sampled given $z$ and $u$ for a fixed image. This kind of experiments could also help to yield insights regarding the previous issue.
    - How does the model depend on the size of the latent space, i.e. the number and dimensionality of the capsules?
    - How is an image transformed, if the latent representation is rolled only for a single capsule? Does this correspond to meaningful transformations?


### Clarity
- Overall, the paper is very clearly written, well-structured and provides the necessary background for readers not familiar to topographic modelling.
- In my view, more details about the models used for the experiments, especially on the choice of neighbourhood structures and the weight normalization, should be provided in the main paper. To obey the page limit, the background and derivation of the model could be described more compactly.
- Figure 1: I do not understand the visualizion of the rolls in the latent space. Visualizing more latent variables per capsule, or smaller shifts would make the figure clearer.
- Some minor issues:
    - Figure 2: There should not be a line between the $Z$s as they are independent.
    - L217: Missing reference / undefined BibTeX entry
    - Equation 1 and 10: Whats the role of $g_\theta(t)$? Can it be subsumed into $p_\theta(x|t)$?

### Significance
- As stated by the authors, topographic organization is an important principle also found in the human brain and has been used in various forms by research related to computational neuroscience. Integrating this idea with VAEs may foster future research in that direction.
- However, no results have been presented that show the advantage of a spatially topographic (i.e., L=0) latent representation, e.g. in terms of the sample quality or model interpretability/controllability. So this model is most likely of less interest to researchers not interested in topography itself.
- The temporal model (L > 0) has shown appealing results regarding the completion of sequences of complex transformations. Compared to related works, e.g. on disentanglement, this has important limitations: The learned transformations have to be presented as sequences during training. More importantly, only image sequences can be encoded, and samples can only be transformed by the transformation observed in the input sequence. So the applicability of the learned equivariance of the latent space is very narrow.
limitations_and_societal_impact: 
- The authors discuss the limitations of the model honestly in Section 7, and the potential societal impact of the work in Section 9.
- As detailed above, the limitation and effects stemming from the fact that the model can only encode the latent $T$ for sequences should be adressed in the experimental section. The discussion in the limitations section is adequate.
- The discussion of the potential societal impact is brief, but given the rather theorectical nature of the paper I think it is okay.
review_title: null

Meta-review: Solid work making an original contribution on the link between topographic representations (such as those found in the brain)  to equivariance learning.
Decision: Accept (Poster)