Title: Topological Obstructions and How to Avoid Them
Venue: NeurIPS 2023
Accepted: yes
Keywords: representation learning

variational autoencoders

homeomorphism

topological

equivariant

lie groups

normalizing flows
Abstract: Incorporating geometric inductive biases into models can aid interpretability and generalization, but encoding to a specific geometric structure can be challenging due to the imposed topological constraints. In this paper, we theoretically and empirically characterize obstructions to training encoders with geometric latent spaces. We show that local optima can arise due to singularities (e.g. self-intersection) or due to an incorrect degree or winding number. We then discuss how normalizing flows can potentially circumvent these obstructions by defining multimodal variational distributions. Inspired by this observation, we propose a new flow-based model that maps data points to multimodal distributions over geometric spaces and empirically evaluate our model on 2 domains. We observe improved stability during training and a higher chance of converging to a homeomorphic encoder.

Topological Obstructions and How to Avoid Them

Abstract

Incorporating geometric inductive biases into models can aid interpretability and generalization, but encoding to a specific geometric structure can be challenging due to the imposed topological constraints. In this paper, we theoretically and empirically characterize obstructions to training encoders with geometric latent spaces. We show that local optima can arise due to singularities (e.g. self-intersection) or due to an incorrect degree or winding number. We then discuss how normalizing flows can potentially circumvent these obstructions by defining multimodal variational distributions. Inspired by this observation, we propose a new flow-based model that maps data points to multimodal distributions over geometric spaces and empirically evaluate our model on 2 domains. We observe improved stability during training and a higher chance of converging to a homeomorphic encoder.

1 Introduction

A well-established idea in machine learning research is that geometric inductive biases can help us learn representations that reflect the underlying structure of a dataset [Bronstein et al., 2021, Higgins et al., 2022]. A key intuition behind this line of research is that such representations make it easier to reason about the similarities of different instances in the dataset, for example by relating inputs using a rotation or translation, which in turn aids interpretability and data-efficiency. Geometric inductive biases have been explored in a wide variety of forms, including models that are defined on hyperbolic or spherical Riemannian manifolds [Lezcano-Casado and Martınez-Rubio, 2019, Ganea et al., 2018], models that are equivariant or invariant with respect to particular symmetry groups [Kondor and Trivedi, 2018, Cohen and Welling, 2016], and work that leverages symmetries to learn disentangled representations that factorize into distinct axes of variation [Higgins et al., 2018].

In this paper, we consider symmetry-based approaches to learning representations in an unsupervised manner by imposing geometric inductive biases on the representation space. In this context, our notion of a representation that 'reflects the underlying structure of the data' is a representation that is homeomorphic to the true generative factors. We are specifically interested in the optimization challenges that arise when encoding to geometric spaces such as Lie groups. While a common intuition is that an inductive bias that matches the underlying topology of the data will guide a model towards a homeomorphic representation, there are also indications that certain inductive biases can make a model more difficult to train in practice [Park et al., 2022, Falorsi et al., 2018, Batson et al., 2021], which limits their practical utility.

^

To understand why encoding to geometric structures can give rise to optimization challenges, we formalize topological defects that can occur in a randomly initialized encoder, such as discrepancies in the winding number or crossing number relative to those in a homeomorphic encoder. We show that these topological defects will be preserved under continuous optimization, which suggests that escaping these local optima relies on the discrete jumps that are employed during optimization.

To circumvent these obstacles to optimization, we propose Group-Flow Variational Autoencoders (GFVAEs), which leverage normalizing flows to model complex multimodal distributions on Riemannian manifolds (Figure 1). We show that if we define the mode of the variational distribution to be the representation, normalizing flows can circumvent some of the challenges associated with local optima due to their multimodal nature. Experiments demonstrate that GF-VAEs can escape local optima during the early stages of training, resulting in more reliable convergence to a homeomorphic mapping and a greater degree of continuity after training.

We summarize the main contributions of this paper as follows:

We characterize topological defects that can arise in encoders that map onto spaces with geometric structure. We show that some obstructions that arise from topological defects cannot be resolved using continuous optimization.

We define evaluation criteria based on the winding number, crossing number, and continuity to measure topological defects and homeomorphism violations in the encoder.

We propose GF-VAEs, a new VAE-based model that that employs normalizing flows to define complex distribution on Lie groups. We empirically show that GF-VAEs are able to aid in circumventing identified optimization obstructions.

2 Problem Statement

Homeomorphic Encoders. Our goal is to learn representations in domains where we have prior knowledge of the geometric structure, specifically structure in the form of a symmetry group that can be associated with the input data. We assume that data lies on a low-dimensional manifold M that is embedded in a higher-dimensional space X := R n via a mapping g X : M→X . This is commonly known as the manifold hypothesis [Bengio et al., 2013]. We denote the image of the mapping by M x := g X ( M ) ⊆ X . Then g X is a homeomorphism, or topological isomorphism, onto its image g : M ∼ -→ M x . That is, g is continuous, bijective, and has a continuous inverse.

We wish to learn an encoder f φ : X → Z such that its restriction f φ | M x : M x →Z is a homeomorphism. Following Falorsi et al. [2018], we will define this mapping in terms of a network h φ : X → Y

that maps to an intermediate space Y := R d , followed by a known projection π : Y → Z ,

Lie Groups. Our work focuses the specific case where the manifolds M and Z are Lie groups. Any set of symmetries may be formally described by a group G , which is a set of invertible transformations which may be composed using a binary operation · : G × G → G . A Lie group is a group that is also a differentiable manifold. Lie groups describe continuous symmetries such as rotations and translations, and are therefore a natural mathematical setting for any system with spatio-temporal symmetries.

Reasoning about Lie groups is difficult due to their non-flat structure. Lie algebras provide a way to study Lie groups by considering the tangent space g of the manifold at the identity element. The exponential map exp : g → G maps points the Lie algebra to points on the group manifold as such. A vector v ∈ g defines a vector field on G using the group action to transport v around G . Then exp( v ) is defined to be the point reached by flowing along this vector field for unit time. For more details on Lie groups, we refer the readers to [Hall and Hall, 2013].

Variational Autoencoders on Lie Groups. In this paper, we primarily focus on variational autoencoder (VAE) [Kingma and Welling, 2014, Rezende et al., 2014] as a means to learn a homeomorphic embedding. In this setting, we define a generative model by composing a uniform prior p ( z ) = U ( z ) on Z with a likelihood model p θ ( x | z ) = N ( x ; f ∗ θ ( z ) , σ 2 x I n ) that is defined in terms of a decoder network f ∗ θ : Z → X . We use the encoder f φ to define a variational distribution q φ ( z | x ) , whose design we discuss below and train the encoder and decoder jointly by optimizing the variational lower bound,

Defining a variational distribution on a manifold is generally not straightforward as it requires finding an expression of the density on the manifold or keeping track of the change of volume when projecting to the manifold from the tangent space. Falorsi et al. [2019] define a reparameterized construction for sampling from a Gaussian distribution on Z by first sampling glyph[epsilon1] ∼ N (0 , I p ) from the p -dimensional Lie algebra associated with Z , then rescaling glyph[epsilon1] by way of element-wise multiplication σ φ ( x ) glyph[circledot] glyph[epsilon1] using a network σ φ : X → R p , and computing the corresponding element z glyph[epsilon1] on the group manifold using the exponential map. By left multiplying this randomly sampled z glyph[epsilon1] by the group element f φ ( x ) ∈ M to move the mode of the final distribution to its intended location, we obtain the construction

We account for the possible change in volume using the change of variable formula. Note that composing group elements f φ ( x ) · z glyph[epsilon1] does not change the density of the resulting point on the manifold because the Haar measure, a standard choice for Lie groups, is left invariant.

Running Example. As a concrete running example, which we will use throughout this paper, we will consider data on the circle M = SO(2) that is embedded into a space of images X . The subset M x of images generated by a function g : SO(2) →M x corresponds to images of an object that is subject to a one-dimensional rotation. We will consider the case of in-plane rotations of images, as well images of a 3-dimensional object that is rotated around a single axis.

The special orthogonal Lie group SO(2) is defined as

where GL(2) is the general linear group, which is the group of invertible 2 × 2 matrices under matrix multiplication. The set of images M x lives on a manifold that is homeomorphic to the rotation group SO(2) , embedded into the space of images by g . Because we assume we know the underlying manifold M , we can design Z to have the same structure. When M = SO(2) , we can use Y := R 2 . The function h φ in Equation 1 is then an ordinary neural network, and the projection to Z is simply the projection onto the unit circle : R 2 → SO(2) := y ↦→ A ( y/ ‖ y ‖ ) .

,

M

Mr

1

OQ

g

3 Optimization Obstructions

In practice, training homeomorphic encoders can give rise to optimization challenges. To develop intuition for these challenges, we will consider the running example M = SO(2) with data in the form of in-plane rotated images on X = R 32 × 32 (Figure 2). We train a V AE in which the network h φ is a standard convolutional network which is paired with a deconvolutional decoder f ∗ : SO(2) → R 32 × 32 (see Appendix A). Figure 2 shows encodings in the intermediate space Y after training with ELBO with 3 random seeds. For the first two seeds, we see that h φ ( M x ) crosses over itself, resulting in an figure '8' shape. When projected onto Z := SO(2) , this results in discontinuities at the crossover points. The second seed in addition also exhibits a sparse region in the intermediate space, leading to a gap in the SO(2) projection. Only the third initialization has converged to the correct topology. These local optima are not unique to this example; obstructions have also been encountered in Falorsi et al. [2018] when trying to learn 3D orientations of a rotating multi-color cube from 2D images using a homeomorphic VAE. Park et al. [2022] show that the homeomorphic VAE cannot generalize well to other shapes and tends to learn a degenerate embedding to a small part of SO(3) .

The main observation that we make in this paper is that imposing a geometric structure on the latent space can introduce topological obstructions during optimization . This insight is distinct from the homological obstructions identified by de Haan and Falorsi [2018], who describe the obstructions that emerge from the choice of parameterization on the Lie group. We will refer to the topological defects that we identify in this paper as 'optimization obstructions'. The problem that we identify here is that randomly-initialized layers have a high probability of exhibiting topological defects (degree, crossing, etc.) that cannot be resolved under continuous optimization using gradient flow. Removal of these topological defects is thus only possible by relying on the jumps coming from performing SGD with a large enough learning rate. This implies that while escaping such local minima is possible, it is difficult and may require many epochs to do so, dramatically slowing training and undercutting the advantages of learning a homeomorphic representation.

We now discuss several specific optimization obstructions. We focus on the case where M is the Lie group SO(2) , with the same example setting described in Section 2. All the optimization obstructions we consider occur in this case and in the case of higher-dimensional manifolds M as well, but are simpler to describe for S 1 . See Appendix B for proofs.

3.1 Figure Eight Local Minima

To more precisely describe obstructions that might arise during optimization, we consider continuoustime training along a gradient flow. We denote the weights of the initialized encoder as φ (0) and the trained weights as φ (1) . We consider the idealized setting in which φ ( t ) is a continuous function of t .

Empirically, either at initialization or after some training, we often observe a 'figure 8' pattern in Y of roughly the form ( h ∞ ◦ g )( θ ) = (cos θ sin θ, sin θ ) glyph[latticetop] . In such a case, the embedding into Z contains two disjoint pieces [ -3 π/ 4 , -π/ 4] ∪ [ π/ 4 , 3 π/ 4] . One half of the circle is mapped to one piece at

the top of the circle and the other half of the circle is mapped to the other disjoint piece at the bottom of circle. Such a mapping takes advantage of the singularity at (0 , 0) 2 into order to embed S 1 in two continuous pieces. The resulting mapping is nearly bijective, failing to reconstruct on only a small region near the two discontinuities. It is also mostly continuous, having only two discontinuities.

Once this local minimum is obtained, it is very difficult to move out of it using gradient descent. It is unlikely for the two pieces to join together and become a homeomorpic embedding since this would require passing one disjoint segment through the other and reversing its orientation which would violate bijectivity and increase the reconstruction loss.

We make this observation precise by noting that continuous optimization preserves the ordering of points on the circle. Let ( z 1 , z 2 , z 3 , z 4 ) denote the four end points of the two disjoint intervals of f φ (0) . The ordering of these points is only defined up to cyclic permutations mod C 4 . Proposition 3.1 states that continuous optimization must preserve ordering mod C 4 . Figure 3 illustrates the proof.

Proposition 3.1 Assume that f φ ( t ) undergoes continuous optimization. Assume that f φ ( t ) ◦ g is injective for all t . The cyclic ordering induced on k points by f φ (0) is equal to f φ (1) . Thus a figure 8 embedding, which corresponds to cyclic order ( z 1 , z 2 , z 4 , z 3 ) mod C 4 , cannot be transformed to a homeomorphic embedding, which has cyclic order ( z 1 , z 2 , z 3 , z 4 ) mod C 4 or ( z 4 , z 3 , z 2 , z 1 ) mod C 4 .

In other words, transition from a figure 8 embedding to a homeomorphic embedding is impossible without violating continuity during optimization. This indicates that escaping a figure 8 local optimum during training would need to rely on discrete jumps and likely the stochasticity of the gradient estimate.

3.2 Degree Obstructions

A second class of topological defects that can arise are encodings with a discrepancy in the winding number. We can compute the degree, or winding number, of a map ψ : S 1 → S 1 around the origin by summing up the differentials along its path on the sphere,

This concept can be extended to arbitrary connected oriented manifolds, where it is usually referred to as the degree of a mapping. Intuitively, it describes the number of times that the domain manifold wraps around the co-domain manifold .

glyph[negationslash]

If the embedded image h φ ( t ) ( M x ) does not contain the origin, then the mapping factors through R 2 \ { (0 , 0) } and is consequently continuous. Therefore, a continuous path in φ yields a continuous path in Z . If f φ | M x is continuous, then the function ψ φ := π ◦ h φ ◦ g : S 1 → S 1 has a well-defined degree, also known as winding number w ( ψ φ ) ∈ Z . In order for f φ | M x to be a homeomorphism, the winding number must be w ( ψ φ ) ∈ {-1 , 1 } . Under random initialization, however, the initial network may have winding number equal to any integer. Assuming continuous optimization and continuous embeddings, then h φ ( t ) ( x ) is a continuous function of both x, t . We assume that h φ ( t ) ( x ) = (0 , 0) for any t, x and thus winding number is defined for any time. The following proposition thus holds.

Proposition 3.2 Winding numbers of the initialized and final model are equal w ( ψ φ (0) ) = w ( ψ φ (1) ) .

In practice, neither the continuous optimization assumption nor the avoidance of the origin holds. Rather h φ is updated by SGD in discrete jumps and h φ ◦ g may map to the origin. Thus, empirically, we do see that the winding number may change during training. However, if the initialization avoids

GF-VAE

t=0

t=1

t=2

t=3

t=4

the origin, then due to the tendency of the magnitude of the unnormalized embeddings h φ ( x ) to grow during optimization (see Section 3.3), the winding number changing becomes more unlikely. This means that defects in the winding number pose significant obstruction to learning homeomorphic embeddings. The winding number is also the primary optimization obstruction which makes it impractical to remove the hard projection π . If instead we decode directly from y ∈ Y but push embeddings to the unit circle using the loss |‖ y ‖ -1 | , then it is far more likely we converge to discontinuous embeddings with the incorrect winding number (Appendix D).

3.3 Magnitude Growth in Y

Empirically, we observe the values of the embeddings in Y := R 2 continually grow during training. This phenomenon makes it more difficult for the embedded data manifold h φ ( M x ) to cross the origin and for the winding number to change. We give a theoretical explanation for this behavior.

Consider what would happen if the embedding y were updated directly based on the gradient of the loss ∇ y L with respect to y . We assume the loss depends only on z = y/ ‖ y ‖ , and so has level sets which are unions of radial rays from the origin. The gradient ∇ y L must then be tangent to a circle about the origin. That is, for y = ( a, b ) , the gradient ∇ y L = ( ± b, ∓ a ) . Under gradient flow, the evolution of y in time y t would thus flow along circles of fixed radius and so ‖ y 0 ‖ = ‖ y t ‖ . Under gradient descent, however, due to the convexity of the flow lines, which are circular, the embeddings y will tend to grow in magnitude. For η ∈ R > 0 , we compute

In practice, however, we do not update y based on ∇ y L but rather based on the gradient with respect to model parameters φ . Let F : Φ →Y be the map from model parameters φ to y given fixed input data x . Then the actual update to y is ˜ ∇ y L = dF T ◦∇ φ L where dF is the total derivative or Jacobian of the map F . Since ∇ φ L = ( dF ) ∇ y L we have ˜ ∇ y L = dF T dF ∇ y L . The angle between ˜ ∇ y L and ∇ y L is bounded by some θ a quantity depending on the eigenvalues of the operator dF . Given that ∇ y L is tangential to the circle, assuming for simplicity ˜ ∇ y L has constant length L and uniform distribution [ -θ, θ ] in angle to ∇ y L , the norm of y still grows in expectation .

Proposition 3.3 Assume a circle of radius R . Let v be a random vector at y on the circle of length L with angle to the tangent uniform in [ -θ, θ ] . Then

4 GroupFlow-VAE

The topological obstructions to optimization that we identified in Section 3 cannot be resolved under continuous optimization. Moreover, even if we allow for a degree of discontinuity, resolving obstructions like a figure 8 will require violating bijectivity, which implies that the reconstruction loss

must increase to escape this defect. This suggests that a V AE with a reparameterized construction as described in Equation 3 will be susceptible to local optima, which aligns with the empirical observation that homeomorphic VAEs can be difficult to train.

This raises the question of whether we can make V AEs less susceptible to topological obstructions by employing a different parameterization from that of Equation 3. More concretely, we will consider a parameterization that admits multiple modes in the variational distribution q φ ( z | x ) , rather than the unimodal construction in Equation 3, and define f φ ( x ) in terms of the mode,

The intuition behind this approach is illustrated in Figure 4. If the variational distribution contains multiple modes, rather than a single peak centered at π ( h φ ( x )) , then it may becomes possible for small changes to result in non-local changes, such as the reordering of points that is needed to untwist a figure 8, by switching between modes in the variational distribution. Moreover, increasing the number of parameters of q φ ( z | x ) is likely beneficial for escaping some of the topological obstructions as some defects such as self-intersection are less likely to occur in high-dimensional spaces.

Motivated by this intuition, we consider a parameterization of q φ ( z | x ) that employs normalizing flows [Rezende and Mohamed, 2015]. In a normalizing flow, the sample z is defined as a push forward of sequence of smooth bijective transformations, which makes it possible to reshape a simple unimodal distribution into a more complex multimodal distribution. The probability of a sample from the final density can be computed by repeatedly applying the rule for change of variables. Concretely, given a base distribution p ( z 0 ) and a sequences of bijective transformations r k : Z → Z , we obtain a sample z = z K and probability by first sampling z 0 ∼ p ( z 0 ) and defining a sequence of transformations

Normalizing flows have been used to define distributions on geometric structures such as Lie groups by either defining a flow on the Lie algebra and computing the push-forward density of the exponential map [Falorsi et al., 2019], or designing structure-specific transformations [Rezende et al., 2020].

In the GF-VAE, we will define a construction in which the encoder network returns the parameters of the flow. We show an overview of the architecture in Figure 1. We define a network h φ : X → Y := R K × l where K and l are the number of flow layers and parameters respectively, a sequence of bijective transformations { r ( · ; y k ) } K k =1 parameterized by { y k } K k =1 , and a base distribution distribution p ( z 0 ) which we define as a distribution on the group. The sequence { r ( · ; y k ) } K k =1 is then used to define a new distribution on Z given an x by transforming the base distribution. This defines a conditional flow that can be trained using a stand lower bound (Eq. 2),

The choice of the flow r is very important here as normalizing flows are typically defined on flat spaces. This means that, for a specific manifold M , additional care must be taken when designing them to ensure that they are a diffeomorphism from Z to itself [Rezende et al., 2020, Durkan et al., 2019, Mathieu and Nickel, 2020]. In this paper, we employ a similar method as Falorsi et al. where we define an affine layer followed by a single layer of spline flow, and a Tanh layer multiplied by π as the last layer to push all the probability density between -π and π .

5 Experiments

We perform a series of experiments to evaluate the difficulty of learning a homeomorphic encoder. Concretely, we investigate how often we fall into one of the failure cases described in Section 3 in standard VAEs with geometric latent spaces in practice. Subsequently, we examine how well GFVAE can circumvent these topological obstructions during training.

Baselines. Throughout our experiments, we compare against (1) a standard VAE, and (2) a supervised VAE where in addition to maximizing the ELBO, the encoder is trained to predict the ground truth representations. These two scenarios serve as extremes on the spectrum of guiding the model to the right representation. We also compare against a deterministic autoencoder (AE) which does not

VAE

VAE

0

2

GF-VAE

GF-VAE

丫

丫

丫

o00o0ooo0ooo0oo

regularize the embedding space. We also tried regularizing the y-space to be close S 1 (by penalizing ( ‖ y ‖ 2 -1) 2 ) in order to mitigate the optimization problem discussed in Subsection 3.3, which we refer to as 'regy ' loss. Finally, we evaluate the β -VAE objective [Higgins et al., 2017] which increases the regularization on the latent space by upweighting the KL term in Eq 2. All models employ a 4-layer CNN architecture for the encoder and decoder with LeakyReLU activations. For the decoder, we also experiment with action-decoder proposed by Falorsi et al., which we found helpful for learning a homeomorphic mapping. The action-decoder uses a special first layer where the group action is applied to a set of learned Fourier coefficients rather than directly being passed as input to the architecture. For further details regarding our experiments, please refer to Appendix A.

Evaluation. We evaluate all models based on two criteria: (1) Has the encoder learned a homeomorphic mapping? and (2) Has the decoder learned a good model of the data? Assessing whether a learned mapping is homeomorphic is challenging. To verify homeomorphism, we follow the evaluation proposed in Falorsi et al. [2018] by examining whether the encoder yields a continuous path when interpolating in the data manifold from -π to π . Details on evaluating continuity are provided in Appendix C. To determine how often the models encounter the topological obstructions described in Section 3, we also report crossing and winding numbers in Appendix E. A crossing number grater than 0 implies a 'figure 8' obstruction, and a winding number that is not equal to 1 or -1 implies winding number obstruction. Lastly, we measure the log-likelihood to assess how well each model approximates the data manifold. If the model has diverged during training due to posterior collapse, we report it as a non-homeomorphic mapping and ignore its continuity score in the average.

5.1 Images: SO(2)

In our first experiment, we train on images of an L-shaped tetromino [Bozkurt et al., 2021], a teapot, and an airplane [Shilane et al., 2004]. The SO(2) manifold corresponding to each object is made by rotating the image of the object around the center. We report our findings in Tables 1 and 3.

Colour

Orientation

Weobserve that even though the types of obstructions vary across images, both V AE and AE in general fail to learn a homeomorphic encoder. The GF-VAE objective improves performance noticeably across both metrics. We also did not find y regularization to be very helpful as it mainly stabilized training towards whatever mapping that was achieved at the early stages of training. We observed the main failure case for most of the non-homeomorphic encoders was due to discontinuity emerging from figure-eight obstruction. Moreover, looking at the learning curves in Figure 9, we observe that the winding number is susceptible to change during training which suggests that the continuity assumption is of importance in proposition 3.2. GF-VAE resolves both issues, which allows us to interpolate nicely in the Z -space (Figure 5). What is very surprising is that in the case of airplanes, we see that even supervised objective fails to overcome these optimization obstructions, while a GFVAE is able to achieve this at a much better rate.

We also observe that increasing the β value for the KL term generally helps. This is perhaps unsurprising given that a high β value encourages the latent space to cover the prior and therefore discourages the winding numbers from being 0 and improves the performance on the continuity metric. In the case of the teapots, we in fact observe that increasing β is sufficient to learn a homeomorphic encoder. However, GF-VAE still generally scores better in terms of continuity and winding numbers.

5.2 Images: Torus

In this experiment, we consider a ring torus as the latent space, which is homeomorphic to the Lie group SO(2) × SO(2) . We create a dataset homeomorphic to this group by independently rotating the L-shaped tetromino in both orientation and color. All models can be extended to Tori by simply duplicating the latent space and learning multiple encodings to SO(2) in parallel. This is done by defining a factorized density q φ ( z (1) , z (2) | x ) = q φ ( z (1) | x ) q φ ( z (2) | x ) , where each distribution q φ ( z ( i ) | x ) defines a distribution on a S 1 . Ideally, we want one subgroup to correspond to colour and the other to orientation. In this setting, we evaluate homeomorphism by picking 10 different values in either colour or orientation and measuring the continuity of the encoded path when interpolating in the data manifold on the other attribute. We identify the encoder as homeomorphic if the average continuity score of all 10 paths is below a certain threshold (see Appendix C). Unsurprisingly, we found that learning a homeomorphic mapping in tori is more challenging compared to circles. Out of 15 runs, the V AE models learn a homeomorphic mapping 1 time, while a GF-VAE manages to learn a homeomorphic mapping 7 times. To be able to align each Lie group with the corresponding attribute, we also used the weakly supervised proposed by Locatello et al., where the model receives a sequence of images in which only the angle or the colour is changing. We show the latent traversal in the decoder model for one of the successful runs of GF-V AE in Figure 6. As we can see, the model has successfully managed to disentangle colour and orientation in the latent space.

6 Related Work

Learning Geometric Representations. There has been a large amount of work concerned with learning representations from data with geometric structure in the unsupervised or weakly supervised setting. One line of work has focused on the use of geometric spaces such as lie groups in latent space [Davidson et al., 2018, Falorsi et al., 2018, Perez Rey et al., 2020, Vadgama et al., 2022]. In [Miao et al.], it was argued that that geometric inductive biases in V AEs should be Incorporated via a deterministic mapping rather than the prior which is consistent with our work. As we show in this

work, naively incorporating a geometric bias leads to topological obstructions. Another line of work has focused on inferring the latent structure by exploring the local connectivity information [Moor et al., 2020, Chen et al., 2021, Lee et al., 2021, Pfau et al., 2020].

Learning Disentangled Representations. Topological group structure can be used to define a notion of disentanglement that is based on equivariant properties under group transformations [Higgins et al., 2018]. There exists a body of work that aims to learn both disentangled and equivariant group representations. One set of methods relies on agent actions to predict the group element [CasellesDupré et al., 2019, Quessard et al., 2020]. An adjacent line of work is to regularize encoders to be equivariant with respect to the group action, using triplets of the form ( x t , m t , x t +1 ) or longer sequences [Guo et al., 2019, Dupont et al., 2020, Tonnaer et al., 2022]. Some approaches focus on learning symmetry-based disentangled representations in fully unsupervised settings or by enforcing commutativity in the latent Lie group [Yang et al., 2022, Zhu et al., 2021]. Another related area disentangles class and pose in the latent space [Marchetti et al., 2022, Winter et al., 2022]. Our work does not focus on disentanglement, but the topological obstructions that we describe are relevant to this domain.

Topological Obstructions in Learning. The topological obstructions that we consider in this work are distinct from the homological obstructions that have been characterized in prior work [de Haan and Falorsi, 2018, Batson et al., 2021, Falorsi et al., 2018, Perez Rey et al., 2020]. Theorem 1 by de Haan and Falorsi [2018] defines homological obstructions as follows: For any latent space Z with non-trivial topology, it is possible to learn an encoder f φ that is continuous when restricted to M x ⊂ X , but this encoder must be discontinuous on the full space X . For this reason, Falorsi et al. [2018] and others [Xu and Durrett, 2018, Meng et al., 2019] use the two-part encoder from equation 1, inserting discontinuous layers π when mapping to circles, spheres, SO( n ) , or other manifolds. This explicit discontinuity circumvents the homological obstruction without forcing the linear layers of the network to approximate discontinuities using large weights, which we and others find leads to instability during training and inferior reconstructions (Section 5).

Normalizing Flows on Manifolds. In recent years, there has been a surge of interest in extending normalizing flows, originally formulated for Euclidean spaces, to Riemannian manifolds [Rezende et al., 2020, Mathieu and Nickel, 2020, Köhler et al., 2021, Durkan et al., 2019]. One approach involves leveraging the Lie group structure of the manifold to define a parametrization of the flow [Rezende et al., 2020], which is also the strategy we adopt in this work. These recent advancements have paved the way for applying normalizing flows to the group SO(3) in order to learn pose estimation in molecular structures [Köhler et al., 2023] and images [Liu et al., 2023, Murphy et al., 2021]. Our work differs from these efforts in that the primary objective of our method is not to target flexible distributions on Riemannian manifolds, but rather to demonstrate that utilizing a flow as a variational distribution aids optimization in learning a homeomorphic embedding.

7 Conclusion

In this paper, we investigate obstructions to optimization that can arise when learning encoders for topological spaces. We classify different types of obstructions, provide evidence these are encountered in practice, and give mathematical explanations for how they occur under certain assumptions. We propose GF-VAE, a novel model that employs normalizing flows as variational distributions to help circumvent these issues and show its effectiveness across several datasets when encoding to circles and tori. This work contains several limitations. Firstly, our theoretical analysis is limited by the idealized assumptions necessary to analyze the method using topological tools which do not exactly match those encountered in practice. Secondly, the metrics we define such as winding and crossing numbers are harder to define and compute for higher dimensional manifolds. Future work includes expanding our analysis and techniques to a wider array of Lie groups and non-group manifolds.

Acknowledgments and Disclosure of Funding

This work was supported by NSF grants #2107256 and #2134178. We also would like to thank Patrick Forré, Sharvaree Vadgama, Marco Federici, and Erik Bekker for helpful discussions.

References

Joshua Batson, C Grace Haaf, Yonatan Kahn, and Daniel A Roberts. Topological obstructions to autoencoding. Journal of High Energy Physics , 2021(4):1-43, 2021.

Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8):1798-1828, 2013.

Alican Bozkurt, Babak Esmaeili, Jean-Baptiste Tristan, Dana Brooks, Jennifer Dy, and Jan-Willem van de Meent. Rate-regularization and generalization in variational autoencoders. In International Conference on Artificial Intelligence and Statistics , pages 3880-3888. PMLR, 2021.

Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veliˇ ckovi´ c. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478 , 2021.

Hugo Caselles-Dupré, Michael Garcia Ortiz, and David Filliat. Symmetry-based disentangled representation learning requires interaction with environments. Advances in Neural Information Processing Systems , 32, 2019.

Xingyu Chen, Chunyu Wang, Xuguang Lan, Nanning Zheng, and Wenjun Zeng. Neighborhood geometric structure-preserving variational autoencoder for smooth and bounded data sources. IEEE Transactions on Neural Networks and Learning Systems , pages 1-14, 2021. doi: 10.1109/TNNLS. 2021.3053591.

Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning , pages 2990-2999. PMLR, 2016.

Tim R. Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M. Tomczak. Hyperspherical Variational Auto-Encoders. In Amir Globerson and Ricardo Silva, editors, Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018 , pages 856-865. AUAI Press, 2018. URL http://auai.org/uai2018/ proceedings/papers/309.pdf .

Pim de Haan and Luca Falorsi. Topological Constraints on Homeomorphic Auto-Encoding. In NIPS Workshop , December 2018. URL http://arxiv.org/abs/1812.10783 . arXiv: 1812.10783.

Emilien Dupont, Miguel Bautista Martin, Alex Colburn, Aditya Sankar, Josh Susskind, and Qi Shan. Equivariant neural rendering. In International Conference on Machine Learning , pages 2761-2770. PMLR, 2020.

Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural spline flows. Advances in neural information processing systems , 32, 2019.

Luca Falorsi, Pim de Haan, Tim R. Davidson, Nicola De Cao, Maurice Weiler, Patrick Forré, and Taco S. Cohen. Explorations in Homeomorphic Variational Auto-Encoding. In ICML 2018 workshop on Theoretical Foundations and Applications of Deep Generative Models , 2018. URL http://arxiv.org/abs/1807.04689 . arXiv: 1807.04689.

Luca Falorsi, Pim de Haan, Tim R Davidson, and Patrick Forré. Reparameterizing distributions on lie groups. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 3244-3253. PMLR, 2019.

Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic neural networks. Advances in neural information processing systems , 31, 2018.

Xifeng Guo, En Zhu, Xinwang Liu, and Jianping Yin. Affine equivariant autoencoder. In IJCAI , pages 2413-2419, 2019.

Brian C Hall and Brian C Hall. Lie groups, Lie algebras, and representations . Springer, 2013.

Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations , 2017. URL https://openreview.net/forum?id=Sy2fzU9gl .

Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a Definition of Disentangled Representations. arXiv:1812.02230 [cs, stat] , December 2018. URL http://arxiv.org/abs/1812.02230 . arXiv: 1812.02230 version: 1.

Irina Higgins, Sébastien Racanière, and Danilo Rezende. Symmetry-based representations for artificial and biological general intelligence. Frontiers in Computational Neuroscience , page 28, 2022.

Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014.

Jonas Köhler, Andreas Krämer, and Frank Noé. Smooth normalizing flows. Advances in Neural Information Processing Systems , 34:2796-2809, 2021.

Jonas Köhler, Michele Invernizzi, Pim de Haan, and Frank Noé. Rigid body flows for sampling molecular crystal structures. arXiv preprint arXiv:2301.11355 , 2023.

Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International Conference on Machine Learning , pages 2747-2755. PMLR, 2018.

Yonghyeon Lee, Hyeokjun Kwon, and Frank Park. Neighborhood reconstructing autoencoders. Advances in Neural Information Processing Systems , 34, 2021.

Mario Lezcano-Casado and David Martınez-Rubio. Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group. In International Conference on Machine Learning , pages 3794-3803. PMLR, 2019.

Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representations , 2019.

Yulin Liu, Haoran Liu, Yingda Yin, Yang Wang, Baoquan Chen, and He Wang. Delving into discrete normalizing flows on so (3) manifold for probabilistic rotation modeling. arXiv preprint arXiv:2304.03937 , 2023.

Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Oliver Bachem. Disentangling factors of variations using few labels. In Eighth International Conference on Learning Representations . OpenReview. net, 2020.

Giovanni Luca Marchetti, Gustaf Tegnér, Anastasiia Varava, and Danica Kragic. Equivariant representation learning via class-pose decomposition. arXiv preprint arXiv:2207.03116 , 2022.

Emile Mathieu and Maximilian Nickel. Riemannian continuous normalizing flows. Advances in Neural Information Processing Systems , 33:2503-2515, 2020.

Yu Meng, Jiaxin Huang, Guangyuan Wang, Chao Zhang, Honglei Zhuang, Lance Kaplan, and Jiawei Han. Spherical text embedding. Advances in Neural Information Processing Systems , 32, 2019.

Ning Miao, Emile Mathieu, N Siddharth, Yee Whye Teh, and Tom Rainforth. On incorporating inductive biases into vaes. In International Conference on Learning Representations .

Michael Moor, Max Horn, Bastian Rieck, and Karsten Borgwardt. Topological autoencoders. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 7045-7054. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/moor20a.html .

Kieran A Murphy, Carlos Esteves, Varun Jampani, Srikumar Ramalingam, and Ameesh Makadia. Implicit-pdf: Non-parametric representation of probability distributions on the rotation manifold. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 7882-7893. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/murphy21a.html .

Jung Yeon Park, Ondrej Biza, Linfeng Zhao, Jan Willem van de Meent, and Robin Walters. Learning symmetric embeddings for equivariant world models. In International Conference on Machine Learning , 2022.

Luis A. Perez Rey, Vlado Menkovski, and Jim Portegies. Diffusion Variational Autoencoders. In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20 , pages 2704-2710. International Joint Conferences on Artificial Intelligence Organization, July 2020. doi: 10.24963/ijcai.2020/375. URL https://doi.org/10. 24963/ijcai.2020/375 .

David Pfau, Irina Higgins, Alex Botev, and Sébastien Racanière. Disentangling by subspace diffusion. Advances in Neural Information Processing Systems , 33:17403-17415, 2020.

Robin Quessard, Thomas Barrett, and William Clements. Learning disentangled representations and group structure of dynamical environments. Advances in Neural Information Processing Systems , 33, 2020.

Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning , pages 1530-1538. PMLR, 2015.

Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning , pages 1278-1286. PMLR, 2014.

Danilo Jimenez Rezende, George Papamakarios, Sébastien Racaniere, Michael Albergo, Gurtej Kanwar, Phiala Shanahan, and Kyle Cranmer. Normalizing flows on tori and spheres. In International Conference on Machine Learning , pages 8083-8092. PMLR, 2020.

Philip Shilane, Patrick Min, Michael Kazhdan, and Thomas Funkhouser. The princeton shape benchmark. In Proceedings Shape Modeling Applications, 2004. , 2004.

Loek Tonnaer, Luis Armando Perez Rey, Vlado Menkovski, Mike Holenderski, and Jim Portegies. Quantifying and learning linear symmetry-based disentanglement. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 21584-21608. PMLR, 17-23 Jul 2022. URL https://proceedings. mlr.press/v162/tonnaer22a.html .

Sharvaree Vadgama, Jakub Mikolaj Tomczak, and Erik J Bekkers. Kendall shape-vae: Learning shapes in a generative framework. In NeurIPS 2022 Workshop on Symmetry and Geometry in Neural Representations , 2022.

Robin Winter, Marco Bertolini, Tuan Le, Frank Noe, and Djork-Arné Clevert. Unsupervised learning of group invariant and equivariant representations. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=47lpv23LDPr .

Jiacheng Xu and Greg Durrett. Spherical latent spaces for stable variational autoencoders. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 4503-4513, 2018.

Tao Yang, Xuanchi Ren, Yuwang Wang, Wenjun Zeng, and Nanning Zheng. Towards building a group-based unsupervised representation disentanglement framework. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=YgPqNctmyd .

Xinqi Zhu, Chang Xu, and Dacheng Tao. Commutative lie group vae for disentanglement learning. In International Conference on Machine Learning , pages 12924-12934. PMLR, 2021.

A Experimental Details

We train all our models for 150 epochs with a batch-size of 600. For optimization, we use the RAdam optimizer [Liu et al., 2019] with a learning rate of 5e-4. For all image datasets, we use a 4-layer CNN with kernel, stride, and padding of size 4, 2 and 1 respectively followed by a leakyReLU activation (Table 2) for the encoder and a Sigmoid activation for the decoder. The network σ 2 glyph[epsilon1] , shares the same architecture with the only difference being the last layer, which is a fully-connected network followed by a Softplus activation which is common in standard VAEs. In all our experiments, we used K = 1 for the GF-VAE models as it was sufficient to avoid the optimization obstructions mentioned in the paper. All models were initialized and trained with 15 3 different random seeds.

B Proofs

We include the proofs for the propositions in the main text.

B.1 Figure Eight Local Minimum

Proposition B.1 Assume that f φ ( t ) undergoes continuous optimization and is thus continuous in t . Assume that π ◦ f φ ( t ) ◦ g is injective for all t . The cyclic ordering induced on k points by f φ (0) is equal to that induced by f φ (1) . Thus a figure 8 embedding, which corresponds to cyclic order ( z 1 , z 2 , z 4 , z 3 ) mod C 4 , cannot be transformed to a homeomorphic embedding, which has cyclic order ( z 1 , z 2 , z 3 , z 4 ) mod C 4 or ( z 4 , z 3 , z 2 , z 1 ) mod C 4 .

glyph[negationslash]

Proof: Since we assume π ◦ f φ ( t ) ◦ g is injective for all t , the path z ( t ) = ( π ◦ f φ (0) ◦ g ( θ i )) 4 i =1 is inside the k -fold configuration space on S 1 defined Conf k ( S 1 ) = { ( z 1 , . . . , z k ) ∈ ( S 1 ) k : z i = z j for i = j } . In order to prove the claim, we will show that the path-connected components of Conf k ( S 1 ) correspond to cyclic orderings of ( z 1 , . . . , z k ) and thus the start and end point of every path share a cyclic ordering.

glyph[negationslash]

Mapping ( z 1 , . . . , z k ) ↦→ ( z k , ( z -1 k z 1 , . . . , z -1 k z k -1 )) gives a homeomorphism Conf k ( S 1 ) ∼ = SO(2) × Conf k -1 ( S 1 \ { 1 } ) ∼ = SO(2) × Conf k -1 ( R ) . Let ˜ z i = z -1 k z i . Consider D = { (˜ z 1 , . . . , ˜ z k -1 ) : ˜ z 1 < . . . < ˜ z k -1 } ⊂ Conf k -1 ( R ) .

We can identify the connected components of Conf k -1 ( R ) . The set D is a fundamental domain for the action of the symmetric group S k -1 on Conf k -1 ( R ) . Thus Conf k -1 ( R ) = ∐ σ ∈ S k σ ( D ) is a disjoint union. Linear interpolation shows D is connected. The sets D and σ ( D ) for σ ∈ S k are not connected. Consider a path from z = ( z 1 , . . . , z k ) ∈ D to σ ( z ) ∈ σ ( D ) . The element σ must reverse the order of at least two elements z j < z i . Thus the function f ( z ) = z i -z j must take the value 0 over the path by intermediate value theorem. Hence the path cannot be in Conf k -1 ( R ) . Thus the connected components of π 0 (Conf k -1 ( R )) ∼ = S k -1 .

Since SO(2) is connected, π 0 (Conf k ( S 1 )) ∼ = S k -1 . That is each connected component of Conf k ( S 1 ) is labeled by an element of S k -1 describing the ordering of ˜ z 1 , . . . , ˜ z k -1 in R . Each ordering of (˜ z 1 , . . . , ˜ z k -1 ) in turn corresponds to a different cyclic ordering of z 1 , . . . , z k in S 1 , that is, a different element of S k /C k . Thus two k -point configurations are homotopic if and only if they have the same cyclic ordering. glyph[square]

B.2 Degree Obstruction

Proposition B.2 The winding number of the initialized model and final model are equal w ( π ′ ◦ h φ (1) ◦ g ) = w ( π ′ ◦ h φ (1) ◦ g ) .

Proof: The winding number of a map is a continuous function t ↦→ w ( π ′ ◦ h φ ( t ) ◦ g ) . Since the output space Z is discreet, the winding number must be constant in t . glyph[square]

B.3 Magnitude Growth in Y

We assume that dF is full rank , which is a reasonable assumption for an overparameterized neural network. In that case M = dF T dF is a positive definite symmetric matrix and can be orthogonally diagonalized M = Q Λ Q T where Q is orthogonal and

and λ i > 0 . The maximum angle between x = ∇ y L and M x = ˜ ∇ y L can then computed in terms of the eigenvalues λ i . This maximum is computed for the case of an n × n symmetric positive definite matrix here 4 . We include the proof for the 2 × 2 case we consider here for completeness.

glyph[negationslash]

Lemma B.1 The maximum angle between x and Mx for x ∈ R 2 =0 is

Proof: The angle is maximized at the minimum value of

It suffices to consider ‖ x ‖ = 1 . Substituting M = Q Λ Q T and y = Qx , we want to minimize

over all ‖ y ‖ = 1 since ‖ Qx ‖ = ‖ x ‖ = 1 . Letting a = y 2 1 and noting y 2 1 + y 2 2 = 1 , this is equal to minimizing

over 0 ≤ a ≤ 1 . Setting the derivative equal to 0 gives

and yields one critical value at a = λ 2 / ( λ 1 + λ 2 ) corresponding to value 2 √ λ 1 λ 2 λ 1 + λ 2 . This is the global minimum since the boundary values a = 0 and a = 1 correspond to maxima with value 1. glyph[square]

Thus the angle between ∇ y L and ˜ ∇ y L is bounded by θ = cos -1 ( 2 √ λ 1 λ 2 / ( λ 1 + λ 2 ) ) . Given that ∇ y L is tangential to the circle, assuming for simplicity ˜ ∇ y L has constant length L and uniform distribution [ -θ, θ ] in angle to ∇ y L , the norm of y grows in expectation .

Proof: Without loss of generality, y = (0 , R ) and v = ( L cos t, L sin t ) where | t | < θ . Then we evaluate

by Pythagorean identity and the fact sin t is odd.

C Continuity Metric

For measuring continuity, we adopt a similar method as Falorsi et al. and evaluate continuity in terms of how the largest 'jump' compare to others when walking a continuous path m i ∈ M for i = 1 · · · N pairwise close points. We define q i to be the ratio distances between the ground-truth and the learned representation

From the set { q i } i , we compute the continuity metric L cont as

In our experiments, we set α = 90 .

There are two differences between how we evaluate continuity compared to [Falorsi et al., 2018]. First, we measure the continuity of ψ φ rather than f φ , which we argue is more relevant. Second, in [Falorsi et al., 2018], the authors are mainly interested in verifying whether the encoder is discontinuous in the topological sense (which they verify by examining the inequality M > γP α for some γ ). We on the other hand report continuity on the spectrum by computing the γ that would make η φ discontinuous. For evaluating homeomorphism, we conclude for an encoder to be homeomorphic if L cont < 10 (empirically, we observed the mapping to appear smooth for a continuity score below this threshold).

glyph[square]

D Decoding From Y

We consider the alternate strategy of removing the projection π and adding a loss so that y stays close to the desired manifold M in Y . Winding number obstructions become far more prominent in this case. In Figure 7, we show the latent space for different random seeds in the teapot case when we train with such an objective.

2

1.0

1.0

2

1.0

1

0.5

0.5

1

0.5

0

×

F00

×

0.0

0

×

0.0

-1

0.5

-1

0.5

-1

0.5

-2

1.0

-1.0

-1.0

-2

0

2

-1

0

-2

0

2

-1

0

-2

0

2

-1

0

Z

C

Z

CC

Z

2

1.0

4

1.0

2

1.0

0.5

2

0.5

0.5

×

0.0

0.0

0

0.0

0

×

0.5

0.5

-1

0.5

1.0

-2

1.0

1.0

0

2

-1

2.5

0.0

2.5

-1

0

0

2

1

Z

C

Z

C

Z

1.0

1.0

1.0

2

2

0.5

1

0.5

0.5

F

×

0.0

0

×

0.0

0

×

0.0

0.5

-1

0.5

0.5

1.0

-2

-1.0

1.0

2.5

0.0

2.5

-1

0

2

0

2

0

0

2

1

0

C

Z

Z

C

Z

1.0

1.0

1.0

0.5

1

Fs'0

2

0.5

0.0

0

×

0.0

0

×

0.0

F

0.5

-1

0.5

0.5

F

-2

1.0

2

-1.0-

1.0

-2.5

0.0

2.5

-1

0

-2

0

2

0

2.5

0.0

2.5

-1

0

C

y

1.0

1.0

2

1.0

0.5

0.5

1

Fs'0

X

F00

0

×

0.0

0

×

0.0

0.5

0.5

-1

0.5

1.0

-1.0

1.0

-2

0

2

-1

0

-2

0

2

-1

0

1

2

0

2

0

E Experiments: Additional Results

We report the full results of our SO(2) experiments in Table 3.

Table 3: Comparison of different VAEs trained on the various image datasets in terms of number of encoders with homeomorphic mappings (# H.), correct winding number (# W.), and correct crossing number (# C.) for 15 random seeds. We additionally report the error on continuity as well as negative Loglilkelihood (lower is better).

F Additional Figures

VAE

VAE

0

2

GF-VAE

GF-VAE

VAE

VAE

0

2

GF-VAE

GF-VAE

0

20

40

60

80

100

120

140

1.0

0.5

0.0

0.5

1.0

VAE (

=4)

Winding Number

0

20

40

60

80

100

120

140

1.0

0.5

0.0

0.5

1.0

1.5

GF-VAE (

=4)

seed

1

2

3

4

5

0

20

40

60

80

100

e

0

50

100

150

200

250

|

y

|2

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors investigate two types of topological obstructions that pose challenges for models aimed at learning a particularly structure in the embedding space. Specifically, the authors identify figure eight local minima and mismatches in winding numbers are two defects that make learning the right latent structure difficult. The authors propose a VAE method to improve the learning procedure in order to better reflect the underlying data’s structure. The authors run experiments on three shapes and measure continuity of the learned space in comparison to existing VAE models.\n\nSTRENGTHS:\n- Understanding challenges arising in the learning dynamics of models to ensure they appropriately reflect the underlying structure of data is an important topic.\n- Authors acknowledge a reasonable set of limitations in the conclusion.\n- It’s nice to see the authors applying tools from topology to examine how well the right structure can be learned by a model’s embedding space.\n\nWEAKNESSES:\n- The writing flow for Section 2 and 3 could use more work to improve clarity\n\t- Problem statement would benefit from an illustrative example of an application for f, h, and pi to help make the setup in line 61-64 more clear. I’d suggest the authors consider moving the running example earlier or using another application to motivate and illustrate this setup.\n\t- Figure 2 isn’t referenced at all in the running examples section and doesn’t appear in the text until Section 3 (line 107). I’d recommend moving the reference to Figure 2 earlier. \n\t- Figure 2: the figure would benefit from additional labels to indicate what each component is. For example, M as I understand is comprised of points for each degree of rotation for the object. The three right hand diagrams should be labeled clearly to indicate they are 3 runs (with different seeds).\n- I found the empricial evidence, especially for the Torus experiment in 5.2 to be quite weak. For example, the authors base the results on only 5 runs grounding the conclusion on the fact that “VAE learns a homeomorphic mapping 0 times” while “GF-VAE learns the mapping 2 out of 5 times”. I don’t find this sufficiently convincing evidence as the authors test a single learning rate and setup across only 5 runs. I would recommend the authors run many more trials to prove this out. \n- Unfair comparison to basleines: In the appendix, the full table shows Beta-VAE (with beta = 4) is quite an effective baseline, but this is absent from the main table 1 in the paper, which only compares against vanilla AE and VAE even though the GF-VAE model uses a beta term. This is not an apples-to-apples comparison to tease out the impact of the author’s proposed method. I suggest the authors include beta-VAEs in the main table and appropriately compare their method to the same beta-VAE baseline.\n- I’m also surprised the authors did not include a comparison to other topological models that impose equivariance to Lie groups, but only to standard VAE models that do not impose group structure in the latent space.\n\nDetails\n- typo in citation line 74\n- typo line 227: “stand” → \"standard”\n- typo 239: extra space\n\nQUESTIONS:\n- Why does the setup include a projection function pi form y to Z? You cite Falorsi, but I couldn’t find intuition or motivation for this at all in the paper. Could you elaborate?\n- How prevalent in practice is the figure eight local minima problem you identify in 3.1? This seems like a very specific topological artificat. Is it found real applications, or does it arise in training of deep networks? \n- Could you explain the justification for the claim on line 178: “In order for f to be a homeomorphism, the winding number must be -1 or 1?” \n- What exactly are the columns in table 1? The caption is quite sparse and doesn’t adequately describe how #H or Continuity are measured. The discussion of continuity seems to be relegated entirely to the appendix. At least a brief description with some intuition should be in the main text.\n- What’s the rough computational cost of training with the additional normalizing flow regularization? \n- How come the baseline “reg-y” model performs so poorly? I’m surprised that explicitly encouraging the latent space to be close to S1 doesn’t improve continuity on two of the three shapes.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper theoretically and empirically characterize obstructions to training Homemorphic encoders with geometric latent spaces, such as local optima due to singularities or incorrect degree or winding numbers.\n\nSTRENGTHS:\nOriginality: The paper is original in its approach to addressing topological obstructions in machine learning models. While previous work has explored the use of geometric inductive biases to improve interpretability and generalization, this paper is one of the first to systematically investigate the topological challenges of encoding to a specific geometric structure from the training perspective and propose a novel solution using adapted normalizing flows. \n\nQuality: The paper is of fair quality, in terms of its theoretical rigor and its empirical evaluation. The authors provide a detailed analysis of the topological obstructions that can arise in encoders with geometric latent spaces, and their proposed GF-VAE model is well-motivated  to address these challenges. \n\nClarity: The paper isfairly written and organized. The authors provide ample background and motivation for their work, and their theoretical and empirical analyses are both presented in an accessible manner. The use of figures and examples throughout the paper helps to illustrate the authors' ideas and make the paper more understandable.\n\nSignificance: The authors' characterization of topological obstructions and their proposed GF-VAE model have the potential to improve the interpretability, generalization, and robustness of topological machine learning models. The paper is likely to inspire further research in this area and has the potential to lead to significant improvements in the performance and reliability of machine learning models.\n\nWEAKNESSES:\nWhile the paper is fairly written with novel contributions to the training of geometric deep learning, there are a few areas where it could be improved:\n\n3. While the paper provides a clear and accessible presentation of the authors' ideas, some of themathematical concepts could be more clearly explained. For example, the motivation of homemorphic encoder and its applications in generative models should be discussed.     The concrete geometries considers in this paper are restrited to $S^1$, which is a commutative group. As th title is for general topological obstructios, the paper could benefit from providing a more general Lie group case.\n\n2. The design of  GF-VAE is to apply flow model as a plug-in prior for the VAE, however combination of VAE and flow model is not new.  Therefore, there are likely other approaches and techniques that could be relevant to GF-VAE that are not discussed in detail.\n Second, The empirical evaluation of the proposed GF-VAE model is also limited to just two domains, and it is unclear how well the model would perform on other types of data. It would be helpful for the authors to provide additional experiments on a wider range of datasets to demonstrate the generalizability of their approach.\n\n3. The paper could benefit from a more detailed discussion of the limitations of the proposed approach. \n\nTypos: stand-> standard in line 227\n\nQUESTIONS:\n1. Why the author selected the figure eight and the degree cases to represent topological obstructions. Are there any principles for classifying topological obstructions?\n\n2. What is the precise definition of the projection from $Y$ to $Z$ in formula (1) ?\n\n3. Figure 2 doesn't contain the deconvolutional decoder $f^*$?\n\n4. How is the flow $r$ realized (in line 227) for general lie groups?\n\n5. The last paragraph of page 5 seemed to have provide a simpler solution to avoide topological obstructions than the  GF-VAE?\n\nLIMITATIONS:\nThe paper could benefit from a more detailed discussion of the limitations of the proposed approach.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the challenges of encoding data into geometric spaces and proposes a solution using Group-Flow Variational Autoencoders (GF-VAEs). The authors discuss how incorporating geometric inductive biases can improve interpretability and generalization but also present obstacles due to topological constraints. They identify two types of local optima that can arise: singularities and incorrect degree or winding number. To overcome these challenges, the authors propose GF-VAEs, which utilize normalizing flows to define multimodal distributions on geometric spaces. The paper characterizes topological defects in encoders, introduces evaluation criteria based on winding number, crossing number, and continuity, and demonstrates that GF-VAEs can escape local optima and achieve a more reliable convergence to a homeomorphic mapping. The main contributions of the paper include characterizing topological defects, proposing GF-VAEs as a solution, and empirically validating their effectiveness.\n\nSTRENGTHS:\n1. *Theoretical and Empirical Analysis*: The paper combines theoretical analysis with empirical evaluations to provide a comprehensive understanding of the challenges and solutions related to encoding data into geometric spaces. This approach strengthens the validity of the proposed methods and their practical implications.\n\n2. *Identification of Obstructions*: The paper effectively identifies and characterizes topological defects that can occur in encoders mapping to geometric structures. By recognizing the specific challenges such as singularities and incorrect degree or winding number, the authors provide a clear understanding of the obstacles that need to be addressed.\n\n3. *Proposal of GF-VAEs*: The introduction of Group-Flow Variational Autoencoders (GF-VAEs) as a solution to the identified obstructions is a significant contribution. The paper explains how GF-VAEs leverage normalizing flows to model complex multimodal distributions on Riemannian manifolds. This proposal offers a practical approach to circumvent local optima and achieve more reliable convergence.\n\nWEAKNESSES:\n1. *Idealized Assumptions*: The paper acknowledges that the theoretical analysis is limited by the idealized assumptions necessary to analyze the method using topological tools. These assumptions may not exactly match the real-world scenarios encountered in practice. This limitation undermines the direct applicability of the theoretical findings to real-world problems and raises questions about the generalizability of the proposed solutions.\n\n2. *Limited Metrics for Higher Dimensions*: The metrics defined in the paper, such as winding number and crossing number, are primarily designed for lower-dimensional manifolds. It is mentioned that these metrics become harder to define and compute for higher-dimensional manifolds. This limitation restricts the applicability of the evaluation criteria to higher-dimensional geometric spaces, potentially limiting the scope of the proposed approach.\n\n3. *Restricted Scope of Experiments*: While the paper presents empirical evaluations on two domains, it does not cover a wide range of datasets or scenarios. The limited scope of the experiments may not fully capture the diversity of real-world applications and datasets, leaving open questions about the performance and generalizability of the proposed GF-VAEs in different contexts.\n\n4. *Lack of Comparative Analysis*: The paper lacks a comprehensive comparative analysis of the proposed GF-VAEs with existing methods. While the empirical evaluations demonstrate the effectiveness of GF-VAEs in escaping local optima and achieving better convergence, a thorough comparison with alternative approaches would provide a clearer understanding of the strengths and weaknesses of the proposed method in relation to existing state-of-the-art techniques.\n\nQUESTIONS:\nSee the Section 'Weaknesses'.\n\nLIMITATIONS:\nThe authors presented in the paper the most important limitations of the proposed approach.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper addressed several topological obstructions that cannot be easily solved during the optimization of VAE. To solve this problem, the paper proposed to train a special NF to escape the defect, which the authors called GroupFlow. The paper then evaluated the proposed method on synthetic image datasets with different kinds of manifolds.

STRENGTHS:
The paper is clearly written and the mathematical formation of the problem is precise. 

The analysis on the obstructions during optimization clearly shows the invariants which lead to defects. 

The proposed method naturally follows the defects and seems interesting to me.

The evaluation results indicate the proposed method achieves the best homeomorphism.

WEAKNESSES:
I am not an expert in topology, so please refer to other reviewers for the comments on theoretical analysis. However, it seems the authors only analyzed a very simple type of manifold with small freedom. It would be more interesting if there are results on more complex manifolds beyond rotation and colorization. There is a similar concern for experiments.

From Table 1 it seems the $\beta$ parameter has large impact on the homeomorphism result. There should be more (intuitive or theoretical) discussion on the relationship between $\beta$ (disentanglement) and homeomorphism. It is also important to demonstrate results from the standard $\beta$-VAE.

QUESTIONS:
Does your theory apply to more complicated manifolds? 

Why does higher $\beta$ leads to better homeomorphism? How about results of the standard $\beta$-VAE?

LIMITATIONS:
The paper has adequate discussion on limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06