Topological Obstructions and How to Avoid Them
Authors: Babak Esmaeili
Robin Walters
Heiko Zimmermann
Jan-Willem van de Meent
Venue: NeurIPS 2023 poster (2023)
Keywords: representation learning
variational autoencoders
homeomorphism
topological
equivariant
lie groups
normalizing flows
TL;DR: Imposing geometric inductive biases in representation learning can lead to topological obstruction during training, but these can be circumvented using multimodal distribution such as normalizing flows as variational distributions.

Abstract: Incorporating geometric inductive biases into models can aid interpretability and generalization, but encoding to a specific geometric structure can be challenging due to the imposed topological constraints. In this paper, we theoretically and empirically characterize obstructions to training encoders with geometric latent spaces. We show that local optima can arise due to singularities (e.g. self-intersection) or due to an incorrect degree or winding number. We then discuss how normalizing flows can potentially circumvent these obstructions by defining multimodal variational distributions. Inspired by this observation, we propose a new flow-based model that maps data points to multimodal distributions over geometric spaces and empirically evaluate our model on 2 domains. We observe improved stability during training and a higher chance of converging to a homeomorphic encoder.

Topological Obstructions and How to Avoid Them
Babak Esmaeili" Generative AI Group
Robin Walters"
Khoury College of Computer Sciences
Eindhoven University of Technology
Northeastern University
I. www.weeerscoooohheteerrr.e edu
Heiko Zimmermann
Jan-Willem van de Meent Amsterdam Machine Learning Lab University of Amsterdam
Amsterdam Machine Learning Lab University of Amsterdam h. zimmerrannn@ua.... ml
j w. vandemeent@uva. -nl
Abstract
Incorporating geometric inductive biases into models can aid interpretability and generalization, but encoding to a specific geometric structure can be challenging due to the imposed topological constraints. In this paper, theoretically and em- pirically characterize obstructions to training encoders with geometric latent spaces. We show that local optima arise due to singularities (e.g. seff-intersection) due to an incorrect degree or winding number. We then discuss how normalizing flows can potentially circumvent these obstructions by defining multimodal vari- ational distributions. Inspired by this observation, we propose a new flow-based model that maps data points multimodal distributions over geometric spaces and empirically evaluate our model on domains. We observe improved stability dur- ing training and a higher chance of converging to a homeomorphic encoder.
Introduction
well-established idea in machine learning research is that geometric inductive biases can help us learn representations that reflect the underlying structure of a dataset [Bronstein et al., 2021, Higgins al., 2022]. key intuition behind this line of research that such representations make it easier to reason about the similarities of different instances in the dataset, for example by relating inputs using a rotation or translation, which in turn aids interpretability and data-efficiency. Geometric inductive biases have been explored in a wide variety of forms, including models that are defined hyperbolic or spherical Riemannian manifolds [Lezcano-Casado and Martinez-Rubio 2019, Ganea et al., 2018), models that are equivariant or invariant with respect to particular symmetry groups [Kondor and Trivedi, 2018, Cohen and Welling, 2016], and work that leverages symmetries to learn disentangled representations that factorize into distinct axes of variation [Higgins et al., 2018].
In this paper, consider symmetry-base based approaches learning representations in an unsupervised manner by imposing geometric inductive biases the representation space. In this context, our notion of representation that "reflects the underlying structure of the data" is a representation that is homeomorphic to the true generative factors. We are specifically interested in the optimization challenges that arise when encoding to geometric spaces such as Lie groups. While a common intuition is that an inductive bias that matches the underlying topology of the data will guide a model towards a homeomorphic representation, there are also indications that certain inductive biases can make a model more difficult to train in practice [Park et al., 2022, Falorsi et al., 2018, Batson et al., 2021], I], which limits their practical utility.
Equual contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

L
y
2
E
Ty. (666l
hig
fg
Figure 1: A GF-VAE consists of an encoder network h& that maps data & to an intermediate parameter space V. The encoded vector is split into K parts, where each sub-vector y/ corresponds to the parameters of a normalizing flow at layer k. We can sample from the variational distribution ds by first sampling from uniform prior defined sequence of K bijective transformations r(: yk). The output of the normalizing flows, denoted as 2k, is then passed to decoder fø that maps from the Lie group 2 the data space X.
the Lie group (SO(2) in this case), followed by
To understand why encoding to geometric structures can give rise to optimization challenges, formalize topological defects that can occur a randomly initialized encoder, such as discrepancies in the winding number crossing number relative those in homeomorphic encoder. We show that these topological defects will be preserved under continuous optimization, which suggests that escaping these local optima relies on the discrete jumps that are employed during optimization.
To circumvent these obstacles to optimization, we propose Group-Flow Variational Autoencoders (GF- VAEs), which leverage normalizing flows to model complex multimodal distributions on Riemannian manifolds (Figure 1). We show that we define the mode of the variational distribution to be the representation, normalizing flows can circumvent some of the challenges associated with local optima due to their multimodal nature. Experiments demonstrate that GF-VAEs can escape local optima during the early stages of training, resulting in more reliable convergence to homeomorphic mapping and a greater degree of continuity after training.
We summarize the main contributions of this paper as follows:
We characterize topological defects that can arise in encoders that map onto spaces with geometric structure. We show that some obstructions that arise from topological defects cannot be resolved using continuous optimization.
We define evaluation criteria based the winding number, crossing number, and continuity to measure topological defects and homeomorphism violations in the encoder.
We propose GF-VAEs a new VAE-based model that that employs normalizing flows to define complex distribution on Lie groups. We empirically show that GF-VAEs are able to aid in circumventing identified optimization obstructions.
Problem Statement
Homeomorphic Encoders. Our goal is to learn representations domains where we have prior knowledge of the geometric structure, specifically structure in the form of symmetry group that can be associated with the input data. We assume that data lies on low-dimensional manifold M that embedded in nigher-dimensional space X' == I via mapping 9x . -> X. This is commonly known as the manifold hypothesis [Bengio et al., 2013]. We denote the image of the mapping by M2 (MM X. Then gx is homeomorphism, topological isomorphism, onto its image g: . - Ma. That is, g continuous, bijective, and has continuous inverse.
We wish to learn an encoder fs: X > Z such that its restriction følM_ M, - 2 is homeomor- phism. Following Falorsi et al. [2018], will define this mapping in terms of a network hs: X y

---

that maps to an intermediate space : RR, followed by known projection _ : 2,
:= fs: X he y 2.
(1)
Lie Groups. Our work focuses the specific case where the manifolds M and Z are Lie groups. Any set of symmetries may be formally described by a group G. which is a set of invertible transformations which may be composed using a binary operation : G x G + G. A Lie group is group that also a differentiable manifold. Lie groups describe continuous symmetries such rotations and translations, and are therefore a natural mathematical setting for any system with spatio-temporal symmetries.
Reasoning about Lie groups is difficult due to their non-flat structure. Lie algebras provide way to study Lie groups by considering the tangent space g of the manifold at the identity element. The exponential map exp g G' maps points the Lie algebra to points the group manifold as such, vector u E defines a vector field on G using the group action to transport v around G. Then exp(e) is defined to be the point reached by flowing along this vector field for unit time. For more details on Lie groups, we refer the readers to [Hall and Hall, 2013].
Variational Autoencoders on Lie Groups. In this paper, we primarily focus on variational autoen- coder ( (VAE) [Kingma and Welling, 2014, Rezende et al., 2014] as a means to learn a homeomorphic embedding. In this setting, we define generative model by composing uniform prior p(z) = (()) on 2 with a likelihood model (=(x=>) = N (a: fğ (2), o In) that is defined in terms of a decoder net- work fō Z + X. We use the encoder f to define a variational distribution go (2 (z),, whose design we discuss below and train the encoder and decoder jointly by optimizing the variational lower bound,
VAF (x) = (x]=) Plog [x) - DEL [=o(=|>) [[p())] < log (xr)..
(2)
Defining a variational distribution on manifold is generally not straightforward as it requires finding an expression of the density on the manifold or keeping track of the change of volume when projecting to the manifold from the tangent space. Falorsi al. [2019] define a reparameterized construction for sampling from Gaussian distribution on Z by first sampling Lie algebra associated with Z, then rescaling by way of clement- multiplication (z) O e using network d X IR,, and computing the corresponding element 2e the group manifold using the exponential map. By left multiplying this randomly sampled 2e by the group element f fo(x) E M to move the mode of the final distribution to its intended location, we obtain the construction
_O0 I,) from the p-dimensional
6 <~ <~:(; Ip), = (x=p(a((())))) 2 == fo(r) Ze: p(zc) = p(c) det dexp(e)
(3)
de
We account for the possible change in volume using the change of variable formula. Note that composing group elements fo (z) 2 does not change the density of the resulting point on the manifold because the Haar measure, standard choice for Lie groups, is left invariant.
Running Example. As a concrete running example, which we will use throughout this paper, we will consider data the circle .M- SO(2) that is embedded into a space of images X. The subset M2 of images generated by a function y: SO(2) + .M corresponds to images of an object that is subject to one-dimensional rotation. We will consider the case of in-plane rotations of images, as well images of a 3-dimensional object that is rotated around a single axis.
The special orthogonal Lie group SO(2) is defined as
SO(2) = {= E (2) de)) = 1} A(v)
y1 y2
U2
y E R2, ||vll 1Ivll= -1)
where GL.(2) is the general linear group, which is the group of invertible 2 x 2 matrices under matrix multiplication. The set of images My lives on a manifold that homcomorphic to the rotation group SO(2), embedded into the space of images by g. Because assume know the underlying manifold M, we can design to have the same structure. When M = SO(2), we can use y := R². The function he in Equation is then an ordinary neural network, and the projection to 2 is simply the projection onto the unit circle R² + SO(2) + A (y//lv/))
3

---

M
Mr
V.2
Figure 2: Example of topological defects in learned encoders for a VAE with data on X' = R 32x32 in the form of rotations L-shaped tetrominoes and Z SO(2) (the unit circle). Our goal is to learn an encoder fe that defines a homeomorphism (a continuous bijection with continous inverse) between the manifold of images of L-shaped tetrominoes M, X' and that of the latent space 2 SO((). The encoder fo = π h& is defined by composing a network h X - y with a projection л y - Z. On the right, we show the intermediate space y/ hg(Ma) and its projection Z,- = x =V²)) for random seeds after convergence. Colours indicate the angle associated with each data point on the manifold M.. Optimization obstructions can arise when the network he maps data onto trajectory Y/+ that exhibits topological defects, such as the crossing in "figure 8" shape. which gives rise to discontinuities in the projection Z onto the latent space.
3 Optimization Obstructions
In practice, training homeomorphic encoders can give rise to optimization challenges. To develop intuition for these challenges. we will consider the running example M = SO(2) with data in the form of in-plane rotated images on = 323x32 (Figure 2). We train a VAE in which the network hes is standard convolutional network which is paired with deconvolutional decoder :SO(2) - RR32x32 (see Appendix A). Figure? 2 shows encodings the intermediate space y after training with ELBO with 3 random seeds. For the first two seeds, we see that (MM)) crosses over itself, resulting in an figure "8" shape. When projected onto == SO(2), this results in discontinuities at the crossover points. The second seed in addition also exhibits a sparse region in the intermediate space, leading to a gap in the SO(2) projection. Only the third initialization has converged to the correct topology. These local optima are not unique to this example; obstructions have also been encountered in Falorsi et al. [2018] when trying to learn 3D orientations
rotating multi-color cube from 2D images using homeomorphic VAE. Park et al. [2022] show that the homeomorphic VAE cannot generalize well to other shapes and tends to learn a degenerate emhedding to a small part of SO(3).
The main observation that we make in this paper is that imposing geometric structure on the latent space can introduce topological obstructions during optimization. This insight is distinct from the homological obstructions identified by de Haan and Falorsi [2018], who describe the obstructions that emerge from the choice parameterization on the Lie group. We will refer to the topological defects that we identify in this paper "optimization obstructions". The problem that we identify here is that randomly- mandolly-iittiliized layers have a high probability of exhibiting topological defects (degree, crossing, etc.) that cannot be resolved under continuous optimization using gradient flow. Removal of these topological defects is thus only possible by relying on the jumps coming from performing SGD with large enough learning rate. This implies that while escaping such local minima is possible, it is difficult and may require many epochs to do so, dramatically slowing training and undercutting the advantages of learning a homeomorphic representation.
We now discuss several specific optimization obstructions. We focus on the case where M the Lie group S2(2), with the same example setting described in Section 2. All the optimization obstructions we consider occur in this case and in the case of higher-dimensional manifolds M as well, but are simpler to describe for S See Appendix B for proofs.
3.1 Figure Eight Local Minima
To more precisely describe obstructions that might arise during optimization, we consider continuous- time training along gradient flow. We denote the weights of the initialized encoder as 6(0) and the trained weights as ((). We consider the idealized setting which ((t) continuous function of t.
Empirically, either at initialization or after some training, we often observe "figure 8" pattern in y of roughly the form (1+00 o (0) = (cos 0 sin 0, sin TT In such a case, the embedding into 2 contains two disjoint pieces [-3r// nT m/4 IT /4, 3m 3/////4 /4. One half of the circle is mapped to one piece at

---

the top of the circle and the other half of the circle is mapped to the other disjoint piece at the bottom of circle. Such a mapping takes advantage of the singularity at (0, 0)2 into order embed S' in two continuous pieces. The resulting mapping is nearly bijective, failing to reconstruct on only small region near the two discontinuities. It also mostly continuous, having only two discontinuities.
Once this local minimum is obtained, it is very difficult to move out of it using gradient descent. It is unlikely for the two pieces join together and become homeomorpic embedding since this would require passing one disjoint segment through the other and reversing its orientation which would violate bijectivity and increase the reconstruction loss.
We make this observation precise by noting that continu- ous optimization preserves the ordering of points the circle. Let (21, 22, 23, 24) denote the four end points of the two disjoint intervals of fo10)- The ordering of these points is only defined up to cyclic permutations mod C4. Proposition 3.1 states that continuous optimization must preserve ordering mod C4. Figure 3 illustrates the proof.
t=
Proposition 3.1 Assume that facs undergoes continuous optimization. Assume that fe(t) o g is injective for all t. The cyclic ordering induced on lo points by fo(o is equal to Je(1)- Thus figure 8 embedding, which corresponds to cyclic order (21, 22, 24, 23) mod C2, cannot be trans- formed to a homeomorphic embedding, which has cyclic order (21, 22, 23,, 24) mod C4 or (z4, 23, 22, 21) mod C4-
21
23
Figure 3: The figure 8 pattern in 3 (grey) maps to two disconnected components in Z. The cyclic order of these endpoints is preserved by homotopy. Following the parameterization of the data mani- fold the cyclic order is (æ1, 22, 24, 23), which is distinct from the cyclic order of a homeomorphic embedding, either (21, 22, Æ3, 24) (24, 23, 22, 21).
In other words, transition from figure 8 embedding homeomorphic embedding is impossible without violating continuity during optimization. This indicates that escap- ing a figure 8 local optimum during training would need to rely on discrete jumps and likely the stochasticity of the
gradient estimate.
3.2 Degree Obstructions
A second class of topological defects that can arise encodings with discrepancy in the winding number. We can compute the degree, or winding number, of a map tb S² + S* around the origin by summing up the differentials along its path on the sphere,
w 2b)
(1).
2T
This concept can be extended arbitrary connected oriented manifolds, where it is usually referred to as the degree of mapping. Intuitively, describes the number of times that the domain manifold wraps around the co-domain manifold.
If the embedded image (M does not contain the origin, then the mapping factors through RR {(0.00) 0)} and is consequently continuous. Therefore, a continuous path in é yields a continuous path in Z. If folM, is continuous, then the function 3b := o hg g S¹ + S has well-defined degree, also known as winding number wr tha) E Z. In order for folM, to be a homeomorphism, the winding number must be w(b)) E (-1, 1). Under random initialization, however, the initial network may have winding number equal to any integer. Assuming continuous optimization and continuous embeddings, then hoce) (z) is continuous function of both xr, We assume that hote) (a:) (0, 0) for any f, r and thus winding number is defined for any time. The following proposition thus holds.
Proposition 3.2 Winding numbers of the initialized and final model are equal w(va())) = w (()))))
In practice, neither the continuous optimization assumption nor the avoidance of the origin holds. Rather ha is updated by SGD in discrete jumps and h c g may map to the origin. Thus, empirically, we do see that the winding number may change during training. However, if the initialization avoids
In practice we never exactly cross (0, 0) but only pass by it with small distance.
5

---

t=0 t=1 t=2 t=3 t=4
GF-VAE
Figure 4: The initialized encoder (t-0) may contain defects as described in Section 3, such as a figure 8 pattern. With standard VAE, where the learned representation (red point) is the mean of conditional gaussian, these defects are unlikely to be resolved during optimization, as it would require passing through intermediate parameters (and hence representations) with higher reconstruction loss. Using a multimodal variational distibution, the parameters and corresponding representations are less tightly coupled, in the sense that continuous change in the parameter space result in discontinues change in representation (i.e. the mode of the distribution) without passing through high-loss areas of the parameters space.
the origin, then due to the tendency of the magnitude of the unnormalized embeddings (a(x:) to grow during optimization (see Section 3.3), the winding number changing becomes more unlikely. This means that defects in the winding number pose significant obstruction learning homeomorphic embeddings. The winding number is also the primary optimization obstruction which makes it impractical remove the hard projection и. If instead we decode directly from y y but push embeddings to the unit circle using the loss |||y|| 1/. then it is far more likely we converge to discontinuous embeddings with the incorrect winding number (Appendix D).
3.3 Magnitude Growth in
Empirically, we observe the values the embeddings in y == R22 continually grow during training. This phenomenon makes more difficult for the embedded data manifold (2M,) to cross the origin and for the winding number to change. We give a theoretical explanation for this behavior.
Consider what would happen if the embedding were updated directly based the gradient of the loss V, with respect to y. We assume the loss depends only on z = y/ /|yl]. and so has level sets which are unions of radial rays from the origin. The gradient V, v must then be tangent to circle about the origin. That is, for y = (a, b), the gradient v,C = (#b, Fa). Under gradient flow, the evolution of y in time Ye would thus flow along circles of fixed radius and ||vo|| l|yell- Under gradient descent, however, due to the convexity of the flow lines, which are circular, the embeddings y will tend grow in magnitude. For Rzu, compute
ly // ,, = (a F n/)² + + rya)² = (a² + 63) + 722) > ||
In practice. however, we do not update y based on v.C but rather based on the gradient with respect to model parameters p. Let F: & - y be the map from model parameters & to y given fixed input data Then the actual update y is v,C dFT V&L where dF is the total derivative Jacobian of the map F. Since VoL (dF) we have Vy dF yc. The angle between V.C. and V v is bounded by some a quantity depending on the eigenvalues of the operator dF. Given that V, C is tangential to the circle, assuming for simplicity V C has constant length L and uniform distribution [-0,0 0) in angle to V,C, the norm of y still grows in expectation.
Proposition 3.3 Assume a circle of radius R. Let be random vector y on the circle of length L with angle to the tangent uniform in -0,0 0). Then
Ellly + w = 1,2
R²
R² 12
4 GroupFlow-VAE
The topological obstructions to optimization that we identified in Section 3 cannot be resolved under continuous optimization. Moreover, even if we allow for degree of discontinuity, resolving obstructions like figure 8 will require violating bijectivity, which implics that the reconstruction loss

---

must increase to escape this defect. This suggests that VAE with reparameterized construction as described in Equation will be susceptible to local optima, which aligns with the empirical observation that homeomorphic VAFs can be difficult to train.
This raises the question of whether we can make VA less susceptible to topological obstructions by employing different parameterization from that of Equation 3. More concretely, we will consider a parameterization that admits multiple modes in the variational distribution qo (z|z)) rather than the unimodal construction in Equation 3, and define fo(r) in terms of the mode,
(1) == arg max qo (z/xr).
The intuition behind this approach is illustrated in Figure 4. If the variational distribution contains multiple modes, rather than single peak centered at π (z)) then it may becomes possible for small changes to result in non-local changes, such as the reordering of points that is needed to untwist figure 8, by switching between modes in the variational distribution. Moreover, increasing the number of parameters of (0(2)z) (r) is likely beneficial for escaping some of the topological obstructions as some defects such
are less likely to occur in high-dimensional spaces.
Motivated by this intuition, we consider a parameterization of (z(())) that employs normalizing flows [Rezende and Mohamed, 2015]. In a normalizing flow, the sample is defined as push forward of sequence of smooth bijective transformations, which makes possible to reshape a simple unimodal distribution into more complex multimodal distribution. The probability of sample from the final density can be computed by repeatedly applying the rule for change of variables. Concretely, given a base distribution p(2o) and a sequences of bijective transformations T Z Z, we obtain a sample 2 = KK and probability by first sampling 2g ~ p(20) and defining a sequence of transformations
i (=(2)-1)
2 = (2)-11)
logp(zx) = log 11) log det
for k = 1...K. K. (5)
a 2k-1
Normalizing flows have been used to define distributions on geometric structures such as Lie groups by either defining a flow on the Lie algebra and computing the push-forward density of the exponential map [Falorsi et al., 2019], or designing dtrucure-specifer transformations [Rezende et al., 2020].
In the GF-VAE, we will define construction in which the encoder network returns the parameters of the flow. We show an overview of the architecture in Figure 1. We define a network he : X' + y :=
K where K and are the number of flow layers and parameters respectively, a sequence of bijective transformations {r(: ) parameterized by {yk ) and base distribution distribution p(20) which we define distribution the group. The sequence {r(: 3k)) ) is then used to define new distribution on Z given an æ by transforming the base distribution. This defines a conditional flow that can be trained using a stand lower bound (Eq. 2).
log (=)(<)) = log (1)) - log det i Dr(z-119M
~ p(2n),
2k = ((=--1:8k)
θ 2g
The choice of the flow r is very important here as normalizing flows are typically defined on flat spaces. This means that, for a specific manifold M, additional care must be taken when designing them to ensure that they are a diffeomorphism from Z to itself [Rezende et al., 2020, Durkan et al., 2019, Mathieu and Nickel, 2020]. In this paper, we employ a similar method as Falorsi et al. where we define an affine layer followed by a single layer of spline flow, and Tanh layer multiplied by as the last layer to push all the probability density between - and æ.
5 Experiments
We perform series of experiments to evaluate the difficulty of learning a homeomorphic encoder. Concretely, we investigate how often we fall into one of the failure cases described in Section 3 in standard VAEs with geometric latent spaces in practice. Subsequently, we examine how well GF- VAE can circumvent these topological obstructions during training.
Baselines. Throughout our experiments, we compare against (1) standard VAE, and (2) a super- vised VAE where in addition to maximizing the ELBO, the encoder is trained to predict the ground truth representations. These two scenarios serve as extremes on the spectrum of guiding the model to the right representation. We also compare against deterministic autoencoder (AE) which does not
7

---

VAE
GF-VAE
4
VAE
GF-VAE
Figure 5: Top: Latent traversals the decoder for a VAE and GF-VAE after training. Bottom: The representations in latent space Z for VAEs and GF-VAEs initialized with 15 different random seeds. For VAEs, the gray line inside the circle show the (scaled-down) y-traversals. For GF-VAEs, we is diffieult visualize Y' space it high-dimensional space. Therefore, inspect for obstructions, we show the traversal of the 2 vectors instead.
regularize the embedding space. We also tried regularizing the y-space to be close Si (by penaliz- ing (llvll2 1)²) in order to mitigate the optimization problem discussed in Subsection 3.3, which we refer to as "reg-y" loss. Finally, we evaluate the 3-VAE objective [Higgins et al., 2017] which increases the regularization
the latent space by upweighting the KL term in Eq 2. All models em- ploy a 4-layer CNN architecture for the encoder and decoder with LeakyReLU activations. For the decoder, we also experiment with action-decoder proposed by Falorsi et al., which found helpful for learning a homeomorphic mapping. The action-decoder uses a special first layer where the group action is applied to set of learned Fourier coefficients rather than directly being passed input to the architecture. For further details regarding our experiments, please refer to Appendix A.
Evaluation. We evaluate all models based two criteria: (1) Has the encoder learned homeo- morphic mapping? and (2) Has the decoder learned a good model the data? Assessing whether a follow the evalu-
learned mapping is homeomorphic is challenging. To verify homeomorphism, ation proposed in Falorsi et al. [2018] by examining whether the encoder yields a continuous path when interpolating in the data manifold from x to Details evaluating continuity provided in Appendix C. To determine how often the models encounter the topological obstructions described in Section 3, we also report crossing and winding numbers in Appendix E. A crossing number grater than O implies a "figure 8" obstruction, and a winding number that is not equal to or -1 implies winding number obstruction. Lastly, we measure the likelioid to assess how well each model approximates the data manifold. If the model has diverged during training due to posterior collapse, we report it as a non -omeoooooooicccc mapping and ignore its continuity score in the average.
5.1 Images: SO(2)
In our first experiment, train images of L-shaped tetromino [Bozkurt et al., 2021], teapot, and an airplane [Shilane et al., 2004]. The SO(2) manifold corresponding to each object is made by rotating the image of the object around the center. We report our findings in Tables and 3.
Table 1: Number of learned homeomorphic encoders and their continuity score for different objectives trained with 15 different random seeds.
Isshpedd Tetromino Continuity
Teapot Continuity
Airplane Continuity
# H.
# H.
# H.
AE
2/15
137.28 + 54.15
0/15
173.37 + 21.99
0/15
132.95 +: 40.61
VAE (3 1)
0/15
117.02 + 23.17
6/15
14.21 + 7.30
0/15
86.72 +: 65.12
VAE (B3 4)
2/15
132.45
58.79
15/15
1.22 + 0.05
5/15
67.88 + 59.20
VAE y-reg
1/15
152.71
+ 74.20
1/15
14.51 81.
3/15
124.10
VAE (5 1)
5/15
50.51 58.18
7/15
21.07 + 24.42
0/15
63.94 + 34.43
VF- (B 4)
9/15
24.04
29.14
13/15
8.16 1. 17.89
7/15
27.70 + 29.23
Action-Gr GF- VAE (3 = 4)
10/15
19.35 + 23.37
13/15
3.14 1.74
9/15
33.26 + 40.03
Sup-VAB VAE
15/15
5.74+10.9
13/15
5.67= 0.34
0/15
96.78 + 38.

---

1
Griomitiin
L
Latent traversals in the decoder of a GF-VAE trained on Tetrominoes with a torus latent spaces. In each traversal, we keep the value for one Lie group fixed and do geodesic interpolation across the other group. The traversals in the latent space highlighted are by red and orange.
Figure 6:
We observe that even though the types of obstructions vary across images, both VAE and AE in general fail to learn homeomorphic encoder. The GF-VAE objective improves performance noticeably across both metrics. We also did not find regularization to be very helpful as it mainly stabilized training towards whatever mapping that was achieved at the early stages of training. We observed the main failure case for most of the non-homeomorphic encoders was due to discontinuity emerging from figure-eight obstruction. Moreover, looking at the learning curves in Figure 9, we observe that the winding number is susceptible to change during training which suggests that the continuity assumption of importance in proposition 3.2. GF- VAE resolves both issues, which allows us to interpolate nicely in the Z-space (Figure 5). What is very surprising is that in the case of airplanes, we see that even supervised objective fails to overcome these optimization obstructions, while a GF- VAE is able to achieve this at much better rate.
We also observe that increasing the 3 value for the KL term generally helps. This is perhaps unsurprising given that a high / value encourages the latent space to cover the prior and therefore discourages the winding numbers from being and improves the performance on the continuity metrie. In the case of the teapots, we in fact observe that increasing 3 is sufficient to learn homeomorphic encoder. However, VAE still generally scores better in terms of continuity and winding numbers.
5.2 Images: Torus
In this experiment, we consider ring torus the latent space, which is homeomorphic the Lie group SO(2) x SO(2). We create a dataset homeomorphic to this group by independently rotating the L-shaped tetromino in both orientation and color. All models can be extended Tori by simply duplicating the latent space and learning multiple encodings to SO(2) in parallel. This is done by defining a factorized density (=))21), =(00(x))) (z) = (=)=(() |xr )a (2(2) where each distribution (s(2(0(zz) (z) defines a distribution on S1, Ideally, we want one subgroup to correspond to colour and the other orientation. In this setting, we evaluate homeomorphism by picking 10 different values in either colour or orientation and measuring the continuity of the encoded path when interpolating in the data manifold on the other attribute. We identify the encoder homeomorphic the average continuity score of all 10 paths is below certain threshold (see Appendix C). Unsurprisingly we found that learning a homeomorphic mapping in tori is more challenging compared to circles. Out of 1.5 runs, the VAE models learn a homeomorphic mapping time, while GF-VAE manages to learn homeomorphic mapping times. To be able to align each Lie group with the corresponding attribute, we also used the weakly supervised proposed by Locatello et al., where the model receives a sequence of images which only the angle or the colour is changing. We show the latent traversal in the decoder model for one of the successful runs of GF-VAE in Figure 6. As can see, the model has successfully managed to disentangle colour and orientation in the latent space.
6 Related Work
Learning Geometric Representations. There has been a large amount of work concerned with learning representations from data with geometric structure in the unsupervised or weakly supervised setting. One line of work has focused on the use of geometric spaces such as lie groups in latent space [Davidson et al., 2018 8, Falorsi et al., 2018, Perez Rey et al., 2020, Vadgama et al., 2022]. In [Miao et al. was argued that that geometric inductive biases in VAFs should be Incorporated via deterministic mapping rather than the prior which is consistent with our work. As we show in this

---

work, naively incorporating geometric bias leads to topological obstructions, Another line of work has focused on inferring the latent structure by exploring the local connectivity information [Moor et al., 2020, Chen et al., 2021, Lee et al., 2021, Pfau et al., 2020].
Learning Disentangled Representations. Topological group structure be used to define notion of disentanglement that is based on equivariant properties under group transformations [Higgins et al., 2018]. There exists body of work that aims to learn both disentangled and equivariant group representations. One set of methods relies on agent actions to predict the group element [Caselles- Dupré et al., 2019, Quessard et al., 2020]. An adjacent line of work is to regularize encoders to be equivariant with respect to the group action, using triplets of the form (Ia, Me, zt+1) or longer sequences [Guo et al., 2019, Dupont et al., 2020, Tonnaer al., 2022]. Some approaches focus on learning symmetry-basee disentangled representations in fully unsupervised settings or by enforcing commutativity in the latent Lie group [Yang et al., 2022, Zhu et al., 2021]. Another related area disentangles class and pose in the latent space [Marchetti et al., 2022, Winter et al., 2022]. Our work does not focus on disentanglement, but the topological obstructions that we describe are relevant to this domain.
Topological Obstructions in Learning. The topological obstructions that we consider in this work are distinct from the homological obstructions that have been characterized in prior work [de Haan and Falorsi, 2018, Batson et al., 2021, Falorsi et al., 2018, Perez Rey et al., 2020]. Theorem by de Haan and Falorsi [2018] defines homological obstructions as follows: For any latent space Z with non-trivial topology, it is possible to learn encoder fo that is continuous when restricted to M2 C X. but this encoder must be discontinuous on the full space X. For this reason, Falorsi et al. [2018] and others [Xu and Durrett, 2018, Meng et al., 2019] use the two-part encoder from equation 1, inserting discontinuous layers т when mapping to circles, spheres, SO(n). or other manifolds. This explicit discontinuity circumvents the homological obstruction without forcing the linear layers of the network to approximate discontinuities using large weights, which we and others find leads to instability during training and inferior reconstructions (Section 5).
Normalizing Flows on Manifolds. In recent years, there has been surge of interest in extending normalizing flows, originally formulated for Euclidean spaces, to Riemannian manifolds [Rezende et al., 2020, Mathieu and Nickel, 2020, Köhler et al., 2021, Durkan et al., 2019]. One approach involves leveraging the Lie group structure of the manifold to define parametrization of the flow [Rezende et al., 2020], which is also the strategy we adopt in this work. These recent advance- ments have paved the way for applying normalizing flows to the group SO(3) in order to learn pose estimation in molecular structures [Köhler et al., 2023] and images [Liu et al., 2023, Murphy et al., 2021]. Our work differs from these efforts in that the primary objective of our method is not to target flexible distributions on Riemannian manifolds, but rather to demonstrate that utilizing a flow as a variational distribution aids optimization in learning homeomorphie embedding.
7 Conclusion
In this paper, we investigate obstructions to optimization that can arise when learning encoders for topological spaces. We classify different types of obstructions, provide evidence these are encountered in practice, and give mathematical explanations for how they occur under certain assumptions. We propose GF- VAE, a novel model that employs normalizing flows as variational distributions to help circumvent these issues and show its effectiveness across several datasets when encoding to circles and tori. This work contains several limitations. Firstly, our theoretical analysis is limited by the idealized assumptions necessary to analyze the method using topological tools which do not exactly match those encountered in practice. Secondly, the metrics we define such as winding and crossing numbers are harder define and compute for higher dimensional manifolds. Future work includes expanding our analysis and techniques to a wider array of Lie groups and non-group manifolds.
Acknowledgments and Disclosure of Funding
This work was supported by NSF grants #2107256 and #2134178. We also would like to thank Patrick Forré, Sharvaree Vadgama, Marco Federici, and Erik Bekker for helpful discussions.
10

---

References
Joshua Batson, C Grace Haaf, Yonatan Kahn, and Daniel A Roberts. Topological obstructions to autoencoding. Journal of High Energy Physics, 2021(4):1-43, 2021.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and muchine intelligence, 3598):1788-1888, 2013.
Alican Bozkurt, Babak Esmaeili, Jean- Baptiste Tristan, Dana Brooks, Jennifer Dy, and Jan- Willem van de Meent. Rate-regularization and generalization in variational autoencoders. International Conference Artificial Intelligence and Statistics, pages 3880-3888. PMLR, 2021.
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13475, 2021.
Hugo Jissllees-uprrss Michael Garcia Ortiz, and David Filliat. Symmetry-basee disentangled representation learning requires interaction with environments. Advances in Neural Information Processing Systems, 32, 2019.
Xingyu Chen, Chunyu Wang. Xuguang Lan, Nanning Zheng, and Wenjun Zeng. Neighborhood geometric structure- presteriin variational autoencoder for smooth and bounded data sources. IEEE Transactions on Neural Networks and Learning Systems, pages 1-14, 2021. doi: 10. 10.1109/N.SS. 2021.305359..
Taco Cohen and Max Welling, Group equivariant convolutional networks. In International conference on machine learning. pages 2990-2999. PMLR, 2016.
Tim R. Davidson, Luca Falorsi, Nicola De Cao. Thomas Kipf, and Jakub M. Tomezak. Hyperspherical Variational Auto-Encoders. In Amir Globerson and Ricardo Silva, editors, Proceedings of the Thirty-Fourth Conference on Uncertainty Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6- 10, 2018, pages 856-865. AUAI Press, 2018. URL http::/ /wwai. org/ htgggaa2018///// proceedings/ Sttpgg///pppr////// 309. påf.
Pim de Haan and Luca Falorsi. Topological Constraints on Homcomorphic Auto-Encoding. In NIPS Workshop, December 2018. URL http: /tatp///axiv.... wwg. aa/////ss////..//////////// 1812. 10783. arXiv: 1812.10783.
Emilien Dupont, Miguel Bautista Martin, Alex Colburn, Aditya Sankar, Josh Susskind, and Qi Shan. Equivariant neural rendering. In International Conference on Machine Learning, pages 2761-2770. PMLR, 2020.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. Advances in neural information processing systems, 32, 2019.
Luca Falorsi, Pim de Haan, Tim R. Davidson, Nicola De Cao, Maurice Weiler, Patrick Forré, and Taco S. Cohen. Explorations in Homeomorphic Variational Auto-Encoding, In ICML 2018 workshop Theoretical Foundations and Applications of Deep Generative Models, 2018. URL
org/ abs/ 1807 04689. arXiv: 1807.04689
Luca Falorsi, Pim de Haan, Tim R Davidson, and Patrick Forré. Reparameterizing distributions on lie groups. In The 22nd International Conference Artificial Intelligence and Statistics, pages 0244-3253. PMLR, 2019.
Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic neural networks. Advances in neural information processing systems, 31, 2018.
Xifeng Guo, En Zhu, Xinwang Liu, and Jianping Yin. Affine equivariant autoencoder. In IJCAI, pages 2413-2419, 2019.
Brian Hall and Brian C Hall. Lie groups, Lie algebras, and representations. Springer, 2013.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017. URL httph:://peeeeeeeee//e //tp://eee///eeee/// http:///rrreeeeeeeee/tra//a//aa/
11

---

Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards Definition of Disentangled Representations. arXiv: 1atp::812200255 [cs, statj, December 2018. URI. http::///araa...oog//b//11111 02230. arXiv: 1812.02230 version: 1.
Irina Higgins, Sébastien Racanière, and Danilo Rezende. Symmetry-based representations for artificial and biological general intelligence. Frontiers in Compulational Neuroscience, page 28, 2022.
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Jonas Köhler, Andreas Krämer, and Frank Noé. Smooth normalizing flows. Advances in Neural Information Processing Systems, 34:2796-2809, 2021.
Jonas Köhler, Michele Invernizzi, Pim de Haan, and Frank Noć. Rigid body flows for sampling molecular crystal structures. arXiv preprint arXiv:2301. 11355, 2023.
Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International Conference on Machine Learning. pages 2747-2755. PMLR, 2018.
Yonghyeon Lee, Hyeokjun Kwon, and Frank Park. Neighborhood reconstructing autoencoders. Advances in Neural Information Processing Systems, 34, 2021.
Mario Lezcano-Casado and David Martunez-Rubio. Cheap orthogonal constraints neural networks: A simple parametrization of the orthogonal and unitary group. In International Conference Machine Learning, pages 3794-3803, PMLR, 2019.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In International Conference Learning Representations, 2019.
Yulin Liu, Haoran Liu, Yingda Yin. Yang Wang, Baoquan Chen, and He Wang. Delving into discrete normalizing flows on so (3) manifold for probabilistic rotation modeling. arXiv preprint arXiv:2304.03937, 2023.
Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch. Bernhard Schölkopf. and Oliver Bachem. Disentangling factors of variations using few labels. In Fighth International Conference on Learning Representations. OpenReview. net, 2020.
Giovanni Luca Marchetti, Gustaf Tegnér, Anastasiia Varava, and Danica Kragic. Equivariant repre- sentation learning via class-pose decomposition. arXiv preprint urXiv:2207.03116 16, 2022.
Emile Mathieu and Maximilian Nickel. Riemannian continuous normalizing flows. Advances in Neural Information Processing Systems, 33:2503-222 2515,, 2020.
Yu Meng, Jiaxin Huang, Guangyuan Wang, Chao Zhang, Honglei Zhuang, Lance Kaplan, and Jiawei Han. Spherical text embedding. Advances in Neural Information Processing Systems, 32, 2019.
Ning Miao, Emile Mathieu, N Siddharth, Yee Whye Teh, and Tom Rainforth. On incorporating inductive biases into vaes. In International Conference on Learning Representations.
Michael Moor, Max Horn, Bastian Rieck, and Karsten Borgwardt. Topological autoencoders. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 7045-7054. PMLR, 13-18 Jul 2020. URL https: / http///poeeeee..eeeeee mlr press/vi19 19/1109/mo00a... html.
Kieran A Murphy, Carlos Esteves, Varun Jampani, Srikumar Ramalingam, and Ameesh Makadia.
Implicit-pdft Non-parametrie representation of probability distributions on the rotation manifold. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 7882-7893. URL https:://roceeeeeeeeee http://pooceee..eee/e/ mlr, press/ watess/vvvv/////ee 44399//rrhy2111 a. html.
PMLR, 18-24 Jul 1.21.
12

---

Jung Yeon Park. Ondrej Biza, Linfeng Zhao, Jan Willem van de Meent, and Robin Walters. Learning symmetric embeddings for equivariant world models. International Conference on Machine Learning, 2022.
Luis A. Perez Rey, Vlado Menkovski, and Jim Portegies. Diffusion Variational Autoencoders. In Christian Bessiere, editor, Proceedings of the Twenty-Niuth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 2704-22002 2710. International Joint Conferences Artificial Intelligence Organization, July 2020. doi: 00.24966/ijeci222003775. URL https: htops://doii. org/10. 24963/1 1/063/ijcai. 2020/375.
David Pfau, Irina Higgins, Alex Botev, and Sébastien Racanière, Disentangling by subspace diffusion. Advances in Neural Information Processing Systems, 33:1 18:11433-174155 2020.
Robin Quessard, Thomas Barrett, and William Clements. Learning disentangled representations and group structure of dynamical environments. Advances in Neural Information Processing Systems, 33, 2020.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning, pages 1530-1538. PMLR, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference machine learning. pages 2278-1286 PMLR, 2014.
Danilo Jimenez Rezende, George Papamakarios, Sébastien Racaniere, Michael Albergo, Gurtej Kan- war, Phiala Shanahan, and Kyle Cranmer. Normalizing flows on tori and spheres. International Conference on Machine Learning, pages 8083-8092. PMLR, 2020.
Philip Shilane, Patrick Min, Michael Kazhdan, and Thomas Funkhouser. The princetonn shape benchmark. In Proceedings Shape Modeling Applications, 2004., 2004.
Loek Tonnaer, Luis Armando Perez Rey, Vlado Menkovski, Mike Holenderski, and Jim Portegies. Quantifying and learning linear symmetry- based disentanglement. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 21584-21608. PMLR, 17-23 Jul 2022. URL httph: httpps://proceedingg. mlr. wttppeeeeee/etenreeeeeeeeee html..
Sharvaree Vadgama, Jakub Mikolaj Tomezak, and Erik J Bekkers. Kendall shape-vae: Learning shapes in generative framework. In NeurlPS 2022 Workshop on Symmetry and Geometry in Neural Representations, 2022.
Robin Winter, Marco Bertolini, Tuan Le, Frank Noe, and Djork-Aré Clevert. Unsupervised learning of group invariant and equivariant representations. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances Neural Information Processing Systems, 2022. URL https: /ttp////pp/eeieeeee///eeeeeeeeee www..e/oooommmdd/
Jiacheng Xu and Greg Durrett. Spherical latent spaces for stable variational autoencoders. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4503 4503-4513, 2018
Tao Yang, Xuanchi Ren, Yuwang Wang, Wenjun Zeng, and Nanning Zheng. Towards building group-based unsupervised representation disentanglement framework. In International Conference on Learning Representations, 2022. URL
nwt. aet/foormm?dd==ggalccttmy
Xinqi Zhu, Chang Xu, and Dacheng Tao. Commutative lie group vae for disentanglement learning. In International Conference on Machine Learning, pages 12924 1124-1293 12934. PMLR, 2021.
13

---

A Experimental Details
We train all our models for 150 epochs with a batch-size of 600. For optimization. use the RAdam optimizer [Liu et al., 2019] with learning rate of 5e-4. For all image datasets, we use 4-layer CNN with kernel, stride, and padding of size 4, 2 and respectively followed by a ReakyReLU activation (Table 2) for the encoder and Sigmoid activation for the decoder. The network o² shares the same architecture with the only difference being the last layer, which a fully-connected network followed by a So /pluu activation which is common in standard VAEs. In all our experiments. we used K = 1 for the GF-VAE models was sufficient to avoid the optimization obstructions mentioned in the paper. All models were initialized and trained with 15a different random seeds.
Table 2: Architecture of the encoders and decoders employed for all image datasets.
Encoder
Decoder
Input 32 x 32 images
Input E R² s.t. || = 1
x 4 conv. 32 stride 2, LeL.
4 deconv. 64, stride 2, ELU.
4 x 4 conv. 32 stride 2, LeakyReLU.
4 4 deconv. 64, stride 2, ELU.
4 X 4 conv. 64 stride 2, LeakyReLU.
4 X 4 deconv. 32, stride 2, ELU.
4 conv. 64 stride LeakyRel.U.
deconv. 3, stride Sigmoid.
F.C. 2, n (y)
/ [yl.
B Proofs
We include the proofs for the propositions in the main text.
B.1 Figure Eight Local Minimum
Proposition B.1 Assume that feit) undergoes continuous optimization and is thus continuous in t. Assume that π o fatt) g is injective for all t. The cyclic ordering induced on k points by faroy equal to that induced by føct)- Thus a figure 8 embedding, which corresponds to cyclic order homeomorphic embedding, which has cyclic order (Z1, 22, 23, 24) mod C4 or (24, 23, 22, 21) mod C4.
(z1, 22, 24, 23) mod CA, cannot be transformed
Proof: Since assume o fe()) og is injective for all I, the path z(() = (IT o Soco o (0(0)) inside the k-fold configuration space on S defined Conf 2y for i * j}. In order to prove the claim, we will show that the path-connected components of (S" correspond to cyclic orderings of (zI,..., and thus the start and end point of every path share a cyclic ordering.
{(21,..., 2x) E (5*)* 2, *
(2=12,, Mapping (ZI,..., 2k) + (#k, (#x 21, 21,...,, 2 1k) gives homeomorphism Conf ~ SO(2) x Confa-1 (1)) = SO(2) x Confx-1(R). Let E, 2/ 2j. Consider D
:2 <...<....
< 24 C (R)
We can identify the connected components of Conf (R). The set D is fundamental domain for the action of the symmetric group S on Conf, (R). Thus Conf (E) II oDD) is disjoint union. Linear interpolation shows is connected. The sets D and a (D) for E S₁ are not connected. Consider path from = (21...., 2x) D to (z) E (D)) The element o must
reverse the order of at least two elements =, < z;. Thus the function f(z) = z; 2j must take the value 0 over the path by intermediate value theorem. Hence the path cannot be in Co (R). Thus the connected components of To (r(Conla-((R))
Sk-1-
Since (0()) is connected, To (Confx ~ S That is each connected component of ) labeled by an element of Sk-1 describing the ordering of ăj.... - 1 in R. Each ordering of (ž,, λ in turn corresponds to different cyelic ordering of ø1, 2 in S that is, a different element of SA/Ck- Thus two k-ppoint configurations are homotopic if and only they have the same cyclic ordering.
We used higher number of random seeds than normal account for the training instability.
14

---

B.2 Degree Obstruction
Proposition B.2 The winding number of the initialized model and final model are equal w(n' he(1) og) w('' ha(1)* g).
Proof: The winding number of map a continuous function w((T' hol()) o g). Since the output space E is discreet, the winding number must be constant in t.
B.3 Magnitude Growth in y
We assume that dF is full rank, which is reasonable assumption for an overparameterizee neural network. In that case M = dFe dF is a positive definite symmetric matrix and can be orthogonally diagonalized M QAQ where is orthogonal and
A1
0 X2
A
and A; >> O. The maximum angle between æ = v, and Mæ = v can then computed in terms of the eigenvalues Å; This maximum is computed for the case of n x n symmetric positive definite matrix here". We include the proof for the 2 x 2 case we consider here for completeness.
Lemma B.1 The maximum angle between x and Mx for x E R22
12/ A1 + A2
cos
The angle is maximized at the minimum value of Proof:
x Mx
|| ae ||ll Mx
It suffices to consider |æ||| = 1. Substituting M = =QQQ and = Qx, we want to minimize
x² =TQTAQr yf JT
T -TQTAQQ= y A2y
over all ||,|| since ||Qu| |||||| Letting a yी and noting vř y/ this is equal to minimizing
aAı + (1
ad + (1 a
over < S 1. Setting the derivative equal to gives
(A1 - A2) a(\i A2))
0
2 (23 + a(A X²))) 3/2
and yields one critical value at a = >2/(A1 A₂) corresponding to value 2/A This is the global Art
minimum since the boundary values a = and correspond to maxima with value 1.
Thus the angle between v,C and V,C bounded by 0 cos (2VA1)e/(A1 + X2)) Given that v,C L is tangential to the circle, assuming for simplicity v, has constant length T. and uniform distribution |0, 0 in angle to V,C, the norm of y grows expectation.
'karakusc (htts://matt.staacc wtw.keecchagg..oommseeeeeeeeeeee (tt.serrs//76550/aaaaakue Maximum angle between vector and its linear transformation Ar, URL (version: 2017-05-06):
http:///athttaaccccchaang...oooo
15

---

Proof: Without loss of generality, y = (O RR) and (L cost L sin where [t] < 0. Then we
evaluate
Blyly =113
|| cost, R L L ||dt
20
0
(L2 cos t R² + 2RL sint t + L2 sin" t)dt
20
(T.² + R2 R2)dt
28
L + +R²
by Pythagorean identity and the fact sin odd.
C Continuity Metric
For measuring continuity, adopt similar method Falorsi et al. and evaluate continuity in terms of how the largest "jump" compare to others when walking a continuous path m; E .M for i = .... N pairwise close points. We define qi be the ratio distances between the ground-truth and the learned representation
dm((no mo(m++1))
dm(m;, m4+1)
From the set {4 we compute the continuity metric Loont as
M
Lcont
M
max go:
a-th percentile of {q))
(6)
Pa
In our experiments, we set a 90.
There two differences between how evaluate continuity compared to [Falorsi et al., 2018]. First, we measure the continuity of the rather than fo, which we argue more relevant. Second, in [Falorsi al., 2018], the authors mainly interested in verifying whether the encoder is discontinuous in the topological sense (which they verify by examining the inequality M > yPa for some 7). We on the other hand report continuity the spectrum by computing the that would make пe discontinuous. For evaluating homeomorphism, we conclude for an encoder to be homeomorphic if L < 10 (empirically, we observed the mapping to appear smooth for continuity score below this threshold).
16

---

D Decoding From )
We consider the alternate strategy of removing the projection a and adding a loss that y stays close to the desired manifold M in V Winding number obstructions become far more prominent in this case. In Figure 7, we show the latent space for different random seeds in the teapot case when train with such an objective.
10
65
05
05
00
+.0
1.0
100
-2
1.0
1.0
10
0.5
05
05
x
-65
055
05
1.0
10
-0
-2
-25
0.0
25
1.0
10
10
x
8.
x
80
0.5
05
05
U3
x
0.
x
80
00
0.5
05
05
0.
0.
X
00
80
1.0
1.0
1.0
Figure 7: y and Z space of Isom-AEs trained on teapots for 15 random seeds, where instead of decoding from we decode from with additional soft regularization that constrain y values have unit length.
17

---

E Experiments: Additional Results
We report the full results of our (2) experiments Table
Table 3: Comparison of different VAEs trained the various image datasets in terms of number of encoders with homeomorphic mappings (# H.), correct winding number (# W.), and correct crossing number (# C.) for 15 random seeds. We additionally report the error on continuity as well negative Loglilkelihood (lower is better).
L-shaped Tetrominoes
# W.
C.
Continuity
-Loglikkelihood
2/15
2/15
2/15
137.28
54.15
9.66 5.63
VAE (33 = 1)
0/15
7/15
0/15
117.02 = 23.17
7.26 1 6.99
VAE (3 4)
2/15
9/15
2/15
132.45 = 58.79
2.67 + 0.30
VA- (3
5/15
14/15
50.51 - 58.18
2.71 1.67
GF-VAE (3
9/15
15/15
24.04 29.14
4.43 1.84
Action GFLn-GG--NNN VAE (3 = 4)
10/15
14/15
19.35 - 23.37
4.43 = 1.67
Teapots
# W.
#C
Continuity
-Loglilkelhhood
0/15
15/15
8/15
173.37 21.99
7.49 0.16
VAE (3 = 1)
6/15
15/15
8/15
14.21 = 7.30
7.61 7.61:0007
VAE (3- = 4)
15/15
15/15
15/15
1.22 1.22 0.05
10.60 0.01
GF-VAE (3
7/15
15/15
21.07 = 24.42
8.48 = 0.38
GF-VAE (3
13/15
14/15
8.16 + 17.89
11.05 0.24
Action-GF-VAL (8=4)
13/15
12/15
3.14 1.74
10.96 0.49
Airplanes
#1.
# W.
C.
Continuity
-Loglikelihhod
0/15
3/15
3/15
132.95 40.61
11.55 2.19
VAE (3 = 1)
0/15
6/15
4/15
86.72 = 65.12
12.61 3.41
VAE (B3 = 1)
5/15
9/15
5/15
67.88 = 59.20
12.53 = 1.77
GF-VAE (3 = 1)
0/15
11/15
63.94 = 31.43
10.88 1.24
GF-VAE (3
7/15
13/15
27.70 + 29.23
13.17 1.64
Action- -Faon--F--MAe GF- VAE (3 =4
9/15
12/15
33.26 = 40.03
1.79
F Additional Figures
18

---

VAE
GF-VAE
VAE
GF-VAE
VAE
GF-VAE
7
2n
VAE
GF-VAE
Figure 8: Top: Latent traversals in the decoder for VAE and GF-VAE after training with 8 4. Bottom:
The representations latent space Z for VAEs and GF-VAEs initialized with 15 different random seeds. For VAEs, the gray line inside the circle show the (scaled-down) y-traversals. For GF- VAEs, we it difficult to visualize Y space as it high- high-dimensional space. Therefore, to inspect for obstructions, we show the traversal of the z vectors instead.
Winding Number
1.0
MW IN WA
VAE (8 44
0.5
-05
-1.0
20
100
120
140
1.5
seed
GF-VAR -A-A (8 4)
0.5
0.0
-1.0
120
epoch
Figure 9: The winding numbers for VAE and GF-VAE trained with different random seeds on the tetromino dataset as function of epoch.
19

---

250
100
50
20
40
60
80
100
Figure 10: ||yl|2 as function of epoch for a standard autoencoder trained on teapots for seed
20


Reviews:
[{"review_id": "QXQwVpjvxb", "reviewer": "Reviewer_qXsa", "summary": "The paper addressed several topological obstructions that cannot be easily solved during the optimization of VAE. To solve this problem, the paper proposed to train a special NF to escape the defect, which the authors called GroupFlow. The paper then evaluated the proposed method on synthetic image datasets with different kinds of manifolds. ", "questions": "Does your theory apply to more complicated manifolds? \n\nWhy does higher $\\beta$ leads to better homeomorphism? How about results of the standard $\\beta$-VAE?", "limitations": "The paper has adequate discussion on limitations.", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly written and the mathematical formation of the problem is precise. \n\nThe analysis on the obstructions during optimization clearly shows the invariants which lead to defects. \n\nThe proposed method naturally follows the defects and seems interesting to me.\n\nThe evaluation results indicate the proposed method achieves the best homeomorphism. ", "weaknesses": "I am not an expert in topology, so please refer to other reviewers for the comments on theoretical analysis. However, it seems the authors only analyzed a very simple type of manifold with small freedom. It would be more interesting if there are results on more complex manifolds beyond rotation and colorization. There is a similar concern for experiments.\n\nFrom Table 1 it seems the $\\beta$ parameter has large impact on the homeomorphism result. There should be more (intuitive or theoretical) discussion on the relationship between $\\beta$ (disentanglement) and homeomorphism. It is also important to demonstrate results from the standard $\\beta$-VAE. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your kind review. We are delighted that you found our theory and proposed approach clear and interesting! We will respond to your questions and comments below:\n\n**Applicability to other Lie groups**:\n\nThank you for your question! Yes, our theory does apply to other compact Lie groups as well. Please see our general response for more details.  \n\n**Role of β**:\n\nThe reviewer is absolutely correct that β indeed makes a difference. We show the results for β-VAE in the Appendix in Table 3. We believe the reason for its effectiveness is that because the prior $p(z)$ here is uniform, regularizing the KL terms corresponds to maximizing entropy which encourages the encodings to cover all of the latent Lie group (which is necessary for a homeomorphic mapping). As the results in Table 3 shows, sometimes increasing the β can be enough to learn a homeomorphic mapping (e.g. teapots). However, as we can see in the other cases, we generally need both a GF-VAE and a high β to achieve a homeomorphic mapping (e.g. Tetrominoes & Airplanes). We will make sure to add the results from β-VAE to the main paper as well. \n"}, {"review_id": "mwzxZbEKuB", "reviewer": "Reviewer_mNP4", "summary": "This paper explores the challenges of encoding data into geometric spaces and proposes a solution using Group-Flow Variational Autoencoders (GF-VAEs). The authors discuss how incorporating geometric inductive biases can improve interpretability and generalization but also present obstacles due to topological constraints. They identify two types of local optima that can arise: singularities and incorrect degree or winding number. To overcome these challenges, the authors propose GF-VAEs, which utilize normalizing flows to define multimodal distributions on geometric spaces. The paper characterizes topological defects in encoders, introduces evaluation criteria based on winding number, crossing number, and continuity, and demonstrates that GF-VAEs can escape local optima and achieve a more reliable convergence to a homeomorphic mapping. The main contributions of the paper include characterizing topological defects, proposing GF-VAEs as a solution, and empirically validating their effectiveness. ", "questions": "See the Section 'Weaknesses'.", "limitations": "The authors presented in the paper the most important limitations of the proposed approach.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. *Theoretical and Empirical Analysis*: The paper combines theoretical analysis with empirical evaluations to provide a comprehensive understanding of the challenges and solutions related to encoding data into geometric spaces. This approach strengthens the validity of the proposed methods and their practical implications.\n\n2. *Identification of Obstructions*: The paper effectively identifies and characterizes topological defects that can occur in encoders mapping to geometric structures. By recognizing the specific challenges such as singularities and incorrect degree or winding number, the authors provide a clear understanding of the obstacles that need to be addressed.\n\n3. *Proposal of GF-VAEs*: The introduction of Group-Flow Variational Autoencoders (GF-VAEs) as a solution to the identified obstructions is a significant contribution. The paper explains how GF-VAEs leverage normalizing flows to model complex multimodal distributions on Riemannian manifolds. This proposal offers a practical approach to circumvent local optima and achieve more reliable convergence.\n", "weaknesses": "1. *Idealized Assumptions*: The paper acknowledges that the theoretical analysis is limited by the idealized assumptions necessary to analyze the method using topological tools. These assumptions may not exactly match the real-world scenarios encountered in practice. This limitation undermines the direct applicability of the theoretical findings to real-world problems and raises questions about the generalizability of the proposed solutions.\n\n2. *Limited Metrics for Higher Dimensions*: The metrics defined in the paper, such as winding number and crossing number, are primarily designed for lower-dimensional manifolds. It is mentioned that these metrics become harder to define and compute for higher-dimensional manifolds. This limitation restricts the applicability of the evaluation criteria to higher-dimensional geometric spaces, potentially limiting the scope of the proposed approach.\n\n3. *Restricted Scope of Experiments*: While the paper presents empirical evaluations on two domains, it does not cover a wide range of datasets or scenarios. The limited scope of the experiments may not fully capture the diversity of real-world applications and datasets, leaving open questions about the performance and generalizability of the proposed GF-VAEs in different contexts.\n\n4. *Lack of Comparative Analysis*: The paper lacks a comprehensive comparative analysis of the proposed GF-VAEs with existing methods. While the empirical evaluations demonstrate the effectiveness of GF-VAEs in escaping local optima and achieving better convergence, a thorough comparison with alternative approaches would provide a clearer understanding of the strengths and weaknesses of the proposed method in relation to existing state-of-the-art techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your kind review. We are delighted that you recognized our contributions as significant! We will respond to your questions and comments below:\n\n**Assumptions in Propositions 3.1 and 3.2**:\n\nThe reviewer raises a valid point about the idealistic nature of our assumptions, as we acknowledge in our paper. However, undertaking theory work in a fully realistic setting is indeed challenging, particularly considering that, to the best of our knowledge, our work represents the first attempt to conduct theoretical analysis on the topic of topological obstructions. While we recognize the limitations of our assumptions, we would like to emphasize that these idealized conditions serve as a foundational step to understand the fundamental constraints and possibilities within the context of topological instructions. Our intention is to establish a theoretical framework that lays the groundwork for future research in this promising area. Moreover, it is essential to note that we adopt a perspective that interprets gradient descent as a discretization of a continuous path. Under this viewpoint, our theory highlights that even if continuous optimization were feasible, obtaining a homeomorphic mapping would not be possible. The presence of topological obstructions fundamentally means we have to break the continuity of the optimization path. This finding is an intriguing and novel result that showcases the inherent challenges in learning such mappings, and it holds significance irrespective of the idealized assumptions. As our field progresses, we anticipate that future works will build upon our theoretical framework to consider more realistic scenarios and explore techniques to address practical challenges.\n\n\n**Limited Metrics for Higher Dimensions**:\n\nWe believe there is a small misunderstanding. The concepts such as winding numbers and crossing numbers both can be defined for Lie groups of higher dimensions as well. It is simply a matter of discretization of the latent space.  Though they become computationally more challenging to compute in higher dimensions. We will make sure to clarify this in the final manuscript. Please see our general response for more details.\n\n**Restricted Scope of Experiments**:\n\nThank you for your constructive comment. We hope that our general response has addressed your concern. If the reviewer had a specific domain in mind, we would be happy to try it.  \n\n**Lack of Competitive Analysis**: \n\nThank you for your constructive comment. Due to the uniqueness of our setting, finding suitable prior works for a fair comparison presents challenges. Let us elaborate on the reasons behind our experimental choices and the considerations we took into account while designing our approach. In our work, we deliberately refrain from imposing constraints on the neural networks, such as equivariance, to maintain generality. This choice as well as the fully unsupervised nature of our setting aligns with our long-term goal of eventually moving towards scenarios where we do not assume prior knowledge of the correct Lie group. Thus, as a foundational step, we opt to utilize an ordinary encoder and solely impose constraints on the latent space itself. It is important to acknowledge that if we were to possess knowledge of the correct structure in advance, employing an equivariance neural network would be the correct choice. However, as mentioned in our related work, existing approaches either heavily rely on equivariance neural networks or assume additional information about the data, such as the presence of group elements 'g' in between pairs. These approaches, while useful in their respective contexts, deviate significantly from our fully unsupervised and unconstrained setting.  Also please note that the primary focus of our experiments was to investigate the hypothesis that multimodal distributions have a higher probability of learning a homeomorphic representation when compared to vanilla VAEs with a Lie group latent space. If the reviewer has any specific baseline that would like us to compare to, we would be happy to hear it.  \n\n"}, {"review_id": "1d3KyCMCiR", "reviewer": "Reviewer_1s1k", "summary": "This paper theoretically and empirically characterize obstructions to training Homemorphic encoders with geometric latent spaces, such as local optima due to singularities or incorrect degree or winding numbers. ", "questions": "1. Why the author selected the figure eight and the degree cases to represent topological obstructions. Are there any principles for classifying topological obstructions?\n\n2. What is the precise definition of the projection from $Y$ to $Z$ in formula (1) ?\n\n3. Figure 2 doesn't contain the deconvolutional decoder $f^*$?\n\n4. How is the flow $r$ realized (in line 227) for general lie groups?\n\n5. The last paragraph of page 5 seemed to have provide a simpler solution to avoide topological obstructions than the  GF-VAE?", "limitations": "The paper could benefit from a more detailed discussion of the limitations of the proposed approach.  ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Originality: The paper is original in its approach to addressing topological obstructions in machine learning models. While previous work has explored the use of geometric inductive biases to improve interpretability and generalization, this paper is one of the first to systematically investigate the topological challenges of encoding to a specific geometric structure from the training perspective and propose a novel solution using adapted normalizing flows. \n\nQuality: The paper is of fair quality, in terms of its theoretical rigor and its empirical evaluation. The authors provide a detailed analysis of the topological obstructions that can arise in encoders with geometric latent spaces, and their proposed GF-VAE model is well-motivated  to address these challenges. \n\nClarity: The paper isfairly written and organized. The authors provide ample background and motivation for their work, and their theoretical and empirical analyses are both presented in an accessible manner. The use of figures and examples throughout the paper helps to illustrate the authors' ideas and make the paper more understandable.\n\nSignificance: The authors' characterization of topological obstructions and their proposed GF-VAE model have the potential to improve the interpretability, generalization, and robustness of topological machine learning models. The paper is likely to inspire further research in this area and has the potential to lead to significant improvements in the performance and reliability of machine learning models.", "weaknesses": "While the paper is fairly written with novel contributions to the training of geometric deep learning, there are a few areas where it could be improved:\n\n3. While the paper provides a clear and accessible presentation of the authors' ideas, some of themathematical concepts could be more clearly explained. For example, the motivation of homemorphic encoder and its applications in generative models should be discussed.     The concrete geometries considers in this paper are restrited to $S^1$, which is a commutative group. As th title is for general topological obstructios, the paper could benefit from providing a more general Lie group case.\n\n2. The design of  GF-VAE is to apply flow model as a plug-in prior for the VAE, however combination of VAE and flow model is not new.  Therefore, there are likely other approaches and techniques that could be relevant to GF-VAE that are not discussed in detail.\n Second, The empirical evaluation of the proposed GF-VAE model is also limited to just two domains, and it is unclear how well the model would perform on other types of data. It would be helpful for the authors to provide additional experiments on a wider range of datasets to demonstrate the generalizability of their approach.\n\n3. The paper could benefit from a more detailed discussion of the limitations of the proposed approach. \n\nTypos: stand-> standard in line 227\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your kind review. We are delighted that you recognized our theoretical and empirical contributions as significant and novel! We will respond to your questions and comments below:\n\n\n**Motivation for learning a homeomorphic representation**:\n\nThank you for your comment. As stated in our introduction, we believe a reasonable notion of a good representation is a representation that reflects the underlying data structure, and we believe a sensible definition of this reflection is a representation that locally preserves the distances between the ground truth representations (and by extension, the data) which is described by the concept of homeomorphism. This should not only help us to perform better in downstream tasks, but also has the additional advantage of interpretability which is a desired property of any learned representation.\n\n\n**Does our theory apply to more general Lie Groups?**\n\nThank you for your constructive comment. We believe there is a small misunderstanding. The concepts such as winding numbers and crossing numbers both can be defined for Lie groups of higher dimensions as well. We hope that our general response has addressed your concern.\n\n\n**Combination of VAEs and Flows**:\n\nThank you for your constructive comment. Just to clarify, we apply to the flow to the inference model $q_\\phi(z|x)$ and not the “prior” as the reviewer wrote. Nevertheless, we certainly did not mean to claim that we are the first paper that combined VAEs and Flows and will try to clarify this in the final manuscript. There are two main differences between our work and previous work that combined VAEs and Flows: (1) Previous work on Flow-VAEs only considered Euclidean latent spaces. To the best of our knowledge, we are the first work that employs a geometric latent space in this setting. This is mainly because defining flows on geometric spaces is a challenging task. (2) The motivation for applying flows to $q_\\phi(z|x)$ in the prior work is to tighten the variational gap and therefore obtain a higher $\\log p_\\theta(x)$, while our motivation is to learn a homeomorphic mapping.  Does this address your concern? \n\n**Limited Scope of experiments**:\n\nWe hope that our general response has addressed your concern. If the reviewer had a specific domain in mind, we would be happy to try.  \n\n**Limitations**:\n\nThank you for your constructive comment. We will expand on the limitations of our work mentioned in the conclusion. \n\n**Other types of topological obstructions**:\n\nThat is a great question! Yes it would be possible to characterize different types of topological obstructions using tools from algebraic topology, namely homology and homotopy theory. Computing such topological invariants would give a classification of possible to obstructions to optimization, although many may be unlikely to occur in practice. The main reason we considered the figure-8 shape and the winding number mismatch was because empirically the combination of these two was the most common obstruction we faced in practice (Figure 5). \n\n**Definition of $\\pi: \\mathcal{Y} \\rightarrow \\mathcal{Z}$**:\n\nIn the case of SO(2), $\\pi(y)$ is just a projection on the circle: $\\pi(y) := y / ||y||$. Thank you for your comment. We will clarify this in the final manuscript. \n\n**Decoder in Figure 2**:\n\nOur intention with this figure was to depict our encoder design as well as various types of potential representations we could learn from this design. As the decoder $f^*$ is just an ordinary neural network, we did to include it in that figure due to space constraints. We are happy to try to include it in Figure 2 if the reviewer feels it helps with clarity. \n\n**Normalizing flows for general Lie groups**:\n\nThis is a great question! As mentioned in the related work Section, defining normalizing flows on geometric spaces is a field of itself. One general way of defining a flow on the Lie group is to define the flow on the Lie Algebra and use the exponential map to map it to the Lie group, which has been defined and discussed for most Lie groups. However, more care needs to be taken to account if we want to avoid having a discontinuous pdf as discussed in [1]. We refer the reviewer to [1,2] for more details on this topic.\n\n**Last paragraph on page 5**:\n\nIf the reviewer is referring to the “directly decoding from $y \\in \\mathcal{Y}$ but push embeddings to the unit circle using the loss $| ||y|| − 1|$” we believe there has been a misunderstanding. As pointed out in the same paragraph and Figure 8 in the Appendix, optimizing this objective could result in the wrong winding number.  \n\n**[References]**\n\n[1] Danilo Jimenez Rezende, George Papamakarios, Sébastien Racaniere, Michael Albergo, Gurtej Kan-war, Phiala Shanahan, and Kyle Cranmer. Normalizing flows on tori and spheres. In the International Conference on Machine Learning, pages 8083–8092. PMLR, 2020.\n\n[2] Luca Falorsi, Pim de Haan, Tim R Davidson, and Patrick Forré. Reparameterizing distributions on lie groups. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3244–3253. PMLR, 2019.  \n"}, {"review_id": "JmLJ452Urv", "reviewer": "Reviewer_WSwE", "summary": "The authors investigate two types of topological obstructions that pose challenges for models aimed at learning a particularly structure in the embedding space. Specifically, the authors identify figure eight local minima and mismatches in winding numbers are two defects that make learning the right latent structure difficult. The authors propose a VAE method to improve the learning procedure in order to better reflect the underlying data’s structure. The authors run experiments on three shapes and measure continuity of the learned space in comparison to existing VAE models.\n", "questions": "- Why does the setup include a projection function pi form y to Z? You cite Falorsi, but I couldn’t find intuition or motivation for this at all in the paper. Could you elaborate?\n- How prevalent in practice is the figure eight local minima problem you identify in 3.1? This seems like a very specific topological artificat. Is it found real applications, or does it arise in training of deep networks? \n- Could you explain the justification for the claim on line 178: “In order for f to be a homeomorphism, the winding number must be -1 or 1?” \n- What exactly are the columns in table 1? The caption is quite sparse and doesn’t adequately describe how #H or Continuity are measured. The discussion of continuity seems to be relegated entirely to the appendix. At least a brief description with some intuition should be in the main text.\n- What’s the rough computational cost of training with the additional normalizing flow regularization? \n- How come the baseline “reg-y” model performs so poorly? I’m surprised that explicitly encouraging the latent space to be close to S1 doesn’t improve continuity on two of the three shapes. \n", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Understanding challenges arising in the learning dynamics of models to ensure they appropriately reflect the underlying structure of data is an important topic.\n- Authors acknowledge a reasonable set of limitations in the conclusion.\n- It’s nice to see the authors applying tools from topology to examine how well the right structure can be learned by a model’s embedding space. \n", "weaknesses": "- The writing flow for Section 2 and 3 could use more work to improve clarity\n\t- Problem statement would benefit from an illustrative example of an application for f, h, and pi to help make the setup in line 61-64 more clear. I’d suggest the authors consider moving the running example earlier or using another application to motivate and illustrate this setup.\n\t- Figure 2 isn’t referenced at all in the running examples section and doesn’t appear in the text until Section 3 (line 107). I’d recommend moving the reference to Figure 2 earlier. \n\t- Figure 2: the figure would benefit from additional labels to indicate what each component is. For example, M as I understand is comprised of points for each degree of rotation for the object. The three right hand diagrams should be labeled clearly to indicate they are 3 runs (with different seeds).\n- I found the empricial evidence, especially for the Torus experiment in 5.2 to be quite weak. For example, the authors base the results on only 5 runs grounding the conclusion on the fact that “VAE learns a homeomorphic mapping 0 times” while “GF-VAE learns the mapping 2 out of 5 times”. I don’t find this sufficiently convincing evidence as the authors test a single learning rate and setup across only 5 runs. I would recommend the authors run many more trials to prove this out. \n- Unfair comparison to basleines: In the appendix, the full table shows Beta-VAE (with beta = 4) is quite an effective baseline, but this is absent from the main table 1 in the paper, which only compares against vanilla AE and VAE even though the GF-VAE model uses a beta term. This is not an apples-to-apples comparison to tease out the impact of the author’s proposed method. I suggest the authors include beta-VAEs in the main table and appropriately compare their method to the same beta-VAE baseline.\n- I’m also surprised the authors did not include a comparison to other topological models that impose equivariance to Lie groups, but only to standard VAE models that do not impose group structure in the latent space.\n\nDetails\n- typo in citation line 74\n- typo line 227: “stand” → \"standard”\n- typo 239: extra space\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your kind review and constructive comments. We appreciate your thorough review and detailed questions! We are also delighted that you recognize the topic of our work as an important one. We believe we can address most of your concerns. We will respond to your questions and comments below:\n\n**Clarity of Section 2 and 3**:\n\nThank you for your comments. We all make sure that figure 2 is referenced in the running example part as well as revising it to make it clear that the plots on the right belong to 3 different random seeds. We will also try to add a separate example at the beginning of Section 2, if there was sufficient space. \n\n**Tori Results**:\n\nThank you for your constructive comment. The reviewer is completely correct. We were slightly short on time during the submission for the Tori experiment and therefore failed to run for 15 seeds or do any other hyperparameter tuning. We have run additional experiments since the deadline for the Tori experiments. GF-VAE (β = 6) converges to a homeomorphic mapping 9/15 times while 0/15 for VAE (β = 6). Hope this has addressed your concern. \n\n**Comparison against β-VAE**:\n\nThank you for your constructive comment. We will make sure to add the β-VAE result in the main paper. As mentioned in our Appendix as well as [1], β was an important hyperparameter as it maximizes entropy which makes sure that all of the Lie group is covered and therefore improves continuity. However, as it is shown in the Appendix, you normally need both β and a GF-VAE architecture to learn a homeomorphic mapping (see β-VAE results for Tetrominoes and Airplanes). \n\n**Comparison against equivariant neural networks**:\n\nThank you for your constructive comment. Equivariant neural networks achieve homeomorphism by definition so the reviewer is correct that if we have the prior knowledge of the Lie group *and* we can design an equivariance network for it, we should. However, as we are trying to build towards cases where we do not assume prior knowledge of the group or at least a design of an equivariant neural network for the group, here we consider a setting where no constrains are imposed on the neural network and only on the latent space. We hope that this and our general response and our response to Reviewer mNP4 have addressed your concern.\n\n**Why does $\\pi$ exist?**\n\nOrdinary neural networks only map from Euclidean to Euclidean space. Therefore, if we want to map to a different space such as a Lie group, we have to add a separate mapping which is the role of $\\pi$ in our design. For example, in the case of $S^1$, we can do this by either encoding to $\\theta \\in \\mathbb{R}^1$ and then map to the circle $[\\cos \\theta, \\sin \\theta]$, or encode to $y \\in \\mathbb{R}^2$ first and then project to the circle $\\pi(y) := y / \\|y\\|$. Hope this clarifies the role of $\\pi$.\n\n**How prevalent is the figure-8 obstruction?**\n\nWe mainly decided to analyze this obstruction because empirically this was the most common obstructions we faced in practice as you can see in the encodings in Figure 5. It is not unique in the sense that sometimes it manifests itself in the form of trefoil or figures with crossing numbers higher than 1 as well. However, we focus on figure 8 in our theory because it is the simplest case. Please see our general response for more details. \n\n**Why should the winding number be either 1 or -1 for a homeomorphic mapping?**\n\nThank you for the question! The winding number can intuitively be explained by how many times do we ‘warp around‘ in the output space if we warp around once in the input space. Winding number 1 means as we go through $-\\pi$ to $\\pi$ (rotating counter-clockwise) in the input, the circle in the output space is also covered once in the same direction. Winding number -1 is basically the same thing except the circle in the output space is warped in the clockwise direction as we go through $-\\pi$ to $\\pi$ in the input space.\n\n**Details of our Evaluation Metrics**:\n \nThank you for the question! We could have done a better job describing the details of our evaluation. The #H column shows the number of runs that have converged successfully to a homeomorphic mapping. Evaluating homeomorphism is difficult in general as it requires verifying continuity across the full domain. Here, we evaluated it based on two criteria: (1) if the continuity score is less than 8  (empirically, we observed the encoding to appear smooth below this threshold), and (2) winding number 1 or -1. The other column is continuity for which we describe the details of its calculation in the Appendix. This is done by taking an equidistant trajectory in the input space and keeping track of the pairwise distances in the output space. We then divide the maximum $q_i$ (where there was the most discontinuity) by the 90-th percentile $\\{q_i\\}$. We will provide more details of our valuation in the final manuscript.\n\n**Additional Computational cost of Flows**:\n\nThis is a valid point and we will add it to our discussion of limitations in the paper. For the set of experiments in the paper, the additional computational cost was relatively small because in our experience, only a single flow layer was enough. Moreover, the flow is applied on the latent space which is very low dimensional therefore does not require a lot of extra computation. As an example, the training run time for a VAE on the Airplane dataset was 1 hour and 17 minutes while for GF-VAE it was 1 hour and 31 minutes. \n\n**$reg\\text{-}y$ performance**:\n\nThis was somewhat of a surprise to ourselves too. What in practice occurred when regularizing $y$-space was that if the mapping was already close to something homeomorphic, then it helped stabilize the convergence to a homeomorphic mapping. However, if the representation was stuck in a “figure 8” minima, then it made it even more difficult for the model to escape this local optimum as most of the arcs of “figure 8” had to be close to the circle and had no room to move. \n"}]

Author rebuttal:
We sincerely appreciate the valuable feedback provided by all the reviewers. Your constructive comments and efforts in evaluating our paper are highly appreciated. We are pleased to see that the general consensus is that our theoretical analysis of topological obstructions, along with the proposed GF-VAE, are significant and novel contributions to the field. There are some shared concerns among reviewers regarding the application of our theory and method to other domains and topological spaces, which we will address below before addressing individual comments:

**Applicability to other Lie groups and topological obstructions**:

All reviewers inquired about the applicability of our theoretical analysis beyond the specific case of $\mathrm{SO}(2)$ and “figure 8” shape. The wording in the conclusion might have made it seem the mathematical concepts we employ are limited to $\mathrm{SO}(2)$, but this is not the case. To clarify, the concepts of winding number and crossing number can be extended to other Lie groups and topological spaces of higher dimensions.  In higher dimensions, the winding number is known as the degree of a continuous mapping and can be computed over any compact manifold. Thus Prop 3.2 may be readily generalized. The crossing number can be generalized in different ways including the measure of the size of the self-intersection. For instance, we can consider a 2D sheet intersecting itself in a manner akin to a twisted ribbon.  Additionally, in the case of the figure-8 example, we note that other shapes with the incorrect crossing number, such as trefoil or quatrefoil, can also exist as a possible encoding and would be part of proposition 3.1. Computing these metrics for other topological spaces primarily involves discretization of the latent space, though we recognize that the computational challenges increase with higher dimensions. We apologize for any confusion caused by our phrasing, and we will rectify this in the final manuscript to make it clear that the defined metrics can be applied to various Lie groups and topological spaces. 

**Limited Scope of Experiments**:

Reviewers mNP4, 1s1k, and WSwE pointed out that we are only considering Circles and Tori in our experiments.  In the field of equivariance/disentanglement, and generally extracting the right Lie group from data, $\mathrm{SO}(2)$ and $\mathrm{SO}(2) \times \mathrm{SO}(2)$ are the two of the most common Lie groups. We could consider adding translation to the features as well (e.g. $(R^2, +) \times \mathrm{SO}(2)$), but translation can be modeled with a standard Euclidean latent space so therefore does not lead to the topological obstructions discussed in the paper. The only common Lie group we did not consider in our experiments was the Lie group $\mathrm{SO}(3)$ which we will try our best to add to the camera ready. However, we would like to point out that besides homeomorphic-VAE, our work would be the only VAE-based work that would manage to learn a homeomorphic mapping from images to $\mathrm{SO}(3)$ in a fully unsupervised manner that doesn’t use an equivariance neural network. Moreover, even in the case of homeomorphic-VAE, it requires two additional regularizers as well as a hard tuned β-scheduler to have a single successful run. While we understand the reviewers' desire to see broader experiments, we feel that given the novelty of our theoretical and methodological contributions, the focus on circles and tori is sufficient for this paper. However, we assure the reviewers that we will explore the extension of our approach to other Lie groups in future work.


Meta-review: This paper explores geometric inductive biases  and shows that it can improve interpretability and generalization but it is challenged by obstacles in the optimization due to topological constraints. Authors discuss  two types of local optimas that can arise: singularities and incorrect degree or winding number. The paper proposes GF-VAEs that uses  normalizing flows to define multimodal distributions on geometric spaces to alleviate these issues. The paper  introduces evaluation criteria based on winding number, crossing number, and continuity, and demonstrates that GF-VAEs  escapes local optima and converges to a homeomorphic mapping. 

Reviewers agreed that the contribution of the paper is significant , although limited in scope in terms of the theory and the applications that consist only in two domains and lack of baseline comparisons. Authors clarified the applicability to other lie groups in the rebuttal and clarified and extended experiments on tori , and  promised to move the beta vae experiment from appendix to main, addressing overall reviewers concerns. Please take into account in the revision all reviewers comments and suggestions.
Decision: Accept (poster)