SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We assume that  $dF$  is full rank, which is a reasonable assumption for an overparameterized neural network. In that case  $M = dF^T dF$  is a positive definite symmetric matrix and can be orthogonally diagonalized  $M = Q\Lambda Q^T$  where  $Q$  is orthogonal and
$$
\Lambda = \left( \begin{array}{c c} \lambda_ {1} & 0 \\ 0 & \lambda_ {2} \end{array} \right)
$$
and  $\lambda_{i} > 0$ . The maximum angle between  $\pmb{x} = \nabla_y\mathcal{L}$  and  $M\pmb{x} = \tilde{\nabla}_y\mathcal{L}$  can then be computed in terms of the eigenvalues  $\lambda_{i}$ . This maximum is computed for the case of an  $n\times n$  symmetric positive definite matrix here. We include the proof for the  $2\times 2$  case we consider here for completeness.
Lemma B.1 The maximum angle between  $x$  and  $Mx$  for  $x \in \mathbb{R}_{\neq 0}^2$  is
$$
\cos^ {- 1} \left(\frac {2 \sqrt {\lambda_ {1} \lambda_ {2}}}{\lambda_ {1} + \lambda_ {2}}\right).
$$
Proof: The angle is maximized at the minimum value of
$$
\frac {x ^ {T} M x}{\| x \| \| M x \|.}
$$
It suffices to consider  $\| x \| = 1$ . Substituting  $M = Q \Lambda Q^T$  and  $y = Qx$ , we want to minimize
$$
\frac {x ^ {T} Q ^ {T} \Lambda Q x}{x ^ {T} Q ^ {T} \Lambda^ {2} Q x} = \frac {y ^ {T} \Lambda y}{y ^ {T} \Lambda^ {2} y}
$$
over all  $\| y \| = 1$  since  $\| Qx \| = \| x \| = 1$ . Letting  $a = y_1^2$  and noting  $y_1^2 + y_2^2 = 1$ , this is equal to minimizing
$$
\frac {a \lambda_ {1} + (1 - a) \lambda_ {2}}{a \lambda_ {1} ^ {2} + (1 - a) \lambda_ {2} ^ {2}}
$$
over  $0 \leq a \leq 1$ . Setting the derivative equal to 0 gives
$$
\frac {(\lambda_ {1} - \lambda_ {2}) ^ {2} (- \lambda_ {2} + a (\lambda_ {1} + \lambda_ {2}))}{2 (\lambda_ {2} ^ {2} + a (\lambda_ {1} ^ {2} - \lambda_ {2} ^ {2}))) ^ {3 / 2}} = 0
$$
and yields one critical value at  $a = \lambda_2 / (\lambda_1 + \lambda_2)$  corresponding to value  $\frac{2\sqrt{\lambda_1\lambda_2}}{\lambda_1 + \lambda_2}$ . This is the global minimum since the boundary values  $a = 0$  and  $a = 1$  correspond to maxima with value 1.
Thus the angle between  $\nabla_y\mathcal{L}$  and  $\tilde{\nabla}_y\mathcal{L}$  is bounded by  $\theta = \cos^{-1}\left(2\sqrt{\lambda_1\lambda_2} /(\lambda_1 + \lambda_2)\right)$ . Given that  $\nabla_y\mathcal{L}$  is tangential to the circle, assuming for simplicity  $\tilde{\nabla}_y\mathcal{L}$  has constant length  $L$  and uniform distribution  $[- \theta ,\theta ]$  in angle to  $\nabla_y\mathcal{L}$ , the norm of  $y$  grows in expectation.
$^{4}$ karakusc (https://math.stackexchange.com/users/176950/karakusc), Maximum angle between a vector  $x$  and its linear transformation  $Ax$ , URL (version: 2017-05-06): https://math.stackexchange.com/q/2266057
Proof: Without loss of generality,  $y = (0,R)$  and  $v = (L\cos t,L\sin t)$  where  $|t| < \theta$ . Then we evaluate
$$
\begin{array}{l} \mathbb {E} [ \| y + v \| ^ {2} ] = \frac {1}{2 \theta} \int_ {- \theta} ^ {\theta} \| (L \cos t, R + L \sin t) \| ^ {2} \| d t \\ = \frac {1}{2 \theta} \int_ {- \theta} ^ {\theta} \left(L ^ {2} \cos^ {2} t + R ^ {2} + 2 R L \sin t + L ^ {2} \sin^ {2} t\right) d t \\ = \frac {1}{2 \theta} \int_ {- \theta} ^ {\theta} \left(L ^ {2} + R ^ {2}\right) d t \\ = L ^ {2} + R ^ {2}. \\ \end{array}
$$
by Pythagorean identity and the fact  $\sin t$  is odd.
C Continuity Metric
For measuring continuity, we adopt a similar method as Falorsi et al. and evaluate continuity in terms of how the largest "jump" compare to others when walking a continuous path  $m_i \in \mathcal{M}$  for  $i = 1 \cdots N$  pairwise close points. We define  $q_i$  to be the ratio distances between the ground-truth and the learned representation
$$
q _ {i} = \frac {d _ {\mathcal {M}} (\eta_ {\phi} (m _ {i}) , \eta_ {\phi} (m _ {i + 1}))}{d _ {\mathcal {M}} (m _ {i} , m _ {i + 1})}.
$$
From the set  $\{q_i\}_i$ , we compute the continuity metric  $L_{\mathrm{cont}}$  as
$$
L _ {\text {c o n t}} = \frac {M}{P _ {\alpha}}, \quad M = \max  _ {i} q _ {i}, \quad P _ {\alpha} = \alpha - \text {t h p e r c e n t i l e o f} \left\{q _ {i} \right\} _ {i = 1} ^ {N}. \tag {6}
$$
In our experiments, we set  $\alpha = 90$
There are two differences between how we evaluate continuity compared to [Falorsi et al., 2018]. First, we measure the continuity of  $\psi_{\phi}$  rather than  $f_{\phi}$ , which we argue is more relevant. Second, in [Falorsi et al., 2018], the authors are mainly interested in verifying whether the encoder is discontinuous in the topological sense (which they verify by examining the inequality  $M > \gamma P_{\alpha}$  for some  $\gamma$ ). We on the other hand report continuity on the spectrum by computing the  $\gamma$  that would make  $\eta_{\phi}$  discontinuous. For evaluating homeomorphism, we conclude for an encoder to be homeomorphic if  $L_{\mathrm{cont}} < 10$  (empirically, we observed the mapping to appear smooth for a continuity score below this threshold).
D Decoding From  $\mathcal{V}$
We consider the alternate strategy of removing the projection  $\pi$  and adding a loss so that  $y$  stays close to the desired manifold  $\mathcal{M}$  in  $\mathcal{V}$ . Winding number obstructions become far more prominent in this case. In Figure 7, we show the latent space for different random seeds in the teapot case when we train with such an objective.
Figure 7:  $\mathcal{V}$  and  $\mathcal{Z}$  space of Isom-AEs trained on teapots for 15 random seeds, where instead of decoding from  $\mathcal{Z}$ , we decode from  $\mathcal{V}$  with an additional soft regularization that constrain  $y$  values to have unit length.
E Experiments: Additional Results
We report the full results of our SO(2) experiments in Table 3.
Table 3: Comparison of different VAEs trained on the various image datasets in terms of number of encoders with homeomorphic mappings (# H.), correct winding number (# W.), and correct crossing number (# C.) for 15 random seeds. We additionally report the error on continuity as well as negative Loglikelihood (lower is better).
L-shaped Tetrominoes
# H.	# W.	# C.	Continuity	-Loglikelihood
AE	2/15	2/15	2/15	137.28 ± 54.15	9.66 ± 5.63
VAE (β = 1)	0/15	7/15	0/15	117.02 ± 23.17	7.26 ± 6.99
VAE (β = 4)	2/15	9/15	2/15	132.45 ± 58.79	2.67 ± 0.30
GF-VAE (β = 1)	5/15	14/15	-	50.51 ± 58.18	2.71 ± 1.67
GF-VAE (β = 4)	9/15	15/15	-	24.04 ± 29.14	4.43 ± 1.84
Action-GF-VAE (β = 4)	10/15	14/15	-	19.35 ± 23.37	4.43 ± 1.67
Teapots
# H.	# W.	# C.	Continuity	-Loglikelihood.
AE	0/15	15/15	8/15	173.37 ± 21.99	7.49 ± 0.16
VAE (β = 1)	6/15	15/15	8/15	14.21 ± 7.30	7.61 ± 0.07
VAE (β = 4)	15/15	15/15	15/15	1.22 ± 0.05	10.60 ± 0.01
GF-VAE (β = 1)	7/15	15/15	-	21.07 ± 24.42	8.48 ± 0.38
GF-VAE (β = 4)	13/15	14/15	-	8.16 ± 17.89	11.05 ± 0.24
Action-GF-VAE (β = 4)	13/15	12/15	-	3.14 ± 1.74	10.96 ± 0.49
Airplanes
# H.	# W.	# C.	Continuity	-Loglikelihood
AE	0/15	3/15	3/15	132.95 ± 40.61	11.55 ± 2.19
VAE (β = 1)	0/15	6/15	4/15	86.72 ± 65.12	12.61 ± 3.41
VAE (β = 4)	5/15	9/15	5/15	67.88 ± 59.20	12.53 ± 1.77
GF-VAE (β = 1)	0/15	11/15	-	63.94 ± 34.43	10.88 ± 1.24
GF-VAE (β = 4)	7/15	13/15	-	27.70 ± 29.23	13.17 ± 1.64
Action-GF-VAE (β = 4)	9/15	12/15	-	33.26 ± 40.03	12.27 ± 1.79
Figure 8: Top: Latent traversals in the decoder for a VAE and GF-VAE after training with  $\beta = 4$ . Bottom: The representations in latent space  $\mathcal{Z}$  for VAEs and GF-VAEs initialized with 15 different random seeds. For VAEs, the gray line inside the circle show the (scaled-down)  $y$ -traversals. For GF-VAEs, we it is difficult to visualize  $\mathcal{V}$  space as it is high-dimensional space. Therefore, to inspect for obstructions, we show the traversal of the  $z$  vectors instead.
Figure 9: The winding numbers for VAE and GF-VAE trained with 5 different random seeds on the tetromino dataset as a function of epoch.
Figure 10:  $\| y\| _2$  as a function of epoch for a standard autoencoder trained on teapots for seed 0.