SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The SIPaKMeD dataset includes 4,049 single-cell Pap smear images for cervical cancer diagnosis and is split into 5 classes [32] (see Section C of SM for examples). We assume O(2) symmetry for the SIPaKMeD dataset. We resized the original SIPaKMeD images to size  $32 \times 32$  for better usability. In accordance with the original data splitting, 3,549 samples were used for training and 500 samples for testing.
IQVAE-M. As we did with rotated MNIST, we conducted the same experiment on the SIPaKMeD dataset. Furthermore, we also as
Table 6: Clustering ARIs and classification test accuracies for SIPaKMeD (repeated five times).
METHOD	k-MEANS	GMM	SVM	RF
AE	33.4±4.0	35.6±1.0	78.6±0.5	76.9±1.1
VAE	33.9±1.5	34.0±1.7	79.4±1.0	76.5±2.1
β-VAE	34.4±1.5	33.7±1.1	79.1±1.0	76.6±1.3
CRVAE	24.9±3.6	31.9±1.3	77.1±0.7	80.6±1.1
QAE	40.9±1.9	41.1±2.9	80.5±0.9	81.6±0.7
FMVAE	39.6±0.3	37.9±1.1	80.0±0.8	75.7±1.2
QVAE	40.6±3.2	40.6±2.5	82.2±0.8	81.7±0.9
IQVAE	39.3±0.4	44.7±2.0	82.7±0.3	80.9±0.7
IQVAE-M	43.0±1.1	46.5±2.9	84.3±0.7	81.6±1.8
sessed the proposed IQVAE-M that incorporates a learnable flexible Riemannian metric as described in (7) and utilizes the latent Mahalanobis distance when computing the radial basis function (RBF) kernel  $K: \mathbb{R}^n \times \mathbb{R}^n \to \mathbb{R}$  of SVMs as  $K(\mathbf{z}_0, \mathbf{z}_1) = \exp \left( -\left(\mathbf{z}_1 - \mathbf{z}_0\right)^T \mathbf{C}_n\left(\mathbf{z}_1 - \mathbf{z}_0\right) \right)$ . To quantitatively analyze the learned representations, we present the downstream task performance of the competing models on the SIPaKMeD dataset in Table 6 (see Section E of SM for UMAP visualizations). As shown in Table 6, the proposed methods exhibit superior performance compared to other models<sup>4</sup>.
We have proposed and demonstrated the effectiveness of IQVAE, a simple yet effective approach that maintains symmetry and geometry in data. However, our work has two main limitations as follows.
Predefined groups of symmetry transformations. We assume the group structure of a given dataset is known in advance. This group invariance is represented using the quotient auto-encoding framework with a Riemannian isometry. However, several recent papers have delved into learning an unknown group directly from data. For example, [35] presents a novel neural network that identifies bispectral invariants. [31] employs Lie algebra to find underlying Lie group invariances. [46] tackles the disentanglement challenge with VAEs, aiming to learn the latent space in terms of one-parameter subgroups of Lie groups. Integrating these methods with our IQVAEs learning approach could be a promising direction for future research.
Euclidean metric tensors. We focus on cases where the metric tensor of the observation space is Euclidean. If the intrinsic dimensionality of the image manifold is significantly lower than that of the ambient observation space, the Euclidean metric of the observation space can well serve as a Riemannian metric for the intrinsic geometry of the image manifold. This reasoning supports our use of the pullback metric as a metric tensor for the latent space. However, when constructing a latent representation that effectively addresses a specific task, a specialized metric tensor might be more appropriate than the standard Euclidean metric of the ambient space. For instance, when tasks involve capturing subtle variations in a local image patch, the pullback metric derived from the entire observation dimensions may not provide the most efficient geometry. In such cases, a specialized metric structure better suited for capturing these local variations should be considered. In this context, a task-specific metric can be determined using prior knowledge about the tasks [27]. Alternatively, IQVAEs can semi-supervisedly learn this specialized metric with minimal labeled data, leveraging metric learning concepts [3]. This approach holds significant potential for future research, as it allows the model to tailor its representation to the specific requirements of the task, thereby improving overall performance.
4When using the Euclidean RBF  $(\mathbf{C}_n = \mathbf{I}_n)$  for SVMs of the IQVAE-M, the accuracy decreases to 81.9; it shows the usefulness of the learned metric and the corresponding Mahalanobis RBF.
Changwook Jeong sincerely acknowledges the support from Samsung Electronics Co., Ltd (IO221027-03271-01, IO230220-05060-01) and is further supported by the Technology Innovation Program (20020803, RS-2023-00231956) and Korea Institute for Advancement of Technology (KIAT) grant (P0023703), both funded by the MOTIE, Korea (1415180307, 1415187475), the National Research Foundation of Korea (NRF) grant (RS-2023-00257666) funded by the MSIT, Korea, the IC Design Education Center (IDEC), Korea, and the research project funds (1.220125.01, 1.230063.01) and supercomputing center of UNIST.
[1] Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg. Latent space oddity: on the curvature of deep generative models. In International Conference on Learning Representations, 2018.
[2] Georgios Arvanitidis, Soren Hauberg, and Bernhard Schölkopf. Geometrically enriched latent spaces. In International Conference on Artificial Intelligence and Statistics, pages 631-639. PMLR, 2021.
[3] Aurélien Bellet, Amaury Habrard, and Marc Sebban. Metric learning. Springer, 2022.
[4] Gregory Benton, Marc Finzi, Pavel Izmailov, and Andrew G Wilson. Learning invariances in neural networks from training data. Advances in Neural Information Processing Systems, 33:17605-17616, 2020.
[5] Tristan Bepler, Ellen Zhong, Kotaro Kelley, Edward Brignole, and Bonnie Berger. Explicitly disentangling image content from translation and rotation with spatial-VAE. Advances in Neural Information Processing Systems, 32, 2019.
[6] Pratik Prabhanjan Brahma, Dapeng Wu, and Yiyuan She. Why deep learning works: A manifold disentanglement perspective. IEEE Transactions on Neural Networks and Learning Systems, 27(10):1997-2008, 2015.
[7] Clément Chadebec and Stéphanie Allassonnière. A geometric perspective on variational autoencoders. arXiv preprint arXiv:2209.07370, 2022.
[8] Nutan Chen, Alexej Klushyn, Francesco Ferroni, Justin Bayer, and Patrick Van Der Smagt. Learning flat latent manifolds with VAEs. In International Conference on Machine Learning, pages 1587-1596. PMLR, 2020.
[9] Nutan Chen, Alexej Klushyn, Richard Kurle, Xueyan Jiang, Justin Bayer, and Patrick Smagt. Metrics for deep generative models. In International Conference on Artificial Intelligence and Statistics, pages 1540-1550. PMLR, 2018.
[10] Taco Cohen and Max Welling. Group equivariant convolutional networks. In International Conference on Machine Learning, pages 2990-2999. PMLR, 2016.
[11] Luca Falorsi, Pim De Haan, Tim R Davidson, Nicola De Cao, Maurice Weiler, Patrick Forre, and Taco S Cohen. Explorations in homeomorphic variational auto-encoding. In ICML 2018 Workshop on Theoretical Foundations and Applications of Deep Generative Models, 2021.
[12] Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to Lie groups on arbitrary continuous data. In International Conference on Machine Learning, pages 3165-3176. PMLR, 2020.
[13] Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael Black, and Bernhard Scholkopf. From variational to deterministic autoencoders. In International Conference on Learning Representations, 2020.
[14] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020.
[32] Marina E Plissiti, Panagiotis Dimitrakopoulos, Giorgos Sfikas, Christophoros Nikou, O Krikoni, and Antonia Charchanti. SIPaKMeD: A new dataset for feature and image based classification of normal and pathological cervical cells in Pap smear images. In 2018 25th IEEE International Conference on Image Processing (ICIP), pages 3144-3148. IEEE, 2018.
[33] Douglas A Reynolds. Gaussian mixture models. Encyclopedia of Biometrics, 741(659-663), 2009.
[34] Tammy Riklin-Raviv and Amnon Shashua. The quotient image: Class based recognition and synthesis under varying illumination conditions. In Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149), volume 2, pages 566-571. IEEE, 1999.
[35] Sophia Sanborn, Christian A Shewmake, Bruno Olshausen, and Christopher J Hillar. Bispectral neural networks. In International Conference on Learning Representations, 2023.
[36] Hang Shao, Abhishek Kumar, and P Thomas Fletcher. The Riemannian geometry of deep generative models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 315-323, 2018.
[37] Samarth Sinha and Adji Bousso Dieng. Consistency regularization for variational auto-encoders. Advances in Neural Information Processing Systems, 34:12943–12954, 2021.
[38] Douglas Steinley. Properties of the Hubert-Arable adjusted Rand index. Psychological methods, 9(3):386, 2004.
[39] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pages 6438-6447. PMLR, 2019.
[40] Binxu Wang and Carlos R Ponce. A geometric analysis of deep generative image models and its applications. In International Conference on Learning Representations, 2021.
[41] Haitao Wang, Stan Z Li, and Yangsheng Wang. Generalized quotient image. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pages II-II. IEEE, 2004.
[42] Junliang Wang, Chuqiao Xu, Zhengliang Yang, Jie Zhang, and Xiaou Li. Deformable convolutional networks for efficient mixed-type wafer defect pattern recognition. IEEE Transactions on Semiconductor Manufacturing, 33(4):587-596, 2020.
[43] Rui Wang and Nan Chen. Wafer map defect pattern recognition using rotation-invariant features. IEEE Transactions on Semiconductor Manufacturing, 32(4):596-604, 2019.
[44] Maurice Weiler, Fred A Hamprecht, and Martin Storath. Learning steerable filters for rotation equivariant CNNs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 849-858, 2018.
[45] Tao Yang, Georgios Arvanitidis, Dongmei Fu, Xiaogang Li, and Søren Hauberg. Geodesic clustering in deep generative models. arXiv preprint arXiv:1809.04747, 2018.
[46] Xinqi Zhu, Chang Xu, and Dacheng Tao. Commutative Lie group VAE for disentanglement learning. In International Conference on Machine Learning, pages 12924-12934. PMLR, 2021.