Learning Identity-Preserving Transformations on Data Manifolds
Authors: Marissa Catherine Connor
Kion Fallah
Christopher John Rozell
Venue: Accepted by TMLR ()

Abstract: Many machine learning techniques incorporate identity-preserving transformations into their models to generalize their performance to previously unseen data. These transformations are typically selected from a set of functions that are known to maintain the identity of an input when applied (e.g., rotation, translation, flipping, and scaling). However, there are many natural variations that cannot be labeled for supervision or defined through examination of the data. As suggested by the manifold hypothesis, many of these natural variations live on or near a low-dimensional, nonlinear manifold. Several techniques represent manifold variations through a set of learned Lie group operators that define directions of motion on the manifold. However, these approaches are limited because they require transformation labels when training their models and they lack a method for determining which regions of the manifold are appropriate for applying each specific operator. We address these limitations by introducing a learning strategy that does not require transformation labels and developing a method that learns the local regions where each operator is likely to be used while preserving the identity of inputs. Experiments on MNIST and Fashion MNIST highlight our model's ability to learn identity-preserving transformations on multi-class datasets. Additionally, we train on CelebA to showcase our model's ability to learn semantically meaningful transformations on complex datasets in an unsupervised manner. 

Published in Transactions on Machine Learning Research (03/2023)
Learning folds
Transformations on Data Mani-
Marissa Connor, Kion Fallah', Christopher J. Rozell School of Electrical and Computer Engincering Georgia Institute of Technology Atlanta, GA 30332 (marissa. connor, kion, crozell)@gatech.co
Reviewed on OpenReview:
Abstract
Many machine learning techniques incorporate identity-preservinn transformations into their models to generalize their performance to previously unseen data. These transformations are typically selected from a set of functions that are known maintain the identity of an input when applied (e.g., rotation, translation, flipping, and scaling). However, there many natural variations that cannot be labeled for supervision or defined through examination of the data. As suggested by the manifold hypothesis, many of these natural variations live on or near a low-dimensiee din-dimensiona,, nonlinear manifold. Several techniques represent manifold variations through a set of learned Lie group operators that define directions of motion on the manifold. However, these approaches are limited because they require transformation labels when training their models and they lack method for determining which regions of the manifold are appropriate for applying each specific operator. We address these limitations by introducing learning strategy that does not require transformation labels and developing method that learns the local regions where each operator is likely to be used while preserving the identity of imputs. Experiments on MNIST and Fashion MNIST highlight our model's ability to learn identity- transformations on multi-class datasets. Additionally, we train on CelobA to showcase our modol's ability to learn semantically meaningful transformations on complex datasets an unsupervised manner.
1 Introduction
A goal of many machine learning models to accurately identify objects as they undergo natural trans- formations a lask that. humans are adept at. According to the manifold hypothesis, natural variations in high-ddmennsionll data lie
near low-dimensional, nonlinear manifold (Fefferman et al.) 2016). Additionally, the manifolds representing different classes are separated by low density regions (Rifai et al. 2011ah. Natural physical laws govern the possible transformations that objects can undergo and many of the identity-preservvnn transformations (e-g., changes in viewpoint, color, and lighting) are shared among classes data. The sharing transformations between classes enables increased efficiency in defining data variations model can represent a limited set of transformations that can describe a majority of variations in many classes. Several machine learning models incorporate specific plentity-preserving transformations that are shared among large number of classes to generalize the performance of their model to unseen data. These include equivariant models that incorporate transformations like translation and rotation into intermediate network layers (Cohen & Welling 2016 Cohen et al.) 2018) and data augmentation techniques
that apply known identity-preserving variations to data while training (Cubuk et al. 2019a Ho et al. 2019
"equll contribution.

---

Published in Transactions on Machine Learning Research (03/2023)
Lim et al. 2019): However, in many datasets there may be natural transformations shared among classes that are not easily prespecified from intuition, making it critical that we develop a model that can learn both 1) representation for these transformations without explicit transformation labels and 2) the context of when each transformation is likely to be relevant.
Manifold learning strategies estimate the low-dimensional manifold structure of data and examine how points are related through natural transformations. Most common techniques learn manifold embeddings of data
points without a method for generating transformed points on the manifold (Tenenbaum et al. 2000 Roweis
& Saul 2000: Belkin & Niyogi 2003 Maaten & Hinton 2008). Other manifold learning techniques learn to
transform points on the manifold either along linear manifold tangent planes (Dollár et al.)
2007
Bengio
& Monperrus 2005
Park et al.) 2015; Rifai et al.] 2011e| Kumar et al., 2017) or through nonlinear Lie
group operators that transverse the manifold (Rao & Ruderman 1999; Miao & Rao 2007 Culpepper &
Olshausen 2009] SShl-Dicksteinn et al. 2010 Cohen & Welling. 2014 Hauberg et al.] 2016 Connor & Rozell
2020; Connor et al.) 2021). Lie group operators represent infinitesimal transformations which can be applied to data through an exponential mapping to transform points along a manifold, and a manifold can be globally defined by set of operators that each move in different directions along it (Hoffman 1966 Dodwell 1983
A Lic group operator model is well-suited for representing natural data variations because the operators can be learned from the data, applied to data points to transform them beyond their local neighborhoods, and used to estimate geodesic paths.
While the Lie group operator models have many benefits, previous approaches demonstrate two shortcomings. First, to learn Lie group operators that represent a data manifold, pairs of training points are selected which lie within a neighborhood of one another. The training objectíve encourages efficient paths between these nearby points, and the choice of training point pairs strongly influences the types of manifold transformations that are learned. Recent papers incorporating Lie group operators into machine learning models have either used predefined operators that represent known transformations groups (e.g., the 3D rotational group 50(3) Falorsi et al.) 2019)), ), required transformation labels for selecting point pairs when training (Connor & Rozell 2020), or randomil selected pairs of points from the same class (Connor et al., 2021). To learn an effective model with datasstss having no labeled transformation structure, we require a strategy for selecting point pairs that automatically identifies points that are related through the transformations the model aims to learn. Second, existing Lie group operator models have lacked method for determining which regions of the manifold are appropriate for each operator, meaning existing approaches assume that every operator is equally likely to be used at every point on the manifold. This is a flawed assumption because, while many transformations are shared between classes, there are also data variations that are unique to specific data classes. Additionally, in a datasel with several manifolds (each representing a different. class), there is a limited extent to which transformation can be applied without moving a point onto another manifold.
The main contributions of this paper are the development of methods to address, in the context of Lie group operator models, the two critical shortcomings of generative manifold models noted above. Specifically, motivated by finding perceptually similar training samples without transformation labels, we first introduce a point pair selection strategy to learn a manifold representation of natural variations shared across multiple data classes without requiring transformations labels. Second, we develop a method that uses a pretrained classifier (measuring operator is likely to be used while preserving the identity of transformed samples. This approach enables us to analyze the local structure of the data manifold in the context of the learned operators and to describe the invariances of the classifier. We demonstrate the efficacy of these strategies in the context of the Manifold Autocncoder (MAE) model (Connor & Rozell 2020) to learn semantically meaningful transformations on MNIST LeCun et al. 1998), Fashion MNIST
of transformed samples) to learn the local regions where each
Xiao et al., 2017 and CelebA (Liu et al. 2015
2 Background
Manifold Learning Traditional manifold learning models estimate the low-dimensional structure of high- dimensional data by utilizing the property that local neighborhoods on the manifold are approximately linear. Some techniques represent the manifold through a low-dimensional embedding of the data points (Tenenbaum)
Code available at:
2

---

Published in Transactions on Machine Learning Research (03/2023)
et al., 2000; Roweis & Saul 2000 Belkin & Niyogi 2003 Maaten & Hinton 2008
Others learn functions
that map high-dimensional data points to linear Langent planes that represent directions of manifold motion in the local neighborhood around the points (Dollár et al. 2007 [engio & Monperrus 2005; Park et al., 2015). While traditional manifold learning approaches that are applied directly to data points are useful lor understanding low-dimensional data structure, in many cases, the input data space is an inefficient. representation of the data. For example, data in the pixel space suffers from the curse of dimensionality and cannot be smoothly interpolated while maintaining identity (Bengio et al., (005).. Many approaches have used neural networks to learn a low- cow-dimensional latent space in which manifold models can be incorporated. The contractive autoencoder (CAF) estimates manifold tangents by minimizing the Jacobian of the encoder network, encouraging invariance of latent vectors to image space perturbations, Tangent directions can then be estimated through lop singular vectors of the encoder Jacobian (Rifai et. al. 2011c b Kumar el. al. 2017). Several methods estimate interpolated geodesic paths in the latent space of a trained variational autoencoder (VAE) model (Arvanitidis ct al.) 2018) Chen et al., 2018 Shao et al., 2018 Arvanitidis ct al. 2019) and show that geodesic paths result in more natural interpolations. Some extend this approach to learn VAEs with
priors that are estimated using the Riemannian metrics computed in the latent space (Aranitidis et al. 2021
Kalatzis et al. 2020
Lie Group Operators A Lie group is a group of contimuous transformations which also defines a manifold by representing infinitesimal transformations that can be applied to input data (Hoffmaan 1966 Dodwell 1983). Several methods incorporate Lie groups into neural networks to represent data transtormations that are identity-preserving within the model (Cohen & Welling 2014 Cohen et al. 2018). A prevalent strategy is to learn dictionary of Lie group operators that mapped to specific group element through the matrix exponential expm(-) (Rao & Ruderman] 1999 Miao & Rao 2007
Culpepper & Olshausen 2009
Sohl-Dickstein et al.] 20100; Cohen & Welling 2014 Comnor & Rozell 2020 Connor et al., 2021). In these models, each operator V Im, called a transport operator, describes a single direction along the manifold and is parameterized by a single coefficient Gn. Given an initial data point 2, the transport operators define a generative model where transformations can be derived from sampling sparse coefficients 6n ~ Laplace (0, C):
M L
2 expm
Cm
z
+
(1)
m=
where n ~ o.). For analysis on the effect of this sparsity prior on the coefficients, see Appendix
L
The manifold autoencoder (MAE) incorporates the transport operator model into the latent space of an autoencoder to learn a dictionary of operators that represent the global, nonlinear manifold structure in the latent space (Connor & Rozell, 2020). This model has been shown to learn operators effectively with Iransformation supervision, and it will provide the context for dernonstrating the effectiveness of the methods developed in this paper. Specifically, we demonstrate a method for learning natural data variations in the MAE latent space using a new strategy for selecting point pairs without transformation labels, as well as a method for learning the regions of the manifold where each operator likely to be used.
3 Methods
The MAE learns a low-dimensional latent representation of the data by defining an encoder function f X' + Z that maps sigh-dimensional data points E RR to low-dimensional latent vectors Rd and a decoder function g 2 + X' that maps the latent vectors back into the data space (Connor & Rozell 2020). Transport operators V are incorporated into the latent space to learn manifold-based transformations. Before learning the transport operators, the autoencoder is pretrained to extract a latent representation of the data using the traditional autoencoder reconstruction ob objective.
After pretraining, the autoencoder weights are fixed and the operators are trained with the following objective, which encourages the learning of transport operators that generate efficient paths between the latent vectors
3

---

Published in Transactions on Machine Learning Research (03/2023)
Zo and z1 (coinciding with fxx) and f(x1)) that are nearby on the manifold:
M
Cv = 2 11 - expmn
Σ m==1
v
Cm
Zo
+ 2
Σ
v m 112
C/|c||2,
(2)
m
where 7,5 0.
Objective (2) is minimized via alternating minimization scheme, as in the linear sparse coding model Olshausen & Field (1997p. Specifically, at each training iteration, points pairs Xo and x1 are selected in the input space and encoded into the latent space 2g and Z1- Then, coefficients с are inferred between the encoded latent vectors (by minimizing (2) with respect to c) to estimate the best path between Zo and Z1. After inference, the coefficients are fixed and a gradient step is laken to minimize 2 with respect to transport operator weights. Once learned, these operators represent different types of motion that traverse the manifold, and they can be combined to generate natural paths the manifold.
After fitting transport operators to the latent space of fixed autoencoder network, there is a final fine- tuning training phase that updates the network weights and transport operators simultaneously using a joint objective that combines the autoencoder reconstruction loss with Lv- This fine-tuning step has been shown to dramatically improve the quality of learned transformations Connor & Rozell (2020) and addresses the potential mismatch between the data manifold and the learned latent structure by adapting the latent. structure to fit the transport operators learned between the selected training point pairs. We empirically find that breaking up training into these three phases increases the stability with which the transport operators can be learned. Given the context of this MAE model, in the following sections we describe our contributions.
3.1 Unsupervised Transformation Learning
There are many possible strategies for seleeting training point pairs for Lie group operator model, and the choice of a stralegy can dictate the quality of transformations that are learned. For example, point pairs may be selected as random samples from the dataset or from within the same class. These points, however, are likely to be outside of local manifold neighborhoods and may not provide representations of the natural, perceptually smooth variations in the dataset. Alternatively, point pairs can be selected as those with the lowest Fuclidean distance in the pixel or autoencoder latent space. Both of these approaches are limited, however, since nearest neighbors in high dimensions has been noted as ineffective Aggarwal et al. (2001) and since unsupervised autoencoder features may not capture semantic transformations of interest through Euclidean distance. In applications, there may be additional information about the dataset that aid in point pair selection, such as rotation angle, semantic label, or time index in a temporal sequence. However, for most complex datasets, information about the transformations of interest is not available. This necessitates a strategy for selecting training point pairs which can learn natural transformations in complex datasots without requiring additional transformation supervision.
We generalize transport operator training to incorporate an unsupervised learning strategy that can be applied to wide array of datasets. In particular, we select point pairs using perceptual loss metric (Johnson et al. 2016 The perceptual loss measures the distance between feature representations of input images that are extracted from a classifier pretrained on large dataset, like ImageNet. It has been shown that features in the penultimate layer of such pretrained classifiers serve as an effective surrogate for visual similarity (Johnson)
et al. 2016 Yosinski et al. 2015). Given an initial point Xo, we define point pairs in a feature embedding r(x) as:
X argmin ||((x)) r(x)
xED
where, in practice, we randomly select x₃ from the N nearest points to Xy in the classifier feature space.
With the proposed approach, we are capable of training transport operators without labels to select point pairs. Furthermore, we find that the percoptual loss results in transport operators that correspond to smooth, semantically meaningful transformations. We compare the effect of training with the proposed approach versus other point pair strategies in Appendix
4

---

Published in Transactions on Machine Learning Research (03/2023)
3.2 Learning Local Transport Operator Statistics to Encourage Identity Preservation
A trained transport operator model provides dictionary of operators that can be applied globally across the entire data space. This model has flexibility to define local manifold characteristics through the coefficients c which specify the combination of operators to apply to given point. The standard prior on the transport operator coefficients a factorial Laplace distribution parameterized by a single scale parameter Ç, which encourages transformations to be defined by a sparse set of operators:
M II m=1 2
(-1cmL (-c
Pc
exp
(4)
By setting fixed prior coefficients across the entire manifold, there is implicit assumption that the manifold structure is the same over the whole dataset and every operator is equally likely to be applied at every point. However, this is flawed assumption because not all transformations in one class of data are present in all other classes and even within classes there may be regions with different manifold structure. Not capturing the local statistics of transport operator usage could result in transformed points that depart from the data manifold and change their identity.
To address this limitation, we introduce coefficient encoder network which maps latent vectors to scale parameters that specify the Laplace distribution the coefficients at those points in the latent space. The goal of this network is to estimate local coefficient distributions that maximize the identity- preservation of points that are transformed with operators characterized by randomly sampled coefficients:
2 = T=Ty(c)z+G c~,
55)
where To = expm (EM= V mm E Rdddd is the transformation matrix corresponding to the transport operators (parameterized by coefficients c E RM).
Given labeled observations (z, y) and a pretrained classifier r(-), we aim to learn network go(c 2) that outputs distribution parameters from which e can be sampled and applied produce augmented samples ă Rd without changing the classifier output r(2) =y
To maximize the likelihood of obtaining the labeled classifier output for augmented samples, we adapt the concept of consistency regularization from semi-supervised learning. Consistency regularization is applied when training a classifier in a semi-supservised setting in order to ensure that known Seentityprrssevvng augmentations cause only small changes in the classifier output (Bachman et al., 2014). In our context, we use a pretrained classifier and dictionary of transport operators and we want to find a distribution on the coefficients individual imput points that results in consistent classification outputs when To (c) is applied to inputs. The specific objective can be chosen from variety of loss functions that encourage similarity in classifier probability outputs. We specifically minimize the KL-divegence between the label and the label of the transformed output r(2).
Unfortunately, the consistency regularization objective can be trivially minimized by setting c = 0, resulting in an identity transformation and the same classifier output. Therefore, we want to encourage the largest coefficient values possible while maintaining the identity of our initial data point. This motivates the addition of KT-ddvergence regularizer that encourages the distribution (c(z)) to be similar to specified Laplace prior with fixed scale Ç like in Our final objective for training the coefficient encoder network is:
E = (x))(())) + DKL (g (c)))
(6)
A more detailed derivation of this training objective can be found in Appendix A
The addition of this coefficient encoder is a principled way to build identity- directly into our model and to identify local manifold characteristics throughout the latent space. Since this model is trained using outputs from pretrained classifier, the resulting encoded coefficient. scale weights can be informative
5

---

Published in Transactions on Machine Learning Research (03/2023)
about classifier invariances. Specifically, we can quantify which operators are associated with large encoded coefficient scale weights in different parts of the latent space, indicating that the classifier is invariant to those transformations. Additionally, we can identify points or regions of space that have small encoded coefficient scale weights, indicating they are near class boundarics and can undergo only small transformations without changing identity.
3.3
Decreasing Computational Complexity of Coefficient Inference
To compute the gradient for the matrix exponential during training and inference, previous works have used a lincar approximation (Rao Ruderman, 1999 Icarnod the operators factored form (Sohl-Dickstein et al. 2010 or used analytic gradients that are not favorable for parallel computations (Culpepper & Olshausen 2009 Connor & Rozell 2020 Connor et al. 2021 The main computational bottleneck for learning transport operators is the coefficient inference step, where objective 2 is minimized with respect to The & norm term is non-smooth, leading to slower convergence when using subgradients found via automatic differentiation Boyd & Vandenberghe 2004).
In this work, we employ a forward-backwer splitting scheme (Beck Teboulle, 2009) that applies automatic differentiation for the matrix exponential term, with numerical approximations that are more favorable for parallel hardware Bader et al. (2019), and proximal gradients for the l norm (Parikh & Boyd] 2014). We show in Appendix B (with further details on the coefficient inference algorithm) that this leads to significant speedup during training, and allows for scaling of transport operators to complex datasets that were not. possible before.
4 Analysis
For our experiments, we examine the ability of our proposed approaches to enable the MAE model to learn natural transformations from complex datasets where the underlying puretty-rreerving transformations are not easily identifiable. We also highlight the benefits of incorporating the coefficient encoder network that captures the transport operator local usage statistics by encoding coefficient scale values that best maintain the identity of latent vectors.
We work with three datasets: MNTST (LeCnn et al. 1998 Fashion MNIST (iaoo al. 2017 and CelebA (Lu al. 2015). We select MNIST and Fashion MNIST because they contain several classes that share natural tions but they do not have transformation labels. We select CelebA to highlight our ability learn natural transformations in a larger, more complex dataset. As classic dataset used in papers that aim to disentangle datasct fcatures (Higgins et al. 2017 Chen et al. 2016 Hu et al. 2018 Lin et al.) 2019), CelebA contains semantically meaningful natural translormations that may be amenable to qualitative labeling.
In all experiments, we followed the general training procodure put forth previously (Connor & Rozell 2020h by separating the network training into three phases: the autoencoder training phase, the transport operator training phase, and the fine-tuning phase. We select training point pairs that are nearest neighbors in the feature space of the final, pre-logit layer of ResNet-18 (He et al. 2015)) classifier pretrained on ImageNet (Russakovsky et al., 2015). After completely training the MAE, we fix the autoencoder network weights and transport operator weights and train the coefficient encoder network with the objective derived in Section 3.2 Additional details on the datascts, network architectures, and training procedure are available in the Appendix.
We compare against the contractive autoencoder (CAE) RRiaii et al. 2011cp and 8- 8-V (Higgins et al. 2017), two other autocncoder based methods that incorporate data structure into the latent space. The CAE represents another technique that learns a manifold representation in a neural net worw latent space. In their model, the manifold is represented by estimated manifold tangent planes at latent point locations. The 3-VAE learns to discntangle factors of variation along latent dimensions through an increase in the weighted penalty on the KL-divergence term in the VAE objective. We choose these methods as they provide approaches to learn natural dataset variations in the latent. space without transformation labels.

---

Published in Transactions on Machine Learning Research (03/2023)
Manifold Autocncoder
Lighting Angle
Receding Hairlice/ Bangs
Skin Color
B-VAE
Liehtine
Receding Hairline, /aags
Skin Color
CAB
Background
Recedlnng Hairlize Banng
Hair Color
(a)
Line Width
Angled Squeeze
Digit Curvature
a
2
2
2
2
2
2
8
g 2
2
2
a
6
6
6
6
6
6
6
6
6
6
6
6 6
6
6
6
6 8 8
3
8
8
8
8
8
8
8
8
8
%
g
8
8
8
(b)
Width
Sleeve Length
Color
11 11
Figure Paths generated by applying a subset of learned transport operators on three datasets. In cach figure, images in the middle column of the image block are the reconstructed inputs and images to the right and left images decoded from transformed latent. vectors in positive and negative directions, respectively. (a) Comparing the transformations generated by three learned transport operators to transformations generated by 3-VAE and CAE. The transport operators learn semantically mcaningful transformations similar to the disentangled 3-VAE representation, while maintaining a higher resolution in image oulputs. (b-c) Transport operators learned using the perceptual point pair selection strategy generate natural transformations on both MNIST and Fashion MNIST.
4.1 Learning Natural Data Variations
First we show how well our proposed methods enable the MAE model to learn natural data variations in the dataset when selecting point pairs using the perceptual point pair selection strategy and we highlight the usefulness of a nonlinear manifold model for generating latent space paths. Fig, [ shows the paths generated by transport operators trained in this model. The image in the middle column in each block of images is the ructedd version of the input image Xg- The images to the left and right of the middle show the reconstructod images generated by an individual lcarned operator applied the encoded latent vector Zo: c -Ney ... No. We see an individual operator generates similar transformation across multiple inputs, and in many cases, the transformations induced by the transport operators are semantically meaningful. We also show that the perceptual point pair selection strategy is effective over a range of datascts.
%. =
In Fig. D we compare the transport operator paths to paths generated by the CAE and the 8-VAE. We compare with the CAE as it also incorporates manifold structure into the latent space, although it uses a tangent approximation that limits the size of reasonable transformations. On the other hand, we choose the 3-VAE as a ubiquitous technique for disentangling factors of variation across Euclidean coordinates. Similarly, the MAE learns disentangled factors of variation via the non-linear paths spanned by the transport operators.
7

---

Published in Transactions on Machine Learning Research (03/2023)
Thar
Pade Scin
Coeficiintt
Contideet
(a)
(b)
Norbil
Procuttoon
Mouth Skgniy Open
Euegasso
No Deard
0.15
Conticient Value
(c)
(d)
Figure 2: Paths generated by the application of two Iearned transport operators with the associated attribute classifier probability outputs for the transformed images. Our model learns operators that correspond to meaningful CelebA dataset attributes like (a) hair color (b) pale skin (c) smiling (d) sunglasses/
The CAE-generated paths represent the directions of motion on the tangent planes estimated at individual points. The B- B-VAE paths are generated by varying the value of one latent dimension, while the others remain fixed. While our method and the 3-VAE learn several qualitatively similar transformations, our method is capable of doing so without significantly sacrificing reconstruction performance. In the Appendix, we include more examples of the transformations generated by each learned transport operator across each dataset.
While many of the operators can be assigned a semantic label through qualitative visual inspection, the CelebA dataset has attribute labels for the images which enable a quantitative analysis of the connection between learned transport operators and dataset attributes. To classify attributes, we fine-tune a ResNet-18 pretrained on ImtgeNett with 16 classification heads for attributes including smile, beard, hair color, and pale skin Mao et al. 2020). With this classifier, we are able to quantify the degree with which specific transport operators correspond to dataset attributes. Fig. 2 shows the classification outputs of the attribute classifier for specific transport operators. Our model learns operators that vary hair color, skin paleness, and several others attributes, as shown in Appendix L
One benefit of Lie group operator models over other manifold-base models is that the learned transport operators provide natural way for estimating nonlinear manifold paths between points in the latent space. To highlight this benefit, in Fig. 3 we compare the MAB-based interpolated and extrapolated paths to those estimated using CAF and 3-VAF, as well as linear paths in our original autoencoder before fine-tuning it to fit the manifold structure defined by the learned transport operators. With each method, we estimate a path between the latent vectors associated with two points Zu and Z1, interpolate between the two points, and extrapolate beyond 2, to extend the path. For MAE, this path is estimated by inferring the set of transport operator coefficients c" between the latent points and then generating the path: z, expm (EM-) Wm Zo-
When the path multiplier t is between 0 and 1 that indicates interpolation and path multipliers beyond indicate extrapolation. To estimate paths in AE, CAE and VAE,, we compute the vector difference between 20 and 21 and interpolate and extrapolated on that vector direction. In these figures, the first block of images corresponds to the interpolated path with the selected final point X1 surrounded in an orange box. The second block of images corresponds to the extrapolated paths.
For quantifying the identity preservation of each transformation, we input each generated image into a pretrained classifier and plot the probability of the class label associated with the inputs. All four methods perform interpolation effectively but our trained model estimates the extrapolaeed paths more accurately. Fig. A shows how that accuracy varies during extrapolation sequences for 4000 samples. The MAE better at generating extrapolated outputs that maintain class identity. The lower classífication accuracy in fashion MNIST is due to both a more challenging dataset and the lower resolution of autoencoder image outputs when compared to input images.
8

---

Published in Transactions on Machine Learning Research (03/2023)
4
2 I Z
4
4
8 2
8
2
2
X Z 2
I
( B-VAE CAEE
8
2
2
2
Cornect Elass
Probughtt
MAT
MAC
0.5
0.5
D-S
Path Mutiiperr
Puth Muckipller
Fath Mustippier
Path Musiiplerr
(a)
3 VVAE
Cornect Cass Trubabiyy
0.5
CAB
Path Mutigleer
Poth Hudipler
Pach Mudiplier
Path Muciiplier
(b)
Figure 3: Identity preservation of transformed paths as quantified by a pretrained classifier output. In the figures on the top, the first block of images corresponds to the interpolated path with selected final point x surrounded in an orange box. The second block of images corresponds to the extrapolated paths. Below the images are plots of the probability of the class label associated with the inputs zo and 21. A path multiplier between 0 and indicates interpolation and path multipliers beyond indicate extrapolation. (a) MNIST (b)
Fashion MNIST
MNIST
Fashion MNIST
Classification Accuracy
Classifiction Accurapy
0.8
0.8
0.8
MAE
0.6
MAE
AE
CAE
CAE
0.4
0.4
0
1
2
3
0
2
3
Path Multiplier
Path Multiplier
(a)
(b)
Figure 4: The average accuracy of the classifier output for images at each point along the interpola- tion/exton extrapooataoion sequence. When the path multiplier is between and that indicates interpolation and path multipliers beyond 1 indicate extrapolation. (a) MNIST (b) Fashion MNIST 4.2 Learning Local Manifold Structure
A After the MAE is trained, we have a dictionary of transport operators that describe manifold transformations and a network with a latent space that is adapted to the manifold. We then train the coefficient encoder from Scction 3.2 to estimate the cocfficient scale weights
function of points in the latent space. To visualize how the use of the transport operators varies over the latent space, we generate Isomap embedding (Tenenbaum) et al. 2000) of latent vectors and color each point by the encoded scale parameter for coefficients associated with each of the transport operators. Fig. 5a and 5b shows these embeddings for MNIST and Fashion MNIST, respectively. Each operator has regions of the latent space where their use is concentrated.
By training the coefficient encoder to maximize the similarity between the classification outputs for an input: sample and for a transformed version of that sample, the network aids in identifying which transport operators can be applied to inputs in regions of the data space without changing the identity of the input. This helps significantly with data augmentation where the goal is to create new samples with in-class variations. To highlight this benefit, in Fig. b we show samples augmented by applying transport operators with randomly sampled coefficients to an input latent vector. In each block of images, the leftmost image (in a green box) is the input image and the images to the right are decoded augmentations. The top row shows samples augmented with transport operators controlled by coefficients sampled from Laplace distributions with encoded coefficient scale weights. The bottom row shows samples augmented with transport operators controlled by coefficients sampled from Laplace distributions with fixed scale parameter. The augmentations
9

---

Published in Transactions on Machine Learning Research (03/2023)
Operator
Operator
Operator
Openakor
Operator
Operatoe
Opanaice
(a)
(b)
Escoded Corficirr Scall
Ereezed Conficier Scale Faevico Whst
Openter
2 &
GM
Opeaszel
100
0 10000
1000 12228
e 0 cl
4
4
4
Trampoa Numbdrr
TurnoCT Openat Haundor
(c)
(d)
Figure 5: Visualizations of the encoded coefficient scale weights for (a, c) MNIST and (b, d) Fashion MNIST. Figure best viewed zoomed in. (a, b): Isomap embedding of the latent vectors with input images overlaid. The scatter plots on the right show the same Isomap embedding colored by the encoded coefficient scale weights for several operators (ycllow indicates large scale weights and blue small scale weights). We see operators whose for each class each transport operator. High scale weights for given operator (yellow) indicate it be applied to given class without casily changing identity. The images on the right show examples of the operators applied to classes with high encoded scale weights (in the top yellow boxes) and classes with low encoded scale weights (in the bottom blue boxes). The examples with low coefficient scale weights change classes more easily than other examples.
localized in regions of the manifold space. (c, d): The average coeflicient scale weights
Enoocle Scale Weights
9
9
9
9
9 9
9 9
8
8
8
8 8
8
Fixed Scale Weights
9 9
9
9
{
8
8
8
8
8
Figure 6: Examples of samples generated by transport operators using coefficients sampled with encoded scale weights (top row) and with fixed scale weight (bottom row). Images in the green box are the input. images and the remaining images in each row are transformed outputs.
in these figures differ from previous ones where augmentations were generated using coefficients interpolated within fixed range, rather than sampled. While both strategies generate realistic variations of the data, using the encoded scale weights improves preettyy-peservationn of the transformed output. The augmentations with the encoded scale weights are better at maintaining the identity of the sampled points.
Because the coefficient encoder uses a pretrained classifier to determine the coefficient distributions in the latent space, the resulting encoded scale weights can inform us about the types manifold transformations the classifier invariant to. Fig. 5c and 5d shows the average coefficient scale weights for each class (rows) and each transport operator (columns) for MNIST and Fashion MNIST, respectively. From this, one can identify classes for which the classifier is both sensitive and robust to transformations, represented by small and large scale weights respectively. We can also examine which classes share the use of the same transformations The images to the right of in Fig. 5c show transport operators being applicd to samples with high encoded scale weights (in a yellow box) and samples with low encoded scale weights (in a blue box). It is interesting transformations that are better suited to some classes than others. For instance, operator 3 in Fig. [c increases the curve the bottom of digits. This is a natural transformation for classes 3, 5, and which all have higher coefficient scale weights for this operator, but when this is applied to 1, that makes it look like an 8.
to examine the characteristics
10

---

Published in Transactions on Machine Learning Research (03/2023)
Table Test Accuracy training a LeNet-5 on MNIST with 10 examples per class with different augmentations applied each training step. Among augmentation strategics, transport operator augmentations sampled from the coefficient encoder has the highest accuracy.
Model
Test Accuracy
No Augmentations
81.49 3.35%
Modified RandAugment
87.94 3.34% 80.40 + 4.02%
Elastic Distortion
Transport Operator w/ Fixed Prior
70.35 + 1.70%
Transport Operator w/ Coefficient Encoder
93.57 + 0.67%
4.3 Applying Augmentations for MNIST Classification
To test how well the learned operators represent data variations and to examine the potential for using transport operators in downstream tasks, we enrich limited MNIST dataset using the learned operators and compare against other common augmentation strategies. We train a LeNet-5 classifier Lecun et al. (1998) using 10 examples per class, sampled with replacement cach training step with a random augmentation applied. This experimental set up was intentionally selected make MNIST more challenging demonstrate the potential benefit of augmentations. We compare transport operator- baseat augmentations against two common image space augmentations, modified version of RandAAgment Cubuk et al. (2019b) and clastic distortion, as described in Simard et al. (2003). To test the effectiveness of our learned coefficient encoder, we compare against transport operator augmentations sampled either from fixed prior or a coefficient encoder trained following the methodology in Section 3.2] To prevent overfitting due to limited training data, we apply a weight decay of le-4 and carly stopping after 10, 000 iterations, measuring accuracy over the full test dataset over multiple trials.
Cubuk
Results can be found in Table where it can be seen that augmentations generated using transport operators controlled by coefficients sampled from the learned coefficient encoder produce the highest classification accuracy. We observe higher variance in the performance of RandAugment and elastic distortion, which apply augmentations that can alter the idcntity of the image. We note that the cocfficient encoder uses supervision to learn the exlent with which the transport operators can be applied, as opposed the fully unsupervised comparison methods. However, this experiment demonstrates that the learned transport operators effectively represent the data variations and the coefficient encoder enables effective augmentations. Future work extend the coefficient encoder to settings with limited (e.g., semi-supervised) no labels.
5 Conclusion
In this work, we develop methods to improve the effectiveness and utility of Lie group operator models for learning manifold representations of datasets with complex transformations that cannot be labeled. We do this by introducing perceptual point pair selection strategy for training operators and by developing a method that uses a pretrained classifier to learn local regions where operators are likely to be used while preserving the identity of transformed samples. We demonstrate the efficacy of our approach in learning natural dataset variations with the MAE. While this is a powerful model, users should be mindful of the biases that are introduced through training with specific supervision methods when drawing conclusions about learned data transformations. This work presents a promising technique for learning representations of natural data variations that can improve model robustness. In future work, we address some limitations by expanding this work to consider methods for localizing manifold structure with limited or no class labels and improving the reconstruction fidelity by applying the model within more complex generative model embeddings.
Since Augment requires 3 image channels, repcat grayscale MNIST images times the channel dimeusion, apply Rugment with N=2 and M=6, and select random channel training example. refer this "Madified Angment".
11

---

Published in Transactions on Machine Learning Research (03/2023)
References
Charu C. Aggarwal, Alexander Hinneburg, and Daniel A. Keim. On the surprising behavior of distance metrics in high dimensional space. In Jan Van den Bussche and Victor Vianu (eds.), Database Theory ICDT 2001, pp. 420-434, Berlin, Heidelbery, 2001. Springer Berlin Heidelberg. ISBN 978-3-540-44503-66
Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg, Latent space oddity: on the curvature of deep generative models. In International Conference on Learning Representations, 2018.
Georgios Arvanitidis, Soren Hauberg, Philipp Hennig, and Michael Schober. Fast and robust shortest paths on manifolds learned from data. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 1506-1515. PMLR, 2019.
Georgios Arvanitidis, Bogdan Georgiev, and Bernhard Schölkopf. A prior-based approximate latent riemannian metric. arXiv preprint arXiv:2103.06 052900 2021.
Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. ar preprint arXiv:1412.4664, 2014.
Philipp Bader, Sergio Blanes, and Fernando Casas. Computing the matrix exponential with an optimized taylor polynomial approximation. Mathematics, 7(12), 2019. ISSN 2227-7390. doi: m0.3390/aath7221174..
URL hwwp htttps:/wwww.mdp..com com mt0p1...cm/222117737//1111111111
Amir Beck and Marc Teboulle. A Fast Iterative Shrinkage-Thhnee Thresholding Algorithm for Linear Inverse Problems.
SIAM Journal on Imaging Sciences, 2(1): 202, Jamuary 2009. ISSN 1936-4954. doi: 10.1137/080716512.
URL http:://ptbs...../////////// siam. http:pppi........orr.r../o//e//e wwwp://////ooo..//// 011.//00011117/007166611 00000//....././/////010010077720
Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation.
Neural computation, 15(6):1373 1396, 2003.
Yoshua Bengio and Martin Monperrus. Non-local manifold tangent learning. Advances in Neural Information Processing Systems, 17:129 136, 2005.
Yoshua Bengio, Olivier Delalleau, and Nicolas Le Roux. The curse of dimensionality for local kernel machines. Techn. Rep, 1258:12, 2005.
Stephen Boyd and Lieven Vandenberghe. Conver optimization. Cambridge University Press, Cambridge, UK New York, 2004. ISBN 978-0-521-83378-3.
Nutan Chen, Alexej Klushyn, Richard Kurle, Xueyan Jiang, Justin Bayer, and Patrick Smagt. Metrics for deep generative models. In International Conference on Artificial Intelligence and Statistics, pp. 1540-1550. PMLR, 2018.
Chen, Yan Duan, Rein Houthooft, John Schulman, Tlya Sulskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets, 2016.
Taco Cohen and Max Welling. Learning the irreducible representations of commutative lie groups. Tn International Conference on Machine Learning, pp. 1755 1763. PMLR, 2014.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pp. 2990 2999. PMLR, 2016.
Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling, Spherical cmns. In International Conference on Learning Representations, 2018.
Marissa Connor and Christopher Rozell. Representing closed transformation paths in encoded network latent space. In AAAI Conference on Artificial Intelligence, volume 34, pp. 3666-3675, 2020.
Marissa Connor, Gregory Canal, and Christopher Rozell. Variational autoencoder with learned latent structure. In International Conference on Artificial Intelligence and Statistics, pp. 2359-33592200 2369. PMLR, 2021.
12

---

Published in Transactions on Machine Learning Research (03/2023)
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVP Conference Computer Vision and Pottern Recognition, pp. 113 123, 2019a.
Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. Randaugment: Practical automated data augmentation with reduced search space, 2019b. URL httpss///arriv.oo/ab/////////// 13719
Benjamin J Culpepper and Bruno A Olshausen. Learning transport operators for image manifolds. In Advances in Newral Information Processing Systems, pp. 423-431, 2009.
Peter C Dodwell. The lie transformation group model of visual perception. Perceplion & Psychophysics, 34 (1):1- 16, 1983.
Piotr Dollár, Vincent Rabaud, and Serge Belongie. Learning traverse image manifolds. In Advonces in Neural Information Processing Systems, pp. 361-368, 2007.
Luca Falorsi, Pim de Haan, Tim R Davidson, and Patrick Forré. Reparameterizing distributions on lie groups. In The 22nd International Conference on Artificial Intelligence and Stalistics, pp. 3244- -223. PMLR, 2019.
Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Society, 29(4):983 1049, 2016.
Manuel Gil. On Rényi divergence measures for continuous alphabet sources. PhD) thesis, Citeseer, 2011.
Søren Hauberg, Oren Freifeld, Anders Boesen Lindbo Larsen, John Fisher, and Lars Hansen. Dreaming more data: Class-dependent distributions over diffeomorphisms for learned data augmentation. In Artificial Intelligence and Statistics, pp.
PMLR, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. ICLR, 2(5):6, 2017.
Daniel Ho, Eric Liang, Xi Chen, Ion Stoica, and Pieter Abbeel. Population based augmentation: Efficient learning of augmentation policy schedules. In International Conference Machine Learning, pp. 2731-2741. PMLR, 2019.
William C Hoffman. The lie algebra of visual perception. Journal of mathematical Psychology, 3(1):65 98, 1966.
Qiyang Hu, Attila Szabó, Tiziano Portenier, Matthias Zwicker, and Paolo Favaro. Disentangling factors of variation by mixing them, 2018.
Mark Ibrahim, Diane Bouchacourt, and Ari Morcos. Robust self-supervised learning with lie groups, 2022. URL https://arxiv..rrt httpg//ab///2...oo./// 13356
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. European conference on computer vision, pp. 694-711. Springer, 2016.
Dimitris Kalatzis, David Fklund, Georgios Arvanitidis, and Søren Hauberg. Variational autoencoders with riemannian brownian motion priors. In International Conference Machine Learning, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference for Learning Representations, 2015.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. In International Conference on Learning Representations, 2013.
13

---

Published in Transactions on Machine Learning Research (03/2023)
Abhishek Kumar, Prasanna Sattigeri, and P Thomas Fletcher. Semi-supervised learning with gans: Manifold invariance with improved inference. Advances in Neural Information. Processing Systems, 2017.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient- based learning applied document recognition. Proceedings of the IEFE, 86(11):2278 2222222-222, 1998. doi: 10.1109/5.7220 5.7299..
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient- learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 222222244,, 1998.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. Advances in Neural Information Processing Systems, 2019.
Jianxin Lin, Zhibo Chen, Yingce Xia, Sen Liu, Tao Qin, and Jiebo Luo. Exploring explicit domain supervision for latent space disentanglement in unpaired image-to-image translation, 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Laurens van der Maaten and Geoffrey Hinton. Visualizing Data using t-SNE. Journal of Machine Learning Research, 1:1-48, 2008. URL http::/www. cs. toronto. www.oooo...uuooootttttooooooeeo. http://hintoooabssstseeaee pdf
Longbiao Mao, Yan Yan, Jing- Hao Xue, and Hanzi Wang. Deep multi-task multi-label enn for effective facial attribute classification, 2020.
Xu Miao and Rajesh PN Rao. Learning the lie groups of visual invariance. Newral compulation, 19(10): 2007.
Bruno A Olshausen and David J Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research, 32333333311 1325,, 1997.
PACE. Partnership for an Advanced Computing Environment (PACE), 2017. URL http://www.pace
gatech.edu
Neal Parikh and Stephen Boyd. Proximal algorithms. Found. Trends Optim., 1(3):127-233) January 2014. ISSN 2167-3858. doi: 10.1561/2400000003. URL http:///oooooo......oo//eo 1511/40000020
Mijung Park, Wittawat Jitkrittum, Ahmad Qamar, Zoltán Szabó, Lars Buesing, and Maneesh Sahani. Bayesian manifold learning: The locally linear latent variable model (Il-vmm. In Advances in Newral Information Processing Systems, pp. 154-162, 2015.
Rajesh PN Rao and Daniel L Ruderman. Learning lie groups for invariant visual perception. In Advances in neural information processing systems, pp. 810-816, 1999.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows, 2016.
Danilo Jimonez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generalive models. In International Conference on Machine Leurning, pp. 1278 1286. PMLR, 2014.
Salah Rifai, Yann N Dauphin, Pascal Vincent, Yoshua Bengio, and Xavier Muller. The manifold tangent. classifier. Advances in Neural Information Processing Systems, 24:2294 24:2294-2302, 2011a.
Salah Rifai, Grégoire Mesnil, Pascal Vincent, Xavier Muller, Yoshua Bengio, Yann Dauphin, and Xavier Glorot. Higher order contractive auto-cncoder. In Joint European conference on machine learning and knowledge discovery in dalabases, pp. 645 660. Springer, 2011b.
Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive auto-cncoders: Explicit invariance during feature extraction. In International Conference on Machine Leurning, 2011c.
Sam T Roweis and Lawrence Saul. Nonlinear dimensionality reduction by locally linear embedding. Science, 290(5500) :2323 2326, 2000.
14

---

Published in Transactions on Machine Learning Research (03/2023)
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 222211221222211111 2015.
Hang Shao, Abhishek Kumar, and P Thomas Fletcher. The riemannian geometry of deep generative models. In Procecdings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pp- 315 315-323, 2018.
P.Y. Simard, D. Steinkraus, and J.C. Platt. Best practices for convolutional neural networks applied visual document. analysis. In Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings., pp. 958-963, 2003. doi: 10.1 N0.1109/TCDAR:2000 /TTC//DAA..0...227777.
Jascha Sohl-Dickstein, Ching Ming Wang, and Bruno A Olshausen. An unsupervised algorithm for learning lie group transformations. arXiv preprint 1x1::1001.1027, 2010.
Steven H. Strogatz. Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chemistry and Engineering. Westview Press, 2000.
Joshua B Tenenbaum, Vin De Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. Science, 290(5500):2319 2323, 2000.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. arXie preprint arXiv:1506. 06379, 2015.
15

---

Published in Transactions on Machine Learning Research (03/2023)
A
Coefficient Encoder Details
E
V
2
2
9 (c(z)
c
Tv (c)c
r(.)
y
Figure 7: System diagram for learning local coefficient statistics.
Our motivation in training an encoder to learn the coefficient statistics is to understand the regions of the manifold in which specific transport operators are applicable and the invariances of a pretrained classifier the learned transport operator-indueed transformations. In this section, we will outline derivation of our coefficient encoder training objective from variational inference perspective by learning variational posterior for the coefficients using deep neural network (Kingmma & Welling 2013 Rezende et al. 2014). Given obscrvations (z, Vi) for i = I,..., N,, and pre-trained classifier r(-), we aim to learn a distribution (c(c/z) from which we can sample to define the transformation Tv (c) that can be applied to z while maintaining class identity. In other words, we want the augmented 2 = Tv (c)z + 6 to result in r(2) =
To encourage identity-preservation of vectors transformed with transport operators that are controlled by sampled coefficients, we maximize the likelihood of class output for an observation under the system model diagrammed in Figure 7 This system connects the observed latent vector to output through transformation defined by We follow the derivation used in Rezende & Mohamed (Rezende & Mohamed 2016 which introduces the variational posterior to estimate the log likelihood of the data.
Consider a parameterized distribution for the conditional likelihood of our observations:
log po (3/ z) = log (e) [po(y] c, z)]
(7)
= log j pc(c)po(y z)dc
(8)
(c
log
2o(c 2)
l z) poly c, z)de
(9)
(c(c)
> co(c log log
[s(c 2)) Polyy (2.2) de
(10)
Ega [log Po(8
c, - DKL (go(c) (c)p Egg [log E.Po(y c, c, DeL (c z)lpe(c))-
(11)
(12)
In 7 and 8 we marginalize over the coefficients as required under our system model in Figure 7
10 we lower bound the conditional likelihood with a variational lower bound derived from Jensen's ss the reparameterization trick to define Z deterministic function of c, z, and the
incquality. Finally,
parameter-free random variable in [2).
In order to sample from (c z) when computing the expectations, we first use the reparameterization trick (Connor et al. 2021 Kingma Welling 2013: Rezende et al. 2014) to define the sampled cocfficients as a
function of a uniform random variable
Unif (- 1)
é - 46 u,, 2) - hz((z)
sgn(u) log(1 2|ul),
(13)
where Io is a defined mapping from a uniform distribution Laplace distribution and the Laplace scale parameters defined by the output of the coefficient. encoder hs that aim to learn.
16

---

Published in Transactions on Machine Learning Research (03/2023)
Using the reparamcterization of è and 2, we can define the expectation:
Egg [log ExPo(y c, c, = Eu [log (y é
(14)
po (v)elo) = -6 2), (0)
log
(15)
JK
Our pre-trained classifier defines which allows us to specify final objective using of r()). Using a KL- divergence as the likelihood function between the ground truth labels ys and the classifier output for the augmented inputs r(z) and simplifying the model by setting to O, we get:
logro 2)
Σ j=1
DEL y3 r(()))) DRL (1a(c 2)(p((c)),
(16)
where in practice we use a single sample = 1, resulting in the objective in (6) which we want to minimize.
The KL divergence between the qo(c| z) and pc(c) has closed form expression Gil 2011):
S -1 hz(z)
DKL (9olc ())pp(()) lz)( log(s) +
(17)
B Coefficient Inference Details
Time Taken for Coefficient Inference
Batch Size 100, Trials 30
100
Conj. Grad. (CPU)
Prox. Grad, (CPU)
Subgrad. (CPU)
Time Taken (secs)
Prox. Grad. (GPU)
Subgrad. (GPU)
60
40
20
8.08 MBC
2.73 sec
0
Optimization Method
Figure 8: Comparison in time taken for various coefficient inference algorithms on fully trained transport operators. Each inference trial is performed over random batches of 100 point pairs drawn from the CelebA dataset (with a latent dimension of 32). 30 trials are run with the same co and tolerance. We used the proximal gradient (GPU) method.
In order perform coefficient inference, previous Lie group operator methods (Connor & Rozell 2020
Connor et al. 2021 use the analytic gradient proposed in (Culpepper & Olshausen 2009 with conjugate gradient descent solver. This method requires an eigenvalue decomposition for computing the gradient which is not favorable for parallel computations. Alternatively, we use the PyTorch implementation of the matrix exponential that allows for automatic differentiation. Furthermore, to handle the non-smooth & norm in objective 2 we apply a proximal gradient step in a forward-backee splitting scheme (Beck & Teboulle,
17

---

Published in Transactions on Machine Learning Research (03/2023)
2009 For the & norm, the proximal gradient has known, closed-form solution in the sattthhresholdig
function (Parikh & Boyd 2014]:
Tx (c) = sign(c) * max (icl - 2.0)
(18)
Vf(e) be a numerical approximation to the gradient of the & term, found through automatic differentiation. Given initial coefficient values co drawn from an isotropic Gaussian with variance 4 x 10-4, our gradient descent step is:
Let Ve3 31 - expm (EM= mC
Zo
c++1 = Tean (c+- Or (((a))
(19)
These steps are iterated upon until either a max iteralion count is reached, or the change in coefficients, ||c++1 - - falls below some threshold. For all experiments we use a max iteration count of 800 and a tolerance of 1 x 10-3, 5. For our step-size, we use Og (0.985)* with dg = 1 x 10-2, We experimented with different acceleration methods (Kingma & Ba, 2015 Beck Teboulle 2009), and found that in many cases they resulted in worse performance.
The main parameter to select for inference is the sparsity-inducing parameter the coefficient prior, Ç. This value is sensitive to the dataset, latent dimension, and y ayper-parameter value. In practice, we fix a y value and pick Ç to be as high as possible, inducing sparsity in the inferred coefficients, while maintaining good roconstruction performance of 2 If Ç is too large, that will result in all cocfficients going to zero, preventing proper reconstructive performance. On the other hand, setting ( to be too small will result in all operators being used to represent the transformation between each point pair, preventing any meaningful structure from being learned during training. Parameters used in specific training runs are included in their respective Appendix sections.
When compared against the coefficient inference implementation from (Connor & Rozell 2020 we perform inference for a batch of 100 samples in an average of 1.67 seconds over 100 trials whereas the inference from (Connor & Rozell 2020) took an average of 101.01 seconds. We also compare the speed-up over a baseline that strictly uses automatic differentiation (subgradients) and compare the benefits from moving to GPU hardware in Figure g We use the proximal gradient (GPU) method from this figure. Experiments were run a machine with an Intel i7-6700 CPU with 4.00 GHz and Nvidia TITAN RTX.
C Training Strategy
In all experiments, we follow the general training procedure put forth previously (Connor & Rozell 2020 We train the MAE with three training phases: the autoencoder training phase, the transport operator training phase, and the fine-tuning phase. During the autoencoder training phase, the network weights are updated using a reconstruction loss objective: EAE = ||x - 211
During the transport operator training phaso, the network weights are fixed and the transport operators are trained between pains of points using the objective 2 Pairs of images 2o, 31 are chosen using the perceptual
point pair selection strategy described in Section 3.1 The images are then encoded into the latent space 2p, 21. For each batch, the first step is to infer the coefficients between all pairs of latent vectors. Coefficient inference is best performed when the entries of the latent vectors are close to the range [-1, Because of this, we define a scale factor that can be applied to encoded latent vectors to reduce the magnitude of their entries prior to performing coefficient inference. In practice, we inspect the latent vector magnitudes after the autocncoder training phase and choose a scale that will adjust the magnitudes of the latent vector entries to be in the range [-1, 1]. This does not have to be a precise range for the latent vector magnitudes but instead is a practical guideline. Coefficient inference is performed on the scaled latent vectors as described in Sections 3.3 and Appendix B After the coefficients are inferred for batch, the weights on the dictionary elements are updated. This phase of training is performed until the loss values reach a platean and the dictionary magnitudes platean.
18

---

Published in Transactions on Machine Learning Research (03/2023)
The fine-tuning training phase begins after the transport operator training phase is complete. In this phase, both the network weights and the transport operator weights are updated using the joint objective:
E ==x(1xx) - 2 + ||=1- - x111)) + + (1- A) Ey.
(20)
where Ey is defined in 2 Using this objective, we altornate between taking steps on the transport operator weights while the network weights are fixed and laking steps on the network weights while the transport operator weights are fixed. Additionally, during the fine-tuning phase we incorporate occasional steps in which we update the network weights using only the roconstruction loss to ensure effective image reconstruction.
In most cases, it is necessary to reduce the 7 parameter in front of the Frobenius norm dictionary regularizer prior to fine-tuning the dictionary magnitudes will reduce to zero. We report the use for transport operator training and fine-tuning in the experimental details sections below. It may also be necessary to decrease the network learning rate during fine-tuning.
The coefficient encoder training requires a network that is trained to classify data from our selected dataset. We train this classifier using training data from a given dataset. For datasets with worse autoencoder reconstruction quality, we train the classifier in the latent space. Otherwise, we train the classifier on images in the data space. With the MAE and classifier trained, train the coefficient encoder network following the strategy described in Section 3.2 and Appendix A
In order to train the CAE, we use the same autoencoder architecture used with the MAE with the addition of a Frobenius norm regularizer on the encoder Jacobian, weighted by a selected value (different from the X in 20 The Jacobian is computed using PyTTrch automatic differentiation. We find the Jacobian norm decreases to the same value irrespective of our choice X, leading us to choose the same architectures outlined in (Higgins et al.) 2017) with B = 10 for CelebA and 3 for MNIST and Fashion MNIST. We find that setting any higher results in poor reconstructive performance. Scripts for training both comparison methods are included in the code repository.
For the 3-VAE we use
tuning for all experiments was performed on the Georgia Tech Partnership for Advanced Computing Environment (PACE) clusters (PACE, 2017). Experiments were performed using a Nvidia Quadro RIX 6000. Runs training the CAE, and 3- 8-VAE on CelebA were all run on a separate machine with a Nvidia TITAN RTX.
Parameter Selection
The MAE model has several hyperparameters that must be tuned and we will provide guidance to determining ideal parameter values for our experiments and future experiments. First, we will describe some signs to look out for to identify if run is succeeding or failing. One indicator that compute the transport operator difference which is:
1 2 21 expm
M Σ
2
1
M Σ
2
Fraaitt
Cy
Z0
21 expm
Cy
2.
(21)
2
where L. the dictionary after the gradient step is taken. A gradient step should decrease the transport operator objective meaning the Evait value should be positive for effective step. Tf there are many gradient steps that result in negative Evais values, that indicates that the parameters are not optimal or that the learning rate is too large. This is inportant metric to observe during fine-tuning to determine whether to select smaller or smaller network learning rate.
Signs of failure of a training run with selected training paramcters:
All operator magnitudes reduce towards zero.
All inferred coefficients between point pairs are zero.
19

---

Published in Transactions on Machine Learning Research (03/2023)
Most of the operators generate transformation paths with latent values that increase quickly to infinity.
Many steps have negative values for Evaint
The operator magnitudes increase which results in unstable training steps with NaN values in the computed objective.
Dictionary regularizer parameter The dictionary regularizer parameter y is the weight the Frobenius norm term in 2 This objective term serves two purposes. First, it balances the effect of the coefficient sparsity regularizer with the parameter Ç. If Ç is large, the spansity regularizer encourages small coefficient magnitudes and one way to achieve that while still effectively inferring paths is to increase the magnitude of the operators. The Frobenius norm term must have a large enough influence to counterbalance this force or the operators will increase in magnitude to the point of being unstable. If run becoming unstable, we recommend decreasing or increasing 7-
The second purpose of the dictionary regularizer term is to identify which operators are necessary to represent the transformations on the data manifold. If an operator is not being used to represent a transformation between 2n and 2 then the dietionary regularizer reduces its magnitude to zero. Therefore, during training we are able to estimate the model order based on how many dictionary elements remain non-zero. In our tests we vary y between 2 x 10-8 and x 10-4, If y is too small, it will not counterbalance the coefficient sparsity term and the dictionaries will grow unstable magnitudes. If y is too large it will reduce the magnitude of all operators to zero. During the fine-tuning steps, we have often found it necossary to reduce the y because, with a larger 7, both the operator magnitudes and the latent vector magnitudes can decrease substantially, which leads to an ineffective manifold model.
Coefficient sparsity parameter The coefficient sparsity parameter ( in (2) controls the sparsity of the coefficients that are used to estimate paths between Zu and 21. We run tests with values of between 0.005 and From dataset to dataset the ideal value varies. If Ç values are too small then all the operators are usod represent all the paths between point pairs. The Ç should be increased so fewer than M coefficients are used for each inferred path. When & is too large, all the coefficients go to zero during inference. This means there is no path inferred between Zo and 21 because of overweighting the sparsity constraint.
Number of dictionary elements As mentioned above, the dictionary regularizer acts as model order selection tool so our strategy for selecting number of dictionary elements M is to increase the number of dictionary clements until some of their magnitudes begin reducing to zero during training. This indicates that some of the operators are not necessary for representing transformations.
Latent dimension The latent dimension of the autoencoder is selected to ensure quality reconstructed image outputs.
Relative weight of reconstruction and transport operator objectives the weight of the reconstruction term relative to the transport operator term. We observe good performance of the model for À between 0.5 and 0.75.
The parameter À determines
E Analysis on the Effect of Coefficient Sparsity
In our work, we impose sparsity in coefficients e that describe the manifold transformation between a point pair in the latent space. Sparse coding has been used to encourage learned representations that are statistically independent. Olshausen & Field 1997). In our context, this means that we expect each transport operator to transform a single semantic attribute in an image.
We investigate the effects of the sparsity penalty by comparing to transport operators trained using dense coefficients (i.e., all coefficients non-zero for every point pair). To train dense coefficient. model, we follow the exact methodology from the main text CelebA, replacing coefficient inference step with the forward
20

---

Published in Transactions on Machine Learning Research (03/2023)
Sparse Coefficients
(b) Dense Coefficients
Figure 9: Qualitative comparison of reconstructing transport operator transformation paths when trained with (a) sparse coefficients and (b) dense coefficients. For each row, the first two columns denote point pair (Xo and x1), the third column shows reconstructed estimate of the second column X after applying transport operators with inferred coefficients c, and the fourth column shows a reconstructed estimate after applying transport operators with a times extrapolation of the inferred coefficients 2c.
pass of learned DNN. We use a fully-connected network with two hidden layers that takes the features of the point pairs concatenated as input (e.g., [zo, zı]) and coefficients as output, similar to the method used in Ibrahim et al. (2022). To prevent coefficients from growing too large, we also apply an & penalty coefficients with a weight of 1.-3. We train our auto-encoder and coefficient inference network together end-to-end using automatic differentiation.
We first compare the qualitative reconstruction capabilitics of sparse and dense coefficients. For both methods, we randomly sample 5, 000 point pairs and perform coefficient inference estimate transport operator paths. Averagod across all point pairs, the dense and sparse coefficients result in transport operator error of 2.80 2.804e-4 and 1.023e-3, respectively. This is to be expected, since the sparse coefficients are constrained to use fewer operators for every point pair, mirroring the intuition of higher training MSE in LASSO over unconstrained least-squares methods. We show qualitative reconstructions in Figure g where the fourth column denotes the reconstruction of extrapolated paths. Extrapolated paths with sparse coefficients have higher reconstruction quality, allributed to better operator stability (see Appendix Fb. To measure this, we take the eigenvalue with the largest real component from each operator and normalize by the average coefficient magnitude across the 5, 000 evaluation point pairs. The modian of the largest normalized cigenvalue over all operators 3.08e-2 and 1.06e-2, for operators trained with dense and sparse coefficients, respectively.
Finally, we compare transformations for both methods by applying single operator with coefficients within fixed range (as in Figure D. We seleet four operators from both methods and find the average cocfficient magnitude across the 5, 000 evaluation point pairs. We then interpolate samples between -5 and times this average magnitude, applying transformations from a single operator. The results are shown in Figure 10 where it can be seen that operators trained with sparse coefficients lead to more semantically meaningful transformations and higher reconstruction quality.
F Comparing Point Pair Selection Strategies
In the past, transport operators have been trained using point pairs that were selected randomly from the same class (Connor al. (021) or with some knowledge of transformation labels (Connor & Rozell 2020). In this work, we establish method that learns natural transformations without requiring transformation labels using a perceptual point pair selection strategy. The perceptual loss metric enables to select point pairs that may share semantically meaningful transformations without being exactly the same. Fig. 11 shows examples of nearest neighbors selected using pixel similarity, similarity in the latent space, and the perceptual
21

---

Published in Transactions on Machine Learning Research (03/2023)
(a) Sparse Coefficients
(b) Dense Coefficients
Figure 10: Transport operator augmentations applicd by interpolating coefficients within a fixed range for (a) sparse coefficients and (b) dense coefficients, as in Figure [ Each row denotes a different operator applied to input point. From left to right, each column denotes a single operator applied to the same input point, with coefficients interpolated between [-Ne....,NNe)
loss metric. This highlights how the perceptual loss metric can be useful for identifying inputs with similar qualitative characteristics. For instance, on the left, the perceptual loss metric identifies another bald man nearest neighbor which has the similar characteristics of the initial image even though the exact image looks quite different.
Pixel NN
Latent NN
Perceptual NN
Figure 11: Examples of nearest neighbors identified through pixel similarity, latent similarity, and the perceptual loss metric. The images in the green boxes are the initial reference images and the images to the right of those show three selected nearest neighbors. The perceptual loss metric identifies neighbors that share similar characteristics like hair style without being exactly the same.
When examining the success of learned operators, one characteristic we care about is how well the operators maintain stability of gencrated paths. We have defined this as a useful characteristic for edentity-presorvatee of applied operators if operators expand latent vectors to infinity, that will very likely lead to a change in identity. Each of the operators can be viewed as the dynamics matrix of a linear dynamical systems model and we can analyze their stability by observing their eigenvalues (Strogatz) 22000). In our model, because the transport operators relate to natural data transformations without temporal component, it does not matter if the transport operators are applied with positive or negative coefficients. Therefore, general stability or instability of the dynamical system associated with a given transport operator has the same impact of causing the latent vectors to increase to infinity because our coefficients can be either positive- or negative-valued. In this setting, we define the stability of our transport operator system in the context of marginal stability of a dynamical system. Marginally stable dynamical systems generate cyclic transformations that never increase or decrease the magnitude of the inputs. A marginal stable system has only imaginary eigenvalues with no real parts (Strogatz 2000). Therefore, we can identify transport operators approaching marginal stability by investigating the magnitude of the real parts of their eigenvalues and specifically the eigenvalue with the maximum magnitude of its real part. When the maximum magnitude of the real part for all cigenvectors associated with an operator is close to zero, that indicates that the operator is closer to marginal stability. Therefore, we quantify the stability of transport operator paths by looking at the maximum magnitude of real parts of eigenvalues associated with each transport operator. Fig [2a shows the sorted maximum magnitude of the real parts of eigenvalues in each of the 16 operators learned in the MNIST experiment when using the parceptual point pair selection strategy and when using a simple strategy of selecting point pairs nearest.
22

---

Published in Transactions on Machine Learning Research (03/2023)
Max Real Value Magnitude
Perceptual Latent NN
Perceptual
Latent NN
1.5
0.5
Imaginary
0 x
0.5
-0.5
X
-1
5
10
15
-2
-1
0
2
Transport Operator Number
Real
(a)
(b)
Analysis of eigenvalues of operators learned in the MNIST experiment. (a) The maximum absolute value of the real parts of eigenvalues computed from each learned operator. (b) Plot of the real and imaginary parts of the eigenvalues for each operator.
Figure 12:
neighbors in the latent space. Except for one operator, all the operators trained using latent space similarity for supervision have larger maximum magnitudes of real eigenvalue components than the operators trained using the percepl ual loss metric. This indicates that the operators trained using the latent space similarity to select training point pairs are farther from marginal stability and can be seen as less stable by our definition of transport operator stability.
To view the effect of these operators more intuitively, we plot the values of the latent dimensions as individual operators are applied in Fig. I3 The paths of the lines in each of these plots show a lot about how the operators influences cach latent dimension. Fig 13a shows the paths generated by transport operators trained with the perceptual point pair selection strategy. The plots with straight lines show the effect of operators whose magnitudes are reduced to 0 during training. Most of the operators learned using the perceptual point pair selection strategy are close to cyclic except for operator 2 (the operator with the large real component magnitude Fig. 12a). In contrast, several of the operators trained with point pains selected as nearest neighbors the latent space extend to infinity (Fig 13bl. This can explain the larger maximum real value magnitudes in Fig. 12a This analysis leads us to conclude that the perceptual point pair selection strategy is more likely to yield identity-preservinng transport operators.
G MNIST Experiment Details
The MNIST dataset is made available under the terms of the Creative Commons Attribution-Shar Alike 3.0 license. We split the MNIST dataset into training, validation, and testing sets. The training set contains 50,000 images from the traditional MNIST training set. The validation set is made up of the remaining 10,000 images. The traditional MNIST testing set is used for our testing set. The input images are normalized so their pixel values are between and 1. The network architecture used for the autoencoder is shown in Table 2 The training parameters for the transport operator training phase and the finc-tuning phase are shown in Tables
3 and
Prior training the coefficient encoder for the MNIST dataset, we train classifier on the labeled MNIST image data which we use to encourage ceentty--reservation during coefficient encoder training. The training parameters for the coefficient encoder are shown in Table 5 The image classifier use is based on the simple Ne architecture with two convolutional layers and three fully connected layers LeCun et al. 1998
23

---

Published in Transactions on Machine Learning Research (03/2023)
2
2
0
0
590
-2
2
-2
-2
-2
2
2
13
14
15
BOACK
-2
-2
-10
0
10
-00
10
10
10
-10
-10
-10
10
11
0
10
10
10
Coefficient Values
(a)
2
-2
2
22
-2
-10
10
-10
10
-10
10
10
-10
-1
0
10
-10
10
-10
10
9
10
11
12
13
14
15
16
2
2
-2
2
-10
0
10
-10
10
-0
10
10
-10
10
10
0
10
10
0
10
10
10
Coefficient Values
(b)
Figure 13: Visualizations of the effect of each learned transport operator on each dimension of an encoded latent vector. In each plot, each line represents single latent dimension and the coefficient magnitude of the transformation varies on the x-axis. (a) Paths from operators learned with percept.al point pair supervision. (b) Paths from operators learned with point pairs selected as nearest neighbors the latent space.
Table 2: Network Architecture for MNIST and Fashion MNIST Experiments
Encoder Network
Decoder Network
Input E 28228
Input R²
conv: chan: 64, kern: 4, stride: 2, pad: 1
Linear: 3136 Units ReLU
BatchNorm: feat: 64
ReLU
conv TonvTranposs: chan: 64, kerm: 4, stride: 1, pad: 1
conv: chan: 64, kern: 4, stride: 2, pad: 1
BatchNorm: feal: 64
BatchNorm: feat: 64
ReLU
ReLU
coanTrapp chann: 64, kern: 4, stride: 2, pad: 2
conv: chan: 64, kern: 4, stride: 1, pad:
BatchNorm: feat: 64
BatchNorm: feat: 64
ReLU
ReLU
TonvTraaposs:: chan: 1, kernel: stride: 2, pad: 1 Sigmoid
Linear: 2 Units
H MNIST Experiment Additional Results
Here we show additional experimental details and results for the MNIST experiment. Fig. 14 shows the magnitude of all 16 operators after the fine-tuning phase. Six of the operators have their magnitudes reduced to zero. Fig. 15 shows the paths generated by transport operators trained on MNIST data.
Fashion MNIST Experiment Details
The Fashion MNIST dataset is made available under the terms of the MIT license. We split the Fashion MNIST dataset into training, validation, and testing sets. The training set contains 50,000 images from the
24

---

Published in Transactions on Machine Learning Research (03/2023)
Table Training parameters for the transport operator training phase of the MNIST experiment
MNIST Transport Operator Training Parameters
batch size: 250
autocncoder training opochs: 300
transport operator training epochs:
latent space dimension (Zdim): 10
M 16
Iraet 10-4
Ir 10-3
0:1.1
7:2x x 10-6
initialization variance for V: 0.05 number of restarts for coefficient inference: nearest neighbor count: 5
latent scale: 30
Table 4: Training parameters for the fine-tuning phase of the MNIST experiment
Table 5: Training parameters for the MNIST Coefficient Encoder
MNIST Fine-tuning Parameters
batch size: 250
MNIST Coefficient Encoder Parameters
transport operator training epochs: 100 Iraet: 10--
batch size: 250
training epochs: 300
Irv 10-3
Ir: 10
Ç: 0.1
Sprior 0.1
y 2 x 10-6
AgI: 0.5
A: 0.75
coefficient spread scale: 0.1
number of network update steps: 50
classifier domain: image
number of V update steps: 50
Dictionary Magnitude
2
1
5
10
15
Transport Operator Number
Figure 14: The magnitudes of the learned operators after fine-tuning
Fashion MNIST training set. The validation set is made up of the remaining 10,000 images. The traditional Fashion MNIST testing set is used for our testing set. The input images are normalized so their pixel values are between 0 and 1. The network architecture used for the autoencoder is the same as in the MNIST experiment and it shown in Table 2 The training parameters for the transport operator training phase and the fine-tuning phase are shown in Tables and
Prior to training the coefficient encoder for the Fashion MNIST dataset, we train classifier the latent vectors associated with labeled Fashion MNIST data which we to encourage dentitt-preserration during
25

---

Published in Transactions on Machine Learning Research (03/2023)
Transport Operator
Transport Operator
Transport
Transport Operator
0
0 D
O
O
O
O
O
O
7
7
8 1
A 2
F
1
1
8
2
2
2
2
2
8
8
2
2
2
2 3
2
2
2
2
2
2
3
3
7 3
3 3
3
D
3
3
3 3
7
3
3
33 49
2
3
3
3
3
4 555
444
4 4
4
4
4
4
4
444
44
5
5
5 5
5
5 55555
5
5
5
5
5555
1555555
6
6
6
6
6
6
6
6
6
6
6
6
6
5
7
7
7
7
7
7 7
7
7
7
7 7
7
77
7
7
7
7
7
7 8 4
2
2
8
8
8
9
9 9
8
V
V
8
8
8
8
8
8
8 9
8 9
7
7
9
9
9
9
4
<
K 4
9
9
9
9
9
4 (d)
Q
9
9
cc)
D
O
0
0
0 8
O O O
0
O
O
b O
2
/
/
7
7
} 1 /
2
2
2
2
2 3
3
3
2
2
2
2
2
2
1
2 2 2 2 2 2 Z
3 4 4 4444
3
3
3
3
3
335 3 3 3 3
2
3
3 4 5 6
3 4 5 6
3
3 3 3 3 3 33
44444
4
4
4 4 4 4 4
5555555
5
555555
5
5
5
5 6 7 7
555
555555
6
6
6
G G
6
6
6
6
6
6
66
7 7
7
7
7
7 7
7
7
7
7
7 8
7 8
7
7 X X
7 8
7
7
7
77
R
Q
8
8
8
8
9
8
8
9 9
8
8
8
8 9
8
8
8
4
9
9
9
9
5
8 Y
9
9
4
4
4
9
9
9
4 4
9
9
9
9
9
(c)
(h)
0
O 1
O
O
O
O
0
o
1
1
2
2 2 2
2 2
3
3
3
3
33
2 2
3
3
3 3
7
4
4
44444
4
4 414440
4 5 6
5 5
5
5
5
5
5
6
6 6
6
6
6
6
6
7
7
177
7
7
7
7 8
8
8
8
8
8
R
8
8
8
8
7 7
4 9
9
4
4
9
9
9
9
(i)
Figure 15: Paths generated by all non-zero transport operators trained on the MNIST dataset. Images in the middle column of the image block are the reconstructed imputs and images to the right and left are images decoded from transformed latent vectors in positive and negative directions, respectively
coefficient encoder training. The training parameters for the coefficient encoder are shown in Table g The latent vector classifier has a simple architecture of Linear(512), ReLU, Linear(10), Softmax.
26

---

Published in Transactions on Machine Learning Research (03/2023)
Table Training parameters for the transport operator training phase of the Fashion MNIST experiment
Fashion MNIST Transport Operator Training Parameters
batch size: 200
autocncoder training epochs: 300
transport operator training epochs:
latent space dimension (Zdim): 10
M 16
Innet 10-4
Irv 10-3
0.5
7:2 x 10-5
initialization variance for V: 0.05 number of restarts for coefficient inference: nearest neighbor count: 5
latent scale: 30
Table 7: Training parameters for the fine-tuning phase of the Fashion MNIST experiment
Table 8: Training parameters for the Fashion MNIST Coefficient Encoder
Fashion MNIST Fine-tuning Parameters batch size: 200
Fashion MNIST Coefficient Encoder Parameters
transport operator training epochs: 150 Innet 10-1
batch size: 200
training epochs: 300
Ir : 10-3
Ir: 10-3
Ç: 0.5
Sprior 0.5
y: 10-6
ARt: 0.5
A: 0.75
coefficient spread scale: 0.1
number of network update steps: 50
classifier domain: latent
number of V update steps: 50
J Fashion MNIST Experiment Additional Results
Here we show additional experimental details and results for the Fashion MNIST experiment. Fig. 16 shows the magnitude of all 16 operators after the fine-tuning phase. Six the operators had their magnitudes reduced zero.
Dictionary Magnitude
4
3
2
5
10
15
Transport Operator Number
Figure 16: The resulting magnitude of the learned operators after fine-tuning the MAE.
To visualize how the use of the transport operators varies over the latent space, we generate an Isomap embedding Tenenbaum et al. 2000) of latent vectors and color each point by the encoded scale paramcter
27

---

Published in Transactions on Machine Learning Research (03/2023)
for coefficients associated with each of the transport operators. Fig. 5b shows these embeddings for Fashion MNTST data. Each operator has regions of the latent space where their use is concentrated. Fig. 5 shows the average coefficient scale weights for each class (rows) and each transport operator (columns) for Fashion MNIST. There are some classes like trouser and sandal (classes and 5) which have large encoded coefficient scale weights for most of the transport operators. This means they are robust many natural transformations. Other classes like coat and shirt (classes and 6) have smaller encoded coefficient scale weights which means they are sensitive
many transformations. The images to the right in Fig, |dd show transport operators being applied to samples with high encoded scale weights (in yellow box) and samples with low encoded scale weights (in blue box). Fig. 17 shows the paths generated by non-zero transport operators trained on Fashion MNIST data.
K CelebA Experiment Details
The CelebA dataset is publicly available for non-commercial research purposes. We split the CelebA dataset into training and testing sets. The training set contains the first 150,000 images accompanied with the entire test set. The input images are normalized so their pixel values are between and 1. The network architecture used for the autoencoder shown Table g The training parameters for the transport operator training phase and the fine-tune fine-tuning phase are shown in Table 10
The attribute classifier is RosNet-18 model adapted from
with 16 classifier heads after the layer with 512 hidden units. Each classifier head corresponds to a single attribute and is modeled as fully-connected linear layer with 256 hidden units, followed by batch normalization, dropout, and ReLU layers. Afterwards, two logits are output for a 0/1 prediction for each classifier output. The training procedure, including the dynamic loss weighting, follows (Mao et al. 2020
Table 9: Network Architecture for CelebA Experiments
Encoder Network
Decoder Network
Input p64x64
Input R-2 Linear: 80,000 Units
conv: chan: 32, kern: 4, stride: 2, pad:
BatchNorm: feat: 32
ReLU
ReLU
conv TowTTranpss: chan: 256, kern: 3, stride: 1, pad: 0 BatchNorm: feat: 256
conv: chan: 64, kern: 4, stride: 2, pad:
BatchNorm: feat: 64
ReLU
ReLU
conv TonTrroposs:: chann: 256, kern: 3, stride: 1, pad: 0
conv: chan: 128, kern: stride: 2, pad: 1
BatchNorm: feat: 256
BatchNorm: feat: 128
ReLU
ReLU
convTranppen chan: 256, kernel: 3, stride: 1, pad:
conv: chan: 256, kern: 3, stride: 1, pad: 1
BatchNorm: feat: 256
BatchNorm: feat: 128
ReLU
ReLU
conv NouTTranpos:: chan: 128, kernel: 3, stride: 1, pad:
conv: chan: 256, kern: 4, stride: 2, pad: 1
BatchNorm: feat: 128
BatchNorm: feat: 256
ReLU
ReLU
conv connTooapss:: chan: 128, kernel: 3, stride: 1, pad:
conv: chan: 128, kern: 4, stride: 2, pad: 1
BatchNorm: feat: 128
BatchNorm: feat: 128
ReLU conv TonyTrappss:: chan: 3, kernel: 4, stride: 2, pad: 0 Sigmoid
ReLU
Linear: Units
L
CelebA Experiment Additional Results
Here show additional experimental results for the CelebA experiment. Fig. 18 shows the paths generated
by the 40 transport operators trained on cclebA data. Each row represents different operator acting on the
28

---

Published in Transactions on Machine Learning Research (03/2023)
Table 10: Training parameters for the CelebA experiment
CelebA Transport Operator Training Parameters
batch size: 500
autoencodor training epochs: 300
CelebA Fine-tuning Parameters
transport operator training epochs: 50
batch size: 500
latent space dimension (Zdim): 32
fine-tuning epochs: 10
M 40
Iraet :10
Iruet 10-4
Ir 10-3
Ir 10-3
Ç: 0.8.8 0.8
1:5
7: 7: 10-7
7:1x 10-5
0.75
initialization variance for V: 0.05 number of restarts for coefficient inference: nearest neighbor count: 5
number of network update sheps:
number of v update steps: 50
latent scale: 2
same input image. Images in the middle column of the image block are the reconstructed imputs and images to the right and left are images decoded from transformed latent vectors in positive and negative directions, respectively. Fig. ?? shows the classification outputs of the attribute classifier for example transport operators. These two operators that vary smiling, beard, and sunglasses attributes.
Fig. 19 shows an interesting feature in our learned operators many of them generate cyclic paths that begin and end at nearly the same point. Also, these cyclic sequences, the transformations seem lead a change in gender. The image sequence in Fig. 19a shows the path generated by single operator. The image in the middle is the reconstructed input image and the images to the left and right are the paths generated by negative and positive coefficients respectively. This operator changes the hairline and quantity of bangs. As we apply the transport operator with a negative coefficient (to the left of center), the woman gains bangs and then becomes a man with a moustache on the far left of the image. As we apply the transport operator with a positive coefficient (to the right of center), the woman's forehead gets higher and then she becomess a man with high forehead and eventually, on the far right the woman transforms into man with bangs, similar to the man on the far left. The similarity between the generated images on the far left and far right is notable because this indicates the transformation path is nearly closed. Fig. 19b shows the change in five of the 32 latent dimensions over the generated path. These paths have nearly cyclic structure.
This is particularly interesting because in (Connor & Rozel] 2020 they learn closed transformation paths by selecting point pairs on known cyclic transformation paths and highight the benefit of the transport operator model for representing these types of paths. In this case, we learn this cyelic path with only perceptual point pair supervision. Additionally, this identifies benefit of the transport operator approach over other models of the manifold in a neural network latent space. We can learn nonlinear paths that keep the generated points in the same neighborhood in the latent space without extending to infinity which is inevitable when linear paths that are used to represent natural transformations.
29

---

Published in Transactions on Machine Learning Research (03/2023)
Transport Operator 1
Transport Operator 2
Transport Operator
Transport Operator 6
(c)
Transport Operator 7
Transport Operator 10
Transport Operator
Transport Operator 12
(e)
g
(h)
Transport Operator 13
Transport Operator 16
ii
Figure 17: Paths generated by all non-zero transport operators trained on the Fashion MNIST dataset. Images the middle column of the image block are the reconstructed inputs and images to the right and left are images decoded from transformed latent vectors in positive and negative directions, respectively
30

---

Published in Transactions on Machine Learning Research (03/2023)
COS
(b)
COO
CED
LAV
LAS
COO
DOS
(d)
Figure 18: Paths generated by all 40 transport operators trained the CelebA dataset. Images in the middle column of the image block are the reconstructed inputs and images to the right and left are images decoded from transformed latent vectors in positive and negative directions, respectively
31

---

Published in Transactions on Machine Learning Research (03/2023)
(a)
0.5
Latent Value
0
-0.5
-2
0
2
Coefficient Value
(b)
Figure 19: An example of an operator that generates a nearly cyclic path in the latent space (a) Image outputs along a path generated by a single learned operator. The images on the far left and far right look similar which indicates that this operator generates nearly closed path that begins and ends at the same point. (b) Paths of five of the 32 latent dimensions as the learned operator is applied to them. This again highlights the cyclic nature of the transport operator generated paths.
32


Reviews:
[{"review_id": "nF94VpIpcA", "reviewer": "Reviewer_mioK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- The strong performance on CelebA is a definite strength of this work, as prior work on Lie group operators has not evaluated on datasets of this scale. \n- The analysis of CelebA attributes as functions of the operator coefficients is a very nice addition. \n- The experiments clearly demonstrate that the proposed MAE method is better at extrapolation than other methods, which suggests that the manifold learned by this method is potentially more useful for, e.g., data augmentation than other methods. \n\n\n**Weaknesses:**\n- While CelebA is a great addition, the only other evaluations are done on MNIST and Fashion MNIST. I only view this as a slight weakness, as it appears that prior work on MAEs evaluates on similar datasets. \n- The approach relies on the existence of a pre-trained classifier on some large, relatively in-domain dataset. I definitely don’t think that this is a deal breaker for this paper, but I am curious if the method could still work outside of the image domain. For instance, it would be extremely interesting if this method could recover known operators from some other data domain such as PDEs–such as the LPSDA operators used in [1]. \n\n[1] https://arxiv.org/pdf/2202.07643.pdf \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a generative manifold learning method based on Lie group operators that aims to address the following two concerns with prior work on Lie group operators learning:\n- the manifold learning problem is sensitive to the choice of point pairs used to learn the operators (which often also requires supervision signal), and\n- current methods wrongly assume that learned operators are uniformly applicable anywhere on the manifold. \n\nThe proposed method is based on prior work on manifold autoencoders (MAEs) that modify autoencoders to learn Lie group operators without direct supervision signal. The authors perform a thorough evaluation and comparison to other unsupervised manifold learning techniques that do not incorporate any supervision signal. \n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Possible changes / questions:**\n- [critical] The baselines appear to not involve any supervision signal at all, while MAE uses pretrained models as a form of alternative supervision to obtain neighborhood information. Unless I’ve misunderstood, it seems that this same form of alternative supervision signal can be used within other Lie group operator learning methods as slightly more comparable baselines. Can the authors comment on this choice? \n- [critical] Relatedly, and I might have missed something, but some commentary on the choices that went into deciding on this set of existing baseline techniques would be useful, and why they are the correct set of baselines to consider in this setting. \n- [not critical but seems odd] “The lower classification accuracy in fashion MNIST is due to both a more challenging dataset and the lower resolution of autoencoder image outputs when compared to input images.” – why are the outputs lower resolution? I may have missed something here, but I imagine that you can just design the decoder network to output the appropriate resolution. \n- [not critical but I think it would really strengthen the work] This is not a deal breaker, but I would be curious to know how well these learned operators work for performing data augmentation (especially given the identity-preserving properties). \n- [a definite nitpick] throughout the text, “FMNIST,” “Fashion-MNIST,” and “Fashion MNIST” are used interchangeably – this could be cleaned up a bit. \n", "broader_impact_concerns": "The authors do briefly mention broader impact: “While this is a powerful model, users should be mindful of the biases that are introduced through training with specific supervision methods when drawing conclusions about learned data transformations.” A minor suggestion: adding a brief remark on the specific usage of face datasets such as CelebA and the potential for modeling potential biases in such data could improve upon the current statement of broader impact. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "38qb85tkpb", "reviewer": "Reviewer_5pDP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "pros:\nThe author proposed methods to address the two drawbacks of the Lie group operator accordingly. The first drawback was addressed through applying a data selection strategy that encourage the selected pairs of data are visually similar. The second drawback was addressed through learning the preferred local regions of each transport operator while preserving identity. The motivation of this paper is clearly stated and the proposed methods are well presented.\nThe author also discussed about the computational complexity regarding the gradient of matrix exponential. The author resorted to a forward-backward splitting scheme and clearly summarized related existing methods.\nThe author designed abundant experiments to analyze the effectiveness of the proposed methods from multiple aspect. \n\n\nCons:\nThe author proposed to compare with CAE and \\beta-VAE in all the analysis part. The author also showed the superior of the proposed method in terms of certain metrics. However, it would be better to first state more about the connection between proposed methods and the above two methods. Especially, why and how the proposed method is superior theoretically or technically. More discussion paragraphs may need to make this point more convincing.\n\nAll the experiment results in this paper are focusing on analyzing the learnt manifold and transformed samples themselves. To verify the effectiveness of the prosed methods, more downstream tasks are needed. Such as the classification results on the reconstructed images. Or generate transformed images as augmented samples and study the performance of models trained with such augmentations. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "n this paper, the author proposed to learn identity preserving transformations through manifold representation learning. The author first proposed a perceptual point pair selection strategy to encourage the learned transport operators induce more semantically meaningful transformations. The author then proposed to learn the locale region preference of each transport operators through introducing an operator coefficient encoder.\nThe author designed multiple experiments to show the effectiveness and utility of the proposed methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As stated in the previous section, listing two adjustment recommendations. Both of which will make the work be more convincing in my view.\n\n1. Adding more discussion concerning the connection with the chosen baseline methods. Showing more theoretical justification other than empirical justifications only.\n2. Adding more downstream experiments to show the effectiveness of the proposed methods.", "broader_impact_concerns": "No ethical concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rmO0CbZKhi", "reviewer": "Reviewer_w5hQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n-----------\nThe abstract and introduction are well-written and easy to follow. I appreciate the incorporation of a background section, making the paper self-contained. \n\nWeaknesses\n--------------\n\nformulation\nThe formulation in the method section is missing some details. E.g, dimensions input and output model functions are missing; the details of how  q_\\theta fits into eq. (5) are missing. What is the role of the added noise in eq (5), why is it needed?\n\nCumbersome training technique\nThe previous method already consisted of three somehow independent training phases. The suggested technique adds another phase for training the coefficient encoder. This raises the concern of how practical it would be to adapt the suggested method to new datasets by other practitioners. Is it possible to reduce the method to consist of only one training phase? Is not why is that so? \n\nMissing evaluation\nthe method is missing a quantitative evaluation. This raises the concern that the examples chosen for the figure are not representative. For example, the interpolation in figure 6 down for non-learnable coefficients looks \"bad\", while in figure 1 it looks \"good\". In addition, it seems that the chosen baselines can be considered of less quality because of their reconstruction capacity. Why not add standard AE, VAE as baselines as well? will they produce the same reconstruction quality as the suggested method?\n\nmissing ablation\nI think it would be interesting to test the effect of the lie group structure model. Would it be possible to design an experiment where the same training phases are used to modify the weights of a learned encoder (maybe only the last layer), according to the loss in eq(2) s..t it organizes the latent space according to the knowledge of the classifier. Maybe that is enough to get the same results without the lie group model?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses the challenge of manifold learning. The paper follows a manifold learning technique based on augmenting the learned latent space of autoencoders with a Lie group structure. The paper identifies two shortcomings in the existing training procedure for learning the lie group manifold: i) the need to specify \"close\" training samples without the available supervision; ii) the assumption of a global lie manifold hinders the ability to learn transformations that are only applied locally. \nThe paper addresses the above challenges by suggesting two corresponding techniques. The first technique to find close training samples is based on the features of a pre-trained imagenet classifier. The second technique is based on conditioning the predicted lie group element on the input with an encoder network.\nThe method is qualitatively evaluated on mnist,fmnist and celebA.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the above weaknesses—specifically, the ones describing the missing ablation and evaluation. ", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors study how to find transformations or invariances in data by learning Lie group elements. Their work does so in an unsupervised way that additionally has ae local refinement. 

The reviews are largely positive. Reviewers agree that the paper is well-written, interesting, and provides evidence for the claims made. In addition, all agree that the authors provided responses to the reviews that further clarified all the key questions.

The only concern was about evaluation, with one reviewer asking for evaluation claims beyond the MNIST level of dataset. Beyond this, but the authors do also include the CelebA set. Overall I believe the empirical evidence provided here is pretty reasonable.

For these reasons, acceptance is appropriate.
Decision: Accept (poster)