Learning Identity-Preserving Transformations on Data Manifolds
Authors: Marissa Catherine Connor
Kion Fallah
Christopher John Rozell
Venue: ICLR 2022 Submitted (2022)
Keywords: manifold learning
unsupervised learning
generative models
lie groups
transport operators
transformation learning

Abstract: Many machine learning techniques incorporate identity-preserving transformations into their models to generalize their performance to previously unseen data. These transformations are typically selected from a set of functions that are known to maintain the identity of an input when applied (e.g., rotation, translation, flipping, and scaling). However, there are many natural variations that cannot be labeled for supervision or defined through examination of the data. As suggested by the manifold hypothesis, many of these natural variations live on or near a low-dimensional, nonlinear manifold. Several techniques represent manifold variations through a set of learned Lie group operators that define directions of motion on the manifold. However theses approaches are limited because they require transformation labels when training their models and they lack a method for determining which regions of the manifold are appropriate for applying each specific operator. We address these limitations by introducing a learning strategy that does not require transformation labels and developing a method that learns the local regions where each operator is likely to be used while preserving the identity of inputs. Experiments on MNIST and Fashion MNIST highlight our model's ability to learn identity-preserving transformations on multi-class datasets. Additionally, we train on CelebA to showcase our model's ability to learn semantically meaningful transformations on complex datasets in an unsupervised manner.

Under review as a conference paper at ICLR 2022
LEARNING IDENTIYY-PESSERVNG TRANSFORMA- TIONS ON DATA MANIFOLDS
Anonymous authors Paper under double-blind review
ABSTRACT
Many machine learning techniques incorporate identity-preservin transforma- tions into their models to generalize their performance to previously unseen data. These transformations are typically selected from set of functions that are known to maintain the identity of an input when applied (e.g., rotation, translation, flip- ping, and scaling). However, there are many natural variations that cannot be labeled for supervision or defined through examination of the data. As suggested by the manifold hypothesis, many of these natural variations live on or near low- dimensional, nonlinear manifold. Several techniques represent manifold varia- tions through a set of learned Lie group operators that define directions motion the manifold. However theses approaches are limited because they require transformation labels when training their models and they lack a method for de- termining which regions of the manifold are appropriate for applying each specific operator. We address these limitations by introducing a learning strategy that does not require transformation labels and developing a method that learns the local regions where each operator is likely to be used while preserving the identity of inputs. Experiments on MNIST and Fashion MNIST highlight our model's s ability to learn we train on CelebA to showcase our model's s ability to learn semantically mean- ingful transformations on complex datasets in an unsupervised manner.
transformations on multi-class datasets. Additionally,
INTRODUCTION
A goal of many machine learning models is to accurately identify objects as they undergo natu- ral transformations - a task that humans are adept at. According to the manifold hypothesis, natural variations in high-dimensional data lie on or near oow-dimeniional, nonlinear manifold (Fefferman jet al. (016). Additionally, the manifolds representing different classes are separated by low density regions Rifai et al., (201la). Natural physical laws govern the possible transformations that objects can undergo and many of the identity- preeeity-ppeeeiiing transformations (e.g., changes viewpoint, color, and lighting) are shared among classes of data. Sharing of transformations between classes enables increased efficiency in defining data variations - a model can represent a limited set of transforma- tions that describe majority of variations in many classes. Several machine learning models incorporate specific adentity-preserving transformations that are shared among a large number of classes generalize the performance of their model to unseen data. These include equivariant mod- els that incorporate transformations like translation and rotation into intermediate network layers 2016 Cohen et al., 2018) and data augmentation techniques that apply known variations to data while training (Cubuk et al. 2019 Ho et al.) 2019 Lim et al., Sohn et al. 2020 He al., 2020 Chen et al., 2020). However, many datasets have natu- ral transformations shared among classes that are not easily prespecified from intuition, making it critical that we develop a model that can learn both 1) representation for these transformations without explicit transformation labels and 2) the context for which locations in the data space each transformation is likely to be relevant.
(Cohen & Welling
2019
Manifold learning strategies estimate the low-dimensional manifold structure of data. A subset of these techniques learn to transform points on the manifold through nonlinear Lie group opera- tors (Rao & Ruderman, 1999; Miao & Rao 2007; Culpepper & Olshausen, 2009 Sohl-Dickstein et al. 2010;[Cohen & Welling 2014: Hauberg et al., 2016; Connor & Rozell, 2020 et al. Lie group operators represent infinitesimal transformations which can be applied data
2021

---

Under review as a conference paper at ICLR 2022
through an exponential mapping to transform points along manifold, and manifold can be glob- ally defined by a set of operators that each move in different directions along it (Hoffman [966, Dodwell 1983). A Lie group operator model is well-suited for representing natural data variations because the operators can be learned from the data, applied to data points to transform them beyond their local neighborhoods, and used estimate geodesie paths.
While the Lie group operator models have many benefits, previous approaches demonstrate the two shortcomings noted above. First, learn Lie group operators that represent a data manifold, pairs of training points are selected which lie within neighborhood of one another. The training objec- tive encourages efficient paths between these nearby points and the choice of training point pairs influences the types of manifold transformations that are learned. Recent papers incorporating Lie group operators into machine learning models have either used predefined operators that represent known transformations groups (e.g., the 3D rotational group SO(3) (Falorsi et al. 2019p), required transformation labels for selecting point pairs when training (Connor & Rozeli| 2020). or randomly selected pairs of points from the same class (Connor et al. 2021). To learn effective model with datasets having no labeled transformation structure, we require a point pair selection strategy that identifies points that related through the transformations the model aims to learn. Second, exist- ing Lie group operator models have lacked a method for determining which regions of the manifold appropriate for each operator, meaning that every operator is equally likely to be used every point on the manifold. This is a flawed assumption because, while many transformations are shared between classes, there are also data variations that are unique to specific data classes. Additionally, in dataset with several manifolds (each representing a different class), there is limited extent to which a transformation can be applied without moving point onto another manifold.
The main contributions of this paper are the development of methods to address the two critical shortcomings of Lie group operator- based manifold models noted above. Specifically, motivated by finding perceptually similar training samples without transformation labels, we first introduce a point pair selection strategy to learn a manifold representation of natural variations shared across multiple data classes without requiring transformations labels. Second, we develop a method that uses a pretrained classifier (measuring local regions where each operator is likely to be used while preserving the identity of transformed samples. This approach enables us to analyze the local structure of the data manifold in the context of the learned operators and to describe the invariances of the classifier. We demonstrate the efficacy of these strategies in the context of the Manifold Autoencoder (MAE) model (Connor & Rozell 2020) to learn semantically meaningful transformations on MNIST (LeCun al. 1998), Fashion MNIST (Xiao et al. 2017). and CelebA (Liu et al. 2015).
of transformed samples) to learn the
2 BACKGROUND
Manifold Learning Manifold learning models estimate the low-dimensional structure of high- dimensional data by utilizing the property that local neighborhoods on the manifold are approxi- mately linear. Traditional techniques represent the manifold through low- now-dimensional embedding of the data points (Tenenbaum et al., 2000 Roweis & Saul 2000 Belkin & Niyogi 2003 Maaten & Hinton 2008) or through estimates of linear tangent planes that represent local directions of man- ifold motion (Dollár et al.] 2007 Bengio & Monperrus 2005 Park et al.) 2015). While these tra- ditional manifold learning approaches are useful for understanding low-dimensional data structure, in many cases the input data space is an inefficient representation of the data. For example, data in the pixel space suffers from the curse of dimensionality and cannot be smoothly interpolated while maintaining identity (Bengio et al. (2005). Many approaches have used neural networks to learn a low-dimensional latent space in which manifold models can be incorporated. The contractive autoencoder (CAE) estimates manifold tangents by minimizing the the Jacobian of the encoder net- work, encouraging invariance of latent vectors to image space perturbations (Rifai et al. 2011c b a Kumar al. 2017). Several methods estimate geodesic paths in the latent space of trained vari- ational autoencoder (VAE) model (Arvanitidis et al., 2018 Chen et al. 2018 Shao et al., 2018 Arvanitidis et al.) 2019) and use this approach to learn VAËS with priors that are estimated using the Riemannian metries computed in the latent space (Arvanitidis et al. 2021 Kalatzis et al. 2020).
Lie Group Operators A Lie group is a group of continuous transformations which also defines a manifold by representing infinitesimal transformations that can be applied to input data (Hoffman
2

---

Under review as a conference paper at ICLR 2022
1966 Dodwell 1983).
Several methods incorporate Lie groups into neural networks to represent
data transformations that are Seeetitt--reservve within the model (Cohen & Welling 2014 Cohen
et al. 2018 Cosentino et al., 2021 A prevalent strategy is to learn dictionary of Lie group operators that are mapped to a specific group element through the matrix exponential expm( (Rao
& Ruderman 1999 Ham & Leel 2006 Miao & Rao 2007 Culpepper & Olshausen 2009 Sohl-
Dickstein et al., 2010; Cohen & Welling, 2014; [Conorr & Rozell 2020] [Connor et al., 2021). ). In
these models, each operator L m. called transport operator, describes a single direction along the manifold and is parameterized by a single coefficient Cm- Given an initial data point z, the transport
operators define generative model where transformations can be derived from sampling sparse coefficients G. ~ Laplace (0,
2 expm Σ
L m Cr
2 7+6 C,
where e ~ oT)
The manifold autoencoder (MAF) incorporates the transport operator model into the latent space of an autoencoder to learn a dictionary of operators that represent the global, nonlinear manifold structure in the latent space (Connor & Rozell 2020). This model has been shown to effectively learn reusable operators with transformation supervision, and it will provide the context for demonstrating the effectiveness of the methods developed in this paper.
Disentanglement One class of methods that also aim to identify factors of variation in data is dis- entanglement methods which learn features that each vary one independent characteristic of the data (Higgins et al. 2018). In supervised learning scenario. there are many disentangling techniques that separate data style from content by encouraging similarity between the content features in sam- ples from the same class (Tenenbaum & Freeman 2000 Reed et al.) 2014; Mathieu et al. 2016 Bouchacourt et al., 2018) or encouraging features to map to known class (Cheung et al. 2015)
or transformation labels (Hinton et al., 2011; Kulkarni et al.) 2015; Yang et al.] 2015). There
also techniques that can disentangle latent representations without labels like InfoGAN (Chen et al. 2016) and 3- 3-VAE (Higgins et al. 2017). The goal of our work is distinct from disentanglement work because, rather than identifying independently varying factors of variation, we aim to learn non-linear operators that correspond to transformations independently). Our approach is advantageous because it can faithfully represent longer transfor- mation paths, learn variations while maintaining image reconstruction, and change the number of learned variations by increasing or decreasing the number of learned operators.
the data manifold (which may not vary
3 METHODS
The MAE learns a low-dimensional latent representation of the data by defining an encoder function f: X' that maps high-dimensional data points x E ER to low-dimensional latent vectors e Rd and a decoder function g 2 - X' that maps the latent vectors back into the data space (Connor Rozell 2020). Transport operators
are incorporated into the latent space to learn manifold- based transformations. Before learning the transport operators, the autoencoder is pretrained to extract a latent representation of the data using the traditional autoencoder reconstruction objective.
After pretraining, the autoencoder weights are fixed and the operators are trained with the following objective, which encourages the learning of transport operators that generate efficient paths between the latent vectors Zo and z (coinciding with fxx) and fxx)) that are nearby the manifold:
Ly
expmn Σ
L m Cm
Z0
+
L m 2 + Cl|e|/2,
(2)
where 7,5> 0.
Objective 22 is minimized via an alternating minimization scheme. Specifically, at each training iteration, points pairs X. and X₃ are selected in the input space and encoded into the latent space Zn and z Then, coefficients e are inferred between the encoded latent vectors (by minimizing ( with respect c) estimate the best path between 2g and 2 After inference, the coefficients
3

---

Under review as a conference paper at ICLR 2022
fixed and a gradient step is taken minimize 2 with respect to transport operator weights. Once learned, these operators represent different types of motion that traverse the manifold and they can be combined to generate natural paths on the manifold.
After fitting transport operators to the latent space fixed autoencoder network, there is a final fine-tuning training phase that updates the network weights and transport operators simultaneously using a joint objective that combines the autoencoder reconstruction loss with Ly- This fine-tuning step addresses the potential of a mismatch between the data manifold and the learned latent structure by adapting the latent structure to fit the transport operators learned between the selected training point pairs. We empirically find that breaking up training into these three phases increases the stability with which the transport operators can be learned. Given the context of this MAE model, the following sections we describe our contributions.
3.1 UNSUPERVISED TRANSFORRATION LEARNING
There are many possible strategies for selecting training point pairs for Lie group operator model and the choice of a strategy can dictate the types of learned transformations. Point pairs may be selected as random samples from the dataset or from within the
class. These points, however, are likely to be outside of local manifold neighborhoods and may not provide representations of the natural, perceptually smooth variations. Point pairs also be selected nearest neighbors in the pixel or autoencoder latent space (denoted with hxx) x and (xx) f(x) in ( respectively):
X1 = arg min [[((xx) hxx)]12
X
However, semantic transformations of interest may be exhibited through pixel similarity nor through unsupervised autoencoder feature similarity. In some applications there is additional infor- mation that can aid in point pair selection, such as rotation angle, semantic label, or time index in a temporal sequence. However, for most complex datasets, information about the transformations of interest is not available. This necessitates a strategy for selecting training point pairs which can be used to learn natural transformations without requiring additional transformation supervision.
We generalize transport operator training to incorporate an unsupervised learning strategy that can be applied to a wide array of datasets. In particular, we assume access to classifier pretrained on a large-scale dataset (likely taken from a model zoo) with similar statistics as our training dataset of interest (gguyen et al. 2020). In this approach, we select point pairs by setting h(x) from (3) equal to the penultimate layer of the classifier, similar to the previously proposed perceptual loss (Johnson let al. 2016). It has been shown that high-level features in such classifier networks correspond to semantic characteristics of the data and finding points nearby in this feature space can result in image pairs that perceptually similar without being exactly the same (Johnson al.) 2016 Yosinski et al. 2015).. Incorporating the classifier features as a supervision strategy for point pair selection enables our model learn transport operators without requiring transformation labels or class labels. We show that this metric leads to learned operators that correspond to semantically identifiable transformations. Discussion of different point pair strategies can be found in Appendix E
3.2 LEARNING LOCAL TRANSPORT OPERATOR STATISTICS TO ENCOURAGE IDENTITY PRESERVATION
A trained transport operator model provides a dictionary of operators that can be applied glob- ally across the entire data space. This model has flexibility to define local manifold characteristics through the coefficients e which specify the combination of operators to apply to a given point. The standard prior on the coefficients is factorial Laplace distribution parameterized by single scale parameter Ç, which encourages transformations to be defined by sparse set of operators:
M
(-lcml) <
(- |cml
(c(c)
II m=1 25
exp
(4)
By setting fixed prior coefficients across the entire manifold, there is an implicit assumption that the manifold structure is the same over the whole dataset and every operator is equally likely to be applied every point. However, this is flawed assumption because not all transformations
4

---

Unfortunately, the consistency regularization objective can be trivially minimized by setting c = O, resulting in identity transformation and the same classifier output. Therefore, we want encour- age the largest coefficient values possible while maintaining the identity of our initial data point. This motivates the addition of KL-divergence regularizer that encourages the distribution (c/z) Under review as a conference paper at ICLR 2022
in class of data present in all other classes and within classes there may be regions with different manifold structure. Failing to capture the local statistics of transport operator usage could result in transformed points that depart from the data manifold and change their identity.
To address this limitation, we introduce a coefficient encoder network which maps latent vectors to scale parameters that specify the Laplace distribution the coefficients those latent points. The goal of this network is to estimate local coefficient distributions that maximize the identity- preservation of points (as measured by pretrained classifier) transformed with operators character- ized by sampled coefficients as in (T) Given labeled observations (z, y) and pretrained classifier on the latent space r(-), we aim to learn network 90 that outputs parameters that can be used to produce augmented samples 7 without changing the classifier output r(z) y.
To maximize the likelihood of augmented input data maintaining the same classifier output, we adapt the concept of consistency regularization from cmi-suuereisee learning (Bachman et al. 2014 (Sonn et al., 2020). our context, we have pretrained classifier and a dictionary of transport operators and we aim to find distribution on the coefficients individual input points that result in consistent classification outputs when (c) is applied. The specific objective can be chosen from a variety of loss functions that encourage similarity in classifier probability outputs. We specifically minimize the KL-diveeencc between the classifier output r(z) and the estimated probability of the transformed output r(Z).
to be similar to a specified Laplace prior with a fixed scale Ç like in ( Our final objective for training the coefficient encoder network is: (a more detailed derivation can be found in Appendix A
E' = (xL((())))()) + DEL (go (c/zz) (c))
(5)
The coefficient encoder introduces a principled way to build model and to identify local manifold characteristics throughout the latent space. Additionally, it can quantify the extent with which operators can be applied in different parts of the latent space without changing class prediction, indicating transformations to which the classifier is invariant. Likewise, points or region that have small encoded coefficient scale weights indicate closeness to class boundaries and can undergo only small transformations without changing identity.
directly into our
3.3 DECREASING COMPUTATIONAL COMPLEXITY OF COEFFICIENT INFERENCE
To compute the gradient for the matrix exponential during training and inference, previous works have used linear approximation (Rao & Ruderman 1999), learned the operators in factored form (Sohi-Dickstein et al.] 22100), or used analytic gradients that are not favorable for parallel compu- lations (Culpepper & Olshausen 2009 Connor & Rozell 2020 Connor et al. 2021). The main computational bottleneck for learning transport operators is the coefficient inference step, where ( minimized with respect to c. The & norm term is non-smooth, leading slower convergence when using subgradients found via automatic differentiation (Boyd & Vandenberghe] (004)). In this work we employ a nawward-backwan splitting scheme (Beck & Tebouile 2009) that alternates be- tween automatic differentiation of the matrix exponential term, and proximal gradients for the & norm (Parikh & Boyd 2014). This significantly speeds up the training process and allows for scal- ing up the learning of transport operators to complex datascts. To the best knowledge of the authors, this is the first work to demonstrate learning of Lie group operators on datasets as large and complex as CelebA. Additional details on the inference algorithm are provided in Appendix B
4 ANALYSIS
For our experiments, examine the ability of our proposed approaches to enable the MAE model to learn natural transformations of datasets where the underlying not easily identifiable. We also highlight the benefits of incorporating the coefficient encoder network that captures the transport operator local usage statistics by encoding coefficient scale values that best maintain the identity of latent vectors.
transformations
5

---

Under review as a conference paper at ICLR 2022
We work with three datasets: MNIST (LeCun et al., 1998). Fashion MNIST (Xiao et al. 2017);
and CelebA (Liu et al., 2015). We select MNIST and Fashion MNIST because they contain several classes that share natural transformations but they do not have transformation labels. We select CelebA to highlight our ability learn natural transformations in a larger, more complex dataset. As classic dataset used in papers that aim disentangle dataset features (Higgins al.) 2017] Chen iet al., 2016 Hu et al., 2018; Lin et al., 2019). CelebA contains semantically meaningful natural transformations that may be amenable qualitative labeling.
In all experiments, we followed the general training procedure put forth previously (Connor & Rozell, 2020) by separating the network training into three phases: the autoencoder training phase, the transport operator training phase, and the fine-tuning phase. The fine-tuning phase adapts the au- toencoder latent space to the learned transformations in the event that that pretrained latent space does not match the desired data manifold. We select training point pairs that are nearest neighbors in the feature space of the final, pre-logit layer of ResNet-18 (He et al. 2015) classifier pretrained on ImageNet (Russakovsky et al. 2015). After completely training the MAE, we fix the autoencoder network weights and transport operator weights and train the coefficient encoder network with the objective derived in Section 3.2] Additional details on the datasets, network architectures, and train- ing procedure are available in the Appendix.
We compare against the contractive autoencoder (CAE) (Rifai et al. 2011c) and 3- VAE (Higgins et al. 2017), two other methods for incorporating data structure into the latent space.. The CAF represents another technique that learns manifold representation in neural network latent space. In their model, the manifold is represented by estimated tangent planes at latent point locations. The 3-VAE learns to disentangle factors of variation along latent dimensions through an increase in the weighted penalty on the KL-divergence term in the VAE objective. We choose these methods because they also learn natural dataset variations in the latent space without transformation labels.
4.1 LEARNING NATURAL DATA VARIATIONS
First, we show how well the perceptual point pair selection strategy described in Section 3.1 enables the MAE model to learn natural data variations in datasets and we highlight the usefulness of a nonlinear manifold model for generating latent space paths. Fig. [ shows the paths generated by transport operators trained in this model. The image in the middle column in each block of images the reconstructed version of the input image Xg- The images to the left and right of the middle show the reconstructed images generated by an individual learned operator applied to the encoded latent vector zo: 2e- = expmi V Pm.c)x Vg. e = -Ne, ...: Ne. We see an individual operator generates similar transformation across multiple inputs, and in many cases, the transformations induced by the transport operators are semantically meaningful. We also show that the perceptual point pair selection strategy is effective over range of datasets.
While many of the operators can be assigned a semantic label through qualitative visual inspection, the CelebA dataset has attribute labels for the images which enable quantitative analysis of the connection between learned transport operators and dataset attributes. To classify attributes, we fine-tune ResNet-18 pretrained ImageNet with 16 classification heads for attributes including smile, beard, hair color, and pale skin (Mao et al., 2020).. With this classifier, we are able to identify transport operators that correspond to specific datasct attributes. Fig. 2 shows the classification outputs of the attribute classifier for example transport operators. Our model learns operators that vary hair color and skin paleness as well as several others which are shown in the Appendix.
In Fig. I we compare transport operator paths to those generated by the CAE and the 9 V-VE The CAE-generated paths are the directions of motion on the tangent planes estimated at individual points. The 8-VAF paths are generated by varying the value of one latent dimension while the others remain fixed. While our method and the V-V learn several qualitatively similar transformations, method is capable of doing without significantly sacrificing reconstruction performance. In the Appendix, we include examples of the transformations generated by each learned operator.
A primary motivation for learning the data manifold is to generate new views of data that cannot arise from pre-defined functions. To highlight the ability of our model to generate new views that preserve identity, we compare both interpolated and extrapolated paths from the MAE to those from the CAE, 3- VAE, and a traditional auto-encoder in Fig. 3 Interpolated paths offer insight in the smooth variations between data points, while extrapolated paths show the capacity generate

---

Under review as a conference paper at ICLR 2022
Mannfoldd
Lighting Angle
Recoding Mareiin Tonge
Sklt Color
8 3-VAE
Lishting
Receiing Hlicting Danne
Skin Coltr
CAE
Backgooadd
Receling Huii fine: Bange
Criur
(a)
Line Width
Angled Squcezee
Digic Curvature
&
a 6
2
2
a a
R 2 2
a
a
2
a
6
6 6
6
6
8
8
8
8 8 8
8 8 88 38
8
8
8
8
8
8
(b)
Width
Sleeve Length
Color
1
Paths generated by applying subset of learned transport operators on three datasets. In each figure, images in the middle column of the image block are the reconstructed inputs and im- ages to the right and left are images decoded from transformed latent vectors in positive and negative directions, respectively. (a) Comparing the transformations generated by three learned transport op- erators to transformations generated by 3-VAF and CAF. The transport operators learn semantically meaningful transformations similar the disentangled 3-VAE representation while maintaining higher resolution in image outputs. (b-c) Transport operators learned using the perceptual point pair selection strategy generate natural transformations on both MNIST and Fashion MNIST.
Figure I:
0.15
0.0
4..5
4055
0.55
215
(b)
Figure 2: Paths generated by the application of two learned transport operators with the associated attribute classifier probability outputs for the transformed images. Our model learns operators that correspond to meaningful CelebA dataset attributes like (a) hair color (b) pale skin.
new views of data. For MAE, the paths are estimated by inferring coefficients c* between two latent points Zn and 2 and then generating the path: 2, = expm (Em= W Zn- When the path multiplier é between and that indicates interpolation and path multipliers beyond indicate extrapolation. For all other methods, Eucilidean distance between features is used for both interpolation and extrapolation. In these figures, the first block of images corresponds to the interpolated path with the selected final point x, surrounded in orange box. The second block of images corresponds to the extrapolated paths.
To quantify the identity preservation of each transformation, we input each generated image into pretrained classifier and plot the prediction the correct class. All four methods perform interpola- tion effectively but our trained model estimates the extrapolated paths more accurately. Fig. shows how that accuracy varies during extrapolation sequences for 4000 samples. The MAE better at generating extrapolated outputs that maintain class identity. The lower classification accuracy in fashion MNIST is due to both a more challenging dataset and the lower resolution autoencoder image outputs when compared to input images.
7

---

Under review as a conference paper at ICLR 2022
4
4
4
8
8
2
2 2
2
Z
Z
2
(a)
AB B-VAE
P:D Hudtcha
Path MUtb
Puth Mart Naciipiio
(b)
Figure 3: Identity preservation of transformed paths as quantified by pretrained classifier output. In the figures on the top, the first block of images corresponds to the interpolated path with selected final point X1 surrounded in an orange box. The second block of images corresponds to the extrapo- lated paths. Below the images are plots of the probability of the class label associated with the inputs 2n and z₁ A path multiplier between and indicates interpolation and path multipliers beyond indicate extrapolation. (a) MNIST (b) Fashion MNIST.
Clacuiaation Aucuraay
Fashion
0.8
CAR
0.4
CAt
Paeh Mullieer
Path Millcoowe
(b)
Figure 4: The average accuracy of the classifier output for images at each point along the interpola- tion/extrapolation sequence. When the path multiplier is between 0 and 1 that indicates interpolation and path multipliers beyond 1 indicate extrapolation. (a) MNIST (b) Fashion MNIST.
4.2 LEARNING LOCAL MANIFOLD STRUCTURE
After the MAE is trained, we have a dictionary of transport operators that describe manifold trans- formations and a network with a latent space that is adapted to the manifold. We then train the coefficient encoder network to estimate the coefficient scale weights as a function of points in the latent space. To visualize how the use of the transport operators varies over the latent space, we generate an Isomap embedding (Tenenbaum et al. 2000) of latent vectors and color each point by the encoded scale parameter for coefficients associated with each of the transport operators. Fig. [al shows these embeddings for MNIST data. Each operator has regions of the latent space where their use is concentrated.
By training the coefficient encoder to maximize the similarity between the classification outputs for an input sample and for a transformed version of that sample, the network aids in identifying which transport operators can be applied inputs in regions of the data space without changing the iden- tity of the input. This helps significantly with data augmentation where the goal is to create new samples with in-class variations. To highlight this benefit, in Fig. 6 we show samples augmented by applying transport operators with randomly sampled coefficients to an input latent vector. In each block of images, the leftmost image (in a green box) the input image and the images to the right are decoded augmentations. The top row shows samples augmented with transport operators con- trolled by coefficients sampled from Laplace distributions with encoded coefficient scale weights. The bottom row shows samples augmented with transport operators controlled by coefficients sam- pled from Laplace distributions with a fixed scale parameter. While both strategies generate some realistic variations of the data, using the encoded scale weights improves dientity-preservale of the transformed output. The augmentations with the encoded scale weights are better at maintaining the identity of the sampled points.
8

---

Under review as a conference paper at ICLR 2022
Opesator
Operator 2
Opecator
Operskor
(a)
(b)
Figure 5: (a) Visualizations of the encoded coefficient scale weights. The leftmost image shows an Isomap embedding of the latent vectors with input images overlaid. The scatter plots on the right show the same Isomap embedding colored by the encoded coefficient scale weights for several operators (yellow indicates large scale weights and blue small scale weights). We see operators whose use localized in regions of the manifold space. (b) The average coefficient scale weights for each class on each transport operator for MNIST. High scale weights for given operator (yellow) indicate can be applied to a given class without easily changing identity. The images on the right show examples of the operators applied to classes with high encoded scale weights (in the top yellow boxes) and classes with low encoded scale weights (in bottom blue boxes). The examples with low coefficient scale weights change classes more easily than other examples.
Encodad Seak
Weights
9
9
9
9 9
9
8
9
8
8
8
8
8
9
9
9
4
8
8
8
8
8
8
Weights
Figure 6: Examples of samples generated by transport operators using coefficients sampled with encoded scale weights (top row) and with fixed scale weight (bottom row). Images in the green box are the input images and the remaining images in each row are transformed outputs.
Because the coefficient encoder uses a pretrained classifier to determine the coefficient distributions in the latent space, the resulting encoded scale weights can inform about the types of manifold transformations the classifier is invariant to. Fig. 5b shows the average coefficient scale weights for each class (rows) and each transport operator (columns) for MNIST. From this we can identify classes for which the classifier is both sensitive and robust to transformations, represented by small and large scale weights respectively. We can also examine which classes share the use of the same transformations. The images to the right in Fig. shos transport operators being applied to samples with high encoded scale weights (in a yellow box) and samples with low encoded scale weights (in blue box). We can examine the characteristics of transformations that are better suited to some classes than others. For instance, operator 3 in Fig. 55) increases the curve at the bottom of digits. This is natural transformation for classes 3, 5, and 6 which all have higher coefficient scale weights for this operator, but when this is applied to a 1, that makes it look like an 8.
5 CONCLUSION
In this work we develop methods improve the effectiveness and utility of Lie group operator models for learning manifold representations of datasets with complex transformations that cannot be labeled. We do this by introducing a perceptual point pair selection strategy for training operators and by developing a method that uses a pretrained classifier to learn local regions where opera- tors are likely to be used while preserving the identity of transformed samples. We demonstrate the efficacy of our approach in learning natural dataset variations with the MAE. While this is powerful model, users should mindful of the biases that are introduced through training with spe- cific supervision methods when drawing conclusions about learned data transformations. This work presents a promising technique for learning representations of natural data variations that can im- prove model robustness. In future work we can address some limitations by expanding this work to consider methods for localizing manifold structure with limited or no class labels and improving the reconstruction fidelity by applying the model within more complex generative model embeddings.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg, Latent space oddity: on the curvature of deep generative models. In International Conference on Learning Representations, 2018.
Georgios Arvanitidis, Soren Hauberg, Philipp Hennig, and Michael Schober. Fast and robust short- est paths on manifolds learned from data. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1506-1515. PMLR, 2019.
Georgios Arvanitidis, Bogdan Georgiev, and Bernhard Schölkopf. A prior-based approximate latent riemannian metric. arXiv preprint 1XXi:2103.002900 2021.
Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. arXiv preprint arxiv: lel:::442444800 4864,, 2014.
Amir Beck and Mare Teboulle. A Fast Iterative Prrinkage-Thresholddin Algorithm for Linear
Inverse Problems. SIAM Journal on Imaging Sciences, 283):183- January 2009. ISSN
1936-4954. doi: 10.1137/080716554. URL http:://ppbb.. aa....m.o.. org. wwtp:ooggooooooooooooooo 1137/ 08C716542
Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation, 15(6):1333-1396 2003.
Yoshua Bengio and Martin Monperrus. Non-local manifold tangent learning. Advances in Neural Information Processing Systems, 17:199-136, 2005.
Yoshua Bengio, Olivier Delalleau, and Nicolas Le Roux. The curse of dimensionality for local kernel machines. Techn. Rep, 1258:12, 2005.
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. In AAAI Conference on Artifi- cial Intelligence, volume 32, 2018.
Stephen P. Boyd and Lieven Vandenberghe. Convex optimization. Cambridge University Press, Cambridge, UK New York, 2004. ISBN 978-0-521-83378-33
Nutan Chen, Alexej Klushyn, Richard Kurle, Xueyan Jiang, Justin Bayer, and Patrick Smagt. Met- rics for deep generative models. In International Conference on Artificial Intelligence and Statis- fics, pp. 1540-1550. PMLR, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets, 2016.
Brian Cheung, Jesse A. Livezey, Arjun K. Bansal, and Bruno A. Olshausen. Discovering hidden factors of variation in deep networks. In Yoshua Bengio and Yann LeCun (eds.), International Conference Learning Representations, ICLR Workshop Track Proceedings, 2015.
Taco Cohen and Max Welling. Learning the irreducible representations of commutative lie groups In International Conference on Machine Learning, pp. 1755-1763. PMLR, 2014.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International confer- ence on machine learning, pp. 2990-2999, PMLR, 2016.
Taco Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns. In International Conference on Learning Representations, 2018.
Marissa Connor and Christopher Rozell. Representing closed transformation paths in encoded net- work latent space. In AÁAI Conference on Artificial Intelligence, volume 34, pp. 3666-3675, 2020.
10

---

Under review as a conference paper at ICLR 2022
Marissa Connor, Gregory Canal, and Christopher Rozell. Variational autoencoder with learned latent structure. In International Conference on Artificial Intelligence and Statistics, pp. 2359 2367.
PMLR, 2021.
Romain Cosentino, Randall Balestriero, Richard Baraniuk, and Behnaam Aazhang. Deep autoen- coders: From understanding to generalization guarantees, 2021.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 113-123, 2019.
Benjamin Culpepper and Bruno A Olshausen. Learning transport operators for image manifolds. In Advances Neural Information Processing Systems, pp. 423 431, 2009.
Peter C Dodwell. The lie transformation group model of visual perception. Perception & Psy- chophysics, 34(1):1-16, 1983.
Piotr Dollár, Vincent Rabaud, and Serge J Belongie. Learning to traverse image manifolds. In Advances in Neural Information Processing Systems, pp. 361-368, 2007.
Luca Falorsi, Pim de Haan, Tim R Davidson, and Patrick Forré. Reparameterizing distributions on lie groups. In The 22nd International Conference on Artificial Intelligence and Statistics, pp- 3244-3253, PMLR, 2019.
Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Society, 29(())983 1049, 2016.
Manuel Gil. On Rényi divergence measures for continuous alphabet sources. PhD thesis, Citeseer, 2011.
Jihun Ham and Daniel D. Lee. Separating pose and expression in face images: manifold learning approach, 2006.
Søren Hauberg, Oren Freifeld, Anders Boesen Lindbo Larsen, John Fisher, and Lars Hansen. Dreaming more data: Class-dependent distributions diffcomorphisms for learned data aug- mentation. In Artificial Intelligence and Statistics, pp. 342-350. PMLR, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition, 2015.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning, 2020.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick,
Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basie visual concepts with constrained variational framework. ICLR, 2(5):6, 2017.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv: (0n:::812.000 (1...82...22222 2018.
Geoffrey E Hinton, Alex Krizhevsky, and Sida Wang. Transforming auto-encoders. Interna- tional Conference on Artificial Neural Networks, pp. 44-51. Springer, 2011.
Daniel Ho, Eric Liang, Xi Chen, Ion Stoica, and Pieter Abbeel. Population based augmentation: Efficient learning of augmentation policy schedules. In International Conference on Machine Learning, pp. 2731-2741. PMLR, 2019.
William C Hoffman. The lie algebra of visual perception. Journal of mathematical Psychology. (1):65-98, 1966.
Qiyang Hu, Attila Szabó, Tiziano Portenier, Matthias Zwicker, and Paolo Favaro. Disentangling factors of variation by mixing them, 2018.

---

Under review as a conference paper at ICLR 2022
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pp. 694-711, Springer, 2016.
Dimitris Kalatzis, David Eklund, Georgios Arvanitidis, and Søren Hauberg. Variational autoen- coders with riemannian brownian motion priors. In International Conference Machine Learn- ing, 2020.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference for Learning Representations, 2015.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. In International Conference
on Learning Representations, 2013.
Tejas D Kulkarni, William Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convo- lutional inverse graphies network. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28, Curran As- sociates, Inc., 2015. URL htt.l: httttttssss//ppppeeeeiiiieeee/ee neurips http://.....aaaoo.o.co////////// pcppppp//pep//01///1eee////
Abhishek Kumar, Prasanna Sattigeri, and Thomas Fletcher. Semi-supervised learning with gans: Manifold invariance with improved inference. Advances Neural Information Processing Sys- lems, 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2788-2324, 1998.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. Advances in Neural Information Processing Systems, 2019.
Jianxin Lin, Zhibo Chen, Yingce Xia, Sen Liu, Tao Qin, and Jiebo Luo. Exploring explicit domain supervision for latent space disentanglement in unpaired image-to-image translation, 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild.
In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Laurens van der Maaten and Geoffrey Hinton. Visualizing Data using t-SNE. Journal of Ma-
chine Learning Research, 1:1 1:1-48, 2008. http::tttp:...... cs S.oronto..dd//hhnneet..
absps/tsne. pdf
Longbiao Mao, Yan Yan, Jing-Hao Xue, and Hanzi Wang. Deep multi-task multi-label enn for effective facial attribute classification, 2020.
Michael F Mathieu, Junbo Jake Zhao, Junbo Zhao, Aditya Ramesh, Pablo Sprechmann, and Yann LeCun. Disentangling factors of variation in deep representation using adversarial training. In Advances in Neural Information Processing Systems, pp. 5040-5048. 2016.
Xu Míao and Rajesh PN Rao. Learning the lie groups of visual invariance. Neural computation, 19 (10):2665-28) 2693, 2007.
Cuong V. Nguyen. Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: A new measure to evaluate transferability of learned representations, 2020.
PACE. Partnership for Advanced Computing Environment (PACE), 2017. URL http://wwwg pace www.chttecc....o...m edu
Neal Parikh and Stephen Boyd. Proximal algorithms. Found. Trends Optim., 1(3):127-239, January 2014. ISSN 2167-3888. doi: 10. 10.1561/2400000003. URL httpps///oo...tt//111.555 24C0000003]
Mijung Park, Wittawat Jitkrittum, Ahmad Qamar, Zoltán Szabó, Lars Buesing, and Mancesh Sahani. Bayesian manifold learning: The locally linear latent variable model (II-Ivm). In Advances in Neural Information Processing Systems, pp. 154-162, 2015.
Rajesh PN Rao and Daniel L Ruderman.
Learning lie groups for invariant visual perception. In Advances in neural information processing systems, pp. 810-816, 1999.
12

---

Under review as a conference paper at ICLR 2022
Scott Reed, Kihyuk Sohn, Yuting Zhang, and Honglak Lee. Learning to disentangle factors of variation with manifold interaction. In International Conference on Machine Learning, pp. 1431- 1439. PMLR. 2014.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows, 2016.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International Conference on Machine Learn- ing. pp. 1278 1286. PMLR, 2014.
Salah Rifai, Yann N Dauphin, Pascal Vincent, Yoshua Bengio, and Xavier Muller. The manifold tangent classifier. Advances in Neural Information Processing Systems, 24:2294-2300 2011a
Salah Rifai, Grégoire Mesnil, Pascal Vincent, Xavier Muller, Yoshua Bengio, Yann Dauphin, and Xavier Glorot. Higher order contractive auto-encoder. In Joint European conference on machine learning and knowledge discovery in databases, pp. 645-660. Springer, 2011b.
Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive auto- encoders: Explicit invariance during feature extraction. In International Conference on Machine Learning, 20116 lc.
Sam T Roweis and Lawrence Saul. Nonlinear dimensionality reduction by locally linear embed- ding. Science, 290(5500):2323 2326, 2000.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 122222,, 2015.
Hang Shao, Abhishek Kumar, and P Thomas Fletcher. The riemannian geometry of deep generative models. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pp. 315-323, 2018
Jascha Sohl-Dickstein, Ching Ming Wang, and Bruno Olshausen. An unsupervised algorithm for learning lie group transformations. arXiv preprint 1001. 2010.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence, 2020.
Steven Strogatz. Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chem- istry and Engineering. Westview Press, 2000.
Joshua B Tenenbaum and William T Freeman. Separating style and content with bilinear models. Neural computation, 12(6): 1247-123, 2000.
Joshua B Tenenbaum, Vin De Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. Science, 290(5500):2319 2323, 2000.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mniss a novel image dataset for benchmark- ing machine learning algorithms, 2017.
Jimei Yang, Scott E Reed, Ming-Hsuan Yang, and Honglak Lee. Weakly-supervised disentangling with recurrent transformations for 3d view synthesis. Advances in Neural Information Processing Systems, 18:1099-11077 2015.
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.066: 2015.
13

---

Under review as a conference paper at ICLR 2022
SUPPLEMENTARY MATERIAL
A COFFFICIENT ENCODER DETAILS
E
V
2
(o(c|2)
Tw (c)z +
r(.)
y
Figure 7: System diagram for learning local coefficient statistics.
Our motivation in training an encoder to learn the coefficient statistics to understand the regions of the manifold in which specific transport operators are applicable and the invariances of pre- trained classifier to the learned transport operator-inducee transformations. In this section, we will outline derivation of our coefficient encoder training objective from variational inference per- spective by learning variational posterior the coefficients using deep neural network (Kingma) & Welling 2013 Rezende al. 2014). Given observations (z,, ys) for i 1, I..... N, and pre- trained classifier r(-). we aim to learn a distribution 9s (c(z)) 2 from which we can sample to define the transformation Tw (c) that be applied to z while maintaining class identity. In other words, want the augmented ğ = Tv (c)z + to result r()) y-
To encourage identity-preserven preservation of vectors transformed with transport operators that trolled by sampled coefficients, we maximize the likelihood of a class output for an observation under the system model diagrammed in Figure 7 This system connects the observed latent vector to an output through a transformation defined by c. We follow the derivation used in Rezende & Mohamed (Rezende & Mohamed] 2016) which introduces the variational posterior estimate the log likelihood of the data.
Consider parameterized distribution for the conditional likelihood of our observations:
log po z) log (c) [po c.
(6)
log Je
pc(c)po(yle. 2)d
(7)
/. c0(clz)
Pc(c)
log
pe (ylc, z)de
(8)
(o(c)))
Pc(c)
(clz))) log
po c, z) de
(9)
[
Egs [log Pa (yl c, 2)) DRL (1o (co(clz))p((c))
(10)
Egg, [log E.po 2) DEL (c)(c)))))((e))
(11)
In equation 6 and equation 7 we marginalize over the coefficients с as required under our system model in Figure In equation we lower bound the conditional likelihood with variational lower bound derived from Jensen's inequality. Finally, we use the reparameterization trick define 2 as deterministic function of c, and the parameter-free random variable in equation []
To sample from (c
when computing the expectations, we first use the reparameterization
trick (Connor et al. 2021 Kingma & Welling 2013 Rezende et al. 2014) to define the sampled
M
coefficients Č as function of a uniform random variable
Unif -
é = (u, z) (z) sgn(u) log(1 2uu)),
(12)
where 1, is defined mapping from uniform distribution Laplace distribution and the Laplace scale parameters are defined by the output of the coefficient encoder hs that we aim to learn
Using the reparameterization of è and 2, can define the expectation:
14

---

Under review as a conference paper at ICLR 2022
Egg [log E,poly z, c, 2)]
Eu [log E.po (ylc = lolu, c,
(13)
K
log
Po
(ylob)= ( -lo(u()),),
(14)
JK
k=1
Our pre-trained classifier defines which allows us to specify final objective using of r(-). Using KL-divergence the likelihood function between the ground truth labels y, and the classifier output for the augmented inputs r(2 and simplifying the model by setting c to O, we get:
(g)l)())z)) 2 DEL (uir(z?")) - DKL (co(c)z)lp<cc))
(15)
where in practice we use single sample DrL. (r(! for test samples for which we may not have class labels. These practical simplifications result in the objective in which we want to minimize. The KL divergence between the qs (c/z) and (c(c) has closed form expression (Gill 2011):
and we estimate DEL (vil((z!))) ~
S -1
DKL (1olc (c)) log(ha(z)) log(S) +
(16)
hz(z)
B COEFFICIENT INFERENCE DETAILS
Time Taken for Coefficient Inference
Batch Size 100. Trials 30
303.63 BBC
100
Conj. Grad. (CPU)
Grad. (CPU)
Subgrad. (CPU)
Time Taken (secs)
80
Prox, Grad, (GPU)
Subgrad. (GPU)
60
40
20
4,03
Optimization Method
Figure 8: Comparison in time taken for various coefficient inference algorithms on fully trained transport operators. Each inference trial is performed over random batches of 100 point pairs drawn from the CelebA dataset (with a latent dimension of 32). 30 trials are run with the same Co and tolerance. We used the proximal gradient (GPU) method.
To perform coefficient inference, previous Lie group operator methods (Connor Rozell 2020
Connor et al. 2021) use the analytic gradient proposed in (Culpepper & Olshausen, 2209) with conjugate gradient descent solver. This method requires an eigenvalue decomposition for computing the gradient which is not favorable for parallel computations. Alternatively, we use the PyTorch implementation of the matrix exponential that allows for automatic differentiation. Furthermore, to handle the non-smooth & norm in objective (2). we apply proximal gradient step in forward- backward splitting scheme (Beck & Teboulle 2009). For the & norm, the proximal gradient has a known, closed-fome form solution in the tort-thresoodding function (Parikh & Boyd 2014):
T (c) = sign(c) * max (1c) - 2,0)
(17)
15

---

Under review as a conference paper at ICLR 2022
f(() be numerical approximation to the gradi- ent of the lg term, found through automatic differentiation. Given initial coefficient values Co drawn from an isotropic Gaussian with variance 4 x 10 10-4, our gradient descent step is:
Let Vo2 |2 expm Em=i
Cm
Zo
Ck+1 = Tcax (c* - a V((c))
(18)
These steps are iterated upon until either max iteration count is reached, or the change in coeffi- cients, - falls below some threshold. For all experiments we use a max iteration count of 800 and a tolerance of 1 x 10-5. For our step-size, we use (tg (0.985)* og with ag x 10-2, We experimented with different acceleration methods (Kingma & Ba 2015 Beck & Teboulle 2009). and found that in many cases they resulted in worse performance.
The main parameter to select for inference is the sparsity-inducing parameter on the coefficient prior, S. This value is sensitive to the dataset. latent dimension, and 7 we fix a y value and pick Ç to be as high as possible, inducing sparsity in the inferred coefficients, while maintaining good reconstruction performance of 21. If Ç is too large, that will result in all coefficients going to zero, preventing proper reconstructive performance. On the other hand, setting C to be too small will result in all operators being used to represent the transformation between each point pair, preventing any meaningful structure from being learned during training. Parameters used in specific training runs are included in their respective Appendix sections.
value. In practice,
When compared against the coefficient inference implementation from (Connor & Rozell 2020), we perform inference for a batch of 100 samples in an average of 1.67 seconds over 100 trials whereas the inference from (Connor& & Rozell 2020) took an average of 101.01 seconds. We also compare the speed-up over a baseline that strictly uses automatic differentiation (subgradients) and compare the benefits from moving to GPU hardware in Figure We use the proximal gradient (GPU) method from this figure. Experiments were run on a machine with an Intel i7-6700 CPU with 4.00 GHz and a Nvidia TITAN RTX.
C TRAINING STRATEGY
In experiments, we follow the general training procedure put forth previously (Connor & Rozell, 2020). We train the MAE with three training phases: the autoencoder training phase, the trans- port operator training phase, and the fine-tuning phase. During the autoencoder training phase, the network weights are updated using a reconstruction loss objective: E = ||x x1|2
During the transport operator training phase, the network weights fixed and the transport oper- ators are trained between pairs of points using the objective (2). Pairs of images Zig 21 are chosen using the perceptual point pair selection strategy described in Section 33. The images are then en- coded into the latent space 20, 21. For each batch, the first step is to infer the coefficients between all pairs of latent vectors. Coefficient inference is best performed when the entries of the latent vectors are close to the range [-1, 1]. Because of this, we define scale factor that can be applied to encoded latent vectors to reduce the magnitude of their entries prior performing coefficient inference. In practice, we inspect the latent vector magnitudes after the autoencoder training phase and choose a scale that will adjust the magnitudes of the latent vector entries to be the range [-1, 1]. This does not have to be precise range for the latent vector magnitudes but instead is a practical guideline. Coefficient inference is performed on the scaled latent vectors as described in Sections [3.3 and Ap- pendix B After the coefficients are inferred for batch, the weights on the dictionary elements are updated. This phase of training is performed until the loss values reach plateau and the dictionary magnitudes plateau.
The fine-tuning training phase begins after the transport operator training phase is complete. In this phase, both the network weights and the transport operator weights are updated using the joint objective:
E E= = [xxo x011 + ||| - x1112) + (1 - A) Ev.
(19)
where Eg is defined in 22. Using this objective, we alternate between taking steps the transport operator weights while the network weights are fixed and taking steps on the network weights while the transport operator weights are fixed. Additionally, during the fine-tuning phase we incorporate
16

---

Under review as a conference paper at ICLR 2022
occasional steps in which update the network weights using only the reconstruction loss to ensure
effective image reconstruction.
In most cases, it is necessary to reduce the parameter in front of the Frobenius norm dictionary regularizer prior to fine-tuning or the dictionary magnitudes will reduce to zero. We report the n we use for transport operator training and fine-tuning in the experimental details sections below. It may also be necessary to decrease the network learning rate during ane-tuning.
The coefficient encoder training requires network that is trained to classify data from selected dataset. We train this classifier using training data from a given dataset. For datasets with worse autoencoder reconstruction quality, train the classifier in the latent space. Otherwise classifier on images in the data space. With the MAE and classifier trained, we train the coefficient encoder network following the strategy described in Section 3.2 and Appendix A
train the
To train the CAE, we use the same autoencoder architecture used with the MAE with the addition of Frobenius norm regularizer on the encoder Jacobian, weighted by a selected X value (different from the À in equation 19). The Jacobian is computed using Py PyTorch automatic differentiation. We find the Jacobian norm decreases to the same value irrespective of our choice of X, leading us to choose À 1. For the 3-V we use the same architectures outlined in (Higgins et al. 2017) with for CelebA and /3 for MNIST and FMNIST, We find that setting any higher results in poor reconstructive performance. Scripts for training both comparison methods are included in the code repository.
Hyper-paramete tuning for all experiments was performed on the Georgia Tech Partnership for Advanced Computing Environment (PACE) clusters (PACE) 2017). Experiments were performed using a Nvidia Quadro RTX 6000. Runs training the CAE, and V-V on CelebA were all run separate machine with a Nvidia TITAN RTX.
D PARAMETER SELECTION
The MAE model has several hyperparameters that must be tuned and we will provide guidance determining ideal parameter values for our experiments and future experiments. First we will describe some signs to look out for to identify if run is succeeding or failing. One indicator that we compute is the transport operator difference which is:
1
1
Eveain
2 21 expm
Σ
T
m Cm
Zo
21
expm
TT
Cm
Zo
(20)
2
where T is the dictionary after the gradient step is taken. A gradient step should decrease the transport operator objective meaning the Exain value should be positive for an effective step. there are many gradient steps that result in negative Evain values, that indicates that the parameters are not optimal or that the learning rate is too large. This is an important metric to observe during fine-tuning determine whether to select a smaller y or smaller network learning rate.
Signs of failure of a training run with selected training parameters:
All operator magnitudes reduce towards zero.
All inferred coefficients between point pairs are zero.
Most of the operators generate transformation paths with latent values that increase quickly
to infinity.
Many steps have negative values for Fvaiit
The operator magnitudes increase which results in unstable training steps with NaN values in the computed objective.
Dictionary regularizer parameter The dictionary regularizer parameter y is the weight on the Frobenius nomn term in equation 2 This objective term serves two purposes. First, balances the effect of the coefficient sparsity regularizer with the parameter Ç. If ( is large, the sparsity regularizer
17

---

Under review as a conference paper at ICLR 2022
encourages small coefficient magnitudes and one way achieve that while still effectively inferring paths is to increase the magnitude of the operators. The Frobenius norm term must have a large enough influence to counterbalance this force the operators will increase in magnitude to the point of being unstable. If a run is becoming unstable, we recommend decreasing ( or increasing 7-
The second purpose of the dictionary regularizer term is to identify which operators are necessary to represent the transformations on the data manifold. If an operator is not being used to represent transformation between Zo and 2i then the dictionary regularizer reduces its magnitude to zero. Therefore, during training we are able to estimate the model order based on how many dictionary elements remain non-zero. In our tests we vary y between x 10 and x 10 If is too small, it will not counterbalance the coefficient sparsity term and the dictionaries will grow to unstable magnitudes. If y is too large will reduce the magnitude of all operators to zero. During the fine- tuning steps, we have often found it necessary to reduce the Y because, with larger % both the operator magnitudes and the latent vector magnitudes can decrease substantially which leads to an ineffective manifold model.
Coefficient sparsity parameter The coefficient sparsity parameter in equation 2 controls the sparsity of the coefficients that are used to estimate paths between 2n and Z1. We run tests with values of Ç between 0.005 and 2. From dataset dataset the ideal value varies. If values are too small then all of the operators are used to represent all of the paths between point pairs. The Ç should be increased so fewer than M coefficients are used for each inferred path. When Ç is too large, all the coefficients go to zero during inference. This means there is no path inferred between Zo and z1 because of overweighting the sparsity constraint.
Number of dictionary elements As mentioned above, the dictionary regularizer acts as a model order selection tool so our strategy for selecting number of dictionary elements M is to increase the number of dictionary elements until some of their magnitudes begin reducing to zero during training. This indicates that some of the operators not necessary for representing transformations.
Latent dimension The latent dimension of the autoencoder is selected to ensure quality recon- structed image outputs.
Relative weight of reconstruction and transport operator objectives The parameter À deter- mines the weight of the reconstruction term relative the transport operator term. We observe good performance of the model for X between 0.5 and 0.75.
E COMPARING POINT PAIR SELECTION STRATEGIES
In the past, transport operators have been trained using point pairs that were selected randomly from the same class (Connor et al. (02T) with some knowledge of transformation labels (Connor & Rozell, 2020). In this work, we establish a method that learns natural transformations without re- quiring transformation labels using a perceptual point pair selection strategy. The perceptual loss metric (which compares the the features values from the penultimate layer of a pretrained classi- fier model) enables us to select point pairs that may share semantically meaningful transformations without being exactly the same. Fig. 9 shows examples of nearest neighbors selected using pixel similarity, similarity the autoencoder latent space, and the perceptual loss metric. This highlights how the perceptual loss metric can be useful for identifying inputs with similar qualitative charac- teristics. For instance, in the of images on the left, the perceptual loss metric identifies another bald man as a nearest neighbor which has similar characteristics of the initial image even though the exact image looks quite different.
When examining the success of learned operators, one characteristic we care about is how well the operators maintain the stability of generated paths. We consider generated paths stable the latent vectors do not expand quickly to infinity. We have defined this as a useful characteristic for of applied operators if operators expand latent vectors to infinity, that will very likely lead change in identity. Each of the operators can be viewed as the dynamics matrix of a continuous time linear dynamical systems model and we can analyze their stability as dynamical systems by observing their eigenvalues (Strogatz] 2000).
18

---

Under review as a conference paper at ICLR 2022
Pixel NN
Latent NN
Perceptual NN
Figure 9: Examples of nearest neighbors identified through pixel similarity, latent similarity, and the perceptual loss metric. In each set of 12 images, the images in the first column (contained in green boxes) are the initial reference images and the images to the right of those show three selected nearest neighbors. The perceptual loss metric identifies neighbors that share similar characteristics, like hair style, without being exactly the same.
In continuous time linear dynamical systems, there are three classifications of system stability. If a system is stable, the real parts of all eigenvalues are below zero. In stable system, the magnitude of an input shrinks as time progresses forward. If system is unstable, the real part of at least one eigenvalue above zero. In unstable systems, the magnitude of an input expands to infinity as time progresses forward. Finally, if a system is marginally stable, then the magnitude of an input remains constant and the real parts of all of its eigenvalues are zere"] (Strogatz, 2000). The transport operator model is unique because operators can represent dynamics matrices that define the natural data variations which do not have the same directionality as temporal systems (where time moves forward). Therefore transport operators can be applied with positive or negative coefficients and the traditional view of stability of temporal systems no longer applies.
To maintain the stability of generated paths, the operators need generate cyclic transformation paths that neither increase nor decrease the latent vector magnitudes. This corresponds to marginal stability in linear dynamical systems, marginally stable system has only imaginary eigenval- ues with no real parts (Strogatz) (000)). Therefore, we can identify transport operators approach- ing marginal stability by investigating the magnitudes of the real parts of their eigenvalues. The maximum real magnitude of an eigenvalue of a dynamics matrix drives the speed with which the generated paths expand to infinity. Therefore this maximum real magnitude is the focus of our inves- tigation into the stability of paths generated by learned operators. When the maximum magnitude of the real part for all eigenvectors associated with an operator is close to zero, that indicates that the operator is closer to marginal stability. Therefore, we quantify the stability of transport operator generated paths by looking at the maximum magnitude of real parts of eigenvalues associated with each transport operator. Fig. T0a the sorted maximum magnitude of the real parts of eigenval- ues in each of the 16 operators learned in the experiment in Section 4.1 whhic learns natural MNIST variations. The blue x's associated with the operators trained using the perceptual point pair selection strategy and the red dots are associated with the operators trained using a simple strategy of selecting point pairs as nearest neighbors in the latent space. With the exception of one operator, all the operators trained using latent space similarity for supervision have larger maximum magni- tudes of real eigenvalue components than the operators trained using the perceptual loss metric. This indicates that the operators trained using the latent space similarity to select training point pairs are farther from marginal stability and can be seen as less stable by our definition of transport operator stability.
To view the effect of these learned operators more intuitively, we plot the values of the latent di- mensions as individual operators are applied in Fig. II Each line in these plots shows the influence of the operators on a single latent dimension (Note that this experiment utilizes a latent dimension of 10). Fig. II shows the paths generated by transport operators trained with the perceptual point pair selection strategy The plots with straight lines (i.e., operators 3, 4, 7, 12, 14, and 16) indicate that those operators had their magnitudes reduced to zero during training and therefore they have no impact on the latent vectors when applied. Most of the operators learned using the perceptual point pair selection strategy are close to cyclic except for operator 2 (the operator with the large real com- ponent magnitude in Fig. 10a). By contrast, several of the operators trained with point pairs selected
Note that a system can also be marginally stable with at least one pair of eigenvalues with zero real part and non-zero imaginary parts and the rest of the eigenvalues with non-positive real parts. For simplification of this analysis, we will focus on striving for systems which have eigenvalues with all real parts equal to O.
19

---

Under review as a conference paper at ICLR 2022
Max Real Value Magnitude
2
Perceptual Latent NN
Perceptual
Latent NN
1.5
0.5
Imaginary
0
0.5
-0.5
-1
5
10
15
-1
2
Transport Operator Number
Real
(a)
(b)
Figure 10: Analysis of eigenvalues of operators learned in the MNIST experiment. (a) The maxi- mum absolute value of the real parts of eigenvalues computed from each learned operator. (b) Plot of the real and imaginary parts of the eigenvalues for each operator.
as nearest neighbors the latent space extend to infinity (Fig. Iib.. This can explain the larger maximum real value magnitudes in Fig. 10a This analysis leads us to conclude that the perceptual point pair selection strategy has a greater potential to yield
transport operators.
2
5
2
2
0
0
10
10
11
0
10
-10
10
-10
10
-10
0
10
-10
0
10
-10
10
110
10
-10
10
10
10
-10
-10
0
10
11
10
10
10
Coafficient Values
(a)
2
3
5
2
2
-10
0
10
10
0
10
-10
10
-10
10
10
0
10
10
0
10
10
10
10
10
10
10
10
10
-10
10
110
10
10
10
10
Coafficient Values
(b)
Visualizations of the effect of each learned transport operator on each dimension of an encoded latent vector. In each plot, each line represents a single latent dimension and the coefficient magnitude of the transformation varies on the x-axis. (a) Paths from operators learned with per- ceptual point pair supervision. (b) Paths from operators learned with point pairs selected as nearest neighbors in the latent space.
Figure 11:
F MNIST EXPERIMENT DETAILS
The MNIST dataset is made available under the terms of the Creative Commons Attribution-Shan Alike 3.0 license. We split the MNIST dataset into training, validation, and testing sets. The training set contains 50,000 images from the traditional MNIST training set. The validation set is made up of the remaining 10,000 images. The traditional MNIST testing set is used for our testing set. The input images
normalized their pixel values between 0 and 1. The network architecture used for the autoencoder is shown in Table I The training parameters for the transport operator training phase and the fine-tuning phase shown in Tables
and
20

---

Under review as a conference paper at ICLR 2022
Prior to training the coefficient encoder for the MNIST dataset, train classifier the labeled MNIST image data which we use to encourage training. The training parameters for the coefficient encoder shown in Table The image classifier we use based on the simple LeNet architecture with two convolutional layers and three fully connected layers (LeCun al. 1998).
during coefficient encoder
Table I: Network Architecture for MNIST and Fashion MNIST Experiments
Encoder Network
Decoder Network
Input E IR28x28
Input E R²
conv: chan: 64, kerm: stride: 2, pad:
Linear: 3136 Units ReLU
BatchNorm: feat: 64
ReLU
conv connToaap chan: 64, kern: 4, stride: 1, pad: 1
conv: chan: 64, kerm: 4, stride: 2, pad:
BatchNorm: feat: 64
BatchNorm: feat:
ReLU
ReLU
conv Tranpooaps chann: 64, kern: 4, stride: 2, pad: 2
conv: chan: 64, kerm: 4, stride: 1, pad:
BatchNorm: feat: 64
BatchNorm: feat: 64
ReLU
ReLU
conv Tranpooe: chan: 1, kernel: 4, stride: 2, pad:
Linear: Units
Sigmoid
Table 2: Training parameters for the transport operator training phase of the MNIST experiment MNIST Transport Operator Training Parameters
batch size: 250
autoencoder training epochs: 300
transport operator training epochs: 50
latent space dimension (zdimm): ): 10
M : 16
Innet 10-4
Irv 10-3
0:0.1
7:2 10-6
initialization variance for V: 0.05
number of restarts for coefficient inference: 1 nearest neighbor count: 5
latent scale: 30
Table 3: Training parameters for the fine-tuning
phase of the MNIST experiment
Table Training parameters for the
MNIST Fine-tuning Parameters
MNIST Coefficient Encoder MNIST Coefficient Encoder Parameters batch size: 250
batch size: 250
transport operator training epochs: 100 o
Iruet 10-4
training epochs: 300
Ing 10-3
Ir: 10-3
ç: 0.1
Sprior 0.1
7 2 x 10-6
ARt 0.5
X: 0.75
coefficient spread scale: 0.1
number of network update steps: 50
classifier domain: image
number of y update steps: 50
G MNIST EXPERIMENT ADDITIONAL RESULTS
Here we show additional experimental details and results for the MNIST experiment. Fig. 12 shows the magnitude of all 16 operators after the fine-tuning phase. Six of the operators have their magni- tudes reduced zero. Fig. 13 shows the paths generated by transport operators trained on MNIST data.
21

---

Under review as a conference paper at ICLR 2022
Dictionary Magnitude
3
2
1
Transport Operator Number
Figure 12: The magnitudes of the learned operators after fine-tuning.
Transport Operator
1
Transport Operstor
Transport Operator 5
Transport Operator 6
D
O
O
0
O
O
O
O
3
7 /
4
8
A
/
/
8
7
1
/
f
f
2
2
2
2 2
2 7
2 2
2
2 3
2
2
2
2
3
7
3 4
3 3
D 3 3
7
3
3
3
2
3
3
3 4 5
4440
4444
5
5555
5
15555
5
555
5555 5 5
5
5 5
5555
65
5555
5
5
5
5
0
6
6
6
6
6 6
6
6
6
6 6
6
6
6
6 6
6
6
7
7
7
7
7
7
7
7
7
77
7
7 8
2
2
8
8
9
3 9
8
8
8
V
V
8
8
8
8
2
8
T
7
9
9
4
4
4
4 (d)
9
7
(c)
D
D
O
O
O
O
D
D
O
O /
O
0
O /
0
O
O
O
b b /
1
/
2 2
2
2
Z
2
2
2
2
2
2
2
2
2 3
2 2
2 22 2 2 2 2 2
7 7
3
3
3
3
2 2 3
3 3 3
4
4
4 4
4
4
4
4
4
4
4
4
4
4
4 4444449
555
5
5
5
55
5
5
5
5
5 5
5
5
5555
5 5
6 6
6
G
6
6
6 7
6 6
6
7 7
7
7
7
7 7
7 7
7
7
R
8
8
9
9
8 7
8
8 9
&
8
<
9
9
9
5
9
9
4
4 4
9
4
9
9
9
9
(e)
(h)
Transport Operator
Transport Operater 15
o
F
1
1
1
2
2
2
2
2
2
2
2
2
2
2
2
3
3
3
3
3 3
Z
2
3
3
>
4
4
4
4
44 5
4
4
4
4
5
5
5
5
5
5
5
5
5
6
6 77 8
6
6
6
6
6
7
1
7
7 8
7 7
8
8
R
R
8
9 9
R 4
9
9
i
Paths generated by all non-zero transport operators trained the MNIST dataset. Im- ages in the middle column the image block are the reconstructed inputs and images to the right and left are images decoded from transformed latent vectors in positive and negative directions, respectively
Figure 13:
22

---

Under review as a conference paper at ICLR 2022
H FASHION-MNIST EXPERIMENT DETAILS
The Fashion-MNIST dataset is made available under the terms of the MIT license. We split the Fashion MNIST dataset into training, validation, and testing sets. The training set contains 50,000 images from the Fashion-MNIST training set. The validation set is made of the remaining 10,000 images. The traditional Fashion-MNIST testing is used for testing set. The input images are normalized so their pixel values are between and 1. The network architecture used for the autoencoder the same as in the MNIST experiment and shown in Table [ The training parameters for the transport operator training phase and the fine-tuning phase are shown in Tables and
Prior to training the coefficient encoder for the Fashion-MNIST dataset, we train a classifier the latent vectors associated with labeled FasHion-MNIST data which we use to encourage identity- preservation during coefficient encoder training. The training parameters for the coefficient encoder are shown in Table 7 The latent vector classifier has simple architecture of Linear(512), ReLU, Linear Softmax.
Table Training parameters for the transport operator training phase of the Fashion-MNIST exper- iment
Fashion- Transport Operator Training Parameters
batch size: 200
autoencoder training epochs: 300
transport operator training epochs: 50
latent space dimension (Zdim): 10
M 16
Iraet 10
Iny 10-3 0.5.
7 7:2 x 10-5
initialization variance for V:: 0.05 number of restarts for coefficient inference: nearest neighbor count:
latent scale: 30
Table 6: Training parameters for the fine-tuning
phase of the Fashion Fashion-MNIST MNIST experiment
Table 7: Training parameters for the Fashion-
Fashion-MNIST Fine-tuning Parameters batch size: 200
MNIST Coefficient Encoder
Fashion-MNIST Coefficient Encoder Parameters
transport operator training epochs: 150
batch size: 200
Iruet : 10-4
training epochs: 300
Iry 10-3
Ir: 10
Ç 0.5
Sprior : 0.5
: x 10-6
Aat 0.5
A: 0.75
coefficient spread scale: 0.1
number network update steps: 50
classifier domain: latent
number of V update steps: 50
FASHION MNIST EXPERIMENT ADDITIONAL RESULTS
Here we show additional experimental details and results for the Fashion-MNIST experiment. Fig. [4 shows the magnitude of all 16 operators after the fine-tuning phase. Six of the operators had their magnitudes reduced zero.
To visualize how the use of the transport operators varies over the latent space, we generate an Isomap embedding (Tenenbaum et al. 2000) of latent vectors and color each point by the encoded scale parameter for coefficients associated with each of the transport operators. Fig. [5a] shows these embeddings for Aathion-MNIST data. Each operator has regions the latent space where their is concentrated. Fig. 15b shows the average coefficient scale weights for each class (rows)
23

---

Under review as a conference paper at ICLR 2022
5
Dictionary Magnitude
4
3
2
Transport Operator Number
Figure 14: The resulting magnitude of the learned operators after fine-tuning the MAE.
and each transport operator (columns) for Fashion-MNIST. There are some classes like trouser and sandal (classes I and 5) which have large encoded coefficient scale weights for most of the transport operators. This means they are robust to many natural transformations, Other classes like coat and shirt (classes and have smaller encoded coefficient scale weights which means they are sensitive to many transformations. The images to the right in Fig. show transport operators being applied samples with high encoded scale weights (in yellow box) and samples with low encoded scale weights (in blue box). Fig. [6 shows the paths generated by non-zero transport operators trained on Fashion-MNIST data.
Operstor
Operator 2
Operator 3
Operator
(a)
Encoded Coufficient Scale -ile-Fass Fashion NhiT
Operator
Operstor
ses
Duta Cloos
Operstor
Operator
Operator
(b)
Figure 15: (a) Visualizations of the encoded coefficient scale weights. The leftmost image shows an Isomap embedding of the latent vectors with input images overlaid. The scatter plots on the right show the same Isomap embedding colored by the encoded coefficient scale weights for several operators (yellow indicates large scale weights and blue small scale weights). We see operators whose use is localized in regions of the manifold space. (b) The average coefficient scale weights for each class on each transport operator for Fashion-MNIST. High scale weights for given operator (yellow) indicate it can be applied the right show examples of the operators applied to classes with high encoded scale weights (in the top yellow boxes) and classes with low encoded scale weights (in bottom blue boxes). The examples with low coefficient scale weights change classes more easily than other examples.
given class without easily changing identity. The images on
24

---

Under review as a conference paper at ICLR 2022
Operator
Operator
Transport Operator
Transport Operator
Transport Operator 11
Transport Operator
(e)
(h)
Transport Operator
Transport Operator 16
Figure 16: Paths generated by all non-zero transport operators trained on the Fashion-MNIST dataset. Images in the middle column of the image block are the reconstructed inputs and images to the right and left images decoded from transformed latent vectors in positive and negative directions, respectively
J CELEBA EXPERIMENT DETAILS
The CelebA dataset is publicly available for non-commercial research purposes. We split the CelebA dataset into training and testing sets. The training set contains the first 150,000 images accompanied with the entire test set. The input images The network architecture used for the autoencoder is shown Table g The training parameters for the transport operator training phase and the fine-tuning phase are shown in Table 9
normalized their pixel values between 0 and 1.
25

---

Under review as a conference paper at ICLR 2022
adapted from with 16 classifier heads after the layer with 512 hidden units. Each classifier head corresponds to single attribute and is modeled as fully-connected linear layer with 256 hidden units, followed by batch normal- ization, dropout, and ReLU layers. Afterwards, two logits output for 0/1 prediction for each classifier output. The training procedure, including the dynamic loss weighting, follows (Mao et al., 2020
The attribute classifier is hettps:/ggithen htg.:aa..ccc....ll/aaaaaaarrttoo
Re
model
Table 8: Network Architecture for CelebA Experiments
Encoder Network Input E R51X61
Decoder Network
Input E R232
conv: chan: 32, kern: 4, stride: pad: 1
Linear: 80,000 Units ReLU
BatchNorm: feat: 32
ReLU
conv convTranpose: chan: 256, kerm: stride: 1, pad: 0
conv: chan: 64, kern: 4, stride: 2, pad:
BatchNorm: feat: 256
BatchNorm: feat: 64
ReLU
ReLU
TonyTranpose: chann: 256, kern: 3, stride: pad: 0
conv: chan: 128, kern: 3, stride: pad:
BatchNorm: feat: 256
BatchNorm: feat: 128
ReLU
ReLU
conv Tranpos:: chan: 256, kernel: 3, stride: 1, pad:
chan: 256, kern: 3, stride: pad:
BatchNorm: feat: 256
BatchNorm: feat: 128
ReLU
ReLU
conv Tranpooe::: chan: 128, kernel: 3, stride: 1, pad:
conv: chan: 256, kern: 4, stride: 2, pad:
1
BatchNorm: feat: 128
BatchNorm: feat: 256
ReLU
ReLU
conv Tranpose: chan: 128, kernel: 3, stride: 1, pad:
conv: chan: 128, kern: 4, stride: pad:
BatchNorm: feat: 128
BatchNorm: feat: 128
ReI.U
ReLU
Tonnorrnp: chan: kernel: stride: 2, pad:
Linear: 32 Units
Sigmoid
Table 9: Training parameters for the CelebA experiment
CelebA Transport Operator Training Parameters
batch size: 500
autoencoder training epochs: 300
CelebA Fine-tuning Parameters
transport operator training epochs: 50
batch size: 500 fine-tuning epochs: 10
latent space dimension (Zdim)): 32
40
Irues 10-4
Innet 10-4
Ire 10-3
Ir 10-3
::0.8
1:5.5
7 x 10-7
7:1x -5
A: 0.75
initialization variance for V/ 0.05 number of restarts for coefficient inference: nearest neighbor count: 5
number of network update steps: 50
number of V update steps: 50
latent scale: 2
K CELEBA EXPERIMENT ADDITIONAL RESULTS
Here we show additional experimental results for the CelebA experiment. Fig. 17 shows the paths generated by the 40 transport operators trained on celebA data. Each row represents different operator acting the same input image. Images in the middle column of the image block are the reconstructed inputs and images to the right and left are images decoded from transformed latent vectors in positive and negative directions, respectively. Fig. 18 shows the classification outputs of the attribute classifier for example transport operators. These two operators that vary smiling, beard, and sunglasses attributes.
26

---

Under review as a conference paper at ICLR 2022
MUW
(d) Figure 17: Paths generated by all 40 transport operators trained on the CelebA dataset. Images in the middle column of the image block are the reconstructed inputs and images to the right and left are images decoded from transformed latent vectors in positive and negative directions, respectively
(c)
Ove loard
Yount Naki
(a)
(b)
Figure 18: Paths generated by the application two learned transport operators with the associated attribute classifier probability outputs for the transformed images. Our model learns operators that correspond to meaningful CelebA dataset attributes like (a) smiling (b) sunglasses/beard
Fig. 19 shows an interesting feature in our learned operators many of them generate cyclic paths that begin and end at nearly the same point. Also, these cyclic sequences, the transformations seem to lead to change in gender. The image sequence in Fig. [9a shows the path generated by single operator. The image in the middle is the reconstructed input image and the images to the left and right are the paths generated by negative and positive coefficients respectively. This operator changes the hairline and quantity of bangs. As we apply the transport operator with a negative coefficient (to the left of center), the woman gains bangs and then becomes man with a moustache on the far left of the image. As we apply the transport operator with a positive coefficient (to the right of center), the woman's forehead gets higher and then she becomes and eventually, on the far right the woman transforms into a man with bangs, similar to the man on the far left. The similarity between the generated images the far left and far right is notable
with high forchead
27

---

Under review as a conference paper at ICLR 2022
because this indicates the transformation path is nearly closed. Fig. 19b shows the change in five of the 32 latent dimensions over the generated path. These paths have a nearly cyclic structure.
This is particularly interesting because in (Connor & Rozell 2020) they learn closed transformation paths by selecting point pairs on known cycli transformation paths and highlight the benefit of the transport operator model for representing these types of paths. In this case, learn this eyclic path with only perceptual point pair supervision. Additionally, this identifies a benefit of the transport operator approach over other models of the manifold in a neural network latent space. We can learn nonlinear paths that keep the generated points in the same neighborhood in the latent space without extending to infinity which inevitable when linear paths that are used to represent natural transformations.
0.5
Latent Value
-0.5
Coefficient Value
(b)
An example of an operator that generates nearly cyclic path in the latent space (a) Image outputs along path generated by single learned operator. The images the far left and far right look similar which indicates that this operator generates nearly closed path that begins and ends at the same point. (b) Paths of five of the 32 latent dimensions as the learned operator is applied to them. This again highlights the cyclic nature of the transport operator generated paths.
Figure 19:
28


Reviews:
[{"review_id": "sVwKjHivnXZ", "reviewer": "Reviewer_98g7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of using features from classification networks is quite interesting. The proposed way of learning local transport also makes sense. However,  the way of MAE is an ill-posed decomposition of encoding-decoding and group actions. The additional step of combining MAE together with another encoder network for local operation makes the decomposition even more ill-posed. It would be nice to have a better explanation of why the desired decomposition (i.e. latent variables, group action, local action regions) can be obtained. ", "summary_of_the_paper": "The manuscript proposes a method for learning group action on data using features from classification networks. The idea is natural and easy to follow. ", "summary_of_the_review": "In fact, I found the proposed method is quite interesting.  A better explanation of why the proposed ill-posed decomposition can provide the desired solution would make the idea more convincing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dctImdd2vAI", "reviewer": "Reviewer_qg7U", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### Strengths\nThe authors consider the important and challenging task of learning identity preserving transformations. Much of the previous work considers physical transformations (e.g., rotations, scaling) rather than semantic transformations which is arguable more challenging due to the lack of transformation labels (consider citing exception [1]). The challenge is compounded further by the presence of multiple classes in the dataset since not all transformations make sense for all classes. The proposed solutions for these challenges is interesting and the empirical analysis shows that they are also effective.\n\n[1] Ham, Jihun, and Daniel D. Lee. \"Separating pose and expression in face images: a manifold learning approach.\" Neural Information Processing-Letters and Reviews 11.4 (2007): 91-100.\n\n#### Weaknesses\nIt is difficult for me to reconcile the use of Lie operators which assume some smoothness in the data manifold and the statement: “However, semantic transformations of interest may not be exhibited through pixel similarity nor through unsupervised autoencoder feature similarity”. Moreover, the NN samples shown in Figure 9 seem to say otherwise. Specifically, the hair style between Latent NN examples appears more similar than those of the Perceptual NN.\n\nI'm not convinced the use of CAE and beta-VAE as baselines is appropriate. Comparison against more recent work in disentangled representation learning (e.g., [2]) would be more convincing. I would have also liked to see a comparison against another approach that uses Lie operators (e.g., [1, 3])\n\nA common dataset for analysis of disentangled representation learning algorithms is [4]. I would have liked to see results on this dataset.\n\n[2] Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, Xin Tong; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 5154-5163\n\n[3] Connor, Marissa, and Christopher Rozell. \"Representing closed transformation paths in encoded network latent space.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 04. 2020.\n\n[4] Locatello, Francesco, et al. \"Challenging common assumptions in the unsupervised learning of disentangled representations.\" international conference on machine learning. PMLR, 2019.\n", "summary_of_the_paper": "The authors introduce an approach for learning identity preserving transformations from data. First the low-dimensional manifold structure of the data is learned using an autoencoding neural network, then the transformations (the Lie group operators and their coefficients of combination) that map between perceptually similar image pairs are learned. The main contribution of the paper is that the learned operators can transform the data semantically. This is a challenging since 1) semantic transformation labels are not usually available; and 2) not all semantic transformations make sense for all elements of the dataset. The authors address these challenges using two auxiliary networks. The first identifies perceptually similar image pairs in the dataset and the second identifies where particular transformations are likely to be used.", "summary_of_the_review": "The proposed work is interesting and addresses an important challenge in representation learning. I'm not convinced the baselines or datasets used in the empirical analysis are most appropriate for this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "No8r7rZJXbk", "reviewer": "Reviewer_uMAL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nMy main concern is with the backbone chosen (MAE) in this paper and its training, to demonstrate the idea of learning manifold transformation effectively. Three-stage training phase for MAE is already something and on top of that, this paper introduces to exploit a pretrained Resnet classifier and a coefficient encoder network with Laplace distribution prior on the coefficients. As a result, the overall approach errs on the side of being a bit complicated. \n\nPresentation:\nI found page 5 quite hard to follow. As a result, I don't understand the fine tuning phase described as such on page 6. 'The fine-tuning phase adapts the autoencoder latent space to fit the learned transformations in the event that that pretrained latent space does not match the desired data manifold' I understand this without coefficient encoder network but not sure what is being matched after the introduction of coefficient encoder network. \n\nAlso, why name Section 3 as Methods? if named MAE, it will help to delineate your contributions from MAE clearly.\n\nThe experimental setup is ok although I don't understand why choose only two baselines, the latest being from 2017. ", "summary_of_the_paper": "This paper proposes to learn natural transformations in datasets, with manifold auto encoder (MAE), where the underlying identity-preserving transformations are not easily identifiable. By using Lie group operator, this problem reduces to learning paths/motion in the latent space of MAE. The main challenge in training MAE is how to choose a transformation pair, a point and its identity preserving transformed counterpart, in an unsupervised way. This paper introduces the idea of using penultimate layer  of a pretrained classifier on Imagenet to choose such point pairs and learn such MAE more effectively. The proposed method is validated on three datasets. ", "summary_of_the_review": "I give marginally above for now but I will revisit it based on other reviews and discussions. The problem of learning manifold transformation is well motivated. Experiments seem ok. The idea to use pre-trained classifier to fit MAE latent space traversal is sensible. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for learning identity-preserving transformations through a set of learned Lie group operators. It builds upon previous work ( (Connor & Rozell, 2020; Connor et al., 2021) addressing two points: (i) how to select semantically related pairs of points, (ii) how to identify which operators are appropriate for a given local region of the manifold. Authors use nearest neighbors computed via the penultimate layers of a pretrained network to address (i), and learn a separate network q(c|z) that predicts the coefficients given a latent input z. Reviewers have two main concerns with the paper -- limited novelty over earlier work that learns the Lie group operators; and the complicated nature of the method which needs training in three stages and uses a pretrained ResNet for finding nearest neighbors. Lack of comparison with relevant baselines is also pointed out by the reviewers. Given these issues, the paper is unfortunately not suitable for publication in ICLR at this point.
Decision: Reject