Manifold Contrastive Learning with Variational Lie Group Operators
Authors: Kion Fallah
Alec Helbling
Kyle A. Johnsen
Christopher John Rozell
Venue: Accepted by TMLR ()

Abstract: Self-supervised learning of deep neural networks has become a prevalent paradigm for learning representations that transfer to a variety of downstream tasks. Similar to proposed models of the ventral stream of biological vision, it is observed that these networks lead to a separation of category manifolds in the representations of the penultimate layer. Although this observation matches the manifold hypothesis of representation learning, current self-supervised approaches are limited in their ability to explicitly model this manifold. Indeed, current approaches often only apply a pre-specified set of augmentations for "positive pairs" during learning. In this work, we propose a contrastive learning approach that directly models the latent manifold using Lie group operators parameterized by coefficients with a sparsity-promoting prior. A variational distribution over these coefficients provides a generative model of the manifold, with samples which provide feature augmentations applicable both during contrastive training and downstream tasks. Additionally, learned coefficient distributions provide a quantification of which transformations are most likely at each point on the manifold while preserving identity. We demonstrate benefits in self-supervised benchmarks for image datasets, as well as a downstream semi-supervised task. In the former case, we demonstrate that the proposed methods can effectively apply manifold feature augmentations and improve learning both with and without a projection head. In the latter case, we demonstrate that feature augmentations sampled from learned Lie group operators can improve classification performance when using few labels.

Published in Transactions on Machine Learning Research (02/2024)
Manifold Contrastive Learning with Variational Lie Group Operators
Kion Fallah
kion @gotech edu
Alec Helbling
Kyle A. Johnsen
Christopher J. Rozell
ML&GT
Georgia Institute of Technology
Atlanta, GA 30332
Reviewed OpenReview: https: opppr/////eervveee.. net/ forum? id= LVE1 VeGQug
Abstract
Self-supervised learning of deep neural networks has become prevalent paradigm for learning representations that transfer to a varicty of downstream tasks. Similar proposed models of the ventral stream biological vision, is observed that these networks lead separation of category manifolds in the representations of the penultimate layer. Although this observation matches the manifold hypothesis of representation learning, current self- supervised approaches are limited in their ability to explicitly model this manifold. Indeed, current approaches often only apply pre-specified set of augmentations for "positive pairs" during learning. In this work, we propose a contrastive learning approach that directly modols the latent manifold using Lie group operators parameterized by coefficients with a sparsity- promoting prior. variational distribution over these coefficients provides generative model of the manifold, with samples which provide feature augmentations applicable both during contrastive training and downstream tasks. Additionally, learned coefficient distributions provide a quantification of which transformations are most likely at each point on the manifold while preserving identity. We demonstrate benefits in self-supervised benchmarks for image datascts, as well as a downstream semi-supervised task. In the former case, we demonstrate that the proposed hods can effectively apply manifold feature augmentations and improve learning both with and without a projection head. In the latter case, we demonstrate that feature augmentations sampled from learned Lie group operators can improve classification performance when using few labels
1
Introduction
Deep learning systems have made remarkable progress in decision-making tasks by leveraging large-scale, unlabeled datasets to learn neural representations. These representations, such as the activations at the penultimate layer of a deep noural network (DNN), have demonstrated efficacy in object recognition tasks with limited labels (Chen et al. (020bb), as well as the capability to generalize performance several downstreemm tasks (Chen et al.) 2020aa Caron al. 2021)). To effectively learn, DNNs must extract salient features, such the natural variations of a category of data, from high-dimensional inputs into low-dimensional latent representations. Among geometric descriptions of this desired property, manifold models offer natural explanation to how data existing in high dimensions can be parameterized by fewer degrees of freedom.
This explicit use of manifold models to represent visual invariances is also central to hypotheses underlying the ability of biological visual systems to achieve performance that generalizes across multiple tasks. For
Code available
com, http://www..oo.//gaaaaaa/////aaa

---

Published in Transactions on Machine Learning Research (02/2024)
Positive Pair
Backbonc
Lie Group Manifold in Latent Space
Conditional VAE Framework
T
26
fx(x)
2
Z
ZiE N
Negative Samples
Prior Encoder
Repel Attract
+
Figure 1: Proposed ManifoldCLR system for incorporating Lie group operators into the fealure space of contrastive learner. Given encoded features from positive pair, manifold feature augmentations T (c,) are applied before computing the contrastive objective. Here, 2j denote the set of negative pairs in the feature space. We apply the conditional VAE framework to learn coefficient distributions for manifold displacement between two points (encoder network (e(c) 2, x4)) and clentity-preserving augmentations from single point. (prior network (c 2.)).
example, the ventral stream of the primate primary visual cortex has been suggested to produce linearly separable category manifolds which are robust to pose change (DiCarlo & Cox 2007 DiCarlo et al. 2012 Majaj et al. 2015). Unfortunately, the computational mechanisms underlying this performance are not understood, and a large gap still exists between such systems and their deep learning counterparts. Indeed, deep learning methods are often expected implicitly learn invariances from large-scale dat datassts (Moskalev et al. )022), but often fail in distinguishing task-relevant and irrelevant changes to the input (Jacobsen et al, 2019 In the context of contrastive learning, part of this gap may be stated as the lack of explicit manifold modeling, limiting the number of unique views of instance the model sees during training. Rather than contrast potential views from some general manifold structure in the dataset, these methods often rely a set of pre-specified augmentations in the data space (Chen et al. 2020a Cubuk et al. 2020
In this work, we take a step towards closing this gap by proposing ManifoldCLR, a system for incorporating manifold structure describing natural data variations within the latent representations of a DNN while con- currently enriching such representations by sampling novel views along the manifold as feature augmentations during training. The proposed manifold model is parameterized by coefficients with a sparsity-promolle prior (Culpepper & Olshausen 2009 Olshausen & Field 1996). with learned distributions that can find both manifold paths between pair of points, as well as identity-preserving feature augmentations from a single point. To make such sampling possible, we propose the variational Lie group operator (VLGO) model built upon recent advances a variational sparse coding (Barello et al. 2018 Tonoliní et al.) 2020 Fallah & Rozell 2022). With added block-diagonal structure on the learned operators, we are able to reduce memory usage of the model with minimal impact downstream performance. We demonstrate the efficacy of our approach self-supervised and semi-supervised benchmarks with image datasets (Krizhevsky
2009
Coates
et al. 2011 Deng et al. 2009).
2
Background
2.1
Manifold Learning
Manifold models have gained interest in the machine learning community for their geometric description of how data collected high ambient dimension (c.g., pixels an image) can vary with few degroes of freedom. In the context of representation learning, this idea formalized by the manifold hypothesis [Fefferman et al. 2016), which suggests that instances from category lie near low-dimensional manifold,
2

---

Published in Transactions on Machine Learning Research (02/2024)
with low probability regions separating categories of data (Bengio et al. 2013 This hypothesis has been validated on many real-world datasets of interest (Pope et al. 2021 with theoretical implications on the sample complexity of learning underlying data distributions (Narayanan & Mitter 2010 As such, manifolds
have been incorporated into learned representations to discntangle factors of variation [Buchacourt et al.)
2021 Connor et al. 2023). provide equivariance to model puts (Cohen & WWeling
2016). or give additional
context when learning with limited labels (Belkin Niyogi 2002 Belkin et al.] 20055
Many classical mothods
seek non-linear embeddings of data (Saul et al. Tenenbaum 2000 Belkin & Niyogi 2001 or tangent plane
approximations (Bengio & Monperrus, 2004 Rifai et al. 2011 but both are limited in their ability to interpolate generate new points on the manifold.
In this work, we build upon a class of methods which provide a generative model of the manifold (Rao & Ruderman 1998 Miao & Rao 2007 Culpepper & Olshausen 2009 Sohl-Dickstein et al.) 2017] by learning the basis a Lie algebra (Hall 2015). Once passed through the matrix exponential, a point in this algebra provides a linear representation of generator of Lie group, denoting a displacement along a manifold. As an example, let T be family of operators that forms a generic group. This means that (1) there exists function f T T + T that maps pairs of transformations to another transformation such that f is associative, (2) there exists a unique identity transformation Et T t(x)) I, and (3) cach transformation E T is invertible (Rao & Ruderman 1998). Furthermore, restricting T to form a Lie group means that it applies contimuous transformations parameterized by a vector of real numbers. Tn the context of this work, this parameterization comes from inferred coefficients that are used to take a linear combination of the learned operators. Although Lie groups form a subset of all differentiable manifolds, they have a long history of use models of images (Rao & Ruderman 1998) and visual perception (Dodwell 1983 For more comprchensive background, refer readers to Hall (0155).
In this work, we describe this basis as dictionary of M operators Vm E R DxD, Given a point on the manifold x, E RD, the Lie group operators model an infinitesimal transformation as weighted sum of the Icarned dictionary
M Σ
x,
L Cm
Xi,
where e E RM modeled with a sparse, factorial prior p(c) II M p(em) (Culpepper & Olshausen 2009
Olshausen & Field 1996). Given batch of point pairs sampled nearby the manifold (x,, x(), the dictionary of operators can be learned through an alternating minimization scheme where coefficients are first inferred with fixed operators (e.g., using sparse inference procedure (Olshausen & Field 1996)), followed by a gradient step with fixed coefficients to minimize the loss with respect to the operators (Culpepper & Olshausen 2009
Lu (x,,xx,,,) =|xx
(c)
(2)
M
Tw(c) = expm
Σ
V Cm
(3)
Learning these operators directly in the data space is often impractical due to dimensionality. As such, later work has extended this model to learning the operators after applying non-linear dimensionality reduction. (x,) RRd where d < D (Connor& & Rozell, 2020 Connor al. 2021 Bouchacourt al. 2021 Connor et al. 2023 Cosentino et al. 2022a3. Note that this setting, the operators describe transformations between latent points and have dimensionality Rd Most of these systems rely on an aulo-encoding objective to prevent collapse to trivial manifold (e.g., projecting every input data to constant value allows one to learn all operators as identity), but are often limited by the fidelity of the decoder network in their ability to apply manifold transformations.
z,
Furthermore, performing exact sparse inference e at each training step becomes overly prohibitive as dimensionality incrcases. Although fast first-order optimization algorithms have been proposed (Beck & Teboulle 2009 Connor et al. (202)). such methods are not practical in deep learning systems that require upwards of hundreds of thousands of training iterations. Hence, alternative inference methods are necessary
3

---

Published in Transactions on Machine Learning Research (02/2024)
when incorporating the operators into high-dimensional feature spaces (e.g., the penultimate layer of ResNet backbone (He et al. 2016)).
2.2 Variational Inference
When dealing with probabílistic models with computationally infeasible inference procedures, variational inference has emerged as an effective approach (Zhang et al. 2018). For example, let pv (x.) be some arbitrary likelihood of imput x; parameterized by v. When maximizing this likelihood, marginalized over a latent variable c, one may apply the evidence lower bound (ELBO) (Jordan et al., 1998) with a learned variational posterior (c x;):
log (x:) = log Bp(c) [p (x;
c)]
x;)
= log Bp(c)
P (x: c)
[ xx)
> Bgg [log pvr (x:
c)]
DKL (ap(e x;) p(c)).
In practice, maximizing this lower bound is often presented as minimizing the negative lower bound. The rest of this work will use this presentation.
Recent methods have learned this variational posterior with a DNN employing the "reparameterization trick" (Kingma & Welling 2014 Rezende al.) 2014), allowing one to simplify inference procedures as a single forward pass. Variational sparse coding approaches extend this procedure by encouraging sparsity in the latent variables sampled from the variational posterior network. These approaches either sparsity promoting priors (Barello et al., 2018), approximations to discrete variables (Tonolini et al., 2020), straight-through estimations of soft-thresholding (Fallah & Rozell 2022) to achieve the desired sparsity. In each of these previous works, inference is performed from a single input x;, with a Gaussian likelihood computed either using a linear dictionary or DNN decoder. In this work, we are interested in applying variational sparse coding the Lie group operator model, using pairs of points for inference.
2.3 Self-Supervised Learning
Self-supervisod lcarning (SSL) has emerged as an effective paradigm of leveraging unlabeled data to learn deep representations that transfer to several downstream tasks (He et al. 2020 such as learning with limited labels (Chen et al. 2020b). For example, contrastive learning (Hadsell et al. 2006) learns representations by encouraging similarity between representations of "positive pair", while encouraging dissimilarity between representations of "negative pairs". Different approaches have been proposed for selecting negative pairs, such
as nearest neighbor instances (Dwibedi et al. 2021), memory banks of features (He et al. 2020: Chen et al.,
2020ch, or other instances within batch (Chen et al. 2020a). Besides contrastive methods, negative-free approaches have emerged that hir apply regularization to learned features (Ermnoovv el. al. 2021 Bardes
et al. 2022) or utilize teacher networks /stpssoorrrrre grad operations to prevent collapse to a trivial point Grril et al,
2020
Chen & Hc 2021 Caron et al. 2021).
To implement the contrastive learning objective, the InfoNCE loss (Oord et al. 2019
often employed.
First, DNN backbone feature pairs (z,, z = (x)(x;), fo()()) corresponding to two views of an image instance
(x, x!)
are found by randomly sampling from set of image space augmentations (Chen et al.
2020a
Sohn
et al. 2020 Then, the temperature normalized InfoNCE objective is computed (Chen et al.
2020a
exp (-1 ho))o - ho (z,2)11//TT
Lar(ziz -log
(4)
EjEN exp (-lhoo/)) - ho (1337) + exp (exp(-11/ag (-llolzz)))))))))))))))))) - ho ho (2())11//7
where N denotes the set of nogative pairs and 7 is temperature hyper-paramcter. Here, he an additional normalized, non-linear projection from the backbone features. Recent work has argued that this projection learns a noisy estimate of the data manifold, encouraging backbone invariance to non-salient image features
4

---

Published in Transactions on Machine Learning Research (02/2024)
Algorithm Variational Sparse Coding
Input: Input positive pair 2; and 2/ whether to use a Soft.hreshhod, threshold paper-pprramter S, number
of samples J.
(11 b,) с- D (sg (4 + 2!])
for j 1 to do
c U[ U1(2222 2'2)
s M4 b; o sign(e)) In (1 (1-2) e ))
Tofthhrohhlld then
c/ + + sg
Tc s s
else
c₂ s end if
C + Lm (Zi 2 c)))
end for C f a-arggmn miny C/
Cosentino et al. 2022b). Oddly, this non-linear projection is often discarded after training, with backbone leatures 2, used instead for downstream lasks.
2.4 Related Work
Other work has considered unsupervised feature learning using manifold models. The sparse manifold transform (SMT) applies slow feature analysis afler a linear sparse coding step to obtain generative manifold embedding CChen et al. 2018), with extensions to learning unsupervised "white-box" representations (Chen ett al. 2023 In this work, rather than applying linear sparse coding to a single point, we infer coefficients to find non-linear manifold paths between a pair of points. Applying slow feature analysis to the sparse coefficients of the Lie group operators is an interesting direction that we leave to future work. Likewise, other work has considered manifold SSL by encouraging maximum manifold capacity under an elliptical manifold model (Yerxa et al., 2023 The model proposed in this work has the advantage of learning arbitrary manifold structure coinciding with Lie group.
In a similar work, Ibrahim et al. (2022) apply Lie group operators for robustness to unseen views in SSL. However, our work differs in two key ways: (1) we propose a variational framework for coefficients rather than a deterministic estimate, allowing one to sample coefficients, and (2) we incorporate feature augmentations from Lie group operators into contrastive lcarning, providing significant improvements in performance and allowing us quantify coefficient distributions which preserve identity. Both of these changes critical for the goal of this work, which is to incorporate manifold feature augmentations into the contrastive learning process. Indeed, deterministic encoder precludes the ability sample coefficients, with the encoder in Ibrahim et al. (0022) requiring information regarding the transformation applied between the point pair as input (see Equation of Ibrahim et al. (022)), making it impossible to get coefficients from a single input.
3 Methods
3.1
Variational Lie Group Operator Model
To effectively learn and apply the Lie group operators, needs method to perform quick coefficient inference at each training step. The quality of the learned operators is completely dictated by how effective inferred coefficients transport from an initial to a larget point on the manifold. Furthermore, not every manifold augmentation can be applied with the same magnitude at every point on the manifold. This means that incorporating fcature augmentations requires knowledge of which operators can be applicd for given input point and the extent to which they can be applied. To address these requirements for quick inference and learned coefficient distributions from which one sample, build upon advances in variational
5

---

Published in Transactions on Machine Learning Research (02/2024)
(a) CIFARIO
(b) STL10
Figure 2: Nearest neighbor visualization of feature augmentations sampled from the learned prior. For cach row, the first column denotes the input to the prior. Each subsequent column is found by taking a random sample from the prior, applying an operator augmentation, and visualizing the image corresponding to the nearest feature in the dataset. can be qualitatively seen that the prior network learns coefficient statistics that result in augmentations that preserve class-identity.
sparse coding (Fallah & Rozell 2022; Tonolini et al. 2020 Barello et al. 2018). Given features close on the manifold (Z, 2,), we sample sparse coeffciints by reparamcterizing distribution parameters encoded by a recognition network с >~ ( z,, x Kingma & Welling 2014
Rezende et al. 2014). We summarize the reparameterization procedure in Algorithm [] with the option incorporate soft thresholding and take J best-of-many samples for effective machine-precision sparsity (Fallah & Rozell
2022 Bhattacharyya al.,
2018
After applying the variational lower bound, we obtain the following variational Lie group operator objective, which includes a weighted KL divergence term (Higgins et al. 2017h:
Lm (Zi, 2 c) 3Lx (zi, 2)
(5)
In this framework, one can either fix prior distribution parameters pa 2:) as in standard variational inference, encode them with separate DNN in the conditional variational autoencoder (CVAB) framework (Sohn et al. 2015). During training, samples are drawn from the encoder network to concurrently train the operators V with encoder weights & (and backbone/prioe weights when applicable).
Like the C'AEE (Sohn et al. 2015), the encoder 96 and prior pe encode distributions to solve different tasks. The encoder outputs cocfficient statisties for reconstruction of z/, analogous class labels in the CVAE framework. On the other hand, the prior network solves a predictive task in encoding statistics to generate other points on the manifold from an initial 2,- Learning the prior network provides several advantages. First, it provides flexibility for the encoder network to deviate from fixed statistics when certain operators are more or less likely to be used for an initial point z; Second, it allows one to sample feature augmentations along the manifold to apply both as a synthetic positive pair on the feature manifold during SSL pretraining, or as strong feature augmentation during tasks like semi-supervised lcarning (Kuo et al. 2020 Sohn et al., 2020). In Section 4.2 we demonstrate that learning the prior leads to better peentity-rrseevvtion than naively sampling from a fixed prior.
3.2 ManifoldCLR: Manifold Contrastive Learning
Given the variational Lie group operator framework, one can learn a manifold describing the natural variations present in a dataset within the representations of a DNN (while also regularizing such representations to match the data manifold). This is done by sampling coefficients from the encoder network ~ go 2;, 2!) to minimize the manifold loss Cm. Furthermore, one can directly sample transformations along the manifold to enrich the number of views that are applied during instance discrimination. To do this, one may sample
1 Features detached, concatenated, and fed through multi-layer perceptron followed by linear projection for each distribution parameter.

---

Published in Transactions on Machine Learning Research (02/2024)
coefficients غ pe z;) from the learned prior network, apply manifold feature augmentations to obtain synthetic view 2; = Tw (c), and contrast the positive pair (2,,%) x4).
This allows one to both enrich the novel views of an instance, as well as learn coefficient statistics in the prior network that preserve identity. We hypothesize that this latter outcome a result negative samples in the contrastive loss, which encourages prior augmentations to remain on the object manifold, visualized on the right in Figure []] We note that previous approaches for learning prreertypres manifold transformations have required labels (Connor et al. 2023 whereas the methods proposed here are entirely unsupervised.
Finally, the resultant learning objective can be written as:
min Lett Zi, 2.) + ALm 24, c) + BL (Zi, 2/2 c)
* 4,0
where L are manifold operator parameters, O are DNN backbone and prior network parameters, Φ are coefficient network parameters, and À and 3 are loss scaling hyper-parameters. All terms and parameters are learned concurrenlly, end-to-end from random weight initialization. We note that this is a simplified training procedure in comparison to previous works with Lie group operators that use multiple training phases (Connor Rozell 2020 Connor et al. 2023) or pretrained backbone models (Connor et al. 2023
Ibrahim ct al. 2022
3.2.1 Modified Manifold Objective
Learning the Lie group operators can be impractical in cases where large due to memory constraints computing the matrix exponential. To solve this, we propose additional block-diagonal constraints onto the learned operators. We do this by breaking up the features of our point pair into segments of length b. Rather than constrain the operators to be block-diagonal at each training step, they are initialized as d/b separate dictionaries V of shape Rdxb, After inferring coefficients for each pair of features, we compute the objective for each segment separately
3
d/b-11 Lm (z, = Σ
Zb] ( (+1))) T (c)zz(i; (j + 1)b]
(7)
Furthermore, we find that including the stop-grad operation, written sg and indicating no gradient computed through the term, on z' helps prevent feature collapse. This technique has been previously applied in systems such as SimSiam (Chen & He, 2021). Under this approach, the number of parameters needed to learn a dictionary of 128 operators in a 512 dimensional ResNet-18 (He et al. 2016) backbone is reduced from 33.6M parameters to 4. 4. parameters. This leads to significant reduction in memory use during training with minimal impact to downstream performance. We ablate the impact of these choices in Section 4.2.3
4 Experiments
4.1 Synthetic Dataset
To measure the effectiveness and speed-up of the VLGO model, we first evaluate on a synthetic dataset, comparing to the training procedure proposed in Culpepper & Olshausen (2009). We train on 5000 points generated on a Swiss roll manifold, using the fast iterative shrinkage-threste tereethhhrhhhiinggggg (FISTA) algorithm (Beck & Teboulle 2009) for exact sparse inference as a baseline. At each itcration, we seloct 500 points and randomly sample the point pair from the 20th 60th nearest neighbors. In total, we train for 1000 epochs and measure three key metrics, (1) the mean-squared error between transported point pairs Lm, (2) the cumulative runtime, and (3) the & loss of inferred coefficients to measure sparsity. For hyper-parameters, weight of the & penalty as 0.6, and the weight of the F-norm penalty on the operators as 1e-3.
set M 6, the
We compare to the VLGO using both a standard Laplacian prior (Barello et al. 2018 and thresholded samples from a Laplacian FFallah& Rozell 2022), evaluating at different sample counts J. We use the same
We use (a: in array notation denote the subset dimensions from from the feature vector.
7

---

Published in Transactions on Machine Learning Research (02/2024)
Cumulative Runtime (s)
222
L1 Loss
2.30
Liplacian samp
Esplacian 1 semp
RETA
MSE
Laplacian 100
Laplacian samp
relerten vemp
Thresh inplacion 10 samp
Thresh. Leclecian 100 samp
Laclacion
1
Iterations
Iterations
Iterations
(a)
(b)
(c)
03
0.5
1.0
0. 03
4.5 0 0.25
0.75
0.5
0.75
(e)
Figure 3: Comparison of Lie group operators trained with exact and variational sparse inference on synthotic Swiss roll dataset. For each method, we plot (a) mean-squared error, (b) cumulative runtime, and (c) & penalty of coefficients. (e.f) Extrapolated paths of the two operators learned by VLGO, Black dots indicate starting point with subsequent path found by applying Lie group operators Tw (c,)x; with coefficients from a range with fixed step size C; = I-NN...., Ne].
apperpparameters the FISTA baseline with fixed prior and 3 = 5c-3. The results shown in Figure B where three main conclusions can be drawn: (1) as the number of samples drawn from the encoder increases, variational methods match FISTA in MSE, (2) when using more samples, thresholding gives the same MSE as a standard Laplacian with much lower é, loss, (3) variational methods are approximately 15x faster in training time than FISTA. We note that FISTA and the thresholded Laplacian result in two non-zero operators after training, whereas the standard Laplacian results three non-zero operators. We attribute this to exact. sparsity, allowing weight decay to shrink unused operators in the FISTA and thresholded Laplacian models.
4.2
Image Datasets
4.2.1
Experimental Setup
Next, we compare the performance of the ManifoldCLR system equipped with VLGO to standard contrastive learning, as well as other baselines for training without a projection head or incorporating feature augmenta- tions into the learning process. For the main results, we train each method using the SimCLR framework for selecting positive and negative pairs (Chen et al. 2020a)), only adding additional feature augmentations positive pair when relevant. Specifically, let (t,, +) ~ T be two augmentations sampled from those presented in SimCLR (described in Appendix 6.2.1 p. For each imput image in a batch x;, we use (x, = (t:(x;), (x,)) as the input point pair for each evaluated method. Although we use this framework to measure the relative benefits proposed methods, we note that they are compatible with other contrastive learning systems that use the InfoNCE loss (Chen et al., 2020c). For example, we compare to NNCLR (Dwibodi et al.) 2021 by selecting nearest neighbors as our point pairs in Table We leave methods for incorporating manifold feature augmentations into other self-supervised frameworks to future work.
8

---

Published in Transactions on Machine Learning Research (02/2024)
We evaluate both using the standard linear readout protocol (Chen et al. 2020a), shown in Table 1 and modified semi-supervised objective, shown in Figure 4 We test on a variety of datasets, including CTFARIO | 2009), STL10 (Coates et al.) 2011), and TinylmageNett (Deng et al. 2009). We build on the experimental setup of Ermoloy et al. (2021), training a ResNet-18 (He et al. 2016) with a batch size of 512 for 1000 epochs across all experiments. For methods that use a projection head, we normalize the projected features before computing the InfoNCE loss. When training without a projection head, we find that using MSE in the InfoNCE provides superior performance (ablated in Table 7 We train all methods with a single NVIDIA A100 GPU, with approximate runtime of 9 hours for baselines and 24 hours for ManifoldCT.R on Net Additional analysis on the computational and memory complexity can be found in Appendix 6.I.I] Full details on the experimental spt and evaluation is included in Appendix 6.2. Before discussing the results, we introduce each of the baselines for feature augmentations.
ManifoldMixupCLR uses the common linear interpolation strategy for feature augmentation proposed Manifold Mixup (Verma et al., 2019a). For each positive and negative pair, a random interpolation constant is sampled X (0, 1), and applied to form feature augmentation 2, X + (1- )x)2.-
ManifoldCLR uses the VLGO model to sample coefficients from a learned prior network ē po(c 24) that is used to apply manifold feature augmentations z, = Ty (C)z,. This is done concurrently with learning the Lie group operators by minimizing f. with coefficients from the encoder network. We set B == 1e-5 for each method and À = 10 and À = with and without a projection head, respectively. Furthermore, we set the dictionary size M- = (16, 64, 128) for datasets (CIFARIO, STL10, TinylmageNet). Finally, we set the block size of our operators
64 for each experiment.
Direct (Jing et al. 2022) contrasts only the first b components of the backbone features when training without a projection head. We apply it as baseline method that improves upon contrastive learning without projection head.
ManifoldDirectLLE combines ManifoldCLR and Direet by applying the manifold loss only to the first b= 64 feature components. Furthermore, augmentations and contrasting is ouly performed on these first. few feature components.
Table shows that when trained with projection head, ManifoldCLR provides an improvement in all datascts over SimCLR. Without projection head, ManifoldCLR performs the best among feature augmental ion methods. Performance benefits over ManifoldMixupLLR indicate the importance in identifying non-lineet paths in the latent space for feature augmentations. Notably, we find that the benefits of ManifoldCLR and DirectCLR are complementary and that combining the two gives a higher performance than SimCLR with a projection head. We find this result satisfying, since affirms the purpose of the projection head as a means to estimate the data manifold (Cosentino al.) 2022b). ManifoldCLR is capable of estimating this manifold structure directly in the backbone with VLGO, removing redundant model weights from the training process while also providing generative model of the manifold.
To test the applicability of the ManifoldCLR framework to different strategics for selecting point pairs, we also compare to NNCLR (Dwibedi el al., 2021) in Table For this experiment, we follow a methodology similar to Table 5 in Dwibedi et al. (2021), where we contrast nearest neighbor point pairs with only random cropping applied as the augmentations (i.c., no color jitter or grayscale). We present our results with a projection head on CIFAR10 laken over three random trials. Although these experiments have lower separability than the SLR augmentations, this is to be expected, based on the results presented in Table 5 in Dwibedi et al. (0021). We deviate from NNCLR in few ways for these experiments. First, to increase the stability of training, we follow the methodology of Connor et al. (2023) and pre-compute all the point pairs using an auxiliary feature space (DINO model (Caron et al., 2020) pret rained on ImageNet). We note that this leads to a significant benefit (approximately +40%) for both the NNCLR baseline and ManifoldCLR. Furthermore, for simplicity, we do not include the additional prediction head used in NNCLR. This, however, has very minimal impact (approximately 0.4%), as shown in Table 7f in Dwibedi et al. 2021
9

---

Published in Transactions on Machine Learning Research (02/2024)
Table 1: Linear probe accuracy on various image datasets with different methods for contrasting point pairs with without a projection head. Results averaged over three trials. We underline the best performing method(s) overall and bold the best method(s) without projection head. See Section 4.2 for details on methods used.
Method
Projection Head
CIFAR10
STL-10
TinyImageNet
SimCLR-MLP
89.04%
86.68%
39.74%
SimCT.R-Linear
V
87.78%
85.18%
36.88%
ManifoldCLR-MMP V
90.03%
87.33%
42.77%
SimCLR-None
X
88.58%
84.53%
36.26%
ManifoldMixupCTAR
86.52%
83.02%
38.33%
ManifoldCLR
X
88.89%
84.99%
38.68%
DirectCLR
X
88.46%
84.46%
36.20%
ManifolDDired
X
89.73%
87.47%
42.22%
Table 2: Linear probe accuracy compared between NNCLR and ManifoldCLR using nearest neighbor point pairs. The images are only augmented with random crop, as described in Appendix 6.2.1 Both approaches use an MLP projection head. Results averaged over three random trials.
Method
CIFAR10
NNCLR-MLP 66.13 + 0.20% ManifolCCRR-MLP
69.43 + 0.17%
4.2.2 Semi-supervised Learning with Manifold Feature Augmentations
To highlight the benefits of generating manifold feature augmentations for down-stream tasks, we evaluate on semi-supervised task using five labels per class and consistency regularization (Sohn al. 2020 in Figure 4 For each dataset, we freeze the weights of the best performing backbone network from Table 1 and train single hidden-layer MLP with different methods for feature augmentations. We train under this protocol since fine-tuning the backbone weights would result in a deviation from the manifold learned during contrastive pre-training. We have had preliminary success in fine-tuning while including the ManifoldCLR objective (hence adapting the manifold to the classification task), but we leave such experiments for future work. Here, we aim to demonstrate that the VLGO can provide feature augmentations that improve classification performance with limited labels. Given labeled features (2), v/) and unlabeled features z we train a classifier re using the following loss:
min Σ
H (H
1 N
Σ
1 2
(
H]
Here, B and 8" are the batch size of the labeled and unlabeled dataset, respectively. q' = r and q re(z"") are the normalized logits from the classifier corresponding to class probabilities for the labeled and unlabeled features, respectively. N Σ (9 2 7] the number of unlabeled examples with confidence above and H(.) is the cross-cntropy loss. We follow the methodology Sohn et al. (2020) and use the class predictions of unlabeled data of which the classifier is sufficiently confident as pseudo-labels % arg max q for predictions group transformations म T (Cs 22 using cocfficients sampled from the learned prior č₁ (c zg).. We focus our comparison to other methods for incorporating feature augmentations.
augmented examples q re (Z)). To get augmentations, apply Lie
We train each dataset and method with B' 32, = 480 randomly sampled examples at each iteration, using a single hidden-layer MLP with 2048 hidden units. We train with the AdamW optimizer [Loshchllvv Hutter 2019) with fixed learning rate and weight decay of 5c-4. For CIFAR10 and STL10, we set J = 0.95. For TinyImageNet, set 0.7. Furthermore, evaluate using exponential moving average (EMA) of
10

---

Published in Transactions on Machine Learning Research (02/2024)
Table 3: Average percent improvement in semi-supervised accuracy over baseline over 50 splits with varying datasets and methods for incorporating feature augmentations using labels/class. Best method(s) for each dataset are bolded.
Method
CIFAR1O
STL-10
TinylmageNet
Pseudo-labeling Manifold Mixup ICT Feah
3.99 + 1.88%
0.34 4.90%
1.25 0.28%
-1.46 + 0.48%
-3.26 = 1.11%
-0.01 = 0.11%
5.08 + 1.20%
1.35 + 0.50%
0.26 0.15%
Variational Lic Group Operators
8.61 + 2.57%
5.47 5.47 2.37%
1.58 0.30%
85
80
Accuracy
80
Accuracy
75
70
65
65
60
Baseline Pseudo-label MMICT
FeatMatch
VLGO
Baseline Pseudo-label
MMICT
FeatMatch
VLGO
(a) CIFARIO
(b) STL10
Figure 4: Semi-supervised experiments using frozen backbone and learned, single hidden layer MLP. Comparison methods leverage unlabeled data to encourage consistency in classifier output after applying feature augmentations. Taken over 50 data splits used for each method.
100
truck
07:28
ship
Accuracy
80
79.01
Learned Prior
monkey
0.20
horse
60
dog
0.15
40
30.7
deer
cat
0.10
20
bird
airplane
CIFAR10 STL10
Tinylmagenet
18
22
26 31
44
46
52
55
56
Operator Index
(a)
(b)
of prior measured by lincar classifier accuracy on frozen features after sampling augmentations. Standard error below 0.21% for each method. (b) Average, normalized coeffi- cients per class sampled from learned prior on STL10 dataset. Coefficients can be used analyze which transformations more or less likely for different input points and classes.
Figure (a)
our classifier weights at each training step with weight 0.999. We train each method for 5, 000 iterations, except FeatMatch (Kuo et al. 2020) which we train for 10, 000 iterations give the attention head sufficient. iterations learn mcaningful augmentations.
To account for high variance in trial performance (e.g., certain splits of five labels per class are challenging baseline that only uses
than others), present the result with respect to percent improvement standard cross-entropy loss on labeled features in Table 3 We note that VLGO provides the highest accuracy across almost every data split (plotted in Figure 7).. Furthermore, we include paired (-lests in Table g showing that accuracy improvements from the Lie group augmentations are statistically significant at the group level. We compare pscudo-labeling without augmentations (Lee) (213)) FeatMatch (Kuo al.) [2020), and Manifold Mixup interpolation consistency training (MMICT) (Verma al. 2019b). For FeatMatch, we all the labeled data the prototypes for the attention-base feature augmentation module with four
11

---

Published in Transactions on Machine Learning Research (02/2024)
Table 4: Linear probe accuracy and manifold fit on TinylnageNet without projection head ablating various components from the ManifoldCLR system. Results averaged over three trials. Soe Section 4.2.3 for details each system.
System
Stop-grad on
Lie Group Augmentations
Manifold Loss
Learned Prior
Linear Probe
Lm
Acc
So
38.68
S1
X
V
31.97
S2
V
X
V
V
37.37
S3
X
36.57
S4
36.85
attention heads. For MMICT, we apply a random linear interpolation sampled from U(0, 1) between two randomly sampled unlabeled points and apply an MSE loss between the mixup of logits predicted by the EMA model and the logits of the interpolated features. We note that this differs from the methodology of Verma al. (2019b) in that interpolate features rather than the input data. This method is meant to compare the benefit of learning non-linear paths with the Lie group operators. Among these methods, it can be seen that VLGO provides the most benefit in Figure A We include experiments using more labels in Appendix 6.2.2 where the trial variance is reduced while the relative benefit of VLGO is also minimized.
To better understand the types of augmentations encoded by the prior, we visualize the nearest neighbor we classify
image to several feature augmentations in Figure 2 To quantify the degree of Seentity-preservee augmented features from our validation set under a linear probe classifier in Figure 5(a] Here, it can be seen that learned prior has significantly superior adentity--ressevvaiion over fixed prior, with the gap widening as the dataset complexity increases. Furthermore, by analyzing the normalized encoded coefficients, averaged per class, from the prior network in Figure 5(b] one can draw insight on the magnitude of transformations that can be applied per class, as well as which transformations are shared between classes. For example, operator 52 is most likely for 'horse' and 'deer', operator 55 for 'dog' and 'cat', and operator 56 for 'truck', 'car', and 'airplane'. We emphasize that previous work with Lie group operators required all class labels learn
coefficient distributions (Connor al.) 2023), as opposed the unsupervised approach in this work.
4.2.3 Ablations
Next, we ablate several components in the proposed system (So) and measure the impact to linear readout accuracy on TinyImageNet trained without projection head in Table 4 First, observe the impact of removing the stop-grad operation in Equation as S₁. Next, test removing the prior augmentations from the contrastive loss in Equation and removing the manifold loss (i.e., x= = 0 in Equation () in systems S2 and S3, respectively. We note similar degradation of performance when using coefficients sampled from the encoder network as augmentations in the contrastive loss. Finally, we measure the impact of replacing the learned prior with a fixed prior for contrastive loss augmentations in St-
To evaluate the effect of peper-parametery on the ManifoldCLR system, we measure linear probe accuracy while varying different hyper- parameters the TinylmageNet dataset with a projection head in Figure 6 First, it can be seen that dictionary size follows inverted U-shape, with too few dictionary entries causing underfitting and too many causing overfitting. We find that this is the hyper-parameter that is most critical to tune. On the other hand, we see that changing the weight. À of the manifold loss L., provides little difference on downstream performance. Lastly, we see that imposing a block diagonal constraint on operators has little impact on linear probe performance above a block size of 64 (with significant benefits in computational and memory costs). As one may expect, however, we find that reducing the block size leads to a reduction of the effective rank (Roy& & Vetterli 2007) of backbone features. We report these values in Table g
12

---

Published in Transactions on Machine Learning Research (02/2024)
5 Discussion & Limitations
In this work, we demonstrate that incorporating novel views via manifold feature augmentations improves the performance of contrastive learning across several image datasets. To learn the manifold structure of data in a way that is amicable to sampling, we propose VLGO, a variational framework for the Lie group operator model. This allows one to quickly infer coefficients describing the manifold displacement between a pair of points, while also learning identity-preeeringg transformations be applied single point in the feature space. We demonstrate that this model can be used in contrastive learning to improve performance with and without projection head, potentially obviating the need for projection head completely. After contrastive pre-training, we also demonstrate that feature augmentations from the prior can be used for improvement in semi-supervised learning.
One limitation of the proposed model is memory usage from the lack of fp16 support current implementations of the matrix exponential (Bader et al. 2019 Paszke et al. 2019). Here, we address this constraint by imposing block-diagonal structure in the learned Lie group operators. Development new numerical methods for the matrix exponential is future work that dramatically improve the applicability of VLGO to larger scale models. Furthermore, although our work focuses the benefits in contrastive learning, interesting future directions may consider evaluation in other sell-supervised frameworks.
Acknowledgements
The authors would like to thank Brighton Ancelin and Coleman DeLude for their helpful discussions about Lie groups. This work partially supported by NSF CAREER award CCF-1350954 and ONR, grant number N00014-15-1-2119.
13

---

Published in Transactions on Machine Learning Research (02/2024)
References
Philipp Bader, Sergio Blanes, and Fernando Casas. Computing the matrix exponential with an optimized
taylor polynomial approximation. Mathematics, 7(12), 2019. ISSN 2227-7390. doi: 1..3390/aat7722117
URL https://www.mdpi wwtp://ddpccooooooeemmommmmm com /ttp222227770////1111111
Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Varranne-umaataarc--ovvaaeee regularization for
soff-sueervied learning. In International Conference on Learning Represenlations, 2022. URL https:
Gabriel Barello, Adam S. Charles, and Jonathan W. Pillow. Sparse-Coding Variational Auto-Encoders.
preprint, Neuroscience, August 2018. URL
1101//999..1
Amir Beck and Marc Teboulle, A Fast Iterative Threshoodggg Algorithm for Linear Inverse Problems.
STAM Journal on Imaging Sciences, 2(1):183 202, Jamuary 2009. ISSN 1936-4954. doi: 10.1137/08071655
URL http://epubs.. siam. org/ wttgg/doi/100/ 1137
Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps and spectral techniques for embedding and clustering. In Proceedings of the 14th International Conference on Newral Information Processing Systems: Natural and Synlhelic, NIPS'01, pp. 585 591, Cambridge, MA, USA, 2001. MIT Press.
Mikhail Belkin and Partha Niyogi. Using manifold stucture for partially labeled classification. In S. Becker, Thrun, and K. Obermayer (eds.), Advances in Newral Information Processing Sys- tems, volume 15. MIT Press, 2002. URL https://prooceedingg. neurips. cc/paperr2002/file/ F9766576894d277a22e7// aaabde/a222777eeee 21873322222222772222222282263332 Paper.ddf
Misha Belkin, Partha Niyogi, and Vikas Sindhwani. On manifold regularization. In Robert G. Cowell and Zoubin Ghahramani (eds.), Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics, volume R5 of Proceedings of Machine Learning Research, pp. 17-24. PMLR, 06-08 Jan 2005. mlr, www.mm..r..../... press/55/ http://bbbeeknnna.bbbaae html Reissued by PMLR on 30 March 2021.
URL httpp:://proceediiiiii
Y. Bengio, A. Courville, and P. Vincent. Representation Learning: A Review and New Perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798 -8281 August 2013. ISSN 0162-8828, 2160-9292. doi: 10. PT100//TTAMM.01..... URL http:// leee. wrgg//ocument/5472200
Yoshua Bengio and Martin Monperrus. Non-Local Manifold Tangent Learning. In L. Saul, Y. Weiss, and L. Bot- tou (eds.), Advances in Newral Information Processing Systems, volume 17. MIT Press, 2004, URL https pdf
//proceedings.nene
Apratim Bhattacharyya, Bernt Schiele, and Mario Fritz. Accurate and diverse sampling of sequences based on "best of many" sample objective. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition (CVPR), June 2018
Diane Bouchacourt, Mark Ibrahim, and Stéphane Deny. Addressing the topological defects of disentanglement via distributed operators, 2021. URL https://arxiv: orgg wtwgg//////g////eeeeeee////////e /212 05623 arXiv:2102.05623 [es].
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin ), Advances in Newral Information Processing Systems, volume 33, pp. 9912 9924. Curran Associates, Inc., 2020. URL 2t7.///800666.11..0222822222222.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) pp. 9650-9660, October 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Hal Daumé III and Aarti Singh (eds.). Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 13- 18 Jul 2020a. URL https:://proceeddnnggeee//// mlr, ppwsss/vevvvve vwtss/vv1//// ht/chennnj .tml
14

---

Published in Transactions on Machine Learning Research (02/2024)
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self self-supervised models are strong semi-supervised learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 22243 22255. Curran Associates, Inc., 2020b. URL http:://prooeddingg.mmmm httpps.cc/ppppppp wwtpcccppppppeeeeeeeeeeee files wwwlee/paper/eeeee pappp//pp///////
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15750-15758, June 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming Hc. Improved basclines with momentum contrastive learning. 2020c. doi: AA0S550/AAXV.20303..0
URL hattps//aarriv..rrr///////00.000
Yubei Chen, Dylan Paiton, and Brumo Olshausen. The sparse manifold transform. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Biancha and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018 URL hattps://proceedinggs neweess.errppp.. cc/
Yubei Chen, Zeyu Yun, Yi Ma, Bruno Olshausen, and Yann LeCun. Minimalistic unsupervised representation learning with the sparse manifold transform. In The Eleventh International Conference on Learning Representations, 2023. URL httpps://peerreview www.kewwwee//ooorreeeeeeeeeeee innt//orummiddinn _nBVKAhhD
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudík (eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pp. 213-223. Fort Lauderdale, FL, USA, 11-13 Apr 2011. PMLR. URL https: 7/proceedings. mlr. www.ppwwssss..eeeeeee. http///ateessle 1athmmllm
Taco Cohen and Max Welling. Group equivariant convolutional networks. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference Machine Learning, volume 48 of Proceedings of Machine Learning Reseurch, pp. 2990 2999, New York, New York, USA, 20- 22 Jun 2016. PMLR. URL https: https: http:p::////coeeengg.... mmr. wttp://p...eeee//////oeoooctoaoa html
Marissa Connor and Christopher Rozell. Representing closed transformation paths in encoded network latent space. Proceedings of the AAAI Conference Artificial Intelligence, 34(04))3666-3377 3675, Apr. 2020. doi: 10.1609/ 111609/aaaiiv3/tt v3404..340
URL https://ojs. aaai www//oigggeeeeee... php/ /At/////rratccle/e//////////
Marissa Connor, Gregory Canal, and Christopher Rozell. Variational autoencoder with Iearned latent structure. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp. 2359 2367. PMLR, 13 15 Apr 2021. URL https: httpps:/prroceedingg.uu http:/diiinn.m.....rr.eeeeeeeeee vatees//v33//cooooer html
Marissa Catherine Connor, Kion Fallah, and Christopher John Rozell. Learning mations on data manifolds. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL
transfor-
https://openreview htwppppwwweeeeeett...aaaaaraaaee .ett.foorummidddyyhiiirr...
Romain Cosentino, Randall Balestriero, Richard Baranuik, and Behnaam Aazhang. Deep autoencoders:
From understanding to generalization guarantees. In Joan Bruna, Jan Hesthaven, and Lenka Zdeborova
(eds.), Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference, volume 145 of Proceedings of Machine Learning Research, pp. 222. PMLR, 16-19 Aug 2022a. URL https: : //proceedings..
.ariinggsmlll........ mlr. www.m.......ee.e.
html
Romain Cosentino, Anirvan Sengupta, Salman Avestimehr, Mahdi Soltanolkotabi, Antonio Ortega, Ted Willke, and Mariano Tepper. Toward a Geometrical Understanding of Self-supervised Contrastive Learning, May 2022b. URL http: wttpi//rrrabbb//25..... 06926 arXiv:2205.0692 [os].
Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with reduced search space. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and
15

---

Published in Transactions on Machine Learning Research (02/2024)
H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 18613-18622 Cur- ran Associates, Inc., 2020. URL
pdf.
Benjamin Culpepper and Bruno A. Olshausen. Learning transport operators for image manifolds. In Y. Bengio, D. Schuurmans, J. D. Lafferty, C. K. I. Williams, and A. Culotta (eds.), Advances in Newral Information Processing Systems 22, pp. 423-431. Curran Associates, Inc., 2009. URL http:://pppper..a..////// peappprrs.ipps.... cc/pper//
htw--eaannnn--taaasa------epeeee -or-oorg-----------a------- maniiolds. pdf
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hicrarchical image database. In 2009 IEEE Conference on Compuler Vision and Pallern Recognition, pp. 248 255, June 2009, doi: 10. RLL.L////V..009.....
ISSN: 1063-6919.
James J. DiCarlo and David D. Cox. Untangling invariant object recognition. Trends in Cognitive Sciences, 1188)3333-441 August 2007. ISSN 1364-6613. doi: 10.1016|).dees2200..600000
James DiCarlo, Davide Zoccolan, and Nicole C. Rust. How Does the Brain Solve Visual Object Recognition? 2012.01..0..........
Neuron, 73(3):415-434 434, February 2012. ISSN 0896-6273. doi:
Peter C. Dodwell. The Lie transformation group model of visual perception. Perception & Psychophysics, 34 (1):1 16, January 1983, ISSN 1532-5962. doi: 10.3758 /FF02058900 URL https://doi. wttp////11.79990 BF03205890|
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: Nearest-neighbor contrastive learning of visual representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9588 9597, October 2021.
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-supervised representation learning- In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 3015-30 3024. PMLR, 18 24 Jul 2021. URL htttps://proceeedingg mlr. wtt.:e/a.w...rr.e..eeeeeeeeeeeee hatass/v139///// htt.33//mmoool22....a. html
Kion Fallah and Christopher J Rozell. Variational sparse coding with Icarned thresholding. In Proceed- ings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 6034-6058, PMLR, 17 23-23.. Jul 2022. URL http::://proceeednnggeeee////e// mlr. www.ppres/..... press/v162/ Ballah22a.html
Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Sociely, 29(4) 9833 1049, February 2016. ISSN 0894-0347, 1088-6834. doi: 10. 00000//aam//2222 URL https: //wp:::///www... ams org/ jatg//aa///// 000000099000000000--000-0------0
Yean-Bastion Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent - new approach to self-supervised learning. Tn H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Pro- cessing Systems, volume 33, pp. 21271 21271-21284 212244 Curran Associates, Inc., 2020. URL htttp::// https://proceeeding: Paper pdf
courips.cc/papee files/ www.s//pppeeeee///e/eeeeeee
R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality Reduction by Learning an Invariant Mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Volume 2 (CVPR 06), volume 2, pp. 1735-1742, New York, NY, USA, 2006. IEEE. ISBN 978-0-7895-2997-6 doi: 10.1109/ C..1099/CPR... 2006...000.100. URL httt:://eeeepplrseee ieee. arg/ddccumet//640096.
Brian C. Hall Lie Groups, Lie Algebras, and Representations: An Elementary Introduction, volume 222 of Graduate Terts in Mathematics. Springer International Publishing, Cham, 2015. ISBN 978-3-319-13666 doi: 10.1007 10.100779978-3-111--36666
6 1394-7-3
URL https://limg http://aw..pppiierrrrrrr//oo//./
978- 978-3-319-13467-3
16

---

Published in Transactions on Machine Learning Research (02/2024)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Compuler Vision and Pallern Recognation (CVPR), pp- 770-778 2016. doi: 10.1109/CVIR CVP...016.9.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pollern Recognilion (CVPR), June 2020.
Trina Higgins, Loie Malthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE, Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 2017. URL https:/ het/forrumidd-SyfaaUee
Mark Tbrahim, Diane Bouchacourt, and Ari Morcos. Robust selfsuueereeeeeee learning with lie groups. 2022. doi: 10.48550/ARXI 2210.13356. URL http::::///xxxv..rr////aa/
13356
Joern-Henrik Jacobsen, Jens Behrmann, Richard Zemel, and Matthias Bethge. Excessive invariance causes adversarial vulnerability. In International Conference on Learning Represenlations, 2019. URL https
//openreview hett//forumiid-Bkfbppaaee
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In International Conference on Learning Representations, 2022. URL https
//openreview.net//eeee wtt.:e/e//ooorooo/deeeeoSSoSSSS
Michael I. Jordan, Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Saul. An Introduction to Variational Methods for Graphical Models. In Michael I. Jordan (ed.), Learning in Grophical Models, pp. 105-161. Springer Netherlands, Dordrecht, 1998. ISBN 978-94--10-61044-9 978-940-1--------111 doi: 10.1007/978--4 0117999799---1--014--90199 5. URL http://link.
Diederik P. Kingma and Max Welling, Auto- Encoding Variational Bayes. arXiv:1312.6114 [es, stat], May 2014. URL
wwwporggg/aaaaaaaae 11t1/.111......//.// arXiv: 1312.6114.
Alex Krizhevsky. Learning multiple layers features from tiny images. pp. 32-33, 2009. URL https: : w/www.cs.trront...o.. toronto.
Chia-Wen Kuo, Yhih-Yao Ma, Jia-Bin Huang, and 7solt Kira. Featmatch: Feature-based augmentation for semi-supervised learning. ar arXiv:2007.08500 CV,, 2020. URL http://///wwwww.r//aaa/a/ 20000000000000
Dong-Hyus Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. ICML 2013 Workshop Challenges in Representation Learning (WREPL), 07 2013.
Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL httpgp//pppeaeeeeee..ee////rrrrr
Najib J. Majaj, Ha. Hong, Ethan A. Solomon, and James J. DiCarlo. Simple Learned Weighted Sums of Inferior Temporal Neuronal Firing Rates Accurately Predict Human Core Object Recognition Performance. doi: 10.1523/
Journal of Neuroscience, 35(3)):13402 (340-)113402-131888,, September 2015. ISSN 0270-6474, 1529-2401.
JNEUROSCI518I-14.201
Xu Miao and Rajesh P. N. Rao. Learning the Lie Groups of Visual Invariance. Newral Computation, 19 210:2666-2693, October 2007. ISSN 0899-7667, 1530-888X. doi: 00.162//c00.207.1910000000 URL
Artem Moskalev, Anna Sepliarskaia, Ivan Sosnovik, and Arnold Smeulders. Liegg: Studying learned li group generators. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 25212-25223. Cur- ran Associates, Inc., 2022. URL https://proceednns nww.ine.rrrcccooooooo. http://pweoosss...o..ccooopppppp 12729326e220000477errrttt2222222
17

---

Published in Transactions on Machine Learning Research (02/2024)
Hariharan Narayanan and Sanjoy Mitter. Sample complexity of testing the manifold hypothesis. In J. Lafferty, C. Williams, J. Shawe-Taallor, R. Zemel, and A. Culotta (eds.), Advances in Newral Information Processing Systems, volume 23. Curran Associates, Inc., 2010. URL https://prrcceddiiii
Paper. pdf
Bruno A. Olshausen and David J. Field. Emergence of simple-cell receptive field properties by learning a sparse
code for natural images. Nature, 381 (6683) June 1996. ISSN 1476-4687. doi: 10. 10.1038/38 /088/30000
URL htllhttpp::. : //www/ nature. wwt./aatiiceeee/1116111101
Number: 6583 Publisher: Nature Publishing Group.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Prodictive Coding, January 2019. URL http://arxiv http:///arddi.orr/aaa///11. 037488 AxXiv:1807.03746 (cs, stat].
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019. URL http://papers. wtwp:/sssnnuirpp.cccepeeoaa pdd
Phil Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact on learning. In International Conference on Learning Regresenlations, 2021. URL
http::/ppenneerrrrreeeeerroroaop net//ooummiid-KJ119KK666 https://openrevier
Rajesh Rao and Daniel Ruderman. Learning Lie Groups for Invariant Visual Perception. In M. Kearns, S. Solla, and D. Cohn (eds.), Advances in Newral Information Processing Sys- tems, volume 11. MIT Press, 1998. URL https://proceeddngs.oo .pdf
cc/paper//198/f111
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Eric P. Xing and Tony Jebara (eds.), Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 1278-1286, Bejing, China, 22-24 Jun 2014. PMLR. URL httpp:://proceedingg: mlr.. www.ppess/v... press/v32/ rezende 14.
Salah Rifai, Yann N Dauphin, Pascal Vincent, Yoshua Bengio, and Xavier Muller. The Manifold Tangent Clas- sifier. In Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011, URL
https://papers. wewaaperssnipp..oomm https://papers.nips.ccc
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. Tn 2007 15th European Signal Processing Conference, pp. 606 610, 2007.
Lawrence K Saul, T Labs, Park Ave, Florham Park, and Sam T Roweis. An Introduction to Locally Linear Embedding. URL
pdf
Jascha Sohl-Dickstein, Ching Ming Wang, and Bruno A. Olshausen. An Unsupervised Algorithm For Learning Lie Group Transformations. arXiv:1001.108 [cs], June 2017. URL http:://rxiii http:://axxiv.og////// aaspa/aa//11100000 1027 arXiv: 1001.1027.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning Structured Output Representation using Deep Conditional Generative Models. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL https: .dd]]
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence.. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (ods.), Advances in Neural Information Processing Systems, volume 33, pp. 596-608. Cur- ran Associates, Inc., 2020. URL https::/ htt.rppp.ccpppp....oo//////////p http:///iieee//pe///////1/// DBGG4aeeeaabbccccc66688888333333
18

---

Published in Transactions on Machine Learning Research (02/2024)
J. B. Tenenbaum. A Global Geometric Framework for Nonlinear Dimensionality Reduction. Science, 290 (5500):2319 23233 December 2000. ISSN 00368075, 10959203. doi: https://wwz sciencemag.
URL
http:///a........///iiicc.eeeeee www.ciieeeeee..........o 2319
Francesco Tonolini, Bjørn Sand Jensen, and Roderick Murray-Smith. Variational Sparse Coding. In Proceedings The 35th Uncertainty in Artificial Intelligence Conference, pp. 690-700. PMLR, August 2020. URL
https://proceedingss wttpeeeedding......pp.p.. wl...prees/v11//tooolllmmm
html ISSN: 2640-3498.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machiie Learning Research, pp. 6438 6447. PMLR, 09 15 Jun 2019a.
URL https://proceecinng mlr www.mmrrr..sss.. wwwpppesvvvr/e /97 wttp:///vv/vve..vv////// vermal9a..tmm
Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for semi-supervised learning. In Proceedings of the Twenty- Eightt International Joint Conference
Artificial Intelligence, IJCAI-19, pp. 3635 3641. International Joint Conferences Artificial Intelligence Organization, 7 2019b. doi: 10.24963 //sj//icii./// htt/://jaaaaa.///////1 2019/504. URL
24933./jaaa....aa...... 2019/504
Thomas Edward Yerxa, Yilun Kuang, Eero P Simoncelli, and YueYeon Chung. Learning efficient coding of natural images with maximum manifold capacity representations. In Shirty-seventhh Conference on Neural Information Processing Systems, 2023. URL http::///w.gpgaeeeeeeeeeee/rrrrp
Cheng Zhang, Judith Butepage, Hedvig Kjellstrom, and Stephan Mandt. Advances in Variational Inference. [cs, stat), October 2018. URL http: org/abs//711..0559 arXiv: 1711.05597.
19

---

Published in Transactions on Machine Learning Research (02/2024)
6 Appendix
6.1
Appendix A: Training Variational Lie Group Operators
When training the VLGO, there are several metrics of success use to evaluate the model. For given point pair xx) and coefficients c, the most important metric is the distance improvement (DI) from the operators:
x Tw (c)x, X
DI
(8)
x
This metric indicates how effective the operators/coeffcientt inference stratogy is in transporting from x; to X4. When training the VL.GO in the feature space of a model, this metric is especially important ensure that the model is not minimizing L. by trivially collapsing the features. is important also measure the average magnitude of non-zero coefficients and the average Frobenius norm of the Lie group operators to ensure they are not increasing without bound. To help ensure stability, we apply gradient clipping to both the Lie group operators and the weights of the coefficient encoder and prior at each training step. Additionally, we clamp the estimated distribution parameters from the encoder and prior network. It is critical to not set the clipping/ clippnggcclmmping values loo high, as it can harm performance.
Tt is also important to learn operators that are approximately stable, with almost entirely imaginary eigenvalues (i.c., real components near zero) defining cyclic paths. This is important to prevent applied augmentations from causing unbounded growth in the magnitude of the features (see appendix of Connor el al. (2023)). We have found that the initialization of the Lie group operators affect the spectra of the final learned operators. As such, we initialize each operator in the dictionary with the following block-diagonal form:
0
pT
a1
0
0
0
0
d//2
d/2
0
8m
d/2
4//2
where at" and 3."h denote the real and imaginary component the ith and i+hth eigenvalue, respectively, Note that imaginary eigenvalues come conjugate pairs. For the ManifoldCLR experiments, we set a 1.0e-4 and 8Tm 6.0.
Finally, in the when learn the prior, apply warmup of the estimated prior distribution parameters. For the first 5000 iterations of training, apply a linear warm-up from the encoded prior parameters and fixed prior parameters. Let (H,, b.) go (z;) be prior Laplacian parameters for input and E [0, 1] be a warm-up scalar. In the first 5000 iterations, we set the prior parameters as:
(1i, bz) = 15/us + (1- k))o, kh; + (1 - x3)bo).
where 14 and by are initial, fixed prior parameters. We use initial prior distribution parameters in our evaluation in Figure 5(a)
6.1.1 Computational and Memory Complexity of Lie Group Operators
In this work, take three steps to improve computational complexity: (1) variational inference to avoid the need for iterative optimization procedure at each training step, (2) block diagonal structure on the learned operators, and (3) reducing the magnitude of the cocfficients via the scale parameter of the prior distribution or additional &2 regularization For the malrix exponential, we use an optimized Taylor series polynomial Bader et al. 2019) with computational complexity OKK)3), where K is a term that depends on the norm of the matrix, as described Equation of Bader al. (2019). Hence, reducing the magnitude of the coefficients leads to decrease in matrix norm in Equation of our manuscript, and hence the computational complexity.
20

---

Published in Transactions on Machine Learning Research (02/2024)
Table 5: Comparison of memory usage and average iteration time between SimCLR and ManifoldCLR on TinyImageNet with batch size of 512 using Nvidia A100 GPU
Method
GPU Memory (GB)
Avg Time per Iter (sec)
SimCLR ManifoldCLR
13.4
0.0902
15.2
0.4284
Table 6: Comparing the effect of block dimension of ManifoldCLR on memory usage and average itcration
time on Net with a batch size of 512 using four Nvidia RTX 6000 GPUs
Block Dimension
GPU Memory (GB)
Avg Time per Iter (sec)
32
25.80
0.4735
256
52.42
0.5279
512
70.68
0.9051
We include an additional experiment comparing the empirical computational and memory complexity of different methods on the Tinylmagenet dataset. From Table 5 it can be seen that ManifoldCLR incurs only a slight memory overhead compared to SimCLR, with the previously stated limitation of a higher computational complexily. This comes with the advantage of learned operators that can be applied for downstream tasks. Fortunately, Table + suggests that imposing block diagonal constraint significantly reduces computational complexity. We caution against comparison between Table 5 anndd Table 0 since they use different GPU architectures and Table 6 requires storing the model on cach GPU.
6.2
Appendix B: Image Experiments
6.2.1
Experimental Setup
For each dataset, we train a ResNet-18 (He et al., 2016) with the Adamn optimizer (Lohhhhlovv & Hutter 2019) for 1000 epochs using a batch size of 512. We set the backbone and projection head learning rate 3.0e-3 3 for CIFARIO and 2.0e-3 for STL10 and TinyImageNet. For every model, set the learning rate of the Lie group operators and cocfficient encoder to 1.0c-3 and 1.0c-4, respectively. We set the weight decay equal to 1.0e-5 for the backbone, projection head, and coelficient encoder and equal to 1.0e-3 for the Lie group operators. We use a cosine annealing scheduler with 10 warm-up epochs and a minimum learning rate of 1.0c-5 5 for all parameters.
For all datasels, we modify the first convolutional layer of the ResNet-18 to have a 3x3 kernel size with a stride and padding of and no bias term. For CIFAR1O, we also remove the max pool layer. We use images of size 32x32 for CIFAR10 and 64x64 for STL10 and TinyImageNet. When applicable, we use single hidden layer projection head with 1024 hidden units, BatchNormid, and an output dimension of 128. To augment the images, we first apply random color jitter with brightness, contrast, and saturation set to 0.4 and hue set to 0.1 with probability 0.8. We then apply a random grayscale with probability 0.1. Afterwards, we apply a random resize/crop with scale between 0.2 and 1.0 and ratio between 0.75 and 1.33. Finally, we apply a random horizontal flip with probability 0.5 and normalize the image using the standard mean and standard deviation value for each datasct.
For our coefficient encoder architecture, we use two-hidden-llayr MLP with leaky ReLU non-linearity. The encoder takes concatenated, detached features as input (dimension equal to 1024) with hidden layers with 2048 units. The output of the MLP is 512, after which a single lincar layer is used to estimate log scale and shift. for each coefficient component. Our prior network uses a similar architecture ure, except with an input dimension 512 and the second hidden layer of the MLP with 1024 units. Due to memory constraints from taking multiple samples, we use a standard Laplacian prior for all ManifoldCI.R experiments except CIFAR10 with a projection head. In that case, we apply threshold Ç 0.01 with J 20 samples. For all methods, set the initial prior shift equal to 0.05 and scale equal to 0.01.
21

---

Published in Transactions on Machine Learning Research (02/2024)
42
40
38
34
32
30
16
128
256
512
0.01
0.1
1.0
10.0
100.0
16
32
64
128
256
512
Dictionary Size
Lm Weight (b)
Block Size
(a)
(c)
Figure 6: Linear probe accuracy on a TinylmageNett model trained with projection head while varying different components of the ManifoldCLR system. These include the (a) dictionary size M, (b) weight on the manifold loss term X, and (c) block size for the dack-diagonal constraint on operators.
Table 7: Linear probe accuracy for CLr (Jing et al. 2022 and Direct ManifodCLLL (named Manifol Evaluated using an InfoNCH loss with normalized features (i.e., cosine similarity) and without normalized features (i.e., MSE). All methods trained without a projection head. Best method(s) for each dataset are bolded.
Method
Normalized
CIFAR10
STL-10
TinylmageNet
SimCLR-None
V
80.49%
78.29%
24.47%
DirectCLR
V
85.55%
82.99%
33.29%
SimCLR-None
X
88.58%
84.53%
36.26%
DirectCLR
X
88.46%
84.46%
36.20%
ManifoldCLR
X
88.89%
84.99%
38.68%
ManifoldDirectCLR
X
89.73%
87.47%
42.22%
To perform the linear probe evaluation, freeze the backbone and encode features of the entire dataset without augmentations. We then train a single linear layer for 500 epochs using the Adam optimizer. We use an exponential decay on our learning rate at the end of each epoch, starting at 1.0e-2 and ending at 1.0e-5.
6.2.2 Additional Results
Table 8: Comparison of effective rank of backbone features between SimCLR and ManifoldCLR with different block sizes in the Lie group operators. All methods trained on TinylmageNet with projection head. Block size of 512 indicates no block-diagonal structure in the operators.
Method Block Size
Effective Rank
SimCLR
6.01
ManifoldCLR
512
5.87
ManifoldCLR
256
5.75
ManifoldCLR
128
5.37
ManifoldCLR
64
5.25
ManifoldCLR
32
4.23
ManifoldCLR 16
3.32
22

---

Published in Transactions on Machine Learning Research (02/2024)
Table 9: Paired I-lest p-values between variational Lie group operators and every other method over 50 random data splits on semi-supervised experiment using 5 labels/chee classs from main text. Alternative hypothesis is that mean of VI.GO accuracy is greater than each method.
Method
CIFAR10
STL-10
TinyImageNet
Baseline Pseudo-labeling
1.33e
1.29e 21
-840e-38
3.58c-11 16
5.85e 10
6.10e-13 13
Manifold Mixup ICT
4.91e 32
2.48e 27
9.63e- 40
Matt
1.25e
2.42e
3.48e-35 35
Baseline Pleeco abeling
MMICT
23
MLG0
Accuracy
Accuracy
Accuracy
Baselino
Baseline
Pocuco-labeling
Pooudo
Pbouut-labetingg labeling
MMIC
MMICT
65
Fautbashh
FeutWach
M.G0
V.30
Dataset Split
Dataset Split
Dataset Split
(a) CIFAR10
(b) STL10
(c) TinnylmaeNeet
Figure 7: Per trial performance, sorted by baseline performance, of semi semi-supervised experiments with 5
labels/ch /class taken over 50 data splits. For almost every split, VLGO provides the highest accuracy.
85
Accuracy
Accuracy 75
Accuracy
22
20
21
Baseline Pheucio-abbl MISICT
Match
Baseline Pheaco-label MICCCT
PeetMatthh
VLGD
Beswine Paessto-labe
MMCIC
Pee:Matthh
VLGD
(a) CIFAR10
(b) STL10
(c) TinylmageNet
Figure 8: Semi-supervised experiments using a frozen backbone and learned, single hidden layer MLP with 5 labels/ cclass, taken over 50 data splits. Best viowed zoomed in.
08
Accuracy
07
540
Recural B0
36 00
36 75
88
830
86 25
84
35
$15
Basefire
Pacudorlaoul MMITT FeutAusc
V_80
Baseire
Paaudorabe
NMITT
VUCO
Dostin::
Pacudtiibte
MW
PootArich
(a) CIFAR10
(b) STL10
(c) TinylmageNet
Figure 9: Semi-supervised experiments using a frozen backbone and learned, single hidden layer MLP with 50 labels /class, laken over 50 data splits. Best viewed zoomed in.
23

---

Published in Transactions on Machine Learning Research (02/2024)
Table 10: Average percent improvement in semi-supervised accuracy over baseline over 50 splits with varying datasets and methods for incorporating feature augmentations using 50 labels /llass. Best method(s) for each dataset are bolded.
Method
CIFAR10
STL-10
TinylmageNet
Pseudo-labeling
0.59 + 0.12%
0.62 0.21%
0.16 0.11%
Manifold Mixup ICT
0.02 + 0.07%
0.02 0.09%
-0.06 = 0.21%
FeatMatch
0.10 + 0.08%
0.08 = 0.08%
0.10 - 0.21%
Variational Lie Group Operators
1.42 0.37%
0.85 = 0.30%
0.22 + 0.12%
Table 11: Average percent improvement in semi-supervised accuracy over baseline over 50 splits with varying datasets and methods for incorporating feature augmentations using 100 labels/ class. Best (sthod for each dataset are bolded.
Method
CIFARIO
STL-10
TinylmageNe
Pseudo-labeling
0.30 + 0.09%
0.31 0.13%
0.11 0.09%
Manifold Mixup ICT
0.02 + 0.08%
0.02 0.08%
0.00 0.22%
FeatMatch
0.01 + 0.07%
0.02 0.10%
0.66 + 0.25%
Variational Lie Group Operators
0.82 + 0.19%
0.46 0.19%
0.14 + 0.10%
86.25
Razurarc
8
$4.
$3.
NMAJI
VLU8O
Bascking
Pazudoliubo
NMMUT
FoulMakn
NuGO
Bostra: Pacudelited
(a) CIFAR10
(b) STL.10
(c) ThyyImggeNet
Figure 10: Semi-supervised experiments using frozen backbone and learned, single hidden layer MLP with 100 labels/class, taken over 50 data splits. Best viewed zoomed in.
24


Reviews:
[{"review_id": "BLoC3QGiIi", "reviewer": "Reviewer_2HHn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n\n* The proposed method is an interesting and relevant synthesis of self-supervised learning and generative model using Lie group transformations. There is a natural synergy between local manifold augmentations generated by Lie groups and those required by self-supervised learning and the direction seems promising.\n* Empirical results show good performance compared to the base SimCLR model and other manifold-based contrastive learning approaches.\n* The authors found ways to tackle computational limitations from the dimensionality of the embedding space by imposing block sparse structure on the learned transformation matrices.\n* Results show a relatively small gap between models trained with and without a projection head, in contrast to SimCLR and other similar models. This could lead to more interpretable embedding spaces accompanied by models of local transformation structure.\n* Each component was ablated on the non-trivial task of Tiny ImageNet linear probe accuracy.\n\n*Weaknesses*\n\n* The method still relies on the use of hand-chosen image augmentations. The learned augmentations in the latent space essentially re-learn hand-chosen image augmentations. Thus, the method does not achieve the ultimate goal of contrasting views from the image manifold rather than views from hand-crafted transformations as discussed in the introduction. Nonetheless, this approach could be a step towards such a method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates learning a self-supervised contrastive image model jointly with a model of local transformations along an image manifold. Local transformations are represented as a Lie group of matrix transformations that take a point on the image manifold as input and output a neighboring point on the image manifold. The local transformations are the weighted sum of a dictionary of learned matrices with coefficients that follow a sparse distribution. The motivation of using a Lie group generative model as part of the self-supervised learning pipeline is to enrich the contrastive embedding space with by modeling latent Lie group transformations. The contrastive model is trained using standard InfoNCE loss along with a variational loss that learns a CVAE modeling the Lie group transforms in the latent embedding space. The CVAE model consists of a learned prior that takes the starting state and learns an output that is reparameterized into a sparse distribution on weighting coefficients, along with an encoder network used for variational approximation that takes a pair of neighbors and outputs parameters for a variational weighting distribution. Samples from the latent Lie group samples are used as augmentations in addition to standard SimCLR augmentations. Experiments show that the applying the method along with SimCLR augmentations outperforms the base SimCLR model and other manifold variants of SimCLR in downstream linear probing tasks. Experiments using features from the model for weakly supervised classification show improved performance compared to baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I do not have any significant requests for changes at this time.", "broader_impact_concerns": "Broader impacts are not discussed, but this does not impact my assessment of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vHiLlwxYJZ", "reviewer": "Reviewer_pMNS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Using Lie groups for manifold learning in the context of contrastive learning is interesting\n- Experimental results are good, showing improvements over competing methods\n- Ablation studies provided for many design choices\n- Paper provides good discussion of related works \n\nWeaknesses:\n- It is unclear from the paper how the different views of the same example/close by examples on the manifold ($z_i$ and $z^{'}_i$) are selected, which is key to the algorithm. I would like to understand how this is done before making a decision on the soundness of the framework. For instance, if they are done using transformations then those should be representable by Lie groups for the framework to make sense.\n- The paper is very similar to Ibrahim et al. (2022) as discussed in the related work, though there are 2 key differences highlighted. Ablation studies showcase the importance of incorporating feature augmentations from Lie group operators into contrastive learning, but there is no study showing the benefit of the variational approach used in this paper.\n- The VLGO component seems to be generic but results are only shown with the SimCLR framework.\n- On the one hand the manifold loss is shown to be important for performance in Table 3, but in a different set of ablation experiments on $\\lambda$, the paper concludes that changing the weight \"provides little difference on downstream performance\". These two results should be somehow reconciled.\n- The paper does not include sufficient background on several of the components to make it self-contained. In particular, an introduction to Lie groups/algebra would be helpful, as would a bit more detail on the variational sparse-coding component in the main text (e.g. what is the factorial prior used).\n\nOther comments:\n- In Figure 2, many of the augmented images appear very similar - are there actually differences (e.g. 2(a) first row 2, 4, 7, 8, 9 and 3, 5, 6)?\n- Would be interesting to understand what the operators in Fig 5b are doing and if they make sense for the classes\n\nTypos:\n- Citation format should be corrected to use parenthetical citations unless referring to authors.\n- (Page 3, just above Eq 1) \"the Lie group operators model an infinitesimal transformation\" - should this be \"Lie algebra operators\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Lie group operators as a way to learn manifold structure in the context of contrastive learning. The operators are applied in a lower dimensional feature space and learnt using variational sparse coding techniques. The resulting training objective is a sum of the usual InfoNCE loss for contrastive learning with learnt transformations applied, the manifold reconstruction loss and the loss for the VAE used for variational sparse coding. A block-diagonal approximation is further applied to make learning the Lie operators tractable. The framework is evaluated in the context of standard self-supervised and semi-supervised settings where it is shown to outperform other contrastive learning and semi-supervised learning techniques.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Clarify how close-by examples on the manifold ($z_i$ and $z^{'}_i$) are selected \n- Provide more background and detail on Lie groups and variational sparse coding\n- Ablation studies to show the effectiveness of the variational approach for learning coefficients over a determinstic estimate (e.g. in Ibrahim et al. (2022))\n- [Optional] Show that VLGO works with other self-supervised learning frameworks", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NzP6ZqL7MH", "reviewer": "Reviewer_FZ1K", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- Proposed a principled way to incorporated manifold modeling into self-supervised learning representations. The method is end-to-end trainable and doesn't require a pre-trained feature extractor, unlike some prior works applying manifold operators. \n- Quantitatively demonstrated their framework, MainfoldCLR, improves performance on self-supervised benchmarks over baseline methods.\n- VLGO provides a flexible way to model manifold variations and sample useful augmentations for downstream tasks like semi-supervised learning.\n- The learned prior distribution provides a way to quantify which augmentations are identity-preserving at different points on the manifold. This is learned in a completely unsupervised way.\n\n**Weakness:**\n- Due to the high computation and memory cost of matrix exponential, it is challenging to scale up this method to larger datasets. Even though the author proposed block diagonal modification to alleviate this issue, the ManifoldCLR spent 3X runtime than baselines on the TinyImageNet dataset, implying the computational cost is still expensive. \n- While effective manifold augmentations are demonstrated for images, it's unclear if this approach would work well for other data modality like audio, video, graphs, etc. Additionally, having experimental results on larger datasets would enhance the generalizability of the conclusions.\n- If it is available, providing intuitions for what semantic transformations the learned operators represent.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Self-supervised learning has become a prevalent paradigm for learning general purpose representations from unlabeled data. However, most of existing methods rely on pre-defined augmentations and do not explicitly enforce manifold structure in the learned representations. This is suboptimal, as manifold models provide a natural framework for capturing invariant features, similar to how biological vision systems are believed to create linearly separable category manifolds.\n\nTo address these limitations, this paper proposes ManifoldCLR - a principled approach to incorporate manifold modeling into contrastive self-supervised learning. The main contribution is a variational Lie group operator (VLGO) model that captures manifold variations and allows sampling of augmentations along the learned manifold. The model is integrated into contrastive learning, where manifold samples provide additional positive pairs to improve representation learning. Experiments demonstrate benefits of ManifoldCLR on self-supervised benchmarks and downstream tasks compared to baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Discussion and major concerns:**\n- I would recommend adding analysis of the computational and memory complexity of the ManifoldCLR framework, either in the main paper or an appendix. Understanding the operational costs is important for practitioners looking to adopt this approach.\n- While the learned augmentations do seem to preserve identity well, an open question remains around whether they provide sufficiently challenging positive samples for contrastive learning. The pre-defined augmentations in standard contrastive learning can sometimes generate very challenging positive pairs, forcing the model to learn more robust and generalizable features. The identity-preserving priors in ManifoldCLR may not distort the data to the same extent. Further analysis or visualization of the augmentations could provide insight into the level of difficulty and study the impact on generalization performance. \n- The infoNCE loss presented in eq(4) differs slightly from the loss defined in other works, such as CPC, SimCLR etc, where they compute the similarity score by taking the inner product of vectors. In eq (4), the square of the $\\ell_2$ norm is calculated instead. The empirical results of ablation Table 4 shows that without normalized feature outperformed normalized features. This is very interesting. I would greatly appreciate it if the author could provide more insights into the motivation behind and advantages of this modification.\n- In Algorithm 1, line 2, what is $g_\\theta$ stands for? Is it a typo, and should it be referring to \"$q_\\theta$\"?\n- I am wondering would the effectiveness of the prior augmentations diminish when labels are available during downstream tasks?", "broader_impact_concerns": "I have No Broader Impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Given that the two criteria (Claims & Evidence and Audience) are satisfied, I recommend acceptance with the following minor revisions:

- As Lie groups are relatively less well-known within the machine learning community, the authors are encouraged to expand the background on Lie group operators and possibly integrate relevant portions of it into the main paper. 
- The NNCLR results should be integrated into the main paper.
- Please check the equations in section 2.2.
- Please ensure proper formatting of references.

In addition, see the comments above regarding claims about the broader applicability of the method and ensure that any claims in the revised version of the paper (i.e. outside Section 4.2.1) are similarly scoped.
Decision: Accept (poster)