Robust Self-Supervised Learning with Lie Groups
Authors: Mark Ibrahim
Diane Bouchacourt
Ari S. Morcos
Venue: Submitted to ICLR 2023 (2023)
Keywords: robustness
computer vision
generalization
self-supervised learning
out-of-domain generalization

Abstract: Deep learning has led to remarkable advances in computer vision. Even so, today’s best models are brittle when presented with variations that differ even slightly from those seen during training. Minor shifts in the pose, color, or illumination of an object can lead to catastrophic misclassifications. State-of-the art models struggle to understand how a set of variations can affect different objects. We propose a framework for instilling a notion of how objects vary in more realistic settings. Our approach applies the formalism of Lie groups to capture continuous transformations to improve models’ robustness to distributional shifts. We apply our framework on top of state-of-the-art self-supervised learning (SSL) models, finding that explicitly modeling transformations with Lie groups leads to substantial performance gains of greater than 10% for MAE on both known instances seen in typical poses now presented in new poses, and on unknown instances in any pose. We also apply our approach to ImageNet, finding that the Lie operator improves performance by almost 4%. These results demonstrate the promise of learning transformations to improve model robustness.

Under review as a conference paper at ICLR 2023
ROBUST SELF-SUPERVISED LEARNING WITH LIE GROUPS
Anonymous authors Paper under double-bling review
ABSTRACT
Deep learning has led to remarkable advances in computer vision. Even so, today's s best models are brittle when presented with variations that differ even slightly from those seen during training. Minor shifts in the pose, color, or illumination of an object can lead catastrophic misclassifications. State-of-the art models struggle to understand how a set of variations can affect different objects. We propose a framework for instilling notion of how objects vary more realistic settings. Our approach applies the formalism of Lie groups to capture continuous transformations improve models' robustness to distributional shifts. We apply our framework on top of bate-of-the-art self-supervised learning (SSL) models, finding that explicitly modeling transformations with Lie groups leads to substantial performance gains of greater than 10% for MAE on both known instances seen in typical poses now presented in new poses, and on unknown instances in any pose. We also apply approach to ImageNet, finding that the Lie operator improves performance by almost 4%. These results demonstrate the promise of learning transformations to improve model robustness!.
INTRODUCTION
State-of-the-arr models have proven adept at modeling a number of complex tasks, but they struggle when presented with inputs different from those seen during training. For example, while classification models are very good at recognizing buses in the upright position, they fail catastrophically when presented with an upside-down bus since such images are generally not included in standard training sets (Alcorn et al., 2019). This can be problematic for deployed systems as models are required to generalize to settings not seen during training ("out-of- -ist-ii-udttiiiiionnn (OOD) generalization"). One potential explanation for this failure of OOD generalization is that models exploit any and all correlations between inputs and targets. Consequently, models rely on heuristics that while effective during training, may fail to generalize, leading to a form of "supervision collapse" (Jo & Bengio, 2017; Ilyas et al., 2019; Doersch et al., 2020; Geirhos et al., 2020a). However, a number of models trained without supervision (self-supervised) have recently been proposed, many of which exhibit improved, but still limited OOD robustness (Chen et al., 2020; Hendrycks et al., 2019; Geirhos et al., 2020b).
The most common approach to this problem is to reduce the distribution shift by augmenting training data. Augmentations SimCLR (Chen et al., 2020). While this approach can be effective, it has a number of disadvantages. First, for image data, augmentations are most often applied pixel space, with exceptions e.g. Verma et al. (2019). This makes it easy to, for example, rotate the entire image, but very difficult to rotate a single object within the image. Since many of the variations seen real data cannot be approximated by pixel-level augmentations, this can be quite limiting in practice. Second, similar to adversarial training (Madry al., 2017; Kurakin et al., 2016). while augmentation can improve performance on known objects, it often fails to generalize to novel objects (Alcorn et al., 2019). Third, augmenting to enable generalization for one form of variation can often harm the performance on other forms of variation (Geirhos et al., 2018; Engstrom et al., 2019), and is not guaranteed to provide the expected invariance to variations (Bouchacourt et al., 2021b). Finally, enforcing invariance is not guarantced to provide the correct robustness that generalizes new instances (as discussed in Section 2).
also key for number of contrastive self-supervised approaches, such as
Code reproduce all experiments will he available upon acceptance.

---

Under review as a conference paper at ICLR 2023
Training Data
Self- Suffrviueee Lie Operator
Evaluate Robustness
Known instances
MAE Lie
encourages
g((2) = z
instance in new pose
+13.7%
atypical pose
SSL encoder
Lie Operator
.Z Wnfer tronstormatoo magrituce
X exp( Σ LL
Uhlnown Instance
+10.3%
typical pose
d leamned sigebra besis
Unknown Instance typical pose
+12.4%
typical pose
Loss
Loss
Lie Loss
nor
[|(())
Natural variation
Structure variation Transformations
Generalize variation across instances
in
space
Figure I: Summary of approach and gains. We generate a novel dataset containing rendered images of objects in typical and atypical poses, with some instances only seen in typical, but not atypical poses (left). Using these data, we augment SSL models such as MAE with a learned Lie operator which approximates the transformations in the latent space induced by changes in pose (middle). Using this operator, we improve performance by >10% for MAE for both known instances in new poses and unknown instances in both typical and atypical poses (right).
For these reasons, we choose to explicitly model the transformations of the data as transformations in the latent representation rather than trying to be invariant it. To do so, we use the formalism of Lie groups. Informally, Lie groups are continuous groups described by set of real parameters (Hall, 2003). While many continuous transformations form matrix Lie groups (e.g., rotations), they lack the typical structure of vector space. However, Lie group have corresponding vector space, their Lie algebra, that can described using basis matrices, allowing to describe the infinite number elements of the group finite number of basis matrices. Our goal will be to learn such matrices directly model the data variations.
To summarize, approach structures the representation space to enable self-supervised models to generalize variation across objects. Since many naturally occurring transformations (e.g., pose, color, size, etc.) continuous, representation space (see Fig. 1). Specifically, the Lie operator learns the continuous transformations observed in data as a vector space, using of basis matrices. With this approach, we make the following contributions:
develop
peooneiicall-motivvtttto operator, the Lie operator, that acts in
1. We generate novel dataset containing 3D objects in many different poses, allowing to explicitly evaluate the ability models to generalize to both known objects in unknown poses and to unknown objects in both known and unknown poses (Section 3).
2. Using this dataset, we evaluate the generalization capabilities of number of standard models, including ResNet-50, VIT, MLP-Mixer, SimCLR, CLIP, VICReg, and MAE finding that all state-of-the-art models perform relatively poorly in this setting (Section 3.2).
3. We incorporate our proposed Lie operator in two recent SSL approaches: masked au- toencoders (MAE, (He et al., 2021)), and Variance- Auriance-Invariance-e wwwaariance-ovvriaaae (VICReg (Bardes et al., 2021)) to directly model transformations in data (Section 2), re- sulting in substantial OOD) performance gains of greater than 10% for MAE and of up to 8% for VICReg (Section 4. 4.1). We also incorporate our Lie model SimCLR (Chen et al., 2020) (Appendix E).
Regularization
4. We run systemic ablations of each term of our learning objective in the MAE Lie model, showing the relevance of every component for best performance (Section 4.3).
2

---

Under review as a conference paper at ICLR 2023
5. We challenge our learned Lie operator by running evaluation on Imagenet as well rotated version of ImageNet, improving MAE performance by 3.94% and 2.62%, respectively. We also improve over standard MAE with MAE Lie when evaluated with finetuning the iLab (Borji et al., 2016) dataset. These experiments show the applicability of our proposed Lie operator to realistic, challenging datasets (Section 5).
METHODS
Humans recognize even new objects in sorts of poses by observing continuous changes in the world. To mimic humans' ability to generalize variation, we propose method for learning continuous transformations from data. We assume we have access to pairs of data, e.g., images, x, x' where x' is transformed version of corresponding to the same instance, e.g., specific object, undergoing variations. This is a reasonable form of supervision which one can easily acquire from video data for example, where pairs of frames would correspond to pairs of transformed data, and has been employed in previous works such as Ick & Lostanlen (2020); Locatello et al. (2020); Connor et al. (2021b); Bouchacourt al. (2021a).
Despite widespread use of such pairs in data augmentation, models still struggle to generalize across distribution shifts (Alcorn et al., 2019; Geirhos al., 2018; Engstrom et al., 2019). To solve this brittleness, we take inspiration from previous work illustrating the possibility of representing variation via latent operators (Connor et al., 2021 Bouchacourt et al., 2021a; Giannone et al., 2019). Acting in latent space avoids costly modifications of the full input dimensions and allows to express complex operators using simpler latent operators. We learn to represent data variation explicitly as group actions on the model's representation space Z. We use operators theoretically motivated by group theory, as groups enforce properties such as composition and inverses, allowing us to structure representations beyond that of an unconstrained network (such as multi-layer perceptron, MLP) (see Appendix A). Fortunately, many common transformations can be described as groups
We choose matrix Lie groups to model continuous transformations, they be represented as matrices and described by real parameters. However, they do not possess the basic properties of a vector space e.g., adding two rotation matrices does not produce rotation matrix, without which back-propagatinn propagation is not possible. For each matrix Lie group G' however, there is a corresponding Lie algebra g which is the tangent space of G at the identity. A Lie algebra forms vector space where addition is closed and where the space can be described using basis matrices. Under mild assumptions over G, any element of G can be obtained by taking the exponential matrix map: exp(tM) where M is a matrix in g. and t is a real number. Benefiting from the vector space structure of Lie algebras, we aim learn a basis of the Lie algebra in order to represent the group acting on the representation space, using the exp map. We refer the reader to Appendix A and Hall (2003); Stillwell (2008) for more detailed presentation of matrix Lie groups and their algebra.
2.1 LEARNING TRANSFORMATIONS FROM ONE SAMPLE TO ANOTHER VIA LIE GROUPS
We assume that there exists ground-truth group element, g. which describes the transformation in latent space between the embedding of x and x'. Our goal is approximate g using its Lie algebra. More formally, given pair of samples (e.g., a pair of video frames), x and x', and an encoder, f()) compute the latent representations and d assume z = f(x')= = (x)())) We aim to learn an operator 9 maximizing the similarity between 2 (x)) and z
fxx and 2 f(x'). We therefore
We construct our operator by using the vector space structure of the group's Lie algebra g: we learn a basis of g as a set of matrices Lk E Rmxm k E {L...., d} m is the dimension of the embedding space 2 and d the dimension of the Lie algebra, i.e. the number of Lie generator matrices. Intuitively, d expresses the degrees of freedom required to model the transformations. Specifically, it corresponds to the dimension of the Lie algebra as a vector space?. We use this basis as a way to represent the group elements, and thus to model the variations that data samples undergo. A matrix M in the Lie algebra thus writes M Ed= L2 for some coordinate vector
In our preliminary experiments, have cross-validates hetween d=1 and d=5 and found it did not seem alter the results, hence we use d=1 in our reported results.
3

---

Under review as a conference paper at ICLR 2023
t (fi....,ttl E IR. Let consider pair 2, z with z" g(4). We assume the exponential map surjective?, thus, there exists Σk=1 I g such that
d
9 = exp
t -xp((t)))
(I)
where = (4.....,42) 12) are the coordinates in the Lie algebra the group element and L is the 3D matrix in R mdxmxx concatenating (L, k {I,...,d]}.
Our goal is to infer 9 such that g(z) (the application of the inferred 9 the representation z) equals z². We denote z (zz). Specifically, our transformation inference procedure consists of Algorithm 1. similar to Connor et al. (2021b); Bouchacourt
Steps 1-4 in Algorithm For step et al. (2021a), we use pair of latent codes to infer the latent transformation. Additionally, we consider that we can access a scalar value, denoted òr,n", measuring "how much" the two samples differ. For example, considering frames from given video, this would be the number of frames that separates the two original video frames x and x', and hence comes for free .hh coordinate vector corresponding to the pair (z, 2 is then inferred using multi-layer perceptron h²:
i h(2, x², 8x,x))
When xx is small, we assume the corresponding images and embedding differ by a transformation close to the identity which corresponds to infinitesimal É. To illustrate this, we use squared Ln-norm constraint on For smaller 6x,x' (e.g., close-by frames in a video), larger similarity between 2 is expected, in which case we want to encourage smaller norm of é:
s(z, z 8x.x ||
where: s: x R + [0,1] measures the similarity between z and z". In our experiments, we simply use s(2, z 5x as function of 8x,x' and compute
s(xx.x)) 1 exp(1xxxx)
When xx decreases, 8 increases which strengthens the constraint in Eq. 3: the inferred vector of coordinates is therefore enforced to have small norm.
2.2 LIE LEARNING OBJECTIVE
We encourage 2 to be close to z", while preventing a collapse mode where every transforma- tion would be inferred as identity. In practice, recall we obtain 2 as the embedding f(x) of corresponding frame. Specifically, we use an InfoNCE loss to match pairs":
exp(sim(z2 (2))))),,)))))))))))))))))))(( 22))
Lzie(z2,2 x²) = log exp(sim(zá, 29)) + ΣBEN exp(sim(2, 29))
(5)
where sim is a similarity function measuring how close two latent embeddings are (e.g., cosine similarity in our experiments), and ž belongs to set of negatives A for z For given z of given instance, the set N is comprised of latent codes of other instances. We also include other instances' transformed versions and their corresponding inferred transformed embeddings negative samples.
Crucially, we also include in the negatives of z (denoted A) the frame encoding in order to prevent the model from collapsing to a point where all the encodings of the same instance (i.e. any transformed version) would be mapped the same point, and the inferred 9 would simply learned as identity. Indeed, one can view transformed pairs simply as another form of data augmentation, such that enforcing that 2 matches z (i.e. implicitly enforcing invariance) would
Thii the case if G is compact and connected (Hall, 2003, Corollary 11.10).
"Assuming that nearby video frames will be restricted to small transformations.
5 'During the learning of the parameters of h, we detach the gradients the latent code with respect to the parameters of the encoder f-
Addiitionllly, forward the latent code through neural projection head proj before computing similarity.
4

---

Under review as a conference paper at ICLR 2023
Algorithm I: Implementing our Lie operator in self-supervised learning model
Input: x, x'
Encode f(x), 2 f(x') with the encoder f;
2 Infer È as i hz,z , z", )x,x));
3 Use the exp map to get 0 exp(ttL), where ÿ E Rm
Compute 2 - O(z). This simple matrix-vector multiplication; 5 Jointly learn f. h. and by minimizing:
C = (Laal(Laa)zz) + Lzal(z))) + Aui Lze 20) + deve - 20112 s(0xxx)]((]
(8)
bring robustness to the transformation g. However, we control for this in our experiments using the SimCLR model with our Lie operator (Appendix E, see SimCLR Frames baseline). We experiment with a baseline that simply encouraged to match pairs of rotated frames in the representation, and SimCLR Lie outperforms this baseline. Thus, this shows that viewing transformations as data augmentation is not sufficient for robust generalization.
To further encourage 2 and 2 to match, we add an Euclidean-type constraint:
Aeue| 2 2
(6)
The addition of the Euclidean loss helps to stabilize training, but using only the Euclidean loss (in place of our Lie loss) would not be enough to prevent the model from collapsing to mapping all transformed samples to the same point in latent space. Indeed, in the case of the MAE model, perform ablations of our Lie model in Section 4.3 and show that all terms in the training objective are important for best performance with the MAE Lie model.
2.3 LEARNING A LIE OPERATOR IN SELF-SUPERVISED REPRESENTATIONS
We use our Lie algebra model on top of two existing self-supervised (SSL. models, MAE and VICReg. In MAE, f trained reconstruct a masked input image, while VICReg is trained to match pairs of samples obtained from the same image via some form of data augmentations (e.g. two crops) and uses Variance-IIvariancee Covariiaee regularization to prevent collapse. In Appendix E also experiment applying our Lie operator on the SimCLR (Chen et al., 2020) model, showing gains in the linear setting in particular. Our loss will balance between two components: the original SSL model loss Lest, applied to both samples of the pair separately, and the transformation loss Lrie- Combining these constraints gives us our final loss,
(7) The parameters Asst and ALiO weight the two losses. Note that Lest can be different given the deploy our Lie operator top of the
L = (est( + (zs)) + Atio (2 22) + AenelIz² - 2011² + s(00
SSL model backbone use. In experiment Variance- Invrriance- Invariance-Covarianee Regularization model (VICReg, (Bardes et al., 2021)), and of the very recent masked autoencoders model (MAE, (He al., 2021)))
This loss is minimized with respect the parameters of the encoder f. the parameters of the network h (multi-layer perceptron with two linear layers connected by leaky ReLU) that infers the Lie algebra coordinates, and the Lie algebra basis matrices Lk, k E {i....,de d} which are used compute ÿ. The total loss is over all the instances (N of them) and all possible pairs of transformed samples we have for each instance (if we have P pose changes, this will be P(P- 1)). During training, we estimate this loss for minibatch by sampling instances first, and sampling a pair of samples for each instance without replacement. The full procedure illustrated in Fig. and in Algorithm 1.
3 EVALUATING GENERALIZATION ACROSS INSTANCES
Our goal is to evaluate how well models generalize variation across instances. We focus on robustness pose changes, common, natural variation of objects. We evaluate the learned representations'
5

---

Under review as a conference paper at ICLR 2023
Known instance
in new piposs
Known inchance now 3D poscs
MAC
VIT
ResNNttSS 5D
ViCRing
SmüLH
MUF-Mxer
OUP
hoccsuuionn
-70%
60%
-50%
-35%
-25%
100%
-75%
-00%
-50%
400%
-30%
-22%
-00%
0h
Linear evalaation gaps
Finstuning gaps
Figure 2: SoTA image classification models fail to generalize across poses and instances, Bar plots show the gap in top-1 accuracy relative to training instances with poses seen during training. Self-occlusion is the gap in accuracy between unknown instances in 3D and 2D poses.
robustness via standard linear evaluation or supervised finetuning protocols. Specifically, we measure classification top-1 accuracy across both known (seen during training) and unknown instances (not seen during training) in typical and new poses. As only a portion of instances (less or equal to 50% in experiments) are seen varying, motivated by works showing the benefit of data reweighting for generalization challenges (Idrissi et al., 2022), we sample data at training so that the model sees many images varying as non varying by upsampling the varying instances where needed.
In order perform well on known instances in new poses, models must learn to recognize an object regardless of its orientation and constitutes our weakest generalization test. In order to perform well on unknown instances, however, models must be able generalize both across instances and poses, a substantially more difficult task.
3.1 DATA VARIATION
To conduct our evaluation, we develop a controlled dataset of common objects such as buses, cars, planes, etc. based on realistic object models from Trimble Inc 7, We render nearly 500,000 images of varying 2D and 3D poses uniformly distributed with a step size of 4 degrees. To simulate natural changes in pose we organize pose changes temporally into frames (i.e. into images). Each frame captures a 4 degree change in pose either in-plane rotation or across all three axes. In addition, we conduct experiments evaluating the Lie operator on the iLab 2M dataset following the preprocessing steps from Madan et al. (2021).
3.2 GENERALIZING POSE CHANGES IS CHALLENGING FOR SOTA VISION MODELS
We first assess state-of-the-an vision models' robustness to pose using a collection of supervised and self-supervised models (CLIP, ResNet-50, ViT-B/16, MLP Mixer, SimCLR, VICReg and MAE). All of these models rely on some form of data augmentation to provide invariance to nuisance features such as pose. We use ImageNet (Deng et al., 2009) pretrained embeddings for our models (with the exception of CLIP, which is pretrained on a large corpus of image-text pairs) (Radford et al.). We only show supervised linear evaluation for VICReg as it is mainly explored in the linear evaluation setting for the fully supervised task in Bardes et al. (2021).
While all models perform well on classification for known instances in typical poses, the same models have a considerable generalization gap to varying poses. In Figure 2, we show results for both linear evaluation and supervised finetuning classification gaps relative to typical poses of known instances. Strikingly, all models we evaluated exhibited large generalization gaps of 30-70% suggesting that even modern self-supervised models struggle to generalize both to new poses and to new instances. As such poor generalization can be problematic at best and dangerous at worst for deployed models, developing ways to improve generalization performance beyond simply augmenting the data is critical. In the next section, introduce new approach in which we encourage models not to be invariant to transformation, but rather to explicitly model and account for transformations.
freely available under General Model license

---

Under review as a conference paper at ICLR 2023
MAE Lie MAE
VICReg Lie
B
VIGReg
Unknown instance in typical pose
+12.4%
+0.2%
Unknown instaance
+10.3%
+1.2%
Known instance in new posa
+13.7%
+7.7%
tap-1 Accurary 18)
10p1 Acouracy (16)
Figure 3: Incorporating our Lie operator to MAE and VICReg improves generalization across poses and instances. Bars indicate the mean top-1 accuracy and brackets indicating the standard error across seeds. In A., we show supervised finetuning for MAE Lie versus a standard MAE. In B., we show supervised linear evaluation for VICReg Lie versus VICReg as the original VICReg model is mainly explored in the linear evaluation setting for the fully supervised task (Bardes et al., 2021).
INCORPORATING A LEARNED LIE OPERATOR IN SELF-SUPERVISED LEARNING MODELS 3.3
To improve robustness, we incorporate a self-supervised learning phase where we structure repre- sentations using our Lie operator. During this phase, half of the instances are seen undergoing pose changes in 2D and 3D. We use the angle difference between frames as an proxy for frame distance (denoted in Section 2). 1) In the case of rotation around all axes, use the mean of three angles. We then evaluate the representations in the same manner as described above.
We implement our Lie operator in two SSL models: Masked Autoencoder (MAE) and Variance- Invariance-Covare Covariance Regularization (VICReg). VICReg is a SSL method, which encourages similar representations among views (generated via data augmentation) of image. MAE is SSL model whose learning objective is based on the reconstruction randomly masked input.
4 RESULTS
4.1 SSL MODELS WITH LIE OPERATORS ARE MORE ROBUST ACROSS INSTANCES AND POSES
We evaluate the robustness benefits of our Lie operator across several generalization settings. In Figure 3, we plot the performance of MAE (A) and VICReg (B) models with and without our Lie operator. Encouragingly, we find explicitly modeling transformations with Lie groups (referred as MAE Lie and VICReg Lie) improves robustness for all generalization settings. For known instances (those seen during training), incorporating our Lie operator improves Top- classification accuracy for poses by 13.7% for MAE Lie and 7.7% for VICReg Lie. In the more challenging settings of unknown instances which are never seen during training, we still observe performance gains for VICReg Lie of 0.2% for instances in typical and 1.2% for those in new poses. For MAE Lie, observe substantial performance gains of 12.4% for unknown instances in typical poses and 10.3% for those diverse poses. We note both MAE and VICReg Lie bring even more substantial gains for settings when data diversity is limited (see Sec. 4.2 Tables and 2).
4.2
LIE OPERATOR ROBUSTNESS GAINS ARE SUSTAINED EVEN WITH LIMITED DIVERSE DATA
For our experiments so far, models have been exposed to a 50/50 mix of typical and atypical poses during supervised linear classification and finetuning. However, we want models to generalize across instances and to atypical poses even when presented with comparatively small amounts of diverse data. We therefore evaluated our models performing supervised linear classification and finetuning with either 50%, 25%, or 5% data featuring atypical poses (Tables and 2). Notably, we observed that while overall performance decreased with fewer diverse instances, might expect, the
7

---

Under review as a conference paper at ICLR 2023
relative gains of the Lie models were largely consistent across settings, and if anything, were larger for settings with less diverse data. These results demonstrate the robustness of our approach to differing quantities of diverse data.
Table 1: Assessing linear evaluation robustness to pose changes as the proportion of diverse instances seen during training varies. Table reports linear evaluation top-1 I accuracy as mean + standard erorr (+ absolute difference, relative multiple) to the baseline model.
instanee new poue
insuace
diverse proporsion
tp sccuracy
1.24K)
05.0
59.7
(2.0- 1.02s)
VICRRa
56.7
608
25.0
13.3 (42.3
1336)
11.5
12.6222
11.0264
12
20.6
11.6225
Table Assessing finetuning robustness to pose changes as the proportion of diverse instances seen during training varies. Table reports finetuning Top-1 Accuracy as mean + standard emorr (+
absolute difference, relative multiple) the baseline model. We do not experiment with VICReg
finetuning as the original paper (Bardes et al., 2021) focuses on linear evaluation.
known instance in new poss
mnknowe instance
diverso propottion
5%
up- accunccy
1.26x)
1.MK)
4.3 ABLATIONS
Table 2. compared to our MAE a model with only the SSL term (and no Euclidean or Lie loss), that is, the baseline MAE, suffers drops of 10.3%, 9.4%, and 8.2% in top-1 accuracy unknown instances, respectively for the 5%, 25% and 50% diversity settings.
In addition, we ran systematic ablations for each loss component in our MAE Lie model. We find consistent decrease in performance across all three diversity settings. Compared to our MAE Lie model, model without the Euclidean loss term (Aeue 0) suffers drops of 10.5%, 9.08%, and 3.9% in top-1 accuracy for unknown instances (when 50%, 25% 5% of instances vary in pose during training, respectively). Similarly, compared to our MAE Lie, a model with only the Lie loss (i.e. Aeue = Agal
suffers a drop of 5.9%, 5.01%, and 1.99% in top-1 accuracy for unknown instances across the three diversity settings. This result illustrates how each loss term is critical to the overall performance of our proposed MAE Lie model.
MAE LIE TRANSFER ON REAL OBJECTS DATASETS
5.1 IMAGENET, ROT IMAGENET
Table 3: Results comparing MAE Lie versus MAE Top-1 Accuracy for ImageNet, Rot ImageNet (random rotations) after finetuning of 20k steps on the ImageNet training set.
Top-1 Accuracy (%)
ImageNet Validation
Rot ImageNet Validation
MAE MAE Lie
63.61
34.50
67.55 (+3.94%)
37.12 (+2.62%)
In order to further demonstrate the benefit of our proposed Lie operator to standard benchmarks, we perform evaluation on ImageNet (Deng et al., 2009) and some of its variants. Rot ImageNet consists of the same validation images which we apply random rotations to assess how well models handle pose changes. To do so, we finetune both the baseline MAE model and MAE Lie for 20k steps on ImageNet with linear classification head. Table 3 reports the performance the ImageNet and Rot ImageNet validation sets. We find MAE Lie outperforms the baseline for rotated ImageNet validation images by considerable +2.62% in top-1 accuracy and the unaltered ImageNet validation set (by
8

---

Under review as a conference paper at ICLR 2023
+3.94% in top-1 accuracy) which contain pose along with other changing factors. Details on the training can be found in Appendix B.1.3.
5.2 ILAB DATASET
To demonstrate the applicability of our approach other datasets, we also evaluate our MAE Lie model on the iLab 2M dataset. The iLab 2M dataset (Borji et al., 2016) is a natural image dataset consisting of 2M images of toy objects, from multiple categories, undergoing variations in object instances, viewpoints, lighting conditions, and backgrounds. We follow the preprocessing steps in Madan al. (2021). We partition the training samples into a validation set (10%) and show results on the iLab test set. We conduct supervised finetuning, the standard evaluation protocol for MAE, We observe MAE Lie yields nearly a 3% improvement on the test set (93.2% + 0.45% MAE Lie vs 90.6% + 0.11% MAE) without re-training our Lie operator the iLab dataset.
6 RELATED WORK
Modeling variation Existing domain adaptation works (Zhou et al., 2020; Robey et al., 2021; Nguyen et al., 202 1) learn transformations that map one domain to the other. In contrast to Ick Lostanlen (2020); Hashimoto et al. (2017), we learn transformations in the latent space rather than image-space. Connor & Rozell (2020); Connor et al. (2021a) learn transformations in the latent space but uses a decoder that performs image reconstruction. which is known to be hard apply to complex, large scale image data. In the context of classification, Benton et al. (2020) use the Lie algebra formalism to learn augmentations, but they require a priori knowledge of the augmentations. Tai et al. (2019) provides invariance to continuous transformations, but they only experiment on small scale data (MNIST (LeCun al., 2010) and SVHN (Netzer al., 20)1)). We also depart from their work we explicitly focus on the challenge of robustness to distributional shifts. Equivariant models (see Cohen & Welling (): Cohen al. (2020) among others) and disentanglement (Higgins et al., 2018; Giannone et al., 2019) also aim to structure latent representations, but are mainly applied to synthetic, small scale data. Disentanglement also often requires costly pixel reconstruction. Spatial Transformer affine transform for classification, resulting
Networks (Jaderberg et al.) learn an instance-depende in behavior such as zooming in on the foreground object. Our goal differs as we focus on the generalization capabilities of various models to pose changes. Capsule networks (Sabour et al., 2017; Mazzia et al., 2021) were designed to better generalize to novel views, however, it is difficult for us to compare approach to these we require strong pre- trained image model. To our knowledge, no such model using capsule networks exists, making a fair comparison very difficult.
OOD Generalization failures Alcom et al. (2019); Engstrom et al. (2019) shed light on the failures of SoTA models on "unusual" examples that slightly vary from the training data, and Engstrom et al. (2019); Azulay & Weiss (2019) show that explicitly augmenting the data with such variations does not fix the problem. Indeed, Bouchacourt et al. (b021b) found that data augmentation does not bring the expected invariance to transformations, and that invariance rather comes from the data itself. Self-supervised models trained with InfoNCE theoretically recover the true data factors of variations (Zimmermann et al., 2021; von Kügelgen et al., 2021), but our experiments show that they do not generalize to distributional shifts.
7 DISCUSSION
We aim to improve the OOD generalization of SSL models by encouraging them to explicitly model transformations using Lie groups. Using novel dataset to evaluate OOD generalization, we found that applying our Lic operator to MAE and VICReg leads to markedly improved performance for both models, enabling better generalization both across poses and instances, both on our synthetic data and on ImageNet and its variations for MAE. Our results highlight that data augmentation alone is not enough to enable effective generalization and that structuring operators to explicitly account for transformations is a promising approach to improving robustness. We focus on modeling pose changes, a common natural factor of variation in objects. However, Lie groups are capable of describing a variety of naturally occurring continuous transformations. Future work could extend the application of Lie operators to these settings, possibly even to model multiple factors of variation at once.

---

Under review as a conference paper at ICLR 2023
8 REPRODUCIBILITY STATEMENT
We made efforts to make all our work reproducible. In the main text, we describe every aspect of our learning objective and all the assumptions taken, with some background Appendix Regarding our experiments, all training details are in Appendix B or in main text, and datasets descriptions are in main text or Appendix C. We will release the code to reproduce all experiments and generate the dataset based on realistic object models from Trimble Inc (freely available under a General Model license) upon acceptance.
REFERENCES
Michael A. Alcom, Qi Li, Zhitao Gong, Chengfei Wang. Long Mai, Wei-Shinn Ku, and Anh Nguyen. Strike (with) pose: Neural networks are easily fooled by strange poses of familiar objects. 2019. URL http: //aaxiv.orgg///aa/...../.a abs/ 1811, 11553.
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small image transformations? Journal of Machine Learning Research, 20( 20(184):1-25, 2019. http: //jml /tt.:////rrrrrrro....... rrrgg/ papers /t///////11//2/// 9. html.
Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Varianee-iivatinnc--ovvaaaa regularization for self-supervised learning. 2021. URL http::/atxii.....///////t//
04906.
Gregory Benton, Mare Finzi Pavel Izmailov, and Andrew Wilson. Learning invariances in neu- ral networks from training data,
H. Larochelle, M. Ranzato, R. Hadsell, F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 17605-17616. Cur- ran Associates, Inc., 2020. URL https: pwwppoeioinnngee neurips..... cc/ wt/paperr/222/f111 00000004d27991cdaee2eeeeeee
Ali Borji, Saced Izadi, and Laurent Itti. ilab-20m: large-scale controlled object dataset investigate deep learning. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2221-2230, 2016. doi: 10. 10.1009/CVP..20162222
Diane Bouchacourt, Mark Ibrahim, and Stéphane Deny. Addressing the topological defects of disentanglement via distributed operators. CoRR, abs/2102.05623, 2021a. URL https: /ttp::::///xxxx.x.//.. 05623.
/tt.httpp////rxxi wwt.:/s.//.//o.o..o...o...s.ss 2202
Diane Bouchacourt, Mark Ibrahim, and Ari Morcos. Grounding inductive biases natural images: invariance stems from variations data. In Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Ad- vances Neural Information Processing Systems, 2021b. https:/ /ttpa//////eeeeeeeeeeeeeeeeeeeee net forum? id-p7Gu jbewnRY.
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. 2020. URL http: /ttp:::///avvv.///aaeeeeppp wwrg/aiwiii...a..aaa...oo..oo.or aba/ 220022 05709.
Taco Cohen and Max Welling. Group equivariant convolutional networks. Maria Florina Balcan Kilian Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 2990-2999, New York, New York, USA, 20-22 Jun 2016.
PMI.R URL http:/ /ptpc//pr/cceieceo ngs. ml prwss//ss//ee. v / wowenccellcccoooooooooooooooo 6. html.
Taco Cohen, Mario Geiger, and Maurice Weiler. A General Theory of Equivariant CNNs on Homogencous Spaces. arXiv: 1800001 Jcs, stat), January 2020. URL http: /tctp://axx.v..o/g///aa abs////////bb/aaa 02017.
arXiv: 1811.02017.
Marissa Connor and Christopher Rozell. Representing closed transformation paths in encoded network latent space. In The Thurty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, FAAI 2020, New York, NY, USA, February 7-12, 2020, pp- 3666-3675. AAAI Press, 2020. ISBN 978-1-57735-8233 URL https: aaai. orggojss......o.ooooo.o. htsp//iiss/i.i////ii///i index. php/ /tt////AAAaaattteeeeaeaaeaeeeeet
Marissa Connor, Kion Fallah, and Christopher Rozell, Learning identity- transformations on data manifolds. CoRR, abs/2106. 12096, 2021a. URL pttttpp:///xiiii org/ /bb./106............bbbbbbb Tttp:/221....2200..... 12096.
Marissa C. Connor, Gregory H. Canal, and Christopher J. Rozell. Variational autoencoder with learned latent structure. 2021b. URL. http:httt///....oggg.goooo/
10597.
10

---

Under review as a conference paper at ICLR 2023
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. Icce, 2009.
Carl Doersch, Ankush Gupta, and Andrew Zisserman. Crosstransformees:: spatially-aware few- shot transfer. Н. Larochelle, M. Ranzato, R. Hadsell. M.F. Balean. and H. Lin (eds.). Advances in Neural Information Processing Systems, volume 33, pp. 21981-21993. Curran Associates, Inc., 2020. URL hetps://proceeddngg weuggis...sss..... wtw.ss...pppap.........e.. wttpp/pppe///////1/..... 4428c6cdfoddbE41a655555 (allaa66c33cca5579--9a55p0 -aw.eaa0----aape..ac
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Exploring the landscape of spatial robustness. In Kamalika Chaudhuri and Ruslan Salakhutdinov (cds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 1802-1811. PMLR, 09-15 Jun 2019. URL https: https:://ptcceeeeeeeeee http///orceeed...cp/o//o wttpinns.........ss..ee www.prreere//... wtt.spres//99 engst roml 9a. html.
R. Geirhos, J. Jacobsen, C. Michaclis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shorteut learning deep neural networks. Nature Machine Intelligence, 2(11):665-673, 2020a. doi: 10.1038/ s42256-020- 012256-020-00257--
Robert Geirhos, Carlos R. M. Temme, Jonas Rauber, Heiko H. Schütt, Matthias Bethge, and Felix A. Wichmann. Generalisation humans and deep neural networks. In S. Bengio, H. Wallach, Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gareett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https: : /poceedd ngs neur ips .p..p. co/ pappr/pprrpp// /tt1:r///01////// 0937fb588 7755844e0065 fb59ae5 (9b5555e559956666 9b5dd6 7a 9- Paper.
Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. On the surprising similarities between supervised and self-super models. In NewrlPS 2020 Workshop SVRHM, 2020b. URL https:://////.... ropenrevii ew net bet//forie forum?i d=q2m1 4CJMHAX.
Giorgio Giannone, Jonathan Masci, and Christian Osendorfer. No representation without transformation. CoRR, abs/1912.03845, 2019. URL http: http://axvvv....... 1912. 00..1.14......
B.C. Hall. Lie Groups. Lie Algebrus, and Representations: Elementary Introductionn Graduate Texts in Mathematics, Springer, 2003. ISBN 9780387401225. URL https: /bwoks.ooo////o google. cww.coo//oooooooooo com/books? 1d=m1 waalivv1@@mmmmmm 8HmEweC.
Tatsunori B Hashimoto, Percy Liang, John Duchi. Unsupervised transformation learning via convex relaxations. In I. Guyon, U. V. Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (cds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Kaiming He, Xinlei Chen, Saining Xie. Yanghao Li, Piotr Dollár, Ross Girshick. Masked autoencoders are scalable vision learners. 2021. URL http: / /wwxivv.vvvvv./////a//aa//aaaaia org/ abs 2111. 06377.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness uncertainty. In Wallach, H. Larochelle, A. Beygelzimer, d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL ht tttttps://ee http://ppoooeeieeeeeeeooe nww.ipuuu....... pttps/ppppppp/////s/LL. (2lo1533/edac1ddee (Tt.///222000211196666228232
pdi..
Irina Higgins, David Amos, David Pfau, Sehastien Racaniere, Loic Matthey Danilo Rezende, and Alexander Lerchner. Towards Definition of Disentangled Representations. 1/X10:1812.022300 [cs, stat], December 2018. URL http: wawpivxiivv..///ii//.aaaaaaaa wwtp://.rrga/b/oboeoo hat///bs//lee /1cb 1812. 02230. arXiv: 1812.02230.
Chris Ick and Vincent Lostanlen. Learning a lie algebra from unlabeled data pairs. volume abs/2009.09321. 2020. URL htttps://arxiv....t
2009 /tt.2.0.9......... 09321.
Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive worst-group-accuraey In First Conference on Causal Learning and Reasoning, 2022.
URL httpt:/t////// /ttl:::///peeeeeee wttp///wppeeeeeeeeeeeeeee net/ forum? wwerruiiddcDDss
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bue, F. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL ht http///poceeennn.cceee httppoeddddg...drr..ppp... wew.eiiinnnnnnininii.ieeee #2c42089244558
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks.
URL htt.ttt::////iiieee abs 1506. 02025.
11

---

Under review as a conference paper at ICLR 2023
Jason Jo and Yoshua Bengio. Measuring the lendency of cans to learn surface statistical regularities. CoRR, abs/1711.1 2017. URL http: wrt.//s/r//////oo/ wttp://////....///..e/... 11711.11661.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scalc. arXiv preprint arXiv: 1.01236, 2016.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: 2, 2010.
http://yann.lee www.ll/www..occd//nnee..
F. Locatello, B. Poole, G. Rätsch. Schölkopf. O. Bachem, and M. Tschannen. -uparis- disentan- glement without compromises. In Proceedings of the 37th International Conference on Machine Learning (ICML), volume 119 of Proceedings of Machine Learning Research, pp. 6348 6359, PMIR, July 2020. URL
/ttp:e//pccceeiiio ings mlr. press v11 199 locatello220 wttp:/0aaooaatelle222 pdf.
Spandan Madan, Timothy Henry, Jamell Dozier, Helen Ho, Nishchal Bhandari, Tomotake Sasaki, Frédo Durand, Hanspeter Pfister, and Xavier Boix. When and how CNNs generalize to -ii-tt--ittiitutionn category-viewpoint combinations. 2021. URL http::// / ataaaaaiivvvo.... abs 080022.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks, arXiv preprint wxXiv:176600600 2017.
Vittorio Mazzia, Francesco Salvetti, and Marcello Chiaberge. Efficient--cassnet: capsule network with self- attention routing. Scientific reports, 11, 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 1. URL http::ttt// /ufidl. stanford. edu http///uuunnnbbrbeeee saipersrrrri.s/r1e11 housenumbers. pdf.
A. Tuan Nguyen, Toan Tran, Yarin Gal, and Atilim Gunes Baydin. Domain invariant representation learning with domain density transformations. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. URL /ttp::///ppeeeeeeeeeeeee net
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. URL https::/t/// /tappii/////vv../////p//apsssm// awwiv.iivvv..ooo//aaaaaaaaaa oogg///g//aa...a...... abaaaaaalll0 22103.103.000 00020v1.
Alexander Robey, George Pappas, and Hamed Hassani. Model-based domain generalization. Thirty- Fifth Conference on Neural Information Processing Systems, 2021. URL /ttttps://penreeee
net//
forum? J0J0 1
Sara Sabour, Nicholas Frosst, and Geoffrey E. Hinton. Dynamic routing between capsules. In Proceedings of the 31st International Conference on Neural Information Prucessing Systems, NIPS' 17, pp. 3859-3869, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964,
J. Stillwell. Naive Lie Theory. Undergraduate Texts in Mathematics. Springer New York, 2008 ISBN 97803877821577. URL httts:://booke http:/oooooo..llllll.oo//// google..oe.ooooooooooooooooo...o bboks iddSuR50AgsyDII..
Kai Sheng Tai, Peter Bailis, and Gregory Valiant. Equivariant Transformer Networks. In International Conference on Machine Learning, 2019.
Trimble Inc. warehouse. nttps: httpps:/3ddwreehoussee wwwpaaseee.keeeeee.. www.skeet.u...... con/. Accessed: 2022-03-07.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume of Proceedings of Machine Learning Research, pp. 6438-6447. PMLR, 09-15 Jun 2019. URL https: httpss//pcccceeddeneeee mlr. press / v77 vermal 9a htmi
Julius von Küggelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. supervid leaming with data augmentations provably isolates content from style. 2021. URL http:// htattp:p//axxivve/// aaxiv. wwwg//oog/ooo..o abs/abbsl0000 2200.. 210.60066.......111....011111116
Ross Wightman.
Pytorch image
models.
https://githin comb rwi ghtman/
wytorch-image-mer 2019.
12

---

Under review as a conference paper at ICLR 2023
Kaiyang Zhou, Yongxin Yang. Timothy M. Hospedales, and Tao Xiang. Deep domain-adversarria image generation for domain generalisation. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposiun on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 13025-13032. AAAI Press, 2020. URL https: / htt.aaaaaaaaaaoaaaaaaaaaa /indxx.. php/AAAI/ wtw.A///aaateeeeeieeeeaeaaaaae
Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. 139:12979-12990, 2021. URL http::/prooceeddngs press/ vv139/1////////// zimne rmann211 la. html.
13


Reviews:
[{"review_id": "lrDGxDjSCy7", "reviewer": "Reviewer_1hzj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to apply the formalism of Lie groups to capture continuous transformationsto improve models robustness to distributional shifts. Specifically, it structures the representation of corresponding vector space of the assumed Lie group by learning some basis metrics and then constructs their Lie operator by using this vector space structure. The general idea is impressive, and the experimental results are promising.", "summary_of_the_review": "In general, the presented method is novel and effective, which is also well-supported by the experimental results.", "strength_and_weaknesses": "Strength: This paper shows great potantial of learning the transformation in laten space induced by changes in pose to enable self-supervised models to generalize variation across objects. The perspective is novel and ablation study shows promising results.\n\nWeaknesses: In Section 2.1, there is no description about how to decide the dimention d of the Lie algebra. In my understanding, there should be a way to specify the dimention, or it is the degree of freedom of the transformation matrix in Li group. Is my understanding correct?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Pg4T1X1bKr", "reviewer": "Reviewer_wpyw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Though deep learning has achieved many advances, many existing methods suffer from poor generalization performance due to the domain-shift impact. To this end, this paper focuses on improving the robustness of self-supervised methods. Specifically, the authors choose matrix Lie groups to model continuous transformations and employ contrastive learning to enhance the generalization. In the experiments, the proposed method is evaluated on multiple datasets.\n\n\n", "summary_of_the_review": "The writing of this paper is not clear. The proposed method only uses the idea of Lie group and contrastive learning. The novelty of this paper is limited. Besides, the proposed method contains multiple operations and is complex.", "strength_and_weaknesses": "\nUsing group theories to improve OOD robustness is an interesting idea.\n\n\n1. The motivation of this paper is not clear. In Introduction Section, the authors directly indicate that self-supervised models own limited OOD robustness. However, I am not clear why self-supervised models are with limited OOD robustness. The authors should give more interpretations.\n\n2. Some indications in this paper are not accurate. For example, the authors indicate that for image data, augmentations can only be applied to the pixels themselves. However, to the best of my knowledge, there exist some feature-level augmentation methods. Besides, the authors should further interpret why augmentation fails to generalize to novel objects.\n\n3. The writing of the method section is very chaotic. The proposed method contains multiple operations. However, the authors do not introduce the motivation of these operations clearly. Besides, it is better to give some theory analyses for the proposed method.\n\n4. In Table 1 and 2, this paper only compares MAE and VICReg. The authors should evaluate the effectiveness on more methods and tasks. The proposed method involves multiple operations. The authors should make sufficient ablation experiments. Finally, the authors should give more training details, e.g., the training curves, and feature-level visualization analyses, which is beneficial for further understanding the proposed method.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IJrYbWMxKc", "reviewer": "Reviewer_iZ5V", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the problem of OOD robustness with respect to pose changes. To this end, a Lie group transformation of latent representations is learned. This groups is parametrized via its Lie algebra in order to properly integrate it into a (vector space-driven) deep learning framework.\n\nTo train the network, pairs of images are used that show the same object at different poses.\nUsing these pairs, an SSL is performed by using a combination of four loss functions:\n1. Standard SSL training loss\n2. InfoNCE to measure the similarity of the latent representation (w.r.t. the algebra estimation)\n3. Euclidean similarity between the latent representation\n4. Task-dependent algebra-regularization\n\nThe proposed method is trained on a dataset of about 500K generated images based on models owned by Trimble Inc.", "summary_of_the_review": "Overall, I like the idea presented in this paper. There are minor concerns like the Lie group's dimension.\n\nMy main concern is with respect to the reproducibility with stems from the license of the used data.\nTherefore, I see the paper marginally below the acceptance threshold.\n\nIf the reproducibility issue can be resolved, I am happy to raise my vote. ", "strength_and_weaknesses": "Strengths:\n+ The used mathematical concept of Lie groups/algebras is nicely explained\n+ The four different loss components are explained in detail and ablation studies are performed to emphasize their usefulness\n+ Newest models (like CLIP) are used to demonstrate the usefulness of this new approach\n\nWeakness:\n- It is not clear why 3D rotations should translate to group operations in the latent space. This two operations seem to be logically disconnected\n- The paper is not clear about the (manifold) dimension of the Lie group. Depending on the latent space, this dimension might be quite high\n- The proposed method uses different hyper-parameters for the four different loss components. Since these are not disclosed, it is difficult to reproduce the results.\n- In Section 2, it is not clear why the scalar `t` is necessary. After all, the Lie algebra is a vector space and is closed under scalar multiplication", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject