Homomorphic Self-Supervised Learning
Authors: T. Anderson Keller
Xavier Suau
Luca Zappella
Venue: Accepted by TMLR ()

Abstract: Many state of the art self-supervised learning approaches fundamentally rely on transformations applied to the input in order to selectively extract task-relevant information. Recently, the field of equivariant deep learning has developed to introduce structure into the feature space of deep neural networks by designing them as homomorphisms with respect to input transformations. In this work, we observe that many existing self-supervised learning algorithms can be both unified and generalized when seen through the lens of equivariant representations. Specifically, we introduce a general framework we call Homomorphic Self-Supervised Learning, and theoretically show how it may subsume the use of input-augmentations provided an augmentation-homomorphic feature extractor. We validate this theory experimentally for simple augmentations, demonstrate the necessity of representational structure for feature-space SSL, and further empirically explore how the parameters of this framework relate to those of traditional augmentation-based self-supervised learning. We conclude with a discussion of the potential benefits afforded by this new perspective on self-supervised learning.

Published in Transactions on Machine Learning Research (11/2023)
Homomorphic Self-Supervised Learning
T. Anderson Keller Apple
Londersso.kkllerr@mmaai..
Xavier Suau Apple
tswaucwadroos@upple..oo
Luca Zappella Apple
taapella@apple.com
Reviewed on OpenReview:
https: openreview. net/ forum? id= tEXqQgbub
Abstract
Many state of the art self-supervised learning approaches fundamentally rely trans- formations applied to the imput in order to selectively extract task-relevant information. Recently, the field of equivariant deep learning has developed to introduce structure into the feature space deep neural networks by designing them as homomorphisms with re- spect to imput. transformations. In this work, we observe that many existing self-supervieer learning algorithms can be both unified and generalized when seen through the lens of equivariant representations. Specifically,
introduce general framework call Homo- morphic Self-Supervised Learning, and theoretically show how it may subsune the use of input- aupuutaugmeetttions provided an augmentation homomorphic feature extractor. We validate this theory experimentally for simple augmentations, demonstrate the necessity of represen- tational structure for feature-space SSL, and further empirically explore how the parameters of this framework relate those traditional ongmentation-basee self-supervised learning. We conclude with a discussion of the potential benefits afforded by this new perspective on self-supervised learning.
sim 1 +
sim
0
|
G-conv
conv
conv
7
W
H-SSL
A-SSL
Figure 1: Overview of Homomorphie-SST (left) and its relation to traditional Angmentatton--asse SSL (right). Positive pairs extracted from the lifted dimension (0) of a rotation equivariant network (G-conv) are equivalent to pairs extracted from the separate representations of two rotated images.

---

Published in Transactions on Machine Learning Research (11/2023)
1 Introduction
Many self-supervised learning (SSL) techniques can be colloquially defined representation learning algo- rithms which extract approximale supervision signals directly from the inpul data itself (LeCun & Misra 2021 b. In practice, this supervision signal is often obtained by performing symmetry transformations of the imput with respect to task-relevant information, meaning the transformations leave task-relovant information unchanged, while altering ieeelirrelevnt information. Numerous theoretical and empirical works have shown that by com- bining such symmetry transformations with contrastive objectives, powerful lower dimensional representations can be learned which support linear-sparrabiityy (Wang et al. 2022 Lee et al. 2021 Tian al. 2020b; Arora
ct al. Federici et al. 2020:
2019 Tosh et al. 2020 identifiability of generative factors (von Kügelgen et al. 2021 Tsai et al.
2020
Ji el. al. 2021). and reduced sample complexity (Grill et al. 2020 Chen et al.
2020
One rapidly developing domain of deep learning research which is specifically focused on learning structured representations of the input with respect to symmetry transformations is that of equivariant neural networks
(Cohen & Welling 2016 2017 Weiler et al.) 2018: Worrall Welling 2019 Finzi et al. 2020: 2021 van der Pol et al. 22000). Formally, equivariant neural networks are designed to be group homomorphisms for transformation groups which act on the input space meaning that their output explicitly preserves the structure of the imput with respect to these transformations. Traditionally, equivariant neural networks have been contrasted with data augmentation in the supervised setting, and proposed as more robust and data efficient method for incorporating prior symmetry information into deep neural networks (Worrall et al.) 2017]. In the self=supervised setting, however, where data augmentation is itself implicitly responsible for extracting the supervision signal from the data, the relationship between data augmentation and equivariance is much more nuanced.
In this work, we study self-supervised learning algorithms when equivariant neural networks are used backbone feature extractors. Interestingly, we find a convergence of existing loss functions from the literature, and ultimately generalize these with the framework of Homomorphic Self-Superuised Learning. Experimentally, we show that, when the assumplion of an augmentattion-ooooooryy subsumes input augmentation, as evidenced by identical performance over a range of settings. We further validate this theory by showing that when our assumption is not satisfied, the framework fails to learn useful representations. Finally, we explore the new generalized parameters introduced by this framework, demonstrating immediate path forward for improvements to existing SSL methods which operate without. input augmentations.
backbone is satisfied, this framework
2 Background
In this section, we introduce the concept of equivariance and show how structured representations can be obtained via G-convolutions (Cohen & Welling 2016). We then review general self-supervised frameworks and how prior literature differs with respect to its use of input augmentations.
2.1 Equivariance
Formally, a map which preserves the st sucture of the input space in the output space is termed a homomorphism. The most prominent example of a homomorphism in modern deep learning is the class of group equivariant neural networks, which are analytically constrained be group homomorphisms for specified transformation groups (such as translation, rotation, mirroring, and scaling). The map f X + Z is said to be equivariant with respect to the group 9 = (G, if
hr, such that f(T,(=)) [x(( G
(1)
where G is the set of all group elements, is the group operation, T₁ is the representation of the transformation g E G in input space X, and I3 is the representation of the same transformation in output space Z. If T, and T are formal group representations (Serre 1977) such maps f are termed group-homomophhisns since they preserve the structure of the group representation Tg in input space with the output representation lg.. There are many different methods for constructing group equivariant. neural networks, resulting in different.
2

---

Published in Transactions on Machine Learning Research (11/2023)
[(11))) (x1,2), z(g (x22y1), (x2y2)
z(g.9o 9o (x3,y3), (x3,44),
z(g {(x))}
(x4,33), (x4,4)
r 9
Channels
conv
conv
conv
F
F
F
Figure 2: Visualization of 'fiber' (left), 'fiber bundle' (center) and a group representation r acting fiber bundle (right). We see that a fiber is defined as all features at an individual group element (in this case all feature channels at an individual spatial dimension), while a fiber bundle is all features at a set of ordered group elements. In this figure, we depict fcature channels stacked along the z-dimonsion, different from the 'lifted' dimension in Figure [ (left).
reprosentations of the transformation in feature space g- In this work, we consider only discrete groups G and networks which admit regular representations for r. In the following paragraph we outline one met hod by which such networks can be built, and thereby demonstrate how the regular representation behaves.
Neural Networks One common way in which group-equivariant networks are constructed is via the group-convolution (G-conv) (Cohen & Welling 2016). For a discrete group G, we denote the pre-activation output of a G-equivariant convolutional layer I as x', with a corresponding input y¹. In practice these values are stored finite arrays with feature multiplicity equal to the order of the group cach space. Explicitly, z' R IRCos x|Guel, and y E RCCx where Gout and Gin are the set of group elements in the output and input spaces respectively. We the following shorthand for indexing z 29(9) = zlng RCoue and y'(g) =yli E RCC. denoting the vector of feature channels at specific group element (sometimes called a 'fiber' (Cohen & Welling 2017)). Then, the value 21((0)) E R of a single output at layer 1, channel e and element is
Cin Σ
(1) =(y)' *()
E BEG
y (g
(2)
where 6/ is the filter between the ith input channel (subscript) and the t output channel (superscript), and is similarly defined (and indexed) over the set of input group elements G.n- We highlight that the composition g-1 h = k E Gin is defined by the action of the group and yields another group element by closure of the group. The representation Hg and can then be defined as [g]x((h)] = x'(g>)) h) for all I >> when 9 = glut = Go From this definition is straightforward to prove equivariance from: T Tolr") (h) = I', [y' * (h)) group, meaning that it acts by permuting features along the group dimension while leaving feature channels intact. Group equivariant layers can then be composed with pointwise non-lineasss and biases to yield fully equivariant deep neural network (c.g. y I+1 = ReLU(z + b) where b E R is learned bias shared over the output group dimensions). For 1 = 0, yd is set to the raw input æ, and typically the input group is set to the group of all 2D integer translations up to height H and width W: gh = (2 +). The output group Pome is then chosen by the practitioner and is typically a larger group which includes translation as a subgroup, e.g. the roto-translation group, or the group of scaling & translations. In this way, the first layer of a group-equivariant neural network is frequently called the 'lifting layer' since it lifts the input from
[=))) Furthermore, we see that Hg is a 'regular representation' of the
3

---

Published in Transactions on Machine Learning Research (11/2023)
the translation group, containing only spatial dimensions, to larger group by adding additional 'lifted' dimension.
Example As a simple example, a standard convolutional layer would have all height (H) and width (W) spatial coordinates as the set Gout, giving & E RCXHW, A group-equivariate neural network (Cohen & Welling 2016) which is equivariant with respect to the the group of all integer translations and 90-degree rotations (p4) would thus have a feature multiplicity four times larger (z RCx4HW), since each spatial element is associated with the four distinct rotation elements (0", 90", 180", 270"). Such a rotation equivariant network is depicted in Figure 1 with the 'lifted' rotation dimension extended along the vertical axis (0). In both the translation and rotation cases, the regular representation r acts by permuting the represcntation along the group dimension, leaving the feature channels unchanged.
Notation In the remainder of this paper we will see that helpful to have a notation which allows for easy reference to the sets of features corresponding to multiple group elements simultancously. These sets are sometimes called 'fiber bundles' and are visually compared with individual fibers in Figure 2 In words, fiber (Ieft) can be described as all features values at specific group element (such as all channels at a given spatial location), and fiber bundle (center) is then all features at an ordered set of group elements (such as all channels for a given spatial patch). We denote the set of fibers corresponding to an ordered set of group elements 9 as: zgg) = [=(g) g E g] E Blal Using this notation, we can define the action of Tg as: [g]]]]]] = 2gg -1 go). Thus g can be seen to move the fibers from 'base' locations Do to a new ordered set of locations g go. as depicted in on the right side of Figure 2 We highlight that order critical for our definition since transformation such rotation may simply permute go while leaving the unordered set intact.
2.2 Supe-Supe Learning
As mentioned in Section 1 self-supervised learning can be seen extracting supervision signal from the data itself, often by means of transformations applied to the input. Many terms in self-supervised learning objectives can thus often be abstractly written as function I(V(1), v(2) I(V((),,())) of two batches of vectors v(1) = ( Ji=1 and ( {o(2)) N where there some relevant relation between the elements of the two batches. Tn this description, we see that there are two main degrees of freedom which we will explore in the following paragraphs: the choice of function I, and the precisc relationship between (1) and (2))
The most prominent SSL loss terms in the literature are often segregated into contrastive Io (Chen et al. 2020 Oord et al. 2018 and non-contrastive INC (Grill et al.,
SSL Loss Functions: Ie and INC
2020 Chen & He 2020) losses. At a high level, contrastive losses frequently rely on a vector similarity function .) (such as cosine similarity), and 'contrast' its output for 'positive' and 'negative' pairs. A general form of a contrastive loss, inspired by the 'InfoNCE' loss (Oord et al., 2018), can be written as:
exp sim (h(v) (") 1()) ((x))(())))))) ()
(V((), v(2))
1 N
log N Σj# Ex, exp sim (h(p)*) (*)) h(e, (1) //)))
(3)
where h is a non-linear 'projection head' h 2 y' and T is the 'temperatrre' of the softmax. We see that such losses can intuitively be thought of as trying to classify the correct 'positive' pair (given by of & x(!²) out of a set of negative pairs (given by all other pairs in the batch). Comparatively, non- non-controstive losses are often applied to the same sets of representations (1) and V but crucially forego the need for 'negative pairs' through other mcans of regularization (such as stop-gradient on one branch (Chen & He 2020 Tian let al.) 2021) observed to regularize the eigenvalues of the representation covariance matrix). Fundamentally this often results in a loss of the form:
IN (v()), v(2) =- (ime(
(4)
where SG denotes the stop-gradient operation. In this work we focus the majority of our experiments on the INCE loss specifically. However, given this general formulation which decouples the specific loss from the
4

---

Published in Transactions on Machine Learning Research (11/2023)
choice of pairs V(O) & V and the fact that our framework only operates on the formulation of the pairs, we will see that our analyses and conclusions extend to all methods which can be written this way. In the following, we will introduce the second degree of freedom which captures many SSL algorithms: the precise relationship between V and V
V & ( Similar to our treatment of SSL loss functions I, in this section we separate the existing literature into two categorics with respect the leveraged relationship between positives pairs. Specifically,
Relationship Between SSL Pairs:
compare methods which rely on input. augmentations, which call Augmentattion-assdd SSL (A-SST.), to methods which operate entirely within the representation of single input, which we call Feature-space SSL (F-SSL). An influential framework which relies augmentation is the SimCLR framework (Chen et al.
2020 Using the above notation, this is given as:
I (11(Calee.))), RM, {1(T 91.92
C CA-SSL (X)
= E
55)
where gg](z] denotes the action of the sampled augmenlation g the inpul, G is the set of all augmentations, and f(x)= = v is the backbone feature extractor to be trained. This loss is then summed over all elements i in the batch before backpropagation. In this work, we consider this SimCLR loss given in Equation 5 as the canonical A-SSL method given its broad adoption and similarity with other augmentation-basse methods. The second class of SSL methods we consider in this work are those which operate without the use of explicit input augmentations, but instead compare subsets of representation for a single image directly. Models such as Deep InfoMax (DIM(L)) (Hjelm et al. Predictive Coding (CPC) (Oord et al.
2019 Greedy InfoMax (GIM) (öwwe et al. 2019), and Contrastive 2018 can all be seen to be instantiations of such Feature-space SSL methods. At a low level, these methods vary in the specific subsets of the representations which are used in the loss (from single spatial clements to larger 'patches'), and vary in the similarity function (with some using a log-bilinean model sim(a, b) = exp (a WWb,, instead of cosine similarity). In this work we define a general Feature-space SST. (F-SST.) loss the spirit of these models which similarly operates in the feature space of a single image, uses an arbitrary spatial 'patch' size [g], and a cosine similarity function. Formally:
C CR-SSI
E I (f {
(z) (zn())))
21.
Ziw refers to sampling a contiguous patch from the spatial coordinates of convolutional feature map, and z,, is the output of our backbone f(x,). In the following section, we show how equivariant backbones unify these two losses into a single loss, helping to explain both their successes and limitations while additionally demonstrating clear directions for their generalization.
where 9
3 Homomorphic Self- Supevvupe Learning
In this section we introduce Homomorphic Self-Supervised Learning (H-SST.) as general framework for SSL with homomorphic encoders, and further show it both generalizes and unifies many existing SSL algorithms.
To begin, consider an A-SSL objective such as Equation when f is equivariant with respect to the input augmentation. By the definition of equivariant maps Equation [ the augmentation commutes with the feature extractor: [x) = [x)).. Thus, replacing f(xx) with its output zn zn (go), and applying the definition of the operator, we get:
SS.. (X)
E Io (
2m (yz" {
go) }m)))
(7)
Ultimately, we see that C--SSL subsumes the use of input augmentations by defining the 'positive pairs' as two fiber bundles from the same represenlation 2n, simply indexed using two differently transformed
In CPC, the authors use an autoregressive encoder eucode element of the positive pairs. In GIM, they find that in the visual domain, this autoregressive encoder is not necessary, and thus the loss reduces simple contrasting the representations from raw patches with one another, as defined here.
5

---

Published in Transactions on Machine Learning Research (11/2023)
base spaces 91 go and 92" go (depicted in Figure 1 and Figure center & right). Interestingly, this loss highlights the base space go parameter choice previously unexplored in the A-SSL frameworks. In Section 4 we empirically explore different choices of go and comment on their consequences.
A second interesting consequence of this derivation is the striking similarity of the CHIsst. objective and other existing SSL, objectives which operate without explicit input augmentations to generate multiple views. This can be seen most simply by comparing CH.SSL from Equation 7 with the CFFSSL objective from Equation 0 Specifically, since 9i & 92 from the F-SSL loss can be decomposed as single base patch go offset by two single translation elements 91 & 92 (e.g. 91 91 go and 92 92 go), we see that Equation 6 can be derived directly from Equation 7 by setting G = frw and the size of the base patch lgol equal to the size of the patches used for each F-SSL case. Consequently, these F-SSL losses are contained in our framework where the set of "augmentations' (g) is the 2D translation group, and the base space (go) is a small subset of the spatial coordinates. Since CH-SSL also derived directly from CA-SSL (when f is oquivariant), we see that provides a mcans to unify these previously distinct sets of SSL objectives. In Section we validate this theoretical equivalence empirically. Furthermore, since CH-SSL is defined for transformation groups beyond translation, it can be seen to generalize F-SSL objectives in way that we have not previously seen exploited in the literature. In Section 4 we include preliminary exploration of this generalization to scale and rotation groups.
4 Experiments
In this section, we empirically validate the derived equivalence of A-SSL and H-SST. in practice, and further reinforce our stated assumptions by demonstrating how H-SSL objectives (and by extension F-SSL objectives) are ineffective when representational structure is removed. We study how the parameters of H-SSL. (topographic distance) relate to those traditionally used in A-SST, (augmentation strength), and finally explore how the new paramcter gencralizations afforded by our framework (such as choices of go and g) impact performance.
4.1 Empirical Validation
For perfectly equivariant networks f, and sets of transformations which exactly satisty the group axioms, the equivalence between Equations 5 and is exact. However, in practice, due to aliasing, boundary effects, and sampling artifacts, even for simple transformations such as translation, equivariance has been shown to not be strictly satisfied (Zhang 2019). In Table ! we empirically validate our proposed theoretical equivalence between CASSSL and CH-SSL showing a tight correspondence between the downstream accuracy of linear classifiers trained on representations Icarned via the two frameworks.
Precisely, for each transformation (Rotation, Translation, Scale), we use a backbone network which is equivariant specifically with respect to that transformation (e.g. rotation equivariant CNNs, regular CNNs, and Scale Equivariant Steerable Networks (SESN) (Sosnovik et al. 2020 ). For A-SSL augment the input at the pixel level by: randomly translating the image by up to + 20% of its height/ /widhh (for translation), randomly rotating the image by one of 90", 180", 270%] (for rotation), or randomly downscaling the image to value between 0.57 & 1.0 of its original scale. These two augmented versions of the image are then fed through the backbone separately, and a single fiber (meaning Igol = 1) is randomly selected. Although the fiber is selected from the same base location for both images, will contain different. features since the underlying images are transformed differently. We investigate the impact of the base space size separately in Section 1.3 For H-SSL we use no imput augmentations and instead rely differently indexed base patches (formed by shifting the randomly selected fiber go by two separate randomly selected group elements 91 & 92). For example, for A-SSL with translation, we compare the feature vectors for two translated images at the same pixel location go- For H-SSL with translation, we compare the feature vectors of a single image at two translated localions 91 "o 8 g go- Importantly, we note that these feature vectors are taken before performing any pooling over the group dimensions in all cases. Ultimately, we see an equivalence between the performance of the A-SSL models and H-SSL models which significantly differs from the frozen and supervised baselines, validating our theoretical conclusions from Section Further details this experimental setup be found in Appendix A
6

---

Published in Transactions on Machine Learning Research (11/2023)
Table 1: MNNST (LeCun & Cortes 2010). CIFAR10 (Krihevsky et al.) and Tiny ImageNet (Le & Yang 2015) top-1 test accuracy (mean = std. over 3 runs) of a detached classifier trained on the representations from SSL methods with different backbones. We compare A-SSL and H-SSL with random frozen and fully supervised backbones. We equivalence between A-SSL and H-SSL from the first two columns, desired, and often see significant improvement in performance for H-SSL methods when moving from Translation to generalized groups such as Scale.
Datasel
Transformation
Backbone
A-SSL
H-SSL
Frozen
Supervised
Rotation
Rot-Eq.
68.2 = 2.5
70.3 + 5.4
87.2 + 0.8
99.4 = 0.1
MNIST
Translation
CNN
95.9 + 0.3
96.0 + 1.3
94.1 + 0.3
99.2 + 0.1
Scale
SESN
98.6 + 0.1
98.3 + 0.2
94.7 + 0.6
99.3 + 0.1
Rotation
Rot-Eq-
46.1 + 0.6
48.3 + 0.5
38.4 + 0.1
73.0 + 1.1
CIFAR10
Translation
CNN
39.2 + 0.5
36.3 + 1.1
40.4 + 0.2
76.2 + 1.4
Scale
SESN
59.4 0.2
56.7 + 0.4
41.1 + 0.6
78.0 + 0.2
Rotation
Rot-Eq. SESN
14.9 + 0.3
13.5 + 0.5
6.1 = 0.2
22.5 = 0.1
Tiny ImageNet
Scale
16.2 + 0.4
14.0 + 1.3
6.4 + 0.2
23. 7 = 0.2
Table 2: An extension of Table | with non-equivariant backbones. We see that the H-SSL methods perform similar to, or worse than, the frozen bascline when equivariance is removed, as expected.
Dataset
Transformation
Backbone
A-SSL
H-SSL
Frozen
Supervised
Translation
MLP
87.6 + 0.2
58.2 + 0.5
83.0 + 0.8
98.6 + 0.1
MNIST
Scale
CNN (6 x CHW)
95.2 + 0.1
87.2 + 2.4
87.2 + 0.6
99.3 + 0.1
CIFAR1O
Scale
CNN (6 CHW)
53.6 + 0.2
37.5 + 0.1
43.6 + 0.3 67.9 + 2.1
4.2 H-SSL Without Structure
To further validate our assertion that CH-SSL requires a homomorphism, in Table we show the same models from Table [ without equivariant backbones. Explicitly, use the same overall model architectures but replace the individual layers with non-equivarianl counterparts. Specifically, for the MLP, replace the convolutional layers with fully connected layers (slightly reducing the total number of activations from 6272 to 2048 to reduce memory consumption), and replace the SESN kernels of the scale-cquivariant models with .ally-parameterizze,, non-equivariant counterparts, otherwise keeping the output dimensionality the same (resulting in the 6 x larger output dimension). Furthermore, for these un-structured representations, in the H-SSL setting, we 'emulate' a group dimension to sample 'fibers' from. For the MLP we do this by reshaping the 2048 dimensional output to (16, 128), and select one of the 16 rows at each iterations. For the CNN, we
th of the elements as if they were scale-equivariant. similarly use the 6 times larger feature space to sample 5
We thus observe that when equivariance is removed, but all else remains equal, c--sst. models perform significantly below their imput-angmentattion counterparts, and similarly to a 'froeen' randomly initialized backbone baselines, indicating the learning algorithm is no longer effective. Importantly, this indicates why existing F-SSL losses (such DIM(L) [Heelm et al. 2019) always act within equivariant dimensions (e.g. between the spatial dimensions of feature map pixels) these losses are simply ineffective otherwise. Interestingly, this provides novel insights into how the successes of carly self-supervised methods may have been crucially dependent on equivariant backbones, polentially unbeknownst to the authors at the time. An intuitive understanding of this result can be given by viewing arbitrary features as being related by some unknown input transformation which may not preserve the target information about the input. In contrast, however, since equivariant dimensions rely on symmetry transforms, contrast over such dimensions is known to be equivalent to contrasting transformed inputs.
7

---

Published in Transactions on Machine Learning Research (11/2023)
Scale (CIFAR) (CIFAR)
%A Test Aac. (relative to max)
% Test (relative
Scale (CIFAR)
-15
Translation (CIFAR)
Scale (MNIST)
Transiation (MNIST)
20
Base Size (|go] relative to (G])
Topographic Dist. (relative max gall(i)
Figure 3: Study of the impact of new H-SSL parameters top-1 test accuracy. (Left) Test accuracy marginally increases as increase total base space size Do- (Right) Test accuracy is constant or decreases we increase the maximum distance between fiber bundles considered positive pairs.
4.3 Parameters of H-SSL
Base size lgol As discussed in Section 3 The H-SSL framework identifies parameter choices such the base space go- This parameter specifically carries special importance since it is the main distinction between the A-SSL and F-SSI. losses in the literature. Specifically, the size of go is set to the full representation size in the SimCLR framework, while it is typically a small patch or an individual pixel in F-SSL losses such as DLM(L) or GIM. To investigate the impact of this difference, we explore the performance of the H-SSL framework as we gradually increase the size of go from (akin DTM(T.) lossos) to |G| - (akin to SimCT.R), with no padding. In each setting, we similarly increase the dimensionality of the input layer for the non-linear projection head to match the multiplicative increase in the dimension of the input representation =(g). In Figure (left) we plot the 7-change in top-1 accuracy CIFAR- 10 for each size. We see minor increase in performance as we increase the size, but note relative stability, again suggesting greater unity between A-SSL and H-SSL.
Topographic Distance Each augmentation in standard SimCLR augmentation stack is typically associated with scalar vector valued 'strength'. For example, this can correspond to the maximum nunber of pixels translated, the range of rescaling, or the maximum number of degrees to be rotated. We note that the same concept is present in the H-SSL framework and is defined by the associated latent representation of the transformation. For networks which use regular representations (as in this work), the degree of a transformation corresponds exactly to the degree of shift within the representation. We thus propose that an analagous concept augmentation strength is topographic distance in equivariant networks, meaning the distance between the two sampled fiber bundles as computed along the group dimensions (i.e. the 'degree of shift'). For example, for convolution, this would correspond to the number of feature map pixels between two patches. For scale, this would correspond to the number of scales between two patches. In Figure 3 (right), we explore how the traditional notion of augmentation 'strength' can be equated with the 'lopographic distance' between 91 and 92 and their associated fibers (with a fixed base size of ggol = =1). Here we approximate topographic distance as the maximum euclidean distance between sampled group elements for simplicity (Il9ı 92 Il),, where a more correct measure would be computed using the topology of the group. We see, in alignment with prior work (Tian et al.) 2020a 2019), that the strength of augmentation (and specifically translation distance) is important parameter for effective self supervised learning, likely relating to the mutual information between fibers as function of distance. We believe that the reason that we do not see the inverted U-shaped relationship between accuracy and lopographic distance as found by Tian et al. (2020a) is that their models have been trained on the much higher resolution DIV2K dataset, allowing for patch offsets of up to 384 pixels. In our case, since we are working on the latent representations of low resolution MNIST and CIFAR10 images, we only have maximum offset of 8-pixels, and therefore believe that we are only capturing fraction of the curve illustrated by others.
8

---

Published in Transactions on Machine Learning Research (11/2023)
4.4 Methods
Model Architectures All models presented this paper are built using the convolutional layers from the SESN (Sosnovik et al.) 2020) library for consistency and comparability. For scale equivariant models,
we used the set of scales [1.0. 1.25, 1.33, 1.5, 1.66, 1.75]. To construct the rotation equivariant backbones, we use only a single scale of [1.0] and augment the basis set with four 90-degree rotated copies of the basis functions at [0",90,, 90", 180", 270%] These rotated copies thus defined the group dimension. This technique of basis or iiter-augmentation for implementing equivariance is known from prior work and has been shown to be equivalent to other methods of constructing aruup-equivariant neural networks (Li et al., 2021). For translation models, we perform no basis-augmentation, and again define the set of scales used in the basis to be a single scale [1.0], thereby leaving ouly the spatial coordinates of the final feat maps to define the output group. On MNIST (LeCun & Cortes 2010), we used backbone network f composed of three SESN convolutional layers with 128 final output channels, ReLU activations and BatchNorms between layers. The output of the final ReLU is then considered our for contrastive learning (for CA-SSI, and CH-SSt.) and is of shape (128, S x R 8, 8) where S is the number of scales for the experiment (either or 6), and R is the number of rotation angles (either 1 or 4). On CIFARIO and Tiny ImageNet we used JEdN-ed ResNet18 and ResNet20 models respectively where the output of the last ResNet blocks were taken as z for castrastive learning. For all models where translation is not the studied transformation, we average pool over the spatial dimensions to preserve consistent input-dimenioonality to the nonlinear projection head.
Training Details For training, we use the LARS optimizer (You et al. 2017) with an initial learning rate of 0.1, and a batch size of 4096 for all models. We use an NCE temperature (r) of 0.1, half-precision training, a learning rate warm-up of 10 epochs, a cosine Ir-update schedule, and weight decay of 1 x 10-4, On MNIST we train for 500 epochs and on CIFARIO and Tiny ImagNet (TIN) we train for 1300 epochs.
Computational Complexity On average each MNIST run took 1 hour to complete distributed across 8 GPUs, and each CIFARIO/TIN run look 10 hours to complete distributed across 64 GPUs. In total this amounted to roughly 85,000 GPU hours. While equivariant models can be slightly more computationally expensive due to expanded feature space dimensionality, this cost is typically not prohibitive for training on full scale datasets such as ImageNet. In practice, we found that equivariant models did not train more than a factor of 2-3 slower than their non-equívariant counterparts, dependent on architecture, and often nearly equivalently fast as non-equivariant baselines. Conversely, however, H-SSL may provide computational complexity reductions in certain settings, since there is not need to perform two forward passes on augmented images, and rather all contrastive learning can be performed in the feature space of a single image. With this new framework, sufficient. future work on layerwise contrastive losses, or new contrastive losses, may uncover computationally cheaper SSL algorithms leveraging H-SSL
5 Related Work
Our work is built upon the literature from the fields equivariant deep learning and self-supervised learning outlined in Sections D
and 2 Beyond this background, our work is highly related in motivation to number
of studies specifically related to equivariance in self-supervised learning.
Undesired Invariance in SSL One subset of recent prior work has focused on the undesired invariances learned by A-SSL methods {Xiao et al. 2021 Tsai et al. 2022) and on developing methods by which to avoid this through learned approximate equivariance (Dangovski et al., 2022 Wang et al.) 2021). Our work is, to the best of our knowledge, the first to suggest and validate that the primary reason for the success of feature-space SSL objectives such as DIM(L) (Hjelm et al. 2019) and GIM (Löwe el al., 2019) is due to their exploitation of (translation) equivariant backbones (i.e. CNNs). Furthermore, while prior work shows benefits to existing augmentation--aaso SSL objectives when oquivariance inducod, our work investigates how equivariant representations can directly be leveraged to formulate new shorrticalll-groundde SSL objectives. In this way, these two approaches may be complimentary.
9

---

Published in Transactions on Machine Learning Research (11/2023)
Data Augmentation in Feature Space There exist multiple works which can similarly be interpreted as performing data augmentation in feat space both for supervised and self-supervised learning. These include Dropout (Srivastava et al., 2014), Manifold Mixup (Verma et al.) 2018), and others which perform augmentation directly in feature space (DeVries & Taylor 2017 Hendrycks et al.) 2020 or through generative models (Sandfort et al. 2019). We see that our work is fundamentally diflerent. from these that it is not. limited to simply performing an augmentation which would have been performed the input in latent space. Instead, it maximally leverages structured representations to generalize all of these approaches and show how others can be included under this umbrella. Specifically, a framework such DIM(L) is not explicitly performing an augmentation in latent space, but rather comparing two subsets of a representation which are offset by an augmentation. As we discuss in Section [ this distinction is valuable for developing novel SSL algorithms which can substitute learned homomorphisms for learned augmentations potentially sidestepping challenges associated with working in input-space directly.
Hybrid A-SSL + F-SSL Some recent work can be seen leverage both augmentation-basee and feature-space losses simultaneously. Specifically, Augmented Multiview Deep InfoMax (Bachman et al. 2019) achieves exactly this goal and is demonstrated to yield improved performance over its non-hybrid counterparts. Although similar in motivation, and perhaps performance, to our proposed framework, the Homomorphic SSL. framework differs by unifying the two losses into a single objective, rather than a sum of two separate objectives.
6 Discussion
In this work we have studied the impact of combining ougmentation-ommmmrryy feature extractors with augnentatioon-aased SSL objectives. In doing so, we have introduced new framework we call Homonorphic--ST. which illustrates an equivalence between previously distinct SSL methods when the homomorphism constraint is satisfied.
Primarily, the H-SSL framework opens the door to further development of augmentation free self-supervieed learning, as was initially pursued in the literature by frameworks such as Deep InfoMax (Hjelm et al. 2019). The necessity for hand-designed augmentations is known to be one of the current limiting factors with respect to self-supervised learming, contributing to biased represcntations (Xiao al. 2021 Tsai et al. 2022: Dangovski et al.] 2022) and more generally determining the overall generalization properties of learned representations (Ji et al. 2021 Wang et al. 2022 von Kügelgen et al. 2021). We therefore believe our work provides valuable new alternative viewpoint through which these limitations may be addressed.
Secondly, with increased augmentation-free SSL methods arises the potential for layerwise 'greedy' SSL, as demonstrated by the Greody InfoMax work (Löwe et al., 2019). We believe a factor which has to-date limited the development of layerwise sell-supervised learning approaches has indeed been the dominance of augmentations in SoTA SSL, and a lack of knowledge about how these augmentations may be performed equivalently in the deeper layers of a neural network. Homomorphic feature extractors exactly provide this knowledge and therefore promise to extend the possibilities for such 'greedy' self-supervised learning methods which have significant. implications for computational efficiency and biological plausibility.
We believe that one of the most promising implications of the H-SSL framework is that the lought goal of 'learned SSL augmentations' may, in this view, be equivalently achieved through learned homomorphisms. While the field of learned homomorphisms is very new and undeveloped, we believe there are already interesting approaches in this direction which would provide an immediate starting point for future work (e.g. Keller & Welling (2021): Keurti et al. (2022): : Connor et al. (021] Dehmamy et al.
Future Work
2021
Pal & Savvides 2018)). Since these approaches differ significantly from their input-space counterparts, it is likely that they may have the potential to circumvent otherwise difficult obstacles of operating in input-space.
Despite the unification of existing methods, and benefits from generalization, we note that this approach is still limited. Specifically, the equivalence between CA-SSL and cH-sst, and the benefits afforded by this equivalence, can only be realized if it is possible to analytically construct a neural network which is equivariant with respect to the transformations of interest. Since it is not currently known how to
Limitations
10

---

Published in Transactions on Machine Learning Research (11/2023)
construct neural networks which are analytically equivariant with respect all input augmentations used in modern SSL, this constraint is precisely the greatest current limitation of this framework. Although the field of equivariant deep learning has made significant progress in recent years, state of the art techniques are still restricted to E(n) and continuous compact and connected Lic Groups (Finzi et al. 2020; 2021: Cesa et al., Weiler & Cesa 2019). We believe in this regard, our analysis sheds some light on the success of methods which perform data augmentation over those which operate directly in feature space recent literature it is simply too challenging with current methods to construct models with structured representations for the diversity of transformations needed to induce sufficient set of invariances for linear separability of classes. We therefore propose this work not as an immediate improvement to the state of the art, but rather as a new perspective on SSL which provides bridge to previously distant literature.
2022
References
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning, 2019. URL https://arxiv.org
abs/1902.09222
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint arx: S5t:1990000910 2019.
Gabriele Cesa, Leon Lang, and Maurice Weiler. A program to build -NN)eqquivarrinn steerable CNNs. In International Conference on Learning Representations, 2022. URL http:s//ppearrriieeneefffouueuee
WE4qe9xInQw
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.0500 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning, 2020.
Taco Cohen and M. Welling. Steerable cnns. ArXiu, abs/ /t///1112.0899.. 2017.
Taco Cohen and Max Welling, Group equivariant convolutional networks. In International conference on machine learning, pp. 1990-2990 2016.
Marissa Connor, Gregory Canal, and Christopher Rozell. Variational autoencoder with learned latent structure. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 Proceedings of Machine Learning Research, pp. 2359 2367. PMLR, 13 15 Apr 2021. URL http:///rroceedings amapiiinnnnmmm......mmm.m..me..m mlr. wwt.:m..........ee.e.em www.pprsssvvv000 connor21a. html
Rumen Dangovski, Li Jing, Charlotte Loh, Seungwook Han, Akash Srivastava, Brian Cheung, Pulkit Agrawal, and Marin Soljacic. Equivariant self-supervised learning: Encouraging equivariance in representations. International Conference on Learning Representations, 2022. URL https:://ppererriew..eo http://pppaariiieeeeeeefforrrccc net./orunn
gKLAAfiytI
Nima Dehmamy, Robin Walters, Yanchen Lin, Dashun Wang, and Rose Yu. Automatic symmetry discovery with lie algebra convolutional network. In A. Beygekzimer, Y. Dauphin, Liang, and J. Wortman Vaughan
(eds.), Advonces in Newral Information Processing Systems, 2021. URL htttpe///penneeveee..eefff
1d=NPOWF ZLFC5
Terrance DeVries and Graham W. Taylor. Dataset augmentation in feature space, 2017. URL https
Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representa-
tions via multi-view information bottleneck, 2020. CRL /ttp:::/aarxx..oog//////00....
Mare Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for cquivariance to lie groups on arbitrary continuous data. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3165-3177 html
PMLRR 13- 18 Jul 2020. URL htt.hitttt//pprceeinggsss http://www..m...eeee//e////////e
11

---

Published in Transactions on Machine Learning Research (11/2023)
Mare Finzi, Max Welling, and Andrew Gordon Gordon Wilson. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 3318-3328. PMLR, 18-24 Jul 2021. URL htttps://proceeddngg mlr
press/v139/finzi22t html
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukeuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: new approach to self-supervised learning, 2020.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distributio generalization. CoRR, abs/2006.1622 16241, 2020. URL
https://arxiv http:///axxiv..orabbbbbeee orgg/b///////////oo 2206. 16241
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildonn Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representalions, 2019. URL htttp::/ wtw.::ww.ee//fornre/ee
1d-BxIr3JOcXX
Wenlong Ji, Zhun Deng, Ryumei Nakada, James Zou, and Linjun Zhang. The power of contrast for feature lcarning: A theoretical analysis, 2021.
T. Anderson Keller and Max Welling. Topographic VAEs learn equivariant capsules. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreeieww wttp::/www..eeeoeeooooaaaaaaaaa
Hamza Keurti, Hsiao- Pan, Michel Besserve, Benjamin F. Grewe, and Bernhard Schölkopf. Homomorphism autoencoder learning group structured representations from observed transitions, 2022. URL. https:
12067
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). URL
http://www.cc. wtt.wwwwcst......oo www.ntorrotooo..oooo wat.atti/odd///ii///cci/aa html
Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge. 2015.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http:://yan. lecun com/exdbmniist/
Yann LeCun and Ishan Misra. Self-supervised learning: The dark matter of intelligence, Mar 2021. URL teaiinn-tth--aaae the-ddrr-mattee--offf matter- mftter--of-inellgennee
Jason D Lee, Qi Lei, Nikunj Saunshi, and JIACHENG ZHUO. Predicting what you already know helps: Provable self-supervised learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 509-323. Curran Associates, Inc., 2021. URL https://prroceeding: hewpnediinnnnnuuiisp..e hwwprps.ccpppppppr////////ee pttpc//ppppp///////1.... D265Gd00FFFF44022933333777 pdf.
Bo Li, Qili Wang, and Gim Hee Lee. Filtra: Rethinking steerable emn by filter transform. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 1525- PMLR, 18-24 Jul 2021. URL
https://proceeding mlr. wttpseawww.....eeeeeeeeememeeeee wttpps///e//ei///./// html
Sindy Löwe, Peter O'Connor, and Bastiaan Veeling. Putting an end to end-to-end: Gradient-isolated learning of representations. In Advances in Neural Information Processing Systems, 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2018. URI. http://axxvvvooo/bbb/a 03748
12

---

Published in Transactions on Machine Learning Research (11/2023)
Dipan K. Pal and Marios Savvides. Non-parametric transformation networks, 2018. URL https://arxiv
org/ /t////a///111.1.// .45200
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and 1lya Sutskever. Learning transferable visual models from natural language supervision, 2021. CRL httpss//aarrav..oorraaaa////
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scout Gray, Chelsea Voss, Alec Radford, Mark Chen, and lly Sutskever. Zero-shot text-to-image generation, 2021. URL https:///artti www.aaiwwi.....ra..aaaaeeeeee wtw..////.............2.
Veit. Sandfort, Ke Yan, Perry J. Pickhardt, and Ronald M. Summers. Data augmentation using genera- tive adversarial networks (CycleGAN) to improve generalizability in CT segmentation tasks. Scientific Reports, 9(1), November 2019. doi: 10.1038/ (00088/441598009-55555 B41596--09-55277--x
URL
Jean-Pierre Serre. Linear representations of finite groups,, volume 42 of Graduate teats in mathematics. Springer, 1977. ISBN 978-3-540-90190-7.
Ivan Sosnovik, Michal Szmaja, and Arnold Smeulders. Scale-cquivariant steerable networks. In International Conference on Learning Representations, 2020. URL
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinor. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56): 1929 1958, 2014. CRL http://jmlr: http:///jll..org/peeee http:///ppppe////ttttvataaaaaaaa
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding, 2019. URL
https
V/axivvorg//bbs
05849
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In Proceedings of the 34th International Conference on Newral Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020a. Curran Associates Inc. ISBN 9781713829546.
Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning with dual deep networks, 2020b. URL httpps:///ari..oo//aa//////// 00578
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervisee dynamics without contrastive pairs. CoRR, 26800102.00810, 2021. URL [BBl//tppp//aaxxxorrr/////220000
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redundancy, and linear models, 2020. URL http:///aaaii...r///////////////
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Pouis-Philippe Morency. Self-supervised learning 05576.
from multi-view perspective, 2020. URL
Yao-Hung Hubert Tsai, Tianqin Li, Martin Q. Ma, Han Zhao, Kun Zhang, Louis-Philippe Morency, and Ruslan Salakhutdinov. Conditional contrastive learning with kernel. In International Conference on Learning Representations, 2022. URL
Elise van der Pol, Daniel Worrall, Herke van Hoof, Frans Oliehoek, and Max Welling. Mdp homomorphic networks: Group symmetries in reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Newral Information Processing Systems, volume 33, pp. 4199-4210 Curran Associates, Inc., 2020. URL https://prroceedings http://.w......cc/ppp/p////////. Dletf1cc8320007722777755666--222 .dd.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, Aaron Courville, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states, 2018. URT. http:///xxxv..o/aaa/////ee/
13

---

Published in Transactions on Machine Learning Research (11/2023)
Julius von Kügglgeen, Yash Sharma, Luigi Grescle, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Newral Information Processing Systems, volume 34, pp. 16451-16467. Curran Associates, Inc., 2021. URL https: pdf
Yifei Wang, Zhengyang Geng, Feng Jiang, Chuming Li, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Residual relaxation for multi-view representation learning. In A. Beygekzmer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL
/ttpp:///peerrvvie.. wett:foruuniddd/BBBeEG...
Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap, 2022.
Maurice Weiler and Gabriele Cesa. General B(2)-Equivariant Steerable CNNs. In Conference Newral Information Processing Systems (NeurIPS), 2019.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, pp. 10402-10413, Red Hook, NY, USA, 2018. Curran Associates Inc.
Daniel Worrall and Max Welling. Deep scale-spaces: Equivariance over scale. In H. Wallach, H. Larochelle, A. Beygekzier, F. d' A'AcheeBu,,, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019. URL https://proceeddiin htw.:/oceddingg.uuurrs...ooom wouripp..ccppper.. pdf
Daniel E. Worrall, Stephan J. Garbin, Daniyar Turmukhambetov, and Gabriel J. Brostow. Harmonic networks:
Deep translation and rotation equivariance. 2017 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), pp- 7168 7177, 2017.
Tete Xiao, Xiaolong Wang, Alexci A Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. In International Conference on Learning Representations, 2021. URL htttps://openrevierr /et.teewwwiwewieeeeee/eee forum?ide 70zmm11122280
Yang You, Igor Gilman, and Boris Ginsburg. Large batch training of convolutional networks, 2017. URL https://arxiv. org/abs//7088
Richard Zhang. Making convolutional networks hift-invvriant again. In International conference on machine learning, pp. 7324 7334. PMLR, 2019.
A Experiment Details
Model Architectures On MNIST (LeCun & Cortes 2010 we used backbone network f composed of three SESN convolutional layers with # channels (32, 64, 128), kernel sizes (11, 7, 7), effective sizes (11, 3, 3), strides (1, 2, 2), padding (5, 3, 3), no biases, basis type 'A', BatchNorm layers after each convolution, and ReLU activations after each BatchNorm. The output of this final ReLU was then considered our z for contrastive lcarning (with CA-SSE and CH-SSL) and was of shape (128, S x R, 8,8) where S was the number of scales for the experiment (either or 6), and R was the number of rolation angles (either or 4). For experiments where the transformation studied was not translation, spatial dimensions before applying the projection head h to achieve a consistent dimensionality of 128. For classification, an additional SESN convolutional layer was placed on top with kernel size 7, effective size 3, stride 2, and no padding, thereby reducing the spatial dimensions to 1, and the total dimensionality of the input to the final linear classifier to 128.
average pool over the
On CIFAR10 we used ResNet20 model composed of an initial SESN lifting layer with kernel size 7, effective size 7, stride 1, padding 3, bias, basis type 'A', and output channels. This lifted representation
14

---

Published in Transactions on Machine Learning Research (11/2023)
was then processed by following SESN convolutional layer of kernel size 7, effective size 3, stride 1, padding 3, no bias, basis type "A', and 64 output channels. This initial layer was followed by a BatchNorm and ReLU before being processed by three ResNet blocks of output sizes (128, 256, 512) and initial strides of (1, 2, 2). Each ResNet block is composed of 3 SESN Basic blocks as defined here (https: V/githu.. wtt./oommmmmaa
ses ses. py#L19). The output of the third ResNet. block was taken as our z for contrastive learning (again for CA-SSL and LH-SsL) of shape (512,5 x R, 7) Again, as for MNIST, for experiments where the transformation studied was not translation, we average pool over the spatial dimensions before applying the projection head h to achieve consistent dimensionality of 512. For classification, the vector z was first max-pooled along the scale/ scale/rotation rrotation group-axis (S x R),
followed by a BatchNorm, a ReLU, and average pooling over the remaining 7 x 7 spatial dimensions. Finally, we apply BatchNorm to this 512-dimensional vector before applying the non-linear projection head h.
On Tiny ImageNet we use Resnet20 model which has virtually the same structure as the CIFARIO model, but instead uses 4 ResNet blocks of output sizes (64, 128, 256, 512) and strides (1, 2, 2, 2). Furthermore, each ResNet. block is composed of only 2 BasicBlocks for TIN instead of 3 for CIFAR10. Overall this results in of shape (512, S R, 4, 4), and final vector for classification of size 512. We note that we do not include Translation results in Table | for Tiny ImageNet preciscly because the spatial dimensions of the feature map with this architecture are too small to allow for effective H-SSL training in the settings we used for other methods.
All models used a detached linear classifier for computing the reported downstream classification accuracies, while the Supervised baselines used an attached linear layer (implying gradients with respect to the classification loss back-propagated though the whole network). All models additionally used an attached non-linear projection head h constructed as an MLP with three linear layers. For MNIST these layers have of output sizes (128, 128, 128), while for CIFAR10 and TIN they have sizes (512, 2048, 512). There is a BatchNorm after each layer, and ReLU activations between the middle layers (not at the last layer).
Empirical Validation For the experiments in Table 1 we use two different methods for data augmentation, and similarly two different methods for selecting the representations ultimately fed to the contrastive loss for the A-SSL and H-SSL settings.
For A-SST we augment the input at the pixel level by: randomly translating the image by up to + 20% of its bhight/widte (for translation), randomly rotating the image by one of (0%, 90", 180 270%) (for rotation), randomly downscaling the image between 0.57 and 1.0 of its original scale. For S-SSL we use no input augmentations.
For both methods we use only a single fiber, meaning the base size ggol is 1. For A-SSL, we randomly select the location go for each example, but we use the same go between both branches. For example, in translation, we compare the feature vectors for two translated images at the same pizel location. Similarly, for scale and rotation, we pick single scale rotation element to compare for both branches. For H-SSL, we randomly select the location g independently for each example and independently for each branch, effectively mimicing the latent operator.
H-SSL Without Structure In Table 2, we use the same overall model architectures defined above (3-layer model or ResNet20), but replace the individual layers with non non-equivariant counterparts. Specifically, for the MLP, we replace the convolutional layers with fully connected layers with outputs (784, 1024, 2048). For the convolutional models (denoted CNN (6 x CHW)), we replace the SESN kernels with palleeparameterizedd non-equivariant counterparts, otherwise keeping the output dimensionality the same (resulting in the x larger output dimension).
Furthermore, for these un-structured representations, in the H-SSL setting, 'emulate' a group dimension to sample 'fibers' from. Specifically, for the MI.P we simply reshape the 2048 dimensional output to (16,128), and select one of the 16 rows at each iterations. For the CNN, we similarly use the 6 times larger feature th space to sample 8 of the elements if they were scale-equivariant.
15

---

Published in Transactions on Machine Learning Research (11/2023)
Parameters of H-SSL For Figure 3 (left), select patches of sizes from 1 to |G] - with no padding. In each setting, we similarly increase the dimensionality of the input layer for the non-linear projection head h to match the multiplicative increase in the dimension of the imput representation =(g). For the topographic distance experiments (right), we keep a fixed base size of g = and instead vary the maximum allowed distance between randomly sampled pairs 91 & 92-
B Extended Background
Our work is undoubtedly built upon the the large literature base from the fields equivariant deep learning and sclf-supervised learning as outlined in Sections [| and 2 Beyond this background, our work is highly related in motivation to a number of studies specifically related equivariance in self-supervised learning. Most prior work, however, has focused on the undesired invariances learned by A-SST. methods (Xiao et al. 2021 Tsai et al. 2022) and developing methods by which to avoid this through learned approximate equivariance (Dangovski et al. 2022 Wang et al. 2021). Our work is, to the best of our knowledge, the first to suggest and validate that the primary reason for the success of feature-space SSL objectives such as DIM(L)
Related Work
(Hjelm et al. 2019) and GIM (Lwwe et al. 2019) is due to their exploitation of equivariant backbones.
(L in H-SSL In this section we outline precisely how the Deep Infomax Local loss (T. relates to the H-SSL framework proposed in Section 3 Specifically, in Deep InfoMax (DIM(L.) the same general form of the loss function applied (often called InfoNCE), but the cosine similarity is replaced with log-bilinear model: sim b) exp (a (D). Additionally, and most importantly to this work, rather than computing the similarity between two differently augmented versions an image, the loss is applied between different. spatial locations of the representation for a single image, again with a head h applied afterwards. If we let ~~ zhaw refer to sampling a contiguous patch from the spatial coordinates of a convolutional feature map, we can write this general Feature- Space InfoMax loss (CFSIM) as:
N
exp
sim (Mzz)())) (M
(X) = Σ
1
B 02 log
(8)
Eags E3, exp sim (h(*; h(z h(zx(())))
To show that this is equivalent to our CH-SSL, we see that the randomly sampled spatial patches gı 92 can oquivalently be described as a single base patch Io shifted by randomly sampled translations I1 and 92- Explicitly,
exp sim (h(z.(0i gul)) h 1(=(12 go)))
Σ G log (X) N
Aoti E3, exp
go))), h( (0) g 90)))
Thus, see that Feature-Space InfoMax losses are included in our framework, and can therefore be seen to be equivalent to inpul-augmentation based losses with equivariant backbone, where the set of augmentations is limited to the translation group G = 2 and the Do base size is a single spatial coordinate (Igol = 1) rather than the size of the full representation (1gol = G|).
C Broader Impact
This work is primarily related to understanding and improving self-supervised learning a training method for deep neural networks which is able to leverage large amounts of unlabeled data from the internet, making it one of the most used methods for state of the art image and text generative models today (Radford et al., 2021 Ramesh et al.] 2021). Such models have significant broader impact and potential negative consequences which are beyond the scope of this work. We refer readers to discussions of those paper for further information. Specifically, this work aims to improve such SSL techniques, thereby inheriting the broader impact of these models.
16


Reviews:
[{"review_id": "XP6gjgcbPP", "reviewer": "Reviewer_yinW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The main idea of the paper is simple and well motivated: combine homomorphic feature extractors with augmentation-based SSL objectives. \n- The intuition from their formulation, regarding the influence of base size and the group under consideration, is simple but interesting. \n- The paper shows that two common formulations in the SSL literature can be seen using the lens of the proposed H-SSL framework. \n- The authors discuss the limitations of their approach with candor.\n\nWeaknesses:\n- The main idea is actually quite simple and intuitive (and this is a good thing), however, it is then incumbent on the authors to include experiments that could convince the practitioner of the usefulness of this approach. The experiments give the impression that they cover the bare minimum needed to support the discussion in the paper, but they are far from compelling. What happens with continuous groups are considered? What if we considered problems in equivariant learning specifically and then did the augmentation along transformations that the backbone network is not made equivariant to, but is nonetheless a symmetry that exists in the data. What are the failure cases of such a SSL approach when there is a symmetry mismatch between the symmetry of the backbone network + augmentations and that of the data. How do such representations generalize? Give the approach is very intuitive, I would like to see at least some results supporting some of these points. Currently, I am finding it a bit difficult to get a take home message from the paper. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "One way of looking at self-supervised learning is that it 'bootstraps' the process of learning informative representations by applying transformations to the input that leave some underlying task-relevant information unchanged. Although learning objectives in SSL are only approximately related to some downstream task, the representations learnt have proven to be powerful across different application areas and domains. Another line of work on group equivariant networks seeks to hard-code equivariance to transformations on the input -- the idea being that it is a more principled and complete way of doing data augmentation when the set of transformations form a group. \n\nThe main idea of the paper is that some ideas in self-supervised learning can be unified and generalized when seen through the lens of equivariant learning. More specifically, it studies SSL when the backbone feature extractor is a group equivariant network. The paper shows that a family of SSL loss functions can be unified and generalized with this viewpoint. First, in section 2.1 is covers the basic ideas behind GCNNs, and in section 2.2 it covers the basic ideas behind SSL. Two common losses in SSL are divided into two categories, one that relies on a vector similarity between pairs of examples and then has a siamese/drLIM type contrastive loss function (e.g. the paper quotes something similar to the neighborhood component analysis objective), and another which could be considered 'non-contrastive\" losses. They could be considered non-constrastive since they don't use negative example as in contrastive setups and substitute it by some approprite regularization instead. Based on the relationship between the example pairs, the authors further divide methods into A-SSL (augmented SSL) and a slightly more general version i.e. F-SSL. Then a simple formulation inspired by GCNNs is considered next and named H-SSL. It is argued that H-SSL includes both F-SSL and A-SSL.\n\nExperiments consider a backbone equivariant network. An example network considered, for instance, is both equivariant to 90 degree rotations and to scale. The selection of the augmentation also follows a straightforward (which is motivated using the language of fibers from the steerable CNNs paper). Finally a non-equivariant backbone H-SSL is also considered. Some results are reported on MNIST, CIFAR10 and tine ImageNet to validate the proposed method. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would love to hear from authors on what they think of additional experiments and could continue the discussion based on that. ", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xlHzvb785h", "reviewer": "Reviewer_7QDP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* The motivation and exposition in the paper are clear and understandable. Investigating the relations between SSL augmentation and network group structure is a relevant and interesting direction.\n* The work provides a novel perspective for understanding feature-based SSL such as Deep Infomax and augmentation-based SSL such as SimCLR within the same framework. The key insight is that group structure in the input space can be studied by group structure in hidden layers or the output space as long as the network is equivariant to the group action.\n* Results in Table 1 give interesting preliminary evidence of the equivalence between augmented SSL and H-SSL in practical settings, and the results in Table 2 clearly show that group equivariant network architectures are vital for effective performance of H-SSL.\n\n**Weaknesses**\n* The primary weakness is the limited scope of the method (which is acknowledged by the authors in the limitations section). The method requires special equivariant network architectures, which are currently limited to a small number of group actions. Therefore the method can currently only be applied to rotation, translation, and scaling augmentations. Strong SSL baselines like SimCLR typically rely on more aggressive augmentations like cutout and colorization which might not be easy to describe in terms of group action, and which do not have special equivariant architectures. Therefore, the method is unlikely to contribute to SOTA SSL results without further development of equivariant architectures to support the method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work studies self-supervised learning (SSL) from the perspective of homomorphic functions. The paper is motivated by the intuition that image transformations with group structure are an essential part of modern contrastive self-supervised learning, which leads to an investigation of group properties of widely used self-supervised learning methods. Given a network which satisfies a certain group equivariance (meaning that a group action on the input space has a compatible group action in the output space), one can define a Homomorphic SSL (H-SSL) objective which uses a contrastive loss of group actions in the output space. This is shown to be equivalent to augmentation-based contrastive learning (such as SimCLR) which uses the corresponding input-space group action as a way to generate augmentations. Furthermore, it is shown H-SSL encompasses patch-based feature space SSL methods such as CPC and Deep Infomax. Experimental results corroborate the equivalence of augmentation SSL and H-SSL by showing very similar results for CIFAR-10, MNIST, and TinyImageNet for downstream linear classification. Ablation studies using a network without equivariance shows that the equivalence breaks down and H-SSL has very poor performance when the input-space group action is not preserved by the network forward pass.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I have the impression that more experiments could be conducted to demonstrate the equivalence of H-SSL and A-SSL. For example, given a network with identical weights, one could measure the correlation between hidden layer activations from group actions on the input space and output space respectively. There should be nearly perfectly correlated (up to numerical inaccuracies from aliasing/boundary etc. that upset the theoretical equivalence).", "broader_impact_concerns": "Broader impacts are appropriately discussed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ybyVbEZYQB", "reviewer": "Reviewer_CJwi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nSA: H-SSL makes sense and it is a nice addition to the literature.\n\nSB: The authors are mostly clear about the limitations of their approach.\n\nSC: It is nice that H-SSL does not need augmentations on the input space.\n\nWeaknesses:\n\nWA. The authors use A-SSL on an equivariant neural network. Typically, in equivariant NNs, there is a global (group) pooling layer at the end, which makes the representation *invariant* to the group action. It seems that if one uses the pooling layer, then the objective of the positive pairs in contrastive learning objective is less clear. I don't think the authors discuss that detail, but it would be nice to clarify my confusion.\n\nWB: What is the qualitative conclusion of Figure 3? Typically, Tian et al. found that there is a \"Sweet spot\" for augmentation strength. Is there such a \"Sweet spot\" in H-SSL?\n\nWC: While H-SSL is input augmentation-free, the complexity is transferred to the architecture and the action on the output representations. This is addressed by the authors, but I would like to further underline it here.\n\nWD: The GPU hours in the Training Details are another limitation that is buried in the text. It seems like training equivariant NNs at scale is very challenging, which makes me skeptical about the scalability of H-SSL to large (more realistic) SSL settings. \n\nMinor: \n\nMA. I would suggest moving Table 2 to the top. Currently, there is a paragraph between Tables 1 and 2, which breaks the flow of the text. \n\nMB: In Figure 3 (right), the CIFAR curves seem to be at a different scale than the MNIST curves. It would be useful to plot a separate figure for CIFAR. For example the \"Scale\" (not to be confused with the *scale* [of the test set drop] in my comment) curve for CIFAR is flat currently. \n\nMC: In Appendix A, when you cite Li et al. (2021) use \\citep instead of \\citet.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce a formalism of Homomorphic SSL (H-SSL) and they show that this formalism can explain both Augmentation-based SSL (A-SSL) and Feature-space SSL (F-SSL). The SSL loss considered in the paper is contrastive-based (InfoNCE). Practically, in H-SSL we act on the output representations space to form positive pairs. Importantly, in order for this SSL to work we need a structural bias, i.e. the architecture we use should be equivariant. The authors support their claims with clear derivation and reasonable experiments. Another interesting take from the paper is a potential explanation of the success of SSL objectives based on CNNs, i.e. that the objectives might have been particularly suitable to the translation-equivariant bias of a CNN.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "CA. More discussion about the equivariant biases and the success of SSL. I feel like currently the discussion is hidden in the text, however, I think this is quite interesting. \n\nCB. How would you explain the success of ViTs in SSL in the context of CA? I am somewhat reminded of that paper: https://openreview.net/forum?id=JL7Va5Vy15J. Curious to hear your take. \n\nCC: I think an experiment with non-contrastive losses would be useful to see the empirical scalability of your approach.\n\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores self-supervised learning (SSL) using homomorphic functions and group-equivariant networks. It introduces a Homomorphic SSL (H-SSL) framework, highlighting the importance of group structures in image transformations for contrastive SSL. The study establishes an equivalence between H-SSL and augmentation-based SSL through experiments. Reviewers acknowledge that the claims have been well-supported with contributions to a novel SSL perspective, and appreciate the clarity and motivation. Somehow they express concerns about the method's limited scope due to special equivariant architectures, suggesting more experiments to enrich the paper.
Decision: Accept (poster)