Contrastive-Equivariant Self-Supervised Learning Improves Alignment with Primate Visual Area IT
Authors: Thomas Edward Yerxa
Jenelle Feather
Eero P Simoncelli
SueYeon Chung
Venue: NeurIPS 2024 poster (2024)
Keywords: Neural Predictivity
Equivariance
Factorization
Self-Supervised Learning
Primary area: neuroscience_and_cognitive_science

Abstract: Models trained with self-supervised learning objectives have recently matched or surpassed models trained with traditional supervised object recognition in their ability to predict neural responses of object-selective neurons in the primate visual system. A self-supervised learning objective is arguably a more biologically plausible organizing principle, as the optimization does not require a large number of labeled examples. However, typical self-supervised objectives may result in network representations that are overly invariant to changes in the input. Here, we show that a representation with structured variability to the input transformations is better aligned with known features of visual perception and neural computation. We introduce a novel framework for converting standard invariant SSL losses into "contrastive-equivariant" versions that encourage preserving aspects of the input transformation without supervised access to the transformation parameters. We further demonstrate that our proposed method systematically increases models' ability to predict responses in macaque inferior temporal cortex. Our results demonstrate the promise of incorporating known features of neural computation into task-optimization for building better models of visual cortex.

Contrastiv--Qquivariat Self-Supervised Learning Improves Alignment with Primate Visual Area IT
Thomas Yerxa
Jenelle Feather Eero P. Simoncelli 1.2 Yue Chung 1.2
Center for Neural Science New York University
*Center for Computational Neuroscience, Flatiron Institute, Simons Foundation
Abstract
Models trained with self-supervised learning objectives have recently matched surpassed models trained with traditional supervised object recogmition in their ability predict neural responses of object-selective neurons the primate vi- sual system. A self-supervised learning objective is arguably more biologically plausible organizing principle, as the optimization does not require large number of labeled examples. However, typical seff-supervieed objectives may result in network representations that are overly invariant to changes in the input. Here, we show that representation with structured variability to input transformations is better aligned with known features of visual perception and neural computation. We introduce novel framework for converting standard invariant SSL losses into "contrastive -euisviriieeuuvviiian"." versions that encourage preservation of input transforma- tions without supervised access to the transformation parameters. We demonstrate that our proposed method systematically increases the ability of models to predict responses in macaque inferior temporal cortex. Our results demonstrate the promise of incorporating known features of neural computation into aast-optimization for building better models of visual cortex.
Introduction
In the past decade, task-optimized deep neural networks (DNNs) have been used to predict responses of object-selective neurons in primates to natural image stimuli (Yamins al.) 2014 [Schrimpf et al.) 2020 Willeke et al., 2023). Such networks have pronounced advantage over more traditional models for explaining responses in deeper areas with more abstract representations, such inferior temporal cortex (IT). This observation naturally leads to the hypothesis that task optimization can provide a normative account for IT neuron tuning properties: late-stage visual representations shaped by the need to perform ecologically relevant tasks.
However, the task that initially led these advances was that of supervised object classification, specific task that relies on an implausibly large number of labeled examples [Lindsay 202T]. More recently, computer vision has undergone "self-supervised learning" (SSL) revolution. Á variety of methods have been proposed to learn representations that match or surpass supervised training on multiple tasks by deriving sources of supervision from the data itself rather than relying on human annotations. For example, many popular SSL strategies aim to unify representations of different transformations of the same image (commonly referred to "views"), while enforcing diversity among representations of distinct images. Additionally, self-supervised representations can predict primate neural responses with fidelity comparable to supervised representations (Zhuang et al. 2021 Konkle and Alvarez 2022 Parthasarathy et al. 2024).
"Corresponding Author: tey214@nyu.ed. edu
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Both of these training objectives are forms of invariance learning: responses of an ideal object classification model should be invariant across different objects from the same class, and self- supervised learning strives to achieve invariance to the transformations used to generate different views. However biological visual representations are not fully invariant across views (DiCarlo and Cox 2007, Kuoch et al., 2024]. Indeed it has been demonstrated that training according to either of these two objectives leads to representations that are invariant to stimulus perturbations that are salient to human observers [Feather et al. 2023), Additionally, even in Area IT, which is thought to subserve invariant object recognition, neural populations encode a significant amount of "category orthogonal" information (e.g., object pose or viewing conditions that are unrelated to semantic category) (Hong et al.) 2016]. Furthermore, such selectivity for object-orthogonal attributes is meaningfully organized within Area IT (Hong et al. 2016 (i.e. object orthogonal attributes are linearly decodable from population responses). Whether such structured variability emerges in invariance-trained networks is likely determined by the uncontrolled inductive biases of the network architecture [Alleman et al. 2024].
Here, we develop equivariant learning framework that encourages such structured variability in network representations. Our contributions are:
We propose a novel framework that converts standard invariance- based self-supervised learn- ing methods into "contrasive--cuuivariitt versions that produce structured, transformation- related variability. Unlike previous approaches, our method does not require supervised access to transformation parameters or costly modifications to the training procedure.
We examine the tradeoff between invariance and structured variability through a series of representational analyses. We find that, relative to networks trained for invariance alone, our contrastive-equivare network learns structured transformation variability that is shared across images and factorized with respect to variability related to changes in image content.
We explore the impact of including an equivariant loss for predicting neural activity in IT,, showing for the first time that explicitly encouraging structured variability via optimization leads to an improved ability to predict cortical responses to natural images.
2 Method
2.1 Transformation-Invan Self-Supervised Learning (iSSL)
The influential work of [Chen et al. 2020 showed that applying two random transformations (often called "augmentations") to batch of images, then training a network to identify which pairs of transformed images originated from the same sample with a cross-entropy style loss (the InfoNCE loss, first formulated in [Gutmann and Hyvärinen 2010) yields representations that are competitive with supervised training for object classification. Many subsequent studies have developed alternative objective functions that produce similar results: Barlow Twins (Zbontar et al. [202]], VICReg (Bardes) et al,, [222], and W-MSE [Ermolov et al., 202T] enforce augmentation invariance along with a constraint that the global covariance matrix is the identity; SimSiam [Chen and He 202T) and BYOL [Grill et al. 2020) employ architectural constraints that regularize towards uniform representations and simply optimize for transformation invariance. Other studies have formalized the problem in terms of maximizing information [Ozsoy et al.) 2022 or capacity (Yerxa et al.) [024]) subject to an invariance constraint, which has enabled connections to normative theories of coding efficiency and
manifold capacity (Barlow et al. 1961 Chung et al.) 2018).
To formalize the definition of iSSL, we denote dataset of images (c.g.. ImageNet) by X E NxD where N is the number of images and D is their dimensionality (number of pixels). Let 7(: RD - _ be function parameterized by p that maps images images (for example, for T random crop operation, P specifies the region to be cropped). The goal of iSSL algorithms is to learn the parameters W of some function f-; W) : R
Rd such that the variability over p is minimal while preserving variability over X (to avoid trivial solutions such as f(: W) = 0 for all inputs). Many methods achieve this by observing pairs of randomly augmented views of a batch of images: X r(X; ). X = P2X; with P1, ~ p(p) where p(p) is pre-chosen probability distribution over augmentation parameters. Generally iSSL frameworks employ an objective function that operates on the outputs of f, A f(xx; W), BB = f(x* B, W). One popular framework is

---

Les SSL
View
7(:0"))
Half
Halve
Half
Orginal
Wew!
Yiew
Los-ssi
LisSt
LiSSL
Sunoe
T(:p0)) 1 Refi
1
Seno
View
Figure I: Diagram of the proposed training method. At the beginning of each training epoch the dataset is randomly split into two non-overlapping halves. Left Gray Panel: corresponding images in each subset augmented using the same set of two random transformations (so the total number of random transformations is halved relative to a standard iSSL training scheme). Every view is passed through a representation network f (ResNet-50 in this work) and the outputs projected into two embedding spaces by different projector networks, 9inz and Jequi- In the invariant embedding space a standard iSSL loss is applied, while in the equivariant embedding space the same iSSL loss is applied to the difference vectors between transformation -positive pairs (visualized on the right).
"Barlow Twins" " Zbontar et al. 2021]. which uses the objective: LBT = C, (C,)))
where C is the cross-correlation matrix between 7 and 7 The first term encourages the outputs in response to the same image subject different augmentations to be correlated, while the second encourages the outputs in response to distinct images to uncorrelated.
Because complete invariance to the transformations employed in iSSL is harmful for downstream tasks, most frameworks employ a learnable "projector network" that maps the outputs the rep- resentation network to an embedding space before applying the loss. The nearly ubiquitous use of this "guillotine regularization" [Bordes et al., 2022]. means that most iSSL. methods aim learn function from which augmentation invariant subypace can be extracted. While this approach does permit some rransfrrmation-rellaedd variability in the representation, there is no explicit control or encouragement of that variability, and no incentive for that variability to be usefully structured.
2.2 Contrastive-Equivariin Self-Supervieed Learning (CE-SSL)
To induce structured variability in learned representations, we require that an equivariant subspace can be extracted from f alongside the invariant subspace described above. A function is equivariant set of input transformations if there exists corresponding induce the same changes. In the self-supervised learning setting this property can be expressed as:
of output transformations that
VTP E P. Vx E X, नD f(T,(=)) = (x))(x))
(1)
where Tp =((pp) and P is the set of possible values of transformation parameters. Note that invariance is special case of equivariance, avoid this degenerate solution, we will require both that similarly transformed inputs be related by the same transformation in the output space, and that differently transformed inputs are related to each other by different transformations.
which T, is an identity transformation for all p. To
Our training methodlogy, summarized in Fig. 1 follows the principle first proposed by [Gupta et al. 2023]: "Equivariance should be learned from pairs of data, as in invariant contrastive learning. First we split our dataset images into two random non-overlapping equal-sized partitions X,, X₂ R x Next we apply randomly selected augmentation to both X, and X2, so that corresponding contain distinct images that have been subjected to the same augmentations.
of x A/B and X A/B

---

Note that this reduces the total number of random samples of by factor of two relative to standard iSSL methods. Finally the resulting representation vectors are each fed through two distinct projector networks, A/BB Fewe (T A/B) and , Gequi (r; A/BB These two embeddings are optimized to be invariant transformations and discriminative across base images, invariant base images and discriminative across transformations, respectively. The overall objective (loss) functions is:
Loverall = (1 A)s + >CCE-SSL
where LISSL [[ 1= 11,,
(2)
and CCE-SSL. L( - -
where both terms are written terms of self-supervieed learning loss function that encourages invariance and uniformity [Wang and Isola] 2020) (e.g., LBT) and À is hyperparameter that de- termines the relative importance of extracting an invariant or equivariant subspace from the shared representation. In the notation of Eq. Q by designing LCE-SSL to encourage similar transfor- mations to induce similar displacements output transformations are of the form (=)=) + Zp. Thus we leverage the principles underpinning contrastive invariance learning to encourage representations that contain useful transformation-related information; this choice differentiates this formulation from previous equivariant self-supervised learning approaches.
the output space, we are implicitly specifying that our
3 Results
3.1 Implementation Details
ResNet-50 architecture (He et af., (0066] as the backbone representation network f. Our training scheme is compatible with any choice of iSSL framework, as specified by the choice of LissL. We experimented with three
Architecture and invariance ojective. For all experiments
different base methods chosen to span the range from "instance contrastive" to "dimension contrastive"
2023a : SimCLR [Chen al. 2020], MMCR (Yerxa al. [024]. and Barlow Twins Zbontar et al. 2021]. In each case, we define Jine using the projector network architecture proposed in the original work. To retain the synergy between the normalization scheme, loss function, and projector architecture achieved by each framework we use the same architecture for both gene and
Garrido al.
Gegui-
Pretraining dataset and augmentations. We train using the ImageNet- Ik dataset and the standard set of augmentations first introduced in (Grill et al.) 2020), which includes random resized cropping, color jittering, Gaussian blurring, solarization, and horizontal flips. See Appendix A.2 for exact training details.
evariance-oquivaaiinne tradeoff. For each of the three choices of LiSSI. we trained networks with hyperparameter values X E {0.0, 0.001,0.1,00 0.2, 0.3, 0.4, 0.5). yielding total of 21 learned representations (note: À == corresponds to standard iSSL). We found that classification performance becomes severely degraded for values of larger than 0.5 (see Appendix A.4).
3.2 Representational Analyses
Bures metric comparisons. We conducted series of experiments to determine the extent to which various sources of variability in our dataset were meaningfully organized. The experiments utilized the Bures metric, which is the Wasserstein ("Earth Mover's") distance between mean-centered Gaussian distributions with covariance matrices C₁ and Cz:
DB(C1, C2) trace (c, + 2 (Ch (
When C and C2 are normalized to have a trace of 1, the maximal distance of 2.0 oceurs when the variabilities lie in orthogonal subspaces (or are completely "factorized" from each other) and the minimum distance of 0.0 occurs when the covariances are equal. More generally, large Bures distance indicates two sources of variability are factorized from each other and low distance

---

Aemmentation-Aggmennnn
Alignment
Augmentation Contrill Aligmment
Spatial-Photonee Photometric Aligoment
Class Class Aligmment
Regg ession
Higher Aigrmentt Algnment
Avenaee
Gross
Over Photematic
Single Photometis,
Average Cuy Dreps
W
Controd Righmer
Diaa-Clan Alg-Cla Algament
Rugehug Abg TE
0.6
0.00
Spaigs Ptt Protommtii
-0.025
0.19
0.70
0.10
00
0.09
0.60
0.075
0.05
0.55
MMCR Barlow SmCLR
-0.0
-0 100
-0.08
0.45
074
0.4
00
0%
0.0
Figure 2: Effects of equivariance on representational geometry. A: Alignment between augmentation manifolds (gray ellipsoids). B: Alignment between augmentation manifolds (gray ellipsoids) and the centroid manifold (blue disk). C: Alignment between spatial and photometric manifolds. Gray ellipsoids represent single augmentation manifolds, and blue/yellow points indicate the mean over the outputs from many transformations of single view obtained via a photometric/spatial transformation, respectively. Expected distance is larger when the two sources of variability are factorized. D: Same A., but for class manifolds. E: A schematic of the parameter regression experiment. In each panel, the bottom row depicts the results of each analysis described in the text of Sections 3.2 and 3..2] Shaded regions indicate 95% confidence intervals (estimated distance is estimated over for A-D and over 5 independent runs of the regression experiments for E). A summary of the sources of variability used to compute C₁ and C2, and the ensemble used to estimate the expected alignment is measured can be found in Table A.5
the same comparisons the expected
indicates shared structure. We first estimate the trace-normalized covariance of the outputs some network f over two sources of variability and compute the Bures metric between the two.
Because we are mainly interested the impact of the equivariance loss relative the invariant baseline, each analysis below is carefully controlled to expose any structural differences. In particular, we estimate Ci and C2 over identical inputs for an invariant network and an equivariant network trained using the same base objective but non-zero value of X. We then can directly compare the measured Bures distance for the invariant network and each equivariant network (A * 0):
ADB = D (C( C2-
DD (CA+ C2*#"): this measure quantifies the amount of alignment between two sources of variability in an equivariant network relative the invariant network baseline. In each of the panels in the bottom row of Fig. we show how E A [DD] (denoted simply "Alignment") evolves function of À when Cj and C2 are estimated over different sources of variability (y- axis labels indicate the ensembles over which the variabilities were estimated). We show the joint distribution of (DB(CA C2=0), (B(CAA C2*(0)) and summarize the sources of variability in each experiment described below Appendix A.S
-agirentttioo--Auugmnntaatonn alignment. First determine the extent to which augmentation variability is shared across base images in each network (Fig. [A). For these experiments, both G and C2 are estimated over many random transformations of single images in the validation set (we will refer to the responses to such group an "augmentation manifold"). The expectation is then over randomly sampled pairs of augmentation manifolds. The positive expected difference of distance indicates the equivariant networks consistently produce lower distance between augmentation manifolds, indicating more shared augmentation variability across base images. This structure is closely related to what is encouraged by the equivariance loss term and the orderly increase function of suggests that we are optimizing effectively.
Augmentation-Centee factorization. We next investigate the extent to which variability over augmentations is factorized from variability over base images (Fig. (B). We use the "centroid
5

---

manifold," (Yerxa et al., 2024] to characterize the variability over base images, by measuring the covariance over base images of the means over augmentations. That is, for these experiments (for each network respectively) C1 is the covariance of the centroids of all augmentation manifolds and C2 is the covariance of randomly selected augmentation manifold. We observe that equivariant networks generally exhibit larger distance between centroid and augmentation manifolds indicating increased factorization (or lower alignment) of image-content variability and variability. This structure was not explicitly encouraged by the objective and can be considered an emergent property of the equivariant learning procedure.
Spatial- Phottme- factorization. Next we ask whether our equivariant training procedure in- duced increased factorization of variability to different types of input transformations (Fig. (C). The standard augmentation procedure involves first taking random crop (spatial variability) of a given image and then applying a series of pixel-level transformations (color-jittering blurring, etc.) (photometric variability). To assess the impact of these two distinet classes of image transformations we first chose 20 random crops a given image, then applied the same set of 20 random photometric transformations each individual crop. yielding 400 different views of each base image. C and C2 are then estimated over network responses that are averaged over different crops or different photometric transformations respectively, and the expectation is taken over different (single) base images. We observe the equivariant networks consistently exhibit increased factorization (i.c. larger Bures distances relative the invariant trained network). This again is an emergent property of the equivariant learning procedure, and is particularly interesting in light of recent work that discovered that this form of transformation-fee factriizzaatooiiiiionnnii is more correlated with neural predictivity than transformation invariance [Lindsey and Issa 2024].
gaussian
Class- Class factorization. Finally we asked whether within-class variability was more or less shared between distinct classes in equivariant networks by estimating C1 and C2 over responses to all images in distinct classes in the validation set (the expectation then taken over different random pairs of classes) (Fig. (p). Increased sharing of variability between class manifolds has been demonstrated to increase manifold capacity, and can make representations better suited for multi-task evaluations [Wakhloo et al.) 2023 2024]. We observe higher alingment (lower expected pairwise Bures distances) in the equivariant networks indicating that the "class manifolds" relative to the invariant networks.
Linear embedding of augmentation--eeater information. While the above experiments demon- strate that equivariant training induces increased alignment of transformation--elateed variability between images, this does not necessarily imply that this variability is coherently organized. To assess this more directly, measure the extent to which augmentation parameters can be linearly decoded from the networks' representations. Specificlly, we regress the concatenated outputs of a clean and transformed image onto the parameters of the applied augmentation. We report the resulting coefficient of determination (Re) on a heldout set of validation images (Fig. Z.. The equivariant training is seen increase the amount of linearly accessible augmentation information relative to invariant training (the leftmost points plotted in Fig 2). We further analyzed a set of equivariant models trained with weaker augmentation parameters (see|A.6ffor details). In these networks, we again observe that equivariant training increased the amount of linearly accessible augmentation information compared to invariant training. This holds not only for augmentation parameters within the training range (left panel ? but also for parameter values beyond the training range (right panel 7 Thus, the equivariance properties of the models generalize beyond the training distribution. Future work could examine generalization to unseen types of augmentations.
3.3 Neural Predictivity
We utilized the BrainScore evaluation pipeline Schrimpl et al. 2018 to measure the extent which each learned representation can linearly predict neural responses measured in macaque area IT,, for four different experimental datasets. At the time of testing, our highest performing model (Barlow Twins objective, À = 0.2) had the 10th highest average predictivity for area IT out of approximately 250 publicly available models on the Brain-Score leaderboard. Across a reasonably large range of values of X, the equivariant models improved the neural predictivity relative to the invariant baseline (A 0) for all four datasets (Fig. 3). Many previous publications have noted that changes in training objective function have small effect neural predictivity, relative to other factors

---

0.0
0.001
0.4
MMCR
Barlow
SimCLR
0.55
0.55
0.55
Neural Prediclivity
0.50
0.50
0.50
0.45
0.45
0.45
0.40
0.40
0.35
0.35
MajajHeng
Sargharjazeii
Sangiawi
SanghaslMorty
MajehHong
Sangharijozei
Sanghavi
SanghavilMury
MajaHHang
Sangharijozeit
Sanghavi
SanghavilMrty
Figure 3: Brain-Score (noise-ceiled predictivity evaluated via ridge regression) for each value of X (different colored bars) for each IT dataset (groups of columns) and base objective functions (different figure panels), For all datasets and base objectives the invariant network (A O, lightest bars) is outperformed by at least one equivariant network, and the spread in predictivity over values of is significantly larger than the spread in predictivity over base objective functions for invariant networks.
such as training dataset (Tuckute et al. 2022, Conwell et al.) 2023 Yerxa et al. 2024). In contrast, encouraging equivariance produced much larger gains than choosing between different base invariant objectives: the range of predictivities over the sweep of X was around 4 times larger than the range of predictivities over objective functions for the invariant baseline. We further contextualize the scale of predictivity improvements in Fig by comparing models to all public submissions on the BrainScore leaderboard; our equivariant training procedure improves performance of the already-strong invariant models to nearly sfute-of-thh-art levels of IT predictivity. By training the most predictive model for 1000 epochs (rather than 100), we achieved 0.5355 mean fraction of explained variance, which makes this the top IT brain prediction model. To ensure that the observed alignment increases not architecture specific, we trained a smaller set of models using different backbone architectures and observed similar trends when using both smaller and larger networks (see Appendix AA.7 for details).
We quantified the correlation between our various representational measurements and the neural predictivity for each of the four electrophysiology datasets in Table [ We observed that the only representational metric with a correlation greater than 0.4 across all four neural datasets was the Peatial-Photmmettre distance, which is the metric most closely related to the factorization score described in [Lindsey and Issa 2024]. While this previous study described a correlation between structured variability and neural predictivity measured from large set of pre-trained models, our results demonstrate that explicitly encouraging such structures can improve alignment between artificial and biological representations. In addition to the previously described representational measurements, we also looked at the linear decoding of the hue modulation parameter in isolation. Hue modulation is one of 12 augmentation parameters that are linearly decoded in the parameter regression measures described in Section 3.2 We observed a strong correlation between neural predictivity and hue modulation, particularly with the Sanghavi-Jozwik dataset, which is the only response dataset that included color image stimuli (last column of Table T.
3.4 Transfer Learning
Several previous studies that aim to reduce augmentation-ineen -iwraiaiieninnniiinieeeeeeeeeeee of self-supervised features have reported that the resulting representations generalize better to out-of-distribution classification datasets (Gupta et al. [2023] Xiao al.) 2020 Suau et al. 2023 Chavhan al. [002]. However most of these studies focused on using the smaller ImageNet- I 00 dataset for training. in one case reporting that the transfer learning gains diminish or disappear when using ImageNet- 1k [Chavhan et al. 2022]. We tested our set of networks on 6 different downstream tasks and found limited evidence that the equivariant features confer an advantage in terms of out-r--distributtin generalization when training on a sufficiently large and diverse dataset (see Table 2). To address this discrepancy with the literature we conducted additional experiments on networks trained using the ImageNet- 100 dataset, and in this case observed improvement in generalization to diverse downstream tasks.
7

---

MajajHong
Sanghavijozwik
Sanghavi
SanghaviMurty
Mean
150 opochs
4-D2:
40
Count
Count
Count
12
A= D2
spacta
72-
15-
1/
04 Noural Predictivity
Naural Predictivity
Neural Predictivity
Neural Predictivity
Noural Prodictivity
Figure 4: Histogram of neural predictivity scores for the 249 models on the public Brain-Score leaderboard at the time of testing, for each of the four considered IT datasets, as well as the mean over the four datasets, In each plot the vertical green line shows the score the invariant Barlow Twins model, the red line shows the score for the equivariant Barlow Twins model with À = 0.2, and the blue line shows a new model trained for 1000 epochs, also using Barlow Twins for the base loss and À 0.2.
Table Absolute values of Pearson correlation coefficients (Rt²) between various representational measurements and the neural predictivity across each of the four IT datasets. The correlation was measured over each value of À and base objective function for a total of 21 networks. Each column corresponds to panel in Fig. 2 except for huc, which is the regression score obtained for the random hue modulation parameter in isolation.
Neural Dataset
Aug- Aug
Aug- Centroid
Spatial- Photometric
Class- Class
Param Regression
Hue Regression
Majaj-Hong
0.03
0.02
0.42
0.10
0.38
0.28
Sanghavi-Jozwit
0.86
0.7
0.83
0.77
0.91
0.91
Sanghavi 0.84
0.84
0.68
0.63
0.85
0.86
Sanghavi-Murty 0.33
0.41
0.42
0.14
0.56
0.48
is also worth noting that CE-SSL trained networks do not outperform their invariant counterparts on in-distribution generalization (see A.2. I and Fig. (5. This is not surprising in light of the fact that the suite of augmentations and architectures employed in SSL. have been some sense optimized by the community in order to improve performance on this task (by aligning the transformation invariance task with the standard in-distribution classification task). However, for outtof-distribution classification tasks where the task-alignment is worse, the equivariance task could mitigate this mis- match. A concrete example is the Flowers- 102 dataset, where the color of petals is much stronger predictor of class than color is in, say, the ImageNet-1k lk dataset (so the color insensitivity induced by the standard augmentations could be detrimental). For this dataset we do see marginal improvements, but note that the improvements (ImageNet- 100). There are at least possible explanations for this: (1) for ImageNet- 1k pretraining the performance of the networks is already quite high, and the task is saturated, or (2) there is more fundamental reason that the improvements transfer learning induced by equivariance decrease as the size and diversity of the pretraining dataset grows. Future work could explicitly disambiguate between these hypotheses to determine why the benefits of transformation--elate variability for out of distribution generalization are outweighed by the gains of scaling the dataset. Furthermore this result shows that the increased neural predictivity we observe in ImageNet-Il lk trained networks cannot be explained by need perform better on variety of invariant-clasiifcatto tasks.
much more pronounced when pretraining
smaller datasets
4 Relationship to Existing Augmentation-Sensiiee SSL Methods
A key feature that differentiates our approach is that it encourages equivariant structure without explicit access to augmentation parameters. This is enabled by the "paired augmentation" data generation procedure, and to the best of our knowledge CARE [Gupta et al. 2023] is the only existing work that shares this feature. Our method has two advantages over CARE: (1) CARE the equivariance loss is applied in the same space as the invariance constraint, and because there are no "negative equivariant pairs" in the CARE framework, learning an invariant representation would perfectly satisfy the equivariant constraint; and (2) in CARE the standard augmentation pipeline is

---

Table 2: Frozen-Lineal Evaluation for invariant and equivariant trained networks on different downstream datasets: Cifar-10/100 Krizhevsky et al. 2009], Oxford-Pets Parkhi al.
(2012)
Describable Textures Database [Cippo et al. 2014]. Flowers- 102 NNilsback and Zisserman, 2008
and Food- 101 [Bossard al. 2014]. We closely follow the evaluation procedure from [Lee al.
2021] (see Appendix A for details) and report topl accuracy for each objective/aatase. In all cases we report the mean over runs of the evaluation procedure, we observed very little variability (maximum of 22,, over all evaluations, we report the standard deviation over runs in Appendix A.4 The equivariant networks are denoted by prepending "CE" before the objective and were trained using À 0.1. which enabled substantial amount of structure variability without significantly impacting frozen-linear classification on the SSL training dataset (see Appendix (A.2). For ImageNet- Ik trained networks out of distribution performance decreased for most evaluations, while for ImageNet- 100 trained networks performance was improved in 15 of 18 cases.
ImageNet-100 Training
Objective
Cifar-10
Cifar-100
Pets
DTD
Flowers-102
Food-101
MMCR
84.3
63.3
67.0
66.1
83.1
60.9
Barlow
87.7
68.7
74.8
67.0
88.3
63.4
SimCLR
87.8
68.8
74.3
66.6
88.5
64.8
CE-MMCR
87.3
69.4
68.9
65.7
87.5
64.1
CE-Barlow
88.0
69.1
73.6
67.3
89.5
65.5
CE-SimCLR
87.9
68.2
72.6
67.5
88.6
65.2
ImageNes-lk IK Training
Objective
Cifar- 10
Cifar-100
Pets
DTD
Flowers-
Food- 101
MMCR
92.2
76.9
85.3
75.4
93.9
73.8
Barlow
91.8
75.8
86.5
73.0
93.8
72.2
SimCLR
91.8
74.7
85.1
74.5
92.7
70.5
CE-MMCR
92.2
76.6
84.3
75.7
94.0
73.8
CE-Barlow
91.8
75.3
85.7
75.6
94.2
73.0
CE-SimCLR
91.0
73.6
82.3
73.9
92.2
70.5
used to optimize the base LISSL loss and paired augmentation are used in parallel to optimize the cquivariance term (so an increased number of passes through the network necessary relative to standard training).
In EquiMod (Devillers and Lefort 2023 the projector network is conditioned the augmentation parameters by appending the parameters to the output of f. The authors theorize that knowledge of the augmentation parameters could allow the projector network better extract an invariant subspace tailored to each transformation, thus allowing for more structured variability in the representation space. Alternatively, the projector network could simply ignore the augmentation parameters,
resulting in a structure that is identical to invariant SSL.. In practice [Garrido et al.) 2023b) have
found this be the case. Split-Invariant Eqqivaarn- (SIE) and Amortised Invariance (AI) learning Garrido et al. (222b) each improve on this principle by using a hypernetwork approach: separate network takes inputs the augmentation parameters and outputs the parameters either both and respectively. While the collapse issue of EquiMod is avoided, this comes at the expense of significantly complicating the network computation, and in the case of AI introduces new parameters that need to be tuned for every downstream task (when augmentation information is not available). Still other methods supplement the standard invariant SSL loss with an auxillary term that involves predicting the parameters of the input transformation [Lee et al.) 2021 Dangovski et al.) 2021) The relationship of our method to these analogous the relationship of transformation-innare self-supervisee learning to supervised classification.

---

5 Discussion
We've developed a new self-supervised objective that explicitly encourages structured variability in networks, and demonstrated that it can produce increased alignment with responses of neurons in primate visual area IT. While we are not the first incorporate a notion of equivariance to self- supervised learning, our method improves on existing work in several ways: it require no extra passes through the network relative to invariance-based learning, encourages diversity in the representation of transformation-relatt does not rely on supervised access to transformation parameters. The parsimony of our approach (applying the same objective to both outputs of individual images and to displacements between similarly transformed images) allows our technique to be easily adapted to other settings such as temporal self- self-supervised learning (discussed below). Although in this work we focused on the visual domain, similar equivariant and invariant objectives could be investigated for other domains such as audio and langauge representation learning.
information by leveraging advances in invariance-baser learning, and it
Our approach induced several interesting features in the learned representations: transformation variability is shared across base images and factorized with respect to variability over base images, the variability induced by distinct types of transformations are factorized from each other, there is increased alignment between class manifolds, and transformation related information is linearly encoded. Some of these properties are closely related to the imposed objective and some are emergent. We also confirmed that several of these representational properties are correlated with increased neural predictivity. Future work can extend these correlative observations to better understand how increasing transformation sensitivity improves neural alignment. For example, one could analyze the residuals of predicted neural firing rates of distinct models to determine how "'vvelappiin"" the variance predicted by each is (or alternatively, attempt to fit the residual variance of one model with another). Such analyses are becoming more feasible with the collection and release of larger scale datasets of neural responses natural images (e.g., [Madan al., 2024)). ). We view this result as demonstrating the promise of incorporating knowledge gained from experimental observations and large scale comparative studies into optimization procedures to produce better models.
Although our experimnents reveal both induced and emergent benefits, the inclusion of an additive cquivariance term in the objective does lead to fewer guarantees regarding the learned structure. For example in schemes where the output transformations (Tp 's) are explicilly represented or learned, the resulting representation is "steerable" by default. It is of interest to investigate whether the output transformations could be reliably recovered from our learned representations. Additionally it would be interesting to consider other types of output transformations (CARE (Gupta et al.. 20023 focuses on orthogonal transformations, and in the case where output transformations are learned they can be computed with nonlinear neural networks).
Finally, it of interest to explore the use of more ecologically relevant sources of training data, e.g.,
by replacing synthetically transformed views of images with temporally adjacent frames of natural videos. This approach is particularly appealing from the perspective of biological plausibility, as the pairing of such training examples is readily available from natural visual experience. Several recent publications have shown that such a strategy can produce representations with competitive neural predictivity and performance on computer vision tasks (Zhuang al. 2021 Parthasarathy ett al., 2023] Venkataramanan et al., 2023). In this context, the typical invariance loss can be thought of as incentivizing representational slowness [Földiāk [1991] Wiskott and Sejnowski] [2002] The equivariance mechanism described in this work could be implemented by applying the same invariance-hasee loss function to the first temporal derivative of the responses, i.e. by encouraging the displacement between successive pairs of frames to be constant. Such a temporal semppral-equivaaiance objective would incentivize representational straightness, which has been used describe features of both human perception and neural activity in the ventral stream [Hénaff et al.] 2021 2019).
Straightness in artificial representations has been found to be correlated with both neural predictivity
and ad adversarial robustness [Lindsey and Issa 2024 Harrington et al. 2022 Niu et al. 2024 These
connections provide an array of promising research directions.
10

---

Acknowledgments and Disclosure of Funding
This work was funded by the Center for Computational Neuroscience at the Flatiron Institute of the Simons Foundation. S.C. is supported by the Slingenstein-Simon Award, Sloan Research Fellowship. NIH award R01DA059220, and the Samsung Advanced Institute of Technology (under the project "Next Generation Deep Learning: From Pattern Recognition to AI"). All experiments were performed on the Flatiron Institute's 's high- nigh-perffrmance computing cluster.
References
Daniel LK Yamins, Ha Hong. Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-opimmized hicrarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, 1 111(23):8619 8619-8624, 2014.
Martin Schrimpf, Jonas Kubilius, Michael Lee, N Apurva Ratan Murty, Robert Ajemian, and James J DiCarlo. Integrative benchmarking advance neurally mechanistic models of human intelligence. Neuron, 2020. URL https: //ww:. cell. cwm/ htwp//euuoo///m http://fuultttttteeeeeeeeeeeeeee /tw.//texxx//006677666663 (2))))66666663)))66)))))) 30605
Konstantin F. Willeke, Kelli Restivo, Katrin Franke, Ame F. Nix. Santiago A. Cadena, Tori Shinn, Cate Nealley, Gabrielle Rodriguez, Saumil Patel, Alexander S. Ecker, Fabian H. Sinz, and Andreas S. Tolias. Deep learning-driven characterization of single cell tuning in primate visual area V4 unveils topoloicac organization. Technical Report
bioRxiv, May 2023. URL
htttp:://di. 1101/2023. 0022..05... 12. 540591
Grace Lindsay. Convolutional neural networks as a model of the visual system: Past, present, and future. Journal cognitive neuroscience, 33(10):0017-2031, 2021.
Chengxu Zhuang, Siming Yan, Aran Nayebi, Martin Schrimpf, Michael Frank, James J DiCarlo, and Daniel LK Yamins. Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences, 11803):201419611 2021.
Talia Konkle and George A Alvarez. A self-supervised domain-general learning framework for human ventral stream representation. Nature communications, 13(1):491, 2022.
N Parthasarathy, O J Hénaff, and E P Simoncelli. Layerwise complexity-matchen learning yields an improved model of cortical area V2. Trans. Machine Learning Research, Jun 2024. URL https://openreviien 10L
Featured Certification.
James J DiCarlo and David D Cox. Untangling invariant object recognition. Trends in cognitive sciences, (1))3333-341) 2007.
Michael Kuoch, Chi-Ning Chou, Nikhil Parthasarathy, Joel Dapello, James J DiCarlo, Haim Sompolin- sky, and Sue Yeon Chung. Probing biological and artificial neural networks with task-dependent neural manifolds. In Conference on Parsimony and Learning, pages 395-418. PMLR, 2024.
Jenelle Feather, Guillaume Leclere, Aleksander Mądry, and Josh H McDermott. Model metamers reveal divergent invariances between biological and artificial neural networks. Nature Neuroscience, 26(11):2017-200 2034444,, 2023.
Ha Hong, Daniel LK Yamins, Najib Majaj, and James DiCarlo. Explicit information for category- orthogonal object properties increases along the ventral stream. Nature neuroscience, 19(4): 613-622, 2016.
Matteo Alleman, Jack Lindsey, and Stefano Fusi. Task structure and nonlinearity jointly deter- mine learned representational geometry. In The Twelfth International Conference on Learning Representations, 2024.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 1597-1607. PMLR, 2020.

---

Michael Gutmann and Aapo Hyvärinen. Noise- contrasite estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 297-304. JMLR Workshop and Conference Proceedings, 2010.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pages 12310-12320. PMLR, 2021.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Sariane-invarianee-ccvaaaan regulaiiziinnn for self-supervised learning. arXiv preprint 1rXiv:2105.04906 2021.
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self- supervised representation learning. In International Conference on Machine Learning. pages 3015-3024. PMLR, 2021.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758, 2021.
BastiBstien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supe learning. Advances in neural information processing systems, 33:21271-21288 2020.
Serdar Ozsoy, Shadi Hamdan, Sercan Ö Arik, Deniz Yuret, and Alper T Erdogan. supe-supe learning with an information maximization criterion. arXiv preprint arXiv:2209.07999 2022.
Thomas Yerxa, Yilun Kuang, Eero Simoncelli, and Sue Yeon Chung. Learning efficient coding of natural images with maximum manifold capacity representations. Advances in Neural Information Processing Systems, 36, 2024.
Horace B Barlow et al. Possible principles underlying the transformation of sensory messages. Sensory communication, 1(01), 1961.
Yue Chung, Daniel D Lee, and Haim Sompolinsky, Classification and geometry of general perceptual manifolds. Physical Review X, 8(3):031003, 2018.
Florian Bordes, Randall Balestriero. Quentin Garrido, Adrien Bardes, and Pascal Vincent. Guillotine regularization: Why removing layers is needed to improve generalization in self-supervised learning. arXiv preprint arXiv:2206. 13378, 2022.
Sharut Gupta, Joshua Robinson, Derek Lim, Soledad Villar, and Stefanie Jegelka. Structur- ing representation geometry with rotationally equivariant contrastive learning. arXiv preprint (tt.:23006139224. 2023.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. International Conference on Machine Learning, pages 9929-9939. PMLR, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann Leeun. On the dual- ity between contrastive and non-contrastive self- self-supervised learning. In ICLR 2023-Elevente International Conference on Learning Representations, 2023a.
Jack W Lindsey and Elias Issa. Factorized visual representations in the primate visual system and deep neural networks. eliife, 13, 2024.
Albert J Wakhloo, Tamara J Sussman, and Yeon Chung. Linear classification of neural manifolds with correlated variability. Physical Review Letters, 131(2))02330,, 2023.
12

---

Albert J Wakhloo, Will Slatton, and Sue Yeon Chung. Neural population geometry and optimal coding of tasks with shared latent structure. arXiv preprint arXiv:2402.1677. 2024.
Martin Schrimpf. Jonas Kubilius, Ha Hong, Najib J. Majaj, Rishi Rajalingham, Elias B. Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, Kailyn Schmidt, Daniel L. K. Yamins, and James J. DiCarlo. Brain-score: Which artificial neural network for object recognition is most brain-like? bioRxiv preprint. 2018. URL https:/tppsett http://wwwwbbbxii.o////// .rgg perg/content/00
1101/407007v2
Greta Tuckute, Jenelle Feather, Dana Boebinger, and Josh H McDermott. Many but not all deep neural network audio models capture brain responses and exhibit correspondence between model stages and brain regions. bioRxiv, pages 2022-09. 2022.
Colin Conwell, Jacob Prince, Kendrick N. Kay, George A. Alvarez, and Talia Konkle. What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines? hioRviv, 2023. doi: 10.1101/2022.03. 28.488868. URI.tttps: h/wp://wwwwbbbxwi. biorxiv org/ wwwtent/earlylyyeeeeeoeoa /atpaallyy//2//////// 07/011100000 /01/ 20222 03 28. 485868
Tete Xiao, Xiaolong Wang. Alexei A Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. In International Conference on Learning Representations, 2020.
Xavier Suau, Federico Danieli, T Anderson Keller, Arno Blaas, Chen Huang, Jason Ramapuram, Dan Busbridge, and Luca Zappella. Duet: 2d structured and approximately equivariant representations. In Proceedings of the 40th International Conference on Machine Learning, pages 2023.
Ruchika Chavhan, Jan Stuchmer, Calum Heggan, Mehrdad Yaghoobi, and Timothy Hospedales. Amortised invariance learning for contrastive self-supervision. In The Eleventh International Conference on Learning Representations, 2022.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs, 2012 IEEE conference computer vision and pallern recognition, pages 3498-3505. IEEE, 2012.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-3613 2014.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 - mining discriminative compo- nents with random forests. In European Conference on Computer Vision, 2014.
Hankook Lee, Kibok Lee, Kimin Lee, Honglak Lee, and Jinwoo Shin. Improving transferability of representations via augmentation-awwrre self-supervision. Advances in Neural Information Processing Systems, 34:17710-17722, 2021.
Alexandre Devillers and Mathieu Lefort. Equimod: An equivariance module to improve visual instance discrimination. In International Conference on Learning Representations, 2023.
Quentin Garrido, Laurent Najman, and Yann Lecun. Self-supervised learning of split invariant equivariant representations. arXiv preprint arXiv:2302.10283 2023b.
Rumen Dangovski, Li Jing, Charlotte Loh, Seungwook Han, Akash Srivastava, Brian Cheung, Pulkit Agrawal, and Marin Soljačíć. Equivariant contrastive learning. arXiv preprint arXiv:2/11.0089 2021.
Spandan Madan, Will Xiao, Mingran Cao, Hanspeter Pfister, Margaret Livingstone, and Gabriel Kreiman. Benchmarking -iitt--iistrbuttion generalization capabilities of dnn-based encoding models for the ventral visual cortex. arXiv preprint 240.:1.:.06..6333 2024.
13

---

Nikhil Parthasarathy, SM Eslami, Joao Carreira, and Olivier Henaff. Self-supervised video pretraining yields robust and more human-aligned visual representations. Advances Neural Information Processing Systems, 36:65743-65766 2023.
Shashanka Venkataramanan, Mamshad Nayeem Rizve, João Carreira, Yuki M Asano, and Yannis Avrithis. Is imagenet worth 1 video? learning strong image encoders from long unlabelled video. arXiv preprint arXiv:2310.08556 2023.
Peter Földiák. Learning invariance from transformation sequences. Neural computation, 3(2):
194-200, 1991.
Laurenz Wiskott and Terrence J Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural computation, 14(4):715-770, 2002.
Olivier Hénaff, Yoon Bai, Julie Charlton, Ian Nauhaus, Eero Simoncelli, and Robbe LT Goris. Primary visual cortex straightens natural video trajectories. Nature communications, 12(1):5982, 2021.
Olivier Hénaff, Robbe LT Goris, and Eero P Simoncelli. Perceptual straightening of natural videos. Nature neuroscience, 22(())984499)))) 991, 2019.
Anne Harrington, Vasha DuTell, Ayush Tewari, Mark Hamilton, Simon Stent, Ruth Rosenholtz, and William T Freeman. Exploring perceptual straightness in learned visual representations. In The Eleventh International Conference Learning Representations, 2022.
Niu, C Savin, and E P Simoncelli. Learning predictable and robust neural representations by straightening image sequences. In Adv. Neural Information Processing (NeurIPS), volume 37, Vancouver, Dec 2024.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint Xiv: 1708. 03888, 2017.
14

---

A
Appendix
A.1
Reproducibility
All code used for pretraining, evaluation, and analyses, will be made available in a public github repository upon publication.
A.2 Additional Pretraining Details
Here we report some additional hyperparameters not included in the main text.
Optimization: For all experiments we trained for 100 epochs using a batch size 2048 and used the LARS optimizer You et al., 2017] with weight decay of le-6 and momentum of 0.9, Note that the LCE-SSL loss is evaluated on pairs of augmented views and thus had an effective batch size of 1024. We use base learning rate of 4.8 and a learning rate schedule consisting of linear warm-up for the first 10 epochs followed by cosine decay throughout training.
Projector Architectures Each trained network uses two projectors with matching architectures, For Barlow Twins we used the architecture proposed in the original work (Zbontar et al. 2021] (3 layer ML. with hidden layer and output layer widths of 8192). For MMCR we also used 3 layer MLP with 8192 hidden width but 512 output units (also in line with the original work (Yerxa et al.] (0244). For SimCLR used the same projector architecture described originally because subsequent work [Garrido et al. 2023a has found that SimCLR benefits from more expressive projector.
MMCR, which is larger than the MLP
A.2.1 ImageNet-1k
For Barlow Twins we set the ABT, which balances the on and off diagonal loss terms, hyperparameter to 5e 3. For SimCLR we used temperature of = 0.15.
A.2.2 ImageNet-100
Besides the change of dataset, the only hyperparameter change in this setting is that increased the number pretraining epochs from 100 to 200 to be more in line with previous work.
A.3
Online-L.inear Evaluation for the Pretraining Dataset
Because frozen-linear evaluation on large datasets is computationally intensive we instead opt for online-linear classification. During pretraining the representation network outputs are detached from the gradient propogation graph and fed through a linear layer that optimized with the standard supervised cross entropy loss. Previous work has shown that online-evaluation is very strongly correlated with frozen-linear evaluation and incurs only minimal cost on top of self-super pretraining. We report the accuracies for ImageNet- Ik trained networks in Fig. and the smaller set of ImageNet- 100 trained models Table 3
Model
Accuracy
MMCR
79.0%
Barlow
81.4%
SimCLR
83.5%
CE-MMCR 79.6%
CE-Barlow 81.6%
CE-SimCLR 82.5%
Table 3: In distribution accuracy on the validation set of ImageNet-100 evaluated by online- classification.
15

---

ImageNet-1k Classification
69
MMCR
Barlow
68
SimCLR
67-
Accuracy
66
65-
64-
63
62
0.2
0.3
0.4
A
Figure 5: In distribution accuracy on the validation set of ImageNet-Ik evaluated by online- linear classification, for each objective and function of X.
A.4 Transfer Learning Evaluation Procedure
We closely follow the evaluation procedure from Lee et al. 2021], we repeat the details here for completeness. First images are resized such that the shortest edge is 224 pixels, then center cropped to 224x224 resolution. Then features are extracted from train, validation, and test splits of each dataset. L-BFGS is used optimize the standard cross entropy loss with L2 regularization, the value of the ridge parameter swept over selected via performance on the validation set. Subsequently the linear classifier is retrained using both the train and validation sets, and report the final accuracy on the held out test set.
This classification procedure was run 5 times with different random initializations, we reported the mean performance 3.4 and report the standard deviation over runs below.
Table 4: Standard deviation of top accuracies 5 independent runs of the transfer learning evaluation procedure.
ImageNer-100 Training
Objective
Cifar- 10
Cifar-100
Pets
DTD
Flowers- 102
Food-101 101
MMCR
le-2
7e-3
4e-2
le-5
7e-2
2e-1
Barlow
le-4
2e-4
2e-2
3e-1
3e-2
9e-3
SimCLR
le-2
7e-3
le-2
4e-2
7e-2
5e-3
CE-MMCR
le-2
2e-2
4e-2
3e-2
le-1
1e-2
CE-Barlow
5e-2
2e-2
3e-2
3e-2
8e-2
le-2
CE-SimCLR
Se-3
2c-2
2c-2
2e-2
5e-2
le-2
ImageNet-Ik Ik Training
Objective
Cifar-10 10
Cifar-100
Pets
DTD
Flowers-102
Food-101
MMCR
le-2
2e-2
8e-2
4e-2
2e-2
7e-3
Barlow
le-2
le-1
le-1
4e-2
3e-2
3e-3
SimCLR
2e-2
7e-3
1e-4
4e-2
2e-2
8e-3
CE-MMCR
9e-3
2e-2
2e-1
2e-1
6e-2
2e-2
CE-Barlow
le-5
le-2
9e-2
le-5
5e-2
9e-2
CE-SinCLR
le-2
le-2
2e-2
2e-2
7e-2
8e-3
16

---

0.001
0.1
0.3
0.5
1.8
Augmentation Distance
Distance
Spatial Photometric Distance
Cless
1.6
1.6
1.0
MMCR
1.4-
144
1.4
1.2
1.0
1.3
0.8
1.0
1.0
0.6
1.0
1.5
1.2
1.4
0.0
1.00
1.25
1.50
1.30
1.25
1.50
175
0g
0g*
0
1.8
1.0
Barlow
1.2
1.4
144
0.8
1.-
1.0-
0.6
1.75
0f
D/-2
0f
153
1.4
SimCLR
144
1.4
2.1.2
1.2
1.3-
1.0
1.2
12
0.8
0.0
0.6
1.3
1.0
1.5
1.2 1.4
1.0
1.5
1.00
1.25
1.50
1.75
02
Dg-
0j*0
06
Figure 6: Joint distributions of invariant and equivariant networks (increasing À increases the importance of the equivariant loss) for all of the Bures metric comparisons detailed in [3.2] The mean value curves in Fig. [ are generated by taking the mean of x for each of these plots, separately for each objective and value of A. The confidence intervals are estimated from the distribution of y values as well. Columns depict Bures distances between covariances estimated over different sources of variability, and columns index different base invariant objective functions.
A.5 Additional Details for Representational Analyses
Below we depict the joint distributions of the distances described 3.2 For each setting (unique objective function and value of À there are 800 unique measurements (i.e. points of unique hue on an individual plot). Meaning, for example, there are 800 random pairs of augmentation manifolds compared in both invariant representation space and equivariant representation space for each equivariant network in the left most column. summarize describe the sources of variability over which covariance matrices are estimated and the variables over which the expected Bures distance is calculated the experiments Fig. A-D.
A.6
-itt-f-diistibbution Equivariance
We aim test whether learning equivariances using weak augmentations induces structured variability in response to stronger augmentations (the extent to which learned equivariances generalize beyond the range of transformations seen during training). We trained models using the Barlow Twins objective and the same sweep over values of À using "weak" augmentations of (1) double the minimum crop size, (2) half the maximum value of color jittering, and (3) half the maximum size of Gaussian blurring kernel. We then repeat the parameter decoding experiments from the main paper Fig. E on these weak augmentations (left panel Fig. 7 and on the non-overlapping part of parameter space between the weak and strong augmentation distributions, i.e only for augmentations whose parameters are in distribution for the models trained as in the main text but out of distribution for the new models trained using weaker augmentations (right panel Fig. 7 In the first panel,
17

---

Table 5: Table defining the sources of variability producing covariance matrices and the random variables that the expected Bures distance is computed over for the experiments in Fig. 2
Column
Source of Variabil- Source of Variabil- ity C2
Ensemble for
ity C1
timating expecta- tion
Mugmentation-AAgmeettaiin Distance (A)
Augmentations of single image.
Augmentations of single image.
Random pairs of distinct images
Augmentation-Cenitn Distance (B)
Augmentation man- ifold centroids over all images
Augmentations of single image.
Random images (those used to compute C2)
Spatial-Photomerne Distance (C)
Photometric aug-
Random crops (of a single image) af- ter averaging many photometric augmentations.
Unique base im- ages.
mentations (of single image) after averaging over many random crops.
Class-Class Distance (D)
(Unaugmented) ex- emplars from one class
(Unaugmented) ex- emplars from one class
Random pairs of distinct classes
0.7
0.71
Thaiting Augmentation Strong
0.4
0.4
0.0
0.2
0.6
0.2
Augmentation parameter decoding performance on held-out test images for networks trained using either strong weak transformations. In the left panel we plot the decoding performance for weak transformations only, and the right panel the performance for strong transformations only (which are not seen by the reak-trained networks during pretraining).
Figure 7:
we can see that the best parameter decoding performance occurs when the pretraining distribution of transformations matched the evaluation transformations (i.e. weak-trained models slightly outperform the strong-trained models at decoding the parameters of weak transformations) In the right panel we see that models trained with strong augmentations have higher decoding performance, but models trained only on weak augmentations still demonstrate significantly increased ability to linearly decode strong augmentation parameters relative the invariant trained models (A models), indicating degree of generalization in the learned equivariances.
A.7 Different Backbone Architectures
To verify that the effect of equivariance neural predictivity observed in the main text is not limited
to specific choice of architecture, we trained invariant (A 0.0) and equivariant (A E [0.1, 0.2)) networks using LBT with smaller (ResNet-34) and larger (ResNet-50) backbones. As shown in Fig. we observe the same trend across different architectures: contrastive-equivariaat training increases alignment to area IT measured by linear predictivity.
18

---

MMCR, ResNet34
Barlow, ResNet101
0.55
0.55
Neural Predictulty
0.50
0.50-
0.45
0.45
0.40
0.40
0.35
MajsjHong
Sanghavijozelk vilzaillliikk
Sanghavi
Sanghavilfury
0.35
MajsjHong
Sanghavijzze Sanghavd
SanghaviMurty
Figure 8: Neural predictivity results for models trained using the MMCR base loss and ResNet-34 backbone (left panel), and the Barlow Twins base loss and a ResNet- 101 backbone (left panel). We similar trends in terms of increased neural predictivity for equivariant trained models as observed using the ResNet-50 backbone in the main text.
A.8 Compute Resources
All pretraining runs used 8 A100 Nvidia GPUs with 40GB of memory each. In our setting pretraining run times were around 15 hours, and we note that CE-SSL training generally increased training time by approximately 10% relative to standard self-supervised training. Subsequent evaluations ran on a single A100.
A.9 Limitations
We discuss some limitations not addressed in the discussion here. Our current training setup requires selection of the hyperparameter À to balance between the equivariant and invariant loss functions. Future work could investigate methods to balance the two losses without explicitly training an individual network for each choice of X. It is worth noting that some invariant SSL methods may be more or less sensitive to this additional hyperparameter. For example, some of the non non-monotonicity of SimCLR curves in Fig. 2 may suggest that SimCLR representations are more difficult to smoothly shape within the CE-SSL paradigm.
Additionally, computational limitations prevent us from doing an extensive architecture search over the two projector networks employed in contrastive equivariant training. In addition varying the depth and width of each projector, it would be of interest to "split" the representation space and have each projector operate on a subset of dimensions as input (as SIE [Garrido al., (0236)). Addi- tonally computational limitations prevented us from extensively evaluating the variability in neural predictivvtty over independent runs of contrastive equivariant training (the BrainScore framework recently stopped providing estimates of the eтrоr of neural predicitivity, but in previous studies the reported error for IT predicitivity was Rs 3e-3, which is small relative to variability we observed across the parameter of interest X). Pilot experiments indicated to us that the variability over training runs was small relative to the variability over values of lambda (we trained two invariant models and two with À 0.1 to get rough estimate of this, in both cases kept the more predictive model for inclusion in all analyses that appear in this paper).
A.10 Broader Impacts
In this work we propose one strategy for inducing increased alignment between artificial and bio- logical visual representations. Better understanding the computational principles underlying visual representations has the potential to benefit the quality of computer vision applications, to offer insights into the structure of primate visual system, and to improve clinical treatment of disorders related to visual perception.
19

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The claims in the abstract and introduction are directly related to the each of the results in the results section of the paper
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
20

---

The limitations are discussed in an appropriately labelled section of the appendix.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-ten system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper only presents empirical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Each experiment is described in the main text and exact details needed to reproduce the results are described in relevant appendices. Furthermore, source code will be released upon publication.
21

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Data sources are clearly documented and source code will be released in the form of public repository upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ptaips.cc//uulic//uuu
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
22

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Most experiments described adequately in the main text and exhaustive details are given in relevant appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All experiments we conducted have clearly defined measures of significance. We relied on a prominent 3rd party software to perform neural predictivity measurements (BrainScore), and measurement of significance provided to us. We discuss this limitation the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Computing resources described in the appendix.
23

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compule required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Ethics.
No special circumstances required any deviation from the NeurIPS Code of
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The broader impacts of the work are described an appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
24

---

Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All creators of assets (in this paper, datasets) are properly cited and the terms of use properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the bataasets.ppprsstttt paperswi theode. com/datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets are released by the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
25

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: There are no participants involved in this study.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "cdqsQvuOsF", "reviewer": "Reviewer_aTov", "summary": "The paper introduces a novel framework, CE-SSL, to address the limitations of traditional self-supervised learning (SSL) objectives, which often result in overly invariant network representations, with a goal to improve the neuronal plausibility of resulting representations. The authors propose a method that incorporates structured variability in response to input transformations, aligning the representations more closely with known features of visual perception and neural computation. The CE-SSL framework converts standard invariant SSL losses into contrastive-equivariant versions, encouraging the preservation of aspects of the input transformation without supervised access to transformation parameters. The method was validated through representational analyses, neural predictivity evaluations using the BrainScore pipeline, and various downstream tasks, demonstrating improved alignment with neural responses and increased structured variability in the representations.", "questions": "How does the method perform with different backbone architectures other than ResNet-50? Have the authors considered testing with more recent architectures?\nHow robust is the method to changes in the choice of augmentations? Have the authors tested with different sets of augmentations?\nWhat are the potential applications of the CE-SSL framework in other domains beyond visual perception, such as audio or text processing?\nHave the authors considered integrating additional forms of self-supervision, such as clustering or reconstruction, to further enhance the learned representations?", "limitations": "The authors have adequately addressed the limitations of their method, including the computational overhead, sensitivity to hyperparameters, and limited out-of-distribution generalization. They have also provided constructive suggestions for future work to address these limitations, such as exploring temporal self-supervised learning and more ecologically relevant data sources. The authors' transparency in discussing these limitations is commendable and aligns with the best practices for responsible machine learning research.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Innovative Framework: The proposed CE-SSL framework is a novel approach that effectively addresses the problem of excess invariance in traditional SSL methods by incorporating structured variability, aligning better with biological visual systems.\nBiological Plausibility: The method is designed to align more closely with neural responses in the primate visual system, particularly the inferior temporal cortex, which is a significant advancement in the field of neural predictivity.\nComprehensive Validation: The authors thoroughly validate their method through multiple analyses, including representational analyses, neural predictivity evaluations using the BrainScore pipeline, and various downstream tasks.\nPractical Implementation: The method does not require supervised access to transformation parameters and involves minimal modifications to the training procedure, making it practical for implementation.\nDetailed Analysis: The paper provides a detailed analysis of the tradeoff between invariance and structured variability, offering valuable insights into the representational properties of the learned models.", "weaknesses": "Sensitivity to Hyperparameters: The performance of the method is sensitive to the choice of the hyperparameter λ, which controls the balance between invariant and equivariant loss terms, requiring careful tuning.\nLimited Generalization: The method showed limited improvement in generalization to out-of-distribution tasks when trained on large datasets like ImageNet-1k, which could limit its applicability in diverse real-world scenarios.\nLimited broader impact: The method improves biological plausibility of learned representations, but the broader impact to representation learning in AI/ML is not clear.\nPresentation: Some concepts are insufficiently described, for example the projection network architecture is not descibed in sufficient detail. Overall, the organization of the paper is a bit convoluted and more work is needed to streamline the narrative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review of our submission. We respond to the listed weaknesses and questions below:\n\n- Introducing a new hyperparameter to tune: we agree that this is a fundamental weakness of our framework. It would be preferable to balance this tradeoff without using two loss functions and a Lagrange multiplier. We will add the following to the limitations section (Line 524): \n\n    “Our current training setup requires selection of the hyperparameter $\\lambda$ to balance between the equivariant and invariant loss functions. Future work could investigate methods to balance the two losses without explicitly training an individual network for each choice of $\\lambda$.”  \n\n- Performance on OOD Classification tasks: Please see our general response for a more detailed explanation for why equivariance leads to little improvement in OOD classification performance (relative to invariant SSL) when trained on a sufficiently large and diverse set of natural images. While we agree that this in some sense limits the applicability of our method in AI/ML representation learning, we feel that this observation is in and of itself a contribution to the community. A similar effect was observed in some past work on Equivariant SSL (see Table 2 vs Table 3 in [1]), but we feel that it can only benefit the community to draw attention to the interplay between inductive biases (enforced via the loss) and the effect of dataset scaling on learned representations. \n\n- Other backbone architectures: We agree that it is important to confirm that the effects we report are not limited to a specific choice of architecture. In response to this concern, we trained a limited set of models using either ResNet-34 or ResNet-101 backbones in place of the ResNet-50 considered in the text. We found similar effects in terms of the neural predictivity of the equivariant networks relative to their invariant counterparts (see Figure R3 in the rebuttal pdf). We acknowledge that this is not a very radical departure from our initial architecture, but time prevented us from considering more modern backbones such as ResNext and ViT. We will endeavor to include experiments with these backbones in the final version of our paper, and will certainly include the experiments from Figure R3 of the Rebuttal PDF as a new Appendix.\n\n- Robustness to choice of augmentations: This is an interesting question to consider. To begin answering this we conducted experiments involving pretraining models using weaker versions of the same transformations (see general response and Figure R1 of the PDF for more details). Of course these experiments do not entirely answer the question of how robust our method and observations are to the choice of input transformations, though they do suggest a form of generalization (from representing weak-to-strong transformations) is possible. We feel the most important and ecologically relevant set of input transformations are those that mimic the visual experience of animals in the world, and as we note in the discussion we believe this is a very promising direction for future work!\n\n- Other input modalities: Great question - we have been thinking about how this technique might be applied to learning representations of audio signals! In this set-up we envision using a speech-in-background setting: where “positive” pairs would consist of the same speech signal with different choices of background noise. Networks can then be trained to be invariant to background noise, or equivariant to background noise using the method described in this work. While these experiments are beyond the scope of this paper, we plan on following up in this direction with subsequent work. We had not considered applying our method to text, though in view of our analogy between invariance/equivariance and slowness/straightness and the recent observations in [2], this may also be an interesting possibility. We will add the following sentence to the discussion line 296 to capture this idea: \n\n    “Although in this work we focused on the visual domain, similar equivariant and invariant objectives could be investigated for other domains such as audio and langauge representation learning.” \n\n- Other forms of Self-Supervision: In terms of clustering, we believe that existing methods that use clustering based contrastive losses (i.e. SwAV) would likely show similar results to the base objective functions considered in this work. A reconstruction loss would certainly also “fight-against” the invariance term, as it explicitly encourages that information about the input pixels be preserved. It would be interesting to see whether such a scheme also produced more predictive representations of IT, though this may be out of the scope of the current work. \n\n- Presentation: We will also add additional details on projector network architectures to Appendix A.2: Additional pretraining details. \n\n[1] Chavhan, Ruchika, et al. \"Amortised Invariance Learning for Contrastive Self-Supervision.\" The Eleventh International Conference on Learning Representations, 2023.\n\n[2] Hosseini, Eghbal, and Evelina Fedorenko. \"Large language models implicitly learn to straighten neural sentence trajectories to construct a predictive representation of natural language.\" Advances in Neural Information Processing Systems 36 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EqkoJNXxFP", "reviewer": "Reviewer_d1rT", "summary": "In their paper ‘Contrastive-Equivariant Self-Supervised Learning Improves Alignment with Primate Visual Area IT’ the authors construct a new kind of contrastive-equivariant loss to optimise ResNet-50 architectures. Their new loss is based on the idea of using both self-supervised learning via object representations invariant across transformations of an image, and equivariant representations of transformation across images. They show that their new loss results in improved alignment to brain data recorded in IT. In a series of additional investigations, they show which part of the achieved representations is most likely causing the alignment to data. They also show that their method shows similar generalization advantages like prior alignment techniques but that this improvement is actually easily compensated for by training on a larger dataset.", "questions": "Apart from the suggestions mentioned in weaknesses, I have some minor comments: \n-\tSome of the in-line references seem to be lacking the brackets in the proper locations, I noticed this in the references in line 86 and 227, where no overall bracket is around the reference but only around the year\n-\tIn line 161 & 162 authors say ‘for each equivariant network’ and I am somewhat confused why there is a plural here? Does this mean the projection relating to each specific transformation?\n-\tIn line 113 you mention a grey inset in Figure 2 but I am not sure what you are referring to with this?", "limitations": "The limitations discussed in the discussion and appendix seem appropriate.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "I believe that the new loss developed in this paper is a very interesting development, routed in ideas which have been discussed in the field. The paper’s descriptions of ideas and results are overall nicely written and give a very balanced account of the investigations. The improved performance on the IT matching score is interesting, especially in light of current debated about self-supervised learning in the brain.", "weaknesses": "I believe that the two weaknesses, one of experimental nature and one on the writing:\n\nExperimental – \nThe main goal of authors was to achieve higher alignment to brain data, but I found the comparison to other methods a bit thin. Authors say their method currently ranks 10th but it would be helpful to at least have a feeling for how far off the method is from 1st place. Many readers will not know whether the differences in the top 10 are in the magnitude of full percentage points or more / less. As such, I would suggest to perhaps at least provide the performance of the 11th and 9th, and 1st. This should be easy to address. \nI also wonder how ‘unique’ the achieved explainability of their method is, i.e. authors contrast their method with mostly task-optimised networks and would they expect that adding their method to task optimised networks would result in an overall best performant network or is the explainability captured by their method also the variance captured by task optimisation so that combining them does not promise any further improvements? I can see that from a biological plausibility perspective the newly proposed technique is still nicer, but if the classification loss achieves learning even more brain-like presentations which also contain a similar trade-off between invariance and equivariance, then perhaps there is multiple ways for the brain to learn such representations and the loss presented here would not achieve any unique advancements. \n\nWriting –\nI found the section ‘3.2 Representational Analyses’ somewhat difficult to follow – I appreciate that authors with Figure 2 tried to provide visualisations of manifolds for different scenarios but perhaps it would be more helpful to visualise which cases are actually compared for which distance? Perhaps this could be done in the form of table in the appendix, if otherwise authors run out of space? From the text and current figures I struggled to exactly get which images / augmentations etc go into which set to construct the manifolds. Also, I believe the line plots at the bottom of Figure 2 would be significantly easier to understand if authors would use something like ‘Strength of equivariance loss (Lambda)’.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive assessment of this submission. Below is our point-by-point response to the weaknesses and questions raised: \n- Lack of baselines/context for brain data: We agree that the contribution is clarified by providing more context for the predictivities for a range of models. See our general response Point 1 and the Figure R2 in the attached PDF for our plan to do so.\n\n- Uniqueness of Improvements: This is indeed a relevant consideration. We suspect that our method induces representations that are explaining disparate portions of the neural response variance from task-trained models for three reasons. . First,  self-supervised trained models provide neural predictivity that is on par with task trained networks (i.e. supervised recognition networks) [1, 2]. Second, above a certain threshold (in the regime that modern computer vision networks occupy) task performance (top-1 accuracy on ImageNet-1k) is actually negatively correlated with neural predictivity [3]. Finally it is worth noting that after retraining our most predictive model for an increased number of epochs, we now obtain SOTA performance on IT predictivity. However, it would certainly be interesting to more directly answer the question of how “shared” the explained variance is between disparate sets of models, for example by examining the structure of the residuals obtained in the Model-to-Brain regression problem [4]. We could begin to answer some of these questions using new and open source datasets of neural measurements in Macaque IT [5] and will highlight the importance of such investigations in the revised manuscript.\n\n- Writing in Section 3.2: We agree that these experiments could be described more clearly. The idea for a table that describes the sources of variability being compared in each panel is a good one and we will include an Appendix with additional details in the revised manuscript. We will also make the suggested updates to axis labels. \n\n- Thank you for pointing out these typographical issues! We will correct these during the revision process.\n\n- Confusion regarding lines 161-162: Thank you for pointing out the ambiguity in our word choice. There need not be a plural in this sentence, the intended meaning was that we repeated this pairwise comparison (between one invariant and one equivariant network) for each of the equivariant networks we trained (different base objective functions and different values of lambda). We propose to change this sentence to: \n\n     \"In particular, we estimate C1 and C2 over identical inputs for an invariant network and an equivariant network trained using the same base objective but a non-zero value of $\\lambda$.\"\n\n- Grey Inset: This was referring to the gray square in Figure 1 and is another typographical error. We apologize for the confusion and will correct this error. \n\n\n[1] Zhuang, Chengxu, et al. \"Unsupervised neural network models of the ventral visual stream.\" Proceedings of the National Academy of Sciences 118.3 (2021): e2014196118.\n\n[2] Conwell, Colin, et al. \"What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines?.\" BioRxiv (2022): 2022-03.\n\n[3] Schrimpf, Martin, et al. \"Brain-score: Which artificial neural network for object recognition is most brain-like?.\" BioRxiv (2018): 407007.\n\n[4] Canatar, Abdulkadir, et al. \"A spectral theory of neural prediction and alignment.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[5] Madan, Spandan, et al. \"Benchmarking Out-of-Distribution Generalization Capabilities of DNN-based Encoding Models for the Ventral Visual Cortex.\" arXiv preprint arXiv:2406.16935 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IbBIOCm0oW", "reviewer": "Reviewer_AoJ5", "summary": "Summary:\n\nThe authors propose a novel approach to self-supervised learning (SSL) by incorporating contrastive equivariant training into existing successful SSL methods based on creating invariance to input transformations. The authors observe that such an incorporation of equivariant contrastive learning produces the following downstream benefits: 1) Improved representational similarity to biological visual representations, and 2) Enhanced transfer learning performance on downstream tasks. Both these findings have been evaluated via experiments performed using the BrainScore benchmark and by computing linear probe classification accuracy of various SSL training methods on downstream image classification tasks.", "questions": "Please refer to my review above.", "limitations": "Limitations are discussed adequately in this submission.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Strengths:\n- The proposed contrastive equivariant SSL (CE-SSL) is a straightforward yet effective extension of existing invariance-based SSL methods. This enhancement has broad potential extending beyond this submission, as CE-SSL-trained encoders may excel in downstream tasks that rely on equivariance, such as image segmentation.\n- The experimental results demonstrate a significant improvement in two key areas: 1) Decoding performance of macaque IT neural recordings as validated by their model performance on BrainScore, and 2) Transfer learning to downstream image classification problems as validated by their linear probe classification experiments presented in Table 2. Evaluation performed broadly over 3 different SSL techniques over multiple $\\lambda$ parameters enhances the technical soundness of this work.\n- The authors provide a clear discussion of the proposed work's relationship to prior art in equivariant self-supervised learning, situating their contributions within the broader context of the field.", "weaknesses": "- Overall, the improvements from using CE-SSL in the BrainScore performance is only quite marginal, hence raising the question of how impactful the current work will be in the broader scheme of methods to enhance neural predictivity.\n- The writing quality of this work could be further improved. Particularly, Section 2 is currently written in a convoluted manner in my opinion and could be further improved to enhance readability. There are other minor writing issues, for e.g.:\n    - Lines 145-146 and equation 2 are in conflict with each other. Throughout the paper, it looks like $\\lambda=0$ refers to the invariant case. In that case, the invariant term of Eqn. 2 should be weighted by $(1-\\lambda)$. Is that correct? Currently the invariant term is weighted by $\\lambda$ in which case $\\lambda=0$ would correspond to training only with CE-SSL.\n    - Line 113, I believe the authors intended to refer to Figure 1 but they incorrectly refer to Figure 2 in the submission\n    - Citation issue in Line 86, Zbontar et al. [2021].\n    - It would help to use different icons to represent $f$ and $g_{inv}$ or  $g_{equi}$ as there are significant structural differences between the backbone and projector networks.\n\n- In Figure 4, as the CE-SSL loss contribution increases, the ImageNet classification accuracy seems to be dropping. This is an issue that needs to be explicitly discussed more clearly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful assessment of our paper. Find below is our point-by-point response to the weaknesses and questions raised: \n- Marginal Effect on Neural Predictivity: Please see our general response Point 1 and Figure R2 in the rebuttal pdf which directly address this concern about the marginal effect on IT predictions. Specifically, the changes observed are substantial in terms of the relative ranking compared to other models on this benchmark. \n- Writing/Clarity Issues: You are correct, there is a misleading typo in Eqn. 2 and the weightings of the two terms are reversed. Thank you for pointing out this mistake, we will correct it in the revised manuscript to  $L_{\\rm overall} =  (1 -  \\lambda) L_{iSSL} + \\lambda L_{CE-SSL}$. Similarly the reference on line 113 should indeed be to Figure 1. Finally, we will also correct the citation error on line 86 and change the icon for projector networks in Figure 1 to better indicate the transition from a convolutional backbone to an MLP projector. \n\n- Performance on ImageNet-1k Classification: Please see our general response Point 3, which provides reasoning for why downstream accuracy on ImageNet-1k degrades as we increase the importance of the equivariance loss. We agree that this issue could be better highlighted in the main text and propose to include language similar to that in our general response in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P5oGtQJEog", "reviewer": "Reviewer_Fzg3", "summary": "This paper presents an equivariant learning framework that modifies standard invariant-based self-supervised learning methods by integrating differences between object classes (Loss_iSSL) and the changes observed in images before and after identical transformations (Loss_CE-SSL). This adaptation enables the model to effectively incorporate transformation-related variability without necessitating additional transformation parameters or extensive modifications to the training protocol. The authors then demonstrate that this approach systematically enhances the model's capacity to predict neural activity in the inferotemporal (IT) cortex, revealing that optimizing for structured variability significantly boosts the accuracy of predicting cortical responses to natural images.", "questions": "1. The SimCLR curve in the lowest part of Figure 2 appears more jagged compared to the other two models. Could this indicate that SimCLR is less suitable for incorporating equivariance?\n2. When predicting IT neuron activity, how can we interpret the variations caused by different $\\lamda$ values across datasets?\n3. In the transfer learning results, why does the addition of an equivariance loss seem to have minimal impact on the outcomes?\n4. Is the equivariance learned by this method limited by the types and varieties of transformations applied to the images? For instance, if the model only learns transformations involving rotations between 0-30 degrees, can it represent equivariance for a 60-degree rotation? Furthermore, can the method effectively learn more complex, non-linear transformations, and can learning multiple transformations enable the model to represent combinations of these transformations equivariantly?\n5. There appears to be a typographical error in line 225: “($\\lamda$ = 0) for all four datasets (Fig. 3.3),” as the figure number seems incorrect.", "limitations": "Yes, the paper concludes with a discussion on some limitations of the model and directions for future research.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The article is clearly articulated and well-defined.\n2. It introduces an effective self-supervised objective function for implementing equivariance, which can be applied across various models including SimCLR, MMCR, and Barlow Twins.\n3. The paper conducts numerous experiments to analyze the impact of this training method on representations. It compares the relationship between different intensities of equivariance error (Loss_CE-SSL) and monkey IT neuron activities, as well as the model's performance in transfer learning tasks.", "weaknesses": "1. The model's performance in transfer learning tasks, as shown in Table 2, does not demonstrate a significant improvement; in many cases, it merely maintains accuracy levels similar to those before implementation.\n2. While the paper analyzes representations based on different values of $\\lamda$, these analyses seem to offer little help in predicting neuronal activity or improving transfer learning. The proposed \"tradeoff between invariance and structured variability\" does not appear to manifest significantly.\n3. The equivariance learned by the method may be limited by the types and varieties of transformations applied to the images, an aspect the paper does not analyze or discuss.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review of our submission. We respond to the questions and limitations point-by-point below:\n\n- Performance on Downstream Classification Tasks (W1, Q3): Please see our general response (Point 3) for our more detailed thoughts on why equivariant training on a significantly large dataset offers little in the way of improvements on downstream classification tasks. In short, our results on the ImageNet-100 training vs. ImageNet-1000 training suggest that the benefits induced by equivariant training “overlap” with those that come from using a large dataset (Line 252 of the submission). With this result, we aim to draw attention to the underappreciated fact that equivariant training seems to be more valuable when the pretraining dataset is smaller. \n\n- Lack of a tradeoff between invariant and equivariant terms (W2): We think there might have been some confusion surrounding our use of the word “tradeoff”; when we refer to a “tradeoff between invariance and structured variability” (Line 54) we are referencing qualities of the artificial representations, not necessarily proposing that IT neurons are explicitly striking some balance between invariance and equivariance. In the artificial representations such a tradeoff is clear, for example, as the importance of the equivariance loss increases the amount of linearly accessible information about transformations applied to the input (Figure 2E), but this comes at the cost of a modest decrease in performance on classification on the in-distribution ImageNet-1k dataset (Figure 4). Furthermore while the improvements in neural predictivity are small in absolute terms, this is common for linear-regression based comparisons, and our improvement does result in a substantial increase in terms of the relative ranking of models on the IT Brain–Score leaderboard (see our general response and Figure R2 for more details).\n\n    If we have misinterpreted this we are more than happy to engage further during the discussion period! We also will remove the reference to a “tradeoff” in line 58 with the following modification, to help avoid confusion: \n    “We explore the impact of including an equivariant loss for predicting neural activity in IT,”\n\n- Dependence on the input transformations (W3, Q4): We fully agree that this is an important point and could be better addressed. Thank you in particular for suggesting we experiment with testing how well the learned equivariances can generalize to unseen (stronger, but similarly typed) transformations. As a step towards answering this question we trained models using the same suite of augmentations but with a uniformly reduced strength, and repeated the parameter decoding experiments from Figure 2. These preliminary results suggest there is a degree of generalization to stronger transformations, see our general response and Figure R1 of the attached PDF for more details. We are currently working on the experiment you described involving rotations, and hope to have additional results to report during the discussion period.\n\n    Considering other more complicated sets of input transformations than those commonly used in SSL is also an interesting direction for further research that we can highlight more directly. We feel the most important and ecologically relevant set of input transformations are those that mimic the visual experience of animals in the world, and as we note in the discussion we believe this is a very promising direction for future work!\n\n- Is SimCLR less suited to incorporating equivariances (Q1): Our results do suggest that SimCLR may indeed be  less amenable to smoothly shaping the representation via an equivariance loss. However another  hypothesis is that these nonsmooth curves arise from the fact that SimCLR is in general more sensitive to various hyperparameter choices, and the one introduced by our equivariance loss is not an exception. For example, it is well known that SimCLR is more sensitive than other self-supervised learning methods to batch size [1]. To definitively determine whether this is a fundamental difference between SimCLR and the other objective functions would require large sweeps over a variety of hyperparamters (batch size, learning rate, etc.), and compute limitations prevent us from conducting such experiments. We will expand Appendix A.7: Limitations, to include this discussion. \n\n- Interpreting the impact of the equivariance loss on neural predictivity (Q2): We agree that gaining better understandings and intuitive interpretations of how different features of learned representations lead to different levels of neural predictivity is an important goal for both our work and the field at large. We attempt to draw conclusions by analyzing how the representational analyses from Section 3.2 correlate with neural predictivity in Table 1. However we agree that there is more work to be done in terms of interpretation. For example one could examining the structure of the residuals and mapping weights obtained in the Model-to-Brain regression problem [2]. We could also begin to answer some of these questions using new and open source datasets of neural measurements in Macaque IT [3] and will highlight the importance of such investigations in the revised manuscript.\n\n- Typos (Q5): Thank you for drawing our attention to this mistake, we will be sure to correct the figure reference in the revised manuscript.\n\n[1] Zbontar, Jure, et al. \"Barlow twins: Self-supervised learning via redundancy reduction.\" International conference on machine learning. PMLR, 2021.\n\n[2] Canatar, Abdulkadir, et al. \"A spectral theory of neural prediction and alignment.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[3] Madan, Spandan, et al. \"Benchmarking Out-of-Distribution Generalization Capabilities of DNN-based Encoding Models for the Ventral Visual Cortex.\" arXiv preprint arXiv:2406.16935 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for their careful consideration of our contributions and thoughtful questions. Several points were raised in multiple reviews and we respond to these below:
1. Lack of Context/marginal improvements on neural predictivity benchmarks: We agree that we provided insufficient context in order to evaluate the effect size of our equivariant training intervention on neural predictivity in IT. To better situate our models performances, we generated histograms that show the performances of all models on the Public Brain-Score leaderboard for each IT dataset considered (Figure R2 in PDF), which we will add to the main text of the paper for the final version. For Barlow Twins, our vanilla model ranked 50th overall and the strongest equivariant model (\lambda = 0.2) ranked 10th overall in terms of mean predicitivity. However, we only trained each ImageNet1000 model in our submission for 100 epochs to reduce computational costs of training many varieties of models (compared to 1000 epochs which the most performant SSL models generally use). We’ve now retrained our best model using twice as many (200) epochs, and the resulting equivariant Barlow Twins model is actually the #1 performing model. In addition to highlighting this via the included figure (Figure R2 in the PDF), we plan to include the following at Line 225 to reflect this:

    _“Additionally, by training the model that best predicts IT data (BarlowTwins, lambda=0.2) for a total of 200 epochs, compared with 100 
    for the other models, we achieved 0.5330 mean fraction of explained variance, which makes this the top IT brain prediction model.”_

2. Dependence on augmentation distribution: We chose to use the standard set of augmentations employed in modern self-supervised learning schemes as these are known to yield models that are performant on downstream tasks and predictive of neural responses, but we agree that considering the sensitivity of our method to augmentation choices is worthwhile. To investigate this with our framework, we trained a new set of models using uniformly weaker augmentations and the Barlow Twins objective (specifically, we doubled the minimum size of random crops, halved the maximum values of color jitter modulations, and halved the maximum std of gaussian blurring), and repeated the parameter decoding experiments from Figure 2E. The results are summarized in Figure R1 of the attached PDF. We find evidence that models trained on weak augmentations do still exhibit structured variability that generalizes to stronger input transformations. We will include this result in the Appendix, and the following at line 217:

    _"We further analyzed a set of equivariant models trained with weaker augmentation parameters (Figure R1) In these networks, we once again observe that equivariant training increased the amount of linearly accessible augmentation information compared to invariant training. This is the case not only for the augmentation parameters they were trained on (left panel) but also for parameters beyond the range of the training distribution (right panel). This suggests that the models represent some equivariance beyond the training distribution, and future work could further investigate the interaction between equivariant training and improved generalization to unseen types of augmentations."_

3. Performance on downstream classification tasks: We believe that multiple factors could contribute to our observation that encouraging equivariance during pretraining has little effect on OOD downstream performance for invariant classification tasks, and can actually harm performance on the in-distribution setting.  One important factor is the design of the augmentations used in this work (and widely across invariance based self-supervised learning methods). This particular set of transformations was refined by the community in order to increase the task alignment between the self-supervised learning and supervised object classification (i.e., the augmentation distribution was tweaked in order to maximize the downstream classification performance on ImageNet). In light of this we actually expected the in-distribution performance on classification to be reduced, and find it notable that in many settings the performance penalty is quite small (< 3%), while still inducing a large amount of new structure in the representation.

    However, for out-of-distribution classification tasks where the task-alignment is worse, the equivariance task could mitigate this mis-match. A concrete example is the Flowers-102 dataset, where the color of petals is a much stronger predictor of class than color is in, say, the ImageNet-1k dataset (so the color insensitivity induced by the standard augmentations could be detrimental). For this dataset we do see marginal improvements, but note that the improvements are much more pronounced when pretraining on smaller datasets (ImageNet-100). There are 2 possible explanations for this: (1) for ImageNet-1k pretraining the performance of the networks is already quite high, and the task is saturated, or (2) there is a more fundamental reason that the improvements in transfer learning induced by equivariance decrease as the size and diversity of the pretraining dataset grows. Our suspicion is that factor (1) is actually the dominant cause, but we certainly agree that disambiguating between these possibilities would be worthwhile. We plan to describe this issue in the Discussion section of the revised manuscript to better highlight these observations.

4. Overall presentation and typographical errors: Thank you for highlighting parts of the paper that were confusing or incorrect. The typos will be corrected for the final version of the manuscript, and we'll make another pass through the text to improve readability, especially for Section 2 describing the methods and section 3.2 describing the representational distance experiments.

Meta-review: This paper describes a novel SSL approach incorporating constraints for learning less invariant and more equivariant representations. 

The reviewers unanimously found the proposal novel and worthy. One main limitation of the work is the seemingly limited improvements in OOD classification compared to current methods. Even in the case of brain predictability, one could argue that the improvements are marginal. Nonetheless, the work appears to be carried well, and there is unanimous agreement that the approach is moving the field in the right direction.

Hence, the AC recommends the paper to be accepted.
Decision: Accept (poster)