Layerwise complexity-matched learning yields an improved model of cortical area V2
Authors: Nikhil Parthasarathy
Olivier J Henaff
Eero P Simoncelli
Venue: Accepted by TMLR ()

Abstract: Human ability to recognize complex visual patterns arises through transformations performed by successive areas in the ventral visual cortex. Deep neural networks trained endto-end for object recognition approach human capabilities, and offer the best descriptions to date of neural responses in the late stages of the hierarchy. But these networks provide a poor account of the early stages, compared to traditional hand-engineered models, or models optimized for coding efficiency or prediction. Moreover, the gradient backpropagation used in end-to-end learning is generally considered to be biologically implausible. Here, we overcome both of these limitations by developing a bottom-up self-supervised training methodology that operates independently on successive layers. Specifically, we maximize feature similarity between pairs of locally-deformed natural image patches, while decorrelating features across patches sampled from other images. Crucially, the deformation amplitudes are adjusted proportionally to receptive field sizes in each layer, thus matching the task complexity to the capacity at each stage of processing. In comparison with architecturematched versions of previous models, we demonstrate that our layerwise complexity-matched learning (LCL) formulation produces a two-stage model (LCL-V2) that is better aligned with selectivity properties and neural activity in primate area V2. We demonstrate that the complexity-matched learning paradigm is responsible for much of the emergence of the improved biological alignment. Finally, when the two-stage model is used as a fixed front end for a deep network trained to perform object recognition, the resultant model (LCL-V2Net) is significantly better than standard end-to-end self-supervised, supervised, and adversarially-trained models in terms of generalization to out-of-distribution tasks and alignment with human behavior. Our code and pre-trained checkpoints are available at https://github.com/nikparth/LCL-V2.git

Published in Transactions on Machine Learning Research (06/2024)
Layerwise complexity-matched learning yields an improved model of cortical area V2
Nikhil Parthasarathy
mp17742mmu.edu
Center for Neural Science, New York University Center for Computational Neuroscience, Flatiron Institute
Olivier J. Hénaff
henaff@ggoggl.. com
Google DeepMind
Eero P. Simoncelli
een.siioonceliimmyu.c
Center for Newral Science, New York University
Center for Computational Neuroscience, Flatiron Institute
Reviewed on OpenReview: https: // openreview. net/ forum? id= LQBsL.fAWhj
Abstract
Human ability to recognize complex visual patterns arises through transformations per- formed by successive areas in the ventral visual cortex. Deep neural networks trained end- to-end for object recognition approach human capabilities, and offer the best descriptions lo date of neural responses in the late stages of the hierarchy. But these net works provide a poor account of the early stages, compared to traditional hand-ent -anddeengeeereed models, or mod- els optimized for coding efficiency or prediction. Morcover, the gradient backpropagation used in end-to-end learming generally considered to be biologically implausible. Here, overcome both of these limitations by developing bottom-up self-supervised training methodology that operates independently successive layers. Specifically, maximize fea- ture similarity between pairs of locally-deformed natural image patches, while decorrelating features across patches sampled from other images. Crucially, the deformation amplitudes are adjusted proportionally to receptive field sizes each layer, thus matching the task complexity to the capacity at each stage processing. In comparison with architecture- matched versions of previous models, we demonstrate that our layerwise complexity-man matched learning (LCI.) formulation produces two-stage model (T.CI.-V2) that is better aligned with selectivity properties and neural activity in primate area V2. We demonstrate that the complexity-mallcee learning paradigm is responsible for much of the emergence of the improved biological alignment. Finally, when the two-stage model is used as fixed front-end for a deep network trained to perform object recognition, the resultant model (LCL-V2Net) is significantly better than standard end-to-end self-supervised, supervised, and adversarially-traie
models in terms of generalization out-of cit-of-dditributtiin tasks and aligmment with human behavior. Our code and pre-trained checkpoints are available at https://github. hot.///iiigaarr//11111.....
1 Introduction
Perception and recognition of spatial visual patterns, scenes and objects in primates arises through cascade of transformations performed in the ventral visual cortex (Ungerleider & Haxby 1994). The early stages of visual processing (in particular, the retina, lateral geniculate nucleus, and cortical area V1), have been studied for many decades, and hand-crafted models based on linear filters, rectifying nonlincaritics, and local gain control provide reasonable account of their responses propertics (Shapley Victor 1979 McLean
& Palmer 1989
Adelson & Bergen 1985 Carandini et al. 1997 Complementary attempls to bottom-

---

Published in Transactions on Machine Learning Research (06/2024)
up normative principles such sparsity, coding efficiency, or temporal prediction have provided successful
accounts of various early visual properties (Atick & Redlich 1990 Van Hateren & van der Schaal] 1998 Lip
1996.
Olshausen & Field, 1996 Bell & Sejnowskil 1997
Schwartz & Simoncelli [000]] Karklin & Lewicki
2009
Wiskott & Sejnowski, 2002 Hoyer & Hyvärinen 2002
Cadicu & Olshausen 2012 Karklin & Simoncelli
2011
But these also have been limited to early stages up to area V1, and have thus far not succeeded in
going beyond.
Deep neural networks (DNNs), whose architecture and functionality were inspired by those of the primate
visual system (Fukushima 1980 Douglas et al. 1989 Heeger et al. 1996 Riesenhuber & Poggio 1999
have offered new opportunity. When trained with supervised and self-supervised end-to-end (E2E)
backpropagation, DNNs have provided the first models that begin to capture response properties of neurons
deep in the visual hierarchy (Yamins et al.] 2014 Zhuang et al.) 2021 Schrimpf et al. 2018 Kubilius
et al. 2019). Early results showed that these DNNs are also generally predictive of the overall category-level
decisions of primates during object recognition tasks (Ghodrati ct al. 2014
Jozwik et al., 2016 Kheradpishch
eet al. 2016); however, they have not been predictive of more detailed behavior, as measured by alignment with individual image confusion matrices (Rajalingham et al. 2018). Nevertheless, as the field has rapidly progressed, more recent results demonstrate that scaling end-to-end task-optimization (both in training data and model size) leads to significant improvements in predicting this trial-by-trial human behavior in matched visual tasks (Geirhos et al. 2021 Sucholutsky et al. 2023
Ironically, despite their historical roots, these same networks have not provided convincing models of early visual areas such as V1 and V2, and do not account for other perceptual capabilities (Berardino et al. 2017 Hénaff et al. 2019 Fel al. 2022; Subramanian et al. 2023 Bowers et al. 2022 Feather et al. 2023 Figure sunmarizes these observations for a set of models with a wide variety of architectures and training paradigms, drawn from the BrainScore platform (Schrimpf et al. 2018). The left panel shows that improvements in object recognition performance are strongly correlated (r (r= 0.57) with improvements in accounting for human recognition capabilities. This is encouraging, but perhaps expected, since the recognition databases used for training represent human-assigned labels. The right panel shows that there is also a positive correlation (albeit weaker) between recognition performance and ability to explain responses of IT neurons recorded in macaque monkeys. Again, this is perhaps not surprising, given that object- recognition behavior can be to some extent explained by linear weightings of neurons in inferior temporal (TT) cortex (Majaj et al. 2015). (Tt is worth noting, however, that for models with very high recognition performance (> 70%), the ability to explain IT neurons has gotten progressively worse (Linsley et al.] (202))) ). Surprisingly, recognition performance is uncorrelated (or even slightly anti-correlated) with the ability to explain responses of visual neurons cortical areas V1 and V2.
Why do these networks, which offer human-liks like performance in complex recognition tasks, and which provide reasonable account of neural responses in deep stages of the visual hierarchy, fail to capture carlier stages? We interpret this as an indication that intermediate DNN layers are insufficiently constrained by end-to-end training on recognition tasks. More specifically, the extremely high model capacity of these networks allows the training procedure to find "shortcuts" that satisfy single end-to-end objectives (both supervised and self-supervised) (Geirhos et al.) 2020a Robinson et al. 2021). As a result, it is common for networks to utilize unreliable feature representations that do not generalize well (Hermann & Lampinen 2020). This is further evidenced by the fact that standard trained networks can be fooled by "adversarial examples' (small pixel perturbations that, can large shifts in internal classification decision boundaries) (Szegedy et al.) 2013
Goodfellow et al.) 2014 Tramer et al., 2017). With this in context, it makes sense that the current best DNN models for V1/V2 seem to be those that are trained to increase robustness to adversarial attacks Maddy et al. 2017). However, the specific solution of adversarial training comes at a significant cost in standard image recognition performance, as well as being computationally expensive. Moreover, this training procedure still propagates gradients via E2E backpropagation, which is generally considered to be biologically implausible.
In this work, hypothesize that representations throughout DNN can be constrained in more aalooicaalyy-llassbbe manner through the use of layerwise self-supervised learning objectives. We propose natural method for matching the complexity (or difficulty) of these objective functions with the compu- tational capacity at each stage of processing. When used to train a two-stage model, the resulting network
2

---

Published in Transactions on Machine Learning Research (06/2024)
T= -0.31
Human Behavior Explained
0.6
0.57
ResNext101 WSL
Neural Responses Explained
0.55
-0.08
0.34
0.50
0.5
0.45
(22)
0.40
0.4
AlexNet
0.35
AlexNet
0.3
0.30
0.25
0.2
0.20
ResNext101 WSL
0.5
0.7 0.8
0.4
0.6
0.8
ImageNet Accuracy (top-1)
ImageNet Accuracy (top-1)
Figure 1: DNN object recognition performance predicts human recognition behavior, but not primate early visual responses. Each plotted point corresponds to a DNN model from the BrainScore database (Schrimpf et al. 2018)). Horizontal axis of both panels indicates recognition accuracy (top-1) on the ImageNet datasot (Krizhovsky et al. 2012). Left: Comparison to alignment with human visual recognition performance (combination of benchmarks taken from Geirhos al. (2021) and Rajalingham et al. 2018)) ). Right: Comparison to neural variance explained by regressing the best-fitting DNN layer to
neural responses measured in macaque V1 (green), V2 (blue) (Freeman et al. 2013 Ziemba et al. 2016 and
IT (black) (Majaj et al. 2015 Sanghavi & DiCarlo, 2021 Sanghavi et al. 2021b
achioves statc-of-the-art predictions of neural responses in cortical area V2. Furthermore, when using this learned model as a front-end for supervised training with deeper networks, we show that (in contrast with adversarial training) the increased neural alignment does come the cost of object recognition perfor- mance, and in fact results in significant improvements in out-of-distribution recognition performance and alignment with human behavior.
2 Methods
We first describe the key conceptual underpinnings of our layerwise training method, and then provide the experimental training and evaluation details.
2.1 Layerwise complexitymmatched learning
Layerwise (more generally, blockwise) methods for DNN training have been previously developed to alleviate the global propagation of gradients required in E2E training training (Hinton et al. 2006 Bengio et al.
2006 Illing et al. 2021 Belilovsky et al. 2019 Siddiqui et al.) 2023 Halvagal & Zenke 2023). Figure 2
illustrate the relationship between the two approaches. Given of inputs x and corresponding output
labels 2, the E2E approach optimizes all network parameters to minimize the loss function LE2F via full backpropagation. Successful training of high-capacity networks has generally been achieved with large amounts of training data and complex objectives: (1) supervised data that encourages object-level semantic invariances (Krizhevsky et al.] 2012), (2) self-supervised data generated using combination of spatial and photometric augmentations (Zbonlar el al. 2021 Chen et. al.) 2020b Grill et al. (020),, or self-supervised masked autoencoding with substantial levels of masking (He et al. 2021). In general, the quality learned features and the success in recognition depends on the complexity (or difficulty) of the learning problem. For example, if we consider supervised classification objective, the difficulty of this problem will depend on factors such the number of classes, complexity of the image content (simple shapes real-world objects).

---

Published in Transactions on Machine Learning Research (06/2024)
forward
fo - (22E(z;0)
fe
X
Z
high model capacity
complex objective (i.e. learning object-level invariances)
backward
fo
Lı(zi; 01) objective complexity grows with
for
f0z
X
model capacity grows with /.
1 < Z2
The standard end-to-end (F2F) learning paradigm used with DNNs. The loss function (LEZE) operates on the network output and is typically chosen to lavor object-level invariances, through supervised training on labelled data or self-supervised training on augmented examples. To solve these E2E objectives, the network f(0), must have a high model capacity (sufficiently large number of parameters and non-lincarities. Bottom: a layerwisc training system, the loss is function of all intermediate outputs (=1, 22,.... Losses each layer Lı are used to train each encoder stage fo, independently, with gradients operating only within stages. For effective training, hypothesize that the loss at each stage, Lt, should be matched in complexity to the model capacity defined by the network up to layer 1.
Figure 2: Layerwise complexity-matched learning. Top:
or the magnitude of the within-class variability (deformations under which each object is seen). Similarly, these training set properties control the complexity of the self-supervised problem (Robinson et al.) 2021 Jing et al. 2021). In contrast, in the layerwise approach, the objective partitioned into sub-objectives that operate scparately on the output of cach layer, and the optimization thus relies on gradients that propagate within (but not between) layers. The model at given layer I is composed of all stages up to that layer: for.... Thus, the computational capacity is low in the early layers (only few non-linearities and small receptive fields) and increases gradually with each successive layer. To achieve successful training in this scheme, we propose to match the complexity of the data diversity and objective function with the effective model capacity at a given loyer.
2.2 self-supervised contrastive objective
We construct a layerwise objective based on the "Barlow Twins" self-supervised loss (Zoonarr et al. 2021 a featuro-contrastive loss that is robust to hyperparameter choices and has recently shown success blockwise learning (Siddiqui et al.) 2023). Briefly, each image x in a batch is transformed into two views, x and x via randomly selected spatial and photometric deformations. Both views are propagated through an encoder network fe and projection head go produce embeddings z go fo(x). We define a cross-correlation matrix over each batch of images and corresponding view embeddings:
Cj VELC VENCRREE E.(
where indexes the batch samples and i and j index the vector components of the projection head embedding. The Barlow Twins objective function is then:
L Lgr (1--04) (11-49) c2 - +
(2)
4

---

Published in Transactions on Machine Learning Research (06/2024)
stop-grad
Full view (CC)
ii z ap 2.
Encoder
Image patch size (feature complexity)
for
f0z
Spatial deformation strength strengthh
11
g0l
g02
Projector
poa
100
Loss Lossgr<Z1224 zf) + Loss (2 22)
D2
Figure 3: Layerwise complexity-matched objective. Left: invariance to feature perturbations by comparing the representation of two augmented views of the same image. For layer 1, the fcature complexity of generated image pair (z/1, xx²) is controlled through choice of patch size, and the magnitude of spatial deformations (translation, dilation). Right: The parameters 01 of the first layer encoder fe, are updated using the Barlow Twins feature-contrastive loss (Zbontar et al. 2021) operating on the two views of the smallest patch size (x1, .F.). This set of views is only propagated this layer output. The parameters 0g of the second layer encoder for are updated with the same loss, but using the views that cover larger spatial region, and include larger spatial deformations.
For each layer, the objective encourages
This loss encourages formation of invariant projection-feature (or equivariant encoder features) across the two views (maximizing the diagonal terms of the correlation) while decorrelating or reducing the redundancy of the components of the output vectors (minimizing the off-diagonal terms). This objective can thus be thought as a "feature-contrastivve" method. As noted in Garrido et al. (2022), there is a strong duality between this loss and with sample-contrastive losses (such as SimCLR (Chen et al., 2020b)). Accordingly,
we achieve similar results in our framework using sample-contrastive losses (more in Sec. 3.4 but find slight improvements in performance and stability with the Barlow Twins objective.
2.3 Training methodology
We apply our Layerwise Complexity- Learning paradigm (LCL) a two-stage model, denoted LCL- V2. The training methodology is depicted in Fig. 3 The loss function aims to optimize feature invariance across augmented views of image, while decorrelating features across different images. We control the complexity (difficulty) of each of these learning problems by changing both the size of input images and the strength of augmentation deformations that are used to compute the per-layer loss functions. Fig. 3 (a) depicts this input processing for an example image, considering our two-layer network. Given the full view æ, patch is cropped for layer (xf)) and layer 2 (xz)). We choose an initial patch size for aa and note that the patch size for layer 2 is simply scaled by a factor of 2, roughly matchod to the scaling of biological receptive field sizes between areas V1 and V2 (Freeman & Simoncelli 2011). For the selected patches, then generate augmentations (xg) xz)) using photometric and spatial deformations. For simplicity, we maintain the same photometric deformations and scale the problem complexity by proportionally adjusting the strength of the spatial deformation by a factor of between the two layers. Visually, we see that this procedure results in paired images for layer that have low feature complexity and small translation and scale differences while the images for layer have higher feature complexity and larger deformations.
Given these inputs, Fig. 3()) shows the procedure for computing the per-layer loss functions. We gener- ate projection embeddings for layer and layer 2 by propagating the corresponding input patches to the
5

---

Published in Transactions on Machine Learning Research (06/2024)
corresponding model blocks:
21 A GAP go, fo, (21)
22 GAP ge, o fo, o for (21)
fe; refers to the encoder blocks and 9e, corresponds to the projection heads for each layer. We project the embeddings at each spatial location of the feature map independently before applying global average pooling (GAP) to create the final projection embedding. z and z/ are computed analogously from xf and x². The loss is then computed as the sum of losses for each layer: Loss = L L.TT + LBr(= , As in Siddiqui et al. (2023), ; the loss computation only requires backpropagation within each layer, and gradients from the layer loss do not affect parameters in fe.-
In summary, we implement a complexitymatthhnn layerwise learning formulation where the difficulty of the learning problem at layer 2 is scaled in comparison with that at layer 1. The model must learn invariant fea- tures across images that have more complex content (larger patch size) that are also more strongly deformed (in scale and translation). This increase in objective complexity accompanies a corresponding increase in model capacity in the second layer (due to growth in receptive field size and number of nonlincaritics).
2.4 Implementation details
Architecture. As in many previously published results (Caron et al. 2018 Gidaris et al., 2018); we chose to use the AeexNet architecture (Krizhevsky et al. 2012) with batch normalization (Tofle & Szegedy 2015). While many recent results make use of more complex architectures (eg, ResNets (He et al., 2016). Vision Transformers (Dosovitskiy et al.] (020) etc.) our method can be more effectively evaluated with a very shallow network, as we can severely restrict model capacity in training these early layers without confounding architectural features such as skip connections, attention blocks etc. Additionally, as mentioned earlier, much of the biological anatomy and computational theories suggest that V2 should be explainable by single layer of transformation given V1 afferents (El-Shamaylch et al. 2013 Ziemba 2016). Given that AlexNet models provide relatively strong baseline architectures for V1 (compared with more complex models such as ResNets and vision transformers), we hypothesize that this simple architecture can provide a starting point towards parsimonions and interpretable models of carly vision.
For L.C.-V we train the first two convolutional stages of the AlexNet architecture and utilize a standard multi-layer perceptron (MLP) with a single hidden layer for the projector networks at each layer. The computational capacity is increased between the tw stages, with each stage incorporating two non-linearities (ReLU activation and MaxPooling). In addition, capacity is scaled by increasing the number of channels (64 to 192) and receptive field size (via (2x) subsampled pooling). In Sec. 3.5| we additionally evaluate the effectiveness of LCL-V2 as a fixed front-end model (similar to Dapello et al. (2020)). We train the remaining AlexNet layers (with batch normalization) on top of the fixed LCL-V2 V2 front-end and refer to this full network as LCL-V2Ned For more specific architecture details, see appendix Sec A.1
Data and Optimization. We train LCL-V2 and its ablations (see Sec. 3.4 on the ImageNet-Ik dataset KKrihhessyy et al., 2012). We resize the original images to minimum size 224x224. For layer 1 we centrally crop a 56x56 patch and generated spatially augmented views of size 48x48 via the RandomeResizedCCrp (RRC) operator with scale = (0.6, 0.9). For layer 2, we central crop a 112x112 patch and generate views of size 96x96 with RRC crops (scale = (0.3, 0.9)). As a result, both the final patch size and crop scale range are doubled between layer 1 and layer 2. For each set of patches, we also apply a fixed set of photometric distortions by weakly varying contrast and luminance, and adding random Gaussian noise with variable standard deviation (details in Sec. A.2). Unlike standard E2E self supervised approaches, we do not use the more aggressive augmentations (large color jitter, flipping etc), which seem less perceptually relevant.
We use the Adam optimizer (Kingma & Ba. 2014 without weight decay and Ir = 0.001. We train the model until the summed validation loss (evaluated on a held-out set of images) does not improve above beyond a fixed threshold. While recent work in self-supervised learning has found benefits from using more complex optimizers and learning rate schedules, we find no significant benefits in our two-layer setting. To train the
The additional GAP pooling is only applied during training.
6

---

Published in Transactions on Machine Learning Research (06/2024)
full C2Ne,,, we fix the pre-trained LCL- V2 front-end and use standard supervised cross-entropy loss to train the subsequent, stages. We train for 90 epochs using the SGD optimizer (Ir 0.1) with a step-wise learning rate scheduler that reduces the learning rate every 30 epochs.
2.5 Experimental setup
Model comparisons. Throughout this work we compare to a variety of previous models of three types (see Sec. A.1 for details):
End-to-end AlexNet models trained with standard supervised or self-supervised objective functions on the ImageNet-1K dataset: Supervised (Krizhevsky et al. 2012). Barlow Twins
E2E (standard):
[Zbontar et al. 2021), and VOneNet (fixed-V V1 stage and supervised learning for downstream stages)
(Dapello et al. 2020
E2E (robust): End-to-end AlexNet models trained with state-of-the-art robustification methods specifically to maintain robustness to adversarial pixel perturbations: standard adversarial train- ing (I.2-AT (e = 3.0)) (Madry et al. 2017). adversarial noise training with a parameterized noise distribution (ANT) (Rusak et al. 2020).
Layerwise training: AlexNet model trained with the Barlow Twins objective using thestandard image augmentation schome applied layerwise (Siddiqui et al. 2023 Latent--rediciiv-llaaing (LPL) (Halvagal & Zenke 2023).
Hand-crafted: Steerable pyramid layer (Simoncelli Freeman 1995) (with simple and complex cell noulinearities), followed by a layer of spatial Le (energy) pooling.
Neural alignment evaluations. We compare all models quantitatively their ability to predict aspects of V2 neurons from the dataset used in BrainScore (Schrrimpf et al. (2018). This dataset, described in Freeman et al. (2013) Ziemba et al. (2016), provides eleetrophysioolgiiall recordings of 103 V2 neurons responding to texture images synthesized with the Portilla-Simonceli texture model (Portilla & Simoncelli 2000). The data include responses to 15 texture samples, in addition to 15 samples of spectrally-mathhel noisc images, for 15 different texture families (a total of 450 images). To measure model predictivily of this neural data, we fit models with the data splits and implementation of the partial least squares (PLS) regression method proposed Schrimpf et al. (2018), and then compute explained-varde variincee scores for each fitted model. Details are provided in Sec. A.3
To better understand the ability of models to capture selectivities of V2 neurons, we provide additional evaluations (Sec. 3.2) that use the texture modulation ratio statistic introduced in Freeman et al. (2013) teiss,-nciseo.. where ler,,i is the response of neuron (averaged across 15 image samples) to texture family and noisen,i is the corresponding response to the spectrally-matched noise for family i.
Specifically, we define: Rmood., tern.:
LCL-V2Net recognition and human behavior evaluations. We primarily use the out-of-distribution (OOD) generalization benchmark of Geirhos et al. (2021) to test the performance of LCL- This dataset consists of 17 OOD classification tasks based adding various kinds of noise, distortions, and shape-biasing transformations to ImageNet images. We evaluate both OOD accuracy and consistency with human behavior. For more information on the benchmark, specific list of distortions, and evaluation metrics see Sec. A.4
We additionally report performance on the original ImageNet-1K (Krizheskyy et al. 2012) validation sot as well as more recent large-scale validation sets (ImageNet-R. (Hendrycks et al. [202]) and ImageNettvvddrooust (Shankar et al. 2021 for testing generalization.
7

---

Published in Transactions on Machine Learning Research (06/2024)
Hand-crafted
E2E (standard)
E2E (robust)
Layerwise
0.40
V2 Exp. Var. (V2 (V2-like p2-like, N=26)
0.25
V2 Exp. Var. (N=103 1031
0.35
0.20
0.30
0.25
0.15
0.20
0.10
0.15
0.05
0.10
0.05
0.00
0.55
0.60
0.65
0.70
0.75 0.80
0.00
V2 Exp. Var. (V1-like, N=26)
Untrained
Vi-SteerPyr+PPool
BarlowTwins Supervised
VOneNet
L2-AT 3.0)
ANT
Block LCL-V2 (Ours)
LPL
Figure 4: The LCL-V2 model outperforms other models in accounting for V2 responses. Left: Median explained variance of models fitted with PLS regression to 103 primate V2 neural responses. For models with more than two layers, all layers are evaluated and the performance of the best layer is provided. Standard deviations over 10-fold cross-validated regression are indicated on each bar. Right: Comparison of median explained variance for "V1-like" and "V2-like" " V2 cells. These categories correspond to the top and bottom quartiles (N=26) of V2 cells sorted by how well they are fit by canonical hand-constructed V1 model (V1-SteerPyr+ Pool). The minimum explained variance of the V1 model over the set of "V1-like" neurons is 57 %. The maximum explained variance of the V1 model over the set of "V2-like" neurons is 4 %. The LCL-V2 and L2-AT models significantly outperform all other models on the V1-like subset, even surpassing the baseline V1 model. The LCL-V2 model also significantly outperforms the L2-AT model on the V2-like subset.
3 Results
3.1 Population fits to neural data
Overall V2 predictivity. The first panel of Fig. 4 shows the overall BrainScore explained variance of the models outlined in Sec. [2.5 For all models, the best layer was chosen by evaluating predictions on validation set prior to fitting the final PLS regression outperforms all architeccture-matcher models, including the T2-AT trained network. In fact, although we only show richitecture-matchee results here, our model provides the best account of the V2 data across all architectures currently on the BrainScore leaderboard (Schrimpf et al. 2018 2020). We provide the detailed mumbers in Table A.4 Interestingly, previous layerwise training methods ('Barlow (layerwise)' (iiddiqui et al.) 2023) and 'IPL'' (Halvagal Zenke 2023)) cxhibit significantly worse performance than the standard end-to-end training. This suggests that the benefits of our method specifically arise from conplexity-matching which we quantify further in Sec. 3.4
the held-out test set. We that LCL-V2
Partitioning V2 with V1-bascline model. The VI-StcerPyr+-Pod model provides bascline measure of how well V2 neurons can be predicted simply by combining rectified and Lz-energy pooled oriented filter responses (as are commonly used to account for V1 responses). Nearly 30 % of the variance across all 103 V2 neurons can be explained given this model, suggesting that there are a number of neurons that are selective for orientation and spatial frequency selectivity. In fact, this aligns with prior studies that have found subsets of V2 neurons with tuning similar to V1 neurons (but with larger spatial receptive fields)
Foster et al. 1985 Levitt et al. 1994 Willmore et al.
2010 Lennie 1998).
2a of publication date: 06/2024 2024

---

Published in Transactions on Machine Learning Research (06/2024)
Given this baseline model, we partition the V2 neural datasets into neurons that are 'VI-like' (top quartile, in terms of how well they are explained by the V1-SteerPyr+ model) and those that are 'V2-like' (bottom quartile). In the right panel of Fig. | we compare the median performance of each of the models on these two subsets. We find that all other non-adversarially trained models (both layerwise and end-to-end) are significantly worse at predicting the 'V1-like' subset than the baseline V1 model. Surprisingly, both LCL. V2 (ours) and L2-AT models outperform the V1 basoline on this subset, suggesting that although these neurons are most-likely orientation and spatial frequency tuned, they also have some selectivity that is not captured by the simple V1 model. On the 'V2-like' subset, the performance of all models is significantly worse; however, there is now an even larger gap (approx 5%) between LCI.-V and the L2-AT model. Thus, the adversarial training achieves better predictions of area V2, primarily by better explaining the neurons that have 'V1-like' properties. LCL-V2 maintains this improvement, but also provides better fits to neurons whose complex feature selectivity is not well described by the baseline V1 model. In the following section, we examine whether the models exhibit known fcature selectivities found in the V2 data.
Per-neuron V2 predictivity. To explore whether the above summary statistics are representative of the predictions of given model at the per-neuron level, we also computed the predictions of our model for each of the 103 V2 neurons. When compared against predictions of the L2-AT and Supervised AlexNet models, we find that there is in fact a wide distribution of explained-variances across neurons but that our model outperforms the two other models on a majority of neurons (see Fig. A.2 for more details). We believe it could be quite valuable in future studies to further analyze this data to understand what properties of individual neurons determine the variability in predictions of given model.
3.2 Model comparisons via texture modulation
Cortical area V2 receives most of its input from V1. A fundamental property of V2 neural responses that is not present in V1 responses is that of texture modulation (Freeman et al. 2013 Ziemba et al. 2016), in which responses to homogencous visual texture images are enhanced relative to responses to spectrally-matched noise. As described in Sec. 2.5 we compute a texture modulation index Rmod,,, for each of the 103 neurons, for each of the 15 texture families. We computed the same modulation index for each neuron in the selected V2-layer from each computational model. In Fig. 5 we compare LCL-V2 against the top two other fully learned models in terms of overall V2 explained variance (L2-AT and standard We exclude the VOneNet model here as it uses a fixed front-end with different architecture.
We first compare the three models in terms of their ability to capture the full distribution of texture modula- tion ratios in the V2 dataset (Fig. B(a)), We compute a modulation ratio for each neuron by averaging over texture families: Rmod, = 1 Σ Rmodd... We use quantile-quantile (Q-Q) plot to compare the quantiles of the distribution of these values to those arising from the modulation ralios of each fitted model neuron. It is visually clear that while none of the models perfectly match the V2 neural distribution, LCL-V2 is significantly closer than the other two.
Next, we compute texture modulation ratios for each texture family by averaging over neurons Rmad, = I N. EA' Rmod... Because different texture classes have different types of feature content, they stimulate V2 neurons differently, relative to their spectrally-matcher counterparts. We compare the rank-ordering of modulations ratios over the texture families for the model and real neurons and scatter-plot the ranks against each other (Fig. B(b)). The texture family ranks of the LCI-V2 model are well-aligned with those of the actual V2 neurons, whereas both L2-AT and Supervised models yield ranks with many more outliers. This is quantified by the Spearman rank correlation for LCL-V2 (o 0.90), which is significantly higher than that of the other two models (p = 0.56, p 0.58). It is worth noting that Laskar et al. (2020) find that this rank correlation can be improved for models by incorporating a subset selection procedure to restrict the specific model neurons used in the comparison.
In summary, although the L2-Robust and Supervised models provide competitive predictivity of the V2 neural responses (Fig. D. the LCL-V2 V2 model provides a better account of the texture selectivity properties of these neurons.
9

---

Published in Transactions on Machine Learning Research (06/2024)
Neuron RMod Quantiles (Model)
LCL-V2 (Ours)
L2-AT (E= 3.0)
Supervised
1.0
-0.5
-1.0
-1.0-0.5
0.0
0.5
1.0
-1.000.5 0.0 0.5 1.0
-1.0 0.0
0.5 1.0
Neuron Rmad Quantiles (V2)
Tex Fam Riod Rank (Model)
15
10
= 0,90
p = 0.56
0,58
10 15
10 15
10
Tex Fam Riod Rank (V2)
Figure 5: The LCL-V2 model outperforms other models in capturing texture modulation prop- erties of V2 neurons. We compare the top 3 (in terms of overall V2 predictivity) fully-learned models: LCL-V2 (Ours), L2-AT, and Supervised. Top: Quantile-quantill (Q-Q) comparison the distribution of texture modulation index values (Rmod; averaged over texture families) for real and model neurons. LCLL-V2 shows better alignment with the physiological distribution (closer to the identity line (dashed)) than the other (wo models. Bottom: Comparison of lexture modulation indices for each of 15 texture families (averaged neurons). The modulation indices for model and real neurons ranked (1 lowest modulation family, 15 highest modulation family), and plotted against cach other. Our model provides better alignment with the V2 data, achieving Spearman rank correlation of p 0.9. P-values were computed to test significance of the difference between the Spearman corvelations using the methodology described in Sec. [A.3.3] We find the difference to be significant vs. both models (L2-AT,, Supervised) with (p 0.040, p=0.047) respectively.
3.3 V1 layer analysis
To demonstrate the generality of LCL approach in learning fcature hicrarchics, evaluate the first- stage (LCL-V1) in terms of aligmment with V1 responses and selectivities. In Fig. 6 we find that the LCL-VI model outperforms all non-adversarially trained models in terms of V1 explained variance (approx 2-4 % improvement on average), and is on par with both adversarially-ttrineed models (ANT and L2-AT). Furthermore, when visualizing and characterizing the learned receptive fields, we find reasonable qualitative similarity with receptive field properties extracted from the V1 data (Ringach) 2002) (for details, see Sec. A.5p. Note, however, that the hand-crafted models (VOneNet and V1-SteerPyr) still provide better accounts for the V1 responses than all learned models. We suspect this is largely due to the limited receptive field sizes of the learned AlexNet-based models, all of which use single convolutional layer with 11x11 kernels.
3.4 Ablations
We now examine the effect of ablations of our architectural and training choices on physiological alignment.
Complexity-mismathh Fig. shows a significant improvement V2 predictivity over all non- adversarially trained models and in particular improves dramatically over the application of the layerwise training approach outlined in Siddiqui et al. (2023). Since that model was also trained with the Barlow Twins
10

---

Published in Transactions on Machine Learning Research (06/2024)
E2E (slandard)
E2E (robust)
Layerwise
Hand-crafted
0.40
v1 Exp. Var. (N=103) 103)
0.35
0.30
0.25
0.20
0.15
0.10
0.05
0.00
Untrained
BarlowTwins Supervised
L2-AT (s 330)
ANT
LPL BlockBarlow LCL-V1 (Ours)
VOneNet
Vi-steerPyr
Figure 6: LCL-VI outperforms learned models in V1 predictivity and approaches the perfor- mance of hand-tailored V1 models. Analogous to the V2 comparisons (Fig. 4. we also evaluate the best model layers for explaining the V1 neural responses from the same dataset. The highest explained variance is obtained by the hand-designed V1-SteerPyr and VOneNet models. The L.CL-V1 model performs similarly to the adversarially robust models, and outperforms all other trained models. Standard deviations over 10-fold cross-validatee regression are indicated for each model.
objective, the primary difference with our model is in the complexity complexity-matching of our objective. Specifically, while they use at cach layer a set of augmentations generally used to train large object recognition networks, we approximately complexity-match the objective with capacity at each stage of our model. To better un- derstand the quantitative impact of this, we evaluate the neural predictivity of our learned model when introducing complexity-mismatt via changes in the patch size (feature complexity) random crop scale (spatial deformation strength). Fig. shows that the optimal performance (in terms of neural predictivity) is achieved only when the relative complexity is matched between the two layers. Specifically, we first see that there is an optimal patch size (48px) and deformation strength (s=1) which produces the most aligned V1 layer. This follows from the hypothesis that the the layer 1 views should contain simple edge-like content with small scale deformations. More importantly, once the optimal parameters for the V1 layer training are fixed, we find that both patch size and spatial deformation strength must be scaled accordingly to achieve the optimal V2 model (highlighted in green). This again justifies the initial choice of these scaling parameters based on the natural approximate doubling of receptive field size between V1 and V2.
Contrastive vs non-contrastive losses. While numerous self-supervised learning objectives have been proposed over the years, they generally can be classified as contrastive or non-contrastive. While the Barlow Twins loss is 'feature-contrastivve there have been studies demonstrating a duality with 'sample-contrastive approaches (Garrido al. 2022 Balestriero LeCun 2022). Non-contrastive losses; however, resort to very different mechanisms for avoiding collapsod solutions, with most using some form of 'stop-gradient" based method with asymuetticc encoder networks (Chen et al. 2020b Grill el al. 2020 In Fig. 7 we show that some form of contrastive term (either Barlow Twins or SimCLR) is necessary for achieving optimal neural alignment (especially in the second stage). For example, using SimSiam (Chen et al. (020b) as a representive non-contrastive loss, greatly hurts the V2 predictivity from the second stage. Therefore, in this context, seems that in addition to the problem of learning invariances, it is also necessary include feature decorrelation or sample discrimination as part of the final objective.
3.5 OOD object recognition and human-alignment
In the spirit of Dapello al. (2020), we hypothesized that a more biologically-alinned model of carly visual areas (specifically area V2) may provide additional benefits for both recognition performance and alignment with human behavior visual tasks. Note that while ultimate goal is to train full model of the
11

---

Published in Transactions on Machine Learning Research (06/2024)
LCL-V1
LCL-V2
Baseline (EZE aug scheme)
V2 Baseline (E2E aug scheme)
0.42
0.42
Explained Variance
0.39
0.36
0.30
0.33
0.33
0.24
0.30
0.20
0.30
48
96
192
non-overiap crop
Barlow (contrast)
SimCLR
SimSiam (non-contrast)
(contrast)
Patch Size (feature complexity)
Spatial deformation strength
Objective function
non-contrastive objectives decreases neural alignment. For complexity ablations (left and middle panels), vary the patch size spatial deformation strength for the V1 layer (LCL-VI). We then hold these parameters at the optimal values for the V1 layer fixed and again vary the parameters used for training the V2 layer. Left: Indicated by the green circle, we see that there is an optimal patch size (feature complexity) for best V1 prediction at 48px and the optimal patch size for V2 is then scaled accordingly (factor of 2 larger). Middle: We see that there is also an optimal spatial deformation strength for each layer that also scaled by a factor of 2 (in minimum crop scale) between each layer. Spatial deformation strength 0 refers no spatial deformation, Deformations (1-3) refer to the minimum random resized crop scale of (0.6, 0.3, 0.08). 'Non-overlap crop' refers to only using non-overlapping crops. Right: We ablate the loss function used to train each layer. We find that performance is very similar with SimCLR, but significantly worse when using a non-contrastive method like SimSiam. Baseline comparisons (dashed lines) indicate performance of the layerwise Barlow method proposed in Siddiqui et al. (2028), which uses the end-to-end training augmentation scheme (details in Sec. A.2 from Zbontar et al. (2021) for each layer.
Figure 7: Substituting complexty-mmismacceet
ventral stream in layerwise fashion, currently, this would require collecting significantly larger images to capture increasing levels of feature complexity. We therefore opt for a simpler experiment and learn a cascade of additional AlexNet stages appended to the fixed LCL-V2 front-ond model via the standard ImageNet-1k supervised object recognition lask. We then evaluate the full network on the benchmark proposed by Geirhos let al. (2021) which tests both out-of-distribution (OOD) generalization and prediction of human behavior on this task. Again, we refer to this trained recognition network as LCL-V2Net.
Object recognition accuracy. We first evaluate the accuracy of our trained network on the ImageNet- 1K (Krizhessky et al. 2012) validation set as well as OOD image set proposed in Geirhos et al. (2021) Fig. g (Left) shows that L2NeV significantly outperforms all arehitecturally-matchh in OOD accuracy by a large margin (4-10 % improvement see Table A.5). This is particularly striking because the other robust. models (VOneNet and I.2-AT) do not exhibit a similar improvement, suggesting potential link between the improved V2 predictions of the LCL-V2 front-end and the gencralization of recognition over shifts in the data distribution.
(AlexNet-baser) models
Human behavioral consistency. In addition to absolute recognition performance, also evaluate the ability of the same models to capture human behavioral performance on the same recognition task. The right panel of Fig. shows that LCL-V2Net has significantly better error consistency with the per-trial human recognition decisions. Compared with standard supervised training (consistency=0.1655) the only models to show significant improvement are those trained for adversarial robustness (ANT and L2-AT). These models each achieve a consistency of 0.176 (a 6.6 % relative improvement). Without the computational overhead of adversarial training procedures, T.CL.V2Net achieves consistency of 0.211 (a 28 % relative improvement).
12

---

Published in Transactions on Machine Learning Research (06/2024)
0.52
E2E (standard)
E2E (robust)
Behavioral Error Consistency
0.24
E2E (standard)
E2E (robust)
OOD Accuracy (top-1) 1)
0.39
0.18
0.26
0.12
0.13
0.06
0.00
0.00
BarlowTwins Supervised VOneNet
L2-AT (E= 3.0) 3.0)
LCL-VZNet (Ours)
ANT
BarlowTwins Supervised VOneNet
ANT L2-AT 3.0)
LCL-V2Net (Ours)
Figure LCL-V2Net improves OOD generalization and human behavior error consistency. Left: Supervised training of the later layers of AlexNet modelon top of LCL-V2 leads to significantly increased OOD accuracy compared to all other architecture-matcine models including those trained with standard self-supervised and supervised objectives well those trained for robustness. Right: LCL- V2Net also shows significantly increased human behavioral alignment, as measured by error consistency on the OOD recognition task. For reference, on this task, humans achieve OOD accuracy of 0.73 and human- human error consistency of 0.43.
4 Discussion
We have developed novel normative theory for learning carly visual representations without end-to-end backpropagation label supervision. We hypothesize that the reason why state-of-the-arl DNNs have failed to predict responses of neurons in early visual areas is because they are insufficiently constrained. We then proposed solution that imposes these constraints through layerwise complexity-matehnne learning (LCL) that leverages a canonical self-supervised objective at each layer. When applicd in a two-stage architecture (2) we showed that our trained model is more effective predicting V1 neural responses the first layer than other architecturally-matchh neural responses. Furthermore, demonstrated that the LCL- V2 model can be used fixed front-end to train supervised object recognition network (LCL-V2H) V2Net) that is significantly more robust to distribution shifts and aligned with human recognition behavior.
models, and achieves state-of- the-art quantitative predictions of V2
As described in the introduction, there is a substantial literature on using normative principles such as sparsity, redundancy reduction, coding efficiency, or temporal prediction to explain early visual properties. These approaches have been mostly limited to early stages (up to and including cortical area. V1), and those that have shown some qualitative success in reproducing V2-like selectivities, have again not scaled well beyond small image patches (Willmore et al. 2010 Hosoya & Hyvärinen] 2015 Rowekamp & Sharpee 2017 Bányai et al., 2019) have been restricted to texture images (Parthasarathy & Simoncelli 2020). These are significant limitations, given recent work showing the importance of training on diverse natural image datasets for achieving strong biological alignment (Conwell et al. 2022). We show that normative approach driven by redundancy reduction (through the Barlow Twins objective) can overcome some of these limitations under our LCL framework. In Sec. [.4] we also show similar results using the SimCLR objective, but we would like do more comprehensive analysis of how more recent promising self-supervised objectives
(eexaa et. al. 2023 Assran al. 2023) perform. More generally, it will be interesting to whether
13

---

Published in Transactions on Machine Learning Research (06/2024)
results rely on form of redundancy reduction or if other principles like sparsity or temporal prediction can be leveraged in a similar manner.
End-to-end trained networks (both supervised and self-supervised) have provided strong accounts of neural responses in late stages of primate cortex as well as recognition behavior (Schrimpf et al., 2018 Geirhos et al.] 2021 Zhuang et al. (021). While unconstrained task-optimization of these models has been the standard for many years, recent efforts demonstrate that constraints on model capacity can lead to better alignment with aspects of biological representation. For example, [aaebi et al. (2023) show that self- supervised (contrastive) E2E training of shallow-networke account well for neurons in mouse visual cortex (due to the limited capacity nature of mouse visual cortex). In primate visual cortex, Margalit et al. (2023) have shown that self-E2E objectives coupled with a layerwise spatial-smoothness regularizer over neural responses produce topographically-aligeee models of both primary visual cortex and IT cortex. In contrast with these studies, we hypothesize that E2E objectives do not sufficiently constrain intermediate reprosentations, and that such constraints are better imposed locally, via per-layer objective functions that do not propagate gradients between layers.
In the machine learning literature, there are several examples of layerwise training of DNNs (Belilovsky et al, 2019 Löwe et al. 2019: Xiong et al.) 2020 Siddiqui et al.) 2023). These efforts have been primarily focused on demonstrating that layerwise objectives can approximate the performance of corresponding end-to-end
backpropagation when evaluating on downstream visual tasks. A few studies (Illing et al. 2021 Halvagal &
Zenke 2023) have emphasized the biological plausibility of layerwise learning from a theoretical perspective, but were not scaled to large-scale training datasets. More importantly, previous studies have not assessed layerwise learning objectives result in more biologically-aligned networks. Here, we've shown that previous layerwise learning approaches (layewwise Barlow (Siddiqui et al. 2023 and LPL (Halvagal & Zenke 2023)), do not offer the same benefits as our framework. We further demonstrate that canonical ceature-contrastio objective only leads to improved biological alignment when the objective is complerity-matched with the corresponding computational capacity of the model stage (Sce Sec. 3.4)
whether these biologically-plausiin
Dapello et al.] (2020) have shown that, a biologically-inspreed V1 stage greatly improves the adversarial robustness of trained networks. But, closer analysis of their results (along with the evaluations presented here) demonstrate that V1-like front end docs not in fact provide significantly better robustness to image distribution ion shilts or aligument with human object recognition behavior. On the other hand, there have been a number of published networks that demonstrate greatly improved behavioral error consistency (Sec. 3.5 at the expense of worse alignment with biological neural responses. These include model scaling (Dehghani et al. 2023), training with natural video datasets, (Parthasarathy et al. 2023), , and use of alternative training paradigms (Jaini et al. 2023) Xie et al. 2021b] Radford et al.] (202]). Our work provides a step towards resolving this discrepancy, by providing an improved model of early visual areas (specifically area V2) that is accompanied by corresponding improvement in model generalization and behavioral alignment.
5 Limitations and Future Work
We briefly describe some of the limitations of this work and opportunities for future work. First, while we have explored the benefits of layerwise training in a two-stage model, there is opportunity to explore extensions to learning of stages deeper in the visual hierarchy. In order to appropriately scale both the feature complexity (image field-of-view) and spatial deformation strength effectively in more layers, we will need to leverage either larger, scene-level images (Xie et al., 2021a) or natural video datasets (Gordon et al. 2020: Parthasarathy et al.] 2023). For many ycars, improvements in task performance of deep networks was correlated with improved predictions of neurons in late visual areas, but the most recent task-optimized networks have shown degradation of neural predictivity (Linsley et al. 2023). The extension of our method to deeper layers may help to address these inconsistencies.
Second, the examples and comparisons in this article focused on a single architecture (AlexNet), but we believe the cuuplexity-matchaee property is of broader applicability. Extending it, however, will require development of more quantitative measures of 1) image content complexity that can be used in place of the current 'patch-size' proxy, and 2) computational capacity of a neural network stage, depending on the specific computations (e.g., number and size of filter kernels, choice of nonlinearity, etc). This is especially important
14

---

Published in Transactions on Machine Learning Research (06/2024)
transformer (Dsoovisskiy et al. 2020) networks. These contain additional computational elements such as "skip connections" and spatially-global computations, making it difficult to appropriately complexit-mmatee an objective with a given layer in these architectures.
for extending recent alternative architectures such residual (He et al. 2016
Finally, from a neuroscience perspective, we see a number opportunities for enhancing and extending the current framework. On the theoretical side, although our layerwise learning is arguably more biologically- plausible than standard supervised, self-supervised, within-layer dependencies. We hope to leverage recently developed methods (e.g., Illing et al. (2021) to bridge this gap in biological plausibility. Experimentally, the current results are also limited to few evaluations on dataset of about 100 neurons and their responses to naturalistic texture and spectrally matched noise images (Freeman et al. 2013 Ziemba et al. 2016). While this dataset informative, it will be important compare the LCI.VV model V2 responses on a more diverse set of stimuli. Perhaps more exciting is the possibility that clucidation of the structure and response selectivities the learned LCL-V2 model will reveal new organizing principles for understanding the mysteries of primate visual area V2 and beyond.
adversarial training, it still relies implausible
Acknowledgments
We thank Corey Ziemba and the Movshon lab at NYU for providing access to the raw electrophysiology dataset used this study, We thank Martin Schrimpf for help with and access to the BrainScore implemen- tation and data splits used in the official benchmark. We also thank Pierre-Etienne Fiquct, Manu Raghavan and Thomas Yerxa for helpful discussions during the development of the work. This work was partially supported by NIH grant R01 EY022428 to EPS.
15

---

Published in Transactions on Machine Learning Research (06/2024)
References
Edward Н Adelson and James R. Bergen. Spatiotemporal energy models for the perception of motion. Josa 2(2))284-229, 1985.
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Sell-supervieel learning from images with joint-embedding predictive archi- tecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 15619-15629, 2023.
Joseph Atick and A Norman Redlich. Towards a theory of early visual processing. Neural computation, 2 (3):308 320, 1990.
Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. Advances in Neurol Information Processing Systems, 35:26671- 26685, 2022.
Mihály Bányai, Dávid G Nagy, and Gergo Orbán. Hierarchical semantic compression predicts texture selectivity in early vision. In Proceedings of the Conference on Cognitive Computational Neuroscience, 2019.
Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Greedy layerwise learning scale to ima- genet. In International conference on machine learning, pp. 583 593. PMLR, 2019.
Anthony J Bell and Terrence J Sejnowski. The "independent components" of natural scenes are edge filters. Vision reseorch, (7(23):3327-333 1997.
Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. Advances in neural information processing systems, 19, 2006.
Alexander Berardino, Valero Laparra, Johannes Ballé, and Eero Simoncelli. Figen-distortions of hierarchical representations. Advances in neural information processing systems, 30, 2017.
Jeffrey Bowers, Gaurav Malhotra, Marin Dujmović, Milton Llera Montero, Christian Tsvetkov, Valerio Biscione, Guillermo Puebla, Foderico Adolfi, John E Hummel, Rachel Hcaton, al. Deep problems with neural network models of human vision. Behavioral and Brain Sciences, pp. 1 74, 2022.
Charles F Cadieu and Bruno A Olshausen. Learning intermediate-level representations of form and motion from natural movies. Neural computation, 24(4):827-88) -66,, 2012.
Matteo Carandini, David J Heeger, and J Anthony Movshon. Linearity and normalization in simple cells of the macaque primary visual cortex. Journal of Newroscience,
1997.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In The Eurooean Conference on Computer Vision (ECCV), September 2018.
Peijie Chen, Chirag Agarwal, and Anh Nguyen. The shape and simplicity biases of adversarially robust imagenet-trained cnns. arXiv preprint arXiv:2006.0953 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive lcarning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020b.
Colin Conwell, Jacob S Prince, Kendrick N Kay, George A Alvarez and Talia Konkle. What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines? BioRaie, pp. 2022 03, 2022.
Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David Cox, and James DiCarlo. Simu- lating a primary visual cortex the front of cnns improves robustness to image perturbations. Advances in Neural Information Processing Systems, 33:13073 13087, 2020.
16

---

Published in Transactions on Machine Learning Research (06/2024)
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. arXiv preprint arXie:2302.05442, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXir:2010. 11929, 2020.
Rodney J Douglas, Kevan AC Martin, and David Whitteridge. canonical microcircuit for neocortex.
Newral computation, :44 48844880888888-8888 1989.
Lyndon Duong, Kathryn Bomnen, William Broderick, Pierre-Friemme Fiquet, Nikhil Parthasarathy, Thomas Yerxa, Xinyuan Zhao, and Eero Simoncelli. Plenoptic: A platform for synthesizing model-optimized visual stimuli. Journal of Vision, 239))55822-5822, 2023.
Yasmine El-Shamayleh, Romesh Kumbhani, Neel Dhruv, and Anthony Movshon. Visual response properties of vl neurons projecting to v2 in macaque. Journal of Neuroscience, 33 (44)):16594-1660))) 2013.
Jenelle Feather, Guillaune Leclere, Aleksander Mądry, and Josh H McDermott. Model melamers reveal divergent invariances between biological and artificial neural networks. Nature Neuroscience, pp. 1-18, 2023.
Thomas Fel, Ivan Felipe, Drew Linsley, and Thomas Serre. Harmonizing the object recognition strategies of deep neural networks with humans. Xve preprint orXiv:2211. Xii:22211004333 2022.
Ronald Aylmer Fisher et al. 014: On the" probable error' of a coefficient of correlation deduced from a small sample. 1921.
KH Foster, James Gaska, M Nagler, and DA Pollen. Spatial and temporal frequency selectivity of neurones in visual cortical arcas v1 and v2 of the macaque monkey. The Journal of physiology, 365(1):3311366 1985.
Jeremy Freeman and Ecro P Simoncelli. Metamers of the ventral stream. Nature neuroscience, 14(9): 1195-1201, 2011.
Jeremy Freeman, Corey M Ziemba, David Heeger, Eero P Simoncelli, and J Anthony Movshon. A functional and perceptual signature of the second visual area in primates. Nature neuroscience, 16(7):974, 2013.
Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffectod by shift in position. Biological cybernetics, 36(4):193-25) 202, 1980.
Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann Lecun. On the duality between contrastive and non-contrastive self-supervised learning. arXiv preprint orXiv:2206.0955 2022.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shorteut learning in deep neural networks. Nature Machine Intelli- gence, 2(11):665-675 673, 2020a.
Robert Geirhos, Kristof Meding, and Felix A Wichmann. Beyond accuracy: quantifying trial-by-trial be- haviour of cnns and humans by measuring error consistency. Advances in Neural Information Processing Systems, 33:13890- 111190-13902,,,,,-11101191099990 2020b.
Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. Advances in Newral Information Processing Systems, 34:23885-23899, 2021.
Masoud Ghodrati, Amirhosscin Farzmahdi, Karim Rajaci, Reza Ebrahimpour, and Scyod-Mahdi Khaligh- Razavi. Feedforward abject-vision models only tolerate small image variations compared to human. Fron- tiers in compulational newroscience, 8:74, 2014.
17

---

Published in Transactions on Machine Learning Research (06/2024)
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. International Conference on Learning Representalions (TCLR), 2018.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint Xrv: 1412.1122... 2014.
Daniel Gordon, Kiana Ehsani, Dicter Fox, and Ali Farhadi. Watching the world go by: Representation learning from unlabeled videos. arXiv preprint or Xix:200003. Xiw:2003.07900, 2020.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 33, 2020.
Manu Srinath Halvagal and Friedemann Zenke. The combination of hebbian and predictive plasticity learns invariant object representations in deep sensory networks. Nature Neuroscience, pp. 1-10, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pallern recognilion, pp. 770 778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. arXiv preprint arXi::2111.06777 2021.
David Heeger, Ecro P Simoncelli, and J Anthony Movshon. Computational models of cortical visual processing. Proceedings of the National Academy of Sciences, 93(2):623 627, 1996.
Olivier J Hénaff, Robbe LT Goris, and Eero P Simoncelli. Perceptual straightening of natural videos. Nature neuroscience, 22(6) 984 1984-991,40 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out- of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1340-83450 2021.
Katherine Hermann and Andrew Lampinen. What shapes feature representations? exploring datasets, architectures, and training, Advances in Newral Information Processing Systems, 33:9995 10006, 2020.
Geoffrey E Hinton, Simon Osindero, and Yee-Whye Tch. A fast learning algorithm for deep belief nets. Newral computation, 18(7):1527-1554, 2006.
Haruo Hosoya and Aapo Hyvärinen. A hierarchical statistical model of natural images explains tuning properties v2. Journal of Neuroscience, 35(29):10412 10428, 2015.
Patrik O Hoyer and Aapo Hyvärinen. A multi-layer sparse coding network learns contour coding from natural images. Vision research, 42(12):1593 110---659-----,- 2002.
Bernd Minin Jean Ventura, Guillaume Bellec, and Wulfram Gerstner. Local plasticity rules can learn deep representations using self-supervised contrastive predictions. Advances in Neural Information Processing Systems, 34:30365-30379, 2021.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint
2015.
Priyank Jaini, Kevin Clark, and Robert Geirhos. Intriguing properties of generative classifiers. arXiv preprint arXiv:230..1677,, 2023.
Li Jing. Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arxin:2 10. 09348, 2021.
18

---

Published in Transactions on Machine Learning Research (06/2024)
Kamila M Jozwik, Nikolaus Kriegeskorte, and Maricke Mur. Visual features stepping stones toward seman- tics: Explaining object similarity in it and perception with non-negative least squares. Neuropsychologia. 83:201 226, 2016.
Yan Karklin and Michael $ Lewicki. Emergence of complex cell properties by learning to generalize in natural scenes. Nature, (57(5225):83 86, 2009.
Yan Karklin and Eero Simoncelli. Efficient coding natural images with population of noisy linear- nonlinear neurons. Advonces in neural information processing systems, 24, 2011.
Saced Reza Kheradpisheh, Masoud Ghodrati, Mohammad Ganjtabesh, and Timothée Masquelier. Deep networks can resemble human feed-forward vision in invariant object recognition. Scientific reports, 6(1): 32672, 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arv preprint arXiv:1412.090 6980, 2014.
Alex Krizhevsky, Πya Sutskever, and Geoffrey Hinton. Imagenct classification with deep convolutional neural networks. In Advances in neurul information processing systems, pp. 1097-1108 2012.
Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Rishi Rajalingham, Ha Hong, Najib Majaj, Elias Issa, Pouya Bashivan, Jonathan Prescott-Roy, Kailyn Schmidt, al. Brain-like object recognition with high- performing shallow recurrent anns. Advances in neural information processing systems, 32, 2019.
Md Nasir Uddin Laskar, Luis Gouzalo Sanchez Giraldo, and Odelia Schwartz. Deep neural networks capture texture sensitivity in v2. Journal of vision, 20(7):21-1, 2020.
Peter Lennie. Single units and visual cortical organization. Perception, 27(())889999) 935, 1998.
Jonathan B Levitt, Daniel C Kiper, and J Anthony Movshon. Receptive fields and functional architecture macaque v2. Journal of neurophysiology, 71(6):2517 2542, 1994.
Zhaoping Li. theory of the visual motion coding in the primary visual cortex. Neural computation, 8(4): 705 (00-730,100 1996.
Drew Linsley, Ivan F Rodriguez, Thomas Fel, Michael Arcaro, Saloni Sharma, Margaret Livingstone, and Thomas Serre. Performance-optimizee deep neural networks are evolving into worse models of inferotem- poral visual cortex. arXiv preprint arXim:2306.0377 2023.
Sindy Löwe, Peter O'Connor, and Bastiaan Veeling. Putting an end to end-to-end: Gradient-isolated learning of representations. Advances in neural information processing systems, 32, 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXie preprint Xixi 1766 06083, 2017.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. Tn Proceedings of the European conference on computer vision (ECCV), pp. 181 196, 2018.
Najib J Majaj, Ha Hong, Ethan A Solomon, and James J DiCarlo. Simple learned weighted sums of inferior temporal neuronal firing rates accurately predict human core object recognition performance. Journal of Newroscience, 35(39):13402-13411 2015.
Eshed Margalit, Hyodong Lee, Dawn Finzi, James DiCarlo, Kalanit Grill-Spector, and Daniel LK Yamins. A unifying principle for the functional organization of visual cortex. bioRziu, pp. 2023-05, 2023.
Judith McLean and Larry Palmer. Contribution of linear spatiotemporal receptive field structure velocity selectivity of simple cells in area. 17 of cat. Vision research, 29(6):675 679, 1989.
19

---

Published in Transactions on Machine Learning Research (06/2024)
Aran Nayebi, Nathan CL Kong, Chengxu Zhuang, Justin L Gardner, Anthony M Norcia, and Daniel LK Yamins. Mouse visual cortex as limited resource system that self-learns an ecologically-general repre- sentation. PLOS Computational Biology, 19(10)=1011506, 2023.
Esezi Isaac Obilor and Eric Chikweru Amadi. Test for significance of pearson's correlation coefficient. International Journal of Inmovative Mathematics, Statistics & Energy Policies, 6(1):11-23, 2018.
Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607, 1996.
Nikhil Parthasarathy and Eero P Simoncelli. supervipe learning of a belogically-inspired visual texture model. arXiv preprint Xiv:20006. 1000006669977 16976, 2020.
Nikhil Parthasarathy, S. M. Ali Eslami, João Carreira, and Olivier J. Hénaff. Self-supervised video pretraining yiclds human-aligned visual representations, 2023.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, el al. Pytorch: An imperative style, high-performance deep learning library, Advances in neural information processing systems, 32, 2019.
Javier Portilla and Fero P Simoncelli. A parametric texture model based on joint statistics of complex wavelet coefficients. International journal of computer vision, 40(1):49-70, 2000.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMT.R, 2021.
Rishi Rajalingham, Elias B Issa, Pouya Bashivan, Kohitij Kar, Kailyn Schmidt, and James J DiCarlo. Large- scale, high-rcsolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of-the-art deep artificial neural networks. Journal of Neuroscience, 38(33):7255 -2677777757777- 2018.
Alexander Riedel. Bag of tricks for training brain-like deep neural networks. In Brain-Score Workshop, 2022.
Maximilian Riesenhuber and Tomaso Poggio. Hierarchical models of object recognition in cortex. Nature neuroscience, 2(11):1019-1025, 1999.
Dario L Ringach. Spatial structure and symmetry of simple-cell receptive fields in macaque primary visual cortex. Journal of newrophysiologu, 88(1):455 463, 2002.
Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanic Jegelka, and Suvrit Sra. Can contrastive learning avoid shortcut solutions? Advances in neural information processing systems, 34:4974-4986, 2021.
Ryan J Rowekamp and Tatyana O Sharpee. Crost-orientation suppression visual area v2. Nature com- munications, 8(1):15739, 2017.
Evgenia Rusak, Lukas Schott, Roland S Zimmermann, Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. A simple way to make neural networks robust against diverse image corruptions. In Compuler Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part III 16, pp. 53 69. Springer, 2020.
S. Sanghavi and J. DiCarlo. Sanghavi2020: Documentation pdf of dataset. https://a01.000/11..17000
DSF ID/ CHW Nov 2021.
S. Sanghavi, K. M. Jozwik, and J. J. DiCarlo. Sanghavijozwik2220: Documentation pdf of dataset. https: //doi. org/ 10. 17605/08F I0/FHY36, Nov 2021a.
S. Sanghavi, N. A. R. Murty, and J. J. DiCarlo. Sanghavimurty2020: Documentation pdf of dataset.
https://dbi.org.. 17605/OSF IO/FCHME Nov 2021b.
20

---

Published in Transactions on Machine Learning Research (06/2024)
Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J Majaj, Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, et al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRriv, pp. 407007, 2018.
Martin Schrimpf, Jonas Kubilius, Michael J Lee, N Apurva Ratan Murty, Robert Ajemian, and James J DiCarlo. Integrative benchmarking to advance neurally mochanistic models of human intelligence. Neuron, 2020. URL URRthttps:: http://wwwwwalllloooo///oooooa// 6273 (2))
Odelia Schwartz and Eero P Simoncelli. Natural signal statistics and sensory gain control. Nature neuro- science, 4(8):819-822 2001.
Vaishaal Shankar, Achal Dave, Rebecca Roclofs, Deva Ramanan, Benjamin Recht, and Ludwig Schmidt. Do image classifiers generalize across time? In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 9661-9669, 2021.
Robert Shapley and Jonathan D Victor. The contrast gain control of the cat retina. Vision research, 19(4): 431-434, 1979.
Shoaib Ahmed Siddiqui, David Krueger, Yann LeCun, and Stéphane Deny, Blockwise self-supervised learning at scale. arXiv preprint arXi:2302.01647, 2023.
Ecro P Simoncelli and William T Freeman. The stecrable pyramid: A flexible architecture for multi-scale derivative computation. In Proceedings., International Conference on Image Processing, volume 3, pp. JEEE, 1995.
Ajay Subramanian, Elena Sizikova, Najib J Majaj, and Denis G Pelli. Spatial-frequency channels, shape bias, and adversarial robustness. Xi preprint arXiv: 13190, 2023.
Ilia Sucholutsky, Lukas Muttenthaler, Adrian Weller, Andi Peng, Andreea Bobu, Been Kim, Bradley C Love, Erin Grant, Jascha Achterberg, Joshua B Tenenbaum, et al. Getting aligned on representational alignment. arXiv preprial arXiv:2310. 13018, 2023.
Christian Szegedy, Wojeiech Zaremba, Tlya Sutskever, Joan Bruna, Dumitru Erhan, Tan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint orXio:1312.6199, 2013.
Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Bonch, and Patrick McDaniel. The space of trans- ferable adversarial examples. arXiv preprint arXiv: orXiv:1704.033 704.03453, 2017.
Leslie G Ungerleider and James V Haxby. "what'and 'where'in the human brain. Current opinion in neurobiology, 4(2):157-165, 1994.
Hans Van Hateren and Arjen van der Schaaf. Independent component filters of natural images compared with simple cells in primary visual cortex. Proceedings of the Royal Society of London. Series B: Biological Sciences, (1394))333 366, 1998.
Ben DB Willmore, Ryan J Prenger, and Jack L Gallant. Neural representation of natural images in visual area v2. Journal of Neuroscience, 3016))210222114 2010.
Laurenz Wiskolt and Terrence J Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Newral computation, 14(4):715-770, 2002.
Jiahao Xie, Xiaohang Zhan, Ziwei Liu, Yew Ong, and Chen Change Loy. Unsupervised object-level repre- sentation learning from scene images. Advances in Neural Information Processing Systems, 34, 2021a.
Zhenda Xie, Yutong Lin, Zhuliang Yao, Zheng Zhangg Qi Dai, Yue Cao, and Han Hu. Self-supervised learning with swin transformers. arXiv preprint arXiv:2105.04555 2021b.
Yuwen Xiong, Mengye Ren, and Raquel Urtasun. Loco: Local contrastive representation learning. Advances in neural information processing systems, 18:11142-11155 2020.
21

---

Published in Transactions on Machine Learning Research (06/2024)
Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-opimmizzd hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 111((3))86199 8624, 2014.
T E Yerxa, Y Kuang, E. P Simoncelli, and SY Chung. Efficient coding of natural images using maximum manifold capacity representations. In Adv. Neural Information Processing Systems (NeurIPS), volume 36, New Orleans, 2023.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphanc Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference Machine Learning, pp. 12310-12320. PML.R,
2021.
Richard Zhang. Making convolutional networks thift-invariant again. In International conference on machine learning, pp. 7324-7334. PMLR, 2019.
Chengxu Zhuang, Siming Yan, Aran Nayebi, Martin Schrimpf, Michael C Frank, James J DiCarlo, and Daniel LK Yamins. Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences, (e013)1100,41,,11,,, 2021.
Corey M Ziemba. Neural representation and perception of naturalistic image structure. Unpublished docloral dissertation). New York: Center for Neural Science, New York University, 2016.
Corey M Ziemba, Jeremy Freeman, J Anthony Movshon, and Eero P Simoncelli. Selectivity and tolerance for visual texture in macaque v2. Proceedings of the National Academy of Sciences, F3140-0333 2016.
22

---

Published in Transactions on Machine Learning Research (06/2024)
A Appendix
A.1 Architecture details
We summarize the details of all architectures used in both the biolonec evaluations and the human behavior evaluations.
A.1.1 Architectures
For the LCL-V2 two-stage model, and for all of the ablation studies in Sec. 3.4 use the following architecture:
Conv2d(in channels-3, out channels- 64, ks=11, stride-4, padding-5, 5, padding mrde-'reflect')
BatchNNorm2((44)
Layer
ReLU()
BlurMaxPPol22(ss-3, 2 padding-1)
Conv2d(i__channes=66A channels=64, out channels=192, ks=5, stride=1, padding-2, padding mode='refiectt)
BatchNorm22(112)
Layer 2
ReLU() BlurMaxPool2d(tts3, stride=2, padding-1)
Table A.1: LCL-V2 Two-layer Architecture. We use the same channel dimensions and non-linearities take from the first two layers of the AlexNet architecture (along with BatchNorm layers). In addition, because train model with small image patches, aliasing artifacts impact model responses more than with large images. As result, replace standard MaxPooling with anti-aliasing blurring followed by max-pooling (as done in Zhang 2019
As shown in Fig. 3 we use projector networks g(0)).9(02) during training to encourage learning of 'equiv- ariant' representations:
Linear(in 6hannels=64, out channels=64) BatchNNorm2d(64)
g(0,)
ReLU()
Linear (in channels=64, out_ channels=204s)
Linear (in channels=192, out channels= 192)
BatchNorm22((19)
s(02)
ReLU() out mnt_cchaneks2208))
Table A.2: LCL-V2 projector network architectures. Projector networks used during self-supervised
pre-training for each layer are MLP networks with single hidden layers. Following Siddiqui et al. (2023) we use output dimensionalities of 2048 for each projector.
For the full CL-VVNNe architecture of Sec. 3.5 we fix the LCL-V2 block and train the following subsequent stages:
23

---

Published in Transactions on Machine Learning Research (06/2024)
Conv2d(192, 384, ks=3, padding-1) BatchNorm2dd384)
ReLU()
Conv2d(384, 256, ks=3, padding-1)
Batch(Norm22(566)
Downstream features
ReLU()
Conv2d(256, 256, ks=3, padding=1)
BatchNorm2d(256)
ReLU()
MaxPool2((ks=3,,ppm stride-=)
AdaptiveAvyPPol2d(..
Dropout(p=0.5)
LDnear(D, 4096)
BatchNorm2((4006)
ReLU()
Classifier
Dropout(p=O..
Linear(4096, 4096)
BatchNNor22d(4096)
ReLU()
Linear 1000)
Table A.3: L2LeV2Net Downstream Architecture. On top of the LCL-V2 stage, we train the subse- quent stages indicated here based on the AlexNet architecture. We include BatchNorm layers as we find this speeds up convergence.)
A.1.2 Vo-SteerPyr+PPol baseline (Hand-crafted)
For the neural response prediction evaluations, we implement baseline model using the Steerable Pyramid (Simoncelli & Freeman 1995). We use a scale, 4 orientation complex pyramid (based on 3rd-order oriented derivative filters, and their Hilbert Transforms) as implemented in the Plenoptic package (Duong et al. 2023 We rectify (Re..U) both the real and imaginary channels to generate total of 'simple cells.. We additionally create 20 'complex-cell' channels by computing the modulus of each complex-valued filter response: Toomplex Vreeal + + mmgg For the V1-baseline model, subsample the output spatial feature map by a factor of 4 to reduce the total number of responses. For the V2-baseline model, we use the response of the V1-stage after applying L2 spatial energy pooling in a channc-inndpenndet way with a 3x3 kernel and additional subsampling by a factor 2. Given a 200x200 grayscale input image, the V1-layer response vector therefore has shape (60, 50, 50) while the V2-layer response vector has shape (60, 25, 25).
A.1.3 VOneNet (Hand-crafted + E2E)
We use the VOneNet-AlexNet network developed in Dapello et al. (2020). We note that this network does not use BatchNorm layers; however, due to inability to re-train this model effectively we use the published pre-trained version. Briefly, this network consist of a VOneBlock front-end model that contains 256 channels (128 simple cell, 128 complex cell) created from a Gabor filter bank basis set with parameters sampled from distributions of recorded macaque V1 cells. This front-end is followed by the full standard AlexNet network architecture from the Pytorch library (aasze al. 2019).
A.1.4 Barlow (layerwise)
We use the same architecture as our LCL--V block defined above. However, we utilize the E2F, augmentation training parameters defined in Zbontar et al. 2021 and Siddiqui et al. (0023 (see Sec. A.2 for details).
We use this model for comparison in both neural response prodiction (Sec. 3.1 and as bascline for model ablations (Sec. 3.4
24

---

Published in Transactions on Machine Learning Research (06/2024)
A.1.5 LPL (layerwise)
We use the training code provided in Halvagal & Zenke [2023). While we attempted to re-train the LPI. method on the Image.Net-Ie dataset, we found that this network would not converge. As a result, we use the AlexNet model trained on the STI-10 dataset for 800 epochs (as done in the original work).
and methods
We only use this model for comparison in the neural response prediction experiment (Sec. additional layerwise learning baseline.
3.1 to provide
A.1.6 BarlowTwins (E2E)
Due to the fact that there not an available pre-trained AlexNet-based version of the Barlow Twins E2E training method from Zbontar al. (2021), we pre-train our own version based on the standard AlexNet architecture (with BatchNorm layers after each convolution/lines /linear layer). We pool the final 'feature' layer such that each image is represented by single 256-d responses vector. As implemented in Zbontar et al.) (2021), before the loss computation, this vector is propagated through a standard MLP projector network with 2 hidden layers. We varied different projector sizes and found the best projector setting to be one with dimensionalitics: (1024, 1024, 1024) for the hidden layers and output layer. We pre-train this network for 100 epochs.
For the comparisons in Sec. 3.5 we train a classifier stage (as defined in Table A.3 that operates on the output of the fixed network.
A.1.7 Supervised AlexNet
For the results this work, we lested two variations of the slandard AlexNet architecture (one with Batch- Norm [Toffe & Szegody, 2015) and one without). Interestingly, although we found benefits of BatchNorm in convergence for our LCL training, we find that the standard AlexNet, without BatchNorm, provides slightly better account of the neural data (and similar performance the OOD behavioral benchmarks), As result, use the standard network (without BatchNorm) for comparison. Because this network is fully- supervised on the ImageNet-1K recognition task (provided in the Pytorch library), we do not perform any extra training for the results in Sec. 3.5
A.1.8 L2-AT (e (33..))
We use existing fully pre-trained version of the AlexNet architecture, trained with L2-AT adversarial training (Madry et al., 2017/. We use the specific pre-trained model from Chen et al. (2020a), which uses = 3.0 as the perturbation threshold. Because this network is fully-supervised on the ImageNet- recognition task (provided the Pytorch library), we do not perform any extra training for the results in Sec.
A.1.9 ANT
We use the code provided at https:///itthb. http:s//gggghbooo////tggg//////e ga to train the standard AlexNet model with the adversarial noise training method provided in Rusak et al. (2020). This method differs from the L2-AT standard adversarial training as uses a parameterized noise distribution (Gaussian) rather than arbitrary pixel perturbations within fixed budget.
A.2 Augmentation details
Standard E2E training
The standard E2E augmentation scheme that we use both train the Barlow (E2E) (Zbontar al., 2021) and Barlow (layerwise) (Siddiqui et al. (2023) baseline models defined as follows: Each image randomly augmented by composing the following operations, each applied with given probability:
25

---

Published in Transactions on Machine Learning Research (06/2024)
1. random cropping: random patch of the image is selected, whose area is uniformly sampled in (s AA, where A is the area of the original image, and whose aspect ratio is logarithmically sampled in [3/4, 4/3]. s is a scale parammer set to 0.08. The patch is then resized to 224 x224 pixels using bicubic interpolation:
horizontal flipping:
3. color jittering: the brightness, contrast, saturation and hue are shifted by uniformly distributed offset;
color dropping: the RGB image is replaced by its grey-scale values;
gaussian blurring with a 23x23 square kernel and standard deviation uniformly sampled from 2.01;
6. solarization: point-wise color transformation x + x- Lec0.5 +1 (1- x) 1.20.5 with pixels r in [0, 1].
The augmented frames x and xh result from augmentations sampled from distributions AA and Ag respec- tively. These distributions apply the primitives described above with different probabilities, and different magnitudes. The following table specifies these parameters.
Parameter
AA
A
Random crop probability
1.0
Flip probability
Color jittering probability
0.8
Color dropping probability
Brightness adjustment max
0.4
Contrast adjustment max
0.4
Saturation adjustment max
0.2
Hue adjustment max Gaussian blurring probability
0.1
1.0
0.1
Solarization probability
0.0
0.2
LCL-V2 Augmentations
Given the limited capacity of two-stage network, drastically reduce the number and strength of the augmentation set. As described in the main text, additionally complexity-matdha to generate the inputs used to train each layer. For our training, each image is first resized such that the smallest. side is 224 pixels. A center crop is then selected from this image, and is randomly augmented by composing the following operations, each applied with a given probability.:
the augmentations
random cropping: a random patch this contral crop is selected, whose area is uniformly sampled [s-A, A., where the area of the original image, and whose aspect ratio is logarithmically sampled in [0.9, 1.1]. is a scale rrperpparametr set different value for each layer of the LCL-V2 architecture. The patch is then resized p xp where p is again dependent on the layer.
2. contrast and luminance jittering: the brightness and contrast are shifted by uniformly distributed offset.
Gaussian noise: additive Gaussian noise added independently each channel of the RGB image. The noise is generated to be mean with random standard deviation uniformly sampled from the range (0.04, 0.1).
The parameters to generate the first layer patches of and of are defined below:
26

---

Published in Transactions on Machine Learning Research (06/2024)
Parameter
Aa
Ab
Central crop size
56
Minimum random crop scale
0.6
Random crop resize output size
48
Color jittering probability
0.8
Color dropping probability
0.2
Brightness adjustment max
0.2
Contrast adjustment max
0.2
Gaussian noise probability
1.0
0.0
The parameters to generale the second layer patches x and x2 are defined below:
Parameter
AA
AB
Central crop size
112
Minimum random crop scale
0.3
Random crop resize output size
96
Color jittering probability
0.8
Color dropping probability
0.2
Brightness adjustment max
0.2
Contrast adjustment max
0.2
Gaussian noise probability
1.0
0.0
A.3 Neural evaluation details
A.3.1
Stimulus pre-pro pre-processing
We use the stimuli from Freeman et al. 2013 Ziemba et al. (016)), which consist of 225 naturalistic texture images from 15 texture families and 225 corresponding spectrally-matchel noise images. In the experimental
Texture
Noise
Figure A.1: Example samples of each of the 15 texture families and their corresponding spectrally-matchel noise models, (Taken from Freeman et al. 2013)).
protocol used in Freeman et al. 2013 images were presented at a 4 deg field-of-view to the individual V1/V2 neurons. Since models have arbitrary input field-of-view (viewing distance), we use the pre-processing in the BrainScore benchmark ltttps://githu: com/ http:///innnoorrrraiiaaiaaaeeaee
which sweeps over field-of-view find the optimal viewing distance for predicting the neural responses. This cross-validation is done on a held-out validation set. For example, for our model (and most ImageNet trained models), we find that an input size of 224 pixels and 8 degree field of view is optimal. As a result, the original texture stimuli are resized and placed in the central 4-degrees (112 pixel diameter). This image is then padded with gray values beyond the central 4 degrees.
A.3.2 BrainScore neural prediction
Following the standard BrainScore pipeline, we preprocess the neural responses binning raw spike counts in 10ms windows. For cach window over the 50ms 150ms range post stimulus presentation, we average these spike counts over 20 repeated stimulus presentations. To evaluate neural predictivity we use the API provided in http://hhh....o/oomoooo/////aaaa
We show scores the private split of the
27

---

Published in Transactions on Machine Learning Research (06/2024)
data which consists of approximately 70% of the original images (official BrainScore split). Briefly, N x xDm model responses are regressed onto the N x xD, neural responses (V1: D, 102, V2: D, = 103). This is done using the PLS regression method with 25 components. The regression is computed using 10-fold cross-validation and pearson correlations 7n (between model predictions and ncural responses across all test images) are oblained for each neuron and for each split. A measure of internal consistency Veeil,n is also computed by splitting neural responses half across repeated presentations of the same image and computing the Pearson correlation coefficient between the two splits across images for each neuron. For more details, see [Schrimpf et al. (2018).
For the overall scores presented in Fig. 4 (Left), The final explained variance for given model is calculated as median, (rat///rrrr///./.rr computed over the specific subset of neurons identified as 'V1-like' or 'V2-like' (based on the V1-SteerPyr model explained variances).
For the subset explained variance scores in Fig. (Right), these medians are
However, medians over neurons can potentially obscure the performance of any given model we also provide additional figure comparing the per-neuron predictions of the LCL-V2 model against the L2-AT (e (30) and Supervised AlexNet models.
1.0
1.0
0.8
0.8
LCL-V2 (Ours)
0.2
0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8 1.0
L2-AT (E (E=3.00 3.0)
Supervised
Figure A.2: Comparing per neuron predictions via the BrainScore methodology. We compare the explained variance (normalized by an estimated noise ceiling) of each of the 103 V2 newrons of our LCL-V2 model against other high-performing models. Left: LCL-V2 vs. L2-AT comparison shows that although there are a reasonable fraction of neurons which the L2-AT network predicts better (40%), there is a larger fraction 60%) which are better explained by our model. Right: LCI.-V2 vs. Supervised comparison shows that for nearly all neurons (>75%), the explained variance of our model is comparable or greatly exceeds that of the Supervised network.
We further estimate the internal consistency between neural responses by splitting neural responses in half across repeated presentations of the same image and computing Spearman-Browd Brown-corrected Pearson corre- lation coefficient bet the two splits across images for each neuroid.
A.3.3 Texture Modulation Analysis
The analyis on texture modulation uses the same stimuli and image pre-processing. For the neural responses, we follow the same pre-processing outlined above; however, instead of binning each neural responses over the fixed 50ms to 150ms window post presentation, we use the method in Freeman et al. 22113 which still selects a 100ms window (but now aligned to the specific response-latency of each neuron). This is done for consistency with the measured texture modulation indices in Freeman it significantly changes the results.
(2013). but we do not find that
28

---

Published in Transactions on Machine Learning Research (06/2024)
As discussed above, for the regression analysis we provide explained variance scores relative to the noise ceiling (internal consistency). This is done to avoid biasing results based on predictions for neurons that are inherently unpredictable. However, for this analysis a noise ceiling on texture modulation cannot be calculated in the same way. For simplicity, we therefore calculate the texture modulations only for biological neurons which have an noise ceiling above a threshold (0.4). This excludes the boltom 10 (out of 103) neurons from the texture modulation calculation. Note, we find the results to be relatively robust to the choice of threshold.
Finally to test significance of the Spearman correlations between texture family modulation rank correlations, we use the following method outlined in Obilor & Amadi (0018)
We first transform the correlations to assume a normal distribution via Fisher's z-transformation Iisher et al. 1921]: 24 = arctanh(p:). Given that our sample size is N=15 (15 texture families), we can calculate the standard error on the difference of correlations: SEs, = V Then, the test statistic Stest = SEA, Finally, the p-value can be calculated as p 2* (1 - (L-CDF(C CDF(((tee))))
21-22
A.4 OOD and Human Behavior Benchmark
Dataset information. We evaluate accuracy and human error consistency using the code from https: We report the OOD accuracy (averaged over samples and distortions) on the full dataset from Geirhos et al. (2021), which consists of 17 OOD distor- tions applied to ImageNet images. These distortions include: cue-conflict, edge drawing, sketch drawing, silhouettes, texture stylization, three levels of eidolon noise, color jitter, false colors, contrast, low/high /high-pass noise, uniform noise, phase scrambling, spectral power equalization, and rotation. We additionally report the behavior error consistency metric on the recognition task with these distorted images.
//github. cctm///etiggellb//moeeeeeeeeeeee www.raaaunaaaaeeeeeeeeeeeeeeeeee
Bchavior error consistency metric. This metric, derived in Geirhos et al. (200b)) measures whether there is above-chance consistency with human per-sample recognition decisions. let (s,m(s) be if both a human observer h and decide either correctly or incorrectly on given sample s, and otherwise. The observed consistency Ch,m is the average of (s) over all samples. The error consistency then measures whether this observed consistency is larger than the expected consistency given two independent binomial decision makers with matched accuracy (see Geirhos et al. (2020b) for the details on this exact computation).
A.5 V1 Receptive Field Comparisons
In addition to the overall predictivity of the V1 data, we also use qualitative and quantitative analyses to probe the selectivities of the learned receptive fields in the first convolutional layer of our model. We compare these learned filters to those learned via adversarial-training and standard supervised training.
Filter Visualization. We visualize the 64 filters of each learned model in Fig. A.3 We visualize the filters in grayscale (even though the filters operate on color channels), to draw comparisons specifically between the spatial receptive field propertics. By inspection, none of the models perfectly capture the nature of real V1 receptive fields; however, they all learn a set of reasonably diverse of multi-scale oriented filters. Our model and the L2-AT model seem to better capture the number of cycles within the oriented filters, whereas the supervised network receptive fields are tend to have high frequency and narrow bandwidth. Both LCL-VI and Supervised models, however, seem to learn more localized non-oriented filters than the adversarially-trained model.
V1 Receptive Field Properties. We briefly explore some of the canonical receptive field properties highlighted in the data collected from macaque V1 neurons in Ringach| (2002). For the three models, we fit linear receptive field model to the filters shown in Fig. A.3 by minimizing the mean squared error between a parameterized 2-d Gabor function (product of a sinuoisoidal grating and a Gaussian envelope) and the model receptive field. We remove those cells that were not fit. well.
We first measure the spatial phase of each receptive field and compare these to the distribution of spatial phase of the macacque data (Fig. A.4). None of the models perfectly match the neural data distribution, but our LCI--VI model seems to best capture the relative bi-modal structure around even and odd-symmetry (0
29

---

Published in Transactions on Machine Learning Research (06/2024)
LCL-V1 (Ours)
L2 Robust (E = 3)
Supervised
Figure A.3: Comparing learned receptive fields. The single-layer (layerwise) trained model (LCL-V1), learns a diverse set of oriented receptive fields. All 64 filters are shown for method, adversarial robust training, and standard supervised training. All models learn diverse set of filters, many of them oriented. Both our model and the adversarially-traanee network learn more low-frequency filters than the standard supervised-trannt trriinedd network. Original filters are RGB, but we show grayscale versions here to focus on the spatial structure comparisons.
plotting the size of the receptive field envelope transverse to, and along its oscillating stripes ((2z, ng), respectively), in units of the spatial period of the underlying sinusoid. We see Fig. A.5 that while there are some outliers, our model receptive fields are distributed similarly to the V1 receptive fields measured in macaque. Compared with the other two models, our model also does better job capturing the distribution of 'blob-like' filters near the origin.
and π. 122.. We next compare the receptive field structure via the method in Ringach 22022)
LCL-V1 (Ours)
L2 12-Robust (a 3)
Supervised
30
30
Model Macaque V1
20
20
20
10
10
10
4
m/2
n/4
m/2
n//
n/2
Figure A.4: Comparing spatial phase selectivity between models and macaque V1 neurons. We compare our LCL-V1 receptive field tuning to the equivalent receptive fields extracted from the L2-AT and Supervised networks. Model histograms of neuron spatial phase shown in blue and real macaque data from Ringach (002) is shown in green.
A.6 Additional V2 BrainScore results
We provide additional more detailed results comparing the BrainScore explained variance of our LCL-V2 model against other top models the BrainScore leaderboard. Many of these models much higher capacity architectures, yet still underperform model.
A.7 Additional Human Behavior Results
30

---

Published in Transactions on Machine Learning Research (06/2024)
LCL-V2 (Ours)
12-Robust (E=3) 3)
Supervised
1.5
1.5
1.5
1.0
1.0
1.0
Macaque
Model
0.5
0.5
0.5
0.0
0.0
0.0
0.5
1.0
1.5
0.5
1.0
1.5
0.5
1.0
1.5
Figure A.5: Comparison of learned oriented receptive field sizes to those measured physiolog- ically. Each point corresponds to single cell (model, or biological), whose receptive field was fit with Gabor function (product of sinusoidal grating with a Gaussian envelope). Axes represent the standard deviation of the Gaussian envelope along the directions transverse to, and along, the sinusoidal stripes, in units of the sinusoidal period (referred to as na and fl, respectively, in Ringach (2002)). Model neurons are plotted in blue and Macaque neurons in green.
Method
V2 Ceiled Explained Variance f
LCL-V2
0.417 + 0.008
ResNet50 L2-AT (e = 3.0) Madry et
2017
0.402 + 0.007
AlexNet L2-AT (e 3.0) (Madry et al.
2017
0.397 = 0.007 0.380
VOneResNet-50 L2-AT (Dapello et al.
2020
Bag-of-Tricks Model [Riedel 2022
0.356
AlexNet Supervised Krizhevsky el al.
2012
0.353 1 0.009
ResNet-50 Supervised (He et al. 2016
0.320 0.01
ResNext-101 (32 x 8d) WST. (Mahajan et al.
2018
0.312
Table A.4: LCL-V2 outperforms all existing models for V2 data BrainScore (including those that are non-architecturally matched). The table compares performance in explaining macaque V2 responses for a variely of models that are either stale-of-the-art BrainScore models (Bag-of-Tricks and ResNext-101 WSL) or prior ttat in V2 prediction (ResNet50 L2-AT), For those models for which we were able to obtain the full set explained variance values, we provide standard deviation over 10 cross- validated splits.
Method
IN-1K acc. 1
OOD acc.
obs. consistency
eтrоr consistency
LCL-V2Net
0.527
0.492
0.643
0.211
Barlow Twins Zbontar et al. 2021
0.459
0.451
0.607
0.166
Supervised
Krizhevsky et al. 2012
0.590
0.443
0.597
0.165
VOneNet Dapello et al. 2020
0.491
0.407
0.585
0.168
L2-Robust
Madry et al. 2017
0.399
0.391
0.573
0.176
Table A.5: OOD accuracy and consistency with human judgments 17 different OOD recognition tests, including multiple types of noise, phaso-scrambling and shape-biased stimuli. In both accuracy and human- alignment, our model trained with the LCL-V2 front-end improves over all end-to-end training approaches.
31


Reviews:
[{"review_id": "tjZI0ml8FF", "reviewer": "Reviewer_SmQ8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n 1. Paper is well written and easy to read\n 1. Presents convincing evidence for its claims\n 1. Repeated stacking of normative approaches such as redundancy reduction or sparse coding has thus far not been successful, making the success of this approach a clear strength\n\n\n\n### Weaknesses\n\n 1. Not sure if TMLR is the right venue for this work\n 1. Some methodological details are unclear from the text and could undermine the claims\n 1. No comparison to or discussion of relationship to V2 model by Freeman & Simoncelli", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a layer-wise self-supervised learning paradigm that results in internal representations which are better aligned with visual area V2 than alternative supervised, self-supervised and layer-wise training paradigms using the AlexNet architecture. When using two layers trained in this way as a fixed frontend to AlexNet, it improves out-of-domain generalization and alignment of model predictions with humans in image classification.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall the paper is well organized, well written, easy and fun to read, and makes a very nice contribution. I really enjoyed reading the paper. Nevertheless, I have a few concerns, which I would like to detail below.\n\n\n### 1. Is TMLR the right venue?\n\nThe paper has great potential for moving vision science and computational neuroscience forward. It could also impact machine learning as a field, but in this regard the contribution is less clear. I therefore wonder whether TMLR is the right venue for this paper. While I leave a specific recommendation/decision to the editorial board and/or the authors, let me explain my reasoning:\n\n - I see the main contribution of the paper as providing a potential normative account of primate area V2, which is also evident in most of the results figures (4–7) being related to model-neuron alignment.\n\n - The improved OOD robustness and human alignment (Fig. 8) is of potential interest to the ML community, but it feels more like a corollary than the main result and is only a teaser. The main questions from an ML perspective would be whether this approach (a) can be scaled to more than two layers and (b) also holds for more modern encoder architectures. While the authors acknowledge these limitations, the defer them to future work.\n\n\n\n### 2. Some methodological details are unclear\n\nThere are a few methodological details that I consider somewhat important, but could not resolve from reading the text:\n\n 1. Do I understand the equation on p. 5 correctly that \"Pool\" refers to a global average (or max?) pooling, which contracts the spatial dimensions to $1 \\times 1$? As the input (for layer 1) is $48 \\times 48$ and the conv kernels are $ 11 \\times 11$ with stride 4, it would reduce to $9 \\times 9$ after the convolution, then to $4 \\times 4$ after maxpool, which then gets reduced to $1 \\times 1$ by the global average pooling? In other words, does $f_{\\theta_1}$ include the $3 \\times 3$ max pooling with stride 2 of AlexNet's first layer? Is the same true for the second layer, $f_{\\theta_2}$? I assume it must, because otherwise it could not be used as a frontend to the later layers of AlexNet for Fig. 8? It would be good to be more explicit on this in the manuscript.\n\n 1. The training as depicted in Fig. 3 suggests that both layers are being trained jointly and only the gradient is not passed backwards from layer 2 to layer 1. Is this important or could you also train sequentially where you first optimize the parameters of layer 1 until convergence and then start with layer 2?\n\n 1. How did you choose the input resolution? Cortical receptive fields scale with the eccentricity of the neuron's receptive field, whereas a convnets' receptive field size is given by the kernel size and the learned features depend on the image size and resolution. As e.g. shown by Cadena et al, PLoS CB 2019, the input resolution has a pretty substantial effect on the neural predictivity. How do your results and conclusions change when you increase or decrease the image resolution by, e.g., 50%?\n\n 1. Related to the previous point, I would like to ask whether the (effective) image resolution (after data augmentation) was the same for layer 1/V1 and layer 2/V2 training. I could not completely understand the data augmentation pipeline and how exactly the different scaling/cropping parameters are used for layer 1 vs. layer 2. Suppose you input a single image with an object of, say, size 20 pixels repeatedly to the data augmentation pipeline. Will the distribution of resulting object sizes after applying the data augmentation be identical for layer 1 and layer 2 training? If not, wouldn't this affect your conclusions? Also, if not, how would this be implemented in the brain, where there is only one set of inputs to the eye?\n\n 1. How exactly did you compute the texture modulation index presented in Fig. 5? The way it is defined in the methods section, it should be between –1 and 1, but the figure shows values roughly between –4 and 4. I can see two ways how this could happen: (a) it was normalized/standardized prior to analysis, in which case it would be good to state what exactly was done; (b) activation values could be negative, in which case the denominator could get arbitrarily small and close to zero, which would mean the index is not well defined (it makes sense only for strictly non-negative quantities) and the whole analysis doesn't really make sense.\n\n\n\n ### 3. Comparison to alternative accounts of V2\n\nCould your model be compared to Freeman & Simoncelli's texture model of V2? You present clear evidence that your approach outperforms your baselines, but they are not particularly strong baselines for V2. The paper would be more convincing if it presented some evidence that your account of V2 is competitive with or even better than Freeman & Simoncelli's model, which they showed can account for the texture selectivity of V2 neurons.\n\n\n\n ### Minor comments:\n\n - The first reference to Fig. 3 at the beginning of Section 2 seems too early. If the figure is relevant there already, why not move it there and call it Fig. 2? If it's not relevant there, why cite it and make readers search for it?\n\n - Fig. 2 bottom: Strictly speaking only $f_{\\theta_l}$ depends on $L_l$, not $f_{\\theta_{1,...,l}}$, correct?\n\n - P.6 \"Additionally, as mentioned earlier, much of the biological anatomy and computational theories suggest that the feed-forward aspect of areas V1 and V2 should be explainable by networks with few computational stages. As a result, we hypothesize that the AlexNet architecture can provide a more parsimonious and interpretable model of these areas.\" Could you provide references for this claim? Cadena et al. 2019 [1] suggest it's quite a few nonlinearities already for V1, and Miao & Tong 2023 [2] show that AlexNet needs to be at least modified quite a bit to get there. Also, your explained variance for V1 is only around 50% of the top scores on BrainScore, arguing somewhat against your claim. Maybe I'm overlooking additional evidence that backs your claim or I misunderstood it?\n\n - Fig. 4+5+8: What does the asterisk signal? It's usually used to depict statistical significance, but I cannot find any information on that.\n\n - Section 3.2, p. 10: Can you provide statistics on the difference of rank correlations between the three models? With n=15, it appears to me quite plausible that a difference of 0.59 vs. 0.8 might arise by random chance.\n\n - Fig. 8: If using a bar plot, please start the y axis at zero. Otherwise it's hugely misleading.\n\n \n ### Requested changes\n\nI would like the authors to address my main concerns about lack of clarity regarding the methodology (point 2). Regarding point 3, it would be great if they could provide a comparison, but I realize that it may be challenging and therefore would not insist on such a comparison.\n\n\n\n ### References\n\n\n[1] Cadena et al., PLoS CB 2019, https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1006897\n\n[2] Miao & Tong, bioRxiv 2023, https://www.biorxiv.org/content/10.1101/2023.08.26.554952v1", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zJmKluVPxl", "reviewer": "Reviewer_RvGn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Reviewer limitations:\n\n1. I apologize to the authors and AE for my lateness turning in a review.\n2. I am not versed in this slice of literature, so my estimate of the paper's grounding in the literature is based on its internal evidence.\n\n\nComment to TMLR:\n\nLine numbers make review much easier. Can these be added to the template?\n\n\nGeneral comments:\n\nThis is a very high quality research program and paper, the kind of thing that I wish I was grown-up enough to produce myself. \nIt is well written and easy to follow. \nIt effectively connects and combines neuroscience and ML perspectives. \nIt effectively describes and builds on existing literature. \nIt offers novel and valuable advances, relevant to both biological and ML-centric perspectives. \n\nIs there a publicly-available codebase? That is always a good feature.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper describes a novel and effective way to train layers using only local (layer-specific) information rather than end-to-end backprop. This extra constraint on layer training yields a closer match to biological primate brain layers V1 and V2.\n\nThe paper also gives evidence that this training method, when it uses contrastive objectives, improves network performance in OOD generalization. This has high potential value in a pure ML sense (independent of any biological lens).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As is usual (though lamentable) in reviews, to save time I have not commented on the many features of this work that I found admirable, clever, insightful, etc.\n\nMinor comments:\n\npg 2: Spell out \"IT\"\n\nsection 2.3: typo \"...augmented versions (x_2^A, x_2^B)...\"\n\nsection 2.3: I did not see detail about the projection heads g_theta. Are these trainable, and what is their structure/architecture?\n\nsection \"Neural alignment evaluations\": This holdout set is qualitatively different from imageNet. How does this impact the results and/or their interpretation?\n\nFig 4, 6 (and other bar plots): can you include +/- std devs? This gives a quick sense of \"significance\" of any performance increases. For example, in fig 6, is the difference between LCL-V1 and BarlowTwins within a noise envelope?\n\nFig 4 caption: \"of the best layer is provided\". I was curious which layers turned out to be the best - were these rational/expected?\n\nFig 4 caption \"The minimum explained .... 14%\": These numbers don't seem to jibe with the right hand plot.\n\nsection 3.2, line 2: typos: \"presentin\", citations missing parentheses.\n\nFig 6: The \"untrained\" model seems to do remarkably well. Can you provide some detail as to what \"untrained\" means exactly, and comment on why it appears to do so well? \n\npg 12 last paragraph: typo \"the the\"\n\nI loved the Discussion section - a good framing of the paper's context and implications.\n\nDiscussion, paragraph 2: Perhaps you can discuss how LCL relates to these other normative methods (sparsity etc), since these are all biologically-grounded. LCL also has biological grounding - are the methods mutually exclusive, or different lens on the same thing, or can they be combined, etc.\n\nLimitations etc: \"These networks contain additional computational elements ...\". Are structures like skip connections supported by biology findings, or is this on a separate track for future work, viz pure ML-centric development as opposed to seeking alignment with biology. Both tracks are valid directions given the findings in the paper.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BELBSDaX4W", "reviewer": "Reviewer_1T7Y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Innovative unsupervised approach to DNN training with layerwise complexity matching which demonstrates improved biological data alignment, out-of-distribution generalization and human behavior alignment.\n\nWeaknesses:\n* The study primarily focuses on a two-stage AlexNet model, it’s unclear whether it's effective for deeper and more advanced network architectures.\n* Focused primarily on cortical area V2, with limited exploration of applicability to other areas.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a novel approach to deep neural network training focused on layerwise complexity-matched learning (LCL). This methodology aligns the complexity of self-supervised learning objectives with the computational capacity of each DNN layer. The study showcases that LCL, when applied to a two-stage model, has better prediction of V2 neural responses. Additionally, when this model is used as a front-end for deeper networks in object recognition tasks, it exhibits improved out-of-distribution generalization and alignment with human behavior.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Although the paper is well written overall, I would like to have the following points to be addressed:\n1. In Figure 4, can you show the V2 Exp. Var. for all the 103 neurons (similar as figure 5)? I’d be interested in the distribution of all the 103 neurons rather than medians or median of medians. \n2.  The description of Barlow twins objective function is confusing: according to the cited paper (Zbontar et al., 2021), b indexes batch sample (instead of batch), and the off-diagonal elements of the cross-correlation matrix “decorrelates the different vector components of the embedding... reduces the redundancy between output units” rather than “decorrelated features across the different images in a batch” or “decorrelating features across different images”", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that the paper is suitable for publication in TMLR. 
I propose a featured certification for the submission. The paper is very well-written with creative and solid technical content. It includes a conceptual advance for layer-wise self-supervised learning that I believe can be highly interesting for the TMLR audience interested in local learning with brain-inspired algorithms.  The proposed model improves out-of-domain generalization and model predictions are well-aligned with data from humans in image classification.
Decision: Accept (poster)