Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream
Authors: Abdulkadir Gokce
Martin Schrimpf
Venue: ICML 2025 spotlightposter (2025)
Keywords: scaling laws
neural alignment
behavioral alignment
computer vision
primate visual ventral stream
Primary area: applications->neuroscience_cognitive_science
TL;DR: We systematically explored scaling laws for primate vision models and discovered that neural alignment stops improving beyond a certain scale, even though behavior keeps aligning better.

Abstract: When trained on large-scale object classification datasets, certain artificial neural network models begin to approximate core object recognition behaviors and neural response patterns in the primate brain. While recent machine learning advances suggest that scaling compute, model size, and dataset size improves task performance, the impact of scaling on brain alignment remains unclear. In this study, we explore scaling laws for modeling the primate visual ventral stream by systematically evaluating over 600 models trained under controlled conditions on benchmarks spanning V1, V2, V4, IT and behavior. We find that while behavioral alignment continues to scale with larger models, neural alignment saturates. This observation remains true across model architectures and training datasets, even though models with stronger inductive biases and datasets with higher-quality images are more compute-efficient. Increased scaling is especially beneficial for higher-level visual areas, where small models trained on few samples exhibit only poor alignment. Our results suggest that while scaling current architectures and datasets might suffice for alignment with human core object recognition behavior, it will not yield improved models of the brain's visual ventral stream, highlighting the need for novel strategies in building brain models.

Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream
Abdulkadir Gokce Martin Schrimpf'
Abstract
Samples
Compur
Brait
When trained on large-scale object classification datasets, certain artificial neural network mod- els begin approximate core object recognition behaviors and neural response patterns in the pri- mate brain. While recent machine learning ad- vances suggest that scaling compute, model size. and dataset size improves task performance, the impact of scaling on brain alignment remains un- clear. In this study, we explore scaling laws for modeling the primate visual ventral stream by sys- tematically evaluating over 600 models trained under controlled conditions on benchmarks span- ning V1, V2, V4, IT and behavior. We find that while behavioral alignment continues scale with larger models, neural alignment saturates. This observation remains true across model archi- tectures and training datasets, even though mod- els with stronger inductive biases and datasets with higher-quality images are more compute- efficient. Increased scaling is especially bene- ficial for higher-level visual areas, where small models trained on few samples exhibit only poor alignment. Our results suggest that while scaling current architectures and datasets might suffice for alignment with human core object recognition behavior, it will not yield improved models of the brain's visual ventral stream, highlighting the need for novel strategies building brain models.
Pizskt
Thank
N Cs
0.6
Alignment Score (S)
Neural Alignment
0.4
0.48 0.55C 0.15
02
0.0
Behavioral Alignment 1.00
-02-
1.40C
0.08
10"
10"
16"
10a
Total Flops (C)
Figure 1: a) For given compute budget (C), we determine the scaling laws for maximal neural and behavioral align- ment to the primate visual ventral stream. b) We find consis- tent scaling laws for brain and behavioral alignment across over 600 models. While we predict models to approach perfect behavioral alignment at large scales, the effect of scaling on brain alignment is already saturating.
tools for simulating the response patterns of neurons within this stream (Yamins et al., 2014; Schrimpf et al., 2018). These advanced models offer precise computational ac- count of how neural mechanisms in the brain give rise visual perception.
The advent of neural networks has revolutionized our derstanding and modeling of complex neural processes. A particularly active area of study is the ventral visual stream in primates, key pathway in the brain responsible for processing visual information (Goodale & Milner, 1992: Grill-Spector et al., 2001; Malach et al., 2002; Kriegeskorte al., 2008). Neural networks, when trained on extensive datasets, have emerged the most accurate quantitative
Recent developments in machine learning have emphasized the significance of both the volume of training data and the complexity of model architectures (Kaplan et al., 2020; Hoffmann et al., 2022: Zhai et al., 2022; Bahri et al., 2022: Antonello al.. 2023; Muennighoff al., 2023: Agha- janyan al., 2023; Isik et al., 2024). These findings raise the question: Can build better models of the brain by scaling up model architectures and dataset sizes? Recent studies have found that in pre-trained models, the number of parameters and dataset samples respectively seem to im- prove predictions of fMRI and behavioral measurements (Antonello al., 2023; Muttenthaler al., 2023). With the
EPFL. Correspondence to: Abdulkadir Gokce <abdulka- dir.ookceccpfl.hh>
Proceedings of the 4dnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream
numerous differences between pre-trained models however, the relative contributions of model parameters and dataset size to brain and behavioral alignment are not clear.
1. Related Work
Primate Visual Ventral Stream. The ventral visual stream, a critical pathway in the primate brain, including humans, plays a key role in visual perception, extending from the occipital to the temporal lobes and serving the "what pathway" for object recognition and form represen- tation (Goodale & Milner, 1992; Grill-Spector et al., 2001; Malach et al., 2002; Kriegeskorte et al., 2008). Beginning in the primary visual cortex (V1). where basic visual infor- mation from retinal ganglion cells is processed, the ventral stream proceeds through areas such as V2, V3, V4, and the inferotemporal cortex (IT), each responsible for increasingly complex features of visual perception (Kandel et al., 2000). Despite decades of research and a wealth of brain data, the precise neural mechanisms underlying visual perception are not well understood.
Despite recent successes in using neural networks as models of the brain, a comprehensive understanding of how model scale-separately and jointly across parameters, dataset size, affects functional alignment with different cortical areas remains elusive. Previous studies have often relied on heterogeneously trained models using off- fff-the-shelff checkpoints (Conwell et al., 2024), or focused narrowly on specific brain areas (c.g., IT only (Linsley et al., 2023)), frequently using proxy quantities such as task performance. Our work addresses these limitations through systematic, from-scratch training protocol spanning over 600 models, enabling controlled comparisons and robust parametric es- timation of scaling laws for both behavioral and neural alignment across the entire ventral visual hierarchy. This approach offers a clearer disentanglement of the respective contributions of architecture, data, and optimization objec- tive to brain modeling.
and compute-alt -ffpptts
Modeling the Primate Visual Ventral Stream. Particular artificial neural networks (ANNs) are the most accurate models brain responses in the visual ventral stream and associated core object recognition behaviors (Schrimpf et al., 2018; 2020). Models optimized for ecologically viable tasks (Yamins & DiCarlo, 2016) in particular have demonstrated strong brain and behavioral alignment (Yamins et al., 2014;
In this paper, we examine how scaling- - of model parameters and training dataset size impacts the alignment of artificial neural networks with the primate ventral visual stream. We systematically train models from a variety of architectural families on image classification datasets which allows us to independently control and observe the effects of model complexity and data volume. To capture the observed trends, we introduce parametric power-law trends that describe the impact of scale on alignment with behavior and brain regions along the visual ventral stream. We summarize the contributions of this work as follows:
Khaligh-Razav Razavi & Kriegeskorte, 2014; Cadena et al., 2019; Schrimpf et al., 2018; Nayebi et al., 2018; Kietzmann et al., 2019; Rajalingham et al., 2018; Zhuang et al., 2021; Geiger et al., 2022) notably these models are trained purely on image classification datasets, without fitting to brain data.
Scaling Laws. Recent advancements in artificial intelli- gence are driven by scaling the model size and training data. Empirical evidence suggests a power-law relationship be- tween model performance and both model parameters and dataset size, indicating that continued scaling will further im- prove performance (Kaplan et al., 2020; Cherti et al., 2023; Zhai et al., 2022; Hoffmann et al., 2022; Dehghani et al., 2023; Henighan et al., 2020; Brown et al., 2020; Bahri et al., 2022; Hestness et al., 2017). The power-law exponents enable the optimal allocation of compute between model parameters and dataset samples, such that performance is maximized (Kaplan et al., 2020; Hoffmann et al., 2022).
While scale initially improves alignment, brain align- ment saturates. Behavioral alignment on the other hand continues to improve.
Increasing both parameter count and training dataset size improves alignment, with data providing more gains over model scaling.
Architectures with stronger inductive bias (c.g., convo- lutions and recurrence) and datasets with higher-quality images are more sample- and compute-efiie -efpptteeffiiintt......
While scaling laws for machine lcarning performance has been extensively studied, the scaling laws for brain align- ment remain unclear. Recent studies suggest an involvement of both model size and data volume in the functional align- ment with brain data (Azabou et al., 2023; Benchetrit et al., 2023; Caro et al., 2024; Antonello et al., 2023). Conversely, Muttenthaler et al. (2023) indicate that sample size is critical for behavioral alignment. We here unify these results, in the realm of the primate visual ventral stream, into quantita- tive scaling laws for how model and dataset sizes relate to alignment with the brain and behavior.
Fitting parametric power-law curves, we find that model alignment with higher-level brain regions and especially behavior benefits the most from scaling.
We publicly release our training code, evaluation pipeline, and over 600 checkpoints for models trained in a controlled manner to enable future research.
2

---

Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream
2. Methods
across 1,000 categories, has long been a benchmark in computer vision, designed challenge and evaluate auto- mated visual object recognition systems. On the other hand, EcoSet is more recent dataset, designed to provide an eco- logically valid representation of human-relevant objects. It contains over 1.5 million images spanning 565 basic-level categories, curated to better reflect the natural distribution of objects in the real world, aligning with human perceptual and cognitive experiences.
Neural & Behavioral Alignment. To evaluate the align- ment of our model with brain function, we utilize a range of benchmarks from Brain-Score (Schrimpf et al., 2018; 2020). These benchmarks assess model performance by comparing model activations or behavior with primate neural data using the same images. Specifically, the VI and V2 benchmarks compare model outputs to primate single-unit recordings from (Freeman et al., 2013), using 315 texture images and data from 102 V1 and 103 V2 neurons. For the V4 and IT benchmarks, 2,560 images are used match model activa- tions to primate Utah array recordings from (Majaj al., 2015), based data from 88 V4 and 168 IT electrodes. A linear regression is trained on 90% of the images to correlate model and neural data, with prediction accuracy for the re- maining 10% evaluated using Pearson correlation, repeated ten times for cross- validation. The behavioral benchmark assesses model predictions for 240 images against primate behavioral data from (Rajalingham et al., 2018) using lo- gistic classifier trained on 2,160 labeled images. Pearson correlation is used to measure the similarity in confusion patterns between model predictions and primate responses. All benchmark scores are normalized to their respective maximum possible values.
To create subsets of ImageNet and EcoSet, we sampled d E 1,3, 10, 30, 100, 300 images per category. For d e 1, 10, 100, we repeated the runs with three random seeds to ensure robustness. For ConvNeXtss (Liu et al., 2022b) and ViTs (Touvron et al., 2022), we used the training recipes developed by the original model authors. The remaining models were trained for 100 epochs using minibatch size of 512. We employed stochastic gradient descent (SGD) optimizer with a cosine decaying learning rate schedule, starting with a peak learning rate of 0.1 and incorporating a linear warm-up phase spanning five epochs. We main- tained the momentum at 0.9 and applied a weight decay of 10-4 Cross-entropy loss was used as the minimization objective. We utilized standard ImageNet data augmenta- tions, specifically random resized cropping and horizontal flipping. We define the model's 's alignment score S (and an inverse Misalignment Score I. 1 S)
1-S) as the average across the V1, V2, V4, IT, and behavioral benchmark scores. Layers are committed to brain regions based on models trained on full dataset, and applied to all variants trained with subsampled datasets. As we reused the same neural and behavioral data both to select the optimal model layer for readout and to assess the model's alignment, we validated the benchmark results on a private split of each dataset Brain-Score. We observed almost perfect correlation between the results on the private and public splits (Appendix C).
Scaling Power-Law Curves. Following previous work scaling laws (Zhai et al., 2022; Hoffmann et al., 2022; Besiroglu et al., 2024), we fit power law functions in the form
L= -X-a
(1)
on the data where L is the misalignment score, and X is an independent variable, such as the number of samples seen (D), number of parameters (N), and the total training floating point operations (FLOPs) (C). Coefficients E, A, and a are found by minimizing
Scaling Models and Data. We trained an array of stan- dard models from several architecture families. Specifi- cally, we used ResNet18, 8, 34,50, 101, 152 from (He al., 2016); EfficientNet-BO0 1, from (Tan & Le, 2019); Vision Transformer ViTT, S, B, L from (Dosovitskiy et al., 2021); ConwNeXIT, S, L.L from (Liu et al., 2022b); CORnet-S from (Kubilius et al., 2019); and AlexNet from (Krizhevsky et al., 2012). We also trained 33 modified versions of ResNet18: 22 models obtained by scaling the network width from 1/16 to times the original size, and 11 models de- rived by adjusting the depth. Similarly, we trained four additional ConvNeXt and ViT models by scaling the width of the ConvNeXt-T and VIT-S architectures.
min a.c.a
Σ
Huber, (LSE(a a log X,, e) - log L,)
HE [# Runs)
(2)
where F. = exp(e), A = exp(a) and LSE is the log- sum-exp operator. We solve Eq. using BFGS mini- mizer with = le 1--3. 3, and use a grid of initialiations as follows: e 0-5.. 1}. E 5 10.,,.... 25). a E {0, 0.5. 22.
To capture the slow initial increase in benchmark scores of modern architectures like ConvNeXt and ViT models in the low- regime, we introduce an additional parameter À to Eq. 2. This parameter allows the fitted curve to saturate at lower scales, better reflecting the observed performance of
For our experiments, we selected two image classification datasets: ImageNet (Deng et al., 2009) and EcoSet (Mehrer et al., 2021). ImageNet, with millions of labeled images
3

---

Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream
these models under limited data conditions:
where we fit and via linear regression of C' and N ND in log-log scale. Then, the updated equations governing the optimal allocation becomes
L == + A (x 10A)
(3)
We minimize the modified equation as before, using À E O, 0.5, 1.0, 1.5, 2.0. To fit the curve described by Eq. 3, we utilize all data points from the ConvNeXt and ViT models. For fitting the remaining curves, select ConvNeXt and ViT runs that were trained on datasets with either 300 sam- ples per class or the full dataset. This approach ensures that the fitted curves accurately represent the scaling behavior of these architectures across different data regimes.
(C) = G(C/m) D
(9)
where a', b', and G are calculated as before.
To evaluate the uncertainty of our model fits, we performed bootstrapping with 1,000 resamples. We compute 95% con- fidence intervals for each point along the fitted curves based on the variability observed across the bootstrapped esti- mates.
Furthermore, we would like to describe the misalignment (T.) as function of both the model and data size (N, D) and predict optimal allocations N and D by solving
Finally, to avoid large constants during curve fitting, rescale the variables C, N,, and D by setting C' = C/10 N = N/10", and D = D/10*.
(N", D*) arg min L(N, D), FLDPP((D)- C (4)
NDD
3. Results
In that regard, following (Hoffmann et al., 2022; Besiroglu et al., 2024) we fit parametric function of the form
3.1. Scaling drives behavioral alignment, but saturates for neural alignment
A Na
B
L(N, D) E
(5)
D
Our experiments show clear and consistent improvement in behavioral alignment as both model size and training dataset size increase. Fig 1.b illustrates this trend across different architectures and scaling axes. The curve S = - 1.40- converges to perfect alignment score of in the limit C.
where the loss (L) is a function of parameter count (N) and number of samples seen (D). In Eq. 5, the first term repre- sents the loss in an ideal data generation scenario (entropy), the second and the third terms reflect the under-performant of a model due to limitations in parameter and data size (Hoffmann et al., 2022; Muennighoff et al., 2023). Fol- lowing the example of Hoffmann et al. (2022), we learn variables { E, A, a, B, }} that characterizes misalign- ment by solving
In contrast to behavioral alignment, neural alignment with specific brain regions demonstrated saturation as training compute scaled up in size. The curve represented by the formula S = 0.48 - 0.55 represents a saturation at 0.48. The diminishing returns in neural alignment imply that merely scaling up models and data is insufficient to achieve better alignment with higher-level neural representations.
arg min e, a. a. b, 8 iEl [Ruus))
Hubera (log La-
(6)
LSB(a a - log N., b - 3 log Di, e)
3.2. Architectural Inductive Bias Influences Alignment and Scaling Dynamics
with 0 = 10-3 and E = exp(e), A = exp(a) B = exp(b). Initialiations of b and 33 follow and a. respectively.
Experimental results indicate that modern architectures, such as ConvNeXt and Vision Transformers (ViTs), exhibit poorer neural alignment compared to models like ResNets and EfficientNets in low data regime. ResNets and Efficient- Nets, which have stronger inductive biases due to their fully convolutional structures, demonstrate high neural alignment even at initialization. In Fig. 2, alignment score of ResNets and EfficientNets increase steadily with additional compute in the form of training samples, however ConvNeXt and ViT requires more compute in order to start rising.
Both Kaplan et al. (2020); Hoffmann et al. (2022) assume that compute follows the relationship C(N, D) 6ND to predict the optimal allocation of compute (C) to N and D using set of equations with the learned variables mentioned above:
(C) = G(C//)), D²(C)= = (C/6)
where a' = 3 8 = a G =
aA
+ 3r
a a+
3B
This difference in initial aligmment also affects how the scal- ing laws evolve for each architecture. Models with weaker inductive biases require more extensive in terms of training data- to achieve levels of neural align- ment comparable to those with stronger inductive biases. Consequently, the scaling curves for ConyNeXt and ViT
However, we observe that C(N, D) ~ 6N D does not hold with different architectures, and various CNN families have a slightly different relationship of C, N, and D. As such, we assume a pawer-law relationship of the form
C(N, D) = DyN
4

---

Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream
a)
ResNet
EfficientNet
D)
Average Alignment
Alignment Score (5)
Betarioral 1.0000
1.00C-a15
Alignment Score (9)
Detarioral 5 1.00 1.30C-209
Reshit C.S Dec
ESiieehee w 3-252-0 750-
0.99
0.90
VIT So 0.80 - 1p 1psta-iine
Algement Score
0.90
0.00
0:30
0:0
0.10
4.1
10
10"
10
10
10"
Total Flops (C)
Total Flops (C)
Total
0.20
ConvNeXt
6.70
VIT
C)
Moddy
Alignment Score (5)
0.90
Neurel - 10 111
Behavionl 10110
Alignment Score
0/00
Behavionl 100- 10 1N
- 0.41 10400 1othtaaaa
Afgirmsnt Score (91
0.30
8:30
0.20
620
10
10"
Total Flops
Flops (C)
Modal Parsmettrrs (5)
Figure 2: Scaling Model Size. a) Neural and behavioral alignments of four architecture families. Models with inductive biases (ResNet, EfficientNet) are more compute-efficient than less constrained models (ConvNeXt, ViT). b) Average alignment per model architecture. All models converge to similar aligmments. c) Increasing parameters improves alignment (models trained on full datasets), but the effects saturate.
models develop differently, highlighting that architectural choices not only impact baseline alignment but also influ- ence the efficiency of scaling strategies.
law relationship.
In contrast, expanding the model size without proportionally increasing the training data results in steeper diminishing returns in alignment performance. Larger models rapidly reach point where additional parameters do not translate into meaningful improvements. Fig. 2c estimates satura- tion level of 0.44 by scaling model sizes with all samples of training data whereas Fig. 3a predicts maximum align- ment of 0.46 and 0.48 for ImageNet and Ecoset respectively. This indicates that scaling training datasets overall improves brain alignment better than models scaling. Furthermore, Fig. 4b demonstrates that larger models of the same archi- tecture family require much more samples to achieve the same level of alignment.
Fig. 3b highlights that architectural priors critically shape alignment dynamics, particularly in low-data settings. COR- net models, which incorporate recurrence, achieve relatively high alignment early in training- outperforming both con- volutional and transformer- based models under limited su- pervision. Yet as training data increases, this initial ad- vantage wanes, and alignment scores across architectures begin to converge. This suggests that while certain induc- tive biases offer sample efficiency, their long-term bene- fits may be outpaced by deeper or more flexible architec- tures given sufficient data. Overall, these findings empha- size that strong inductive biases-suche as convolution and recurrence- facilitate better alignment when data is lim- ited, whereas extensive task-driven optimization on larger datasets eventually mitigates differences across architec- tures.
To quantitatively capture the joint interaction between data and model scaling, we fitted a parametric curve based on 5q.5, as shown in Fig.4a. This curve effectively models how compute (C), dataset size (D), and model size (N) collectively influence brain alignment. Utilizing the para- metric relationships described in Eq. 9, we estimate that ad- ditional compute should be allocated following the scaling laws D s C..7 and N s3 C These exponents indicate that, for optimal brain alignment, computational resources should be predominantly invested in increasing the dataset size rather than the model size.
3.3. More Data Is Better Than More Parameters
Our analysis reveals that increasing the size of the train- ing dataset has more significant impact improving brain alignment than simply enlarging the number of model parameters. While both strategies lead to performance en- hancements, the benefits from data scaling exhibit less se- vere diminishing returns compared to model scaling, Specif- ically, models trained on larger datasets consistently demon- strate superior neural and behavioral alignment with the primate ventral visual stream, following predictable power-
3.4. Ordered efffect of scale on alignment
Our study reveals graded effect of scaling on alignment across the cortical hierarchy of the primate visual system.
5

---

Scaling Laws for Task- Bask-Optimized Models of the Primate Visual Ventral Stream
a)
a)
Imageliet 5 0.400 11000.000 003905m00
Imagetee2
S 0.48 0.470-121
0.52
10
Allgnment Score (5)
0.43
Niturolal
Alighment Score
100g Enduugg 10 A02UFT
Places365
16
2:33
0.30
Wechet Conmeac
CIFAR1O
0.22
CoReet &
16
2.20
IMNNST
0.00
12
10
10"
10"
Total Flops (C)
10
10'
10'
16"
16
b)
Number of Samples Seen (D)
b)
Alignment Score (5)
0:40
Alignment Score (8)
0:30
0.50
Archilecuure
0:25
Audee
Commant
0.00
Gondhett
0.10-
Weofat
5:00
0.00
Flops (C)
10
10'
Number of Samples Seen (D)
Figure 4: Optimal Compute Allocation. a) Alignment as function of both model and training dataset sizes. Marker size is log- proportional to model size. Compute should be spent 0.3/0.7 on model/dataset size respectively. b) Models start out at different alignments but converge to the same saturating point.
Figure 3: Scaling Dataset Size. a) Training on larger datasets enhances brain alignment. The aligmment scaling curves derived from ImageNet and EcoSet closely estimate the alignment achieved when using ImageNet2 lk. In con- trast, datasets with specialized image distributions such as Places365-fall below the alignment scaling laws estab- lished by these generalist datasets. In the extreme case of handwritten digits (infiMNIST), the impact of training is minuscule on the alignment. b) Model families with weaker inductive bias start at a lower alignment and require more data to improve.
per region, highlighting how higher cortical areas benefit more from scaling efforts. This ordered effect suggests that regions higher up in the visual hierarchy, such as the Inferior Temporal (IT) cortex and behavioral outputs, gain more substantially from additional data and increased model complexity. In contrast, early visual areas like VI and V2 exhibit smaller alignment gains with increased compute, indicating potential saturation effect.
Specifically, we observe that the benefits of increased train- ing compute- compute-achieved achieved through larger datasets and more complex models-var vary systematically among different brain regions, reflecting their position in the visual processing pathway. Fig. 5.a illustrates the alignment as a function of training compute across various brain regions. We catego- rized the models into two groups based on their architec- tural inductive biases. Group includes most models with strong inductive biases, such ResNets and EfficientNets. These models start with higher neural alignment scores even at initialization due to their fully convolutional archi- tectures. Group 2 consists of models with weaker induc- tive biases, specifically ConvNeXt and Vision Transformers (ViT'). These models exhibit lower neural alignment in the low-data regime and require more compute to achieve similar alignment levels.
4. Discussion
We establish scaling laws governing the effect of model and dataset scale on behavioral and brain alignment with the primate visual ventral stream. While scale is necessary component for all brain-like models, model architectures with priors such as convolutions, and datasets with high- quality images are more sample efficient, leading to align- ment with smaller compute requirements. Scale especially improves alignment with higher-level visual regions, but brain alignment saturates across all conditions tested here whereas behavioral alignment continuously improves with increased scale.
To quantify the impact of scaling on each brain region, we define the alignment gain per region as A10 where A and a are parameters of Eq. 2. Our findings indicate that higher regions in the cortical hierarchy show greater benefits from increased compute. Fig. 5b illustrates the alignment gain
We find a saturation of neural alignment under current mod- cling approaches, consistent with trends reported in prior work (Linsley et al., 2023; Conwell et al., 2024; Mutten- thaler et al., 2023). Critically, our results reveal a disconnect between neural and behavioral alignment: while behavioral
6

---

Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream
a)
V2
V4
Prie 6.66 6.6-0.12
Stringg Pror 0.35 -.6--
Aligrment Store
Altgrament Spore
Alignmant Sccrr
4.4
0.40
0.40
String 0.500---7
Wose Prir 5 1011110-100
0.00
0.20
10
10"
10
Total Flops (C)
Total Flops (C)
Total Flops (C)
Behavioral
b)
12.
Storg 1114--135
String
Aligmment Spore (8)
0.3-
Week Piar 6.53 10-5700
Alignment Spore (5)
Weak Prir 0..5 10 10
Cooffciient
1.--
68-
642
040
0.-
6:1-
0.18
10
10"
')"
Renoded
Total Flops (C)
Total Flops (C)
Regions
Figure 5: Graded Effect of Scale across Cortical Hierarchy. a) Alignment function of training compute across different brain regions. Group 1 contains most models except those with low inductive bias (Group 2; ConwNeXt, ViT). b) Alignment gain per region, defined as A10". Regions higher in the cortical hierarchy show greater benefits from increased compute (Behavior > IT > V4 > V2 > VI).
alignment continues to improve with increased scale, neu- ral alignment plateaus. By quantifying scaling laws across model families and data regimes. we show that improve- ments in brain alignment are more efficiently achieved by increasing dataset size rather than model parameters. These findings offer concrete guidance for developing brain-like models more effectively, emphasizing the importance of dataset diversity and biologically inspired architectural pri- ors over brute-force model scaling.
et al., 2023), neural alignment eventually saturates, indicat- ing that factors other than task performance influence neural alignment.
Generalization Beyond Supervised Training. We as- sessed whether alternative training paradigms can overcome the limitations observed in neural alignment under super- vised learning. Figure 7a illustrates the scaling of alignment as a function of compute spent during self-supervised train- ing of ResNet models using SimCLR (Chen et al., 2020) on ImageNet. The results confirm the trends observed in supervised training: behavioral alignment continues to im- prove with increased compute, following a strong power-law relationship, while neural alignment approaches a saturation point. This consistency suggests that the saturation in neural alignment is not exclusive supervised learning but may be inherent to the models or datasets employed.
Dissociation of behavioral and neural alignment. Our findings reveal dissociation between behavioral and neural alignment as models are scaled with more parameters and larger datasets. While behavioral aligument continues to improve consistently with increased model parameters and training data exhibiting strong power-law relationship- neural alignment reaches a saturation point beyond which additional scaling yields minimal gains. This divergence suggests that behavioral alignment benefits more substan- tially from scaling efforts, whereas neural alignment may require alternative approaches beyond merely increasing model size and data volume to achieve further improve- ments.
The region-specific breakdown (as illustrated in Supp. Fig. S6) further reinforces this observation. Even in a self- supervised learning context, higher-level visual areas like and behavioral outputs demonstrate more pronounced improvements with increased compute, while early visual ar- eas like V1 and V2 show minimal gains. This suggests that the hierarchical nature of neural alignment is a fundamental characteristic that transcends specific training methods.
This disparity is further highlighted by the correlation be- tween task performance and alignment depicted in Figure 6. Behavioral alignment closely tracks validation accuracy, improving hand-in- hand-in-hand as models become more accurate. Consistent with prior (Sck(Snh et al., 2018; Linsley
Additionally, we explored the impact of adversarial fine- tuning on alignment performance. In Figure 7b, ResNet models trained on subsets of ImageNet were fine-tuned
7

---

Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream
8)
b)
c)
Arerge
Behavtoral Allgnmant
Neural Alignment
Nurder Camp Seen
Correlation
02-
64-
Amdle: Colnee
211
62-
Sastiet
62
Valldation Accuracy
Validation Accuracy
Regions
Figure 6: Correlation between Task Performance and Alignment. a,b) Correlation between validation accuracy (ImageNet & EcoSet) and brain (a) and behavioral (b) alignment. Behavioral alignment strongly correlates with task performance, whereas neural alignment shows a non-linear trend, reaching saturation. e) Pearson's correlation coefficient per region, with all p-values less than 10-40.
adversarially for 10 epochs using the Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2015; Wong et al., 2020). Importantly, the scaling curves were estimated solely from the non-adversarial runs, yet the adversarially fine- tuned models exhibited improvements along these existing scaling curves. This indicates that adversarial training can enhance alignment without deviating from the established scaling behavior.
under supervised learning compared to when trained with self- sele-suuer objectives like DINO and SimCLR. In con- trast, the ResNet50 model, which possesses strong induc- tive biases due to its convolutional architecture, exhibits relatively consistent aligmment dynamics across different learning signals. This robustness implies that models with strong inductive biases less affected by the choice of training objective, whereas architectures like ViT-S benefit more substantially from rich, self-supervised feedback to achieve optimal alignment.
Impact of Architectural Inductive Biases on Alignment Dynamies. Our evaluation of alignment during training re- veals that the alignment behavior varies significantly across different model architectures. Figure 7.c shows that while various models eventually converge to similar alignment levels with sufficient training, fully convolutional architec- tures-such ResNets and EfficientNets exbbibt substan- tially higher alignment scores at the very beginning of train- ing. This early advantage suggests that these architectures possess inherent features that align closely with neural data from the primate ventral visual stream even before learning from data occurs.
Limitations and Future Directions. Our study has sev- eral limitations. First, the extrapolation of our scaling func- tions is constrained by the specific range of model sizes and dataset volumes we examined. While we observed power-law relationships between scaling factors and brain alignment, these functions may not generalize beyond the scales tested.
Second, evaluated subset of models focusing primarily on standard and modern convolutional neural networks (e.g.. ResNets and ConyNeXts). mansformer-based architectures (e.g. ViTs) and recurrent networks (CORnet-S). While these architectures cover a range of inductive biases and complex- ities, they do not encompass the full spectrum of possible neural network designs, such more biologically plausible models. We see scaling laws opportunity to extrapo- late the alignment of models at scale, even if their current training is canpute-constrained.
Further analysis in Figure S3 confirms that this initial high alignment is due the strong inductive biases present in fully convolutional networks. These biases enable the mod- els to start with representations already well-suited for neu- ral alignment. Figure S4 reinforces this finding by demon- strating that models with strong inductive biases achieve higher initial alignment compared to architectures like Con- vNeXt and VIT. which have weaker inductive biases.
Third, our experiments utilized a subset of training datasets primarily from ImageNet and EcoSet. Although these datasets are extensive and widely used, they may not cap- ture all the nuances of visual stimuli relevant to the primate ventral visual stream. Therefore, models trained on other datasets might exhibit improved scaling properties.
Influence of Learning Signals on Alignment Dynamics. Our investigation reveals that the type of learning signal plays crucial role in the dynamics of alignment during training. Figure 7d illustrates the alignment trajectories of ResNet50 and ViT-S models trained on ImageNet using su- pervised learning, SimCLR, and DINO (Caron et al., 2021) methods. Notably, the VIT-S model requires significantly more training steps achieve the same level of alignment
While transformer based architectures like ViTs dominate in many ML tasks due to their flexibility and strong task perfor- mance, our findings highlight a clear dissociation between
8

---

Scaling Laws for Task- SiskkOOtimmzed Models of the Primate Visual Ventral Stream
a)
SimCLR Training
b)
Adversarial Finetuning
Alignment During Supervised Training d)
Alignment During Training SSL
Nowe
Algoment Score 15
Algroment Score (80
Mooi
110
Total Plopa (5)
Total Plops (C)
Training Epoch
Training Epoch
Figure 7: Alignment Scaling of Alternative Training Strategies. a) SimCLR Training: ResNet models trained with SimCLR improve behavioral alignment with compute, while neural alignment saturates. b) Adversarial Robustness: Fine- tuning ResNets with adversarial training (FGSM) enhances alignment along the scaling curve. c) Supervised Training Dynamics: ResNets and EfficientNets exhibit higher initial alignment in early phases of training, due to strong inductive dynamics vary with training objectives,
biases, unlike ConvNeXt and ViT. d) Self-supervieed Training but converge to the same alignment saturation.
behavioral and neural alignment. Convolutional models exhibit significantly better neural alignment, especially in early training and low-data
training with brain data integraing neural recordings di- rectly into the training process- could enhance both neural and behavioral alignment, paving the way for more accurate and efficient brain-like models.
that bi- ologically inspired inductive biases play unique role in approximating cortical representations. Interestingly, we also show that recurrence (c.g., CORnet) offers benefits in sample efficiency, but its advantage diminishes with more data. These observations motivate hybrid architectures that blend convolutional, recurrent, and transformer components.
Acknowledgements
This work was partly supported by the Swiss National Sci- ence Foundation (SNSF) through the Spark project CRSK- 2282279 We thank the members of the EPFL NeuroAI Lab for their valuable discussions and feedback throughout the project.
Taken together, our results demonstrate that while scaling both model parameters and training data size enhances be- havioral alignment with human visual perception, it leads to saturation in neural alignment with the primate ventral vi- sual stream. Data scaling proves more effective than model scaling in improving alignment, emphasizing the critical role of extensive and diverse training datasets. We also find that architectural choices significantly influence align- ment efficiency, with models possessing strong inductive biases-such as fully convolutional networks-achieving higher neural alignment even at initialization. Additionally, the impact of scaling varies across different brain regions, benefiting higher cortical areas more than early visual ar- eas. These findings suggest that merely increasing scale is insufficient for modeling the intricate neural representations of the brain's visual system. Future work should investi- gate new approaches, including alternative architectures and training strategies, to develop models that more accurately reflect the complexities of neural processing in the primate visual cortex.
Software and Data
We open-source our training and analysis code, along with benchmark results and model checkpoints from our model zoo. All resources are available at:
Impact Statement
This work seeks to advance our understanding of how seal- ing model size and dataset size influences alignment with primate visual processing. While the primary goal is sci- entific-improving models of the visual cortex-it also in- tersects with broader concerns about resources and impact. Training large-scale models has nontrivial computational costs and environmental implications. At the same time, more accurate brain-like models could yield downstream benefits for neuroscience and machine learning applications (for instance, improved medical imaging or more robust computer vision systems). We do not foresee direct ethical risks specific to this research beyond the usual resource con- siderations in large-scale model training, and we have used existing public neurophysiological and behavioral datasets.
To push neural alignment beyond current saturation levels, future research should explore adversarial training methods that encourage models to learn more robust, brain-like rep- resentations. Leveraging biologically inspired architectures such as VOneNets (Dapello et al., 2020) may lead to more compute-efficient models achieving higher neural alignment without extensive scaling. Additionally, investigating co-
9

---

Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream
References
Buslaev, A., Iglovikov, V. I., Khvedchenya, E., Parinov,
A., Druzhinin, M., and Kalinin, A. A. Albumentations: Fast and flexible image augmentations. Information, 11 (2), 2020. ISSN 2078-2489 doi: 10.3390/imfo1 1020125, URL https: /tt:://wwwww www. www.wwdwp...... com com 2000- 2tt.c///m0022229///0 2/125.
Aghajanyan, A., Yu, L., Conneau, A., Hsu, W.-N., Ham- bardzumyan, K. Zhang, S., Roller, S., Goyal, N., Levy, O., and Zettlemoyer, L. Scaling laws for generative mixed-modal language models. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Ma- chine Learning Research, pp. 265-279. PMLR, 23-29 Jul 2023. https httpcttps://rroeeeee http:///ooccccccoccoo .a..nngsmmm.... m....pres// v202 aatp janyanaaaaaaan2na 3a html.
Cadena, S. A., Denfield, G. H., Walker, E. Y., Gatys, L. A.. Tolias, A. S., Bethge, M., and Ecker, A. S. Deep con- volutional models improve predictions of macaque vl responses to natural images. PLOS Computational Biol- ogy, (544)e100689)) April 2019. ISSN 1553-7358, doi: 10. htst://jouaal..bi.00000000. o org /10 11111//11////orraaae// pcbi 1006897.
URL http: : / dx .oii
Antonello, R. J., Vaidya, A. R., and Huth, A. G. Scaling laws for language encoding models in fmri. Advances in Neural Information Processing Systems, 36, 2023.
Caro, O., de Oliveira Fonseca, A. H., Rizvi, S. A., Rosati, M., Averill, C., Cross, J. L., Mittal, P., Zappala, E. Dho- dapkar, R. M., Abdallah, C., and van Dijk, D. BrainLM: A foundation model for brain activity recordings. The Twelfth International Conference on Learning Represen- tations, 2024. URL https: opwnreviewweeeeeeoooop net/ forum? ieeeeidddwwa R=-IIII2EE...
Azabou, M. Arora, V., Ganesh, V., Mao, X., Nachimuthu, S. B., Mendelson, M. J., Richards, B. A., Perich, M. G., Lajoie, G., and Dyer, E. L. A unified, scalable frame- work for neural population decoding. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: https://ppppeevvveeeeee net forum? (d=sw2Y0sirtM.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Bahri, Y., Dyer, E., Kaplan, J., Lee, J., and Sharma, U. Explaining scaling laws of neural network generalization, 2022. URL https: / /opereevieww wttpeviwwwweeeeeeeeeeeeeeeeeeeee net forum? id-FvfV64rovnk.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In III, H. D. and Singh. A. (eds.), Pro- ceedings of the 37th International Conference on Ma- chine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 13-18 Jul 2020. httpt htt.ttppss/pppoceeeeeeeeeeo http://ppoceeen.ceeeee/e/ee .araggsmmmllr. mir.ppess. v119 chen20j html.
Benchetrit, Y., Banville, H., and King, J.-R. Brain decod- ing: toward real-time reconstruction of visual percep- tion. In Submitted to The Twelfth International Confer- ence on Learning Representations, 2023. URL https: //openreview net for://orummmm -ieimmiddddllll 3y
Besiroglu, T., Erdil, E., Barnett, M., and You, J. Chinchilla scaling: A replication attempt, 2024. URL https:// arxiv. iisivv.oo org/ abs/abbs/2444/ 22444.1044..122
Cherti, M., Beaumont, R., Wightman, R., Worts- man, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., and Jitsev, Reproducible scaling laws for contrastive language-image learning. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2818-2829, Los Alamitos, CA, USA. jun 2023. IEEE Computer Society. doi: 10.1109/CVRR5272222223000 https: //t.:///oii//////o www.eccmmpterrecccci...........e wtt.:eeeerrrreee.r.oooo.... .org/ N0.1109/CVPR5270 2023. 00276.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, D.,
Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Herbert-Voss. A., Krueger, G.,
Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https:: http::/cceediiig....ri.iieee neurips. ht /c//pppe__flleesspprrr////////// 4457Cc6eeea390111115000000222222 pdf.
URL
Conwell, C., Prince, J. S., Kay, K. N., Alvarez, G. A.. and Konkle, T. A large-scale examination of induc- tive biases shaping high-level visual representation in brains and machines. Nature Communications. 15 (1), October 2024. ISSN 2041-1723. doi: 10.1038/ -41467-02--534774 URL http: / dx.ddi..ooo. 10.1038/s41467-D22
10

---

Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream
Dapello, J., Marques, T., Schrimpf, M., Geiger, F., Cox, D., and DiCarlo, J. J. Simulating a primary visual cortex at the front of cnns improves robustness to image perturbations. In Larochelle, H., Ranzato, M., Hadsell, R., Balean, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 13073-13087. Curran Associates, Inc., 2020. URL cc ca/paper_fileeeee [igaperr_fileespppprr/222////// 98b1 3B817f0066569977661111bbaae pdf.
dates needed to produce primate ventral stream. In International Conference on Learning Representations (ICLR) Spotlight, 2022. https:: httpp::///pppneeeeeeeeeee /oppneevin.. net form/form?? 11teiid11111222111
Goodale, M. A. and Milner, A. D. Separate visual pathways for perception and action. Trends in Neurosciences, 1992.
neurips.
Goodfellow, I., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015. URL http: www.iiii.grg.ooooo abss////////// / 1412. 6572.
Paper.
Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J.. Gilmer, J.. Steiner, A. P., Caron, M., Geirhos, R., Alabdulmohsin, I., Jenatton, R., Beyer, L.., Tschan- nen, M., Amab, A., Wang. X., Riquelme Ruiz, C., Min- derer, M., Puigcerver. J., Evci, U., Kumar, M., Steenkiste, S. V., Elsayed, G. F., Mahendran, A., Yu, F., Oliver, A., Huot, F., Bastings, J., Collier, M., Gritsenko, A. A., Birodkar, V., Vasconcelos, C. N., Tay, Y., Mensink, T., Kolesmikov, A., Pavetic, F. Tran, D., Kipf, T., Lu- cic, M., Zhai, X., Keysers, D., Harmsen, J. J., and Houlsby, N. Scaling vision transformers to 22 billion parameters. In Krause, A., Brunskill, E., Cho, K., En- gelhardt, B., Sabato, S., and Scarlett, J. (eds.), Pro- ceedings of the 40th International Conference chine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 7480-7512. PMLR, 23-29 Jul 2023. URL www.tttp:///prooe http://pooceedeeeceeoe mmr. m.l..press/ v202 htt///dhhhaann22a.a.aa2 htmm.
Graham, B., El-Nouby, A., Touvron, H., Stock, P., Joulin, A., Jegou, H., and Douze, M. Levit: A vision transformer in convnet's clothing for faster inference. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 12259-12269 October 2021.
Grill-Spectori K., Kourtzi, Z., and Kanwisher, N. The lateral occipital complex and its role in object recognition. Vision Research, 41(10- 11): 11111:11440 10::114994449 1422, 2001.
He, K., Zhang, X., Ren, S., and Sun, J. Deep Residual Learning for Image Recognition. In Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR '16 pp. 770-778. IEEE, June 2016. doi: 10. 10.1109/CVVR.22116.00 URL http: //teeexpeeeee.eeee ieee owt. wtwooocuuettt7700000
Ma-
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., Hallacy, C., Mann, B., Radford, A., Ramesh, A., Ryder, N., Ziegler, D. M., Schulman, J., Amodei, D., and Mc- Candlish, S. Scaling laws for autoregressive generative modeling, 2020.
Deng, J., Dong, W., Socher, R., Li, L.- Li, K., and Fei-Fei, L. Imagenet: A scaee hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. leee, 2009.
Ding, M., Xiao, B., Codella, N., Luo, P., Wang, J., and Yuan, L. Davit: Dual attention vision transformers. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV, pp. 74-92. Springer, 2022.
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically, 2017.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de las Casas, D., Hendricks, L. A., Welbi, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J. W., and Sifre, L. An empirical analysis of compute-optimet optimal large language model training. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https: / //openreview, net Forum?s (omumiddii
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https:// openreview wwwpareviewnnetteeeee ? d= cbFaNTTy.
Freeman, J., Ziemba, C. M., Heeger, D. J., Simoncelli, E. P., and Movshon, J. A. A functional and perceptual signa- ture of the second visual area in primates. Nature Newro- science, 1677)9944-91, 2013. doi: 10. 10.1038/mn.3400
Isik, B., Ponomareva, N., Hazimeh, H., Paparas, D., Vassil- vitskii, S., and Koyejo, S. Scaling laws for downstream task performance of large language models, 2024. URL https: haapss///rxivv http::///arxvooorr/bbb22222222 wwt.v//rr//bb/.......oo 04177.
Geiger, F., Schrimpf, M., Marques, T., and DiCarlo, J. Wiring up vision: Minimizing supervised synaptic up-

---

Scaling Laws for Task-Optimizer Models of the Primate Visual Ventral Stream
Kandel, E. R., Schwartz, H., and Jessell, T. M. Principles of Neural Science. McGraw-Hill, 2000.
Linsley, D., Rodriguez Rodriguez, F., FEL, T., Arcaro, M.,
Sharma, S., Livingstone, M., and Serre, T. Performance- optimized deep neural networks are evolving into worse models of inferotemporal visual cortex. In Oh, A., Nau- mann, 'T,, Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Pro- cessing Systems, volume 36, pp. 28873-28891. Curran Associates, Inc., 2023.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models, 2020.
Khaligh-Raune Razavi,, S.-M. and Kriegeskorte, N. Deep super- vised, but unsupervised, models may explain it cor- tical representation. PLoS Computational Biology, 10 (11)=e1003915, November 2014, ISSN 1553-7358. doi: 10. htl. org/10. 11.100...1.1111 /ournal pcbi. 003915.
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. A convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022a.
URL http: //xx /oi.
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. A convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022b.
Kietzmann, T. C., Spoerer, C. Sörensen, I.. K. A., Cichy, R. M., Hauk, O., and Kriegeskorte, N. Recurrence is re- quired to capture the representational dynamics of the hu- man visual system. Proceedings of the National Academy of Sciences, 116(43):21854 21863, October 2019. ISSN 1091-6490. doi: 10. 10.1073/pnaa.19955540 1605444116. URL http: //dx fidxxdooi.oo wtg. 1073 /pnas.ppp/a..//.////////// 190554411e 6.
Loosli, G., Bottou, L., and Simard, P. projects:infimnist [leon.bottou.org Ceeonbotttouoog. https: //lonn bottou org/ wwt.oo/.rrggrroooo//// cessed 16-11-2024).
2007. [Ac-
Kim, H. Torchattacks: A pytorch repository for adversarial attacks, 2020.
maintainers, and contributors. Torchvision: Pytorch's computer vision library. https://githin con/ gytorch/visionn 2016.
Kriegeskorte, N. et al. Matching categorical object repre- sentations in inferior temporal cortex man and monkey. Neuron, 60(6): 126-1141, 1141, 2008
Majaj. N. J., Hong, H., Solomon, E. A., and DiCarlo, J. J. Simple learned weighted sums inferior tempo- ral neuronal firing rates accurately predict human core object recognition performance. The Journal of Neu- roscience, 14022-13143 2015. jneurosci.51811 111-511.1111..115 14.2015.
Krizhevsky, A., Nair, V., and Hinton, G. Cifar- 10 (canadian institute for advanced research). URL http:/ / /ww. cs.toronto. www.nntt..duueeeeem wew/eduuukeeee -atitt//kiiiiiiiiee
html.
doi: 10.1523/
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Percira, F., Burges, C., Bottou, L., and Weinberger, K. (eds.), Advances Neural Information Processing Systems, volume 25, Curran Associates, Inc., 2012.
Malach, R., Levy, I., and Hasson, U. The topography of high-order human object areas. Trends in Cognitive Sci- ences, 6(4):176-184. 2002.
Kubilius, J. Schrimpf, M., Hong, H., Majaj, N. J., Rajaling- ham, R., Issa, B., Kar, K., Bashivan, P., Prescott-Rov Roy, J. Schmidt, K., Nayebi, A., Bear, D., Yamins, D. L. K., and DiCarlo, J. J. Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs. In Wallach, H., Larochelle, H.. Beygelzimer, A., A'Alcéé-Bar Buc, F., Fox, E., and Garnett, R. (eds.), Neural Information Pro- cessing Systems (NeurIPS), pp. 12785 17785-117766. Curran Associates, Inc., 2019.
Mehrer, J., Spoerer, C. J., Jones, E. C., Kriegeskorte, N., and Kietzmann, T. C. An ecologically motivated image dataset for deep learning yields better models human vision. Proceedings of the National Academy of Sciences, 118(8), 2021. doi:
Mehta, S. and Rastegari, M. Mobilevit: Light- general-purpose and mobile mobile-friendly vision transformer. In International Conference on Learning Representations, 2022. URL tatttps://ppeeeeee /pwpeeeeeeeeeeeee////pp/p. wtt.riiwwweeeeeeeeeeeeeeeeeeeeee forum? id=vh- OsUt
LeCun, Y., Cortes, C., and Burges, C. The mnist database. https: //yssn//yan.... lecun. com com/exdb/ exdio/ mnist/, 1998. [Accessed 16-11-2024].
Muennighoff, N., Rush, A. M.. Barak, B., Scao, T. L., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and Raffel, C. Scal- ing data-constrained language models. In Teirth-sevnnth Conference on Neural Information Processing Systems, 2023. URL https: aattps://openrevieee /pppnreveeeeeeeeeeeeeeeeeeeeeeee net/ forum? id=j5BuTrRj35.
Li, W., Wang, L., Li, W., Agustsson, E., and Gool, L. V. Webvision database: Visual learning and understanding from web data. CoRR, 000.11080008622. 2017. URL http::/arxiv
http://aaaxvv.oogaa////e/e/ wtt.v/.rrrbbb//////./ 028662.
12

---

Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream
Muttenthaler, L., Dippel, J., Linhardt, L., Vandermeulen, R. A., and Komblith, S. Human alignment of neu- ral network representations.
L. K., and DiCarlo, J. J. Brain-score: Which artifi- cial neural network for object recognition is most brain- like? bioRxiv preprint, 2018. URL ht /www. Fiorxiv.orge
In The Eleventh In- ternational Conference on Learning Representations, 2023. URL htttps::/ppenrevieeee nete forum? id-ReDQ1OUQRoX.
Schrimpf, M., Kubilius, J.. Lec, M. J., Murty, N. A. R., Ajemian, R., and DiCarlo, J. J. In- tegrative benchmarking to advance neurally mech- anistic models of human intelligence. Neuron, 2020. URL https: /www. Fulltext/S0896 5273 (20) 30605-X.
Nayebi, A., Bear, D., Kubilius, J., Kar, K., Ganguli, S., Sussillo, D., DiCarlo, J. and Yamins, D. L. Task-driven convolutional recurrent models of the visual system. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Newral Information Process- ing Systems, volume 31. Curran Associates, Inc., neurips. Sbbe3ff78Sfaa0004777800000000023 pdf.
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. Laion-400m: Open dataset of clip- filtered 400 million image-text pairs, 2021.
2018. URL / ce/paper_fie
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C. W., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., Schramowski, P., Kundurthy, S. R., Crowson, K., Schmidt, L., Kaczmarczyk, R., and Jitsev, J. LAION-5b: An open large-scale dataset for train- ing next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https: / /oopeneeveww net wwtt/oorumidd-33333 4mmMMYY.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H. V., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F. El-Nouby, A., Howes, R., Huang, P.-Y., Xu, H., Sharma, V., Li, S.-W., Galuba, W., Rabbat, M., As- sran, M., Ballas, N., Synnaeve, G., Misra, I., Jegou, H., Mairal, J., Labatut, P., Joulin, A., and Bojanowski, Dinov2: Learning robust visual features without supervi- sion, 2023.
Susmelj, I., Heller, M., Wirth, P., Prescott, J., and Ebner, M. Lightly, 2020.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., Vetooo, Z., Raison, M.. Tejani. A.. Chilamkurthy, S., Steiner, B., Fang. L., Bai, J., and Chintala, S. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024- 8035. Curran Associates, Inc., 2019.
Tan, M. and Le, Q. EfficientNet: Rethinking model scaling for convolutional neural networks. In Chaud- huri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learn- ing, volume 97 of Proceedings of Machine Learning Research, pp. 6105-6114. PMLR, 09-15 Jun 2019. URL. https::t///........ //pp:://procee wwwpprooodinnnee ings mll wal...pees/v9]. tanl 9a html.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J.. Krueger, G., and Sutskever, Learning transferable visual models from natural language supervision, 2021.
Team, T. M. M. composer. https://github. com/ mosa i cml hwtp///opposee/e////oooo 2021.
Utthttss::/ http://aaxvvvv/. or..orggg/ooo.. abs//1033 00020.
Rajalingham, R., Issa, E. B., Bashivan, P., Kar, K., Schmidt, K., and DiCarlo, J. Large-scale, high-resolution com- parison of the core visual object recognition behavior of humans, monkeys, and oftt-oo--hhe-r deep artificial neural networks. The Journal of Neuroscience, 38(33): 7255-7269, 2018. doi: 10. 10.1523/jneuroscc.03888 18.211110000.00088081110011
Touvron, H., Cord, M., and Jégou, H. Deit ii:: Revenge of the vit, 2022. URL 2204. 07118.
Tu, Z., Talebi, H., Zhang, H., Yang, F,, Milanfar, P., Bovik, A., and Li, Y. Maxvit: Multi-axis vision transformer. ECCV, 2022.
Ridnik, T., Ben-Baruch, F.., Noy, A., and Zelnik-Manor, L. Imagenet-21k pretraining for the masses, 2021.
Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., and Belongie, S. The inaturalist species classification and detection dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Schrimpf, M., Kubilius, J., Hong, H., Majaj, N. J., Ra- jalingham, R. Issa, E. B., Kar, K., Bashivan, P., Prescott-Roy, Roy, J., Geiger, F,, Schmidt, K., Yamins, D.
13

---

Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream
Vasu, P. K. A., Gabriel, J., Zhu, J., Tuzel, O., and Ranjan, A. Fastvit: A fast hybrid vision transformer using structural reparameterization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Wightman, Pytorch image models. https://githu/ com rwi wwtggghtttpggggg wwwtte/pytttcchhiiiiieee meytorch-image-ooonne 2019.
Wong, F.,, Rice, l.,, and Kolter, 7. Fast better than free:
Revisiting adversarial training. In International Confer- ence on Learning Reprexentations, 2020. URL https: //openreview.o net wwt. Btte:/oooommnid-.deeee0
Yamins, D. L., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., and DiCarlo, J. J. Performance-optimizze hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, 111(3))8619-8624, 2014. doi: 10.1073/pnas. 12111.
Yamins, D. L. and DiCarlo, J. Using goal-driven deep learning models understand sensory cortex. Na- ture Neuroscience, 19(3)3556-36, February 2016. ISSN 1546-1726. doi: 10. 10.1038/nnn4244, URL http://dx: doi. wwt.oorgg/1... 038/ pn..n4.4444
Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1204-12113, June 2022.
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Tor- ralba, A. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
Zhuang, C., Yan, S., Nayebi, A., Schrimpf, M., Frank, M. C.,, DiCarlo, J. J., and Yamins, D. L. Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences, 118(3), 2021. doi: 10. 10.1003/ppas.2000 201.20111999 18.
14

---

Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream
A. Implementation Details
Our experiments are conducted using the PyTorch framework (Paszke et al., 2019), with Composer (Team, 2021) employed as the GPU orchestration tool efficiently manage computational resources.
For image augmentations, we leverage the Albumentations Buslaev et al., 2020 library due to its rich set of augmentation techniques, which are crucial for enhancing model robustness and preventing overfitting. In experiments involving self- supervised learning, we use the Lightly (Susmelj et al., 2020) library to facilitate the implementation of self-supervised losses, augmentations, and model heads. This library streamlines the process of setting up models for SimCLR and DINO training methods.
To generate adversarial examples for adversarial fine-tuning, we employ the Torchattacks library (Kim, 2020). Specifically, we use the Fast Gradient Sign Method (FGSM) to create perturbations that challenge the models, aiming to enhance their alignment with neural representations by exposing them to adversarial inputs.
B. Additional Image Datasets
To further validate our findings across diverse image distributions and to estimate scaling curves across different sample scales, we trained ResNet18 models on subsets several large-scale image datasets: ImageNet-21kkP, WebVision-P, iNaturalist, and Places365. Below, we provide detailed descriptions of each dataset.
B.1. ImageNet2Ik-P
ImageNet-2Ik-P is a processed subset of the full ImageNet-21X dataset (Ridnik et al., 2021), which originally contains over 14 million images organized into more than 21,000 categories following the WordNet hierarchy. The "P" denotes a enhance dataset quality, This results
pruned version where classes with insufficient images or noisy labels are filtered in refined dataset that maintains the richness of the original ImageNet-2Ik while improving label accuracy and image relevance. The resulting dataset contains approximately 11 million training images across 10,450 classes.
B.2. Web Vision-P
The WebVision dataset (Li et al., 2017) is large-scale web image dataset designed provide real-world, noisy alternative to ImageNet. It originally contains over 16 million images categorized into 5,000 classes. The images are collected from the internet using queries from search engines like Google and Flickr, leading to a dataset that includes label noise, varying image resolutions, and diverse visual contexts. Due to classes with very few available samples, we processed the Web Vision dataset similarly to ImageNet-21k-2 to remove classes with insufficient images. The resulting dataset, which we denote as Web WebVision-P, contains approximately 13.5 million training images across 4, 1,98 categories.
B.3. iNaturalist
iNaturalist (Van Horn et al., 2018) contains 2.7 million photographs of organisms in their natural enviromments, representing 10000 species. The dataset features highly specialized fine-grained categories and natural backgrounds, offering insight into how domain-specific visual features influence alignment scaling.
B.4. Places365
Places365 (Zhou al., 2017) is large-scale scene-centric dataset containing approximately 1.8 million training images across 365 scene categories. Unlike object-centric datasets such ImageNet, Places365 focuses on the recognition of environmental scenes, including natural landscapes, urban settings, and indoor environments. Each category includes wide variety of images to capture the diversity within scene types.
B.5. infiMNIST
The MNIST dataset (LeCun et al., 1998) is classic benchmark in machine learning, comprising 70,000 grayscale images of handwritten digits (0-9), each sized 28x28 pixels. To expand this dataset for more extensive experimentation, we utilize the Infinite MNIST (Infimnist) tool Loosli et al., 2007, which generates additional MNIST-like samples through data augmentation techniques. We create an extended dataset by modifying the original training dataset 19 additional times,
15

---

Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream
resulting in total of 1.2 million images. This enlarged dataset allows for more thorough evaluation of scaling effects the alignment.
B.6. CIFARIO
CIFAR-I0 (Krizhevsky et al.) is widely used benchmark of 60,000 low-resolution (32x32) color images divided evenly into 10 object classes. The dataset comprises 50,000 training images and 10,000 test images, with 6,000 samples per class. To match our scaling protocol, created class-balances subsets by sampling d E {10, 30, 100, 300, 1000, 3000} images class. Because CIFAR-10 is two orders of magnitude smaller than our other baselines, in addition to our standard 100-epoch runs we also trained models on the full dataset for extended durations of 250, 1000, and 2500 epochs to assess convergence and scaling effects.
C. Validation on private data
V1
V2
V4
0.30
R2:0.97
R²: 0.95
R²:0.99
0.25
p-value #3.18e-22
p-value #6.75e-19
p-value 11.06-22
Private
0.20
Private
0.2
Private
0.4
0.15
0.1
0.2
0.05
0.0
Public
Public
Public
IT
Behavioral
Average
R²:1 00
R²: 0.99
1.0000
p-value =1 87e-32
p-value =4, 790-27
p-value =9 626-36
0.4
Private
Private
Private
0.2
0.2
0.2
0.0
0.0
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.2
0.4
0.1
0.2
0.3
0.4
Public
Public
Public
Figure S1: Public benchmarks used in this study correlates highly with private benchmarks Brain-Scon
As described in Section 2 a diverse set of models on private benchmarks on Brain-Score platform. All Ri2 values above 0.95 with p-values less than 10
16

---

Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream
D. Pretrained Models
As part our comprehensive evaluation, we benchmarked a diverse set of pretrained models sourced from both torchvision (maintainers & contributors, 2016) and the t timm (Wightman. 2019) libraries. We tested total of 94, including ViT (Dosovitskiy et al., 2021), DaViT (Ding al., 2022), VeViT (Graham al., 2021), ConvNeXt (Liu et al., 2022a), Vib (Mehta & Rastegari, 2022), Max (Tu et al., 2022). FastViT (Vasu et al., 2023). Each model varies in parameter count, training sample size, dataset source, and training objective, providing broad spectrum for analysis.
To verify the generalizability of our findings, we conducted evaluations with these pretrained models, including larger networks like CLIP (Radford et al., 2021) and DINOv2 (Oquab et al., 2023), which are pretrained on richer and more diverse datasets such as LATON (Schuhmann al., 2021; 2022). We also compared variations of these models by examining base pretrained models alongside their fine-tuned counterparts on ImageNet, aiming investigate the impact of fine-tuning on scaling behavior.
Our results indicate that models with extensive pretraining achieve enhanced behavioral alignment, likely due to their exposure to richer and more varied data. However, similar to models trained solely on ImageNet or EcoSet, these pretrained models still exhibit saturation effect in neural alignment with the primate visual ventral stream (VVS). This suggests that while larger and more diverse datasets improve behavioral predictability, they do not substantially extend the scaling of neural alignment beyond the observed plateau.
The curves in Figure S2 closely follow the scaling patterns estimated for our trained models shown in Figure 2.c, further validating that the observed saturation is consistent across different pretraining regimes and dataset scales. This reinforces conclusion that scaling alone is insufficient to overcome the limitations in neural alignment and highlights the need for alternative approaches to improve alignment with neural representations.
0.50
Alignment Score (S)
0.40
ConvNeXt
DaVit
EfficientNet
FastVIT
LeViT
MaxxVIT
MobileViT
ResNet
0.30
ViT
0.43
10
10
10
10
Model Parameters (N)
Figure S2: Alignment of pretrained vision models as a function of model parameters
17

---

Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream
E. Training Evolution
ResNet
EfficientNet
Aligmment Score (5)
0,:00
Allgiment
0:30
020
510
0/0
12
10
107
10
10'
Training Epoch
Training Epoch
VIT
ConvNeXt
V2
Alignient Sopte (5)
0.50
6:40
0:2
0.00
Training Epoch
Training
Figure S3: Evolution of per-region alignment throughout training. Models with stronger rriors-such ResNet and EfficientNet-extee higher neural alignment initially. However, the gap in representational power diminishes more generalist models are trained on data.
18

---

Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream
Neural Alignment
Alignment Score (S)
0.3
Trained
False
0.2
True
0.1
0.0
ResNett8
ResNa134
ResNer101 ResNa150
ResNet 62
EfficienNeet EffciennNet-B
EffcientNel-Be
ConvNext- ConvNext-S
ConvNexta
ComWeelL
VIT-
VT--S
VIT-B
VIT-L
Model
Behavioral Alignment
0.5
Alignment Score (S)
0.3
Trained
False
True
0.1
-0.1
ReaNet1a
ResNet34
ReaNel50
ResNer101
EffidentNet BB0
EffidanNhets
EfficianNee-B
ConNNeXI-T
ConVNeXLs
ConvNoxtB
ComAieXLL
VIT-T
ViTss
VIT-B
VIT-L
Model
Figure $4: Some untrained models demonstrate non-zero neural alignment at initialization. Nevertheless, all models start with almost zero behavioral alignment, indicating that initial neural alignment arises from architectural biases rather than learned behavior.
19

---

Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream
F. Effect of Training Objective
Effect of Learning Paradigms
Alignment Score (S)
0.36
0.34
0.32
4TT-B
umss
Rostet-5
Rewtee-15 1a
Model
ViT-S Aligment
Trairing Melhod
0.5
DINO
Alignment Score (S)
SimOLR
0.3
6.1
V1
V2
Bahevbnn
Region
ResNet-50 Aligment
0.6
Trairing Melhod
Supervised SimCLR
0.5
Alignment Score (S)
V1
v2
Batteviond
Region
Figure S5; Effect of training objective on alignment
20

---

Scaling Laws for Task-Optimizer Models of the Primate Visual Ventral Stream
SimCLR Training Regions
V2
0.70
0.70
0.70
Alignment's Score (5)
3.80
0/90
0:50
8.50
0.00
Allgrmmet Score
0,50
8.40
0.00
0:00
1:00
-..00 41
5=0.001 -1200 1220
1.497
Flops (C)
Flops
Flops (C)
IT
Behavioral
Average
Alignment Score (5)
Allgrmmnt Score (5)
Allgnment Score
0.30
0:00
0.30
1.2
020
0.20
3.10
810
6:10
0.50
3,00--911
1.20
2.39
0.49 0.810-311
12
10
10"
16"
Total Flops (C)
Total Flops
Total Flops
Figure S6: Scaling of SimCLR training across regions
21

---

Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream
G. Region-wise scatter plots
Aiginnert Siow Sirm
Accurccy
Angriment Soret 21
Alginse down for 12
Abgin Betaviorel VV
Algined Boonn de
Sehawordl
Alge Stcrette
Bethortordl 34d
Behevtoral
Behavbrrl Accuray
Magreset
Figure S7: Region vs. Region Comparisons: This figure shows how the alignment scores for each brain region correlate
with those other regions. The diagonal plots illustrate the relationship between the alignment score of each region and the validation accuracy on ImageNet and EcoSet datasets.
22


Reviews:
[{"review_id": "6t2gSFPt1f", "reviewer": "Reviewer_weze", "summary": "The paper introduces scaling laws for task-optimized models in fitting neural recordings. These laws stem from the observation that neural networks trained on classification tasks have emerged as the most effective models for decoding neural activity in the brain. The study then evaluates a measure of alignment across neural and behavioral benchmarks using publicly available datasets from BrainScore. Finally, the authors draw conclusions about how dataset size and architecture type influence behavioral responses, shedding light on their roles in neural decoding performance.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review - we’re glad you found the paper informative and comprehensive in its overview of models and progress on neural and behavioral benchmarks.  We respond to each of your comments below.\n\n> Claims And Evidence - limited model set:\n\nThe reviewer might have missed this in the paper, but a core focus of our work is to evaluate a variety of architectures  (Figs 2, 3, S2) from standard CNNs (ResNet, AlexNet, EfficientNet) to recent ones (ConvNeXt), Vision Transformers (ViT, DaVit, FastViT, LeViT, MaxxViT, MobileViT) and CORnet. The recurrent architecture CORnet is notably present in all our analyses (lines 133-145 left column; Figs 3, 4, 6) but we do not observe a difference in its alignment at larger scales. Interestingly in Fig 3b, CORnet initially exhibits improved alignment in low-data conditions, indicating that recurrence might provide a meaningful inductive bias for efficiency under limited training samples. However, as the amount of training data increases, the advantage offered by recurrence diminishes, suggesting that deeper feedforward models can approximate recurrent processing by effectively \"unrolling\" recurrent computations through additional layers. Thus, recurrence remains beneficial primarily in data-constrained scenarios, providing sample efficiency advantages, but larger datasets reduce its relative advantage. Inspired by the reviewer’s question, we will highlight this point more prominently in the updated paper.\n\n> The paper offers a wide range of experiments that validate the scaling laws, however the results described seem to resemble facts that were reported before. \n\nPlease see our first response to the reviewer **6Eek**.\n\n> Weaknesses\n\nPlease see the answer below for your first question.\n\n> How can the scaling laws inform future selection of architectures data to improve our understanding of the visual cortex? \n\nOur scaling laws explicitly guide future architecture and dataset development across varying computational budgets. Specifically, our results suggest that architectures with biologically-inspired inductive biases—like convolution or recurrence—achieve superior alignment more efficiently, especially under limited compute (Figs 2, 3b, 4b). Furthermore, our findings strongly advocate for developing richer and more ecologically valid datasets, which consistently yield greater alignment improvements compared to increasing model complexity alone (Fig 3a, Sec 3.3). Additionally, the graded effect of scaling observed across the cortical hierarchy (Fig. 5) suggests that early visual areas (V1/V2) benefit relatively little from scaling alone, highlighting the need for stronger biologically-informed inductive biases early in the processing pathway, e.g. VOneNet. Finally, we hypothesize that to push neural encoding models beyond the current alignment plateau, explicitly incorporating neural recordings into model training may be necessary. Our preliminary results on fine-tuning models on IT neural data (Papale et al., 2024) are promising: https://imgur.com/a/okfmRu8 (anonymous link for figures). Fine-tuning on a single region from another dataset improves neural alignment, with a stronger effect for a weakly-biased model (ViT-Small) than for a CNN (ResNet50).\n We will clearly discuss these directions and practical implications in our revised manuscript.\n\n> How biological inductive bias can be quantitatively described so it can be easy to understand the claim \"Architectural Bias influences alignment in behavior?\n\nWe note that the exact subtitle (\"Architectural Inductive Bias Influences Alignment and Scaling Behavior\") was not specifically about behavioral alignment but rather about scaling properties in general, i.e. the “behavior of scaling”. We see how the phrasing can be misleading and will revise it for clarity.\n\nOur results show that biological inductive biases mainly affect the scaling properties of neural alignment, especially in low-data regimes. Specifically, Figure 5b illustrates clearly that behavioral alignment benefits significantly more from task optimization compared to neural alignment. Further, Figure 5a demonstrates that for behavioral alignment, models with strong versus weak inductive biases closely follow the same scaling trajectory, while significant differences remain for neural benchmarks. Indeed, after extensive training, models with weaker inductive biases (e.g., ViTs) even achieve slightly higher absolute behavioral alignment scores than models with stronger inductive biases.\n\nQuantitatively, these biases shift neural alignment scaling more than behavior. Exploring other priors—like local connectivity or predictive coding—is an exciting direction.\n\nThank you again for your review. We kindly ask the reviewer to consider raising their score and take into account the results that might have been missed in the first pass, such as the diversity of architectures, the novelty of our findings, and the impact on future model building.", "claims_and_evidence": "While the claim that 'architectures with stronger inductive biases are more sample- and compute-efficient' is compelling, the evidence provided is limited to convolutional neural networks (CNNs). Other architectures with well-established inductive biases—such as CorNet or recurrent neural networks (RNNs)—are notably absent from the analysis. Including these architectures would strengthen the validity of the claim, as their exclusion leaves critical gaps in testing the generalizability of the hypothesis.", "methods_and_evaluation_criteria": "yes.", "theoretical_claims": "I checked the derivation for scaling laws, it seems correct.", "experimental_designs_or_analyses": "I checked the experimental design for: \n\n* Total flops vs Alignment score. \n* Parameter size Alignment score. \n* Number of samples vs Alignment score. \n* Learning dynamics and architecture nature. \n* Adversarial finetunning and benefits to neural fit.", "relation_to_broader_scientific_literature": "The paper offers a wide range of experiments that validate the scaling laws, however the results described seem to resemble facts that were reported before. For instance, the effects of model size on neural alignment and  has been previously reported (Linsley et al, 2023, Conwell et al, 2022 ).", "essential_references_not_discussed": "The cites cover most of the literature on the field.", "other_strengths_and_weaknesses": "Strenghts: \n\nThe paper offers a good view of the progress in the field in terms of neural  and behavioral benchmarks. \nIt offers a good overview, covering  a very long list of models. \n\nWeakness: \n\nThe main limitation I find of the paper is that it is little indication on how to move forward and how the scaling laws can impact and inform architecture or data diet development?", "other_comments_or_suggestions": "No.", "questions_for_authors": "* How can the scaling laws inform future selection of architectures data to improve our understanding of the visual cortex? \n* How biological inductive bias can be quantitatively described so it can be easy to understand the claim \"Architectural Bias influences alignment in behavior?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J2bjpH120w", "reviewer": "Reviewer_6Eek", "summary": "In the paper \"Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream\", the authors investigate scaling laws for alignment of machine learning models to the primate visual processing. They assess neural alignment as well as behavioral alignment, and find that there is a saturation point in neural alignment, but not in behavioral alignment. They further evaluate the properties of models with strong inductive bias versus weak inductive bias, and find that models with stronger inductive bias are more sample- and compute-efficient. Further analyses include the individual assessment of areas along the visual hierarchy and the comparison of different training strategies.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive review and are glad you found the paper well-written, the results clear, and the evidence supporting our claims solid. Below, we address your questions point by point.\n\n>  In light of these previous works, it might be expected that scaling up models and data would be insufficient to achieve better alignment for neural data. Therefore, the main findings of the paper are not all that surprising and I find the novelty of the paper to be limited. The findings are empirical in nature and do not provide a deeper understanding for the reasons behind the observed scaling laws, other than the inductive biases of the models.\n\nOur work substantially advances previous literature by providing a more systematic, controlled, and comprehensive analysis that isolates how individual factors—model parameters, dataset size, and compute resources—each influence alignment with multiple brain regions and behavior. Earlier studies on the other hand used heterogeneously pre-trained models and compared to limited brain datasets. \n- **Generalization of brain regions:** Prior works focused narrowly (e.g., only IT in Linsley et al., 2023) or used noisy fMRI signals (Conwell et al., 2022). We evaluate alignment across the full primate ventral stream (V1–IT) using high-resolution intracortical recordings, along with object recognition behavior—both novel in this context.\n- **Model comparability:** Previous studies used pretrained models with inconsistent training recipes, making comparisons hard. Our models are all trained from scratch under uniform conditions, enabling a fair, apples-to-apples comparison across architectures and datasets.\n- **Our comprehensive study led to updated findings:** We show that task performance does correlate with neural alignment (Fig. 6a), but with diminishing returns—contrary to prior conclusions. Moreover, our work quantifies the individual effects of data, parameters, and compute (Figs. 2–4), and provides a parametric framework for understanding scaling (Figs. 4–7), going well beyond simple correlations in earlier work.\n\n- Finally, we contribute new insights such as a graded scaling effect across the visual hierarchy and a clear behavioral/neural alignment dissociation (Fig. 5), offering deeper understanding of how brain-like representations emerge. We’ll emphasize these distinctions more clearly in the revised manuscript and welcome suggestions for where to further highlight them.\n\n\n> Figure panel 3b is not described in the text.\n\nWe thank the reviewer for pointing out this oversight. Figure 3b illustrates that model families with weaker inductive biases (e.g., ViT and ConvNeXt) begin training with lower neural alignment scores and consequently require larger training datasets to achieve alignment comparable to architectures with stronger inductive biases (e.g., ResNet, EfficientNet). Additionally, we find that recurrence, as implemented in CORnet architectures, provides a significant initial advantage in alignment relative to purely convolutional models, particularly in low-data regimes. However, this advantage diminishes with increased training data. Overall, these findings emphasize that strong inductive biases—such as convolution and recurrence—facilitate better alignment when data is limited, whereas extensive task-driven optimization eventually mitigates differences across architectures. We will integrate this explanation into the main text of the revised manuscript.\n\n> In line 313, right column, you speculate that factors other than task performance influence neural alignment. Do you have any ideas about what these factors might be?\n\nOur findings suggest that architectural inductive biases significantly influence neural alignment independently from task performance. Additionally, as illustrated in Figure 5b, higher cortical regions and behavioral alignment benefit more from task optimization compared to early visual areas. Moreover, supplementary analysis (Figure S7) shows that correlations between alignment and task performance also follow the cortical hierarchy. These observations imply that incorporating stronger biologically-inspired priors (e.g., convolutional constraints, local receptive fields, recurrence) specifically for modeling early visual regions, combined with more flexible, data-driven layers for higher cortical areas—akin to the design philosophy of architectures like VOneNet—may yield improved neural alignment. Furthermore, explicitly incorporating neural data into training procedures (via co-training or fine-tuning, see our response to the reviewer **weze**) represents a promising additional strategy to surpass the current limitations of purely task-optimized models.\n\nThanks again for your review. In light of the systematic advances over prior work—such as our controlled experimental setup, broader neural benchmarks, and novel findings on graded scaling and behavioral dissociation—we kindly ask you to consider raising your score.", "claims_and_evidence": "The authors claim that there is a saturation point in neural alignment, but not in behavioral alignment. They also claim that increasing both parameter count and training dataset size improves alignment, with data providing more gains over model scaling. Architectures with stronger inductive bias and datasets with higher-quality images are more sample- and compute-efficient. Finally, they claim that model alignment with higher-level brain regions and especially behavior benefits the most from scaling. They support these claims with a wide range of experiments and fits of scaling laws to the data. Given the range of models and analyses that they carried out, the evidence for their claims looks solid to me.", "methods_and_evaluation_criteria": "The authors use benchmarks from Brain-Score, which includes comparisons of model outputs to recordings from the primate visual ventral system, as well as a comparison of model predictions with behavioral data. They use a range of model architectures, including AlexNet, ResNets, CORnet-S, EfficientNets, Vision Transformers, and ConvNeXt, and fit various functional forms of the scaling laws to account for inductive biases in the model architectures and their data requirements. Models were assessed on a range of image datasets, including ImageNet and EcoSet, as well as a range of other image datasets. They also assessed alternative training strategies, including SimCLR, adversarial training, supervised learning, and self-supervised learning. Bootstrapping was used to quantify uncertainty. The selection of models includes state-of-the-art models and a wide range of architectures, which is a strength of the paper. The authors also provide a detailed analysis of the training strategies, which is important for understanding the results. The evaluation criteria are well-defined and the methods are appropriate for the research questions.", "theoretical_claims": "The paper does not make theoretical claims.", "experimental_designs_or_analyses": "There are experiments assessing model size and dataset training size, as well as the effect of inductive bias on alignment. The authors also assess the alignment of models with different brain areas and behavior. Then there are experiments using alternative training strategies. The breadth of the experiments is a strength of the paper, and the authors provide a detailed description of the methods used.", "relation_to_broader_scientific_literature": "The functional form of the misalignment follows Hoffmann et al., 2022, a study for the scaling of large language models. There are changes in the function for optimal allocate of compute to account for empirical observations. The benchmarking itself follows Schrimpf et al., 2018. Linsley et al., 2023 showed that large DNNs rely on different visual features than than those encoded by area IT. In light of these previous works, it might be expected that scaling up models and data would be insufficient to achieve better alignment for neural data. Therefore, the main findings of the paper are not all that surprising and I find the novelty of the paper to be limited. The findings are empirical in nature and do not provide a deeper understanding for the reasons behind the observed scaling laws, other than the inductive biases of the models.", "essential_references_not_discussed": "To the best of my knowledge, the paper discusses all essential references.", "other_strengths_and_weaknesses": "The paper is well-written with good structure, sufficient details for making the work reproducible and a clear presentation of the results.", "other_comments_or_suggestions": "Figure panel 3b is not described in the text. It would be helpful to include a brief description of the figure in the main text.", "questions_for_authors": "In line 313, right column, you speculate that factors other than task performance influence neural alignment. Do you have any ideas about what these factors might be?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3KALAmTR7W", "reviewer": "Reviewer_Jfvv", "summary": "This paper seeks to measure scaling laws for task-optimized models of the primate visual ventral stream. Several models from multiple families were trained with different amounts of compute and training data. They were then compared on their alignment to different areas of the visual cortex (V1, V2, V4 and IT), as well as on behavior data from non-human primates. Scaling laws were then fit to relate how neural and behavioral alignment scale with flops and data. The top result is that neural alignment asymptotes with scaling. The paper then breaks down these results in terms of areas, architectures, dataset dependence, etc.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback and thoughtful comments. We are particularly encouraged that the reviewer highlighted the key strengths that we were indeed most excited about in our paper:\n\n- Our systematic, extensive evaluation of scaling laws across hundreds of models and diverse benchmarks, providing a new insight into the role of architectural bias and choice of training, and a recipe for optimal compute allocation.\n- Thorough comparisons across various architectures (ResNet, EfficientNet, ViT, ConvNeXt, CORnet), multiple datasets (ImageNet, EcoSet, iNaturalist, infiMNIST, Places365), and alternative training methods (SimCLR, DINO, adversarial training).\n- Clear evidence showing neural alignment saturates with increased model and dataset scaling, providing a significant conceptual step forward to encourage the exploration of new modeling ideas.\n- The depth and rigor of our analyses, including validation with held-out benchmarks and alignment dynamics during training, showing for instance that under controlled model training, task performance and brain alignment remain weakly correlated.\n- Comprehensive results specifically targeted at the primate visual cortex, offering deeper insights beyond prior work, such as the ordered effect of scale on alignment and the role of inductive biases.\n\nWe greatly appreciate the reviewer's supportive assessment and enthusiasm for our study.", "claims_and_evidence": "This is a very thorough investigation of scaling laws in visual cortex, fitting hundreds of models on a broad range of downstream tasks. The authors' top-line result, that alignment saturates, is surprising (though in line with prior literature); it's very well-supported; and it is a significant conceptual advance for the field.", "methods_and_evaluation_criteria": "The authors fit a broad range of models: ResNet, AlexNet, EfficientNet, ViTs, ConvNext, Cornet. They selected two different core datasets: imagenet and ecoset; and alternatives including iNaturalist, infiniMNIST, Places365, etc. They also try out alternative training scenarios beyond supervised learning, including SimCLR, DINO and adversarial fine-tuning. This is a very broad range of models that supports their main conclusions.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The methods are all standard and well-accepted; where the paper innovates is in the thoroughness of its evaluation. I have no qualms.", "relation_to_broader_scientific_literature": "Much has been written about the fact that Brain-Scores and similar measures are saturated, and that multiple models converge on the same (wrong) representations; examples include Conwell et al. (2024) and Linsley et al. (2023). However, this submission stands out for its completeness, the depth of its analysis, and its focus on primate vision.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GrJc9MPAS4", "reviewer": "Reviewer_XLky", "summary": "This paper explores how scaling model size, dataset size affects the alignment of artificial neural networks with primate visual ventral stream behaviors and neural responses. The scaling law is investigated over diverse models on benchmarks including v1, v2, v4, IT and behavior data.\nThe authors offer interesting findings w.r.t. neural alignment and behavioral alignment.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review — we’re glad you found the paper well-written and organized, appreciated the breadth of our experiments and visualizations, and found the findings insightful. We respond to each of your comments below point-by-point.\n\n\n> The practical implications of the authors' conclusions remain unclear. For instance, while the authors highlight that models with strong architectural inductive biases, such as convolution-based ResNets, show better neural alignment trends than transformer-based ViTs, ViTs are still more widely adopted in the community due to their superior overall performance compared to convolution nets.\n\nSince our primary goal is modeling brain information processing (i.e., neuroscience), we consider task performance mainly as a proxy rather than the ultimate target. Indeed, our experiments show ViTs excel at behavioral alignment (Figs. 5a, 6b) but underperform relative to convolutional models in neural alignment benchmarks (Figs. 5a, 6a). This also holds for very large models trained on massive datasets (Fig S2, ConvNeXt vs ViT).  Practically, this suggests a promising path forward could involve hybrid architectures that combine biological inductive biases (e.g., convolutional or hierarchical connectivity) with transformer-based scalability and flexibility such as a VOneBlock (Dapello et al. 2020) combined with ViTs (Dosovitskiy et al. 2020), thus balancing task performance with improved neural alignment. Specifically, one viable strategy could be distilling representations from high-performing transformer models into architectures with hierarchical inductive biases resembling biological circuitry. \n\n> The authors reveal the scaling laws between neural alignment, behavioral alignment, and data size, model size through a series of impressive experiments. However, what is the relationship between improving neural alignment and enhancing downstream performance? What insights can these scaling laws provide for improving model design?\n\nIn this study, we primarily focus on building better models of the human brain, rather than developing better  ML models for various tasks. In this context, our findings show a clear correlation between behavioral alignment and task performance (Figs. 1, 6b), consistent with the widely accepted notion in the ML community that increased compute leads to better-performing models. In contrast, neural alignment shows positive but diminishing returns with improved object recognition performance (Fig. 6a), indicating that improving neural alignment may not always directly translate into better downstream task performance.\n\nPractically, our scaling laws offer concrete guidance for future model design targeting neural alignment. First, our results strongly suggest prioritizing larger, richer, and more diverse datasets, as they consistently yield greater neural alignment improvements compared to merely scaling model complexity (Fig. 3a). Second, our findings emphasize the crucial role of biologically inspired inductive biases, particularly in scenarios constrained by limited compute or data, as these priors substantially enhance alignment efficiency (Figs. 2, 3b). Lastly, the graded scaling effect across the cortical hierarchy (Fig. 5) indicates that early visual regions (V1/V2) are the most challenging to align through scale alone, suggesting that architectures incorporating stronger biologically-informed priors (e.g., VOneNet-style architectures) may be necessary to achieve further improvements.\n\nThanks again for your review. We ask the reviewer to consider raising their score in light of this paper’s placement in computational neuroscience and a focus on modeling the brain.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The scaling laws of large models have been extensively studied and applied. This paper explores another scaling law of models from the perspective of brain alignment. The authors offer an interesting perspective on how mainstream models align with human core object recognition behavior, and the impact of this biological alignment.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths\n\n- The paper is well-written, well-organized and easy to follow.\n- The paper provides extensive experiments across various model architectures and well-illustrated visualizations.\n- The authors provide several interesting findings, including (1) scaling is particularly beneficial for higher-level visual areas, (2) neural alignment saturates in most conditions whereas behavioral alignment continuously improves when scaling.\n\n\n\nWeaknesses\n\n\n- The practical implications of the authors' conclusions remain unclear. For instance, while the authors highlight that models with strong architectural inductive biases, such as convolution-based ResNets, show better neural alignment trends than transformer-based ViTs, ViTs are still more widely adopted in the community due to their superior overall performance compared to convolution nets.\n\n- The authors reveal the scaling laws between neural alignment, behavioral alignment, and data size, model size through a series of impressive experiments. However, what is the relationship between improving neural alignment and enhancing downstream performance? What insights can these scaling laws provide for improving model design?", "other_comments_or_suggestions": "None", "questions_for_authors": "See Other Strengths And Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper systematically investigates how behavioral alignment (=task performance) and neural alignment (=representational similarity with brain) scale with model size across a set of visual recognition models. The authors conclude that while behavioral alignment keeps improving, neural alignment has saturated. While this observation is not entirely new, the current paper presents a thorough, in-depth analysis of the problem that goes substantially beyond prior work in its breadth and level of systematic investigation. As a result, I believe this is a significant and important contribution that identifies pure scaling as a dead end and implies that other approaches (e.g. direct fit to neural data, more diverse behavioral task) are necessary. I strongly recommend acceptance.
Decision: Accept (spotlight poster)