Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream

## Abstract

Abstract When trained on large-scale object classification datasets, certain artificial neural network models begin to approximate core object recognition (COR) behaviors and neural response patterns in the primate visual ventral stream (VVS). While recent machine learning advances suggest that scaling model size, dataset size, and compute resources improve task performance, the impact of scaling on brain alignment remains unclear. In this study, we explore scaling laws for modeling the primate VVS by systematically evaluating over 600 models trained under controlled conditions on benchmarks spanning V1, V2, V4, IT and COR behaviors.
We observe that while behavioral alignment continues to scale with larger models, neural alignment saturates.
This observation remains true across model architectures and training datasets, even though models with stronger inductive bias and datasets with higher-quality images are more compute-efficient.
Increased scaling is especially beneficial for higher-level visual areas, where small models trained on few samples exhibit only poor alignment.
Finally, we develop a scaling recipe, indicating that a greater proportion of compute should be allocated to data samples over model size.
Our results suggest that while scaling alone might suffice for alignment with human core object recognition behavior, it will not yield improved models of the brain’s visual ventral stream with current architectures and datasets,
highlighting the need for novel strategies in building brain-like models. 1 1 1 We open-source all code and checkpoints: github.com/epflneuroailab/scaling-primate-vvs .

## 1 Introduction

The advent of neural networks has revolutionized our understanding and modeling of complex neural processes. A particularly active area of study is the ventral visual stream in primates, a key pathway in the brain responsible for processing visual information Goodale & Milner (), Grill-Spector et al. (), Malach et al. (), Kriegeskorte et al. (). Neural networks, when trained on extensive datasets, have emerged as the most accurate quantitative tools for simulating the response patterns of neurons within this stream Yamins et al. (), Schrimpf et al. (). These advanced models offer a precise computational account of how neural mechanisms in the brain give rise to visual perception.

Recent developments in machine learning have emphasized the significance of both the volume of training data and the complexity of model architectures Kaplan et al. (), Hoffmann et al. (), Zhai et al. (), Bahri et al. (), Antonello et al. (), Muennighoff et al. (), Aghajanyan et al. (), Isik et al. (). These findings raise the question: Can we build better models of the brain by scaling up model architectures and dataset sizes? Recent studies have found that in pre-trained models, the number of parameters and dataset samples respectively seem to improve predictions of fMRI and behavioral measurements (Antonello et al., , Muttenthaler et al., ).
With the numerous differences between pre-trained models however, the relative contributions of model parameters and dataset size to brain and behavioral alignment are not clear.

In this paper, we examine how scaling – of model parameters and training dataset size – impacts the alignment of artificial neural networks with the primate ventral visual stream.

We systematically train models from a variety of architectural families on image classification datasets which allows us to independently control and observe the effects of model complexity and data volume.
To capture the observed trends, we introduce parametric power-law trends that describe the impact of scale on alignment with behavior and brain regions along the visual ventral stream.

We summarize the contributions of this work as follows:

- •
While scale initially improves alignment, brain alignment saturates whereas behavioral alignment continues to improve.
- •
Increasing both parameter count and training dataset size improves alignment, with data providing more gains over model scaling.
- •
Architectures with stronger inductive bias (such as convolutions) and datasets with higher-quality images are more sample- and compute-efficient.
- •
Fitting parametric power-law curves, we find that model alignment with higher-level brain regions and especially behavior benefits the most from scaling.
- •
We publicly release our training code, evaluation pipeline, and over 600 checkpoints for models trained in a controlled manner to enable future research.

Figure: Figure 1: a) For a given compute budget ($C$), we determine the scaling laws for maximal neural and behavioral alignment to the primate visual ventral stream. b) We find consistent scaling laws for brain and behavioral alignment across over 600 models. While we predict models to approach perfect behavioral alignment at large scales, the effect of scaling on brain alignment is already saturating.
Refer to caption: x1.png

## 2 Related Work

#### Primate Visual Ventral Stream.

The ventral visual stream, a critical pathway in the primate brain, including humans, plays a key role in visual perception, extending from the occipital to the temporal lobes and serving as the ”what pathway” for object recognition and form representation Goodale & Milner (), Grill-Spector et al. (), Malach et al. (), Kriegeskorte et al. ().
Beginning in the primary visual cortex (V1), where basic visual information from retinal ganglion cells is processed, the ventral stream proceeds through areas such as V2, V3, V4, and the inferotemporal cortex (IT), each responsible for increasingly complex features of visual perception Kandel et al. (). Despite decades of research and a wealth of brain data, the precise neural mechanisms underlying visual perception are not well understood.

#### Modeling the Primate Visual Ventral Stream.

Particular artificial neural networks (ANNs) are the most accurate models of brain responses in the visual ventral stream and associated core object recognition behaviors Schrimpf et al. (; ). Models optimized for ecologically viable tasks (Yamins & DiCarlo, ) in particular have demonstrated strong brain and behavioral alignment (Yamins et al., , Khaligh-Razavi & Kriegeskorte, , Cadena et al., , Schrimpf et al., , Nayebi et al., , Kietzmann et al., , Rajalingham et al., , Zhuang et al., , Geiger et al., ) – notably these models are trained purely on image classification datasets, without fitting to brain data.

#### Scaling Laws.

Recent advancements in artificial intelligence are driven by scaling the model size and training data. Empirical evidence suggests a power-law relationship between model performance and both model parameters and dataset size, indicating that continued scaling will further improve performance Kaplan et al. (), Cherti et al. (), Zhai et al. (), Hoffmann et al. (), Dehghani et al. (), Henighan et al. (), Brown et al. (), Bahri et al. (), Hestness et al. (). The power-law exponents enable the optimal allocation of compute between model parameters and dataset samples, such that performance is maximized Kaplan et al. (), Hoffmann et al. ().

While scaling laws for machine learning *performance* has been extensively studied, the scaling laws for *brain alignment* remain unclear.
Recent studies indeed suggest an involvement of both model size and data volume in the functional alignment with brain data Azabou et al. (), Benchetrit et al. (), Caro et al. (), Antonello et al. (). Conversely, Muttenthaler et al. Muttenthaler et al. () indicate that sample size is critical for behavioral alignment.
We here unify these results, in the realm of the primate visual ventral stream, into quantitative scaling laws for how model and dataset sizes relate to alignment with the brain and behavior.

## 3 Methods

#### Neural & Behavioral Alignment.

To evaluate the alignment of our model with brain function, we utilize a range of benchmarks from Brain-Score Schrimpf et al. (; ). These benchmarks assess model performance by comparing model activations or behavior with primate neural data using the same images. Specifically, the V1 and V2 benchmarks compare model outputs to primate single-unit recordings from Freeman et al. (), using 315 texture images and data from 102 V1 and 103 V2 neurons. For the V4 and IT benchmarks, 2,560 images are used to match model activations to primate Utah array recordings from Majaj et al. (), based on data from 88 V4 and 168 IT electrodes. A linear regression is trained on 90% of the images to correlate model and neural data, with prediction accuracy for the remaining 10% evaluated using Pearson correlation, repeated ten times for cross-validation. The behavioral benchmark assesses model predictions for 240 images against primate behavioral data from Rajalingham et al. () using a logistic classifier trained on 2,160 labeled images. Pearson correlation is used to measure the similarity in confusion patterns between model predictions and primate responses. All benchmark scores are normalized to their respective maximum possible values.

We define the model’s alignment score $S$ (and an inverse Misalignment Score $L=1-S$) as the average across the V1, V2, V4, IT, and behavioral benchmark scores. Layers are committed to brain regions based on models trained on a full dataset, and applied to all variants trained with subsampled datasets.
As we reused the same neural and behavioral data both to select the optimal model layer for readout and to assess the model’s alignment, we validated the benchmark results on a private split of each dataset on Brain-Score. We observed an almost perfect correlation between the results on the private and public splits (Appendix A).

#### Scaling Models and Data.

We trained an array of standard models from several architecture families. Specifically, we used ResNet${18,34,50,101,152}$ from He et al. (); EfficientNet-B${0,1,2}$ from Tan & Le (); Vision Transformer ViT${\textrm{T},\textrm{S},\textrm{B},\textrm{L}}$ from Dosovitskiy et al. (); ConvNeXt${\textrm{T},\textrm{S},\textrm{B},\textrm{L}}$ from Liu et al. (); CORnet-S from Kubilius et al. (); and AlexNet from Krizhevsky et al. (). We also trained 33 modified versions of ResNet18: 22 models obtained by scaling the network width from $1/16$ to $4$ times the original size, and 11 models derived by adjusting the depth. Similarly, we trained four additional ConvNeXt and ViT models by scaling the width of the ConvNeXt-T and ViT-S architectures.

For our experiments, we selected two image classification datasets: ImageNet Deng et al. () and EcoSet Mehrer et al. (). ImageNet, with millions of labeled images across 1,000 categories, has long been a benchmark in computer vision, designed to challenge and evaluate automated visual object recognition systems. On the other hand, EcoSet is a more recent dataset, designed to provide an ecologically valid representation of human-relevant objects. It contains over 1.5 million images spanning 565 basic-level categories, curated to better reflect the natural distribution of objects in the real world, aligning with human perceptual and cognitive experiences.

To create subsets of ImageNet and EcoSet, we sampled $d\in{1,3,10,30,100,300}$ images per category. For $d\in{1,10,100}$, we repeated the runs with three random seeds to ensure robustness. For ConvNeXts (Liu et al., ) and ViTs (Touvron et al., ), we used the training recipes developed by the original model authors. The remaining models were trained for 100 epochs using a minibatch size of 512. We employed a stochastic gradient descent (SGD) optimizer with a cosine decaying learning rate schedule, starting with a peak learning rate of 0.1 and incorporating a linear warm-up phase spanning five epochs. We maintained the momentum at 0.9 and applied a weight decay of $10^{-4}$. Cross-entropy loss was used as the minimization objective. We utilized standard ImageNet data augmentations, specifically random resized cropping and horizontal flipping.

#### Scaling Power-Law Curves.

Following previous work on scaling laws Zhai et al. (), Hoffmann et al. (), Besiroglu et al. (), we fit power law functions in the form

$$ $L=E+AX^{-\alpha}$ (1) $$

on the data where $L$ is the misalignment score, and $X$ is an independent variable, such as the number of samples seen ($D$), number of parameters ($N$), and the total training floating point operations (FLOPs) ($C$). Coefficients $E$, $A$, and $\alpha$ are found by minimizing

$$ $\min_{a,e,\alpha}\sum_{i\in[\#\textrm{Runs}]}\textrm{Huber}_{\delta}\left( \textrm{LSE}(a-\alpha\log X_{i},e)-\log L_{i}\right)$ (2) $$

where $E=\exp(e),\>A=\exp(a)$ and LSE is the log-sum-exp operator. We solve Eq. 1 using BFGS minimizer with $\delta=1e-3$, and use a grid of initialiations as follows: $e\in\{-1,-0.5,\ldots,1\}$, $a\in\{0,5,\ldots,25\}$, $\alpha\in\{0,0.5,\ldots,2\}$.

To capture the slow initial increase in benchmark scores of modern architectures like ConvNeXt and ViT models in the low-data regime, we introduce an additional parameter $\lambda$ to Eq. 2. This parameter allows the fitted curve to saturate at lower scales, better reflecting the observed performance of these models under limited data conditions:

$$ $L=E+A\left(X+10^{\lambda}\right)^{-\alpha}$ (3) $$

We minimize the modified equation as before, using $\lambda\in{0,0.5,1.0,1.5,2.0}$. To fit the curve described by Eq. 3, we utilize all data points from the ConvNeXt and ViT models. For fitting the remaining curves, we select ConvNeXt and ViT runs that were trained on datasets with either 300 samples per class or the full dataset. This approach ensures that the fitted curves accurately represent the scaling behavior of these architectures across different data regimes.

Furthermore, we would like to describe the misalignment ($L$) as a function of both the model and data size ($N$, $D$) and predict optimal allocations $N^{*}$ and $D^{*}$ by solving

$$ $(N^{*},\;D^{*})=\operatorname*{arg\,min}_{N,\;D}L(N,\;D)\quad\textrm{s.t}\quad \textrm{FLOPs}(N,\;D)=C$ (4) $$

In that regard, following Hoffmann et al. (), Besiroglu et al. () we fit a parametric function of the form

$$ $\hat{L}(N,D)=E+\frac{A}{N^{\alpha}}+\frac{B}{D^{\beta}}$ (5) $$

where the loss ($\hat{L}$) is a function of parameter count ($N$) and number of samples seen ($D$). In Eq. 5, the first term represents the loss in an ideal data generation scenario (entropy), the second and the third terms reflect the under-performance of a model due to limitations in parameter and data size Hoffmann et al. (), Muennighoff et al. (). Following the example of Hoffmann et al. (), we learn variables $\{E,\;A,\;\alpha,\;B,\;\beta\}$ that characterizes misalignment by solving

$$ $\operatorname*{arg\,min}_{e,\;a,\;\alpha,\;b,\;\beta}\quad\sum_{i\in[\#\textrm {Runs}]}\textrm{Huber}_{\delta}\left(\textrm{LSE}(a-\alpha\log N_{i},b-\beta \log D_{i},e)-\log L_{i}\right)$ (6) $$

with $\delta=10^{-3}$ and $E=\exp(e)$, $A=\exp(a)$ $B=\exp(b)$. Initialiations of $b$ and $\beta$ follow $a$ and $\alpha$, respectively.

Using these learned variables, we predict the optimal allocation of compute ($C$) to $N$ and $D$ with a set equations:

$$ $\begin{split}&N^{*}(C)=G(C/6)^{a},\quad\quad D^{*}(C)=G^{-1}(C/6)^{b}\\ &\textrm{where}\quad\quad a^{\prime}=\frac{\beta}{\alpha+\beta}\>,\quad b^{ \prime}=\frac{\alpha}{\alpha+\beta}\>,\quad G=\left(\frac{\alpha A}{\beta B} \right)^{\frac{1}{\alpha+\beta}}\end{split}$ (7) $$

assuming that compute follows $C(N,D)\approx 6ND$ Kaplan et al. (), Hoffmann et al. () relationship. However, we observe that this does not hold with different architectures, and various CNN families have a slightly different relationship of $C$, $N$, and $D$. As such, we assume a power-law relationship of the form

$$ $C(N,\;D)=m(ND)^{n}$ (8) $$

where we fit $m$ and $n$ via linear regression of $C$ and $ND$ in log-log scale. Then, the updated equations governing the optimal allocation becomes

$$ $N^{*}(C)=G(C/m)^{a^{\prime}/n},\quad\quad D^{*}(C)=G^{-1}(C/m)^{b^{\prime}/n}$ (9) $$

where $a^{\prime}$, $b^{\prime}$, and $G$ are calculated as before.

To evaluate the uncertainty of our model fits, we performed bootstrapping with 1,000 resamples. We compute 95% confidence intervals for each point along the fitted curves based on the variability observed across the bootstrapped estimates.

Finally, to avoid large constants during curve fitting, we rescale the variables $C$, $N$, and $D$ by setting $\tilde{C}=C/10^{13}$, $\tilde{N}=N/10^{5}$, and $\tilde{D}=D/10^{4}$.

## 4 Results

Figure: Figure 2: Scaling model size. a) Neural and behavioral alignments of four architecture families. Models with inductive biases (ResNet, EfficientNet) are more compute-efficient than less constrained models (ConvNeXt, ViT). b) Average alignment per model architecture. All models converge to similar alignments. c) Increasing parameters improves alignment (models trained on full datasets), but the effects saturate.
Refer to caption: x2.png

### 4.1 Scaling drives behavioral alignment, but saturates for neural alignment

Our experiments show a clear and consistent improvement in behavioral alignment as both model size and training dataset size increase. Fig 1.b illustrates this trend across different architectures and scaling axes. The curve $S=1-1.29\tilde{C}^{-0.07}$ converges to perfect alignment score of $1$ in the limit of $C$.

In contrast to behavioral alignment, neural alignment with specific brain regions demonstrated saturation as training compute scaled up in size. The curve represented by the formula $S=0.43-0.55\tilde{C}^{-0.07}$ represents a saturation at $0.43$. The diminishing returns in neural alignment imply that merely scaling up models and data is insufficient to achieve better alignment with higher-level neural representations.

### 4.2 Architectural Inductive Bias Influences Alignment and Scaling Behavior

Experimental results indicate that modern architectures, such as ConvNeXt and Vision Transformers (ViTs), exhibit poorer neural alignment compared to models like ResNets and EfficientNets in low data regime. ResNets and EfficientNets, which have stronger inductive biases due to their fully convolutional structures, demonstrate high neural alignment even at initialization. In Fig. 2, alignment score of ResNets and EfficientNets increase steadily with additional compute in the form of training samples, however ConvNeXt and ViT requires more compute in order to start rising.

This difference in initial alignment also affects how the scaling laws evolve for each architecture. Models with weaker inductive biases require more extensive scaling—specifcally in terms of training data—to achieve levels of neural alignment comparable to those with stronger inductive biases. Consequently, the scaling curves for ConvNeXt and ViT models develop differently, highlighting that architectural choices not only impact baseline alignment but also influence the efficiency of scaling strategies.

### 4.3 More Data Is Better Than More Parameters

Figure: Figure 3: Scaling dataset size. a) Training on larger datasets improves brain alignment. Both ImageNet and Ecoset curves estimates ImageNet21k alignment quite closely, whereas datasets with lower-quality images (Places365 and WebVision) fall below the alignment scaling laws of higher-quality datasets. b) Model families with weaker inductive bias start at a lower alignment and require more data to improve.
Refer to caption: x3.png

Our analysis reveals that increasing the size of the training dataset has a more significant impact on improving brain alignment than simply enlarging the number of model parameters. While both strategies lead to performance enhancements, the benefits from data scaling exhibit less severe diminishing returns compared to model scaling. Specifically, models trained on larger datasets consistently demonstrate superior neural and behavioral alignment with the primate ventral visual stream, following a predictable power-law relationship.

In contrast, expanding the model size without proportionally increasing the training data results in steeper diminishing returns in alignment performance. Larger models rapidly reach a point where additional parameters do not translate into meaningful improvements. Fig. 2c estimates a saturation level of $0.44$ by scaling model sizes with all samples of training data whereas Fig. 3a predicts maximum alignment of $0.48$ and $0.50$ for ImageNet and Ecoset respectively. This indicates that scaling training datasets overall improves brain alignment better than models scaling. Furthermore, Fig. 4b demonstrates that larger models of the same architecture family require much more samples to become s good.

Figure: Figure 4: Optimal compute allocation. a) Alignment as a function of both model and training dataset sizes. Marker size is log-proportional to model size. Compute should be spent XX/YY on model/dataset size respectively. b) Models start out at different alignments but converge to the same saturating point.
Refer to caption: x4.png

To quantitatively capture the joint interaction between data and model scaling, we fitted a parametric curve based on Eq.5, as shown in Fig.4a. This curve effectively models how compute ($C$), dataset size ($D$), and model size ($N$) collectively influence brain alignment. Utilizing the parametric relationships described in Eq. 9, we estimate that additional compute should be allocated following the scaling laws $D\approx C^{0.74}$ and $N\approx C^{0.26}$. These exponents indicate that, for optimal brain alignment, computational resources should be predominantly invested in increasing the dataset size rather than the model size.

### 4.4 Ordered effect of scale on alignment

Our study reveals a graded effect of scaling on alignment across the cortical hierarchy of the primate visual system. Specifically, we observe that the benefits of increased training compute—achieved through larger datasets and more complex models—vary systematically among different brain regions, reflecting their position in the visual processing pathway. Fig. 5.a illustrates the alignment as a function of training compute across various brain regions. We categorized the models into two groups based on their architectural inductive biases. Group 1 includes most models with strong inductive biases, such as ResNets and EfficientNets. These models start with higher neural alignment scores even at initialization due to their fully convolutional architectures. Group 2 consists of models with weaker inductive biases, specifically ConvNeXt and Vision Transformers (ViTs). These models exhibit lower neural alignment in the low-data regime and require more compute to achieve similar alignment levels.

To quantify the impact of scaling on each brain region, we define the alignment gain per region as $A10^{\alpha}$ where $A$ and $\alpha$ are parameters of Eq. 2. Our findings indicate that higher regions in the cortical hierarchy show greater benefits from increased compute. Fig. 5b illustrates the alignment gain per region, highlighting how higher cortical areas benefit more from scaling efforts. This ordered effect suggests that regions higher up in the visual hierarchy, such as the Inferior Temporal (IT) cortex and behavioral outputs, gain more substantially from additional data and increased model complexity. In contrast, early visual areas like V1 and V2 exhibit smaller alignment gains with increased compute, indicating a potential saturation effect.

Figure: Figure 5: Graded effect of scale across cortical hierarchy. a) Alignment as a function of training compute across different brain regions. Group 1 contains most models except those with low inductive bias (Group 2; ConvNeXt, ViT). b) Alignment gain per region, defined as $A{10}^{\alpha}$. Regions higher in the cortical hierarchy show greater benefits from increased compute (Behavior $>$ IT $>$ V4 $>$ V2 $>$ V1).
Refer to caption: x5.png

## 5 Discussion

We establish scaling laws governing the effect of model and dataset scale on behavioral and brain alignment with the primate visual ventral stream.
While scale is a necessary component for all brain-like models, model architectures with priors such as convolutions, and datasets with high-quality images are more sample efficient, leading to alignment with smaller compute requirements.
Scale especially improves alignment with higher-level visual regions, but brain alignment saturates across all conditions tested here whereas behavioral alignment continuously improves with increased scale.

#### Platonic representations.

Our results are consistent with the view that representations in deep neural network models are converging. This observation has been likened to Plato’s concept of an ideal reality (Huh et al., ).
Our findings support this view in that neural network models at scale tend to converge toward similar brain and behavioral alignment, regardless of their initial architectural differences.
However, we find that the representational convergence in current models is *distinct from representations in the brain*’s visual ventral stream. Intriguingly, behavioral choices on the other hand seem to converge to human-like core object recognition, suggesting that there is no unique solution to human-like behavior.
Even models with differing inductive biases, such as ConvNeXt and ViT (which start with poorer neural alignment), eventually converge to the same behavioral, and, saturating brain alignment as models like ResNet and EfficientNet.
Building models with representations that are consistent with the brain might thus require substantial changes to current architectures and training approaches.

#### Dissociation of behavioral and neural alignment.

Our findings reveal a dissociation between behavioral and neural alignment as models are scaled with more parameters and larger datasets. While behavioral alignment continues to improve consistently with increased model parameters and training data – exhibiting a strong power-law relationship – neural alignment reaches a saturation point beyond which additional scaling yields minimal gains. This divergence suggests that behavioral alignment benefits more substantially from scaling efforts, whereas neural alignment may require alternative approaches beyond merely increasing model size and data volume to achieve further improvements.

This disparity is further highlighted by the correlation between task performance and alignment depicted in Figure 6. Behavioral alignment closely tracks validation accuracy, improving hand-in-hand as models become more accurate. Consistent with prior work(Schrimpf et al., , Linsley et al., ), neural alignment eventually saturates, indicating that factors other than task performance influence neural alignment.

#### Limitations.

Our study has several limitations. First, the extrapolation of our scaling functions is constrained by the specific range of model sizes and dataset volumes we examined. While we observed power-law relationships between scaling factors and brain alignment, these functions may not generalize beyond the scales tested.

Second, we evaluated a subset of models focusing primarily on standard convolutional neural networks (e.g., ResNets, EfficientNets) and transformer-based architectures (e.g., ConvNeXts, ViTs). While these architectures cover a range of inductive biases and complexities, they do not encompass the full spectrum of possible neural network designs, such as recurrent networks or more biologically plausible models. We see scaling laws as an opportunity to extrapolate the alignment of models at scale, even if their current training is compute-constrained.

Third, our experiments utilized a subset of training datasets primarily from ImageNet and EcoSet.
Although these datasets are extensive and widely used, they may not capture all the nuances of visual stimuli relevant to the primate ventral visual stream.
Therefore, models trained on other datasets might exhibit improved scaling properties.

Taken together, our results demonstrate that while scaling both model parameters and training data size enhances behavioral alignment with human visual perception, it leads to saturation in neural alignment with the primate ventral visual stream. Data scaling proves more effective than model scaling in improving alignment, emphasizing the critical role of extensive and diverse training datasets. We also find that architectural choices significantly influence alignment efficiency, with models possessing strong inductive biases—such as fully convolutional networks—achieving higher neural alignment even at initialization. Additionally, the impact of scaling varies across different brain regions, benefiting higher cortical areas more than early visual areas. These findings suggest that merely increasing scale is insufficient for modeling the intricate neural representations of the brain’s visual system. Future work should explore new approaches, including alternative architectures and training strategies, to develop models that more accurately reflect the complexities of neural processing in the primate visual cortex.

Figure: Figure 6: Correlation between task performance and alignment. a,b) Correlation between validation accuracy and brain (a) and behavioral (b) alignment. Behavioral alignment strongly correlates with task performance, whereas neural alignment shows a non-linear trend, reaching saturation. c) Pearson’s correlation coefficient per region, with all p-values less than $10^{-40}$
Refer to caption: x6.png

## Appendix A Validation on private data

Figure: Figure 7: Private vs Public benchmarks
Refer to caption: x7.png

As described in Section 3 we test a diverse set of models on private benchmarks on Brain-Score platform. All $R^{2}$ values are above $0.95$ with p-values less than $10^{-18}$.

## Appendix B Pretrained Models

As part of our comprehensive evaluation, we benchmarked a diverse set of pretrained models sourced from both PyTorch’s native library and the timm library. We tested a total of 121, including ViT Dosovitskiy et al. (), DaViT Ding et al. (), LeViT Graham et al. (), ConvNeXt Liu et al. (), MobileViT Mehta & Rastegari (), MaxVit Tu et al. (), FastViT Vasu et al. (), and EfficientViT Cai et al. (). Each model has varying parameter counts, training sample size, training dataset source, and training objective.

Figure: (a) Average alignment
Refer to caption: x8.png

## Appendix C Training Evolution

We observed that the alignment scaling behavior varies significantly across different model architectures, prompting an investigation into their training dynamics to understand the underlying causes. As illustrated in Fig. 9, while the average alignment across various models eventually converges to similar levels with sufficient training, fully convolutional architectures—such as ResNets and EfficientNets—exhibit substantially higher alignment scores at the very beginning of training.

Fig. 10 reveals that this early advantage is primarily due to these models having high neural alignment at initialization, even before any learning from data occurs. This suggests that the architectural features inherent to fully convolutional networks endow them with representations that are already closely aligned with neural data from the primate ventral visual stream.

Further emphasizing this point, Fig. 11 demonstrates that strong inductive biases present in these architectures are key determinants of a model’s initial alignment. These findings underscore that while different architectures may ultimately reach similar levels of alignment after extensive training and scaling, their initial conditions and the efficiency with which they achieve alignment can vary greatly.

Figure: Figure 9: Evolution of brain alignment throughout training
Refer to caption: x10.png

Figure: Figure 10: Evolution of per region alignment throughout training
Refer to caption: x11.png

Figure: Figure 11: Trained vs random models
Refer to caption: x12.png

## Appendix D Effect of Training Objective

We are exploring alternative training strategies that may lead to better brain alignment than supervised training alone. Our ongoing experiments indicate that models trained on the same datasets using unsupervised objectives for extended durations can achieve higher alignment scores. Another promising direction is adversarial fine-tuning; we have found that even a weak form of adversarial fine-tuning using the Fast Gradient Sign Method can enhance neural alignment, especially in lower visual regions. These approaches suggest that incorporating unsupervised learning objectives and adversarial training techniques may be effective in further improving the alignment between artificial neural networks and the primate visual system.

Figure: (a) Comparison of training objectives
Refer to caption: x13.png

Figure: (a) Brain alignment evolution across checkpoints
Refer to caption: x15.png

Figure: (a) SimCLR training of ResNet models
Refer to caption: x17.png

## Appendix E Region-wise scatter plots

Figure: Figure 15: Region vs Region
Refer to caption: x19.png

Figure: Figure 16: Region vs Region - KDE fits
Refer to caption: x20.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 4
questions: 1. On Figure 6b, that's a beautiful correlation. How far can you take it out? Just eyeballing I'd guess it would get near 0.7. Perhaps a pivot for the paper, to get the positive result I think it needs, would be to focus on this scaled-up model of behavior? Just a thought.

2. Why do you think neural scaling laws are different for different brain regions and also for behavior? This is a complex question of course, and I don't expect a definitive answer, but perhaps there's something interesting here.
soundness: 2
strengths: The authors completed an extensive sweep through model architectures, compute, and data budgets, in order to give a detailed view of how model scale relates to neural and behavioral brain scores. The key findings here are important (although with debatable novelty): (1) Neural fits asymptote or worsen with scale, (2) behavioral fits are linear with scale (although scale alone appears to be insufficient), (3) the ceiling and form of scaling laws is different for each visual area region that was investigated. Overall, this is a nice capstone on BrainScore, and perhaps is most notable for showing how methods from AI are not always applicable for explaining brain and behavior.
summary: The authors investigate so-called neural scaling laws for predicting visual behavior and neural activity. "Scaling laws" are empirical trends that show a relationship between model scale (e.g., compute used or amount of data used in training) and its loss on a pretraining task. Here, the authors show different functional forms of scaling laws for predicting neural activity vs. behavior, where the latter is far more promising than the former.

**Update**
I'm on the fence with this paper. I think there's tons of well-done experiments, and I think the message is important to the field of NeuroAI albeit not totally a novel one. I think the line fits are also still problematic and telling a story that's not totally backed up by the data, although I appreciate that the authors are trying to establish a parallel with work in AI on scaling laws. If there were a clear direction forward then this would be a no-brainer accept. As is, I believe it's borderline. I am increasing my score to reflect this.

Also on a separate note, my apologies to the authors for neglecting to respond to all of their points. I was confused by the threading of the responses and mistook the authors' responses to gmHr for responses to my own questions.
weaknesses: 1. The power of scaling laws in domains like language (IMO) is that they imply "all you need is scale." That is, and in the spirit of the bitter lesson, there are no conceptual barriers to achieving a criterion level of performance, only engineering ones. If this were the case in brain science it would be a true world changer. But as this paper (and others which were cited) show, this is not the case. DNNs + scale are not the solution to explaining the variance in brainscore visual system recordings. In that sense I see a large overlap between the findings and result of [1] in which they found a trade-off between ImageNet performance and BrainScore fits. In both cases, these are null results. It is great to show this result, but the lack of a direction forward is concerning.

To drive the point home, in Fig 3, the authors show that training on ImageNet21k (but curiously not WebVision which has more images) leads to better fits. Indeed this would seem to be a scaling law... but the effect size makes it impractical at best: the model maxes out around 0.45 alignment even after all of that data.

For these reasons I struggle to see how this paper makes a strong contribution to the field. It feels better served as a memo or blog post than a conference or journal paper.

2. I think some of the line fits are overly optimistic. For example, in Fig 1, the neuro line is monotonically increasing. But if I squint and just look at the dots, it looks more like a subtle decrease in fits, on average, as a function of compute. This issue is in many of the plots. This relates to my thoughts in (1) about what this all means and whether or not the findings are novel. See fig 2 ViT behavioral line fits for an example where it's not just for neural data. I am marking down the "Soundness" of the paper because of these line fits, but to be honest I don't have any great suggestions about how to improve the fits while maintaining interpretable "laws" when you have what look like non-monotic changes like with the Neural data in Fig 1c.

3. The y limits of the plots should be fixed to one range. It looks like 0-0.7 captures everything. Theres too much bouncing around between different ranges in different subplots. Also could you label what dataset the validation accuracy is derived from on plots where you report it?

[1] Linsley et al. Performance-optimized deep neural networks are evolving into worse models of inferotemporal visual cortex.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: * What are the implications of this work, given the limitations already presented in the paper? 

* What would be the predictions for a model that closely resembles the  visual cortex  such as  CorNET ? 

* Given that the paper focuses on scaling, Have the authors considered how their scaling laws might apply to or change for models pre-trained on much larger datasets like LAION before fine-tuning on ImageNet? This could provide insights into whether the observed plateaus persist across different pre-training regimes
soundness: 3
strengths: The paper is well written. The introduction offers a good view of the literature and it is easy to follow the procedure they use to make the evaluation. The results are clearly presented and explained.  It provides a good overview of the current landscape of models in the context of neural and behavioral alignment.
summary: The paper introduces  a way of calculating scaling laws for neural and behavioral alignment with respect of training data and parameter size of models. It offers an interesting overview of the current status  of models and its performance on these alignment challenges.
weaknesses: My main observation about this work is that, while it provides valuable insights and a well-illustrated overview of the current landscape of models and their alignment with  neural of behavioral benchmarks, it could benefit from more clarity on how these findings might guide future advancements. The paper mentions previous work with similar findings, as noted in the discussion; however, it would be helpful to understand more concretely how this work can serve as a foundation for the next steps in the field and how scaling laws can truly help scientists develop the next generation of more brain-like models. For instance what kind of hypothesis can be drawn from scaling laws that can be tested by adding or removing samples/compute of models being constructed to be more brain-like? 

Although the limitations section mentions that ‘these functions may not generalize beyond the scales tested,’ this suggests a natural boundary for the impact of these results. Could the authors estimate, based on their scaling laws, what order of magnitude increase in dataset or parameter size might be needed to significantly improve neural alignment beyond the observed plateau?

While I understand that this point is mentioned in the limitations section, I feel it is a significant oversight not to include recurrent models. It is encouraging that the paper mentions that inductive bias in the form of convolution seems to yield faster returns, but this feels limited, given that most of the models tested in these benchmarks are much deeper than what might be expected for an architecture resembling the visual cortex. For instance, would be interesting to see how the scaling laws would apply to CorNet? Is it the case that the more brain like the easier it is to scape the scaling laws? that would be very impactful for the community. 


I may have missed it, but did not see mention on self supervised models or robust models and how the scaling laws operate on models trained on these type of frameworks?

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: * Would the authors like to highlight how different training signals would influence alignment to brain / behavior? Humans have a rich multimodal perception of the world, they use depth perception, and predominantly learn without supervision. Are the authors able to tease apart the effects of any such factors in their analyses?
soundness: 3
strengths: * This paper sheds light on the similarity of neural network representations to biological visual representations as a function of model size, compute, and training dataset size. The authors have presented these results in a sound theoretical framework by drawing inspiration from analyses of neural scaling laws. 
* It is super interesting that different areas of the ventral visual stream have varied effects to scaling of neural architectures/datasets. I have not seen this in prior work to the best of my knowledge and this will raise interesting discussions at ICLR.
* I appreciate that the paper is well-written, the figures are legible and accompanied with reasonably detailed captions.
summary: In this paper, the authors study the relationship between the size / compute requirement of popular neural network architectures and their training dataset sizes vs alignment to the biological ventral visual stream. The authors analyze the alignment of various architectures to the primate VVS using the publicly available Brain-Score benchmark and claim that (1) scaling models by increasing parameter count produces diminishing neural alignment beyond a saturation point in model size, but behavioral alignment continues to increase with model size, (2) Alignment scales with training dataset size, (3) Higher visual areas in the cortical hierarchy show stronger gains in alignment with respect to scaling.
weaknesses: * **Lacking evaluation of what model behaviors give rise to alignment.** My main point of feedback to further improve this paper is to address what other factors of artificial neural networks contribute to enhancing similarity to biological vision. It is interesting that there exist scaling laws between model / dataset sizes and neural / behavioral alignment, but this has already been documented in prior studies. I urge the authors to further study the qualitative factors (for e.g. sensitivity to the same spatial frequencies that humans are sensitive to) that give rise to enhanced similarity between ANNs and human vision.
* **Missing evaluation of more recent multimodal models.** There has been a surge in multimodal vision language models that, if evaluated in the same framework established by this paper, would produce really intriguing findings on model scaling and alignment. I encourage the authors to include publicly available large vision language models to increase the impact of their findings, as these VLMs are more widely in use now.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: Could there be additional context on the novelty of this work relative to existing literature on model size effects?

Is it possible to control inductive biases more rigorously, either quantitatively or qualitatively?

In Figure 1, what value does alignment saturation reach?

Is “Utah” in l130 a reference or typo?

Would increasing the benchmark sample size for V1, V2 make the results more robust?

Could the paper benefit from additional discussion on neural versus behavioral alignment, and how better control of inductive biases might enhance interpretability?
soundness: 3
strengths: The core claim—model size influencing alignment—is well supported by the results.

Investigating neural and behavioral alignment is a relevant area with potential applications for improving model interpretability and guiding architecture design.

The study contributes to understanding the role of model scale in alignment, a valuable area for both theoretical insights and practical applications in AI research.
summary: This paper explores how varying model sizes impact neural and behavioral alignment, seeking insights into the relationship between model architecture and its ability to mimic human-like neural responses and behaviors.
weaknesses: Inductive biases might need better control, either quantitatively or qualitatively, to improve result clarity.

Minor issues: typo at l100 (“ecology”), unclear reference in l130 (“Utah”), and Fig 1 could specify the saturation value.

Benchmark sample size for V1 and V2 is relatively small (315), which may impact result generalizability.

Equation 7’s clarity is limited without referencing equations 8 and 9; introducing C(N, D) = 6ND earlier could help.

Meta-review: The paper dives deep into the alignment of neural network models and neural response patterns in the visual ventral system. After checking myself, the quality of the paper and the visuals is top, and the findings are indeed intriguing and thought provoking, showing a great deal of work and craftmanship. On the other hand, the majority of reviewers point out that this is definitely positive, but the paper is missing actionable insights that set it apart from other papers, which have already published on this direction. This, with the fact that there were 7 (!!) updates on the manuscript point to having this paper revised and resubmitted so that the paper shines.

Decision: Reject