Prune and distill: similar reformatting of image information along rat visual cortex and deep neural networks
Authors: Paolo Muratore
Sina Tafazoli
Eugenio Piasini
Alessandro Laio
Davide Zoccolan
Venue: NeurIPS 2022 Accept (2022)
Keywords: convolutional neural networks
computational neuroscience
rat
visual cortex
ventral stream
intrinsic dimensionality
vision
representation analysis

Abstract: Visual object recognition has been extensively studied in both neuroscience and computer vision. Recently, the most popular class of artificial systems for this task, deep convolutional neural networks (CNNs), has been shown to provide excellent models for its functional analogue in the brain, the ventral stream in visual cortex. This has prompted questions on what, if any, are the common principles underlying the reformatting of visual information as it flows through a CNN or the ventral stream. Here we consider some prominent statistical patterns that are known to exist in the internal representations of either CNNs or the visual cortex and look for them in the other system. We show that intrinsic dimensionality (ID) of object representations along the rat homologue of the ventral stream presents two distinct expansion-contraction phases, as previously shown for CNNs. Conversely, in CNNs, we show that training results in both distillation and active pruning (mirroring the increase in ID) of low- to middle-level image information in single units, as representations gain the ability to support invariant discrimination, in agreement with previous observations in rat visual cortex. Taken together, our findings suggest that CNNs and visual cortex share a similarly tight relationship between dimensionality expansion/reduction of object representations and reformatting of image information.

Prune and distill: similar reformatting of image information along rat visual cortex and deep neural networks
Paolo Muratore', Sina Tafazoli"," Eugenio Piasini', Alessandro Laio Davide Zoccolan 8.1
International School for Advanced Studies (SISSA), Trieste, Italy
Princetonn Neuroscience Institute, Princeton University, Princeton, NJ, United States of America Abbus Salam International Centre for Theoretical Physics (ICTP), Trieste, Italy
To whom correspondence should be addressed: zoccolan@sissa..it
Abstract
Visual object recognition has been extensively studied in both neuroscience and computer vision. Recently, the most popular class of artificial systems for this task, deep convolutional neural networks (CNNs), has been shown to provide excellent models for its functional analogue in the brain, the ventral stream in visual cortex. This has prompted questions on what, if any, are the common principles underlying the reformatting of visual information as it flows through a CNN or the ventral stream. Here we consider some prominent statistical patterns that are known to exist in the internal representations of either CNNs the visual cortex and look for them in the other system. We show that intrinsic dimensionality (ID) of object representations along the rat homologue of the ventral stream presents two distinct expansion-contraction phases, as previously shown for CNNs. Conversely, in CNNs, show that training results in both distillation and active pruning (mirroring the increase in ID) of low- to middle- image information single units, as representations gain the ability to support invariant discrimination, in agreement with previous observations in rat visual cortex. Taken together, our findings suggest that CNNs and visual cortex share a similarly tight relationship between dimensionality expansion/reduccion of object representations and reformatting of image information.
Introduction
Deep Convolutional Neural Networks (CNNs) currently stand as our best class of models of visual processing in the brain I showing success in: (1) predicting the tuning of individual neurons at various stages of the ventral stream; (2) accounting for the ability of
4 and bold responses ventral stream neurons to encode a variety of object properties 6 and (3) controlling their activity via This suggests that the objective-optimizzion
synthetic stimuli inferred through model inversion framework of deep learning offers a parsimonious explanation of the inner workings complex, hierarchical brain circuits (9) although the latter are likely shaped by very different learning processes (e.g., unsupervised adaptation to the spatiotemporal statistics of the visual input [10] TT]. Despite this success, key differences between biological and artificial hierarchical networks exist (e.g., in sensitivity to noise or adversarial examples 12 13 ). possibly highlighting core dissimilarities in how information is processed in the systems.
In this study, we investigated whether a similar reformatting of image information takes place along rat visual cortex and representative CNN (VGG-16). We started from the observation that the
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

A
intrinsic dimension
160
AlexNet
C
mutual information (bits)
object identity
decoding accuracy (%)
120
VGG-16
0.003
62
60
80
0.002
56
40
0.001
52
0.000
50
input
output
V1
LM
LL
relative depth
B
mutual information (bits)
0.10
D
fraction of sel selective
0.4
orientation
11.0
fraction of selective neurons (monkey)
corner
0.08
neurons (rat)
0.3
0.06
0.2
0.5
0.04
luminosity contrast
0.1
0.02
0.00
0.0
0.0
V1
LM
LI
LL
V1
LI/V4
LL/IT
visual area
visual
Figure 1: Summary of previous results in CNNs and visual cortex (A) Intrinsic Dimension (ID) of object representations in two deep CNNs, plotted function of relative depth. The error bars are standard deviations of the ID across repeated estimates (reproduced from [14)). (B) Median
information (+ SE) conveyed by single neurons recorded along the rat ventral stream (i.e., areas VI,
LM, LI and LL) about luminosity and contrast of visual objects (reproduced from [5] (C) Red curve: median information (# SE) conveyed by single neurons in the four areas about the identity of visual objects, each presented across variety of distinct views. Blue curve: performance of binary linear classifiers that were trained to discriminate two visual objects (each presented across a set of different views), based on the responses evoked by the objects in the neuronal population recorded in a given area (performance was measured over a set of validation views). Error bars are SE computed over different object pairs. Both panels are reproduced from (15). (D) Fraction of orientation tuned neurons (solid lines) and corner tuned neurons (dashed lines) in three different areas of the rat (red:
V1, LI and LL) and monkey (blue: VI, V4 and IT) ventral stream (reproduced from (16) and
Intrinsic Dimension (ID) of object representations follows a characteristic hunchback profile across the layers of deep networks, with an initial expansion phase, followed by a strong contraction [14] (see examples in Figure [A]. This trend is so systematic across network architectures to raise the question of what information processing goals the two phases underlie. A possibility is that the initial expansion reflects the need of removing gradients of task-irrelevant, low-level features (c.g.. luminosity and contrast) that are present in the visual input, while the decrease underlies a gradual reformatting of the data in pursuit of representation better suited for the classification task [14 [8] This idea is reminiscent of the pruning of luminosity and contrast information, and the increase of invariance of object representations that takes place along the rat homologue of the primate ventral visual pathway [15] (Figures MB and [T). This parallel suggests an intriguing similarity the level of core lata-reformatting processes between artificial and natural visual hierarchies. However, it unclear if the dimensionality of object representations follows a hunchback trend in rat visual cortex, and, conversely, if luminosity and contrast information is actively discarded across CNN layers. Moreover, other (middlc-level) tuning propertics exist that follow charactcristic trends of variation along both the monkey and rat ventral streams [66]. These are: (I) the fraction of neurons tuned for orientation, which decreases from primary visual cortex (V1) to higher-order areas (Figure ID solid lines); and (2) the fraction of neurons tuned for multiple orientations (Figure DP, dashed limes), a property thought to reflect the ability to encode corners [17.. which instead increases from VI to downstream areas. A few studies have reported similar trends in deep networks when probed with oriented gratings 6 16 19 although we still lack general assessment of how the encoding of orientation and corner information found in natural images evolve across the layers of CNNs.
The goal of our study is to understand which these image reformatting trends found either CNNs or visual cortex also a signature of information processing in the other system. Specifically,

---

first analysed neuronal recordings from [5] to measure the ID of object representations across the rat ventral stream, finding also in the rat the two distinct expansion-contee contraction phases first described in [4] for CNNs. We then measured the information encoded by single units in VGG-16 about variety of visual properties of increasing complexity, finding that training the network both actively distills and prunes low- to middle-level image information, in agreement with biological observations. Finally, we tracked the evolution of information about object identity at both the single-unit and population level across VGG-16 layers, exposing how such high-level information emerges sharply in late layers, again in agreement with biological findings and previous analyses CNNs [18]
2 Methods
2.1 Analysis of neural data
To measure how the intrinsic dimension of object representations evolves across a visual cortical processing hierarchy we analyzed the dataset recorded by [15]. These data consist of extracellular neuronal responses sampled from four visual cortical areas (228 units from V1, 131 from LM, 260 from LI, and 152 from LL), while anesthetized rats were presented with battery of 380 stimulus conditions - i.e.. 38 different views (obtained by scaling, translation, rotation, etc.) of 10 visual objects. As summarized in Figure IB-D, object representations along this pathway were found to encode stimulus information in way that is consistent with the existence of functional object processing hierarchy. our study, we computed the response of each recorded unit to every stimulus as the average number of spikes fired by the neuron across repeated presentation of the stimulus within a neuron- specific spike-count window, as defined in [15.. Each stimulus condition could thus be represented by a neuronal population vector, whose components were the responses of the neurons recorded a given area to that stimulus. The cloud of population vectors associated to the whole set of 380 object conditions formed a data manifold, whose intrinsic dimension was measured using the nonlinear estimator defined in 20 and previously applied to the analysis of CNNs in 14]
2.2 Dataset, network architecture and estimation of the mutual information between unit activation and image properties
We studied the behavior of the PyTorch implementation of VGG-16 21 either randomly initialized or pre-trained on the full ImageNet dataset, with the goal of understanding how different image properties were encoded by individual units of the network as the result of training. We selected random sub-population of 250 units from each convolutional layer (before the ReLU activation) and from the final fully-commn cullecconeetted layers, and recorded their activations when exposed to 1500 input images taken from the 12S22 ImageNet validation dataset. Inspired by the approach applied by 15) to study rat visual cortex (see Figure B-D), we computed Shannon mutual information 7 (X;, Y?) between the activation Y? of the i-th unit in layer l (referred to as of in what follows) and a given image feature X (e.g., luminosity or contrast). The network architecture imposes for each unit uf a receptive field (RF), namely sub-patch (denoted as ImageRF) of the whole image that the unit processes. As detailed in the next sections, the feature metric X is computed by applying specific function feat that maps ImageRF to a real number (e.g., the luminosity intensity in the image patch) or a combination of real numbers (e.g., the two main orientations in the image patch), i.e., feat: : ImageRF - IR or feat: : ImageRF + R.2.
Given unit uf. the values taken by its activation Y over the set of input images yield a unit-specific activation distribution (y) (for simplicity, we dropped the unit and layer indexes). Similarly, the values taken by the feature metric X! yield unit-specific (i.e., RF-specific) distribution px (a) (e.g., of luminosity intensity levels). In our experiments, both distributions were discretized into 20 equi-spaced bins. By computing the joint distribution of activation and feature values px,y (s, y). we estimated, for each unit, the mutual information between activation and feature metric:
Px. (z, v) px (x) PY (v)
Σ Px, (x,1) log ZER
To allow for a better comparison among the various layers and the different image features used in our analysis, the mutual information was normalized by the entropy of the distribution of the feature metric
3

---

H (x))=-EzeN Px (I) logpx (z). The final estimate of the information conveyed by the units of given layer about the feature metric was computed as U (XlY) E,, [If (X); / (x;)]. where E, is the expected value over all units i of layer l. Importantly, although such unit-averaging was performed on sub-population of O (102) units, the variability of U across independent experiment realizations (different units and stimuli) was very low, as shown by the error bars reported in Figures and 5 The limited sampling bias for the mutual information was corrected with the the Panzeri- Treves method [22 23). Finally, we stress here how US, being a single-unit information estimate, is not bound by the data processing inequality and can in general express non-monotonic behaviours as a function of layer depth f.
2.3 Definition of the metries to quantify visual features
In our analysis, each image patch Imagenr falling within the RF of a unit was quantified by an array of four different visual properties of increasing complexity: 1) luminosity; 2) contrast; 3) orientation of the dominant edge (if any); and 4) orientations of the two dominant edges (if any), which is a proxy for the orientation and width of the dominant corner. Therefore feat E {luminosity, contrast, orientation, corner).
Luminosity can be easily defined as the average pixel-intensity in the image path: luminosity = mean (Imagegr). Contrast quantifies the amount of luminosity variation in the patch and was computed as: contrast = mean (Sobel s Imagege). where Sobel denotes the Sobel kernel and is the convolution operator (the Sobel transform standard approach to compute image gradients
24
The dominant orientation of in image patch is less straightforward to quantify, because of the large variation in RF size across the layers of the network and the complexity of the natural scenes in ImageNet. At very low resolution, such for individual units in early layers in VGG-16 (which have 3 x 3 RF size), no meaningful orientation can be computed. For units in late layers, which process the entire scene, multiple prominent orientations might coexist or not exist at all. More generally, image patches span a spectrum of scene orientation strength, ranging from those containing one or more sharp edges those featuring none. To deal with such variability, we developed a two-stage, compute-and-filet approach. The orientation estimation routine is based on Fourier Analysis and defines the dominant orientation of the patch 6* as the angle of highest power of its Fourier spectrum (see Algorithm I in the Supplementary Material for detailed pseudo-code of the pipeline). In addition, the function provides an orientation strength index { E [0, 1]. which peaks for images containing at least one very sharp edge. Before measuring orientation information, we ranked the pool of sampled units in each layer by computing. for each unit, the average of the orientation strength index & across the full set of 1500 input images. Out of the initial population of 250 units we only retained the 200 units with the largest average index. In addition, for each selected unit, we only considered the 500 images with the largest index
The corner feature was quantified as the pair of orientations of the two most prominent edges in the image patch. Specifically. the corner estimation routine applies Fourier analysis and lenn-iinding algorithms to identify the two dominant orientations 0 and 85 in patch, along with a corner strength index Ç E [0, 1] 17 [6]. which is large when at least two orientations with similar power are detected the Fourier spectrum, while it becomes negligible both for no-peak and single-peaked angular spectra (see Algorithm 2 in the Supplementary Materials for a pseudo-code description of the complete pipeline). We used this index following the same rationale as for the & index of orientation strength, this time ranking units and input images based on their corner strength Ç and retaining a population 200 units, each tested with a sample of 400 images.
3 Results
3.1 Intrinsic dimension of object representations along the rat ventral stream
We applied the nonlinear ID estimator Two-NN 220 to compute the intrinsic dimension of object representations in four visual cortical areas (VI, LM, LI and LL) of the rat ventral stream, as a function of the number of units included in the population vector space (Figure 2 solid lines), up to the maximal number of units available in each area (circles). In addition, we extracted the asymptotic values of the ID (stars in Figure 23) via power-law fits (dashed lines) to control for

---

A
25
B
35
full data
intrinsic dimension
20
intrinsic dimension
30
asympt.
value
15.
25
V1
20
10
LM
15.
5
LL
0
750 number of neurons
250 500
1000
10
V1
LM
LI visual area
LL
Figure 2: Intrinsic Dimension of neural representations (A) Estimation of the ID as a function of the number of neurons considered in the four visual areas (solid lines). Shadings correspond the SD of multiple estimates with randomly sampled neuronal sub-populations, while circles mark the estimates obtained with the full populations in each area. Dashed lines are power-law fits to the data, (B) The ID estimates obtained for the full populations (circles) and for the asymptotic values of the fits (stars) are plotted as a function of the rank of the areas along the rat ventral stream. Error bars are the standards deviation of the values returned by the fit.
finite-size effects. At any population size considered, the ranking of the visual areas in terms of the estimated ID was remarkably stable, with VI featuring the lowest ID, LI the highest, and LM and LL reaching intermediate values. More importantly, plotting the ID in each area as function of its rank along the cortical processing hierarchy (Figure 2 revealed characteristic "hunchback" profile, with initial expansion (from V1 to LI). followed by contraction (from LI to LL). This trend is consistent with the one observed in deep networks (see Figure [T] by ([4), who conjectured that the initial ID expansion was due to the pruning of low-level image information (e.g.. luminosity and contrast). Our result strongly supports this intuition, since the alternation of the expansion-contrattio phases is observed along an object processing pathway where such pruning has been shown to take place (see Figure 3 [5]
3.2 Encoding of low- to middle-level visual features in single units of VGG-16
We now turn to the other question addressed in our study, namely investigating if the information about low-level image features actively discarded in artificial networks a manner that resembles the one observed in rats. Having defined set of metrics to quantify image features of increasing complexity (see Section 2.3). we measured how much information about these features was encoded by the activation of individual units across the layers of VGG-16 (see Section 2.2 for details).
We found that information about luminosity was monotonic decreasing function of the layer's depth, with training producing very large luminosity information loss in the very first layer (compare blue and green curves in Figure BA). Intuitively, this can be explained by the fact that learning spatially structured convolutional kernels will tend to produce both positive and negative weights with balanced, near-zero average, which are poorly sensitive to the mean luminosity within unit's s RF. By contrast, randomly assigned weights will often have the same sign, at least for the small kernels of the early layers, yielding activations that are proportional to the luminous energy falling within a unit's 's' RF. This intuition was confirmed by comparing the distributions of the average weights for the 3 x 3 kernels of the first layer in the trained and untrained network (bar plot in the inset). The gradual monotonic decrease that nevertheless observed in the untrained network is explained by the fact that randomly assigned weights, in case of increasingly larger kernels, will progressively tend to the zero-average condition (inset, red line).
training produces spatially structured kernels, units in early layers should not only lose sensitivity to luminosity, but also become sensitive to image contrast. Our mutual information analysis confirmed this intuition, showing that the units of the initial layers encoded a larger amount of contrast infor- mation in the trained network, as compared to the untrained one (Figure B, blue vs. green curve). In addition, as result of training, contrast information grew steadily the early convolutional layers, reaching peak in the third one, but then decayed sharply in the following layers, eventually
5

---

A
luminosity
B
contrast
norm. mutual Information
0.3
norm. mutual information
0.06
convolution
HH
linear
0.2
0.04
P n mm
0.03 0.00 0.03
mean weights
0.1
0.02
0.0
0.00
1
13 15
1
5
7
9
11
13
15
C
norm. mutual information
x10-2
orientation
norn. mutual information
3.4
x10-2
corner
trained random
3.3
4.5
3.1
3.0
o
ooo
2.9
2.8
3.5
2.7
7 13
15
13 15
layers
layers
Figure 3: Distillation and pruning of image information in VGG-16 (A) Mean normalized infor- mation conveyed by VGG-16 units about image luminosity for trained (blue) and random (green) network. Error bars are standard deviations over five realizations of the experiment (independent sampling of units, images and random weights). Circles represent convolutional layers, while stars indicate fully connected layers. Inset: distribution of the average weights of the units in the first layer of the trained and random network (blue and green bars) and in the forth convolutional layer of the latter. (B-D) Same as in A, but for the information conveyed by VGG-16 units about image contrast, orientation and corners (i.e., joint orientation of two prominent edges).
attaining values that were lower than those the untrained network. This suggests that learning representations that useful to process and classify natural images requires to first distill contrast information in the units of early layers, followed by actively discarding such information in later processing stages. The pruning of contrast and luminosity information matches the results in visual cortex 15 (see Figure BB). We note that the analysis of rat data did not reveal the initial rise of contrast information found in VGG-16, but this unsurprising, given that the rat dataset did not contain recordings from the processing stages that precede V1, i.c., retina and thalamus, where center-surround contrast detectors first emerge in the visual system and that would correspond to VGG-16 very first layers.
We next considered visual features of increasing complexity, measuring the amount of information encoded by VGG-16 units about the dominant orientations of the image patches falling within their RFs. This analysis was applied only units for which enough input images existed that contained, in the patch falling within the units' RFs, a sufficiently prominent oriented edge (see Section 2.3). Moreover, we excluded from the analysis the first two layers of the network, because their units have RFs that are too small for the orientation estimate to be meaningful. When visualized as a function of layer depth, orientation information in the trained network followed hunchback profile (Figure BC, blue curve), raising sharply and reaching a peak in the fifth convolutional layer, i.c., at later stage than contrast information (see Figure 3B), consistently with the hierarchically higher nature of the orientation feature. Following the peak, orientation information dropped sharply in the deeper layers. As for luminosity and contrast, this trend was the result of training, it was not observed in the randomly initialized network (green curve). And again, as for contrast, orientation information, once distilled in individual units of early layers, was then actively discarded in finding consistent with the loss of orientation tuning found along the ventral stream [16) (see Figure DD, solid lines). Just like for contrast, no initial rise of orientation tuning observed along the rat ventral stream,
following processing stages, becoming lower than for the untrained network

---

because no data were available from subcortical areas where orientation tuning is known to be much less prominent (25).
Finally, considering features of even greater complexity, we measured the information conveyed by VGG-16 units about the joint orientation of two dominant edges, i.e., the corner information (again, this analysis was applied only to cases where the image patch falling within a unit RF contained a sufficiently prominent corner; see Section 2.3). As for orientation, also corner information varied across the layers of the trained network according to a hunchback profile (Figure BD, blue curve), again peaking in the fifth convolutional layer, and again being discarded in deeper processing stages, but more gradually than orientation information, reaching sort of plateau in middle layers. Once more, this trend was not observed in the untrained network (green curve) and was instead consistent with the increase of neurons tuned for pairs of orientations found along the ventral stream [16] (see Figure I dashed lines).
Importantly, all these feature information trends were largely preserved when assessed on the activations following the ReLU non-linearities (see Appendix C) and when tested on other networks of the VGG family (see Appendix D
3.3 Effective pruning of low-level information requires training
One of the most intriguing findings of our experiments is that training is necessary not only to build sensitivity for low- and middle-level visual features, but also plays the complementary role of pruming this information, once it has been distilled in individual units of early layers. To better understand the extent to which information pruning is actively enforced by training, we considered a hybrid VGG-16 network constructed as follows: layers S l" shared the same weights the fully-trained (on ImageNet) VGG-16, while weights in layers E >> E" were left randomly initialized. By letting e vary, one could visualize the effect of random transformations after given checkpoint (e*) and ask whether the observed decay of feature information (Figure 5 is a direct consequence of training (active information pruning) or is merely the result of architectural constraints. We found that training played an active role in pruning luminosity information (Figure A with the information profile of the fully-trainee network (blue curve) being consistently lower with respect to the profiles obtained for hybrid networks with intermediate !" checkpoints (green curves). The effect of training was even more striking for contrast information (Figure BB), which, in the hybrid networks, displayed a growing trend through the random convolutional layers, before finally dropping in the second fully-connected layer. In the case of orientation (Figure results were only partially consistent with those of luminosity and contrast. Training the network only up to layer 5 (i.e., up to the peak of orientation information), still yielded a large drop of information from layer onward when these layers are left randomly initialized. However, orientation information did not reach the same low values attained by the fully-trained network in the last layers: information here remained substantially higher. This indicates that a reduction of orientation information after the peak in layer 5 is achieved
A
B
D
luminosity
contrast
orientation
corner
nor.. mutuea Information
0.09
U.P
0.050
0.33
0.07
0.07
0.32-
0.045
0.05
0.3:-
0.03
0,040
0.30-
0.09
0.01
0.035
layer
layer
layer
layer
Figure 4: Training results in active pruning of low-level image information (A) Mean normalized
information conveyed by VGG-16 units about image luminosity for trained network (thick blue line; same curve as in Figure and for three additional hybrid network configurations (green lines) that have been trained only until layer Ex (with weights in the following layers having been left randomly initialized). The gradients of color (from green to blue) correspond to progressively larger ex values, i.e., E {1, 2, 33. (B) Same as in A, but for the information conveyed by VGG-16 units about image contrast. Here the thick blue line is the same curve as in Figure 3 and e E {3. 5, 7, 9, 113. (C-D) Same as in A, but for the information conveyed by VGG-16 about image orientation and corner.
7

---

A
B 1.0
norm. mutual information
0.04
trained
random
decoding accuracy
0.8
0.03
convolution
trainset
fully-connected
0.6
validation
0.02
0.4
0.2
chance baseline
0.1
1
5
9 layers
11
13
15
3
5
7
9 layers
11
13
15
Figure 5: Evolution of category information across VGG-16 layers (A) Mean normalized informa- tion conveyed by 16 units about image category for trained (blue line) and a random (green line) VGG-16 network. As in Figure B error bars are standard deviations over five realizations of the experiment. (B) Training and validation performance (dashed and solid lines respectively) of linear SVM classifiers that were trained to predict the labels of images belonging to 10 selected Imagenet categories (250 and 50 images per category were used, respectively, for training and validation), based on the activations of a pool of 250 VGG-16 units sampled from each layer. Data points are averages (+ SD) over 200 sub-populations of 100 units that were randomly sampled from each pool of 250 units.
even without training, likely because of architectural constraints (e.g., increase in receptive fields). However, further pruning of orientation information in the last layers of the network still requires training, consistently with the results found for luminosity and contrast information. qualitatively similar behavior, albeit noisier, was found for corner information (Figure 4D).
3.4 Information on object identity emerges in late layers at both the single unit and population level
The VGG-16 network used in our experiments was pretrained to achieve high classification per- formance on Imagenet. Thus, all the information about the low- to middle-level visual features explored in our analyses must have been harvested (and then pruned) by the network in the attempt to maximize the separability of image categories in output layer. Having reported how information about several such features peaked in early convolutional layers, we next asked how information about image category evolved across the network. Intuition suggests that should peak in the very last layer, where readout takes place. It is however unclear how such information varies along the network depth, especially the one encoded by individual units. In the rat ventral stream, information about object category encoded by single neurons has been found to rise from low to high visual areas, with matching increase in the ability of neuronal populations to support invariant recognition [5] (see Fig- ure (C). In (18), using neighbourhood regularity metric, it was shown how representation-suppee for image category emerged sharply in late layers of various CNN architectures.
Here, we measured the category information encoded by VGG- units by using the label of the 1500 test images as the feature variable X in Eq. ( We found that this metric remained low and stable for about half the depth of the trained network, increasing smoothly in the last convolutional layers and then abruptly in the fully connected ones (Figure 5A, blue curve), while no trend was observed for the random network (green curve). This result resonates with that in and sharp rise in image category information.
[8] again indicating late
Next, we investigated how easily accessible was such category information encoded by single units. To this aim, we trained linear SVMs predict the image labels based on the activity of pool of 250 units in layer (Supplementary Material, section B). We found growth of decoding accuracy (Figure 3, blue curve) that tracked the increase category information observed, at the single unit level, in the second half of the convolutional layers (compare to Figure 5A). This suggests that, similarly to what observed along the rat ventral stream, the concentration category information in single units plays role in supporting the linear readout of category label at the population level.

---

As expected, the decoding accuracy matched the expected close-t--perfec performance in the final, fully connected layers (see Figure B. blue lines). Interestingly, even the random network supported above-chance decoding beyond the training domain (solid green curve), with both the trained and random configurations remaining closely tied for half of the computation depth (mirroring, again, what observed for the category information at the single unit level).
Conclusions and Discussion
The ventral visual stream 26 and deep convolutional neural networks (27] are very effective solutions to the problem of object vision. The precise extent to which these two classes of systems process visual information based on similar principles is an open question and an active field of investigation
2 3. Here, compared the rat ventral stream and widely used CNN (VGG-16) 16) by tracking how image representations are progressively reformatted across the two hierarchies. Differently from previous studies, our goal was not to use CNNS models to predict the tuning of visual cortical S 28 29 5.. Rather, we took inspiration from recent studies showing that some key statistical properties of image representations follow very specific trends of variation across either the artificial or the cortical processing hierarchies [5
neurons or the spatial structure of their RFs A 7
and we looked for them in the other system.
In this study, we focused on the rat brain as the biological term of our comparison with CNNs, rather than on the primate brain, which certainly possesses more advanced visual system. This is because, despite the long tradition of vision neuroscience in primates, most of the information processing trends we explored in this work have been systematically measured only in the rat [15]. This is related to the fact that, in monkeys, it is difficult to record from the whole ventral stream using the same battery of visual stimuli. Typically, no more than pair of cortical areas are investigated in a single study (e.g., V4 and IT (30)) and object representations in VI are often simulated rather than measured
By contrast, in rats it is possible to probe with the same stimuli V1 and the whole progression of lateral extrastriate areas (LM, LI and LL) that play the functional role of an object-procesiing This makes it possible to analyze cortical hierarchy that is deep enough for meaningful comparison with a CNN. It is worth stressing that we specifically focused on the rat rather than on the mouse visual system, because evidence for the existence of an soject-processing pathway is way more limited in mice (but see 33 and, not surprisingly, deep CNNs have been found to be
pathway 31 15| 52
poor models of mouse visual cortex
29
Our analyses yielded three main results. First, the ID of object representations across the rat ventral stream varied according to the same hunchback profile previously found by [14] in CNNs (compare Figure _ to Figure [ Here it should be noted that, compared to the dramatic ID contraction observed in the final layers of CNNs, the drop found from LI to LL was much smaller. This is not surprising, because LL., despite its high rank along the rat visual cortical progression, is not the final stage of the hierarchy. In fact, the rat ventral stream possesses least one higher-order area (TO, [3] 34]; not probed in (15).. Additionally, the deepest layers of CNNs contain representations that are highly specialized for the classification tasks they were trained on, suggesting a better match with cortical regions involved in memory and decision making (such as perirhinal, posterior parietal and prefrontal cortex) than with purely sensory areas. in these regions that representations may be expected to become as low dimensional as those found in CNNs' final layers 35 36
Our second main result is that information about low-level features (luminosity and contrast) encoded by individual units of 16G- was progressively pruned across the processing hierarchy, as previously found by [5 across rat visual cortical areas (compare Figure A Figure BB). A similar pruning
was observed also in the case of orientation and corner information (Figure 5 D), with the difference
that sensitivity to these higher-order features started low and had first to be progressively distilled through processing along the first layers (a trend that, although less prominent, was also observed for contrast information). These trends are consistent with the drop of orientation tuning and the increase of tuning for multiple orientations (corners) found in both the rat and monkey ventral streams [66 (Figure ID). Again, as for the case of the ID trend discussed above, this consistency between cortex and VGG holds in terms of global trends and not at the level of a one-to-ons area-to-layer match. In fact, while for the CNN we have access to the entire hierarchy, for the rat visual system we only have access to a subset of processing stages. In particular, the neuronal data set did not include data from both deep memoryldecision areas and early processing stages (retina and thalamus). In CNNs, this would be equivalent to missing the first few convolutional layers, as well some of
9

---

the deepest convolutional layers and the fully connected ones (i.e., rat visual areas VI, LM, LI and LL could roughly be equated to layers 5-8 in VGG-16). A tighter match between rat visual areas and CNN layers may become possible using more advanced neuronal recording technologies that allow targeting a larger numbers of visual processing stages (e.g., using Neuropixel probes (37). Moreover, it would be interesting to extend our approach to fMRI data spanning the entire human ventral pathway [5] and estimate how much information is conveyed about the various low-to-middle level features tested in our study by the activity of individual voxels.
The feature information trends reported in our work are also consistent with the orientation tuning profiles reported across CNN layers by a other studies 6 16 19] and with the way the spatial structure of convolutional kernels evolves across CNN layers, where early Gabor-like kernels are replaced by filters with more complex geometries in late layers [38 []. Importantly, our analyses clearly show that information pruning not a trivial byproduct of architectural constraints (e.g., RFs becoming larger as a function of layer depth), but is an active process that takes place, across the whole network, as the result of training (Figure 3 and D. This suggests that, in hierarchical visual processing systems (biological and artificial alike), sensitivity to features that are required for the buildup of higher abstractions (e.g., contrast for edges; edges for corners; corners for shapes; etc.) might become useless or even harmful for further learning in deep layers. Thus, our work extends the
findings of previous modeling studies of visual cortex using CNNs 4
2
7
g 28 29 5 by explicitly
measuring the existence of non-mone monoooooonic trends of feature information across CNN layers and by
establishing their dependence on training.
Finally, our experiments revealed that the growth of classification accuracy afforded by image representations across V6G-16 layers closely tracked the increase of category information encoded by individual units (Figure 5 This result is consistent with the tight relationship found, in the rat ventral stream, between the view-invariant object information encoded by single neurons and the power of neuronal populations to support invariant object recognition (see Figure IC). Overall, this suggests that low/middle-lewe image information and higher-order categorical information trade off along visual processing hierarchies, echoing previous observations that have emphasized the role of learning in suppressing irrelevant information
33
Taken together, these findings point to the existence of functional relationship between dimensional- ity expansion/reducciinn of object representations and distillation/pruning of various kinds of image information, suggesting that such relationship is likely a fundamental property of both biological and artificial visual processing architectures. Further experiments are necessary to probe the generality of this conclusion across natural visual systems (i.e., recording from more visual areas and different species) and across a larger variety of artificial neural networks. In our study, we focused on VGG because it is one of the most popular CNN architectures in visual neuroscience, commonly used either as a model of the ventral stream or as a benchmark against which such models should be tested (sec c.g.
45) ). More importantly. VGG, as any other simple feedforward convolutional network, allows for a very natural definition of the receptive field of individual units. This is fundamental for our analysis, because a notion of receptive field is required for the estimation of image feature information. By contrast, more modern architectures, such as ResNets or ViT's, allow for non-trivial paths of information flow (residual connections, skip connections, the attention mechanisms), which make the identification of receptive fields more challenging. Exploiting gradient backpropagation the input level other reature-visualization techniques (e.g. gradCAM (46)) may be a viable approach to overcome this issue, thus allowing extending our analysis beyond the purely feedforward convolutional framework.
Acknowledgments and Disclosure of Funding
We thank A. Ansuini for his help on getting started with the computation of intrinsic dimension in deep nets and neuronal data, We thank D. Doimo and L. Porta for suggestions on the implementations of our analyses. We thank A. Benucci for his feedback on the interpretation of our findings.
This work was supported by European Research Council Consolidator Grant (project no. 616803- LEARN2SEE to D.Z). We acknowledge the HPC Collaboration Agreement between SISSA and CINECA for granting access to the Marconi cluster.
10

---

References
[1] Nikolaus Kriegeskorte. "Deep neural networks: a new framework for modeling biological vision and brain information processing". In: Annual review of vision science (2015), pp. 417- 446.
[2] Daniel LK Yamins and James J DiCarlo. "Using goal-driver deep learning models to under- stand sensory cortex". In: Nature neuroscience 19.3 (2016), pp. 356-365.
[3] Grace Lindsay. "Convolutional neural networks as a model of the visual system: Past, present, and future". In: Journal of cognitive neuroscience 33.10 (2021), pp. 2017-2031. [4] Daniel LK Yamins al. PPerformance-optimizee hierarchical models predict neural responses in higher visual cortex". In: Proceedings of the national academy of sciences 111.23 2. (2014), pp.
[5] Umut Güçlü and Marcel AJ van Gerven. "Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream". In: Journal of Neuroscience 35.27 (2015), pp. 10005-10014.
[6] Ha Hong et al. "Explicit information for category- object properties increases along the ventral stream". In: Nature neuroscience 19.4 (2016). pp- 613-622.
[7] Carlos Ponce al. "Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences". In: Cell 177.4 (2019), pp. 999-1009,
[8] Pouya Bashivan, Kohitij Kar, and James J DiCarlo. "Neural population control via deep image synthesis". In: Science 364.6439 (2019), eaav9436.
[9] Blake A Richards et al. "A deep learning framework for neuroscience". In: Nature neuroscience 22.11 (2019). pp. 1761-1770.
[10] Nuo Li and James DiCarlo. "Unsupervised natural experience rapidly alters invariant object representation in visual cortex". In: science 321 $21.5895 (2008), pp. 1502-1507.
[11] Giulio Matteucci and Davide Zoccolan. "Unsupervised experience with temporal continuity of the visual environment is causally involved in the development of V1 complex cells". In: Science advances 6.22 (2020), eaba3742.
[12] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. "Explaining and harnessing adversarial examples". In: arXiv preprint arXiv: 1412.6572 (2014).
[13] Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. "Adversarial examples in the physical world". In: Artificial intelligence safety and security. Chapman and Hall/CRC, 2018, pp. 99- 112.
[14] Alessio Ansuini et al. "Intrinsic dimension of data representations in deep neural networks".
In: Advances in Neural Information Processing Systems 32 (2019).
[15] Sina Tafazoli et al. "Emergence of transformation-toleraatt representations of visual objects in rat lateral extrastriate cortex". In: Elife 6 (2017), e22794.
[16] Giulio Matteucci et al. "Nonlinear processing of shape information in rat lateral extrastriate cortex". In: Journal of Neuroscience 39.9 (2019), pp. 1649-1670.
[17] Stephen V David, Benjamin Y Hayden, and Jack L Gallant, "Spectral receptive field properties explain shape selectivity in area V4". In: Journal of neurophysiology 96.6 (2006), pp. 3492- 3505.
[18] Diego Doimo et al. "Hierarchical nucleation in deep neural networks". In: Advances in Neural Information Processing Systems 33 (2020), pp. 7526-7536.
[19] Andrea Benucci. "Motor-related signals support localization invariance for stable visual perception".. In: PLoS computational biology 18.3 (2022), e1009928.
[20] Elena Facco et al. "Estimating the intrinsic dimension of datasets by minimal neighborhood
information". In: Scientific reports 7.1 (2017), pp. 1-8.
[21] Karen Simonyan and Andrew Zisserman. "Very deep convolutional networks for large-scale image recognition". In: arXiv preprint arXiv:1409.1 1556 (2014).
[22] Stefano Panzeri and Alessandro Treves. "Analytical estimates of limited sampling biases in different information measures". In: Network: Computation in neural systems 7.1 (1996),
p. 87.
[23] Stefano Panzeri et al. "Correcting for the sampling bias problem in spike train information measures". In: Journal of neurophysiology 98.3 (2007), pp. 1064-1072.

---

[24] Rafael C Gonzalez and Richard E Woods. "Digital image processing, prentice hall".. In: Upper Saddle River, NJ (2008).
[25] Séverine Durand et al. "A comparison of visual response properties in the lateral geniculate nucleus and primary visual cortex of awake and anesthetized mice". In: Journal of Neuroscience 36.48 (2016), pp. 12144-12156.
[26] James J DiCarlo, Davide Zoccolan, and Nicole C Rust. "How does the brain solve visual object recognition?" In: Neuron 73.3 (2012), pp. 434.
[27] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. "Deep learning.. In: nature 521.7553 (2015), pp. 434-444.
[28] Edgar Y Walker et al. "Inception loops discover what excites neurons most using deep predictive models", In: Nature neuroscience 22.12 (2019), pp. 2060-2065.
[29] Santiago A Cadena et al. "How well do deep neural networks trained on object recognition characterize the mouse visual system?" In: (2019).
[30] Nicole C Rust and James DiCarlo. "Selectivity and tolerance ("invariance") both increase as visual information propagates from cortical area V4 to IT"". In: Journal of Neuroscience 30.39 (2010), pp. 12978-12995.
[31] Ben Vermaereke et al. "Functional specialization in rat occipital and temporal visual cortex". In: Journal of neurophysiology 112.8 (2014), pp. 1963-1983.
[32] Eugenio Piasini et al. "Temporal stability of stimulus representation increases along rodent visual cortical hierarchies". In: Nature communications 12.1 (2021), pp. 1-19.
[33] Emmanouil Froudarakis et al. "Object manifold geometry across the mouse cortical visual hierarchy". In: BioRxiv (2021), pp. 2020-08.
[34] Kasper Vinken et al. "Neural representations of natural and scrambled movies progressively change from rat striate to temporal cortex". In: Cerebral Cortex 26.7 (2016), pp. 3310-3322.
[35] Javier Orlandi et al. "Distributed context-dependent choice information in mouse dorsal- parietal cortex". In: (2021).
[36] Scott I. Brincat et al. "Gradual progression from sensory to task-related processing in cerebral cortex". " In: Proceedings of the National Academy of Sciences 115.30 (2018), E7202-E7211.
[37] Joshua H Siegle al. "Survey of spiking in the mouse visual system reveals functional hierarchy". In: Nature 592. 7822 (2021), pp. 86-92.
[38]
Matthew D Zeiler and Rob Fergus. "Visualizing and understanding convolutional networks".
In: European conference on computer vision. Springer. 2014, pp. 818-833.
[39] Ravid Shwartz-Ziv and Naftali Tishby. Opening the Black Box of Deep Neural Networks via Information. Apr. 2017. DOI: [0. 48550/ www.4485550.......1 arxiiv.1703 000.17773001100000 Xiv: 0000000810 [cs]
[40] Katharina Dobs et al. "How Face Perception Unfolds over Time" In: Nature Communications
10.1 (Mar. 2019), p. 1258. ISSN: 2041- 1723. DOI: 10. 1038/ 0118///4467--19--0 019-0999---99999--1111
[41] Shany Grossman et al. "Convergent Evolution of Face Spaces across Human face-Selective Neuronal Groups and Deep Convolutional Networks". In: Nature Communications 10.1 (Oct.
2019), p. 4934. ISSN: 2041-1723. DOI: 10. wwe.111//4446---1 -19-3----166
[42] Andrew Jaegle et al. "Population Response Magnitude Variation in Inferotemporal Cortex Predicts Image Memorability". In: eLife 8 (Aug. 2019). Ed. by Timothy E Behrens, Tatiana Pasternak, and Elizabeth A Buffalo, e47596. ISSN: 2050-084X. DO1: 10. 77554/eLife.444
[43] Nicole C. Rust and Vahid Mehrpour. "Understanding Image Memorability". In: Trends in Cognitive Sciences 24.7 (July 2020), pp. 557-568. ISSN: 1364-6613. DOI: 10. 1016/ tics
2020. 02.04..00 001
[44] Kasper Vinken and Hans Op de Beeck. "Using deep neural networks to evaluate object vision tasks in rats". In: PLoS computational biology 17.3 (2021), c1008714.
[45] Irina Higgins et al. "Unsupervised Deep Learning Identifies Semantic Disentanglement in Single Inferotemporal Neurons". In: Nature Communications 12.1 (Dec. 2021), p. 6456. ISSN: 2041-1723. DOT: 10. 1038 1030/0014447700 9414//444477-2225 26751-5 arXiv: 2006. 14304 [q-bio]
[46] Ramprasaath R Selvaraju et al. "Grad-cam: Visual explanations from deep networks via gradient-based localization". In: Proceedings of the IEEE international conference on computer vision. 2017, pp. 618-626.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "qkhUmwSgHMK", "reviewer": "Reviewer_sQm2", "summary": "The paper investigates a parallel between two processing systems: (1) a deep convolutional neural network (VGG-16), and (2) the rat visual cortex. More precisely, the paper considers previous results obtained on one system, and attempts to replicate them on the other system. These results are related to the processing of information along each system.\n\nTo investigate which information is represented in each layer of each system, the paper considers a number of metrics computed on the intermediate representations: \n- intrinsic dimensionality \n- mutual information with an image metric computed on the input (average luminosity, contrast, orientation, couple of orientations)\n- decoding accuracy\n\nBy considering the trend of these metrics over the layers of each system, the paper demonstrates a parallel between the two systems.\nThe results also describe the difference in representations between a pretrained CNN and a network with random weights. ", "questions": "1. The paper uses neurophysiological recordings in rats. The justification for this choice is that it enable recording simultaneously over many part of the visual hierarchy. Can the authors discuss the possibility of using fMRI in humans, which records simultaneously over more parts of the visual hierarchy?\n2. To best match VGG-16 and the visual cortex, which subset of layers would best correspond to the four visual areas (V1-LM-LI-LL) ? Is there a way to select a subset of layers with consistent trends with the four visual areas?\n3. Why is figure 4 limited to luminosity and contrast, and does not include orientation and corners as figure 3?", "limitations": "4. The authors briefly discuss the fact that the dataset considered is not capturing the entire visual hierarchy, thus limiting the parallel with the CNN architecture.\n\n5. The authors have not discuss the choice of VGG-16 compared to other existing CNN architectures.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Clarity**\n\nThe paper is very clearly written in all aspects: exposition of the problem, relation with the literature, proposed method, results, and discussion.\nIt is well structured, and the figures are of top quality. \n\nThe results seem sufficiently described to be reproducible. Additionally, sharing the code to reproduce all the figures is appreciated (not tested).\n\n**Originality**\n\nThe investigation is relatively original.\nA large number of paper have investigated mappings between artificial and biological neural networks, particularly CNN and the visual cortex (in humans, monkeys, rats, and more), using methods such as encoding models or representation similarity analysis.\n\nThis paper follows this line of work, but it is original in that it investigates a number of manually computed metrics. These metrics have a somewhat limited complexity, but they can be computed over different layers to investigate explicitly how the information is processed in each system.\n\n\n**Quality**\n\nThe experiments are sound, seem well executed, and the results are reasonable.\n\nThe paper is structured around the replication of figure 1 (ABCD) in the other system:\n- figure 1A vs figure 2. The hunchback profile in the visual cortex is not as neat as in the CNN, but the results are somewhat consistent. The consistency assumes we ignore (roughly) layers 9-16.\n- figure 1B vs figure 3A/B. The decreasing trends are consistent, but only if we ignore layers 1-3.\n- figure 1D vs figure 3C/D. Here the trends are not as consistent as stated. In the visual cortex, the orientation representation decreases with depth, and the corner representation increases with depth. In the CNN, the orientation trend is only decreasing if we ignore layers 3-5, and the corner trend is only increasing if we ignore layers 5-16.\n- figure 1C vs figure 5. The normalized mutual information mostly increases in the last layers (say 7-16).\n\nOverall, the trend consistency between the visual cortex and the CNN is a bit overstated. The trends are only consistent if we consider a subset of layers, and this subset is not the same depending on the representation considered.\n\n\n**Significance**\n\nThe results are not particularly surprising. A lot of studies have already shown a parallel between CNN and the visual cortex in mammals.\nThis new study adds up to this parallel, describing it with an original approach.\n\nThe pretrained-vs-random results are all unsurprising, but nicely quantified in the different experiments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VcjBdAHhmgw", "reviewer": "Reviewer_vMFS", "summary": "In this work, the authors have studied the similarity in the emergence of certain statistical patterns across neural activity recorded from the rat ventral visual stream and a (trained / randomly initialized) deep convolutional network optimized to perform large-scale object recognition on ImageNet. The authors present 3 interesting observations: 1) similar trend of intrinsic dimensionality of object representations across the rat ventral stream and pretrained deep convolutional networks 2) observed the pruning of low-level and mid-level information which followed their distillation in early layers of the rat ventral visual stream and VGG-16, these effects being a result of training and not merely a byproduct of the hierarchical processing architecture, 3) object categorization decoding accuracy closely tracked the increase of object-specific information across the VGG-16 hierarchy.\n", "questions": "- In Fig. 5, why do the authors think there is a sharp decrease in the decoding accuracy of the last two fully connected layers of the randomly initialized network, compared to the rest of the conv and fully connected layers that produce significantly better decoding of object category? ", "limitations": "- As mentioned in the weaknesses, I would like the authors to please discuss limitations and provide directions for extending the current work.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- There has been an explosion of recent studies exploring the similarity between biological and machine visual perception via representation similarity analysis and direct comparison of neural representations on a common stimulus set. This work on the other hand presents a less commonly studied yet important complementary direction of observing commonly emergent statistical trends in the representations of biological and artificial neural networks and highlights the change in these trends as a function of training.\n- While the current study is restricted to comparing object recognition trends of deep networks with the rat ventral stream, the general methodology presented can be extended to primate analyses once the technology to record primate data across the ventral visual stream for a considerably large set of images is available.\n- The authors provide adequate mathematical background in the Methods section to clearly follow the key results presented in the subsequent sections in comparing the statistical trends between rat ventral stream and VGG-16 representations.\nTransparent disclosure of which neurons were discarded from the analyses and the criteria for discarding them (poor selectivity to the stimulus features being analyzed) in section 2.3.\n- I found the paper to be very well-written and organized with clearly labeled plots (for the most part) along with detailed captions describing the key results are very helpful.\n\n\nWeaknesses\n- While the current analyses are interesting and sufficient to convey the key observations made by the authors, a high-level comment would be to relate the observed statistical trends with those obtained by the RSA studies measuring brain-CNN similarity. For e.g. it appears that a related finding is shown in https://arxiv.org/abs/1807.00053 in Fig 5, wherein there exists a correspondence between the decoding accuracy of successive layers of a pre-trained deep neural network with successive areas of the primate ventral visual stream. \n- Nit: Some of the plots may be hard to read due to the minor difference in the color or linestyle of the various markers. E.g. the authors must please check if Fig 4’s coloring scheme is accessible to color-blind readers. It was difficult for me to initially identify the solid vs dashed lines in Fig. 5 as they look quite similar here. Similar to Fig. 3, I would add plot titles “orientation” and “contrast” to Fig 4.A and 4.B for improved readability.\n- The authors could potentially discuss more the limitations of the current work and highlight areas of improvement/extension that could be useful to inspire future work.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "T6Oy_KeUALr", "reviewer": "Reviewer_2Y7t", "summary": "The paper compares internal representations of what could be called the rat's ventral visual stream and the VGG-16 network trained on ImageNet. The authors find that the intrinsic dimensionality (ID) of rat ventral stream and VGG-16 follow a similar pattern, with ID first increasing and then decreasing again. In addition, they estimate the mutual information between some low-level image features and internal representations and also find similar trends.\n", "questions": " 1. While I find it quite interesting that the ID follows a similar pattern in both brain and ANN, several questions remain:\n\n    - How general is this finding across different ANN architectures (e.g. ResNet, ViT)\n\n    - The peak seems to be much earlier in VGG/AlexNet than in the brain. How do you explain this result?\n\n 1. How can the mutual information increase over layers (Figs. 3–5)? Information cannot be created by processing (data processing inequality), so all information that is there about the stimulus in, say, layer 5 also has to be there already in layer 3. Thus, I believe that the fact that the MI curves increase in the first couple of layers in Figs. 3–5 is probably just an artifact of estimating MI from finite data. This makes me wonder how informative these curves are in general.\n\n 1. Assuming the analyses were meaningful, it is not clear to what extent they represent general properties of artificial neural networks trained on ImageNet. Would these findings hold across a variety of more modern architectures? Do they depend on the training objective? Many questions remain open, and just analyzing VGG-16 will not be able to answer them.\n\n 1. Suppose the representational similarities indeed existed, how would this result change how we think about or advance machine learning – the goal of this conference – or neuroscience? Wouldn't we want to know what causes these similarities (network architecture, training objective, dataset, etc.) or what we have to change things to destroy them?\n", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths \n + Understanding similarities and differences of real and artificial neural networks is an important topic\n + Paper is clearly written and easy to follow\n + Finding that ID follows a similar pattern in brain & ANN might potentially be interesting\n\n### Weaknesses\n - Analysis way to restrictive, considering only one (fairly old) CNN architecture\n - Increasing MI values in Fig. 3–5 are implausible (-> data processing inequality)\n - Unclear how the analysis advances either ML or neuroscience\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RDpsLKB12wU", "reviewer": "Reviewer_DRfj", "summary": "The paper analyzes whether low- and mid-level feature information is clearly represented in early CNN layers, but not as clearly represented in later CNN layers, a property that is present in early and late vision processing systems in mice. In particular, the low- and mid-level features of luminosity, contrast, orientation, and corners are investigated. Additionally, the paper shows that the intrinsic dimension of representations is relatively low in early CNN layers, but much higher in later CNN layers.", "questions": "One thing to note is that prune and distill have defined meanings in machine learning model compression (Graves, 2011; Hinton et al., 2014; Blundell et al., 2015), so their use in this paper could be jarring to some readers.\n\nCadena et al. (2019) found that ImageNet trained CNNs are not a particularly good model of the mouse visual cortex. Discussing how their results relate to your approach in the introduction would be good.\n\nHow does your work on the dimensionality of different neural network layers add to the work presented in Güçlü and van Gerven (2015), who show that the Kolmogorov complexity of layer representations increases with depth?\n\nWhy look at the pre-activations when analyzing the activations of a neural network layer? Isn't the information accessible by the next layer dependent on what is let through by the activation function?\n\nGraves, A. (2011). Practical variational inference for neural networks. Advances in neural information processing systems, 24.\n\nHinton, G., Vinyals, O., & Dean, J. (2014). Distilling knowledge in a neural network. NIPS 2014 Deep Learning Workshop.\n\nBlundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015, June). Weight uncertainty in neural network. In International conference on machine learning (pp. 1613-1622). PMLR.\n\nCadena et al. (2019). How well do deep neural networks trained on object recognition characterize the mouse visual system? NeurIPS 2019 NeuroAI Workshop\n\nGüçlü, U., & van Gerven, M. A. (2015). Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. Journal of Neuroscience, 35(27), 10005-10014.\n", "limitations": "The authors should add a discussion on the limitations of their work to the Conclusions and Discussion section. One limitation that potentially could be discussed is that high level statistical properties can be shared by systems that process information differently.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the paper is reasonably written and effectively conveys what the authors did. However, the originality and significance of the paper are, in my opinion, not very high. While specifically looking at luminosity, contrast, orientation, and corner information is nice, the idea that early feature information is prominent in early CNN layers and is lost in later layers is not new. Nor is the idea that the complexity of layer representations increases from early to late CNN layers. One extremely relevant paper on the topic is Güçlü and van Gerven (2015), which showed that information present in early human visual areas is prominent in early CNN layers, but not in later CNN layers. That paper also showed that the Kolmogorov complexity of layer representations increases with depth. The analysis of mutual information between layer representations and labels is not new, either [35]. The comparison of CNNs to mouse data is, also, not new (Cadena et al., 2019). Cadena et al. (2019) found that CNNs were a poor model of the visual system of mice. If the submitted paper could address, and present evidence for, why their results contradict the findings of Cadena et al. (2019), it would strengthen the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept