Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper analyzes whether low- and mid-level feature information is clearly represented in early CNN layers, but not as clearly represented in later CNN layers, a property that is present in early and late vision processing systems in mice. In particular, the low- and mid-level features of luminosity, contrast, orientation, and corners are investigated. Additionally, the paper shows that the intrinsic dimension of representations is relatively low in early CNN layers, but much higher in later CNN layers.
strengths_and_weaknesses: Overall, the paper is reasonably written and effectively conveys what the authors did. However, the originality and significance of the paper are, in my opinion, not very high. While specifically looking at luminosity, contrast, orientation, and corner information is nice, the idea that early feature information is prominent in early CNN layers and is lost in later layers is not new. Nor is the idea that the complexity of layer representations increases from early to late CNN layers. One extremely relevant paper on the topic is Güçlü and van Gerven (2015), which showed that information present in early human visual areas is prominent in early CNN layers, but not in later CNN layers. That paper also showed that the Kolmogorov complexity of layer representations increases with depth. The analysis of mutual information between layer representations and labels is not new, either [35]. The comparison of CNNs to mouse data is, also, not new (Cadena et al., 2019). Cadena et al. (2019) found that CNNs were a poor model of the visual system of mice. If the submitted paper could address, and present evidence for, why their results contradict the findings of Cadena et al. (2019), it would strengthen the paper.
questions: One thing to note is that prune and distill have defined meanings in machine learning model compression (Graves, 2011; Hinton et al., 2014; Blundell et al., 2015), so their use in this paper could be jarring to some readers.

Cadena et al. (2019) found that ImageNet trained CNNs are not a particularly good model of the mouse visual cortex. Discussing how their results relate to your approach in the introduction would be good.

How does your work on the dimensionality of different neural network layers add to the work presented in Güçlü and van Gerven (2015), who show that the Kolmogorov complexity of layer representations increases with depth?

Why look at the pre-activations when analyzing the activations of a neural network layer? Isn't the information accessible by the next layer dependent on what is let through by the activation function?

Graves, A. (2011). Practical variational inference for neural networks. Advances in neural information processing systems, 24.

Hinton, G., Vinyals, O., & Dean, J. (2014). Distilling knowledge in a neural network. NIPS 2014 Deep Learning Workshop.

Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015, June). Weight uncertainty in neural network. In International conference on machine learning (pp. 1613-1622). PMLR.

Cadena et al. (2019). How well do deep neural networks trained on object recognition characterize the mouse visual system? NeurIPS 2019 NeuroAI Workshop

Güçlü, U., & van Gerven, M. A. (2015). Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. Journal of Neuroscience, 35(27), 10005-10014.

limitations: The authors should add a discussion on the limitations of their work to the Conclusions and Discussion section. One limitation that potentially could be discussed is that high level statistical properties can be shared by systems that process information differently.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this work, the authors have studied the similarity in the emergence of certain statistical patterns across neural activity recorded from the rat ventral visual stream and a (trained / randomly initialized) deep convolutional network optimized to perform large-scale object recognition on ImageNet. The authors present 3 interesting observations: 1) similar trend of intrinsic dimensionality of object representations across the rat ventral stream and pretrained deep convolutional networks 2) observed the pruning of low-level and mid-level information which followed their distillation in early layers of the rat ventral visual stream and VGG-16, these effects being a result of training and not merely a byproduct of the hierarchical processing architecture, 3) object categorization decoding accuracy closely tracked the increase of object-specific information across the VGG-16 hierarchy.

strengths_and_weaknesses: Strengths:
- There has been an explosion of recent studies exploring the similarity between biological and machine visual perception via representation similarity analysis and direct comparison of neural representations on a common stimulus set. This work on the other hand presents a less commonly studied yet important complementary direction of observing commonly emergent statistical trends in the representations of biological and artificial neural networks and highlights the change in these trends as a function of training.
- While the current study is restricted to comparing object recognition trends of deep networks with the rat ventral stream, the general methodology presented can be extended to primate analyses once the technology to record primate data across the ventral visual stream for a considerably large set of images is available.
- The authors provide adequate mathematical background in the Methods section to clearly follow the key results presented in the subsequent sections in comparing the statistical trends between rat ventral stream and VGG-16 representations.
Transparent disclosure of which neurons were discarded from the analyses and the criteria for discarding them (poor selectivity to the stimulus features being analyzed) in section 2.3.
- I found the paper to be very well-written and organized with clearly labeled plots (for the most part) along with detailed captions describing the key results are very helpful.


Weaknesses
- While the current analyses are interesting and sufficient to convey the key observations made by the authors, a high-level comment would be to relate the observed statistical trends with those obtained by the RSA studies measuring brain-CNN similarity. For e.g. it appears that a related finding is shown in https://arxiv.org/abs/1807.00053 in Fig 5, wherein there exists a correspondence between the decoding accuracy of successive layers of a pre-trained deep neural network with successive areas of the primate ventral visual stream. 
- Nit: Some of the plots may be hard to read due to the minor difference in the color or linestyle of the various markers. E.g. the authors must please check if Fig 4’s coloring scheme is accessible to color-blind readers. It was difficult for me to initially identify the solid vs dashed lines in Fig. 5 as they look quite similar here. Similar to Fig. 3, I would add plot titles “orientation” and “contrast” to Fig 4.A and 4.B for improved readability.
- The authors could potentially discuss more the limitations of the current work and highlight areas of improvement/extension that could be useful to inspire future work.

questions: - In Fig. 5, why do the authors think there is a sharp decrease in the decoding accuracy of the last two fully connected layers of the randomly initialized network, compared to the rest of the conv and fully connected layers that produce significantly better decoding of object category? 
limitations: - As mentioned in the weaknesses, I would like the authors to please discuss limitations and provide directions for extending the current work.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper compares internal representations of what could be called the rat's ventral visual stream and the VGG-16 network trained on ImageNet. The authors find that the intrinsic dimensionality (ID) of rat ventral stream and VGG-16 follow a similar pattern, with ID first increasing and then decreasing again. In addition, they estimate the mutual information between some low-level image features and internal representations and also find similar trends.

strengths_and_weaknesses: ### Strengths 
 + Understanding similarities and differences of real and artificial neural networks is an important topic
 + Paper is clearly written and easy to follow
 + Finding that ID follows a similar pattern in brain & ANN might potentially be interesting

### Weaknesses
 - Analysis way to restrictive, considering only one (fairly old) CNN architecture
 - Increasing MI values in Fig. 3–5 are implausible (-> data processing inequality)
 - Unclear how the analysis advances either ML or neuroscience

questions:  1. While I find it quite interesting that the ID follows a similar pattern in both brain and ANN, several questions remain:

    - How general is this finding across different ANN architectures (e.g. ResNet, ViT)

    - The peak seems to be much earlier in VGG/AlexNet than in the brain. How do you explain this result?

 1. How can the mutual information increase over layers (Figs. 3–5)? Information cannot be created by processing (data processing inequality), so all information that is there about the stimulus in, say, layer 5 also has to be there already in layer 3. Thus, I believe that the fact that the MI curves increase in the first couple of layers in Figs. 3–5 is probably just an artifact of estimating MI from finite data. This makes me wonder how informative these curves are in general.

 1. Assuming the analyses were meaningful, it is not clear to what extent they represent general properties of artificial neural networks trained on ImageNet. Would these findings hold across a variety of more modern architectures? Do they depend on the training objective? Many questions remain open, and just analyzing VGG-16 will not be able to answer them.

 1. Suppose the representational similarities indeed existed, how would this result change how we think about or advance machine learning – the goal of this conference – or neuroscience? Wouldn't we want to know what causes these similarities (network architecture, training objective, dataset, etc.) or what we have to change things to destroy them?

limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper investigates a parallel between two processing systems: (1) a deep convolutional neural network (VGG-16), and (2) the rat visual cortex. More precisely, the paper considers previous results obtained on one system, and attempts to replicate them on the other system. These results are related to the processing of information along each system.

To investigate which information is represented in each layer of each system, the paper considers a number of metrics computed on the intermediate representations: 
- intrinsic dimensionality 
- mutual information with an image metric computed on the input (average luminosity, contrast, orientation, couple of orientations)
- decoding accuracy

By considering the trend of these metrics over the layers of each system, the paper demonstrates a parallel between the two systems.
The results also describe the difference in representations between a pretrained CNN and a network with random weights. 
strengths_and_weaknesses: **Clarity**

The paper is very clearly written in all aspects: exposition of the problem, relation with the literature, proposed method, results, and discussion.
It is well structured, and the figures are of top quality. 

The results seem sufficiently described to be reproducible. Additionally, sharing the code to reproduce all the figures is appreciated (not tested).

**Originality**

The investigation is relatively original.
A large number of paper have investigated mappings between artificial and biological neural networks, particularly CNN and the visual cortex (in humans, monkeys, rats, and more), using methods such as encoding models or representation similarity analysis.

This paper follows this line of work, but it is original in that it investigates a number of manually computed metrics. These metrics have a somewhat limited complexity, but they can be computed over different layers to investigate explicitly how the information is processed in each system.


**Quality**

The experiments are sound, seem well executed, and the results are reasonable.

The paper is structured around the replication of figure 1 (ABCD) in the other system:
- figure 1A vs figure 2. The hunchback profile in the visual cortex is not as neat as in the CNN, but the results are somewhat consistent. The consistency assumes we ignore (roughly) layers 9-16.
- figure 1B vs figure 3A/B. The decreasing trends are consistent, but only if we ignore layers 1-3.
- figure 1D vs figure 3C/D. Here the trends are not as consistent as stated. In the visual cortex, the orientation representation decreases with depth, and the corner representation increases with depth. In the CNN, the orientation trend is only decreasing if we ignore layers 3-5, and the corner trend is only increasing if we ignore layers 5-16.
- figure 1C vs figure 5. The normalized mutual information mostly increases in the last layers (say 7-16).

Overall, the trend consistency between the visual cortex and the CNN is a bit overstated. The trends are only consistent if we consider a subset of layers, and this subset is not the same depending on the representation considered.


**Significance**

The results are not particularly surprising. A lot of studies have already shown a parallel between CNN and the visual cortex in mammals.
This new study adds up to this parallel, describing it with an original approach.

The pretrained-vs-random results are all unsurprising, but nicely quantified in the different experiments.
questions: 1. The paper uses neurophysiological recordings in rats. The justification for this choice is that it enable recording simultaneously over many part of the visual hierarchy. Can the authors discuss the possibility of using fMRI in humans, which records simultaneously over more parts of the visual hierarchy?
2. To best match VGG-16 and the visual cortex, which subset of layers would best correspond to the four visual areas (V1-LM-LI-LL) ? Is there a way to select a subset of layers with consistent trends with the four visual areas?
3. Why is figure 4 limited to luminosity and contrast, and does not include orientation and corners as figure 3?
limitations: 4. The authors briefly discuss the fact that the dataset considered is not capturing the entire visual hierarchy, thus limiting the parallel with the CNN architecture.

5. The authors have not discuss the choice of VGG-16 compared to other existing CNN architectures.
review_title: null

Meta-review: metareview: This paper examines the intrinsic dimensionality (ID) of representations in the rat brain and CNNs. The authors show that the rat brain, like the CNNs studied, have distinct expansion-contraction phases, and that in the CNNs one also observes a distillation and pruning of low- and mid-level information, similar to what is seen in the brain. The authors also show that in the CNNs high-level object information only emerges after these steps of distillation and pruning. This illustrates potentially interesting parallels between information processing in the real brain and CNNs.

The reviewers were split on this paper. The initial reviews identified issues of novelty and insight. But, after the author responses, three out of four agreed that the work was interesting and technically sound, and they found the paper well-written. One reviewer was still concerned that the paper does not do enough to show that these similarities in ID and information retention tell us anything meaningful. They were also concerned that comparisons were not made across enough architectures. Nonetheless, given the balance in the reviews and post response scores, an accept decision was reached.
confidence: Less certain
Decision: Accept