Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper uses recordings from monkeys to co-train a classifier with a second head from layer conv-3-1 (of VGG-19) that predicts the cortical responses in V1 to the same pictures. The authors test this model on 14 different distortions of images and find that it is more robust than a baseline model trained only to classify the images. They show an upper-bound on robustness via a network trained on distorted images from the same distributions, and show that training on scrambled neural responses provides no advantages. 

Finally, they analyze the network in two ways, qualitatively and quantitatively. First they reconstruct several input images corrupted by gaussian noise from conv-3-1 in their model and the various control networks. They find that the monkey-trained network appears to focus more on the object than the noise compared to a network trained only on the images. This is obviously subjective, so they go on to measure how much the monkey-trained network attends to salient regions (based on Deep-Gaze-2) compared to the other models, and find that it is generally more sensitive to the salient regions.
main_review: Originality: While several previous works have used neural recordings (fMRI, monkey cortex) to regularize their models, this is (I believe) the first to co-train a model on categorization of the image as well as regression to the monkey recordings. Furthermore, I don’t believe anyone has measured the robustness to such a broad array of types of distortions in a neurally-constrained model. 

Quality: Everything appears technically sound. The results are strong, and there are thorough control networks, each controlling for different things, or testing different aspects. There is a baseline model that is only trained on clean images. Then there is the main model, named “MTL-monkey”, which instantiates the main idea of co-training on clean images and predicting V1 neural recordings. Actually, the “neural recordings” are from a network trained to produce them. This is done to give the MTL network a single target for each image, since the monkey recordings vary across presentations. It is unclear why this was necessary; one wonders what would happen if the MTL-monkey network had been trained directly on the data. However, (see below), this neural prediction network is used to make “pseudo-recordings” on images never shown to monkeys, expanding the dataset.

There are two “oracle” models as well. The first, simply titled the “Oracle”, is trained on a 50-50 split of clean and distorted images, using both classification and SimCLR-type representation learning between clean and distorted images at the conv-3-1 level. Now that the front end (up to conv-3-1) has been trained to be robust to distortions, the rest of the network is re-trained on clean images. This represents an upper bound on robustness where the front end of the network is what is robust to noise. 

The second oracle network is a multi-task trained network, called MTL_Oracle. It is identical to the MTL-monkey model in structure, but it is trained on neural predictions from the Oracle network. To provide these predictions, the Oracle network is frozen and V1 predictions are trained from the representations at conv-3-1. These neural predictions from the Oracle network are thus trained from a front-end that has been “robustified” to distortions, and provides an upper-bound on the amount of transfer learning possible from the neural signal. Then MTL-Oracle is trained only on clean images, but the targets are from a network sensitive to image distortions. This results in a network that is nearly as good as the Oracle network, and better than the baseline network, demonstrating that transfer learning is possible in principle. 

Clarity: The paper is written in a pretty confusing manner. It was not easy to extract the above description from the paper. There are a dizzying array of both single-task and MTL models. For example, the “Models” section starts on line 92. But at the end of the paragraph above that, the paper describes a single-task network that is simply trained to map from the images to the neural recordings. This network is then used to predict neural recordings for all of the TIN (Tiny Image Net) images. These predictions are then used in the following models. So, it seems that not all of the neural recordings that the MLT-monkey model is trained on are actually neural recordings, but instead are **predictions** of neural recordings. This makes sense, as it expands the data that can be used to train the neural responses, given the issue that monkey recordings are difficult, and one might not have neural recordings for all images. However, this should be made clearer in the paper. An example of where this leads to confusion is line 167, where you describe your model as being trained on “real neural data.” It isn’t, if I understand correctly.

Furthermore, the Models section doesn’t describe all of the models. Instead, it describes generically what network is used, and how it is modified to be fully-convolutional. 

Another confusing part is the description of the Oracle training. This is reported in the “Results” section - it probably should be back in the “Training” section, or the “Models” section. On line 141, it describes how the Oracle network, trained on 1:1 clean:distorted images, is frozen up to layer conv-3-1 and then the rest of the network is re-trained only on clean images. But then on line 142-144, they mention further training on the frozen part using SimCLR-style training to map the distorted and clean images closer together in layer conv-3-1. If you are doing that, it isn’t frozen! So, I infer that you are describing how the 1:1 network was trained **before** it was frozen. These sentences should be rearranged. Oh, and it isn’t clear that after you have frozen the network up to layer conv-3-1, that you re-initialized the rest of VGG-19 to random weights again, i.e., trained that part from scratch. I assume that’s what you did.

It would be best to systematically describe each model, and why you created it. First, there’s the neural recording predictor, used to create single targets for each image, and targets for images that were never actually shown to monkeys. This is the “neural data” that you train the MTL model on, as well was what you trained the frozen Oracle model on to make MTL-Oracle. The reason for MTL-Oracle is to demonstrate that transfer learning is possible. It might make the paper clearer to leave out MTL-Oracle altogether, since you demonstrate transfer learning with MTL-Monkey, but that’s up to you. Also, you should describe MTL-shuffled in the methods before the results. Why did you create MTL-shuffled? Just to show that adding on a second head doesn’t help if it isn’t making correct predictions (this seems a bit silly, but ok). Because the reader encounters Figure 3 before the descriptions of all of the different models, many of the names given there are somewhat mysterious. 

Under the Models section, you mention using pooling; I assume you are referring to global average pooling? If so, say so.

It’s a bit weird to say that you learn to balance the two losses, but then say you can control it with the batch ratio. Then you use a batch ratio of 1:1! It is only later that this is used in Figure 3C - it might be good to say why you have this earlier. In the batch ratio, I assume that in order to implement this, you have instances where you only collect gradients from the neural head when the ratio favors the neural recordings and vice versa? Also, in Figure 3C, you mean the regression is from neural performance to robustness, right? Not the other way around. 

It would be good to have a figure showing a clean image along with the 14 distortions. We shouldn’t have to go back to the Hendrycks and Dietterich paper to find out what they look like. 

I notice there are no error bars on the baseline network. 

Line 172: What was the range of batch ratios used? 

Line 174: You describe this as “two-factor regression” - I read that to mean you had two independent variables. What are the two factors in the regression? There’s an independent variable (neural prediction) and a dependent variable (robustness), right? How can there be a p-value for both?

lines 248-251 (sentence starting “It could be…”) I can’t parse that sentence. Please rewrite it.

Line 260: I don’t know what you mean by all pixels up to a cumulative sum of 0.7. Pixels go from 0-255. What are you summing? Why not just do a median split?

In the discussion, you mention that it would be interesting to use higher brain areas in training. I think this has already been done in Kietzmann et al,, (2019) Recurrence is required to capture the representational dynamics of the human visual system. PNAS 116(43):21854-21863. There, they used the dynamics of RSA over time to train a network to have those dynamics. 

Significance: The results here are very interesting: Training a network to also predict neural responses makes it more robust. It isn’t obvious that this should be so, so this is an important result. I just wish the paper had been written more clearly. 




limitations_and_societal_impact: They mention some limitations, but I think one is that they used neural predictions rather than raw neural data. This is either a bug or a feature, depending on how you look at it. I think it should be promoted as a feature, myself, but it could be seen as a limitation.

I see no societal impacts of this work, and neither do they.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This study follows a recent trend of results that use similarity to neural representations in early visual areas as a regularizer during training of deep neural networks (DNNs) for object recognition. Here the authors jointly train a DNN for image classification (grayscale version of Tiny ImageNet) and predicting neuronal responses in macaque primary visual cortex (V1), observing that this joint training procedure leads to increased robustness to common image corruptions. Finally, the authors claim that this improvement in robustness is likely to result from an increased sensitivity to salient regions of an image.

[Score updated 5 -> 6]

main_review: Overall the paper is well written, clear and the experiments appear to be properly done. I found the analysis on the saliency maps particularly interesting, deserving further exploration.  However, I have several concerns that negatively affect my score. 


1) Main results are not entirely novel.
The authors state that the neural co-training hypothesis by Sinz et al was untested prior to this study. In my opinion, this statement is not entirely true since similar approaches have been used in the past. Specifically, Li et al NeurIPS 2019 used mouse V1 data to regularize CNNs towards more brain-like representations. This is very similar to the approach used here. While the method for introducing the regularization, the dataset (CIFAR vs Tiny ImageNet), and the species (mouse vs monkey) are different, this study is at most an extension of the existing approach, which is perfectly fine. However, as an extension it fails to make significant novel contributions. Dapello et al NeurIPS 2020 had already shown that introducing a neuroscientific model of monkey V1 leads to improved robustness to image corruptions on color ImageNet, which is a considerably larger dataset and closer to real-world applications than the one used on this study (check model trained without stochasticity for a better comparison). Furthermore, when comparing the improvements of these two studies on a corruption by corruption basis, the results are strikingly similar (improved robustness for noise, blur and digital corruptions and worse robustness for weather effects). So in summary, this study extends Li et al to a larger dataset and to predicting monkey V1, and in the process replicates findings already described in Dapello et al - macaque V1 similarity is correlated with robustness and making the early stages of the model better approximate V1 leads to improved robustness.
 
2) Clean accuracy missing.
Clean accuracy is only reported for the MTL-Monkey models in Figure 3C. This is problematic since the main comparisons are done using a robustness metric which normalizes accuracy to corrupted images by the clean accuracy. Also, the clean accuracy scores reported for the MTL-Monkey models appear to be surprisingly low (under 50%). If this is true, then the usefulness of this approach for improving models seems to be considerably limited.

3) Great variability in the absolute accuracy scores across corruption types.
While some perturbations show overall very large accuracies throughout the severity ranges (brightness between 30 and 50%), others have extremely low accuracies even at the lowest severity levels (defocus blur < 3%). This is very surprising and not consistent with the ImageNet-C accuracy levels. This suggests that the implementation of these corruptions for the grayscale Tiny ImageNet dataset is not appropriate and makes these results harder to interpret.

4) I am not a fan of the glass blur corruption being left out from the test dataset. Given all the compute used in this study, I fail to see how producing the glass blur perturbation for the grayscale TIN validation-set would represent a substantial increase in compute. This particular corruption affects other defense methods differently from the remaining blur corruptions and its exclusion here should be avoided. 

Minor issues:

- What is the range of repetitions per images during the neurophysiological recordings (measured per neuron) for both the training and the test images? This information should be reported in the paper

- The input size of 64x64 with 14.5ppd when matching the V1 recordings means that the model’s input size in physical quantities is ~4.5deg, is this correct? This is smaller than what was considered optimal by Cadena et al 2019. Why the change? 

- The authors claim that CNNs are state-of-the-art for neural response prediction of primate V1 (line 47). I would say that this statement is still somewhat under debate since Cadena et al 2019 reported only minor improvements over a very simple classical V1 model and more recently Marques et al 2021 argued that some optimized neuroscientific V1 models outperform CNNs in several V1 neural benchmarks.

[Score updated 5 -> 6]
limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of this study.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduces a new method of regularizing deep networks with neural data. Specifically, the network is co-trained on a main task and regression of neural recordings (or at least deep-network predictions of these recordings for new images). The analysis shows that a network trained on neural responses is sensitive to different features, and encodes gaze-salient image regions more strongly, than control networks. 
main_review: Originality: As the authors point out, the work is closely related to past work using representational similarity from neural recordings to regularize deep networks. The authors also mention that task/neural-data-reconstruction co-training has been suggested before (Sinz et al. 2019), but I think the idea has been in the air for some time before that, see e.g. (particularly the latter): 
  Rezai, O., Jentsch, P. B., & Tripp, B. (2018). A video-driven model of response statistics in the primate middle temporal area. Neural Networks, 108, 424-444.
  Arai, K., Keller, E. L., & Edelman, J. A. (1994). Two-dimensional neural network model of the primate saccadic system. Neural Networks, 7(6–7), 1115–1135. 
Nonetheless, this work seems to be the best example so far of carrying out this idea. 
 
Quality: I think the work is sound and well developed. It could potentially be explored more thoroughly using different tasks, datasets, networks, and other measures of robustness. 

Clarity: The writing is basically clear. The explanation of methodological details is compact (e.g. Gaussian readout, a recent and non-obvious approach, is only briefly mentioned), so I think it would be helpful to share the code. There are a few small points that could be further clarified, e.g.: 
1) Link, reference, and description of the TINS dataset. Also explain the choice of dataset. 
2) Which network was used to produce the predicted responses? I’m not sure whether line 96 refers to the model that produces predicted responses, or the site of co-training. I think the process is described on line 147, but it would help to expand the explanation around line 96. 

Significance: I have three reservations about the significance: 
1) Given that the approach is closely related to, e.g., Federer et al., I think it would be useful to compare the current approach to that one. Which one works better? Does each have advantages and disadvantages? 
2) The main practical benefit of co-training on neural data seems to be that the network becomes nearly as robust to corruptions as a network that is trained on corrupted images. However, neural data is expensive and corrupted images are cheap, so the practical significance may be minor, or perhaps the authors could discuss it in more detail. 
3) The pipeline for achieving the above benefit included corrupted images. Removing these from the pipeline, the MTL-Monkey model was notably less robust. 

limitations_and_societal_impact: These issues were not addressed, but I don’t think there are problematic impacts.
review_title: null

Meta-review: This paper shows that co-training on primate neural data improves the robustness of CNNs.  There were some questions about the relevance of the work with respect to other previous work, but this was handled well by the authors and the reviewers were satisfied by those clarifications (which should also be worked into the paper).  There were some ethical concerns raised, but these also were handled well.  In general, I believe the review process worked well here, the paper is stronger because of it, and should be accepted.
Decision: Accept (Poster)