Venue: NeurIPS 2019 Workshop Neuro_AI

Reviews:

final_score: rating: null
review_content: importance_comment: This paper shows data that refutes a previous result that representations from unsupervised models are not good at predicting visual areas (mainly IT). Instead, the authors find that a model trained to predict the next frame in a video is more correlated with visual areas.  

Minor: Which unsupervised models did the 2014 paper use? Should we call PredNet self-supervised?

Only using ResNet and AlexNet are not enough as baselines as a lot of recent work has been done in this area.
rigor_comment: The paper appears sound. RSA with Spearman correlation is widely used in the field but I don't think that it is the best approach to relate different representations as its properties are not theoretically analyzed. There are other theoretically derived similarity measures or other approaches such as encoding models that I believe lead to more interpretable results. It is also not mentioned how the noise ceiling was estimated and accounted for.

The authors mention all results are statistically significant. Does that mean that the individual correlations are higher than chance or the bolded correlations are significantly higher than the others? Has multiple comparison correction been made? All details are useful and there is space for them.
clarity_comment: This paper is well written but misses many details that are important to understand everything that was done (see other comments).
intersection_comment: This paper is exactly at the intersection of (cognitive) neuroscience and AI.
review_title: Representations from unsupervised models are correlated with brain activity, details missing

final_score: rating: null
review_content: importance_comment: The authors seek to develop unsupervised training models that exhibit image responses correlated to observed fMRI and MEG activity. This work fits into a tradition of similar inquiries and shows that a predictive, unsupervised network can exhibit higher correlation to neural activity than a supervised convolutional network. 
rigor_comment: The use of multiple datasets contributes to the rigor of the paper. While the similarity measures reported in the paper are based in literature, as reported they are quite opaque, and more detailed analysis would be needed to be persuasive. More detailed reporting of similarity (e.g. example trials or time-courses, alternate criteria) could be helpful. 

clarity_comment: The paper poses an interesting question and develops it clearly. The empirical results reported are not straightforward to interpret.
intersection_comment: The paper asks whether artificial neural networks can be useful models of brain activity, and therefore sits comfortably at the interesection of neuroscience and artificial intelligence.

comment: The authors present evidence that an unsupervised, predictive-coding model of vision is more correlated to neural data in its responses than popular, supervised models. This is further evidence that supervised feedforward models fail to capture something substantial about natural vision, although the particular predictive implementation described here might or might not be descriptive of reality. Without detailed insight into the similarities, it is difficult to evaluate whether the similarities are persuasive.

While the authors report similarity between PredNet representations and data as compared to feedforward architectures, there’s a question that seems to be unaddressed — which may be technically difficult to address fully, but is necessary to understand at least qualitatively: is the performance of the predictive network, at the volume of training data used in this study, comparable to the performance of the feedforward networks? An answer to this question in either direction would not detract from the results presented in the study, but would clarify what existing models have and have not captured about image processing in the brain.
review_title: Prediction-based network is similar to neural data; more detailed analysis would be helpful.

final_score: rating: null
review_content: importance_comment: I believe the concept of using predictive coding and unlabeled video data to train convnets is a great idea. However, the contribution of the authors does not appear to extend beyond combining existing data sets with existing network architectures.
rigor_comment: The work is lacking a discussion of the most recent work in the similarity of visual processing in convnets to brain data, which incorporate recurrence into convnets (Nayebi et al. 2018, Kubilius et al. 2018 and 2019), thereby potentially allowing for similar behavior as a PredNet. How would you expect those networks to perform when trained on unlabeled video data?

It would have been useful to put these in context of the results of the algonauts contest, which pitched supervised methods such as Alexnet against user-submitted content. Does PredNet outperform other user-submitted models?

For this result to be convincing, I would like to see some reasons why the authors think PredNet is outperforming previous models. For example, is there something different about the feature maps that support this? What precisely about predictive coding makes the similarity to brain data expected?
clarity_comment: Results were presented quite clearly, although datasets and methods rely entirely on previously published work, such that digging into previous work on PredNet and the Algonauts project was necessary for a full understanding.
intersection_comment: The question of how the visual world is represented in the brain is an essential question in neuroscience as well as for building successful machine learning techniques for artificial vision.
comment: It does not seem like predictive coding is the main thing going on in V1 (Stringer et al., Science 2019), so I’d be curious how the authors think that should be taken into account in the future.

Typo line 24 – “Moreover, we show that as (we) train the model…”
Typo line 87 – “Second, the model does not rely on labeled data and learn(s)”
review_title: Training of predictive networks on unlabeled video data appears to increase similarity to brain data

Decision: Accept (Poster)