Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work critically reanalyzes previously published deep neural network models of grid cell emergence. It shows that grid-cell-like representations arise only given overly specific implementations of the path integration task and might even depend on undocumented implementation choices. Hence, the training task alone does not predict the emergence of grid cells. The authors suggest that the overly specific occurrence of grid-cell-like representations in NNs trained on path integration reflects the posthoc nature of these modeling studies: Presumably, the implementations of these models were informally tweaked until they reproduced the existing biological phenomenology. The authors analyze the emergent grid-cell-like representations and provide some analytical results on why they arise. Finally, the authors argue that neural network-based modeling must rely more strongly on biological and theoretical constraints. Multiple training objectives may map to the same neural activity pattern. Conversely, the "correct" objective will not lead to the empirical neural activity pattern without the relevant biological constraints.
strengths_and_weaknesses: This is a deep, quantitive critique of current connectionist approaches to explaining grid cells. It is systematic and well presented. 

Having said that, I am left unconvinced by the claim of many-to-one mappings of training tasks and activity patterns (figure 8 top); the empirical results seem to demonstrate the opposite pattern - surprisingly high sensitivity of grid-cell emergence to the specific operationalization of the training objective. In principle, this result can be interpreted as a strength of the NN-based modeling approach - we can reject incorrect operationalization. This sensitivity contrasts with traditional neurobiological "word models" that always magically work since they do not commit to an implementation. Therefore, I believe it would be fairer and more constructive to frame the paper's results as a description of a considerable remaining modeling gap and avoid sweeping statements such as  "...that one often gets neither".
questions: * Why shouldn't we think about the specificity of grid-cell-like representation emergence as a feature rather than a bug of deep neural network modeling? (see above).
* How was the grid score defined? I could not find the relevant equation.
* Is there a repository with the code required for reproducing the study's experiments and results? Adhering to open research practices would increase the impact of this work (and my assigned NeurIPS score). It would also be fairer, given that this study was enabled by the open research practices of the criticized works.

comments:
* Pie charts (figure 1) are an inefficient, outdated, and bias-prone visualization method. An empirical CDF or histogram of grid scores or a scatter plot of position error vs. grid score would convey much more information to the readers.
limitations: Limitations seem to be sufficiently addressed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper casts doubt on whether DNNs can serve as good mechanistic models to understand the brain. In particular, this paper revisits the existing literature on how task-related loss functions (e.g., path integration) of RNNs produce grid-like neural tuning, a key finding supporting the applications of DL models in neuroscience. The authors leveraged the published code in earlier work and performed large-scale hyperparameter sweeps across several dimensions, including network architectures, activation functions, optimizers, supervised targets, loss functions, and other miscellaneous aspects of model training. Surprisingly, almost all trained networks learn to perform path integration tasks very well but only a few exhibit grid-like codes. In particular, they found that networks whose neurons are assumed to have DoG receptive fields exhibit grid-like codes, but even under this regime grid-like cells only emerge for a small range of receptive fields parameter values. The authors also provide theoretical explanations from the perspective of Fourier analysis for why only a small range of parameter values of DoG tuning curve can lead to lattice cells. The authors also criticize for inconsistency between method descriptions in paper and true code implementations. Lastly, the authors investigate the intrinsic dimensionalities of RNN and how it could help predict neural activity.
strengths_and_weaknesses: Before going through this paper, I want to be transparent that I am familiar with DNN and RNN training but not an active researcher in the field of gird cells and spatial navigation. So I may misunderstand some aspects of this paper. I explicitly list the parts I don't understand here for meta-reviewers or area chairs to accurately evaluate this manuscript.

I understand the majority of this paper except that:

* The  Fourier analysis part for why some specific DoG tuning curve values can lead to grid-like codes in Section 6.
* Related to the above point, the difference between DoG and DoS in Section 8.



# Strengths

* This paper is interesting and should be published. All researchers in the field should be aware of this. I appreciate the thorough analyses on where the trained networks do or do not produce grid-like codes. Although DNNs have recently been touted as good models in neuroscience, the field needs this sober consideration and careful examination of the validity of existing results. People should be more objective and critical to papers even published in prestigious journals.
* The analyses are good. At least I cannot find any fundamental flaw in terms of analyses. 
* In Discussion, the authors also sketch out the possible ingredients that could promote existing RNNs to obtain grid-like codes, which is very constructive.


# Weakness:

My major concern is that, although we should criticize the seemingly exaggerated claims in existing literature, we should also be careful and objective to the claims in this paper. I think some claims here are going to the opposite extreme. See my questions below.

Minor
* Ref. 20 and 21 seem identical
questions: 
* Choosing appropriate hyperparameters is a fundamental aspect of machine learning and neuroscience research. For example, in the majority of instances, we say a model "state-of-the-art" only on the condition of certain hyper-parameters. Commonly, a model may no longer be the best for other hyper-parameter settings. In neuroscience, certain neural properties only hold for some task parameters (e.g., stimulus size, contrast). This is why reproducibility rises as an issue in neuroscience because readers typically have no idea of the boundary of appropriate hyper-parameters/experimental settings. In my opinion, hyper-parameter _per se_ is not an issue. The issue here is whether a paper is transparent for the hyper-parameters being chosen such that others can reproduce their results. Do you think a claim can be accepted only after it is confirmed on every possible hyper-parameters? I don't think so. I hope the authors could clarify this. 
* In section 9. The authors investigated the correlation between dimensionalities and neural prediction scores, and speculate that DL models outperform alternatives because they provide higher-dimensional bases. I don't quite get the logic. We can evaluate a model from many perspectives. One important aspect is its explanatory power for data. Higher-dimensional bases usually indicate richer representational power albeit possible suffering of overfitting. I don't think high-dimensional bases _per se_ is bad if it is well cross-validated. Also, this cannot be easily generalized to vision and language models. Vision DNN models typically have no temporal domain and they are not dynamic systems. 
* The authors seem to overly generalize their pessimism to other fields in neuroscience, e.g., vision and language. This is another form of exaggeration in the opposite direction. I do agree that this paper reports a disaster failure of RNN models in explaining grid-like codes. But critics should be reviewed case by case. In other words, unless similar analyses are done in vision or language models to illustrate their deficiency, we should not make such analogy to DNN in other fields.
* One central claim in the Discussion part is universally true and applies to all computational models, not specific to DNNs. Given non-linearity in the majority of computational models, there always exists the possibility of multiple solutions for a given loss and multiple losses for a given solution. The question is the extent of such bad interpretability. For example, trained sparse coding models contain neurons whose receptive fields bear striking resemblance to receptive fields in V1. And people conclude that sparse coding may be one of the loss functions the visual system attempts to optimize. We can certainly find other loss functions giving rise to similar results. But the sparse coding studies are still informative and valuable. 
limitations: I find no specific limitations here.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The theme of this paper is to demonstrate that directly using deep learning models as models of brain functions might lead to wrong or incomplete conclusions. By performing extensive hyper-parameter searches, authors show that the emergence of grid cell patterns by training a path integration RNN depends solely on particular design choices such as DoG firing fields rather than the path integration task itself. Such observations can hardly support conclusions made by previous papers that the emergence of grid cell patterns is due to performing the path integration task by an RNN. As a result, authors argue that neuroscientists should be more cautious to use deep learning models directly as models of brain function.
strengths_and_weaknesses: Strengths:
+ The considered question is important and remains an open and challenging one.
+ The perspective adopted by this paper is interesting, namely to demonstrate fundamental limitations of previous approaches.

Weakness:
- This paper does not figure out any potential solutions to mitigate the observed mismatch between deep learning models and brain functions.
- Many assumptions such as DoG firing fields or special regularizations are indeed clearly claimed in previous papers. So it seems natural that without these assumptions the grid cell patterns should disappear. 
- Only one example (grid cells) is used to show the no-free-lunch claim. 
questions: 1. Whether there are other examples in neuroscience that use deep learning but lead to misleading conclusions?
2. Any new ideas to correctly and cautiously use deep learning models to interpret brain functions?
3. Possible comparisons of expert-designed computational models (CANN) and deep learning models?
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper it is argued that the relevance of deep learning models for computational neuroscience should be carefully assessed by more explicitly stating the inductive biases needed to reproduce the target biological phenomena. Through extensive experiments, the authors show that recent deep learning simulations supposed to explain the emergence of grid cells in the hippocampal formation might have been driven by specific (and not always clearly justified) implementation choices, and use these findings to challenge the notion that deep learning is a panacea for building predictive models in neuroscience.
strengths_and_weaknesses: Originality: I found this paper interesting and provocative. Related research is adequately referenced (actually, there might even be too many references and the authors might consider removing the less relevant ones). At the same time, the overall novelty of the work appears limited, since the basic argument is that neuroscientists exploring computational models are affected by strong confirmation biases and seek models that display the kind of emergent behavior they are looking for. Though important, such claim is nothing new in the modeling literature.

Quality: The authors put lot of efforts to reproduce previously published models (through open access source code) and to carefully investigate the impact of hyperparameters in driving model behavior. Still, the hyperparameter sweeps were not exhaustive, and it seems that in some cases the level of detail required to reproduce the original simulations was not achieved. Though the latter issue is mostly due to the limited information provided in the original publications, it might still have affected the present findings. Moreover, it seems that in some cases the results and interpretations depend on the arbitrary choice of some threshold (e.g., grid scores in Fig. 2) over which there might not be an objective consensus in the literature.

Clarity: The paper is well-written and appropriately structured. The reference list should be double-checked and shortened (there are also some duplicates, e.g., [20] vs. [21]). Readability of the figures can be improved, for example by better organizing the spacing layout and by increasing font size.

Significance: This paper could stimulate further research work. However, in my opinion the fact that a result (e.g., emergence of lattice cells) depends on the careful choice of some model’s hyperparameters does not necessarily undermine the relevance of the model. Rather, it calls for further investigations to better establish how and why certain hyperparameters could assume such specific values. The work carried out by the authors is certainly useful (and remarkable): given that the replication crisis is extending from experimental sciences to modeling sciences, pointing out to issues with published models is a good way to better define their implicit assumptions and possible shortcomings. However, I am not particularly impressed by the finding that only certain combinations of hyperparameters lead to the simulation of relevant emergent phenomena. For example, I do not think that deep learning models should necessarily be robust to variations in the type of activation function: if only one kind of function “works”, then the modeler task should be to try explaining which biological features might justify the presence of such activation function. Overall, these findings/discussions seem more appropriate for a computational neuroscience conference / journal (and maybe one focused on hippocampus).
questions: Despite the results presented in this paper are interesting and provocative, I think that they do not undermine the key role that deep learning models are having for advancing computational (cognitive) neuroscience. The argument made by the authors should be better justified and contextualized in the broader field of computational cognitive neuroscience, not just on the specific case of entorhinal-hippocampal circuit: there are many successful stories of deep learning models that lead to important advances in neuroscience, and those should also be taken in consideration when drawing conclusions.
limitations: The authors did not explicitly mention any limitations of their research approach.
review_title: null

Meta-review: metareview: This paper examines previous deep learning models of the entorhinal-hippocampal formation that have been reported to exhibit the emergence of grid cells. The authors show that this phenomenon only occurs under very specific hyperparameter settings and with a variety of post-hoc modelling decisions, suggesting that it is not a robust finding based on the cost functions used, as suggested in the original work. The authors use this as a case study to make the point that inductive biases and consideration of biological knowledge should be considered critical ingredients for models in computational neuroscience, and that more care needs to be taken when reporting emergent phenomena in deep learning models for neuroscience.

The reviewers were generally positive about the paper, and agreed it makes a worthwhile contribution. They had some concerns related to potential over-reach in the paper's statements/claims, but after engaging with the authors, the majority of such concerns were addressed. As such, consensus was reached to accept this paper.
confidence: Certain
Decision: Accept