Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies code comprehension in the human cortex by evaluating the ability of four different functional networks (multiple-demand, language, auditory, visual) to decode different code properties. The main questions posed are:
- What is the functional selectivity of different brain regions? (for ex., syntactic or semantic code properties)
- Do different regions encode similar properties?

The main results include:
- Differential selectivity for code properties between the language and multi-demand systems.
- Dependence of decoding accuracy on token-level information
- No significant role of the auditory cortex as expected and the processing of low-level properties by the visual cortex.


strengths_and_weaknesses: Edit after rebuttal: I thank the reviewers for their response. I believe they have sufficiently answered my questions and recommend this paper for acceptance.

##################################################################################################

Strengths:

1. Clarity (of writing): the paper was very well written and easy to understand.
2. Originality & Significance: Functional selectivity in code comprehension is under explored and this paper effectively investigates this across multiple systems and properties.
3. Quality: The analyses presented here are thorough and analyze several possible sources of confounds/relationships between the systems and feature spaces. I believe that the claims made are well supported by the evidence.
4. Misc.: All the data and code will be made freely available.

Weaknesses:
1. Clarity (of visualizations & results): Since the paper conducts several analyses and uses multiple baselines, the methodological details were lost on me at several places. For example, what is the significance test being in used in panel 2 of FIg. 3 and how is this different from the token projection baseline in the interpretation? (since the paper reports significant performance over chance even in cases where the performance is no different from the token projection features). Furthermore, the authors make several key, interesting observations in the text but this is not apparent from looking at Fig. 3 alone. For example, the preference of static vs. dynamic features between MD & LS or the models that performed significantly better than token projection. I understand the severe space constraints, but I would urge the authors to try alternate visualizations for the main paper that highlight the results presented in the text.

NB: I have reviewed a previous version of this paper and am happy to see that the authors have considerably revised the analyses and manuscript.
questions: 1. Could the authors elaborate on the significance testing procedure and what “labels” refers to in line 213? Relatedly, what is the significance test used to identify significant  significant effects in Analysis 2 (multi-system partial regression analysis)? Although not identical, this is similar to variance partitioning done in encoding models and the paper could benefit from adding a citation to Deniz et al., 2019.
2. The difference in properties encoded better by MD vs. LS is very interesting! (lines 329-338) What do the authors make of these differences in terms of hypothetical roles of each system in processing code? Can this be related, perhaps, to functional specificity found in MD & LS for language or the timescales of processing? The paper would benefit from some discussion on the same.
3. The results presented in fig 3, panel 2 are also interesting! In my opinion, it is not apparent that the auditory cortex is performing above baseline solely because of token-level matches + refer to comments on clarity in previous section.
4. Is this following necessarily true? “It is currently unclear what mechanisms drive code comprehension. If we find one class of ML models (say, masked language models) to be more predictive than another (say, autoencoders), **it is reasonable to suspect that our brains optimize objectives more similar to that of masked-LMs than that of autoencoders when comprehending code.** ” I would argue that the brain and machines need not be optimizing the same objective to capture similar information. Without systematic analyses of replicated behaviors, prediction performance is not solely enough to answer such a question.
5. Was the right hemisphere left out from the language system? If so, why? (there has been a huge body of work in the past decade showing evidence of robust, reliable RH activity during language comprehension)

limitations: -NA-
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: In this paper the authors show that they can decode various properties of computer code (e.g. python programs) from brain images taken while subjects read and try to interpret those programs. More specifically, the authors separate the brain data into language and multiple-demand (MD) networks and show that (1) many code properties are represented in both networks, but (2) more information about several properties is represented in the MD network.

strengths_and_weaknesses: Strengths:
* Overall the paper is fairly clear, the analyses are well-motivated, and the results are well-explained.
* The methods are clearly and thoroughly explained (bravo!).
* The statistical analyses are of high quality and well-motivated.

Weaknesses:
* Methodologically, this paper is very simple. Oftentimes the best approach is the simplest (linear decoders DO work better than most other approaches on fMRI data), especially when it yields a compelling result. But here the results are, unfortunately, not very compelling, and I struggle to find their significance. There are some small differences between the findings here and those from earlier work on this dataset (Ivanova et al., 2020), particularly in assigning a more general role during code comprehension to the language network. However, it’s not clear how this should be interpreted in light of other knowledge about the language network and its representations.
* The most methodologically interesting analysis, in which code representations from self-supervised sequence models are compared to the brain, is underdeveloped. It seems like something much more interesting could be said about what these models represent, and how those representations relate to the MD and language networks.
questions: I would challenge the authors to go a little further afield with their analyses, which currently hew very closely to standard practices. Can anything be said about what kinds of properties the different self-supervised models represent? Do they capture similar information to the theoretically-defined properties? Is there any relationship between different properties captured by the models and representations in different brain areas? This seems like a rich and underexplored area, so it is a bit frustrating to see this paper do such shallow analyses!

limitations: None noted.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: 
This paper studies the parallels in the representation of computer code comprehension, both in neural activity as it is recorded in fMRI and in an artificial intelligence system. The paper first studies the state of the art and presents the scientific question by describing the different aspects of computer code comprehension. It then presents the different hypotheses that can be made at the level of the brain organization and how the understanding of a computer code can be assigned to certain regions of the cortex. The paper then proposes an artificial intelligence architecture, of the deep learning type, which allows building a model of code representation that is evaluated in two experiments that are those described in an existing dataset. The results show a similarity between the results obtained in the neural data and the artificial model.

strengths_and_weaknesses: 
The main strength of the paper is the originality of the study and its potential societal impact in terms of better understanding of the code that can be performed by computer scientists. The analyses seem correct, even if I am not a specialist in the field. The main weakness of the paper is to understand to what extent the comparison results obtained in figure 3 can be interpreted as a homology between the processing that is actually done in the brain and the one represented in the artificial system. (minor) Note that in figure 2, the code that is shown as an illustration does not seem valid. 

questions: 
I have a question about the paper: does a person who knows how to code in several languages (Python, C++, ...) acquire representations specific to each property of the code that are superimposed in the brain as well as in the artificial system, in the same way as people who are polyglot?

limitations: 
The authors have correctly specified the limitations (but no potentially negative societal impacts) of this paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work examines the relationship between fMRI recordings of people who read short programs and different properties and representations of the programming code. The aim of the work is to understand what properties of code are encoded by different brain systems, and to understand how similar the representations of code in the brain are to those encoded by self-supervised language models that are pretrained to encode programming code. The authors find that several program properties can be significantly decoded from 2 brain systems (the multiple demand system and the language system). They further find that representations of the programs extracted from several machine learning models of varying complexity can also be significantly related to these brain systems.
strengths_and_weaknesses: Disclaimer: I have reviewed this work previously. The authors have addressed almost all of my previous concerns and the work is much improved. I will focus on the few remaining concerns that I have in this review. I am leaning towards acceptance because of the thorough experimentation and improved discussion and motivation.

Strengths:
- The discussion of related literature is thorough
- The analyses are well motivated and the interpretation of the results is sound
- The writing is clear and concise

Weaknesses:
- There is a disconnect between the two main parts of the paper -- decoding hand-engineered code features from the brain, and decoding representations from code models from the brain, and it currently feels like it's two projects put into one manuscript. The authors motivate well why studying both is important, but it's not clear what is learned from the code models over what can be learned from the hand-engineered code features. I agree that in *theory* investigating code model representations may give us additional insight because they may contain additional information, but that is not shown in the current work, as far as I can see. An improvement would be to show that certain parts of the brain can be predicted better using the code properties than with all of the hand engineered features, via an encoding analysis, and ideally show that code models can explain additional variance in the brain recordings. This will actually show that the code models contain additional brain-relevant information over what is encoded by the hand engineered features.
questions: I think this work is a solid contribution. I do have some questions that I am curious about.
1. I'd appreciate the author's response to the Weakness I raised above.
2. L65-L74 try to motivate why the question of investigating code comprehension is of interest. One reason given is that it's poorly understood, which is fine. The other reason is difficult for me to understand -- the authors state that two previous works have found that the neural bases of code comprehension are different from those of natural language comprehension. And then they ask "do code models mimic human cognition of programs? Do language models?". It's not clear what the difference between code and language models is for the purposes of this work -- aren't all code models just language models that are trained on code rather than natural language, and don't the authors *only* investigate representations from code models and not from natural language models? It's not clear what the expectation here is. Much previous work, as discussed by the authors, shows that language models predict large part of cortex (including the language and multi-demand regions) during natural language comprehension. On the other hand, work on code comprehension shows that its neural bases are different from those of natural language comprehension. So then what did the authors expect to learn by using representations from language models to predict activity during code comprehension? Were these expectations met? Addressing this would strengthen the motivation.
3. L377-380: An intriguing discussion of how the current findings that the language system encodes some properties of code can be understood alongside the findings from previous work that the language system does not show a consistent response to code. If the language system indeed does not show consistent response to code and yet it encodes important code properties, what does this imply about these system-based analyses in general? The whole reason to look at these specific systems is that they are believed to have some function, based on work that examines their average responses. Perhaps a whole-brain analysis would be more telling of the functional organization of code comprehension w.r.t. different stimulus properties.




Minor comments:
- The language "MD/LS network decodes ..." sounds strange to me, and it's used throughout the paper. I would suggest using "encodes" or "[FEATURE] is decodable from MD/LS network"
- L313: "These findings establish the unique role of the MD system in encoding code-simulation and execution related information" -- this should be reworded because it can be misinterpreted as these properties being uniquely encoded in the MD system, which is not the case (as the authors show).
- Typo: L314 "an important of aspect of code" -> "an important aspect of code"
- Table 11 in the appendix is illegible

limitations: Limitations are sufficiently discussed
review_title: null

Meta-review: metareview: This work examines the relationship between fMRI recordings of people who read short programs and different properties and representations of the programming code. 
The aim of the work is to understand what properties of code are encoded by different brain systems, and to understand how similar the representations of code in the brain are to those encoded by self-supervised language models that are pretrained to encode programming code. 
More specifically, the authors separate the brain data into language (LS) and multiple-demand (MD) networks and show that (1) many code properties are represented in both networks, but (2) more information about several properties is represented in the MD network.

The paper presents a fairly original idea, namely that of generalizaing brain encoding of language to code.
The work presents the thorough experimentation and improved discussion and motivation: thorough discussion of the literature, good motivation of the analyses, sound interpretation of the results, with clear and concise writing.
What is less clear overall is the results, that may not bring as much insights as one would hoped. There are some small differences between the findings here and those from earlier work on this dataset (Ivanova et al., 2020), particularly in assigning a more general role during code comprehension to the language network. Yet these differences are hard to interpret in view of current knowledge.
Furthermore, the MD system is overall better at decoding every representation than LS, which does not reveal specific functional characteristics. On the other hand, novel experiments show that simple token-level models are a better match to LS, while the TF-IDF model, which is also token-level, is better correlated with the MD decoding than LS. These results hardly provide a consistent picture.

Besides, one may wonder whether the strategy of focussing on such networks (that are not as homogeneous as claimed by the authors) is really a good one.

For this reason, there is a large variance between reviewers ---and a very long discussion, which remained open. In particular, the authors are providing new results in Appendix, which complement the ones described in the original submission, but are also hard to gather with the original results.

Overall, my feeling is tha the paper opens an interesting direction, and could be accepted at NeurIPS to further feed the discussion.

confidence: Less certain
Decision: Accept