Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this work, the authors explore the relationship between 4 different brain regions (multi-demand network, language network, visual system, auditory system) and different features of program code. Specifically, they look at hidden state representations of code language models (seq2seq, CodeBERTa, CodeTransformer, XLNet), tf-idf and BoW representations of the input. For non-LM based features, they look at a code vs. sentence contrast, variable language (English vs. Japanese variable names), data types (strings vs. numerals) and control flow (for vs. if vs. no branching) To analyze relationships between the BOLD signal and stimulus features, they build linear classifiers/regressors from the BOLD activity of each system to each of the stimulus features. Models are evaluated using classification accuracy and linear correlation for regression. In the case of hidden-state features, model are evaluated using rank accuracy.

Overall, the authors find that the visual system is capable of significantly predicting several of the hand-crafted code features suggesting that these features are correlated with low-level stimulus properties like program length. While differences between MD & LS are not significant, these models successfully predict 5/6 hand-crafted features. In the code representation prediction task, the authors find that the MD, LS and Visual systems are able to rank significantly above change. However, the LS and visual systems do not beat a random token embedding baseline. Aside from CodeBERTa, the MD system is also not significantly above the random baseline.
main_review: ##########

Updated review (apologies for the delay!):

I commend the authors for engaging thoroughly with the reviews, providing new evidence and making convincing arguments. Having gone through their updated draft, rebuttals and new identification experiment (for the visual system confound), I am convinced that the MD/LS results show code processing distinct from pure visual features and that the random baseline performance in result 2 is perhaps an effect of the simple dataset, not that the regions/models only care about token-level information. To this end, I am increasing my score by 2 points.

Having said that, I agree with the other reviewers that this has been a productive discussion and the paper could benefit from more additions- particularly, stronger/cleaner results to show what information is differentially processed in MD vs. LS etc.. I am still unconvinced that the current approach is more promising in deciphering brain function than encoding (with variance partitioning). Hence I am unable to fully endorse this paper at this time.

##########

Strengths: The paper was well-written and the research question is interesting + useful. The use of publicly available data and code release is beneficial to the community.

The authors note that the visual ROI can effectively classify 4 different stimulus properties but it does not significantly predict the token count. This to me suggests that the 4 properties evaluated here are in fact strongly confounded with program length, presence of letters etc. and may not specifically correspond to code comprehension. What do the authors think about such confounds being relevant in other areas as well and its impact on the conclusions reached in the paper?

Overall, I am not convinced that classification or identification accuracy is useful in gauging what properties of code are encoded by different brain regions. It would be useful to introduce more contrasts or control for alternate exploratory variables. While the authors allude to several of the metrics being correlated (like in the static analyses and byte code), not only would it be useful to add a correlation matrix but also use a more systematic approach like variance partitioning to see what unique properties of code these feature spaces explain. (Both against each other and confounds like program length) The authors are making definitive statements about the observed performance patterns based on prior knowledge of the MD system, LS system etc. However, given the possibility of confounds and correlations between the metrics itself, I am unable to view these results as strong evidence for one mechanism over the other.

Re experiment 1:
- Why isn’t the baseline just chance probability instead of a round-about way with randomly sampled assignments? For example, it should 1/3 for control flow or 0.5 for variable language.
- There are no details on how the functional ROIs were identified: did you use specific contrasts, FreeSurfer annotations etc.? If indeed AC refers to primary AC,  perhaps the significance test being used is weak and cannot meaningfully identify signifiant effects.

Re Analysis 1:
1. Ivanova et al., 2020 effectively demonstrate that the multi-demand network uniquely responds to code. However, it is not clear to me what new findings are proposed in the current work beyond this.
2. Control flow and data type: The authors make claims about accuracy differences between MD & LS. However, these differences are not statistically signifiant by their own admission: 
> We follow this up by using t-tests to examine whether any one brain region decodes any given property more significantly than another, but find no differences between the MD system and the Language system for any properties (Table 8, Appendix E).
3. Same comment holds for the contrast in dynamic and static analyses:
> We additionally test if any brain region has a preference for a specific code property over another. For instance, is the evidence seen in Figure 4, of MD more accurately decoding dynamic analysis properties than static analysis properties, statistically significant? Likewise, does the LS predict static analysis properties significantly more accurately than the dynamic analysis properties? We do not find any significant differences (Table 9, Appendix E).

Re Experiment 2: 
- What layer of the models were used? Is it the logits or softmax output?
- Can the authors provide more details on how rank accuracy was computed and how the regression task was set up?
- It is unclear to me what scientific question this experiment is answering: Yes, different brain regions have good identification accuracy for different brain models, but isn’t this expected from a) prior work showing these regions respond to code and 2) these models capturing non-linear features of the code stimuli? What does this tell us about brain function?
- I found weak evidence for the identification accuracies presented here since several of the tests were not statistically significant and did not beat the random embedding baseline.
- Re the following: 
> The fact that MD maps to other complex models other than CodeBERTa, like CodeTransformer and seq2seq, as accurately as the Random embedding model is a strong indicator that the representations in these brain systems is strongly driven by token-level information in the input programs. 

I do not follow this claim. Do the authors mean brain systems or code models? Couldn't the same result be observed if the high-level code information was not linearly decodable as permissible by the linear regression setting here?

Other questions:
1. Since the magnitude of the predicted value is important here, I am confused by the use of linear correlation as opposed to RMSE.
2. For discrete properties, this is clearly not regression. The authors should clarify that they build classification/regression models with an L2 penalty.
3. What do the authors mean by “zero-shot” here since iiuc, the model was never actually tested on stimulus decoding following training on code representation identification?
> We show that it is possible to perform zero-shot decoding of the computer program being comprehended using a proxy representation produced by a suite of code models.
4. How were code representations extracted from XLNet and seq-2-seq? Were the models fine-tuned on code?
summary_of_the_review: While the proposition of understanding what aspects of code processing are represented in different brain regions is interesting, in its current form, I think the paper doesn’t successfully disentangle this. Several of the effects reported are not significant above chance or random baselines and further, not significantly different across brain regions. The features considered are also confounded with low-level properties of code comprehension that haven’t been controlled for. To this end, it is hard to infer what properties different systems are encoding and if there is a take away beyond prior work that identifies regions robustly responding to code.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a systematical framework to discover the relationship between the brain representations of programs and their corresponding code models. This framework helps us to understand the code properties encoded in the human brain so that we could evaluate whether ML models faithfully represent human brain representations of computer code comperhensions. 
This paper focuses on answering two questions by showing the results of related experiments on a dataset of 72 programs and 24 persons' brain recordings: First of all, the authors show that how well each of the four brain systems considered in this paper including Multiple Demand, Language, Vision, and Auditory systems encode specific code properties using a ridge regressor. Then they demonstrate another ridge regressor that can map brain representations to the corresponding learned representations by computational language models of code with different model complexity.

main_review: Major comments and questions: 
- The results of experiment 1 show the accuracy of the brain representation of code properties. It shows some correlation between brain regions and code properties. However, the t-test results show no significant differences for any brain regions having any preferences for a specific code property. This experiment has been done only on one dataset with Python code. How much is this kind of correlation reliable and pervasive in other datasets?
- As we know, Python is a high-level scripting language that might not be quite distinguishable from short sentences. Have you tried similar experiments on any other datasets or other lower-level languages such as C/C++ or even assembly? 
- Is there any improvements in the brain code representations in this work compared to previous work such as Ivanova's given that you are selecting a vector of roughly 1000 voxels for each region?
- Ivanova's paper reported some activities in the MD system during code comprehension. They showed moderate activity in Language systems corresponding to Python code comprehension and no activity regarding visual programming like ScratchJr. They also indicated that LS is responsive to Python code, while visual areas are responsive to ScratchJr. Given that, one other natural experiment to support the mapping brain representation to the code model could be to try it on the brain representations of visual programs (mostly in vision and MD systems) and the corresponding machine-learned encodings using visual recognition models such as convolutional neural nets. Have you tried such experiments on ScratchJr codes similar to what you have done for Python codes?

Minor comments:
- The explanation in the paper is fine, however, adding more diagrams of the whole procedure of the framework as well as mathematical descriptions of the models and their input and output as matrices/vectors would have helped a lot to understand the whole idea faster and easier. 
summary_of_the_review: The paper is well written, the motivations are clear, and the references are enough. Related work is adequately established the existing research in the field and compared the goal of the current research to previous work. However, the ideas of the paper are incremental and mostly applications of existing methods which have been put together to approach an interesting open problem. Moreover, usually further experiments on more datasets are required to evaluate any correlation between the code models and the brain representations in the proposed framework.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper investigates encoding of computer code in the human brain. The author build decoding models for fMRI responses to predict 1) various properties of python code and, 2) representations of python code derived from different machine learning models. The main conclusion is that the responses from the Multiple Demand system in the brain is capable of provide significance decoding performance of properties and model representations of computer code, such as runtime information.
main_review: Strengths:
This paper is very clear and well written. It investigates an interesting question of what kinds of information is decodable from brain representations of computer code. Experiments are well thought out and carefully controlled with a decent set of properties of interest and model representation results are compared with relatively extensive sets of models. 

Weakness:
I am not sure whether this paper actually provides a lot of new insights for both audiences from the neuroscience community and machine learning community. 

It is not surprising to me that the Multiple Demand system in the brain is able to provide above chance decoding performance of the selected properties of the computer code. The Multiple Demand system includes a large part of the prefrontal cortex that is generally responsible for any executive control and cognitive processing human do. It would be more interesting, for example, to run a searchlight algorithm over the brain to pin down specific locations where different properties of computer code are represented.

For experiment 2, other than model complexity, another useful control would be the information integration window (or context window) of different models. The models in comparison have very different mechanism to integrate information across the time during which a computer program is presented. Therefore it is unclear to me how the integration of information over time or length of program is going to affect the brain mapping to these representations. The mapping from brain representations to context window is also potentially helpful for machine learning researchers in designing better models for code representations. However, the analysis in the paper so far, largely limited to decoding based methods and only compared with limited number of controls, are not very informative.


summary_of_the_review: The paper has done thoughtful analysis and comparisons to investigate how the brain represent computer code and has great potential to be accepted but more in-depth analysis are needed to make it a more informative paper for both the neuroscience and/or machine learning audience.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work examines the relationship between fMRI recordings of people who read short programs and different properties and representations of the programming code. The aim of the work is to understand what properties of code are encoded by different brain systems, and to understand how similar the representations of code in the brain are to those encoded by self-supervised language models that are pretrained to encode programming code. The authors find that several program properties can be significantly decoded from 3 brain systems (the multiple demand system, the language system, and the visual system). They further find that representations of the programs extracted from several machine learning models of varying complexity can also be significantly decoded from these brain systems.



main_review: Strengths:
- code made publicly available, and the paper also uses publicly available brain data
- mostly clear exposition, with only a few places where more details are needed (see questions below)
- an in-depth and timely empirical investigation 

Weaknesses:
- limited methodological novelty
   - this work uses previously established methods for brain decoding, and does not make advances either in the methodology or in the interpretation of existing methodology. By itself, this is not a fatal flaw (a paper can have value to the community without such novelty).
- limited evidence for some of the stated conclusions
   - see Question 1 below for more detailed comments 
   - one general comment: when reading the main paper, it is not always clear what differences between brain systems or ML models are actually significant
- limited discussion
  - the investigation of the relationship between language models and the brain recordings has the potential to be very interesting, but the discussion of these results is extremely limited. The paper can benefit from a more thorough discussion of what is learned by the comparison with the language models. For example, it appears that a very simple model like BoW reaches similar decoding performance to that of the most complex model (CodeBERTa)--so then, what do we learn by the comparison with the complex model?

Questions for authors:
1. I have some concerns about the conclusions for 3 of the 4 types of experiments in Section 5.1 under Analysis 1:
   - code vs. sentences: Fig 3 shows that 3 of the brain systems (multi-demand, language, and visual) can significantly distinguish programs according to whether they were written using code or using sentences, with very similar accuracy. In conclusion, the authors state:
> The underlying operations required to solve both types
of programming problems were the same (e.g. both might have
required summing elements in a list). Yet, despite the mental
operations remaining mostly the same, the MD system is able to
discriminate between the two contrasting conditions. This supports the claims made by Liu et al. (2020) and Ivanova et al.
(2020) that the MD system is responsible for code comprehension in addition to being responsive to mentally tracing the execution flow of code snippets–a typical working memory task.

      Since the two conditions are easily distinguishable based on visual properties (as evidence by the very high significant decoding accuracy in the visual system), how can the authors make any claims about what the multi-demand or the language systems are doing based on the decoding accuracy of these two conditions?
    - variable language: Here the authors examine the decoding differences between two conditions, one in which the variable names are in English and the other where the variable names are in Japanese. The authors say that they expect a significant decoding performance in the language system. Would they not expect a significant decoding performance in the visual system? Shouldn't the most obvious difference be in the visual system, because of the systematic visual differences between English and Japanese? Fig 3 shows no significant decoding performance for any brain systems (neither the language nor the visual system). Could this be due to the way the brain data is aggregated over TRs for this analysis?
   - control flow and data type: could the differences in decoding performance for the multi-demand and language systems be due to a difference in the signal-to-noise ratio in the two systems, and how to the authors control for this possible SNR difference between brain systems?
2. Can the authors comment on why they chose to do decoding analyses over encoding analyses (i.e. predicting brain activity from representations of programs)? It seems that encoding could allow for a more in-depth analysis of the differences in what properties different regions/brain systems respond to (i.e. by comparing the weights of the learned encoding models for different brain regions/systems).
3. Can the authors comment on why only left-lateralized regions of interest were examined? Several works in natural language comprehension with naturalistic stimuli show that both the left and right hemispheres are engaged during language comprehension (Wehbe et al. 2014, Huth et al. 2016, Jain and Huth 2018,..) Do the authors expect that the mirror brain regions in the right hemisphere do not engage in the same way during programming comprehension? And if so, do the authors believe that this is a function of their short stimuli (i.e. if the stimuli were longer pieces of code, we would expect both hemispheres to be engaged).
4. One of the questions that was thoughtfully examined by the authors was how the complexity of the ML model affects its relationship with the brain recordings. The authors clarify that what they mean by complexity is the number of model parameters. Another related model property that may affect the relationship with the brain recordings is the dimensionality of the output embeddings--did the authors also examine this property (it's somewhat related to the number of parameters in the model but is not a perfect predictor, i.e. BERT and GPT-2 have the same embedding size but different numbers of parameters overall)? 
4. Clarifications needed about:
   - representations of programs extracted from the models: how exactly were they extracted? Was the whole program provided to the model at once? The authors say that the output of the encoder was used, but what about for auto-regressive models (i.e. the last state, or something else?)? 
   - brain experiment: was the program presented at once, or word-by-word? for how long? what were the subjects instructed to do?
   - brain data: assuming that each program was presented for multiple TRs, how was the data from multiple TRs handled in the decoding procedure?

summary_of_the_review: Overall, the paper investigates an interesting and timely question, but offers limited technical and empirical insights. The work can be strengthened by a more in-depth discussion about what is learned by the comparison with complex ML models, and by additional analyses that strengthen the empirical conclusions (see under Questions).
review_title: null

Meta-review: This paper aims to relate brain activity (of people reading computer
code) to properties of the computer code. They relate the found
representations to those obtained from ML computational language 
models applied to the same programs.  The paper is clearly written and
an interesting idea.

There was a lot of discussion and the author(s) updated their paper a
lot.  Program length as a potential confound was raised and 
successfully rebutted.  The extent of novelty from Ivanova et al 2020
was also discussed and successfully rebutted.  In the end, the main
issues the reviewers had were 1) that the paper had been updated
substantially since submission (and would therefore benefit from a
thorough re-review) and 2) whether the results provide enough new  
insights about the brain or about ML language models.  

To summarize, the authors spent a lot of time addressing issues in the 
rebuttal phase and the paper got a lot better with the reviewers'
suggestions, but reviewers agreed it would benefit from more work and
further review before acceptance.  I agree with this assessment.
Decision: Reject