Convergent Representations of Computer Programs in Human and Artificial Neural Networks
Authors: Shashank Srikant
Ben Lipkin
Anna A Ivanova
Evelina Fedorenko
Una-May O'Reilly
Venue: NeurIPS 2022 Accept (2022)
Keywords: Code representations
Brain representations
Cognitive neuroscience
Neuroimaging
Multivoxel pattern analysis
Representation decoding analysis
Representation similarity analysis
fMRI analysis
ML for PL/SE
ML4code

Abstract: What aspects of computer programs are represented by the human brain during comprehension? We leverage brain recordings derived from functional magnetic resonance imaging (fMRI) studies of programmers comprehending Python code to evaluate the properties and code-related information encoded in the neural signal. We first evaluate a selection of static and dynamic code properties, such as abstract syntax tree (AST)-related and runtime-related metrics. Then, to learn whether brain representations encode fine-grained information about computer programs, we train a probe to align brain recordings with representations learned by a suite of ML models. We find that both the Multiple Demand and Language systems--brain systems which are responsible for very different cognitive tasks, encode specific code properties and uniquely align with machine learned representations of code. These findings suggest at least two distinct neural mechanisms mediating computer program comprehension and evaluation, prompting the design of code model objectives that go beyond static language modeling.
We make all the corresponding code, data, and analysis publicly available at https://github.com/ALFA-group/code-representations-ml-brain

Convergent Representations of Computer Programs in Human and Artificial Neural Networks
Shashank Srikant
Benjamin Lipkin
Anna A. Ivanova 12,33
Evelina Fedorenko
Una-May O'Reilly
* Equal contribution BCS, MIT
CSAIL, MIT
3 McGovern Institute for Brain Research MIT-IBM Watson Al Lab (shash, lipkinb, annaiv, evelina9) Gmit t edu, unamay0csail mit..du
Abstract
What aspects of computer programs are represented by the human brain during comprehension? We investigate this question by analyzing brain recordings de- rived from functional magnetic resonance imaging (fMRI) studies of programmers comprehending Python code. We first evaluate selection of static and dynamic code properties, such abstract syntax tree -rSat-related and runtime-relane metrics and study how they relate to neural brain signals. Then, to learn whether brain representations encode fine-grained information about computer programs, we train a probe to align brain recordings with representations learned by a suite MI. models trained on code. We find that both the Multiple Demand and Lan- guage systems-brain systems which responsible for very different cognitive tasks, encode specific code properties and uniquely align with machine learned representations of code. These findings suggest at least two distinct neural mech- anisms mediating computer program comprehension and evaluation, prompting the design of code model objectives that go beyond static language modeling. We make all the corresponding code, data, and analysis publicly available at https: / /tt///itttt /ggggtthbbb.. ctcchuu..omm//e cwm ALFA group/ represerrrtttttttiiio mllbbain
Introduction
Computer code comprehension is a complex task which recruits multiple cognitive processes from syntactic parsing to mentally simulating programs. Despite the prevalence of this task, the representa- tions of code processed the human brain during code comprehension remain uninvestigated. Is it possible that common code properties and program semantics are faithfully represented in brain activity patterns when code is read and evaluated? A few prior works have used data derived from functional magnetic resonance imaging (fMRI) and glectroencephalographe (EEG) locate physical regions in the brain involved code-related activities like code comprehension and debugging (Siegmund al., 2017, Floyd al., 2017, Peitek al., 2018, Castelhano et al., 2019, Ivanova et al., 2020, Liu et al., 2020, Ikutani et al., 2021, Peitek et al., 2021], code writing [Krueger et al., 2020, Karas et al., 2021], and data structure manipulation [Huang et al., 2019]. These studies have helped determine whether code-related activities join other activities supported by those brain regions, such working memory (processed by of brain regions known the Multiple Demand system) or language processing (processed by the Language system-anothhr set brain regions). While these results improve understanding of the brain regions involved in code comprehension, it still remains unclear what specific code-related information these regions encode. For example, does the response in a region seen during code comprehension encode specific syntactic or semantic code properties? Do responses from multiple brain regions correspond to the same set of properties? Or, are different code properties encoded in different regions?
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

A possible approach and its limitation. One way to learn what information is encoded in the brain to decode a code property of interest from recordings of brain signals when reading code (through fMRI or FEG). Being able to decode the property accurately from a specific region of the brain establishes that information related to that code property is faithfully represented in that brain region. A question central to such a decoding analysis is the choice of the target code property-whal code properties should be investigated? We can hand-select a set of fundamental properties of code and test if they can be decoded. While helpful, such set will not preclude other, more complex aspects of code possibly being encoded.
ML models of code as tool to reverse engineer what is encoded. To address the limited scope of hannd-elected properties, we look to machine learning (ML) models trained on code. Dubbed code models, they are trained on large corpora of code. in an unsupervised manner, to learn ML model representations of computer programs. Code models are increasingly being used in software engineering workflows [Allamanis et al., 2018a], and have been shown to perform well on tasks like code summarization [Alon et al., 2019], detecting variable misuse [Bichsel et al., 2016], and more recently, code auto-completion [Chen et al., 2021]. These representations have been shown to encode and describe complex code properties [Bichsel et al., 2016, Allamanis et al., 2018b, Srikant et al., 2021]. Successfully decoding these code model representations from brain activity data then allows us to probe whether complex code properties are also encoded in the brain.
Besides serving as a tool to reverse engineer what is encoded in different brain regions, another distinct advantage of decoding code model representations its potential to serve as a tool to reverse engineer our cognitive processes. It is currently unclear what mechanisms drive code comprehension. If we find one class of MI. models (say, masked language models) to be more predictive than another (say, autoencoders), it is reasonable to suspect that our brains optimize objectives more similar to that of masked-LMs than that of autoencoders when comprehending code. As a corollary, a poor correspondence between the information encoded by our brains and ML models suggests the possibility of unexplored neural architectures and objectives which may better model our cognition, which in turn may outperform extant ML models. Yamins et al. [2013] first showed how information encoded in our visual system resembles what convolutional neural networks learn when trained to recognize images. We attempt to establish similar correspondence between code models and the human brain in comprehending code.
Why code comprehension? We focus on code comprehension in this work because very little of this important skill has been analyzed from a cognitive neuroscience perspective while steady advances are being made in training ML models to understand code and increase programmer productivity [Fedorenko et al., 2019, Hellendoorn and Sawant, 2021]. Extant ML models for understanding programming are direct adoptions of the state-of-the-art in language processing research. However, recent works in neuroimaging like Liu et al. [2020] and Ivanova et al. [2020] suggest that code comprehension does not share the same neural bases as natural language comprehension. Do code models then mimic human cogmition of programs? If not, can we consider other model architectures and training objectives which are directly inspired by results from neuroscience? Our work provides early considerations towards these directions.
Further, characterizing the representations of different code properties in the human brain can inform us about the nature of human algorithmic problem solving more generally. We can possibly characterize signatures of brain activity corresponding to fundamental logic operators like iterative reasoning, conditional reasoning, retrieving calculations that were previously computed, etc. seen in individuals when solving carefully designed code comprehension tasks. Such signatures will then allow us to characterize other tasks which involve similar logic operators, but which are not naturally described as computer programs e.g. general logic reasoning, problem solving, and more generally, our ability to employ abstractions and form concepts [Rule et al., 2020].
Our setup. We introduce framework to evaluate the code properties encoded in human brain repre- sentations of code by analyzing fMRI recordings of programmers comprehending Python programs. We present two means of proceeding one, probing of brain region representations for specific code properties, and two, analyzing the mapping of these representations onto various code models with differing model complexity. We utilize the publicly available dataset from Ivanova et al. [2020] for all our analyses as offers high quality, granular brain response data on code comprehension stimuli controlled across multiple code properties. We train affine models brain representations to predict hand-scle sel--see code properties which express syntactic and semantic behavior of programs. Similarly,
2

---

we predict code representations obtained from suite of code models. We investigate the effects that brain regions, the nature of code properties, and the complexity of the code models have on the accuracies of these prediction tasks.
Key findings. We explore two questions: (1) Do brain systems encode specific code properties? Are there differences in how well each brain system encodes each code property? (2) Do brain systems encode more complex properties of code, derived from the representations of code models? To answer these, we perform three critical comparisons: between code properties and code models, code models and brain systems, and brain systems and code models. We further provide preliminary analysis into the relationship between code models and the brain in the context of what properties code model representations encode. We find the Multiple Demand system and the Language system consistently encode both hand-selected code properties and data-derived code model representations, using features which cannot be explained by only low-level visual characteristics of the code. Within this set, the Multiple Demand system most effectively decodes runtime properties of code like the number of steps involved in a code's s execution, while the Language system decodes syntax-relan properties like the number of tokens in a program and the control structures present in it. These results improve our understanding of the functional organization of the two brain systems-MD and the Language systems-thie that we study, and provide initial evidence for incorporating the roles of these two distinct brain systems in the design of training objectives of code models.
We provide an ooen-source framework to replicate our experiments, and we release our data and anal- ysis publicly. Link https: h //gttth.t../.../// com/ALFA- group/ ctt///odderrreeeeee weweee-reeeeeeeeeeeeeeeeee wwwprseenttiiin--eeeeeee malesmmm--a brain. This should enable authors from other neuroimaging studies or code model developers to collaborate and analyze data across these works, which will also help amortize the high costs of carrying out such experiments.
2
Related Work
Of the prior works that have investigated the neural bases of programming through fMRI and EEG techniques [Siegmund et al., 2017, Floyd et al., 2017, Peitek al., 2018, Castelhano et al., 2019, Huang et al., 2019, Krueger et al., 2020, Ivanova et al., 2020, Liu et al., 2020, Ikutani et al., 2021, Peitek et al., 2021] and through behavioral studies [Prat et al., 2020, Casalnuovo et al., 2020, Crichton et al., 2021], I], the following probe brain recordings for program properties encoded in them.
Floyd et al. [2017] learn linear model to successfully classify whether an observed brain activity corresponds to reading code or reading text. Ikutani et al. [2021] study expert programmers and show that it is possible to classify code into the four problem categories-math, search, sort, and string from the brain activations corresponding to the code. Similarly, Liu et al. [2020] classify whether a brain signal corresponds to code implementing an if condition not. Peitek et al. [2021] analyze correlations between brain recordings of participants reading code and a set of code complexity metrics.
In testing for code properties, our work uses a similar methodology linear model trained on fMRI data), but we evaluate larger set of static and dynamic code properties, often reflecting key programming constructs like control flow. Further, we perform these tests in the brain regions identified by Liu et al. [2020] and Ivanova et al. [2020] as being responsive specifically to code comprehension, offering finer insight into the content of these specific regions' representations. In addition, we study representations generated by suite of MI. models with varying complexity and compare those learned representations to brain representations.
Brain representations have also been studied in domains like natural language, vision, and motor control. Among related works in natural language, a domain that resembles programming languages, Mitchell et al. [2008]. Pallier et al. [201 1], Brennan and Pylkkänen [2017], Jain and Huth [2018]. Gauthier and Levy [2019]. Schwartz et al. [2019], Wang et al. [2020], Schrimpf et al. [2021], Cao et al. [2021]] Caucheteux et al. [2021], Toneva and Wehbe [2019] have studied brain representations of words and sentences by relating them to representations produced by language models. While the broader tools we use to investigate these representations, like multi-voxel pattern analysis (MVPA), are similar to some of these prior works, our focus is on properties specific to code and not natural language.
3

---

3 Background
We provide brief background on fMRI signals as a proxy for brain representations and describe the brain systems that we probe in this work.
Measuring brain activity with fMRL. Functional magnetic resonance imaging (IMRI) is a brain imaging technique used to measure brain activity in specific brain regions. When a brain region is active, blood flows into the region to aid its processing. An MRI machine measures this change in blood flow, and reports BOLD (blood oxygen level dependent) values sampled at the machine's 's frequency [Glover, 201 I, usually 2 seconds]. The smallest unit of brain tissue for which BOLD signal is recorded is called voxel (an equivalent of a 3D pixel); it comprises several cubic millimeters of brain tissue. For our analyses, we select subsets of voxels belonging to specific brain systems. Following common practice, the parameters of a general linear model, fit to time-varying BOLD values, are used as a measure of the overall activation in each voxel in response given input. is the values of these parameters that, in concordance with common practices in the neuroscience community, can be accessed as brain representations.
Brain systems. A system of brain regions can span different areas of the brain but behaves unit, showing similar patterns of engagement across given cognitive task. We probe the following sys- tems in our work: (a) Multiple Demand (MD) sys- tem: this system of regions known engage in cognitively demanding, domain agnostic tasks like problem solving, logic, and spatial memory tasks. Liu et al. [2020] and Ivanova et al. [2020] reported that this system is active during code comprehension. (b) Language system (LS): this system responds dur- ing language production and comprehension across modalities (speech, text) and languages (across 11 language families, including American sign language). (c) Visual system (VS): these regions respond primarily to visual inputs. (d) Auditory system: these regions respond primarily to auditory inputs.
holistic
Left homisphere
Right hemisphere
Left hemisphere
whow
Muttiple Demand (MD) Bystem
Language System
Figure The approximate locations of MD and the Language systems in the human brain.
The regions depicted are used as a starting point to functionally localize these systems in individual participants.
We probe the Visual system and Auditory system since they are involved in general perception. While we do not expect activity in Auditory system, we expect the Visual system to reflect low-level visual properties of the code (e.g. code length and indentation reflect code-related properties. See Appendix A for detailed description of these brain systems along with relevant references.
4 Brain and Model Representations
We describe in this section the method we follow to gather representations of code in the brain (Section 4.1,, evaluate the different code properties they encode (Section 4.2), and how we compare brain representations to those generated by ML models (Section 4.3).
4.1 Brain representations and decoding
We provide summary of how we process activation signals in the brain elicited by code comprehen- sion, to probe whether they encode any specific code properties. We provide details in Appendix B.
Dataset. We use the publicly available brain recordings released as part of the study by Ivanova et al. [2020] (MIT license). It contains brain recordings of 24 participants, each of whom gave consent and is not personally identifiable according to Ivanova et al. [2020] protocol requirements, reading 72 programs from a set of 108 unique Python programs. The 72 programs were presented in 12 blocks of 6 programs each. These programs were 3- 10 lines in length and contained simple Python constructs, such as lists, for loops and if statements. A whole program was presented at once, and the task required participants to read the code and mentally compute the expected output, press button when done, and select one of four choices presented to them which matched their calculated output.
From dataset to brain representations. The original dataset contains 3D images of the brain of each participant. Each voxel value these images an estimate of the response strength in this voxel when particular code (or sentence) problem is presented. To determine which brain systems contain

---

What code properties do brain representations encode?
Static, dynamic analysis of code
Code properties
RNX
RNXI
Experiment 1
Human brain
Neural representation of code
How similar are machine representations and brain representations?
Code
RNDDD
B NxDM
Code language model
Machine representation of code
Experiment 2
Figure 2: Overview. The goal of this work is to relate brain representations of code to (1) specific code properties and (2) representations of code produced by language models trained on code. In Experiment I, we predict the different static and dynamic analysis metrics from the brain MRI recordings (each of dimension DB) of 24 human subjects reading 72 unique Python programs (N) by training separate linear models for each subject and metric. In Experiment 2, we learn affine maps from brain representations the corresponding representations generated by code language models (each of dimension DM) on these 72 programs.
information about particular code properties, we focus our analyses to four systems MD, Language, Visual, and Auditory (Section 3). vector of voxels' activation values in each brain system is then taken to constitute that system's s representation of computer program and serves as an input to all our analyses. See Appendix B for details on data processing and voxel selection.
Analyzing brain representations of code. We probe brain representations from each participant separately. We do not average data across participants since the regions which respond any task (comprehending code in our case) need not align anatomically. For each of two experiments decoding different code properties, and mapping gegression/classficaati models which take as imput normalized brain representations per participant. We hence learn 24 different regression models, for each code property or code model (one per participant), and then report the mean performance of these models across participants. This procedure also referred as multi-voxel pattern analysis (MVPA) [Norman al., 2006]. Linear models are conventionally preferred for probes into brain representations since there has been evidence supporting the idea that other brain areas linearly map information from such brain representations [Kamitani and Tong, 2005, Kriegeskorte, 2011]. We choose linear model primarily control for over-fitting in light of the relatively small dataset.
code model representations- -we train ridge
A remark on data scarcity. For each participant, we train linear regression/clasification model with L2 12-reuularization for on unique cross- voss--vlldtted leave-one-run-our folds of the 72 programs they attempted (or 48 programs when sentences are removed). On the order of 1000 voxels were selected from each brain region responding to any given program per participant, thus resulting in a feature set of dimensions 72 x 1000. Such dimensions allow for strong statistical tests to support the robustness of the predictions made. As a baseline for the model predictions, we use the accuracy of null permutation distribution generated from sampling 1000 random assignments of the labels.
4.2 Code properties
We attempt decode the following code properties from brain representations. See Appendix B for details. (a) Code vs. sentences: classify whether an input stimulus is code or an English sentences describing code (referred as sentences). See Figure B, Appendix for an example. (b) Variable language: classify whether program contains variable names written in English or Japanese (written in English characters). (c) Control flow: predict whether program contains loop (for loop), a branch (if condition), or has sequential instructions.
(d) Data type: predict whether (e) Static analysis: predict static properties of program like token count: number of tokens in the program, node count: number of AST nodes, cyclomatic complexity, and Halstead difficulty. The latter two metrics have been used by software engineering practitioners to quantify the complexity of code, and to quantify the difficulty a human
a program contains string or numeric operations.
5

---

would experience when comprehending code respectively. We defer predicting other advanced static analysis metries such as tracking abstract interpretation joins, data flow analysis-related metries, etc. to future work. (f) Dynamic analysis: predict information about a code's execution behavior like runtime steps: number of instructions executed in the program, and bytecode ops: number of bytecode operations executed in running the program.
Program length as potential confound. Since the properties we examine can also potentially be differentiated using program length and other low-level code features, it is a potential confound in our experiments. We measured the corer-corsslations of these properties, and their correlation to the number of tokens in the program (program length; see Appendix H). We expectedly found the four static analysis properties to be highly correlated to each other and to bytecode ops. We hence use one representative metric each from the two categories of properties for the rest of our analysis token count for static analysis, and runtime steps for dynamic analysis. Importantly, the other properties we examine cannot be explained by program length alone, and therefore program length is not confound in our experiments.
Mapping to code properties. The brain representations (Section 4.1) are mapped to each of the code properties by training a ridge regression (for static analysis and dynamic analysis properties; continuous values) or classification model each for every participant-rooeerly pair. To evaluate model performance, we use classification accuracy when the predicted values are categorical (e.g. string vs. numeric data types), and the Pearson correlation coefficient when the predicted values are continuous (e.g. number of runtime steps). We choose Pearson correlation over RMSE, the canonical distance metric for continuous values, for its simplicity and interpretability. See Appendix J for the RMSE results, which lead to the same conclusions as the correlation results. When testing for the significance of these predictions, we perform false discovery rate (FDR) correction for the number of brain systems tested and the number of properties tested. See Appendix for a detailed description of model hyper-parametent valsdvani settings, and significance testing.
4.3 Model representations and decoding.
We evaluate a bench of unsupervised language models, spanning from count-based language models transformer neural networks [Vaswani al., 2017]. These models were all trained on large (~ IM programs) Python datasets [Husain et al., 2019, Puri et al., 2021]. We use the output of the trained encoders (raw logits) in each of the neural network models as representations of the code input to the model. We vary the general complexity of these models to test whether that variation is meaningful in establishing the quality brain to model fits. Model complexity here is the number of model's learnable parameters. We evaluate the following models, ordered by their increasing model complexity: simple frequency-based language woddels-bg-oo-wweee
TF-IDF; auto-encoder based unsupervised models seq2seq [Sutskever et al., 2014], Code Transformer [Zügner et al., 2021] I,, CodeBERT [Feng et al., 2020], CodcBERTa [HuggingFace, 2020]; auto-regressive models with similar model Cimplexxty-XLNe [Yang et al., 2019], CodeGPT [Microsoft, 2021].
Baseline: Token projection model. We compare the results of the above models against an aggressive baseline (relative to the null-distribution labeling baseline), a token projection model provided by using unique couussan--distibuun
random vectors for the token embeddings in a vocabulary, and returning the sum of these token embeddings across program. The resultant embedding is not transformed by any model or any weights-it instead serves as a proxy for the tokens that appear in the program. The results of this baseline model should be interpreted as the level of performance achievable from the presence of tokens alone with no semantic or syntactic information.
Mapping to code model representations. The brain representations (Section 4.1) are mapped to code representations by training another set of ridge regression models to learn an affine map, and a ranked accuracy metric is used to compare outputs. Ranked accuracy scores are commonly used in information retrieval where several elements in range are similar to the correct one. In our case, the top-ranked prediction by the linear model indicates the closest fit (Euclidean distance) to the code model's 's representation. When reporting result significance, we perform false discovery rate (FDR) correction for the number of brain systems and the number of models. See Appendix B for a detailed discussion of the implemented code models and the corresponding metrics. All experiments this work were run single 8-core laptop in under hour following setup.

---

MD
Language Visual
Ciessificatio Accuracy (SS)
80
Auditorry
Pewrson Correlation tn
Ranked Accuracy (Ng
60-
60
0.2
55-
40
50-
20
Code ix. Sentenca
variable Language
Data Type
Contral Plase
Static Analris
Dyramic Analysio
Zaken Pojectio
Godellatt
Code GadrRaanfomer
Costelterr
Gesterary
Kitet
Sea2
Bag vibrns
Code Properties
Code Model
Figure 3: Affine models are learned on brain representations to predict each the code properties described in Section 4.2, and a collection of code models described in 4.3, for each of the 24 participants. The mean decoding score across subjects is shown here, and error bars reflect the 95% confidence interval of the mean subject score. A solid line on each bar presents the empirical baseline for a null permutation distribution on shuffled labels. All decoding scores were compared to this permuted null distribution using a one-sample z-test, and the significance threshold was defined at < 0,001; significant results are denoted with a *, marked at the base of the bars. Additionally, capped lines denote selected significant paired t-tests (p < 0.05; FDR).
for the number of tests in each panel (FDR). Statistically
Experiments & Results
Our experiments address two research questions:
Experiment How well do the different brain systems encode specific code properties? Do they encode the same properties?
Experiment 2. Do brain systems encode additional code properties represented by computational language models of code?
5.1 Experiment 1 How well do the different brain systems encode specific code properties? Do they encode the same properties?
We analyze the classification models and regressions trained on brain representations to predict each of the code properties described Section 4.2. The results our analyses are summarized in Figure 3. The classification and regression tests are marked the x-axis of the left and right subplots respectively: the classification accuracy or Pearson correlation for each of the tasks marked on the y-axes. We plot dynamic and static properties separately from the others because their basclines different due to a difference in the similarity metric (classification accuracy vs. Pearson correlation). The basclines for the categorical code properties differ from each other due to variation in the number of target classes.
Auditory and Visual systems. The Auditory system and the Visual system serve as negative and baseline controls for the other systems. Since our code comprehension task is visual, we do not decode any meaningful information between programs from Auditory system, although we do observe decoding of code vs sentences, perhaps explained by previous work showing auditory cortex activation during silent sentence reading [Perrone- [Perrone-Bertoloti et al., 2012]. The Visual system serves as a bascline for low-level visual features of the code (the layout and indentation of the code, the presence of letters and alphabets in the programs, etc.). In Analysis 2, we show that the MD and the Language systems encode more than such visual features. The MD and the Language systems yield the following observations.
How accurately are different properties predicted by MD and LS? The MD and LS together decode all the properties well above chance barring variable language. The variable language finding is consistent with Ivanova al. [2020], who show a lack of any significant difference in the aggregate neural activity between English and Japanese variables hames-variale seem to not be encoded any differently in the context of programs we study. To analyze the other results, we use paired two-sample t-tests (p- < 0.05; FDR-correcte) and examine whether for
Analysis 1
names
7

---

given property, any one brain system decodes it significantly more effectively than another (details in Table 8, Appendix E). We find that the MD system decodes the dynamic analysis property better than the Language system. We additionally test if any brain system has preference for a specific code property over another. We find that the MD decodes the dynamic analysis property significantly better than the static analysis property. These findings establish the role of the MD system in encoding code-simulation and execution related information important of aspect of code comprehension.
Analysis 2 Multi-system partial regression analysis. The decoding performance of the VS is comparable to that of the MD and LS (Figure 3). To assess the possibility that all three systems MD, LS, and VS encode the same properties (all potentially related to low level program features), we employ multi-system partial regression analysis. For each brain system, MD and LS (S.), we train two models-one one which decodes from VS, and another which decodes from Ss+ If the difference in the prediction accuracies between the two models is significant, we conclude that S; encodes at least some information which is orthogonal to the information encoded by the Visual system. This method is similar to a variance-partinn partitiooiin analysis which is often employed in encoding models, e.g. Deniz et al. [2019]. For all core properties, control flow, data type, dynamic analysis, and static analysis, we find the MD to encode information orthogonal to the VS. For control flow and static analysis, the LS also encodes information orthogonal to the VS. This suggests that low-level code properties are insufficient to explain the key results from Experiment 1. Other combinations in the regression model reveal that the MD encodes information orthogonal to the LS when predicting code vs. sentences and dynamic analysis. Detailed results are tabulated in Table 15, Appendix F.
Key learnings from Experiment 5.1. All core code properties are decodable from the representations of high-level brain systems, and this information is beyond that which can be explained from low-level visual information alone. Although no property is exclusively encoded in any one brain system, the MD system significantly encodes dynamic analysis- encodes, and more than static analysis properties. Similarly, we find evidence for the LS to also significantly encode static analysis and control flow related properties. These are new results on the nature of code properties different brain systems seem to process and encode. To explore properties which may not be specified by the set we investigate in this experiment, in the following section, we leverage code models as sypothesis-free proxy representations for code syntax and semantics, and see if any one system preferentially encodes a code model.
properties-more than what the LS
5.2 Experiment 2 computational language models of code?
Do brain systems encode additional code properties encoded by
We train ridge regression models with brain representations of programs from specific brain systems to predict code model representations of the same programs. The set of results from this experiment are summarized in Figure 3.
Auditory and Visual systems. Similar to Experiment 1, these systems perform as expected, with the Auditory system exhibiting the lowest decoding performance across code models, and the Visual system acting as a proxy for low-level information (Table 10, Appendix E).
How well do brain representations in MD and LS map to code model representa- tions? We find that the MD and LS map to all the models in our suite significantly more accurately than the null permutation baseline (Figure 3). Further, we find that the MD ranked accuracy is higher than LS for CodeBERTa, two-sample 1-tests; p < 0.05; FDR-corrected. See Table 10, Appendix E).
Analysis 1-
seq2seq and TF-IDF (differences evaluated using
Analysis 2 The effect of model complexity on decoding to code models. We investigate the impact model complexity has on the performance of the mapping between brain and code representations.
We compare each of these code models against the Token Projection baseline model, which only encodes the presence of specific tokens, with no contextual or distributional information. We find that the MD system maps to all the models more accurately than the Token Projection, but this not observed for the LS. In set of paired two-tailed t-tests (p < 0.05; FDR-corrected we find that the MD maps to three models: CodeBERTa, Transfrrsformmrr and bag-of-word significantly more accurately than to the Token Projection model (Table 11, Appendix E). Curiously, since all but 3 of these mappings do not significantly surpass the model which can be explained using only token- level information, these data suggest that the brain signals we access primarily encode token-level
8

---

information. To investigate this further, we analyze the correspondence between brain representations and code models in the context of the code properties they encode.
Analysis 3 Code model and brain representations in the context of code properties, We first evaluate whether the different code properties we investigate in this work be decoded from code model representations. We find that all the code properties are strongly encoded in all models (Table 4, Appendix C.2). Since we have computed the mapping accuracies from code models + code properties (mentioned above), and from brain representations + code properties (Tables & 2, Appendix C.1), we compute how well brain representation decoding results map to code model decoding results using Spearman rank correlation (details in Appendix G). We find two clusters in decoding performance across the code properties, each reflecting a distinct computational motif (Table 17, Appendix G). We see perfect rank correspondence between the 2-scores of the decoding results for the MD and three transformer architectures: CodeBERRa,, CodeBERT,, and Codeeraansformer, and between LS and two boken-based models: bag-of-words and Token Projection.
Key learnings from Experiment 5.2. The MD decodes representations from four models of code significantly more accurately than LS, providing evidence that some aspects of code are more faithfully represented in the MD. follow up partial regression analysis, as in 5.1, reveals that for most models, MD encodes information orthogonal to the LS, and each system encodes information above low-level VS (Table 16, Appendix F).
Analyzing the correspondence between code models and brain representations based on the code properties they each encode reveals that the MD) system maps preferentially complex code models, encoding more than just token-level information. We discuss these results further in the following section.
6 Discussion
Through this study, we learn what computer program-related information can be decoded from the brain, and which brain systems primarily encode that information.
Brain representations and code properties. We show that the MD system preferentially encodes dynamic maalysis-related properties when compared to other brain systems and other properties. Further, we find the Language system encodes syntax-related properties like control flow and static analysis. These findings are complementary to the results from Liu al. [2020] and Ivanova et al. 22020]-they show how the MD system is recruited in both mentally simulating code and comprehending code. They however do not find any consistent response in the Language system to either code simulation or comprehension. Our results show that despite not exhibiting significant average responses, these systems do encode code-specifie properties, improving our understanding of these brain systems' functional organization.
Brain representations and code model representations. Another key contribution of our work, from Experiment 2, is demonstrating that it is possible to map brain representations to representations learned by code models. In particular, we observe encoding of the properties represented by code models in the MD and LS, with 4 models more accurately mapped from MD. This is particularly noteworthy since these models, which are trained source code symbols, can be mapped more faithfully from the representations of a network implicated in problem-solvine than one associated with composition in languages.
We also considered model complexity as a relevant feature, and note that the MD and LS map to combination of token projection embeddings almost as well as to complex models like XL.Net. One plausible explanation for this surprising result is that the program stimuli are simple enough to allow the different properties evaluated in work (control flow, data type, etc.) to be discerned from token level information alone (as validated in Experiment 1), which is likely why the Token Projection model also predicts these properties very well (Table 4, Appendix C.2). Taken together, these data suggest that the information being decoded from brain activations in these two regions is driven at least by the information conveyed by tokens in the programs. This finding is notably consistent with work in the field of natural language which show swaths of cortex are predicted primarily by the token-level properties of sentence stimuli [Toneva et al., 2022].
While we see that information encoded in both the MD and the Language systems are driven by token-level information, clearer trend emerges when evaluating brain-code model mapping in the

---

context of specific code properties. We find that the MD shows decoding performance consistent with complex model architectures, suggesting that it may encode more than just ooken-related information, as complex code models capture contextual information as well. This is new result, which along with Experiment 1, suggest distinct roles for the L.S (purely token-level) and MD (more complex interactions) with respect to computer code comprehension and execution.
Way forward. Our findings have the potential improve our understanding of the organization of the human brain, which can in turn lead to the design of better code models. In computer vision, results by Tschopp et al. [2018]. Schrimpf et al. [2020a] show how deep network architectures that mimic the visual system exhibit superior image classification rates on image recognition tasks. Our findings prompt yet another reconsideration of the current design of ML models of code. Extant code model architectures do not explicitly model the Multiple Demand system in any way-they only model syntactic information and infer dependency information from program ASTs. Taking inspiration from the role of the Multiple Demand system we identified in this work, modeling dynamic runtime information as well as static code structure should be explored. See Srikant and O' O'Reilly [2021] for a related discussion.
We also provide initial results supporting the ability to decode specific and fundamental code-related primitives like control flow information. This sets us up to study other complex human cognitive processes which involve such primitives, but which are not naturally described in terms of code, like general problem solving, employing hierarchical and complex decision making strategies, etc. Such a study also promises to support the recent proposal of a child as a hacker Rule et al. [2020].
Our work also promises to enhance code prosthetics-artfficaal interfaces that can help the physically challenged engage with programming environments. Such systems generally rely on designing and constructing brain decoders-models that convert brain activity to electrical impulses modulating external devices, which remains an open challenge. See the discussion in Nuyujukian et al. [2018] and Andersen et al. [2019] for details.
The average program in any software project exhibits non-trivial control and data de- pendencies, object manipulation, function calls, types, and state changes. However, the programming tasks in Ivanova et al. [2020] short snippets of procedural code with limited program properties. Responses to longer programs, with more complex properties and across multiple languages, should be studied larger number of participants in the future, in order to build on the trends provided by our work. Further, while there are multiple equally important aspects to programming like designing solutions, selecting appropriate data structures, and writing programs, here we have chosen study only single specific activity-code comprehension. Future work should explore these other aspects. Our results do not allow features of code that are used to discriminate between code properties and code models, so future work might consider an encoding analysis. Finally, as with all neural decoding analyses, extracting information from the mental states of participants should be done with caution to ensure it is not used for any exploitative purpose.
Limitations.
infer whether the MD and LS are driven by the same underlying
Acknowledgement. This research was partially supported by NSF grant 1744809. SS and UMO'R 'R were supported by a grant from the MIT-IBM Watson AI Lab. Al was supported by a postdoctoral fellowship from the Quest Initiative. EF was additionally supported by NIH grants DC016607, DC016950 and NS121471, the Quest Initiative, and research funds from the McGovern Institute for Brain Research, the Brain and Cognitive Sciences department, and the Simons Center for the Social Brain. We thank our anonymous reviewers for their thorough and helpful feedback.
10

---

References
Janet Siegmund, Norman Peitek, Chris Parnin, Sven Apel, Johannes Hofmeister. Christian Kästtner Andrew Begel, Anja Bethmann, and André Brechmann. Mcasuring neural efficiency of program comprehension. In Proceedings of the 2017 1 Ithh Joint Meeting on Foundations of Software Engineering, pages 140-150, 2017.
Benjamin Floyd, Tyler Santander, and Westley Weimer. Decoding representation code in the brain: An fmri study of code review and expertise. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE), pages 175-186. IEEE, 2017.
Norman Peitek, Janet Siegmund, Chris Parnin, Sven Apel, Johannes C Hofmeister, and André Brechmann. Simultaneous measurement of program comprehension with fmri and eye tracking: A case study. In Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pages 1- 10, 2018.
Joao Castelhano, Isabel C Duarte, Carlos Ferreira, Joao Duraes, Henrique Madeira, and Miguel Castelo-Branco. The role the imsula intuitive expert bug detection computer code: an firi study. Brain imaging and behavior, 13(3):623-377, 2019.
Anna Ivanova, Shashank Srikant, Yotaro Sucoka, Hope H Kean, Riva Dhamala, Una-May O' O'Reilly, Marina Bers, and Evelina Fedorenko. Comprehension of computer code relies primarily on domain-general executive brain regions. Elife. 9:e58906, 2020.
Yun-Fei Liu, Judy Kim, Colin Wilson, and Marina Bedny. Computer code comprehension shares neural resources with formal logical inference in the fronto-pane fronto-parietal network. Elife, 9:e59340, 2020.
Yoshiharu Ikutani, Takatomi Kubo, Satoshi Nishida, Hideaki Hata, Kenichi Matsumoto. Kazushi Ikeda, and Shinji Nishimoto. Expert programmers have fine-tuned cortical representations of source code. Eneuro, 8(1), 2021.
Norman Peitek, Sven Apel, Chris Parnin, André Brechmann, and Janet Siegmund. Program comprehension and code complexity metries: fmri study. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pages 524-536. IEFE, 2021.
Ryan Krueger, Huang. Xinyu Liu, Tyler Santander. Westley Weimer, and Kevin Leach. Neurological divide: an fmri study of prose and code writing. In 2020 JEEE/ACM 42nd International Conference on Software Engineering (ICSE), pages 678-690, -900. IEEE, 2020.
Zachary Karas, Andrew Jahn, Westley Weimer, and Yu Huang. Connecting the dots: rethinking the relationship between code and prose writing with functional connectivity. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium the Foundations of Software Engineering. pages 767-779, 2021.
Yu Huang. Xinyu Liu, Ryan Krueger, Tyler Santander, Xiaosu Hu, Kevin Leach, and Westley Weimer. Dis- tilling neural representations data structure manipulation using fmri fnirs. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pages 396-407. IEEE, 2019.
Miltiadis Allamanis, Earl Barr, Premkumar Devanbu, and Charles Sutton. A survey of machine learning for big code and naturalness. ACM Computing Surveys (CSUR), (14) 2018a.
Uri Alon, Omer Levy, and Eran Yahav, code2seq: Generating sequences from structured representations of code. In International Conference on Learning Representations, 2019. URI httphtt/oppeeeeeiiieeeee net/forum? id-HigKYo09tX..
Benjamin Bichsel, Veselin Raychev, Petar Tsankov, and Martin Vechev. Statistical deobfuscation of android applications. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pages 343-355, 2016.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders. Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.
11

---

Miltiadis Allamanis, Mare Brockschmidt. and Mahmoud Khademi. Learning to represent programs with graphs. In International Conference Learning Representationss 2018b. URL https://openreeiev. net//Eorum//d-B00EETT..-
Shashank Srikant, Sijia Liu, Tamara Mitrovska, Shiyu Chang, Quanfu Fan, Gaoyuan Zhang, and Una-May O'Reilly. Generating adversarial computer programs using optimized ohfuscations. In International Confer- ence on Learning Representations, 2021. URL wtw.httpa///ooo //tp:////pepeeieeeeeeeeeeee wtw.a/eeeeeeeeeeeenneiieiiaa
Daniel L Yamins, Ha Hong, Charles Cadicu, and James DiCarlo. Hicrarchical modular optimization of convolutional networks achieves representations similar to macaque it and human ventral stream. In J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013. URL https: /ptp/peeeieggaa../ peooceeding..nuuip.... neurips. cc/paperr/013/fflle/ /ttta//////76760000/42222201/100 126777bbbbbb692cccccb0 Papee .df...
Evelina Fedorenko. Anna Ivanova, Riva Dhamala, and Marina Umaschi Bers. The language programming:
a cognitive perspective. Trends in Cognitive Sciences, 23(7):525-525 2019.
Vincent Hellendoorn and Anand Ashok Sawant. The growing cost of deep learning for source code. Communi- cations of the ACM, 6111) 2021.
Joshua S Rule, Joshua B Tenenbaum, and Steven Piantadosi. The child as hacker. Trends in cognitive sciences, 24(1 (A(11):900-915 2020.
Chantel S Prat, Tara M Madhyastha, Malayka J Mottarella, and Chu-Hsuan Kuo. Relating natural language aptitude to individual differences in learning programming languages. Scientific reports, 10(1):1-10, 2020.
Casey Casalnuovo, Kevin Lee. Hulin Wang, Prem Devanbu, and Emily Morgan. Do programmers prefer predictable expressions in code? Cognitive Science, 44(12)=e12211, 2020.
Will Crichton, Maneesh Agrawala, and Pat Hanrahan. The role of working memory in program tracing. In Proceedings of the 202/ CHI Conference Human Factors Computing Systems, pages 1-13, 2021.
Tom M Mitchell, Svetlana V Shinkareva, Andrew Carlson, Kai-Min Chang, Vicente L Malave, Robert Mason, and Marcel Adam Just. Predicting human brain activity associated with the meanings of nouns. science, 320 (5880): 1911-119911111111111111111111111 2008.
Christophe Pallier, Anne- aon--Domnique Devauchelle, and Stanislas Dehaene. Cortical representation the constituent structure of sentences. Proceedings of the National Academy of Sciences, 108())22522-2527, 2011.
Jonathan R Brennan and Liina Pylkkänen. Meg evidence for incremental sentence composition in the anterior temporal lobe. Cognitive science, 41:1515- 15331, 2017.
Shailee Jain and Alexander Huth. Incorporating context into language encoding models for fmri. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gameett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Jon Gauthier and Roger Levy. Linking artificial and human neural representations of language. arXiv preprint arXiv:1910.01 1910.0024,, 2019.
Dan Schwartz, Mariya Toneva, and Leila Wehbe. Inducing brain-relevant bias in natural language processing models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neurul Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Shaonan Wang, Jiajun Zhang, Nan Lin, and Chengqing Zong. Probing brain activation patterns hy dissociating semanties and syntax in sentences. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05): 9201-9208, Apr. 2020. doi:
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. The neural architecture language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences. 118(45), 2021.
Lu Cao, Dandan Huang, Yue Zhang, Xiaowei Jiang, and Yanan Chen. Brain decoding using fnirs. Proceedings of the AAA/ Conference on Artificial Intelligence, (5(14):12022-128 May 2021.
Charlotte Caucheteux, Alexandre Gramfort, and Jean- Remi King. Decomposing lexical and compositional syntax and semantics with deep language models. In International Conference on Machine Learning, pages 1336- 1348. PMLR, 2021.
12

---

Mariya Toneva and Leila Wehbe. Interpreting and improving natural-language processing (in machines) with natural langunge-proeessing (in the brain). In H. Wallach, 1. Larochelle. A. Beygelzimer, d'Alché-Bue, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Syxtems, vol- ume 32, Curran Associates, Inc., 2019, URL httpp://ppet //pp::////////ngss/ www.snneerrrrcooo http:/pp.ccpppppr//11////// 0a55:662221888177777622202338883
Paper-pil .df.
Gary И Glover. Overview functional magnetic resonance imaging. Neurosargery Clinics, 22(2):133-139, 2011.
Kenneth Norman, Sean Polyn, Greg Detre, and James V Haxby, Beyond mind-reading; multi-voxel pattern analysis of fmri data. Trends in cognitive sciences, 1009)4244430, 2006.
Yukiyasu Kamitani and Frank Tong. Decoding the visual and subjective contents of the human brain. Nature neuroscience, 8(5):699 685, 2005.
Nikolaus Kriegeskorte. Patter--inf information analysis: from stimulus decoding to computational-mode testing. Neuroimage, (6(2):11-422) 2011.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon. U. V. Luxburg. S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances Neural Information Processing Systems, neuripe. cctppppr////11/////
volume 30. Curran Associates, Inc., 2017. URL pestps://proceediie Paper.-pa
file/ (at5ee22557aee1111b03aeeeeeeaeed
Hamel Husain, Ho- Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Mare Brockschmidt. Codesearchnet challenge: Evaluating the state of semantic code search. CoRR, abs/1999.094433 2019. URL /atp://rriv.
hrg/abs/1909 1909.094396
Ruchir Puri, David Kung, Geert Janssen, Wei Zhang. Giacomo Domeniconi, Vladmir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks, 2021.
Ilya Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URL https: ttttps://poottt http://ppcceeeen..///// Paw.:aaae-.daa-ae...pppp
Daniel Zügner, Tobias Kirschstein, Michele Catasta, Jure Leskovec, and Stephan Günnemann. Language- agnostic representation learning of source code from structure and context. In International Conference on Learning Representations (ICLR), 2021.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang. et al. Codebert: pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08// 2020.
HuggingFace. Codeherta roherta-like model trained the codesearchnet dataset from githuh. Hugging 2020. httpst /ttpg//gggggggcee.e////o/// co/huggingf ace/ /Coeeeeeeeeaa-eeeeeeeeaeeaaaaaae
act.oeeBeeceCeeeCeTaeaaa..a small-vi.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. XInet: Gener- alized autoregressive pretraining for language understanding. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, F. Fox, and R. Garnett, editors, Advances in Neural Information Processing Syxtems, vol- ume 32. Curran Associates, Inc., 2019. URL. httpa:/ hatp:///////o///s....... wew..pa.cccpppprr/001.. 0ac67776557776444666667838686812 waw.ee6p66..-........... .df.
Microsoft. Codegpt-sman small-py. py. CodeGPT- small-pa py.
HuggingFace, 2021. URL https://nnggingaace.oo// htt.://gg...ggg.ecooooocoooooooo
Marcela Perrone-
Jan Kujala, Juan R. Vidal, Carlos M. Hamame, Tomas Ossandon, Olivier Bertrand, Lorella Minotti, Philippe Kahane, Karim Jerbi, and Jean-Philippe Lachaux. How silent is silent reading? intracerebral evidence for top-down activation of temporal voice areas during reading. Journal of Neuroscience, 32(49): 17562, 2012.
Bertolotti, Perrone-Bertolotti,
Fatma Deniz, Anwar O Nunez-Elizalde, Alexander G Huth, and Jack I. Gallant. The representation of semantic information across human cerebral cortex during listening versus reading is invariant to stimulus modality. journal of neuroscience. 39, 2019.
Mariya Toneva, Tom M. Mitchell, and Leila Wehbe. Combining computational controls with natural text reveals new aspects of meaning composition. bioRxiv, 2022.
13

---

Fabian David Tschopp, Michael B. Reiser, and Srinivas C. Turaga. A connectome based hexagonal lattice convolutional network model of the drosophila visual system, 2018.
Martin Schrimpf, Jonas Kubilius, Ha Hong. Najib J. Majaj, Rishi Rajalingham, Elias B. Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, Kailyn Schmidt, Daniel L. K. Yamins, and James J. DiCarlo. Brain-score: Which artificial neural network for object recognition most brain-like? bioRxiv. 2020a.
Shashank Srikant and Una- May O'Reilly. Can cognitive neuroscience inform neuro souro-yymolii inference models? In Is Neuro- Symbolic SOTA still a myth for Natural Language Inference? The first workshop. 2021. URL httpa://openrevier nwt/tfeurtmeeeee htt.:oorrmdddx///apppp
Paul Nuyujukian, Jose Albites Sanabria, Jad Saab, Chethan Pandarinath, Beata Jarosiewicz, Christine Blabe, Brian Franco. Stephen T Memoff, Emad N Eskandar, John D Simeral, et al. Cortical control of a tablet computer by people with paralysis. PloS one, :ee24)6,, 2018.
Richard A Andersen, Tyson Aflalo, and Spencer Kellis. From thought to action: The brain-machine interface in posterior parictal cortex. Proceedings of the National Academy of Sciences, 11665))22274-262799 2019.
John Duncan. The multiple-demand (md) system of the primate brain: mental programs for intelligent behaviour. Trends in cognitive sciences, 14(4):172-175) 2010.
Evelina Fedorenko, John Duncan, and Nancy Kanwisher. Broad domain generality in focal regions of frontal and parietal cortex. Proceedings of the National Academy of Sciences, (10441)::6616-1662) 2013.
Marie Amalric and Stanislas Dehaene. A distinct cortical network for mathematical knowledge in the human brain. Neurolmage, 189:19-31, 2019.
David Glenn Clark and Jeffrey L. Cummings. Aphasia. In Neurological Disorders, pages 265-275. Elsevier, 2003.
Evelina Fedorenko, Po-Jang Hsieh, Alfonso Tieto-Castaniós Susan Shaitield-Gabrieile and Nancy Kanwisher. New method for fiuri investigations of language: defining rois functionally in individual subjects. Journal of neurophysiology. 104(2):1177-)) 1177-1144, 2010.
Jennifer Hu, Hannah Small, Hope Kean, Atsushi Takahashi, Leo Zekelman, Daniel Kleinman, Elizabeth Ryan, Victor Ferreira, and Evelina Fedorenko. The language network supports both lexical access and sentence generation during language production. bioRxiv, 2021.
Idan A Blank and Evelina Fedorenko. Domain-general brain regions do not track linguistic input as closely as engguage-selective regions. Journal of Neuroscience, 37(41):9999 10009999910011111111111110111111 2017.
Aaron J Newman, Ted Supalla, Nina Fernandez, Elissa L Newport, and Daphne Bavelier. Neural systems supporting linguistic structure, linguistic experience, and symbolic communication in sign language and gesture. Proceedings of the National Academy of Sciences, 112(37):1 1684-11689, 2015.
Dima Ayyash, Saima Malik-Moraleda, Jeanne Gallée, Josef Affourtit, Malte Hoffman, Zachary Mineroff, Olessia Jouravlev, and Evelina Fedorenko. The universal language network: A cross-linguistic investigation spanning 45 languages and 11 language families. bioRxiv, 2021.
David H. Hubel and Torsten N. Wiesel. Receptive fields single neurones the cat's striate cortex. The Journal of Physiology, 148, 1959.
Thad A. Polk, Matthew Stallcup, Geoffrey K. Aguirre. David C. Alsop, Mark D' D'Esposito, John A. Detre, and Martha J. Farah. Neural Specialization for Letter Recognition. Journal of Cognitive Nearoscience, 14(2): 145-159, 2002.
Russell A. Epstein and Nancy Kanwisher. A cortical representation of the local visual environment. Nature, 392: 601. 1998.
Nancy Kanwisher, Josh McDermott, and Marvin M. Chun. The fusiform face area: A module in human extrastriate cortex specialized for face perception. The Journal of Neuroscience, 17:4302 4311, 1997.
Sam Norman-Haagnere, Nancy G Kanwisher, and Josh H McDermott. Distinct cortical pathways for music and speech revealed by hypothesis-free voxel decomposition. Newron, 88(6): 1296-1296, 2015.
Sam Norman-
Norman-Haiinere. Haignere, Nancy Kanwisher, Josh H McDermott, and Bevil Conway. Divergence in the functional organization of human and macaque auditory cortex revealed by finri responses to harmonic tones. Nature newroscience, 12(7):1057-1066 2019.
14

---

John Ashburner, Jesper L.R. Andersson. and Karl J. Friston. High-dimensional image registration using symmetric priors. Neurolmage, 96):619-628, 1999.
Edmund T. Rolls, Chu-Chung Huang, Ching-Po Lin, Jianfeng Feng, and Mare Joliot. Automated anatomical labelling atlas 3. Neurolmage, 206:116189, 2020.
Thomas J McCabe. A complexity measure. IEEE Transactions software Engineering, pages 308-320, 1976.
Martin Shepperd. A critique of cyclomatic complexity as a software metric. Software Engineering Journal, 3(2): 30-36, 1988.
Maurice H Halstead. Elements of Software Science (Operating and programming systems series). Elsevier Science Inc., 1977.
Martin Schrimpf, Idan Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosscini, Nancy Kanwisher, Joshua Tenenbaum, and Evelina Fedorenko. Artificial neural networks accurately predict language processing in the brain. bioRxiv, 2020b.
15

---

Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] Main claims are accurately reflected. (b) Did you describe the limitations of your work? [Yes] Limitations noted in Section 6: Limitations.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Societal implications are noted in Section 6: Limitations.
(d) Have you read the ethics review guidelines and ensured that your paper conforms them?
[Yes]
you are including theoretical results... (a)
Did you state the full set of assumptions of all theoretical results? [N/A] There are no theoretical proofs presented this work.
(b) Did you include complete proofs of all theoretical results? [N/A] There are no theoretical proofs presented in this work. 3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material as URL)? [Yes] All code and data is available at the public link code-represee rite--repreeettiionee
com/ -LLFu// brain, mentioned on the first page of this draft. A sin- gle command is provided to wrap the entire pipeline from environment setup to final plots and tables.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Appendix B: MVPA cross-validation and details.
for all training
(c) Did you report eπrоr bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] See Figure 3 for error bars.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.3: Model representations and decoding. The experiments we present this work are not compute-heavy and can be run on single 8-core laptop in under an hour.
4. If you are using existing assets (e.g., code, data, models) curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [Yes] We use public dataset by
Ivanova et al. [2020] and cite the authors.
(b) Did you mention the license of the assets? [Yes] We mention the license in 4.1: Dataset.
(c) Did you include any new assets either in the supplemental material or as URL? [N/A] No new assets were included, except results derived from existing data. All original links to public data and code operating over them are made available and appropriately licensed to do
(d) Did you discuss whether and how consent was obtained from people whose data you're 're're using/curating? [Yes] We mention participant consent in Section 4.1: Dataset, and link the original data source for more information.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] We mention participant unidentifiability Section 4.1: Dataset, and link the original data source for more information. No offensive content is included.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applica- ble? [N/A] We did not conduct research with human subjects, but this information is available in Ivanova et al. [2020]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A] We did not conduct research with human subjects, but
this information is available in Ivanova et al. [2020]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] We did not conduct research with human subjects, but this
information is available in Ivanova et al. [2020]
16


Reviews:
[{"review_id": "zcEmirx6a-", "reviewer": "Reviewer_zm5y", "summary": "This work examines the relationship between fMRI recordings of people who read short programs and different properties and representations of the programming code. The aim of the work is to understand what properties of code are encoded by different brain systems, and to understand how similar the representations of code in the brain are to those encoded by self-supervised language models that are pretrained to encode programming code. The authors find that several program properties can be significantly decoded from 2 brain systems (the multiple demand system and the language system). They further find that representations of the programs extracted from several machine learning models of varying complexity can also be significantly related to these brain systems.", "questions": "I think this work is a solid contribution. I do have some questions that I am curious about.\n1. I'd appreciate the author's response to the Weakness I raised above.\n2. L65-L74 try to motivate why the question of investigating code comprehension is of interest. One reason given is that it's poorly understood, which is fine. The other reason is difficult for me to understand -- the authors state that two previous works have found that the neural bases of code comprehension are different from those of natural language comprehension. And then they ask \"do code models mimic human cognition of programs? Do language models?\". It's not clear what the difference between code and language models is for the purposes of this work -- aren't all code models just language models that are trained on code rather than natural language, and don't the authors *only* investigate representations from code models and not from natural language models? It's not clear what the expectation here is. Much previous work, as discussed by the authors, shows that language models predict large part of cortex (including the language and multi-demand regions) during natural language comprehension. On the other hand, work on code comprehension shows that its neural bases are different from those of natural language comprehension. So then what did the authors expect to learn by using representations from language models to predict activity during code comprehension? Were these expectations met? Addressing this would strengthen the motivation.\n3. L377-380: An intriguing discussion of how the current findings that the language system encodes some properties of code can be understood alongside the findings from previous work that the language system does not show a consistent response to code. If the language system indeed does not show consistent response to code and yet it encodes important code properties, what does this imply about these system-based analyses in general? The whole reason to look at these specific systems is that they are believed to have some function, based on work that examines their average responses. Perhaps a whole-brain analysis would be more telling of the functional organization of code comprehension w.r.t. different stimulus properties.\n\n\n\n\nMinor comments:\n- The language \"MD/LS network decodes ...\" sounds strange to me, and it's used throughout the paper. I would suggest using \"encodes\" or \"[FEATURE] is decodable from MD/LS network\"\n- L313: \"These findings establish the unique role of the MD system in encoding code-simulation and execution related information\" -- this should be reworded because it can be misinterpreted as these properties being uniquely encoded in the MD system, which is not the case (as the authors show).\n- Typo: L314 \"an important of aspect of code\" -> \"an important aspect of code\"\n- Table 11 in the appendix is illegible\n", "limitations": "Limitations are sufficiently discussed", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Disclaimer: I have reviewed this work previously. The authors have addressed almost all of my previous concerns and the work is much improved. I will focus on the few remaining concerns that I have in this review. I am leaning towards acceptance because of the thorough experimentation and improved discussion and motivation.\n\nStrengths:\n- The discussion of related literature is thorough\n- The analyses are well motivated and the interpretation of the results is sound\n- The writing is clear and concise\n\nWeaknesses:\n- There is a disconnect between the two main parts of the paper -- decoding hand-engineered code features from the brain, and decoding representations from code models from the brain, and it currently feels like it's two projects put into one manuscript. The authors motivate well why studying both is important, but it's not clear what is learned from the code models over what can be learned from the hand-engineered code features. I agree that in *theory* investigating code model representations may give us additional insight because they may contain additional information, but that is not shown in the current work, as far as I can see. An improvement would be to show that certain parts of the brain can be predicted better using the code properties than with all of the hand engineered features, via an encoding analysis, and ideally show that code models can explain additional variance in the brain recordings. This will actually show that the code models contain additional brain-relevant information over what is encoded by the hand engineered features.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dRi5Dl--2Pn", "reviewer": "Reviewer_jQhV", "summary": "In this paper the authors show that they can decode various properties of computer code (e.g. python programs) from brain images taken while subjects read and try to interpret those programs. More specifically, the authors separate the brain data into language and multiple-demand (MD) networks and show that (1) many code properties are represented in both networks, but (2) more information about several properties is represented in the MD network.\n", "questions": "I would challenge the authors to go a little further afield with their analyses, which currently hew very closely to standard practices. Can anything be said about what kinds of properties the different self-supervised models represent? Do they capture similar information to the theoretically-defined properties? Is there any relationship between different properties captured by the models and representations in different brain areas? This seems like a rich and underexplored area, so it is a bit frustrating to see this paper do such shallow analyses!\n", "limitations": "None noted.", "rating": 3, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Overall the paper is fairly clear, the analyses are well-motivated, and the results are well-explained.\n* The methods are clearly and thoroughly explained (bravo!).\n* The statistical analyses are of high quality and well-motivated.\n\nWeaknesses:\n* Methodologically, this paper is very simple. Oftentimes the best approach is the simplest (linear decoders DO work better than most other approaches on fMRI data), especially when it yields a compelling result. But here the results are, unfortunately, not very compelling, and I struggle to find their significance. There are some small differences between the findings here and those from earlier work on this dataset (Ivanova et al., 2020), particularly in assigning a more general role during code comprehension to the language network. However, it’s not clear how this should be interpreted in light of other knowledge about the language network and its representations.\n* The most methodologically interesting analysis, in which code representations from self-supervised sequence models are compared to the brain, is underdeveloped. It seems like something much more interesting could be said about what these models represent, and how those representations relate to the MD and language networks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZOkH38U3Rr9", "reviewer": "Reviewer_Mhum", "summary": "This paper studies code comprehension in the human cortex by evaluating the ability of four different functional networks (multiple-demand, language, auditory, visual) to decode different code properties. The main questions posed are:\n- What is the functional selectivity of different brain regions? (for ex., syntactic or semantic code properties)\n- Do different regions encode similar properties?\n\nThe main results include:\n- Differential selectivity for code properties between the language and multi-demand systems.\n- Dependence of decoding accuracy on token-level information\n- No significant role of the auditory cortex as expected and the processing of low-level properties by the visual cortex.\n\n", "questions": "1. Could the authors elaborate on the significance testing procedure and what “labels” refers to in line 213? Relatedly, what is the significance test used to identify significant  significant effects in Analysis 2 (multi-system partial regression analysis)? Although not identical, this is similar to variance partitioning done in encoding models and the paper could benefit from adding a citation to Deniz et al., 2019.\n2. The difference in properties encoded better by MD vs. LS is very interesting! (lines 329-338) What do the authors make of these differences in terms of hypothetical roles of each system in processing code? Can this be related, perhaps, to functional specificity found in MD & LS for language or the timescales of processing? The paper would benefit from some discussion on the same.\n3. The results presented in fig 3, panel 2 are also interesting! In my opinion, it is not apparent that the auditory cortex is performing above baseline solely because of token-level matches + refer to comments on clarity in previous section.\n4. Is this following necessarily true? “It is currently unclear what mechanisms drive code comprehension. If we find one class of ML models (say, masked language models) to be more predictive than another (say, autoencoders), **it is reasonable to suspect that our brains optimize objectives more similar to that of masked-LMs than that of autoencoders when comprehending code.** ” I would argue that the brain and machines need not be optimizing the same objective to capture similar information. Without systematic analyses of replicated behaviors, prediction performance is not solely enough to answer such a question.\n5. Was the right hemisphere left out from the language system? If so, why? (there has been a huge body of work in the past decade showing evidence of robust, reliable RH activity during language comprehension)\n", "limitations": "-NA-", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Edit after rebuttal: I thank the reviewers for their response. I believe they have sufficiently answered my questions and recommend this paper for acceptance.\n\n##################################################################################################\n\nStrengths:\n\n1. Clarity (of writing): the paper was very well written and easy to understand.\n2. Originality & Significance: Functional selectivity in code comprehension is under explored and this paper effectively investigates this across multiple systems and properties.\n3. Quality: The analyses presented here are thorough and analyze several possible sources of confounds/relationships between the systems and feature spaces. I believe that the claims made are well supported by the evidence.\n4. Misc.: All the data and code will be made freely available.\n\nWeaknesses:\n1. Clarity (of visualizations & results): Since the paper conducts several analyses and uses multiple baselines, the methodological details were lost on me at several places. For example, what is the significance test being in used in panel 2 of FIg. 3 and how is this different from the token projection baseline in the interpretation? (since the paper reports significant performance over chance even in cases where the performance is no different from the token projection features). Furthermore, the authors make several key, interesting observations in the text but this is not apparent from looking at Fig. 3 alone. For example, the preference of static vs. dynamic features between MD & LS or the models that performed significantly better than token projection. I understand the severe space constraints, but I would urge the authors to try alternate visualizations for the main paper that highlight the results presented in the text.\n\nNB: I have reviewed a previous version of this paper and am happy to see that the authors have considerably revised the analyses and manuscript.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "P0_kUCJkhMq", "reviewer": "Reviewer_6vc3", "summary": "\nThis paper studies the parallels in the representation of computer code comprehension, both in neural activity as it is recorded in fMRI and in an artificial intelligence system. The paper first studies the state of the art and presents the scientific question by describing the different aspects of computer code comprehension. It then presents the different hypotheses that can be made at the level of the brain organization and how the understanding of a computer code can be assigned to certain regions of the cortex. The paper then proposes an artificial intelligence architecture, of the deep learning type, which allows building a model of code representation that is evaluated in two experiments that are those described in an existing dataset. The results show a similarity between the results obtained in the neural data and the artificial model.\n", "questions": "\nI have a question about the paper: does a person who knows how to code in several languages (Python, C++, ...) acquire representations specific to each property of the code that are superimposed in the brain as well as in the artificial system, in the same way as people who are polyglot?\n", "limitations": "\nThe authors have correctly specified the limitations (but no potentially negative societal impacts) of this paper.\n", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nThe main strength of the paper is the originality of the study and its potential societal impact in terms of better understanding of the code that can be performed by computer scientists. The analyses seem correct, even if I am not a specialist in the field. The main weakness of the paper is to understand to what extent the comparison results obtained in figure 3 can be interpreted as a homology between the processing that is actually done in the brain and the one represented in the artificial system. (minor) Note that in figure 2, the code that is shown as an illustration does not seem valid. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept