Representations of Computer Programs in the Human Brain
Authors: Shashank Srikant
Benjamin Lipkin
Anna A Ivanova
Evelina Fedorenko
Una-May O'Reilly
Venue: ICLR 2022 Submitted (2022)
Keywords: ML for PL/SE
ML models of code
Code representations
Brain representations
Cognitive neuroscience
Multivoxel pattern analysis
Representation decoding analysis
Representation similarity analysis
fMRI analysis

Abstract: We present the first study relating representations of computer programs generated by unsupervised machine learning (ML) models and representations of computer programs in the human brain. We analyze recordings---brain representations---from functional magnetic resonance imaging (fMRI) studies of people comprehending Python code. We discover brain representations, in different and specific regions of the brain, that encode static and dynamic properties of code such as abstract syntax tree (AST)-related information and runtime information. We also map brain representations to representations of a suite of ML models that vary in their complexity. We find that the Multiple Demand system, a system of brain regions previously shown to respond to code, contains information about multiple specific code properties, as well as machine learned representations of code. We make all the corresponding code, data, and analysis publicly available.

Under review as a conference paper at ICLR 2022
REPRESENTATIONS OF COMPUTER PROGRAMS IN THE HUMAN BRAIN
Anonymous authors Paper under double-blind review
ABSTRACT
We present the first study relating representations of computer programs generated by unsupervised machine learning (ML) models and representations computer programs in the human brain. We analyze recordings-brain representations- from functional magnetic resonance imaging (fMRI) studies of people compre- hending Python code. We discover brain representations, in different and specific regions of the brain, that encode static and dynamic properties of code such as abstract syntax tree (AST)-related information and runtime information. We also map brain representations representations of suite of ML models that vary in their complexity. We find that the Multiple Demand system, system of brain regions previously shown to respond to code, contains information about multiple specific code properties, as well as machine learned representations of code. We make all the corresponding code, data, and analysis publicly available
INTRODUCTION
A question of interest, as yet uninvestigated, is what aspects of computer programs, i.e. code, are encoded by the human brain when comprehending them. Is it possible that common code properties and the semantics of programs are encoded in brain activity patterns when code is comprehended?
A few prior works have investigated the neural bases of programming (Siegmund et al.) 2017 Floyd
et al.... 2017: Peitek et al. 2018 Castelhano et al. 2019 Huang et al. Kruggr et al. 2020 Prat
al. 2020 Ivanova al. 20200 Liu al. 2020 Ikutani et al. 2021 Peitek et al. 2021). hese
works use analysis techniques like functional magnetic resonance magingg (IMRI) and electroen- cephalography (EEG) to investigate brain regions involved in coding tasks like code comprehension, code writing, and data structure manipulation. The primary quest of these works though has been to locate physical regions in the brain involved in these activities, with the goal of determining whether code-related activity joins other activities supported by those brain regions. It remains unclear what specific code-related information these regions encode. Do these regions encode specific syntactic or semantic code properties? Further, are different regions involved in encoding different such prop- erties? To date, no systematic method exists to generate the evidence from which to draw answers. These questions form the focus of this work.
One way to learn what information is encoded in the brain is by recording brain signals when read- ing code (through fMRI or EEG), and then decoding from the recorded signals a code property of interest. Being able decode the property accurately from specific region of the brain establishes that information related to that code property is faithfully represented that brain region. A ques- tion central to such a decoding analysis would be the choice of the target code property-what code properties should be investigated? We can hand-select set of fundamental properties of code and test if they can be decoded. While helpful, such set will not preclude other, more complex aspects of code being encoded.
To address the limitation of hand-selected properties, we look to recent advances in unsupervised machine learning (ML) models trained on code. Dubbed code models, they process large corpora pollpptterpoorraaea
of code to learn ML model representations of computer programs. These models are increasingly being used in software engineering workflows (Allamanis al. 2018a), and have been shown to perform well on tasks like code summarization (Alon et 2019). detecting variable misuse (Bich- sel et al., 2016). and more recently, code auto-completion (Chen et al. 2021). These continuous representations likely encode syntax and semantics of programs (Bichsel al. 2016 Allamanis let al.] [2018b [Srikant et al.) (02T). Beyond the fundamental, hele-selected properties, they may en- code and describe more complex code properties. Decoding these continuous representations from brain activity data then promises to inform us whether such additional properties are also encoded
Project URI. http:///w/ggttt..ooo/////oo/oroa

---

Under review as a conference paper at ICLR 2022
in the brain. Further, comparing representations from code models to the code-related information encoded in the brain also informs us whether code models resemble humans in the knowledge they learn and encode. Yamins et al. (2013) first showed how information encoded in our visual system resembles what convolutional neural networks learn when trained to recognize images. A similar correspondence can possibly be established between code models and the human brain.
In this work, we study code comprehension, and utilize the fMRI recordings dataset from Ivanova eet al. (2020) to investigate human brain representations of code. We present two means of seoceddnn--robing of brain region representations for specific code properties, and analyzing the mapping of these representations onto various code models with differing model complexity. We learn affine maps from brain representations predict hand-selected code properties which summarize the syntactic and semantic behavior of programs, and similarly predict code repre- sentations of different code models. We investigate the effects that brain regions, the nature of code properties, and the complexity of the code models have on the accuracies with which decode brain representations. We find that we can decode code-related information from brain representations, and can map representations of code models from brain representations. We find the Multiple Demand system, followed by the Language system, to consistently encode specific code properties and map with machine learned representations of code. We provide open-source framework to replicate our experiments, and we release our data and analysis publicly. Link /stps://ggttube github......... con/ anonmyous author/ author/anonynonn anonymous code The framework allows both brain data and program metrics to be easily added to extend the current set of experiments. This should enable authors from other neuroimaging studies code model developers to collaborate and analyze data across these works, which will also help amortize the high costs of carrying out such experiments.
2 RELATED WORK
Of the prior works that have investigated the neural bases of programming through IMRI and EEG
techniques (Siegmund et al. 2017 Floyd et al. 2017 Peitek et al. 2018; Castelhano et al. Huang et al. 2019 Krueger et al. 2020 Ivanova et al. 2020 Liu et al. (2200] Ikulan et al.
2019
202T
Pettek et al. 2021 and through behavioral studies (Prat et al. 2020; Casalnuovo et al. 20200 Cnch-
[on ct | 2021), the following probe brain recordings for program properties encoded In them.
Floyd et al. (2017) learn a linear model to successfully classify whether observed brain activity corresponds to reading code or reading text. [kutani et al. (202I) study expert programmers and show that it is possible to classify code into the four problem categorics-matt search, sort, and string from the brain activations corresponding to the code. Similarly, [Lu et al.J (2020) classify whether a brain signal corresponds to code implementing an if condition not. Peitek et al. (2021) analyze correlations between brain recordings of participants reading code and a set of code complexity metrics.
In testing for code properties, our work uses a similar methodology (a linear model trained on fMRI data), but we evaluate a larger set of static and dynamic code properties, We also systematically test whether key programming constructs like control flow and data operations are encoded by these brain representations. A systematic analysis of multiple such properties has not been performed in any of these works. Further, we perform these tests in those brain regions identified by Liu et al. (2000) and Ivanova et al. (2020) as being responsive specifically to code comprehension. This gives us finer insight into how different regions encode these properties. In addition these tests, we study representations generated by suite of ML models with varying model complexity and compare those representations to brain representations.
Brain representations have also been studied in domains like natural language, vision, and motor control. Among related works in natural language, a domain that resembles programming languages, Mitchell et al. (2008); y; Pallier et al.) (201 Dj [Brennan & Pylkkänen (2017): Jain & Huth| (2018), Gauthrer & Levy (evy)2011))) Schwartz et al. (2019); Wang et al. 2020))(chrimm et al. (2020) Cao et al. (202)) have studied brain representations of words and sentences by relating them to representations
produced by language models. While the broader tools we use to investigate these representations, like -uxti-voxel pattern analysis (MVPA), are similar to some of these prior works, our focus is on properties specific to code and not natural language.
3 BACKGROUND
We provide a brief background on fMRI signals as a proxy for brain representations and describe the brain systems that we probe in this work.
2

---

Under review as a conference paper at ICLR 2022
Measuring brain activity with fMRL Functional magnetic resonance imaging (fMRI) is brain imaging technique used measure brain activity specific brain regions. When a brain region is active, blood flows into the region to aid its processing. An MRI machine measures this change in blood flow, and reports BOLD (blood oxygen level dependent) values sampled at the machine's frequency (Glover 2011 I usually 2 seconds). The smallest unit of brain tissue for which BOLD signal is recorded is called voxel (an equivalent of 3D pixel); it comprises several cubic mil- limeters of brain tissue. For our analyses, we select subsets of voxels belonging to specific brain systems. Following common practice, the parameters of a general linear model, fit to time-varying BOLD values, are used measure of the overall activation in each voxel in response to given input. It is the values of these parameters that, according the neuroscience community, are brain representations.
Brain systems. A system of brain regions can span different areas of the brain but behaves holistic unit, showing similar patterns of engagement across a given cognitive task. We probe the following sys- tems of regions in our work. See Appendix [A for a detailed description of these systems along with rel- evant references.
Left hemisphere
Right hemisphere
Left hemisphere
Muttiple Demand (MD) System
Language System
Multiple Demand (MD) system. This system of regions is known to engage in cognitively de- manding, domain agnostic tasks like general prob- lem solving, logic, and spatial memory tasks. [Liu et al. (2020) and Ivanova et al. (2020) reported that this system is active during code comprehension.
Figure I: The approximate locations of MD and the Language systems in the human brain. The regions depicted are used as a starting point to functionally localize these systems in individual participants.
. Language system (LS). This system responds dur- ing language production and comprehension across
modalities (speech, text) and languages (across 10 language families, including American sign lan- guage). SSiegmund et al. (2014; 2017) reported activity in regions resembling the LS during code
comprehension, whereas Floyd (217); Liu al. (2020); Krueger et al. (2020) concluded that
brain regions processing language and code are distinct. [vanova al. (2020) reported moderate
levels of activity within the LS in response to Python, but not to code in a visual programming lan- guage, ScratchJr.
We also probe brain representations in two brain systems responsible for perception.
Visual system (VS). These regions respond primarily to visual inputs. We expect activity this system to reflect low-level visual properties of the code (e.g. code length and indentation).
Auditory system. These regions respond primarily to auditory inputs. We do not expect activity in this system to reflect cele-related properties.
4 BRAIN AND MODEL REPRESENTATIONS
We describe in this section the method we follow to gather representations of code in the brain (Section[4.]). evaluate the different code properties they encode (Section 4.2), and how we compare brain representations to those generated by ML models (Section 4.3)
4.1 BRAIN REPRESENTATIONS AND DECODING
We provide summary of how we process activation signals the brain elicited by code comprehen- sion, to probe whether they encode any specific code properties. We provide details in Appendix B
Dataset. We use the publicly available brain recordings released as part of the study by Ivanova et al. (020). It contains brain recordings of 24 participants reading 72 programs from a set of 108 unique Python programs. The 72 programs were presented in 12 blocks of 6 programs each. These programs were 3-10 lines in length and contained simple Python constructs, such as lists, for loops and if statements. A whole program was presented at once, and the task required participants to read the code and mentally compute the expected output, press button when done, and select one of four choices presented to them which matched their calculated output.
From dataset to brain representations. The original dataset contains 3D images of the brain of each participant. Each voxel value in these images is an estimate of the response strength in this voxel when a particular code (or sentence) problem is presented. To determine which brain systems contain information about particular code properties, we focus our analyses four systems MD, Language, Visual, and Auditory (Section 3. A vector of voxels' activation values in each brain system is then taken to constitute that system's 's representation of a computer program and serves as an input to all our analyses. See Appendix B for details on data processing and voxel selection.
3

---

Under review as a conference paper at ICLR 2022
What code properties do brain representations encode?
Static, dynamic analysis of code
Code properties
NXDE
R Nx1
deeeity inteecaenn Love ypini
Experiment
Human brain
Neural representation of code
How similar are machine representations and brain representations?
pois
Code
Nx
NKD
Code language model
Machine representation of code
Experiment 2
Figure 2: Overview. The goal of this work to relate brain representations of code to (1) specific code properties and (2) representations of code produced by language models trained on code. In Experiment 1, we predict the different static and dynamic analysis metrics from the brain MRI recordings (each of dimension D3) of 24 human subjects reading 72 unique Python programs (N) by training separate linear models for each subject and metric. In Experiment 2, we learn affine maps from brain representations to the corresponding representations generated by code language models (each of dimension D on these 72 programs.
Analyzing brain representations of code. We probe brain representations from each participant separately. We do not average data across participants since the regions which respond any task (comprehending code in our case) need not align anatomically. For each of two experiments- decoding different code properties, and mapping to code model representations--ww we train ridge regressionCllassffcatiin models which take input normalized brain representations per partici- pant. We hence learn 24 different regression models, for each code property or code model (one per participant), and then report the mean performance of these models across participants. This procedure is also referred to as multi-voxel pattern analysis (MVPA) (Norman et al. 2006). Lin- ear models are conventionally preferred for probes into brain representatrons since there has been evidence supporting the idea that other brain areas linearly map information from such brain rep- resentations (Kamitani & Tong 2005 Kriegeskorte 201T). We choose a linear model primarily to control for over-hitung light of the relatively small dataset.
remark on data scarcity. We train linear regression/clasiffcation model with L2-regularization for every participant on unique cross- validated leave-one-run-our folds of the 72 programs they attempted (or 48 programs when sentences are removed). On the order of 1000 voxels were selected from each brain region responding any given program per participant. As baseline for the model predictions, we use the accuracy of null permutation distribution generated from sampling 1000 random assignments of the labels.
4.2 CODE PROPERTIES
We attempt to decode the following code properties from brain representations.
Code vs. sentences. The dataset provides control condition where code problems are described
in English sentences (referred to as sentences). See Figure B Appendix B for an example. We classify these two stimuli categories from brain data.
Variable language, We classify programs containing meaningful variables names from brain representations. The dataset has half its programs with variables English, and the other half in Japanese (written in English characters). which do not convey any contextual meaning.
Control flow. We predict whether a program contains loop (for loop), a branch (i. condition), or has sequential instructions without any control branching. The dataset contains equal number of programs with each of the three conditions.
Data type. We predict whether a program contains string operations or numeric operations in them. Half the programs in the dataset exclusively have string operations within them, while the other half are exclusively numeric.
Static analysis. We decode static properties of program. Specifically, we predict token count (number of tokens in the program) and node count (number of AST nodes). We also consider cyclo- matic complexity, and Halstead difficulty (details in Appendix BB. which have been used by software engineering practitioners to quantify the complexity of code, and quantify the difficulty human
4

---

Under review as a conference paper at ICLR 2022
would experience when comprehending code respectively. We defer predicting other advanced static analysis metries such as tracking abstract interpretation joins, data flow analysis-related metrics, etc. to future work.
Dynamic analysis. We decode information about a code's execution behavior. We evaluate two properties-runting steps (number of instructions executed in the program) and bytecode ops (num- ber of bytecode operations executed in running the program). We expect these properties to be correlated with the number of mental instructions needed to arrive at the output.
Program length as a potential confound. Program length is a potential confound in our experi- ments because the properties we examine can also potentially be differentiated using program length and other low-level code features. We measured the inter-correlations of these properties. and their correlation to the number of tokens in the program (program length) (see Appendix (G). We expect- edly found the four static analysis properties to be highly correlated to each other and to bytecode ops. We hence use one representative metric each from the two categories of properties for the rest of our analysis-tokes tantly, the other propertics we examine cannot be explained by program length alone, and therefore program length is not confound in our experiments.
count for static analysis, and runtime steps for dynamic analysis. Impor-
The brain representations (Section 4.1 are mapped to each of the code properties by training ridge regression or classification model each for every participant-proeers formance, we use classification accuracy when the predicted values are categorical (e.g. string vs. numeric data types), and the Pearson correlation coefficient when the predicted values are contin- uous (e.g. number of runtime steps). We choose Pearson correlation over RMSE, the canonical distance metric for continuous values, for its simplicity and interpretability. See Appendix | for the RMSE results, which are similar to the correlation results. When testing for the significance of these predictions, we perform false discovery rate (FDR) correction for the number of brain systems tested and the number of properties tested. See Appendix B for detailed description of model hyper-parameter and cross- validation settings employed.
pair. To evaluate model per-
cross-validation
4.3 MODEL REPRESENTATIONS AND DECODING.
We evaluate a bench of unsupervised language models, spanning from count- language models to transformer neural networks (Vaswani et al. 2017). These models were all trained on large (~ programs) Python datasets (Husan et al. 2019 Pun et al. 2021). We use the output of the trained encoders (raw logits) in each the neural network models as representations of the code input to the model. We vary the general complexity of these models to test whether that variation is mean- ingful in establishing the quality of brain to model fits. Model complexity here is the number of a model's 's learnable parameters. We evaluate the following models, ordered by their increasing model complexity: simple frequency-basee language models laseell-bgg--e bbag-of-wordd TF-İDF; auto-encoder based unsupervised models seq2seq (Sutskever et al.] 2014), Transforrrr (Zügner et al.. 2021), CodeBERTa (HuggingFace)) 2020); auto-regressiv model with a model complexity similar to that of transformers and CodeBERTa XLNet (Yang et al. 2019).
We compare the results of the above models against an aggressive baseline (relative to the null- distribution labeling baseline), a token projection model provided by using unique Gaussian- distributed random vectors for the token embeddings in a vocabulary, and returning the sum of these token embeddings across program. The resultant embedding is not transformed by any model or any weights-it instead serves as a proxy for the tokens that appear in the program. The results of this baseline model should be interpreted as the level of performance achievable from the presence of tokens alone with no structural information.
The brain representations (Section 4.1 are mapped to code representations by training another set of ridge regression models to learn an affine map, and a ranked accuracy metric is used to compare outputs. Ranked accuracy scores are commonly used in information retrieval where several elements in range are similar to the correct one. In our case, the top-ranked prediction by the linear model indicates the closest fit (Euclidean distance) to the code model's representation. When reporting result significance, we perform false discovery rate (FDR) correction for the number of brain systems and the number of models. See Appendix [ for a detailed discussion of the implemented code models and the corresponding metrics.
5 EXPERIMENTS & RESULTS
Our experiments address two research questions:
Experiment 1. How well do the different brain systems encode specific code properties? Experiment 2. How do the brain representations code correspond to the representations learned by computational language models of code?
5

---

Under review as a conference paper at ICLR 2022
100
MD
Language Visual
0.4
Classification Accuracy (%)
80
Auditory
Pearson Correlatio (r)
60
0.2
40
0.1
Code vs, Sentence
Variable Language
Data type
Control Flow
Static Analysis
Dynamic Analysis
Code Properties
Figure 3: In Experiment 1, a linear model is trained on brain representations to predict each of the code properties described in Section 4.2 Error bars represent 95% confidence interval of individual subject scores. Dotted lines signify the empirical baseline for a null permutation distribution on shuffled labels. All results were compared to this permuted null distribution, and p-values for the number of comparisons in this experiment were corrected for false discovery rates (FDR). Statisti- cally significant results are denoted with a *, marked at the base of the bars.
5.1 EXPERIMENT 1 HOW WELL DO THE DIFFERENT BRAIN SYSTEMS ENCODE SPECIFIC CODE PROPERTIES?
Here, we analyze the classification models and regressions trained on brain representations predict each of the code properties described in Section 4.2 The results of our analyses are summarized in Figure The classification and regression tests are marked on the x-axis of the left and right subplots respectively; the classification accuracy or Pearson correlation for each of the tasks is marked on the y-axes. We plot dynamic and static properties separately from the others because their baselines are different due to difference in the similarity metric (classification accuracy vs. Pearson correlation). The baselines for the categorical code properties differ from each other due to variation in the number of target classes.
Auditory and Visual systems. The Auditory system serves as negative control for the other systems. Given that the code comprehension task we use is visual, we do not expect to decode any meaningful information from it. We find that, out of 6 tested properties, only code vs sentences can be decoded from the Auditory system, and even then the decoding performance is much lower than in other systems. The Visual system also serves as a control for low-level visual properties of the code (as opposed to abstract semantic or syntactic features). Given the visual attributes of a program such as the layout and indentation of the code, the length of the program, and the presence of letters and alphabets in the programs (Park et al.) 2012) Roux et al., 2008 Polk et al.) 2002), this system might reflect at least some of the properties we evaluate. This Is indeed what we observe: 4 out of 6 tested properties can be decoded from the visual system above-chance. The MD and the Language systems yield the following specific observations. In a follow-up up analysis (A (Analysis 3), we investigate the extent to which the MD and the Language systems represent decodable information beyond that which is represented in the Visual system.
Analysis How accurately are different properties predicted by MD and LS? We analyze how accurately each of MD and LS predict all the properties we evaluate.
Code and sentence properties are decoded most accurately among all the properties we test. The MD, LS, and VS decode with 90% accuracy. The high accuracy from MD validates claims from previous works suggesting the involvement of these regions in code compre- hension.
Code vs. sentences
Variable language This requires distinguishing between sets of nearly identical programs, with the only variation being in the language of the variable names (English vs. Japanese). We hence expect the Language system will most accurately encode this information, since it is sensitive to the presence of new words. However, no such effect is observed- no brain system shows any significant decoding. This unexpected finding is however consistent with [vanova et al.) (2020), who show lack of any significant difference in the aggregate neural activity between the two variations. Both suggest that these brain systems do not seem to rely on variable names as a meaningful feature. Control flow and data type Both control flow and data type tasks are most accurately predicted by the MD, followed by the I.S In the case of control flow, the more accurate predictions by the MI)

---

Under review as a conference paper at ICLR 2022
High
Model Complexity
Low
Language
Visual
Auditory
Ranked Accuracy (%)
Baseline Model
CodeBERTa
code
XLNet
Seg2Seq
TF-DDF
Bag of Words
Random Embedding
Model
Figure 4: In Experiment 2, an affine map is learned from brain representations to the representations produced by machine learning models. Error bars represent 95% confidence interval of individual subject scores. Dotted lines signify the empirical baseline from a null permutation distribution on shuffled labels. Random embedding is an aggressive baseline of using random but unique embed- dings for vocabulary tokens. Statistically significant results are denoted with a *, marked at the base of the bars.
can possibly be explained by the larger number of mental operations needed for programs with loops when compared to programs with one or no branches. The difference seen in data type prediction is unclear. While programs with string operations contain more literals and words from the English alphabet, we do not see any preference in the LS being able to decode this property more accurately. Static and Dynamic analysis We observe the LS predicts static analysis properties with the high- est accuracy. This is expected, since activity in the LS known to be sensitive to the length of any text being read (independent of any specific program instructions in our case).
For the dynamic analysis property runtime steps. we see the MD system has the highest accuracy. This again is reflective of the workings of the MD system, which is sensitive to working memory tasks, and the number of steps executed by program matches the number of mental calculations performed by person tracing through the program to calculate its output.
How accurately do MD and LS encode different properties? We use t-tests to examine whether for a given property, any one brain system decodes it significantly more than another. We find no differences between the MD system and the Language system for any properties (Table 8 Appendix E We additionally test if any brain system has a preference for a specific code property over another. For instance, is the evidence seen in Figure 3 of MD more accurately decoding dynamic analysis properties than static analysis properties, statisfically significant? We do not find any significant differences (Table ( Appendix encode any of the code propertics we explore in this work, we do find that the two brain systems individually encode these properties, which is a new result.
Analysis 2-
(E) While neither MD nor LS preferentially
Analysis 3 Multi-system regression analysis. The decoding performance of the Visual system comparable to that of the MD and the Language systems (Figure 3.. It is then possible that all three systems MD. LS, and VS encode the same properties (all potentially related to program length and other low level program features). We employ a multi-systen this possibility. For each brain system, MD and LS (S,), we train two models-on one which decodes from S,. and another which decodes from SS+VS If the difference in the prediction accuracies between the two models is significant, we conclude that S, encodes at least some information which is orthogonal to the information encoded by the Visual system. For control flow, data type, and code find the MD to encode some information orthogonal to the VS. For control flow
regression analysis to test
ws sentences, and code vs sentences, the LS also encodes some information orthogonal suggests that low-level code properties are insufficient to explain the key results from Experiment 1. Other combinations in the regression model show that the MD encodes information orthogonal to the LS when predicting code vs sentences. Detailed results are tabulated in Table 15 Appendix
what the VS does. This
5.2 EXPERIMENT 2 HOW DO THE BRAIN REPRESENTATIONS OF CODE CORRESPOND TO THE REPRESENTATIONS LEARNED BY COMPUTATIONAL LANGUAGE MODELS OF CODE?
Here, we train ridge regression models with brain representations of programs from specific brain system to predict code model representations of the same programs. The full set of results from this experiment are summarized in Figure
Auditory and Visual systems. Similar to Experiment 1, the Auditory system performs as expected, exhibiting the lowest decoding performance across all code models. The VS exhibits the second
7

---

Under review as a conference paper at ICLR 2022
lowest performance across code models with the exception of CodeBERTa to which it maps more accurately to than LS, but not MD. However, this result is not statistically significant (Table To Appendix[E] EE
Analysis 1 How well do brain representations in MD and LS map to code model repre- sentations? To ease our analysis of the MD and LS systems, we re-plot in Figure [ the de- coding accuracies of just these two brain systems from Figure 4 We find that the MD system and Language system map to all the models in our suite signihcantly more accurately than the null permutation baseline (baseline accuracy of 50% in Figures 4 and 5). Further, from Figure 5 we see the MD maps more accurately than the LS across all code models. We test the signin- cance of the differences between the two systems and find that the MD ranked accuracy is higher than LS for CodeBERTa, seq2seq, and IF-IDF, while the differences are not significant for the
other models (Table 10 Appendix E We discuss the implication of these results in Section 6
The effect of model complexity on decoding to code models. We investigate the im- pact model complexity has on the performance of the mapping between brain and code representations. Model complexity here is the number of model's s learnable parameters. In analysis above, we saw that the MD system mapped to the representations of CodeBERTa, seq2seq. and TF-IDF significantly more accurately than the Language system. These three models vary substantially in their complexity. thus suggesting the lack of any relationship between model complexity and brain systems.
Analysis 2
Ranked Accuracy (%)
MD Language
60-
MD Random
Embedding
55
Lang Random Embedding
50
Baseline
Code
CT
XLNet
S25
TF-DF BOW
BERTa
Figure 5: Mapping MD and LS to code model representations.
We also compare each of these code models against the Random embedding model. This baseline model assigns a random vector of numbers to every unique token that appears in the training vo- cabulary, and sums the vectors of all the tokens that appear in program. By its design, this model encodes only the presence of specific tokens in programs. The mapping accuracies of random em- beddings in the MD and the Language systems are marked in blue and green horizontal lines in Figure 5] We find that the Multiple Demand system maps to all the models more accurately than the Random embedding. This is not seen in the LS, which confirms the lack of variabilility of ranked accuracies observed in the LS.
In a set of pairwise significance tests, we find that the MD maps to CodeBERTa significantly more accurately than Random embedding while other systems are more accurate numerically, but not significantly (Table 11 Appendix E). We discuss its implications in the following section.
6 DISCUSSION
Through this study, we learn what computer program-related information can be decoded from the brain, and which brain systems primarily encode that information. In Experiment 1, we investigate whether a set of manually-selecter
code properties are encoded in different brain systems. In Ex- periment 2, by comparing brain representations to code model representations, we investigate if the brain encodes more than just the properties investigated in Experiment 1. Because the code models are trained on a much more expressive set of programs than the ones used in our Experiment we also can learn what additional information these models learn about code.
Key among our contributions is demonstrating that it is possible to predict a range of code properties from brain system activity (representations). Experiment 1 shows that control flow and data type information, two fundamental program properties, well dynamic analysis and static analysis properties of code, can be decoded with high performance from the MD and the Language systems. In several cases, this decoding performance reflects decoding from information beyond what can be predicted from low-level features of the programs encoded in the Visual system alone. This is new finding which indicates that the MD and the Language systems encode higher-level program-related information. Future work should additionally experiment with larger set of programs that have more complex properties across multiple programming languages.
Another key contribution of our work, from Experiment 2, is demonstrating that it is possible to map brain representations to representations learned by code models. Unlike many prior decoding works in cognitive neuroscience, which record multiple responses to each of their experimental stimuli (done generally to control for the noise present in fMRI data), our decoders are trained on individual trials. This demonstrates the feasibility of probing aspects of code comprehension in the brain without the need present the same program multiple times to participant.
8

---

Under review as a conference paper at ICLR 2022
The performance of mappings between brain and code model representations does not appear to be correlated the complexity of the models. However, we do observe a preferential encoding of the properties represented by code models in the MD over LS for the models CodeBERTa, seq2seq, and TF-ID.. Such preferential encoding of properties in MD over LS was not seen for the properties evaluated in Experiment 1, which provides evidence that the brain activation data encodes more than just the code properties we evaluated.
The MD and LS map to complex models like seq2seq and Code'raansfrmer almost as well as to a combination of random token embeddings. One plausible explanation for this surprising result is that the MD and LS signals we have access mostly encode token-level information, and not richer structural information from the programs. The program stimuli are simple enough to allow the different properties evaluated in our work (control flow, data rype, etc.) to be discerned from token level information alone (as validated in Experiment 1), which is likely why the random embeddings model is also able predict these properties very well (see Table 4 Appendix C.2
Taken together, this suggests that the information being decoded from brain activations in these two regions is driven at least by the information conveyed by tokens in the programs. We cannot come to this conclusion from the code properties investigated in Experiment alone.
CodeBERTa alone, for which the response is significantly greater than random embeddings, and which maps more accurately to the MD system than any other brain system, provides initial evidence for the MD system encoding more than just token information. This will need more investigation. Given the current data and the number of observations we have, we can only conclude that tokens do get encoded well in both brain activations and code models. This is new finding on what aspects of code comprehension get encoded by our brains which can be successfully read out using a linear model.
Limitations. The average program in any software project exhibits non-trivial control and data de- pendencies, object manipulation, function calls, types, and state changes, which our dataset does not possess. However, the programming tasks [vanova et al. (2020) are short snippets of proce- dural code with limited program properties. Responses to longer, more realistic programs should be studied on a larger number of participants in the future, building on the understanding of simpler snippets provided by our work. Further, while there are equally important aspects to programming like designing solutions, selecting appropriate data structures, and writing programs, we have cho- sen to study a very specific activity related to proranminggccompehenniii
programs.
One aspect that cannot be inferred from these results, is whether the MD and LS are driven by the same underlying features of code that are used discriminate between code properties, and to map to different code models. Future work should consider an encoding analysis, where we predict the activity of voxels in different brain systems from code properties, in order to establish the relative contributions of those properties to the activations of individual voxels in these systems.
Broader impact. Our findings have the potential to improve our understanding of the organization of the human brain, which can in turn lead to the design of better code models. computer vision, recent results by Tschopp et al. (2018) show how deep network architectures that mimic the visual system in fruit flies exhibit superior image classification rates on image recognition tasks. One im- mediate outcome is reconsidering the current design of MI. models of code. Extant code model architectures do not explicitly model the Multiple Demand system in any way-they only model syn- tactic information and infer dependency information from program AST's. Taking inspiration from the role of the Multiple Demand system we identified in this work, modeling both static and dynamic information should be explored.
This work could perhaps also enhance code waahettics-artfficinn interfaces in the body that can help the physically challenged engage with programming environments. Such systems generally rely on brain decoders-model that convert brain activity data to electrical impulses modulating external devices. An open challenge is to improve them. See the discussion in Nuyujukian et al. (2018) and Andersen et al. (2019) for details.
Finally, our aspiration is that the framework we release in this work will bring together the neuro- science and machine learning communities to better understand the cognitive and neural bases of programming.

---

Under review as a conference paper at ICLR 2022
7 REPRODUCABILITY STATEMENT
All the results, and corresponding tables, plots, and intermediate results we introduce in this paper are fully reproducible. We have made the source code to our work publicly available through this anonymized code repository hittps://githun http://gggtuub..cc/////mmmoo//
anorymous-code
The repository contains detailed instructions for setting up the source code and running the two main experiments we introduce this work. Further, a copy of the dataset released by Ivanova et al. Ivanova et al. (2020) is also available for in the repository, along with the source code.
All the intermediate results which through Appendix !|
to arrive conclusions available in Appendix A
10

---

Under review as a conference paper at ICLR 2022
REFERENCES
Miltiadis Allamanis, Earl Barr, Premkumar Devanbu, and Charles Sutton. survey of machine learning for big code and naturalness. ACM Computing Surveys (CSUR), 51(4):1-37, 2018a. 8a.
Miltiadis Allamanis, Mare Brockschmidt, and Mahmoud Khademi. Learning to represent programs with graphs. In International Conference on Learning Representations, 201 8b. URL https:
//openzeview. net wor.mooruum///// Ba-EBJOEBXXRRR
Uri Alon, Omer Levy, and Eran Yahav. code2seq: Generating sequences from structured rep-
resentations of code. In International Conference Learning Representations, 2019. URL
https://openrevier net Fo
Marie Amalric and Stanislas Dehacne. A distinct cortical network for mathematical knowledge in the human brain. Neurolmage, 189:19-31, 2019.
Richard Andersen, Tyson Aflalo, and Spencer Kellis. From thought to action: The machie interface in posterior parietal cortex. Proceedings of the National Academy of Sciences, 116(52): 26274-26279, 2019.
John Ashburner, Jesper L.R. Andersson, and Karl J. Friston. High-dimensional image registration using symmetric priors. Neurolmage, (6)):19-662) 628, 1999.
Dima Ayyash, Saima Malik-Moraleda, Jeanne Gallée, Josef Affourtit, Malte Hoffman, Zachary Mineroff, Olessia Jouravlev, and Evelina Fedorenko. The universal language network: A cross- linguistic investigation spanning 45 languages and 11 language families. bioRxiv, 2021.
Benjamin Bichsel, Veselin Raychev, Petar Tsankov, and Martin Vechev. Statistical deobfuscation of android applications. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 343-355, 2016.
Idan A Blank and Evelina Fedorenko. Domain-general brain regions do not track linguistic input as closely as language-selective regions. Journal of Neuroscience, 97(41)9999-10011 2017.
Jonathan R Brennan and Liina Pylkkänen. Meg evidence for incremental sentence composition in the anterior temporal lobe. Cognitive science, 41:1515-1531, 2017.
Lu Cao, Dandan Huang, Yue Zhang, Xiaowei Jiang, and Yanan Chen. Brain decoding using fnirs. Proceedings of the AAAI Conference on Artificial Intelligence, 25(14):12202-1261 May 2021.
Casey Casalnuovo, Kevin Lee, Hulin Wang, Prem Devanbu, and Emily Morgan. Do programmers prefer predictable expressions in code? Cognitive Science, 44(12):e 12921, 2020.
Joao Castelhano, Isabel C Duarte, Carlos Ferreira, Joao Duraes, Henrique Madeira, and Miguel Dastelo-Branco. The role of the insula in intuitive expert bug detection in computer code: fmri study. Brain imaging and behavior, 13(3):223-677, 2019.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray. Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert- Noss,, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Me- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.
David Glenn Clark and Jeffrey L Cummings. Aphasia. In Neurological Disorders, pp. 265-275. Elsevier, 2003.
Will Crichton, Maneesh Agrawala, and Pat Hanrahan. The role of working memory in program tracing. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-13, 2021.
John Duncan. The multiple-demand (md) system of the primate brain: mental programs for intelli- gent behaviour. Trends in cognitive sciences, 17(4):172-179 2010.

---

Under review as a conference paper at ICLR 2022
Russell A. Epstein and Nancy Kanwisher. A cortical representation of the local visual environment. Nature, 392:598 601, 1998.
Evelina Fedorenko, Po-Jang Hsieh, Alfonso Nieto-Castañón, Susan Whirfeld-GGabrilli, and Nancy Kanwisher. New method for fmri investigations of language: defining rois functionally in indi- vidual subjects. Journal of neurophysiology, (04(2):1177111) 2010.
Evelina Fedorenko, John Duncan, and Nancy Kanwisher. Broad domain generality in focal regions of frontal and parietal cortex. Proceedings of the National Academy of Sciences, 110(41):16616- 16621, 2013.
Benjamin Floyd, Tyler Santander, and Westley Weimer. Decoding the representation of code in the brain: An fmri study of code review and expertise. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE), pp. 175-186. IEEE, 2017.
Jon Gauthier and Roger Levy. Linking artificial and human neural representations of language. arXiv preprint arXiv: 1.N::1990001244 2019.
Gary H Glover. Overview of functional magnetic resonance imaging. Neurosurgery Clinics, 22(2):
133-139, 2011.
Maurice H Halstead. Elements of Software Science (Operating and programming systems series). Elsevier Science Inc., 1977.
Yu Huang, Xinyu Liu, Ryan Krueger, Tyler Santander, Xiaosu Hu, Kevin Leach, and Westley Weimer. Distilling neural representations data structure manipulation using fmri and fnirs. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pp. 396-407. IEEE, 2019.
David H. Hubel and Torsten N. Wiesel. Receptive fields of single neurones in the cat's striate cortex. The Journal of Physiology, 148, 1959.
HuggingFace. Codeberta a roberta-like model trained on the codesearchnet dataset from github. HuggingFace 2020. URL hetts:///houggnnngaaee...////ogg
CodeBERTa- smal 1-v1
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. Code- searchnet challenge: chalennge Evaluating the state of semantic code search. CoRR, abs/1909.094366, 2019. UtRt://p://arat org/ abs/ 09436
Yoshiharu Ikutani, Takatomi Kubo, Satoshi Nishida, Hideaki Hata, Kenichi Matsumoto, Kazushi Ikeda, and Shinji Nishimoto. Expert programmers have fine-tuned cortical representations of source code. Eneuro, 8(1), 2021.
Anna A Ivanova, Shashank Srikant, Yotaro Sueoka, Hope H Kean, Riva Dhamala, Una-May O'Reilly, Marina U Bers, and Evelina Fedorenko. Comprehension of computer code relies pri- marily on domain-general executive brain regions. Elife, 9:e58906, 2020.
Shailee Jain and Alexander Huth. Incorporating context into language encoding models for fmri. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Yukiyasu Kamitani and Frank Tong. Decoding the visual and subjective contents of the human brain. Nature neuroscience, 855)66796 2005.
Nancy Kanwisher, Josh McDermott, and Marvin M. Chun. The fusiform face area: A module in human extrastriate cortex specialized for face perception. The Journal of Neuroscience, 17:4302 - --111,111 1997.
Nikolaus Kriegeskorte.
analysis: from stimulus decoding to computational- model testing. Neuroimage, 56(2):411 111-2)24111111111 2011.
Ryan Krueger, Yu Huang, Xinyu Liu, Tyler Santander, Westley Weimer, and Kevin Leach. Neuro- logical divide: an fmri study of prose and code writing. In 2020 IEEF/ACM 42nd International Conference on Software Engineering (ICSE), pp. 678-690. IEEE, 2020.
Yun-Fei Liu, Judy Kim, Colin Wilson and Marina Bedny, Computer code comprehension shares neural resources with formal logical inference in the fronto-parietal network. Elife, 9:e59340, 2020.
12

---

Under review as a conference paper at ICLR 2022
Thomas J McCabe. A complexity measure. IEEE Transactions on software Engineering, pp. 308- 320, 1976.
Tom M Mitchell, Svetlana V Shinkareva, Andrew Carlson, Kai-Min Chang, Vicente L Malave, Robert A Mason, and Marcel Adam Just. Predicting human brain activity associated with the meanings of nouns. science, 230(5880)1111111119 2008.
Kenneth A Norman, Sean M Polyn, Greg J Detre, and James V Haxby. Beyond mind-reading: multi-voxel pattern analysis of fmri data. Trends in cognitive sciences, 10(9):424-430, 2006.
Sam Norman-Haignere. Nancy G Kanwisher, and Josh H McDermott. Distinct cortical pathways for music and speech revealed by hypothesis-free voxel decomposition. Neuron, 1866):22111126, 2015.
Sam V Norman-
Norman-Haiinnere, Haignere, Nancy Kanwisher, Josh H McDermott, and Bevil R Conway. Divergence in the functional organization of human and macaque auditory cortex revealed by fmri responses to harmonic tones. Nature neuroscience, 22(7): 1057-1000,, 2019.
Paul Nuyujukian, Jose Albites Sanabria, Jad Saab, Chethan Pandarinath, Beata Jarosiewicz, Chris- tine H Blabe, Brian Franco, Stephen T Mernoff, Emad N Eskandar, John D Simeral, et al. Cortical control of a tablet computer by people with paralysis. PloS one, 13(11) (ee1))e0004566 2018.
Christophe Pallier, Anne- anne-Domniqqu Devauchelle, and Stanislas Dehaene. Cortical representation of the constituent structure of sentences. Proceedings of the National Academy of Sciences, 108 (6):2522 2527, 2011.
Joonkoo Park, Andrew Hebrank, Thad A. Polk, and Denise C. Park. Neural Dissociation of Num- ber from Letter Recognition and Its Relationship to Parietal Numerical Processing. Journal of Cognitive Neuroscience, 24(1):39-50, 2012.
Norman Peitek, Janet Siegmund, Chris Parnin, Sven Apel, Johannes C Hofmeister, and André Brechmann. Simultaneous measurement of program comprehension with fmri and eye track- ing: A case study. In Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pp. 1-10, 2018
Norman Peitek. Sven Apel. Chris Parnin, André Brechmann, and Janet Siegmund. Program com- prehension and code complexity metries: An fmri study. 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pp. 524-554444 IEEE, 2021.
Thad A. Polk. Matthew Stallcup, Geoffrey K. Aguirre, David C. Alsop, Mark D'Esposito, John A. Detre, and Martha J. Farah. Neural Specialization for Letter Recognition. Journal of Cognitive Neuroscience, 14(2):145-159, 2002.
Chantel S Prat, Tara M Madhyastha, Malayka J Mottarella, and Chu-Hsuan Kuo. Relating natural language aptitude to individual differences in learning programming languages. Scientific reports, 10( 10(1):1-1,, 2020.
Ruchir Puri, David Kung. Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladmir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks, 2021.
Edmund T. Rolls, Chu-Chung Huang, Ching-Po Lin, Jianfeng Feng, and Marc Joliot. Automated anatomical labelling atlas Neurolmage, 206:1 16189, 2020.
Emmanull Roux, Vincent Lubrano, Valerie Lauwers-Cancee Carlo Giussani, and Jean- François Démonet. Cortical areas involved in arabic number reading. Neurology, 70:210-217, 2008.
Martin Schrimpf, Idan Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Joshua Tenenbaum, and Evelina Fedorenko. Artificial neural networks accurately predict lan- guage processing in the brain. bioRxiv, 2020.
Dan Schwartz, Mariya Toneva, and Leila Wehbe. Inducing brain-relevant bias in natural language processing models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buu E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019.
13

---

Under review as a conference paper at ICLR 2022
Martin Shepperd. A critique of cyclomatic complexity software metric. Software Engineering Journal, 3(2):30-36, 1988.
Janet Siegmund, Christian Kästner, Sven Apel. Chris Parnin, Anja Bethmann, Thomas Leich, Gunter Saake, and André Brechmann. Understanding understanding source code with functional mag- netic resonance imaging. In Proceedings of the 36th International Conference on Software Engi- neering, pp. 378-389, 2014.
Janet Siegmund, Norman Peitek, Chris Parnin, Sven Apel, Johannes Hofmeister, Christian Kästner, Andrew Begel, Anja Bethmann, and André Brechmann. Measuring neural efficiency of program comprehension. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software En- gineering, pp. 140-150, 2017.
Shashank Srikant, Sijia Liu, Tamara Mitrovska, Shiyu Chang, Quanfu Fan, Gaoyuan Zhang, and Una-May O' O'Reilly. Generating adversarial computer programs using optimized obfuscations. In International Conference on Learning Representations, 2021. htthttt:://// /oppr//veee/eeeeee.. net/ Lttr/Eouuunddd-HHPPPP 1d=PH5PH920. 4
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neu- ral networks. In 7. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Wein- berger (eds.), Advances
Neural Information Processing Systems, volume 27. Curran Asso- ciates, Inc., 2014. URL /ttpsstttss///prreeeennn nee.gssneerrii neurips 2ttp:///ee..pppss.a////////222/2 a14ac55a4 (2laac55442747255555 777444755944ee wtw.:6444eccc37743333233 Paper. .pdf]
Fabian David Tschopp. Michael B. Reiser, and Srinivas C.
Turaga. A connectome based hexagonal lattice convolutional network model of the drosophila visual system, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL httpp:::/proceeeddddit 3f5ee243547dee wtt.eee11oo00331113333a5ee0 Paper. pdf
U. V. Luxburg, S. Bengio,
neurips...pppp.. httpp//ss...rrrssoe..p//////////
Shaonan Wang, Jiajun Zhang, Nan Lin, and Chengqing Zong. Probing brain activation patterns by dissociating semanties and syntax in sentences. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):9201 -9208, Apr. 2020. doi: 10. 1..6909/aai...o http:/aaa.ii33i......oo v34105.6477..
Daniel Yamins, Ha Hong. Charles Cadieu, and James DiCarlo. Hierarchical modular opti- mization of convolutional networks achieves representations similar to macaque it and human ventral stream. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Wein- berger (eds.), Advances in Neural Information Processing Systems, volume 26. Curran Asso- ciates, Inc., 2013. URL https:::: httttp::/pproceeddngg.. neur wurerursps.... http:r/ps...cppp/////22333. 22
pdd
Zhilin Yang. Zihang Dai, Yiming Yang. Jaime Carbonell, Russ Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In H. Wallach, H. Larochelle, A. Beygelzimer. F. d'Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- nwwriis........................s httpppps.../ppppe///////// ciates, Inc., 2019, URL http:////.ppppcee...........///a
730e666333eeeee cc69 www.cccccpppp....eee
Daniel Zügner, Tobias Kirschstein, Michele Catasta, Jure Leskovec, and Stephan Günnemann. Language-agnostte representation learning of source code from structure and context. In Interna- tional Conference on Learning Representations (ICLR), 2021.
14

---

Under review as a conference paper at ICLR 2022
A BRAIN REGIONS
Regions of Interest (ROIs). We investigate the representations of code in well-studied systems of brain regions-the MD system, Language system, Visual system, and the Auditory system (details below). A region (also referred to as parcel) here denotes contiguous chunk of brain mass involved in a cognitive task. A system of regions (also referred to network) can comprise multiple disjoint regions that exhibit shared activity patterns across range of tasks. For many cognitive tasks, a region marks only the approximate location of voxel populations involved in a cognitive task. Functional ROIs (fROIs) are voxels within these broad regions that respond most strongly to given task (language, working memory, efc.). The use of fROIs enables accounting for the exact anatomical locations of these task-sensitive voxels, which vary across individuals.
Multiple Demand (MD) system. Located in the prefrontal and parietal areas of the brain, this system of regions is active in host of tasks requiring working memory and general problem solving skills, including math and logic (Duncan 2010 Fedorenko et al. 2013 Amalrii & Dehaene (019).
Language system. These regions have been identified respond to both comprehension and pro- duction of language across modalities (written, speech, sign language), respond to typologically di- verse languages (>50 languages, from across 10 language families), form functionally integrated system, reliably and robustly track linguistic stimuli, and have been shown be causally important for language (Clark Cummings) 2003) Fedorenko et al., 2010: Blank Fedorenko 2017 Ayyash ct al. 2021).
Visual system. These regions in the occipital lobe of the brain respond to visual input, ranging from level features like lines and edges, through intermediate categories like shapes, letters. and numbers, through higher order structures like faces and scenes. (Hubel & Wiesel 1959. Polk al., 2002
Epstein & Kanwisher 1998 Kanwisher et al. 1997).
Auditory system. The auditory system is located in the superior temporal region of the brain. This region uniquely encodes pitch, speech, and music, but is not involved in high-level language com- prehension and production (Norman-Haignee programming language comprehension, we use the activity seen in the auditory system as a negative control.
al. 2013 2019). In our experiments pertaining to
15

---

Under review as a conference paper at ICLR 2022
B METHOD DETAILS
Selecting brain representations. For each trial in the fMRI experiment, stimulus responses in each voxel were extracted from the parameters of a General Linear Model (GLM) fit to the time- varying BOLD signal in which each experimental condition was modeled with a boxcar function convolved with the canonical hemodynamic response function (HRF), For the localizer experiments, conditions were modeled as the entire block. For the Python program comprehension experiment, individual programs were modeled using the period from the onset of the code/sentence problem until the button press. The predictors for the GLM included trial II) (equívalent to problem ID), run number, and motion regressors. The voxels were then filtered using gray-matter masking and (for MD and the Language systems) network localization. Although fMRI measurements return whole brain responses, only thin layer of cortex dubbed gray matter contains BOLD signal of interest to these analyses. Gray matter voxels were selected using a Bayesian segmentation of the anatomical brain image into standard tissue types, and then returning the set of indices where the posterior gray matter probability exceeds 0.70 (Ashburner et al.) 1999). Next, these sets of voxels were filtered separately for each of the brain regions outlined in Appendix [ For the visual and auditory networks, primary sensory areas were identified using an anatomical atlas (Rolls et al. 2020j. For the MD and the Language systems, voxels were functionally localized as those containing the top 10% of responses to their respective functional localizer tasks, as described in Ivanova et al. (020). See Fedorenko et al. (2010) for a discussion of the functional localization approach as It pertains to the language network. GLM modeling and gray matter segmentation was performed using SPM12; functional localization was performed using the toolbox released by [vanova et al. (2020). Once voxel responses within each brain region were extracted for each trial of each run, some additional preprocessing was required before finalizing the brain representations, and passing them to downstream models. Due to differences in MRI sensitivity across runs, each of the 12 runs of programs for a given subject appeared with nonuniform mean and variance. In order to normalize these signals, so as to leverage data across participant's 's entire scanning session, brain representations were z-transformed within each run to achieve common scale. In order to avoid data-leakage from this preprocessing step, all downstream analyses were cross- valsddaadd via a leave- one-run-out approach, such that no intra-run rescaling could be used for prediction.
MVPA cross-validation and hy hyperparameters. For each brain system and each code property or code model, we run a separate MVPA analysis. For each subject, train separate ridge regres- sion model on each cross-validatt using the brain representations from the left-out run. We use L2 regularization to control for model complexity, and employ a leave-one-out cross validation scheme to select A. We score resulting pre- dictions using series of different metrics to maximize interpretability. For the categorical code properties, we report classification accuracy of model estimates.
leave-one-run-owt fold, and then predict the remaining targets
For the continuously scaled code properties, we report Pearson correlation between the model es- timates and true targets. For the model representations, we report rank accuracy between model representation predictions and the true model representations of left-out samples. We calculate rank accuracy as the percentile of a predicted and true target pairing within the full set of pos- sible pairings sorted by a Euclidean distance metric. Rank accuracy was chosen for this experi- ment as it returns a more interpretable 50% baseline through which to evaluate the brain to model mappings. Following metric calculation each cross-validation fold, we take the mean of those estimates to report an out-of-sample score over the entire dataset for each subject. We then take the mean of those scores across subjects to derive an overall performance measure. As a baseline, we repeat the above process in its entirety 1000 times for each MVPA analysis, to provide null permutation distribution against which to compare scores. These null distributions are fit with univariate Gaussian and used calculate z-statistics and p-values for MVPA scores. All statistical tests were corrected for false discovery rate. The code to run our MVPA analysis is available in htt.bb////.oomooussscoomm
https://githi
Code properties. The code properties consider are the following
Token count. The total number of tokens that appear in the program. This is a sanity check to measure brain activity against, since it is natural to expect the activity in the brain to be correlated the length of a program, and in general, the amount of content being comprehended. By their design, the dataset had program lengths with a small standard deviation.
Node count. The total number of AST nodes that appear in a program. Similar to the number of tokens, we verify if brain activity correlates a proxy for the amount of syntactic content in it. Runtime steps. We execute the programs and measure the number of instructions the program steps through. While two programs can have the same program length, the number of instructions executed can differ (e.g. for I: 10 vs. for 1 :0). We measure if any brain regions capture the number of mental operations needed to compute the output of program.
16

---

Under review as a conference paper at ICLR 2022
Bytecode ops. We execute the programs and count the number of bytecode operations performed.
This metric should mimic the number of mental operations performed needed to compute the output of program.
Cyclomatic complexity. This metric (McCabe, 1976) is used to measure the general complexity of software systems. It is defined function of the number of nodes, edges, and the connected components program's 's control flow graph. While its efficacy as a metric to measure software complexity has been contested (Shepperd 1988), we include to see if brain activity is correlated to any explicit syntactic constructs of program.
defined measure how difficult any piece of software would be for a programmer to comprehend or write. is defined as a function of the number of tokens, operations, vocabulary that appears in a program.
Halstead difficulty. This metric (Halstead 1977)
code
sentence
height 5 weight 100
Your height is 5 feet and your weight is 100 pounds. The BMI is defined as the ratio between the weight and the square of the height of a person. What your BMI?
bmi = weight / (height"height) print(bmi)
Figure 6: An example of code problem and its sentence equivalent from Ivanova et al. (020)
The inter-correlations between these metrics have been tabulated in Table 18 Appendix G
Code models Model details. We evaluate the following models in this work-
bag-of-words, TF-IDF These are count-based language models. They predict the likelihood of a token appearing in a program based on vocabulary statistics and frequencies.
seq2seg (Sutskever et al. 2014), Code Transformer (Zügner et al. 2021), CodeBERTa (Hug-
gingFace 2020). We evaluate these three autoencoder (AEE models with increasing model com- plexity. AE based pretraining reconstructs the original program from corrupted input.
XLNet (Yang et al. 2019). We evaluate an auto-regressive (AR) model with model complexity similar to that of transformers and CodeBERTa. AR language modeling estimates the probabil- ity distribution of a text corpus one direction-eithe forward or backward, while AE models capture dependencies in both directions. We use an AR model to mimic a top-down, single pass comprehension style by humans.
Random embedding. We compare the results of the above models against an aggressive baseline provided by using unique coussian-distributee random vectors for the token embeddings a vocabulary, and returning the sum of these token embeddings across a program. The resultant embedding is not transformed by any model or any weights-it instead serves proxy for the tokens that appear in the program. This sets higher bar than the mull-distribution labeling baseline (Section 5.2).
Code models Configuration details. In setting up our code model bench, we aimed to select a collection of models ranging in their complexity, namely bag-of-words, TF-IDF (Term frequency- Inverse document frequency), seq2seq, XLNeí, Toderransformer, and CodeBERTa, as well Random embedding embedding model.
The hag-of-wordss TF-ID, seq2seq, and Random embedding models all use the same custom tokenizer defined by the current authors, whereas XLNet, TodeTransformes and Code use
tokenizers defined by the original training authors (Zügner et al., 2021: HuggingFace 2020).
The tokenizer set up for the models trained by the current authors establishes unique token in the vocabulary for each Python keyword, each Python builtin function, one token for all numeric types, one token for all string types, and N tokens for each of the N variables in a given program. In the case of the Random embedding baseline, this tokenizer was used to map each token in piece of source code distributed random matrix (D 128). The sum of these random token projections was calculated to return a unique embedding for each program.
index in the vocabulary. which in turn used to index Gaussian
For bag-of- words and TF-IF, the validation split of the code-search-net Python dataset (Husain et al. 2019) was used to enumerate vocabulary and token occurrence statistics. These data were then used transform each program to vector where each dimension represented the raw (bag- of- words or document-weighted (TF-IDF) count of the unique items in the vocabulary. If given set of programs viewed by a subject never included specific token, leading that column equal the zero vector, then those dimensions would be filtered out.
17

---

Under review as a conference paper at ICLR 2022
We used the dataset released by Project Codenet (Puri al. 2021 to pretrain seq2seq model (Sutskever et al. 2014). The dataset contains Python programs that are solutions to olympiad- style programming problems data structures and algorithms. We trained a seq2seq model with a GRU unit for 15 epochs, with a dropout probability of 0.2, dot product attention, and a maximum
sentence length of 500.
For XLNet, Code Transformer, and CodeBERTa, each program was passed through that model's tokenizer and model pipeline as defined by the original authors. The representations of each pro- gram were extracted from the pretrained encoders. XLNet and trained the Python subset of the code-search-nee dataset, whereas CodeBERTa originally trained on the full code-search-net dataset, which also incorporates go, java, javascript, php, and ruby.
were originally
18

---

Under review as a conference paper at ICLR 2022
C ADDITIONAL RESULTS
C.I EXPERIMENT
Using representations from localized brain regions, we attempt to decode static and dynamic prop- erties of comprehended code, and learn maps to code model representations of that same code. We report decoding performance of each brain region to the original Ivanova al. (2020) code prop- erties Table T the complexity-related code properties
Table 2 and the code model mappings in Table 3 We find that the MD system and the Language system decode all code properties and code models except variable language significantly above baseline, established through null permutation test. The Visual system decodes 4 of the code properties and 6 of the 7 code models, whereas the Auditory system only decodes the code vs sentences property.
Brain Representation Code Properties
MD
Language
Visual
Auditory
Empirical Baseline
Code vs. Sentence
0.55
0.88 (+0.33)
0.87 (+0.32)
0.88 (+0.33)
0.64 (+0.09)
Control Flow
0.33
0.49 (+0.16)
0.45 (40.12)
0.39 (40.06)
0.36 (+0.03)
Data Type
0.49
0.63 (+0.14)
0,58 (+0.09)
0.61 (+0.12)
0.53 (+0.04)
Variable Language
0.50
0.53 (+0.03)
0.51 (+0.01)
0.53 (+0.03)
0.50 (-0.00)
Table I: Brain region decoding performance on original |vaoova et al. (2020) code properties. Scores represent classification accuracy and are contrasted with an empirical baseline from the null permu- tation analysis. Values in parentheses units above baseline.
Brain Representation
MD)
Language
Visual
Auditory
Code Properties
Static Analysis
0.17
0.24
0.09
0.00
Dynamic Analysis
0.33
0.20
0.17
0.12
Table 2: Brain region decoding performance on static analysis and dynamic analysis code properties. Scores represent Pearson correlation between predicted and true code properties.
19

---

Under review as a conference paper at ICLR 2022
C.2 EXPERIMENT 2
Brain Representation
MD
Language
Visual
Auditory
Code Models
Empirical Baseline
Random Embedding
0.50
0.36 (+0.06)
0.35 (+0.05)
0.54 (+0.04)
0.53 (+0.03)
Bag Of Words
0.50
0.58 (+0.08)
0.56 (+0.06)
0.54 (+0.04)
0.52 (+0.02)
TF-IF
0.50
0.57 (+0.07)
0.54 (+0.04)
0.53 (+0.03)
0.52 (+0.02)
Seq2Seq
0.50
0.57 (40.07)
(40.04)
0.51 (40.01)
0.51 (40.01)
XLNet
0.50
0.57 (+0.07)
0.55 (+0.05)
0.54 (+0.04)
0.52 (+0.02)
Code Transformer
0.50
0.59 (+0.09)
0.56 (+0.06)
0.55 (+0.05)
0.52 (+0.02)
CodeBERTa
0.50
0.60 (+0.10) 0.57 (+0.07)
0.58 (+0.08)
0.53 (+0.03)
Table 3: Brain region decoding performance code model mappings. Scores represent rank ac- curacy and are contrasted with an empirical baseline from the null permutation analysis. Values in parentheses are units above baseline.
We additionally evaluate whether code models contain linearly decodable information about the code properties we explore in this work (Section 4.2p. The models decode with near perfect accu- racy on most classification tasks (avg. accuracy= 0.96 0.01), and with high correlations on the regression tasks (avg. = 0.89 + 0.02). This is expected for the classification tasks, since the dataset contains tokens which help with perfectly separable decision boundaries. For example, for the control flow code property, the dataset contains programs with if, for, or neither, but never both. These results suggest that the code models we evaluate faithfully encode the set of properties we evaluate them on.
Code Properties
Control Flow
Data Type 0.48
Dynamic Analysis
Static Analysis 0.00
Empirical Baseline Model Representation
0.31
0.00
Random Embedding
1.00
0,94
0.88
1.00
Bag Of Words
1.00
0.92
0.93
0.98
TF-IDF
1.00
0.94
0.94
0.89
Seq2Seq
0.95
0.97
0.87
0.91
XLNet
0.92
0.92
0.76
0.88
CodeTransforie
1.00
0.95
0.90
0.91
CodcBERTa
0.96
0.98
0.81
0.79
Table Decoding performance of all models on all tasks. Scores represent classification accuracy for control flow and data type, and Pearson correlation for the remaining benchmarks. These can be contrasted with an empirical baseline from the null permutation analysis.
20

---

Under review as a conference paper at ICLR 2022
D ANALYSIS OF VARIANCE
We run series one-way ANOVA statistical tests to assess whether decoding performance varies across brain regions for each code property and code model, and whether decoding performance varies across code models for each brain region. We report these results in Tables 6, and We find that decoding performance varies across brain region for all code properties and code models, except variable language property and the Random embedding model, and decoding performance varies across code models for the MD system and the Visual system.
p (corrected)
Is Significant?
Code Property
Code vs. Sentence
53.57
4.03e-20
2.42e-19
Control Flow
12.52
6.13e-07
1.84e-06
Static Analysis
6.71
3.82e-04
7.65e-04
Data Type
4.46
5.66e-03
8.50e-03
Dynamic Analysis
4.15
8.38e-03
1.01e-02
Variable Language
0.82
4.84e-01
4.84e-01
0
Table Results from statistical testing of variance across brain regions for each code property.
F
p
p (corrected)
Is Significant?
Code Model
CodeTransformer
11.85
1.25e-06
7.95e-06
CodeBERTa
11.28
2.27e-06
7.95e-06
TF-IDF
8.46
5.08e-05
9.48e-05
Seq2Seq
8.40
5,42e-05
9.48e-05
Bag Of Words
5.66
1.33e-03
1.86e-03
XLNet
3.48
1.90e-02
2.22e-02
Random Embedding
2.32
8.04c-02
8,04e-02
Table 6: Results from statistical testing of variance across brain regions for each code model.
F
(corrected) Is Significant?
Brain Region
Visual
4.00
9.17e-04
3.67e-03
MD
2.72
1.54e-02
3.08e-02
Language Auditory
2.18
4.70e-02
6.27e-02
0.43
8.57e-01
8.57e-01
Table 7: Results from statistical testing of variance across code models for each brain region.
21

---

Under review as a conference paper at ICLR 2022
E PAIRWISE ANALYSIS
To extend the findings from the ANOVA analyses towards specific pairwise comparisons between brain regions and code models, we compare scores between brain regions for each code property or code model, and compare scores between code models and subset of code properties for each brain region, using two-sample two-tail t-tests with unequal variance. We report all pairwise brain region comparisons in Tables and TO and report all pairwise code property and model compar- isons in Tables 9 and I1 To highlight a few key results from code properties, we find that the MD system decodes runtime steps and control flow significantly better than the Visual system (Table Additionally, the Language system decodes static analysis and control flow significantly better than Visual system. Moving onto code models, we find that the MD system decodes CodeBERTa, seq2seq, and TF-IDF significantly more accurately than the Language system, and additionally de- codes TodeTransformer, seq2seq, TF-IDF, and bag-of- words significantly more accurately than the Visual system (Table I0). Finally, an investigation into model complexity reveals that CodeBERTa significantly more accurately decoded from the MD system than the Random embedding model.
Brain Region A
Brain Region B
(corrected)
Is Significant?
Code Property
Code Sentence
Language
Auditory
8.82
5.33e-1
6.40e-10
Code Sentence
MD
Auditory
9.79
1.28e-11
4.62e-10
Code vs. Sentence
Visual
Auditory
9.06
2.65e-11
4.78c-10
Control Flow
Language
Auditory
4.28
1.02e-04
6.100-04
Control Flow
Language
Visual
2.60
1.26e-02
3.30e-02
Control Flow
MD
Auditory
5.59
1.65e-06
1.49c-05
Control Flow
MD
Visual
3,92
2.95e-04
1.52e-03
Data Type
MD
Auditory
3.16
2.98e-03
1.21e-02
Data Type
Visual
Auditory
2.93
5.33e-03
1.88c-02
Dynamic Analysis
MD
Auditory
3.13
3.03e-03
1.21e-02
Dynamic Analysis
MD
Visual
2.59
1.28c-02
3.30e-02
Static Analysis
Language
Auditory
4.70
2.97e-05
2.14e-04
Static Analysis
Language
Visual
2.92
5.75e-03
1.88e-02
Static Analysis
MD
Auditory
2.79
7.68e-03
2.30e
Code Sentence
Language
Visual
-0.28
7.83e-01
8.30e-01
Code vs. Sentence
MD
Language
0.49
6.27e-01
6.84e-01
Code vs. Sentence
MD
Visual
0.18
8.55e-01
8.55e-01
Control Flow
MD
Language
1.44
1.58e-01
2.84e-01
Control Flow
Visual
Auditory
1.50
1.40c-01
281c-011
Data Type
Language
Auditory
2.30
2.62e-02
6.30e-02
Data Type
Language
Visual
-0.94
3.55e-01
4.91c-01
Data Type
MD)
Language
1.48
1.47e-01
2.81c-01
Data Type
MD
Visual
0.64
5.24e-01
6.18c-01
Dynamic Analysis
Language
Auditory
1.29
2.03e-01
3.04c-01
Dynamic Analysis
Language
Visual
0.57
5.73e-01
6.45e-01
Dynamic Analysis
MD
Language
2.11
4.01e-02
9.03e-02
Dynamic Analysis
Visual
Auditory
0.75
4.55e-01
5.85e-01
Static Analysis
MD
Language
-1.34
1.89e-01
3.04e-01
Statie Analysis
MD)
Visual
1.32
1.94c-01
3.04e-01
Static Analysis
Visual
Auditory
1.47
1.48c-01
2.81e-01
Variable Language
Language
Auditory
0.66
5.15e-01
6.18c-01
Variable Language
Language
Visual
-0.63
5.32e-01
6.18c-01
Variable Language
MD
Auditory
1.40
1.68e-01
2.88c-01
Variable Language
MD
Language
0.84
4.06e-01
5.41e-01
Variable Language
MD
Visual
0.23
8.20e-01
8.44e-01
Variable Language
Visual
Auditory
1.23
2.26e-01
3.25e-01
Table 8: Results from pairwise t-tests of brain regions for each code property. +t reflects A > B, whereas -1 reflects A < B.
22

---

Under review as a conference paper at ICLR 2022
Code Property
Code Property
p
p (corrected)
Significant?
Brain Region
Auditory
Static Analysis
Dynamic Analysis
-1.93
6.00e-02
1.20e-01
Language
Static Analysis
Dynamic Analysis
0.85
4.00e-01
4.00e-01
MD
Static Analysis
Dynamic Analysis
-2.45
1880-02
7.18e-02
0
Visual
Static Analysis
Dynamic Analysis
1..3
1 1.88c-01
2.51e-01
0
Table 9; Results from pairwise t-tests of continuous code metries across brain regions. Only this
subset of properties was selected so as evaluate scores with a consistent metric and baseline. reflects A > B, whereas -t reflects < B.
Brain Region A
Brain Region B
(corrected)
Is Significant?
Code Model
Bag Of Words
Language
Auditory
2.70
9.69c-03
2.60e-012
Bag Of Words
MD
Auditory
4.42
6.03e-05
6.33e-04
Bag Of Words
MD
Visual
1.26e-02
2.93e-02
CodeBERTa
Language
Auditory
3.29
1.94e-03
7.41e-03
CodeBERTa
MD
Auditory
5.80
5.85e-07
2.46e-05
CodeBERTa
MD
Language
2.67
1.05c-02
2.60e-012
CodeBERTa
Visual
Auditory
3.80
4.28c-04
2.25e-03
Code Tranffrrsfrmm
Language
Auditory
4.12
1.55e-04
1.09e-03
Code Transformer
MD
Auditory
5.61
1.42e-06
2.99c-05
Code Transorranorrm
MD)
Visual
2.67
1.05e-02
2.60l-112
Code Transformer
Visual
Auditory
3.09
3.40a-03
1.19e-02
Random Embedding
MD
Auditory
2.69
9 9.95e-03
2.60e-02
Seq2Seq
MD
Auditory
3.94
2.72e-04
1.63e-03
Sey2Seq
MD
Language
2.48
1.71e-02
3.78c-02
Seq2Seq
MD
Visual
4.25
1.16c-04
9.71e-04
TF-IDF
MD
Auditory
4.95
1.06c-05
1.48e-14
TF-IDF
MD
Language
2.86
6.39e-03
2.06e-02
TF-IDF
MD
Visual
3.42
1.33e-03
6.20e-03
XI.Net
MD
Auditory
3.39
1.53e-03
6.45e-113
Bag Of Words
Language
Visual
1.29
2.03e-01
2.80e-01
Bag Of Words
MD
Language
1.50
1,40e-01
2.03e-01
Bag Of Words
Visual
Auditory
0.98
3.33e-01
3.88e-01
CadeBERTa
Language
Visual
-0.81
4.20c-01
4.65e-01
CodeBERTa
MD
Visual
1.66
1.04e-01
1.62e-01
Code Transformer
Language
Visual
0.69
4.92c-01
5.30e-01
Code Transformer
MD
Language
2.21
3.23e-02
6.46e-02
Random Embedding
Language
Auditory
1.70
9.56c-02
1.54c-01
Random Emhedding
Language
Visual
0.32
7.52c-01
7.81e-01
Random Emhedding
MD
Language
1.13
2.63e-01
3.18e-01
Random Embedding
MD
Visual
1.28
2.07e-01
2.80e-01
Random Embedding
Visual
Auditory
1.16
2.53e-01
3.18e-01
Seq2Sey
Language
Auditory
1.87
6.85e-02
1.20e-01
Seq2Seq
Language
Visual
1.98
5.36c-02
1.02--01
Scq2Scq
Visual
Auditory
0.14
8.87e-01
8.87c-01
TF-IDF
Language
Auditory
2.28
2.71e-02
5.69e-12
TF-IDF
Language
Visual
0.92
3.64e-01
4.13e-01
TF-IFF
Visual
Auditory
1.13
2.65e-01
3.18c-01
XI.Net
Language
Auditory
1.58
1.22c-01
1.83e-01
XLNet
Language
Visual
0.30
7.62e-01
7.81e-01
XLNet
MD
Language
1.71
9.49e-02
1.54e-01
XLNet
MD
Visual
1.91
6.25e-02
1.14e-01
XLNet
Visual
Auditory
1.13
2.64c-01
3.18e-01
Table 10: Results from pairwise t-tests brain regions for each code model.
+1 reflects A > B,
whereas -/ reflects A < B.
23

---

Under review as a conference paper at ICLR 2022
Code Model A
Code Model B
(eonected)
Is Signifeent!
Brain Region
MD
Ramdum Einbedding
CodeBERTa
9.63e-04
2.70--03
Visnl
CuleBERTs
Seq2Sey
511
7.0l-066
19--04
ColeBERTs
TF-IDF
1,6
7.96-04
2.70--02
CudeTmanformer
Seq2Sag
1.35
Audivery
8.23-01
Audivrr
CRds
CodeTTansformer
Audiory
CodeB
Sey2Seu
1.15
2.58c-01
6.37c-1
Andiory
CodeB
TF-IDF
4.68-01
740-01
Andiory
Code
NT.Nes
1.10-01
9,8c-01
Audiorry
Cod-Tmaeformer
Words
4.93.-01
7.00-01
Audiorry
Seg2Sey
0.30
7.61e-11
9.44-01
Audiory
TF-IDF
-0.27
7.87e-01
9455-01
0
Auditrry
Codeennnformrr
NL
-0.75
4 4.57c-01
7.40-01
0
Auditery
Rondom Finhediing
Rag Or Work
0.07
9 9.45--01
1.004000
Auditry
Rondom Emihdding
CodeBERTa
-0.16
K70x-01
9.75c-01
Auditory
Raadom Emhedding
CodeTrandormer
0.80
1.29c-011
7.35c-01
Auditory
Kaadom Embedding
Sea2Scq
1402
3 3.14c-01
6. 6.82--01
Anditery
Ramdum Embhedding
TF-IDF
0.58
5.68-01
7 7.74-00
Anditory
Randum Emhedding
X1.Net
0.08
9.40c-01
1 1.0++000
Anditory
SaytSeq
Roe Or Words
-191
166p-01
6 83-00
Anditory
SegtSeq
TF-DFF
-1.56
5.81c-01
7. 7.74-001
Auditory
XL
Bag Of Words
0.00
1.00e4000
1.00.000
Auditory
XLNNer
Sea2Seg
0.9%
3.33p-01
6.83-001
Aulitory
TF-IDF
0.52
6.77-011
Langsage
TF-IDF
1.64
110e-01
Langaage
CadeRFRTs
Ray Words
7.72-011
Langsage
Code
CodeTrandformer
5.0e-01
7.74-001
Langsage
Code
Sea2Slag
282
7.176-03
1.15c-01
Language
CodellEIia
TF-DF
1.70-002
1,33c-01
Language
Code
XLNes
8.27--02
81c-00
Langsage
Words
Langage
CodeTundfrrerr
2
2.Ke-122
1.33-00
Language
CodeTmnsformer
TE-DF
491c-02
2.37c-01
Langsage
codeTmasfrmmer
XLNet
1.35
1.83e-01
4.81c-11
Language
Raadom Embedding
Words
2.09-001
5.52c-01
Kaadom Embedding
CodeBtKKa
-205
4.1c-02
2.57e
Language
Random Embedding
CodeTTandfrrme
1159
1.20e
3.
Language
Random Finhedding
Seq25a
0.85
4 102:-01
745--01
Language
Raadom Emhedding
TF-DF
0.34
7. 7.33c-01
9. 19-011
Language
Random Emhedding
XLNNt
0.00
1 1.00x000
1.001000
Language
Seg25iq
Hag Or Words
-2.01
5.08e-02
2.57c.01
Language
Seq2Seq
TF-DF
-1.5
5.77e-01
01
Language
XLNNtt
Bag Of Words
-1.12
2.0% 01
6.46c
Language
X1 Net
Seq25cn
0.72
75p--01
744-0
Language
XLNNer
TE:DF
0.29
7.76e-01
9.550011
MD
Bog Words
TF-DFF
0.88
3.81e-01
6.83c001
MD
CodeslaKIa
Bog Of Words
1.84
7.18c-02
2.57.001
MD
Code
Codelhanaformer
0.74
4.0le-01
7.40-001
MD
Code
Sea2Seq
2.60
1.27c
1.28c.01
MD
CideB
TF"IDF
2.77
8.21e-03
1.15e.01
Code
XL.Ner
2996-012
1.77c-01
CodeT ranfforme
Bag Words
3.716-01
Sea2Scg
1.68
1.00-011
3:11c011
9.54c-02
Trunsfomme
1.67e-01
Embedding
7.67e-02
Ruadom Embedding
Sea25cq
4.86e-01
0.
Kaadom Embedding
TF-DF
299e-01
6.82c.01
Random Embedding
XLNes
3.76e
6182c
MD
Seg2Sey
Bag Of Words
0.95
3.45c 01
6.82c
MD
SealSoy
TF-DDF
-021
8.35e-01
9. 01
MD
XLNett
Bug Of Words
-0.7
5.05e-01
7.4c-01
MD
XLNNt
Seq25eq
0.21
8.31e-01
9.
MD
XLNNtt
TE-DE
0.85
9.65e-01
1.00+000
Visual
Big Of Words
TF-DF
0.35
5.84c-01
01
Visual
CoTe
Bag Of Words
2.43
1.9e 02
1133e
Visua
CodeBERTa
TolefTransfrrmr
1.86
6.97e-02
2.87-011
Visnl
CodeBERTs
XLNet
2.56
1.38--02
1.28c-01
Visnl
CudkTransforma
Bur Or Words
0.89
3 30e0-0
6 83-00
Visual
Code1 ransformet
IF-DDE
1.79
S.08e-02
2.8%-01
Visual
Code Iansformer
XLNNt
0.93
3.57c-01
6.82c001
Visual
Random Embedding
Bag Of Words
0.05
9.58c-01
1,00++00
Visul
Random Embedding
CodeBERTa
-.67
1.0le-02
1,25c.01
Visnl
Randum Embedding
CodeTTansformer
-0.5
3.48e-01
K2c-00
Visml
Rundum Embedding
Seq[5ey
2.11
4.76-002
2.25-01
Rundom Einheding
TF-IDF
0.70
4.86-01
740-001
Embedding
XLNes
9,70e-01
1,00c+00
Seg2Soq
IFF
1.2e-01
Bug Words
1.92
2.74-01
XI
TF-IDF
5.47-01
Table 11: Results from pairwise 1-tests of code models for each brain region. +1 reflects A B, whereas -/ reflects A < B.
24

---

Under review as a conference paper at ICLR 2022
F MULTI-SYSTEM REGRESSION ANALYSIS: SCORES AND SIGNIFICANCE
In order to investigate whether each of the brain systems included contribute unique information towards the decoding tasks, we combine brain representations from different systems in paired com- binations and evaluate effects on downstream decoding performance across all experiments. We find that the addition of MD L.S VS, relative to VS alone, improves downstream decoding of code vs sentences. The same effect is observed for the combination of MD and L. compared to either alone. These data suggest that the MD system and the Language system encode unique variance relevant to the decoding of code sentences, and this is above and beyond the information encoded in the Visual system each other individually. Additionally, VS, relative to VS alone, improves downstream decoding of control flow and data type. This same effect is observed for the addition of LS to Visual system for control flow. These data suggest that the MD system and the Language system encode unique information above and beyond information encoded in the Visual system for the control flow decoding task, and the MD system encodes unique information above and beyond information encoded in the Visual system for the data type decoding task (Table 15). We repeat this process for Experiment 2, where we find that the addition of MD to VS improves decoding for 5 models, the addition of MD to LS improves decoding for 4 models, and the addition of LS to Visual system improves decoding for models (Table 16).
observe that the addition of MD to
Brain Representation Code Properties
L+V
MD+L
MD+V
Empirical Baseline
Code Sentence
0.56
0.94 (+0.38)
0.94 (+0.38)
0.93 (+0.37)
Control Flow
0.33
0.45 (+0.12)
0.49 (40.16)
0.49 (+0.16)
Data Type
0.50
0.62 (+0.12)
0.63 (+0.13)
0.68 (+0.18)
Variable Language 0.50
0.53 (+0.03)
0.53 (+0.03)
0.53 (+0.03)
Table 12: Multi-system regression analysis on original Ivanova et al. (0200) code properties. Scores represent classification accuracy and are contrasted with an empirical baseline from the null permu- tation analysis. Values in parentheses are units above baseline.
Brain Representation
L+V
MD+L
MD+V
Code Properties
Dynamic Analysis
0.22
0.32
0.31
Static Analysis
0.19
0.21
0.20
Table 13: Multi-system regression analysis on static and dynamic code properties. Scores represent Pearson correlation between predicted and true code properties.
Brain Representation
L+V
MD+L
MD+V
Code Models
Empirical Baseline
Random Embedding
0.50
0.57 (+0.07)
0.56 (+0.06)
0.57 (+0.07)
Bag Of Words
0.50
0.57 (+0.07)
(+0.09)
0.59 (+0.09)
TF-IDF
0.50
0.56 (+0.06)
0.58 (40.08)
0.57 (+0.07)
Seq2Seq
0.50
0.55 (+0.05)
0.57 (+0.07)
0.56 (+0.06)
XLNet
0.50
0.57 (+0.07)
0.58 (+0.08)
0.57 (+0.07)
Code Taldssrasfrr
0.50
0.58 (+0.08)
0.60 (+0.10)
0.60 (+0.10)
Code BodeBERTa
0.50
0.60 (+0.10)
0.62 (+0.12)
0.62 (+0.12)
Table 14: Multi-system regression analysis on code model mappings. Scores represent rank accu- racy and are contrasted with an empirical baseline from the null permutation analysis. Values in parentheses units above baseline.
25

---

Under review as a conference paper at ICLR 2022
Brain Region A
Brain Region B
P
p (corrected)
Is Significant?
Code Property
Code vs. Sentence
L+V
Language
3,94e
5.20e-03
Code Sentence
L+VV
Visual
3.56
9.82c444
7.07e-03
Code vs. Sentence
MD+L
Language
3.77
5.96c-04
5.36e-03
Code vs. Sentence
MD+L
MD
3.83
4.33e-04
5.20e-03
Code vs. Sentence
MD+VV
MID
3.25
2.29c-03
1.37e-02
Code vs. Sentence
MD+V
Visual
2.96
5.36e-03
2,54e-02
Control Flow
L+V
Visual
5.63e-03
2.54e-02
Control Flow
MD+V
Visual
4.11
1.63-44
5.20c-03
Data Type
MD+V
Visual
2.86
6.41e-03
2.56c-02
Control Flow
L+V
Language
0.15
8.80c-01
1.00e+00
Control Flow
MD+L
Language
1,46
1.50c-01
3.18c-01
Control Flow
MD+L
-0.11
9.16e-01
1.00c+00
Control Flow
MD+V
9.73e-01
1.00+400
Data Type
L+V
Language
1.50
1.39c-01
3.13e-01
Data Type
L.VV
Visual
0.44
6.59c-01
9.49c-01
Data Type
MD+L
Language
1.93
6.00e-02
1.80e-01
Data Type
MD+L
MD
0.24
8.11e-01
9.94c-01
Data Type
MD+V
1.76
8.67c-02
2.23e-01
Dynamic Analysis
L+V
Language
7.01e-01
9.71e-01
Dynamic Analysis
L+V
Visual
1.00
3.24c-01
6.48c-01
Dynamic Analysis
MD+L
Language
1.97
5.51e-02
1.80e-01
Dynamic Analysis
MD+L
MD
-0.05
9.63e-01
1.00e+000
Dynamic Analysis
MD+V
MD
-0.28
7.84c-01
9.94e-01
Dynamic Analysis
MD+V
Visual
2,51
1.56e-02
5.61e-02
Static Analysis
L+V
Language
-0.94
3.54c-01
6.71e-01
Static Analysis
L+V
Visual
1.69
9.87e-02
2.37e-01
Statie Analysis
MD+L
Language
-0.70
4.89c-01
8.80e-01
Static Analysis
MD+L
MD
0.57
5.68e-01
9.25e-01
Static Analysis
MD+V
MD
0.50
6.17e-01
9.25e-01
Static Analysis
MD+V
Visual
1.76
8.50e-02
2.23e-01
Variable Language
L+V
Language
5.36e-01
9.18e-01
Variahle Language
L+V
Visual
-0.00
1.00++00
1.00c+00
Variable Language
MD+L
Language
0.53
5.98c-01
9,25e-01
Variable Language
MD+L
MD
-0.27
7.86e-01
9.94e-01
Variahle Language
MI+VV
MD
-0.22
8.28c-01
9.94c-01
Variable Language
MD+V
Visual
-0.00
1.00++00
1.00c+00
Table 15: Results from pairwise t-tests of paired brain systems with their individual components for each code property. +t reflects A > B, whereas -f reflects A < B.
26

---

Under review as a conference paper at ICLR 2022
Brain Region
Brain Region
P
p (corrected)
Is Significant?
Code Model
Bag Words
MD+V
Visual
3.01
4,41e-03
2.65e-02
CodcBERTa
L+V
Language
2.57
1.36c-02
4.76e-02
CodeBERTa
MD+L
Language
3.40
1,45e-03
1.52e-02
CodeBERTa
MD+V
Visual
2.62
1.17e-02
4.76e-02
Troe
MD+L
Language
2.84
6,80e-03
3.51e-02
Code' Transformrr
MD+V
Visual
3.30
1.87e-03
1.57e-02
Seq2Seq
L+V
Visual
3.02
4.21e-03
2.65e-02
Seq2Seq
MD+V
Visual
3,89
3.54e-04
9.44e-03
TF-IDF
L+V
Visual
2.80
7.53e-03
3.51e-02
TE-IDF
MD+L
Language
3.66
6.74c-04
9.44c-03
TF-IPF
MD+V
Visual
3.72
5,49c-04
9.44c-03
XI.Net
MD+L
Language
2.58
1.32c-02
4.76e-112
Bag Of Words
L+V
Language
0.64
5.24e-01
6.47e-01
Bag Of Words
L+V
Visual
1.77
8.35e-02
1.59e-01
Bag Of Words
MD+L
Language
2.25
2.92o-02
7.65e-112
Bag Of Words
MD+L
MD
0.82
4.15e-01
5.44e-01
Bag Of Words
MD+V
MD
0.60
5,49e-01
6.59e-01
CodeBERTa
L+V
Visual
1.46
1.52e-01
2.45e-01
Coda
MD+L
MD
0,88
3.84--01
5.20e-01
CodeBERTa
MD+V
MD
1.11
2.74c-01
4.05c-01
Code' Transformer
L+V
Language
1.61
1.15e-01
1.94--01
Code Transfommer
L+V
Visual
2.18
3.48e-02
8,47e-02
Code Transformer
MD+L
MD
0.36
7.21e-01
7.97e-01
Code Traeforrasorrm
MD+V
MDD
0.48
6.31e-01
7.16e-01
Random Emhedding
L+V
Language
1.72
9.15e-02
1.66e-01
Random Embedding
L+V
Visual
1.80
7.91e-02
1.58e-01
Random Embedding
MD+L
Language
9,46e-02
1.66e-01
Random Embedding
MD+L
MD
0.51
6.11e-01
7.13e-01
Random Embedding
MD+V
MD
1.09
2.80c-01
4.05e-01
Random Emhedding
MD+V
Visual
2,30
2.65c-02
7.56c-112
Seq2Seq
L+V
Language
1.10
2.78c-01
4.05e-01
Seq2Seq
MD+L
Language
2.43
1,99--02
6.41e-02
Seq2Seq
MD+L
MD
0.20
8.39e-01
8.39e-01
Seq2Seq
MD+V
MD
40.28
7.78e-01
8.17e-01
TF-IDF
L+V
Language
2.16
3.63e-02
8.47e-02
TF-IFF
MD+L
MD
1.00
3.22e-01
4.50e-01
TF-IDF
MD+V
MD
7.60e-01
8.17e-01
XLNet
L+V
Language
1.91
6.23e-02
1.31e-01
XeN
L+V
Visual
2.11
4.07e-02
8.99e-02
Net
MD+L
MD
0.79
4.35e-01
5.53e-01
XLNet
MD+V
MD
0.24
8.10e-01
8.30e-01
XLNet
MD+V
Visual
2.29
2.70e-02
7.56e-02
Table 16: Results from pairwise t-tests of paired brain systems with their individual components for each code model. +t reflects > B, whereas -t reflects < B.
27

---

Under review as a conference paper at ICLR 2022
G ALL CODE PROPERTIES: SCORES AND CORRELATIONS
We analyzed a series of code properties as part of the static and dynamic code analysis. As several of these properties (token count, node count, Halstead difficulty, cyclomatic complexity, and bytecode ops) were revealed to be highly correlated in a post-hoc analysis, we report only one measure for this subset in Experiment 1, token count, but include all scores here for completeness (Table [7]. We also report the correlation matrix between code properties that led us select token count as the representative property for this subset (Table 18).
Brain Representation
MD
Language
Visual
Auditory
Code Properties
Token Count
0.17
0.24
0.09
0.00
Node Count
0.12
0.20
0.03
0.01
Halstead Difficulty
0.11
0.17
0.10
-0.01
Cyclomatic Complexity
0.18
0.24
0.10
0.01
Bytecode Operations
0.15
0.18
0.03
0.02
Runtime Steps
0.33
0.20
0.17
0.12
Table 17: Brain region decoding performance on all static analysis and dynamic analysis code properties. Scores represent Pearson correlation between predicted and true code properties.
Datatype
Conditional
Iteration
Tokens
Nodes
Halstead
Cyclomatic
Runtime
Bytecode
Datatype
1.00
0.00
0.00
0.41
0.33
0.39
0.18
0.13
0.28
Conditional
1.00
-0.50
0.44
0.46
0.50
0.61
-0.41
0.49
Iteration
1.00
-0.10
-0.14
-0.35
-0.09
0.90
Tokens
1.00
0.97
0.89
0.80
0.08
0.95
Nodes
1.00
0.86
0.79
0.00
0.96
Halsteud
-0.17
Cyclomatic
1.00
0.02
0.84
Runtime
1.00
0.13
Bytecode
1.00
Table 18: Correlation matrix across all code properties. The control flow property was split into two binary properties here, conditional and iteration. Of relevance is that token count presents with r 0.8 for static analysis properties in the set and as such is used as the representative static analysis code property in Experiment 1.
28

---

Under review as a conference paper at ICLR 2022
ND
Language
Visual
Auditory
Vappping Sore thank 800k
0.50
060
TF-Dr
HagbfiWards
0.30
035
0.50
2:30
0.20
020
9.30
Codeeckia
Coderraratommer
0.5
682
0:7
0.50
0.00
0.50
230
2
22
23
27
200
23
22
2
27
24
200
Embooding Dimendsnnaing
Embeading Ommeserrlityy
Enrbedding Dimensinalitty
Cimbeslingg Dieeessniiyy
Figure 7: Sensitivity of brain representation mapping to model output dimensions. Each subplot contains the decoding results from a given brain network, and each line reflects a unique code model across a range of controlled embedding dimensions.
H SENSITIVITY OF BRAIN MAPPING TO MODEL OUTPUT DIMENSIONS
Of potential interest to the decoding framework is not just the complexity of code models, but their intrinsic dimensionality as well. In order to investigate to what extent brain model mappings are robust to changes in code model dimensionality, and to assess which model representations most sensitive compression and expansion, we rerun our current decoding framework while controlling for embedding size.
For each brain network to code model mapping task, prior to the MVPA analysis, we control for the dimensionality of the code model embedding via projection through a Gaussian random matrix Id xda drawn from N (0, 1/d2) where d1 is the original code model embedding dimensionality and d2 is the desired dimensionality.
We observe that models of lower complexity (e.g., bag-of-words, TF-IDF, seq2seq, XLNet) ap- pear relatively robust to compression, whereas the most complex models (c.g., CodeTransformer, CodeBERTa) gain considerable performance from higher dimensional expression, and suffer con- siderably when constrained.
This indicates that higher dimensions of the encoder in complex models encode relevant neural information. Additionally, these effects appear to be most pronounced when decoding from the brain region whose representations yield the strongest mappings, the MD system.
We note here however that we could be observing an interaction effect between model complexity and dimensionality output in these results. Since cannot fully control for the complexity of these models (by making them all 'equally complex') ) this experiment alone cannot drive definitive conclusions.
These results instead constitute a preliminary exploration into the effects of code model dimension- ality on brain to model representation mappings, and suggest avenue for future investigation.
29

---

Under review as a conference paper at ICLR 2022
I ROBUSTNESS OF RESULTS TO REGRESSION METRIC
For the decoding analysis of the continuous valued dynamic analysis and static analysis properties, is reasonable to ask why the Pearson correlation metric was chosen as opposed to RMS F,, as is typically customary for regression tasks. While we present the results using the Pearson correlation metric in the core results for interpretability via the zero-baselne, here confirm that the of an RMSEE metric leads to the same conclusions. As we see here, MD, LS, and Visual system decode the dynamic analysis property, and MD and LS decode the static analysis property with significantly lower RMSE than the null permutation baseline. These results precisely confirm and mirror the patterns observed in Figure 3
Brain Network
Code Property
RMSE
Null RMSE
Significant?
MD
Dynamic Analysis
3.49
4.03 0.08
MD
Static Analysis
7.68
8.27 0.15
Language
Dynamic Analysis
3.68
3.95 + 0.08
Language
Static Analysis
7.34
8.00 0.15
Visual
Dynamic Analysis
3.79
4.05 0.08
1
Visual
Static Analysis
7.99
8.28 0.16
Auditory
Dynamic Analysis
3.79
3.98 + 0.07
0
Auditory
Static Analysis
8.04
8.10 # 0.15
0
Table 19: RMSE between observed and predicted continuous-valuee code properties from the MVPA regression task Experiment confirming the pattern of results observed in 5.1
30


Reviews:
[{"review_id": "zGiCdcjw9Wq", "reviewer": "Reviewer_txju", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Major comments and questions: \n- The results of experiment 1 show the accuracy of the brain representation of code properties. It shows some correlation between brain regions and code properties. However, the t-test results show no significant differences for any brain regions having any preferences for a specific code property. This experiment has been done only on one dataset with Python code. How much is this kind of correlation reliable and pervasive in other datasets?\n- As we know, Python is a high-level scripting language that might not be quite distinguishable from short sentences. Have you tried similar experiments on any other datasets or other lower-level languages such as C/C++ or even assembly? \n- Is there any improvements in the brain code representations in this work compared to previous work such as Ivanova's given that you are selecting a vector of roughly 1000 voxels for each region?\n- Ivanova's paper reported some activities in the MD system during code comprehension. They showed moderate activity in Language systems corresponding to Python code comprehension and no activity regarding visual programming like ScratchJr. They also indicated that LS is responsive to Python code, while visual areas are responsive to ScratchJr. Given that, one other natural experiment to support the mapping brain representation to the code model could be to try it on the brain representations of visual programs (mostly in vision and MD systems) and the corresponding machine-learned encodings using visual recognition models such as convolutional neural nets. Have you tried such experiments on ScratchJr codes similar to what you have done for Python codes?\n\nMinor comments:\n- The explanation in the paper is fine, however, adding more diagrams of the whole procedure of the framework as well as mathematical descriptions of the models and their input and output as matrices/vectors would have helped a lot to understand the whole idea faster and easier. ", "summary_of_the_paper": "This paper introduces a systematical framework to discover the relationship between the brain representations of programs and their corresponding code models. This framework helps us to understand the code properties encoded in the human brain so that we could evaluate whether ML models faithfully represent human brain representations of computer code comperhensions. \nThis paper focuses on answering two questions by showing the results of related experiments on a dataset of 72 programs and 24 persons' brain recordings: First of all, the authors show that how well each of the four brain systems considered in this paper including Multiple Demand, Language, Vision, and Auditory systems encode specific code properties using a ridge regressor. Then they demonstrate another ridge regressor that can map brain representations to the corresponding learned representations by computational language models of code with different model complexity.\n", "summary_of_the_review": "The paper is well written, the motivations are clear, and the references are enough. Related work is adequately established the existing research in the field and compared the goal of the current research to previous work. However, the ideas of the paper are incremental and mostly applications of existing methods which have been put together to approach an interesting open problem. Moreover, usually further experiments on more datasets are required to evaluate any correlation between the code models and the brain representations in the proposed framework.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "vkArmCEm_gL", "reviewer": "Reviewer_mt3z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- code made publicly available, and the paper also uses publicly available brain data\n- mostly clear exposition, with only a few places where more details are needed (see questions below)\n- an in-depth and timely empirical investigation \n\nWeaknesses:\n- limited methodological novelty\n   - this work uses previously established methods for brain decoding, and does not make advances either in the methodology or in the interpretation of existing methodology. By itself, this is not a fatal flaw (a paper can have value to the community without such novelty).\n- limited evidence for some of the stated conclusions\n   - see Question 1 below for more detailed comments \n   - one general comment: when reading the main paper, it is not always clear what differences between brain systems or ML models are actually significant\n- limited discussion\n  - the investigation of the relationship between language models and the brain recordings has the potential to be very interesting, but the discussion of these results is extremely limited. The paper can benefit from a more thorough discussion of what is learned by the comparison with the language models. For example, it appears that a very simple model like BoW reaches similar decoding performance to that of the most complex model (CodeBERTa)--so then, what do we learn by the comparison with the complex model?\n\nQuestions for authors:\n1. I have some concerns about the conclusions for 3 of the 4 types of experiments in Section 5.1 under Analysis 1:\n   - code vs. sentences: Fig 3 shows that 3 of the brain systems (multi-demand, language, and visual) can significantly distinguish programs according to whether they were written using code or using sentences, with very similar accuracy. In conclusion, the authors state:\n> The underlying operations required to solve both types\nof programming problems were the same (e.g. both might have\nrequired summing elements in a list). Yet, despite the mental\noperations remaining mostly the same, the MD system is able to\ndiscriminate between the two contrasting conditions. This supports the claims made by Liu et al. (2020) and Ivanova et al.\n(2020) that the MD system is responsible for code comprehension in addition to being responsive to mentally tracing the execution flow of code snippets–a typical working memory task.\n\n      Since the two conditions are easily distinguishable based on visual properties (as evidence by the very high significant decoding accuracy in the visual system), how can the authors make any claims about what the multi-demand or the language systems are doing based on the decoding accuracy of these two conditions?\n    - variable language: Here the authors examine the decoding differences between two conditions, one in which the variable names are in English and the other where the variable names are in Japanese. The authors say that they expect a significant decoding performance in the language system. Would they not expect a significant decoding performance in the visual system? Shouldn't the most obvious difference be in the visual system, because of the systematic visual differences between English and Japanese? Fig 3 shows no significant decoding performance for any brain systems (neither the language nor the visual system). Could this be due to the way the brain data is aggregated over TRs for this analysis?\n   - control flow and data type: could the differences in decoding performance for the multi-demand and language systems be due to a difference in the signal-to-noise ratio in the two systems, and how to the authors control for this possible SNR difference between brain systems?\n2. Can the authors comment on why they chose to do decoding analyses over encoding analyses (i.e. predicting brain activity from representations of programs)? It seems that encoding could allow for a more in-depth analysis of the differences in what properties different regions/brain systems respond to (i.e. by comparing the weights of the learned encoding models for different brain regions/systems).\n3. Can the authors comment on why only left-lateralized regions of interest were examined? Several works in natural language comprehension with naturalistic stimuli show that both the left and right hemispheres are engaged during language comprehension (Wehbe et al. 2014, Huth et al. 2016, Jain and Huth 2018,..) Do the authors expect that the mirror brain regions in the right hemisphere do not engage in the same way during programming comprehension? And if so, do the authors believe that this is a function of their short stimuli (i.e. if the stimuli were longer pieces of code, we would expect both hemispheres to be engaged).\n4. One of the questions that was thoughtfully examined by the authors was how the complexity of the ML model affects its relationship with the brain recordings. The authors clarify that what they mean by complexity is the number of model parameters. Another related model property that may affect the relationship with the brain recordings is the dimensionality of the output embeddings--did the authors also examine this property (it's somewhat related to the number of parameters in the model but is not a perfect predictor, i.e. BERT and GPT-2 have the same embedding size but different numbers of parameters overall)? \n4. Clarifications needed about:\n   - representations of programs extracted from the models: how exactly were they extracted? Was the whole program provided to the model at once? The authors say that the output of the encoder was used, but what about for auto-regressive models (i.e. the last state, or something else?)? \n   - brain experiment: was the program presented at once, or word-by-word? for how long? what were the subjects instructed to do?\n   - brain data: assuming that each program was presented for multiple TRs, how was the data from multiple TRs handled in the decoding procedure?\n", "summary_of_the_paper": "This work examines the relationship between fMRI recordings of people who read short programs and different properties and representations of the programming code. The aim of the work is to understand what properties of code are encoded by different brain systems, and to understand how similar the representations of code in the brain are to those encoded by self-supervised language models that are pretrained to encode programming code. The authors find that several program properties can be significantly decoded from 3 brain systems (the multiple demand system, the language system, and the visual system). They further find that representations of the programs extracted from several machine learning models of varying complexity can also be significantly decoded from these brain systems.\n\n\n", "summary_of_the_review": "Overall, the paper investigates an interesting and timely question, but offers limited technical and empirical insights. The work can be strengthened by a more in-depth discussion about what is learned by the comparison with complex ML models, and by additional analyses that strengthen the empirical conclusions (see under Questions).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7Kg4nGzf7Kf", "reviewer": "Reviewer_vpX5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "##########\n\nUpdated review (apologies for the delay!):\n\nI commend the authors for engaging thoroughly with the reviews, providing new evidence and making convincing arguments. Having gone through their updated draft, rebuttals and new identification experiment (for the visual system confound), I am convinced that the MD/LS results show code processing distinct from pure visual features and that the random baseline performance in result 2 is perhaps an effect of the simple dataset, not that the regions/models only care about token-level information. To this end, I am increasing my score by 2 points.\n\nHaving said that, I agree with the other reviewers that this has been a productive discussion and the paper could benefit from more additions- particularly, stronger/cleaner results to show what information is differentially processed in MD vs. LS etc.. I am still unconvinced that the current approach is more promising in deciphering brain function than encoding (with variance partitioning). Hence I am unable to fully endorse this paper at this time.\n\n##########\n\nStrengths: The paper was well-written and the research question is interesting + useful. The use of publicly available data and code release is beneficial to the community.\n\nThe authors note that the visual ROI can effectively classify 4 different stimulus properties but it does not significantly predict the token count. This to me suggests that the 4 properties evaluated here are in fact strongly confounded with program length, presence of letters etc. and may not specifically correspond to code comprehension. What do the authors think about such confounds being relevant in other areas as well and its impact on the conclusions reached in the paper?\n\nOverall, I am not convinced that classification or identification accuracy is useful in gauging what properties of code are encoded by different brain regions. It would be useful to introduce more contrasts or control for alternate exploratory variables. While the authors allude to several of the metrics being correlated (like in the static analyses and byte code), not only would it be useful to add a correlation matrix but also use a more systematic approach like variance partitioning to see what unique properties of code these feature spaces explain. (Both against each other and confounds like program length) The authors are making definitive statements about the observed performance patterns based on prior knowledge of the MD system, LS system etc. However, given the possibility of confounds and correlations between the metrics itself, I am unable to view these results as strong evidence for one mechanism over the other.\n\nRe experiment 1:\n- Why isn’t the baseline just chance probability instead of a round-about way with randomly sampled assignments? For example, it should 1/3 for control flow or 0.5 for variable language.\n- There are no details on how the functional ROIs were identified: did you use specific contrasts, FreeSurfer annotations etc.? If indeed AC refers to primary AC,  perhaps the significance test being used is weak and cannot meaningfully identify signifiant effects.\n\nRe Analysis 1:\n1. Ivanova et al., 2020 effectively demonstrate that the multi-demand network uniquely responds to code. However, it is not clear to me what new findings are proposed in the current work beyond this.\n2. Control flow and data type: The authors make claims about accuracy differences between MD & LS. However, these differences are not statistically signifiant by their own admission: \n> We follow this up by using t-tests to examine whether any one brain region decodes any given property more significantly than another, but find no differences between the MD system and the Language system for any properties (Table 8, Appendix E).\n3. Same comment holds for the contrast in dynamic and static analyses:\n> We additionally test if any brain region has a preference for a specific code property over another. For instance, is the evidence seen in Figure 4, of MD more accurately decoding dynamic analysis properties than static analysis properties, statistically significant? Likewise, does the LS predict static analysis properties significantly more accurately than the dynamic analysis properties? We do not find any significant differences (Table 9, Appendix E).\n\nRe Experiment 2: \n- What layer of the models were used? Is it the logits or softmax output?\n- Can the authors provide more details on how rank accuracy was computed and how the regression task was set up?\n- It is unclear to me what scientific question this experiment is answering: Yes, different brain regions have good identification accuracy for different brain models, but isn’t this expected from a) prior work showing these regions respond to code and 2) these models capturing non-linear features of the code stimuli? What does this tell us about brain function?\n- I found weak evidence for the identification accuracies presented here since several of the tests were not statistically significant and did not beat the random embedding baseline.\n- Re the following: \n> The fact that MD maps to other complex models other than CodeBERTa, like CodeTransformer and seq2seq, as accurately as the Random embedding model is a strong indicator that the representations in these brain systems is strongly driven by token-level information in the input programs. \n\nI do not follow this claim. Do the authors mean brain systems or code models? Couldn't the same result be observed if the high-level code information was not linearly decodable as permissible by the linear regression setting here?\n\nOther questions:\n1. Since the magnitude of the predicted value is important here, I am confused by the use of linear correlation as opposed to RMSE.\n2. For discrete properties, this is clearly not regression. The authors should clarify that they build classification/regression models with an L2 penalty.\n3. What do the authors mean by “zero-shot” here since iiuc, the model was never actually tested on stimulus decoding following training on code representation identification?\n> We show that it is possible to perform zero-shot decoding of the computer program being comprehended using a proxy representation produced by a suite of code models.\n4. How were code representations extracted from XLNet and seq-2-seq? Were the models fine-tuned on code?", "summary_of_the_paper": "In this work, the authors explore the relationship between 4 different brain regions (multi-demand network, language network, visual system, auditory system) and different features of program code. Specifically, they look at hidden state representations of code language models (seq2seq, CodeBERTa, CodeTransformer, XLNet), tf-idf and BoW representations of the input. For non-LM based features, they look at a code vs. sentence contrast, variable language (English vs. Japanese variable names), data types (strings vs. numerals) and control flow (for vs. if vs. no branching) To analyze relationships between the BOLD signal and stimulus features, they build linear classifiers/regressors from the BOLD activity of each system to each of the stimulus features. Models are evaluated using classification accuracy and linear correlation for regression. In the case of hidden-state features, model are evaluated using rank accuracy.\n\nOverall, the authors find that the visual system is capable of significantly predicting several of the hand-crafted code features suggesting that these features are correlated with low-level stimulus properties like program length. While differences between MD & LS are not significant, these models successfully predict 5/6 hand-crafted features. In the code representation prediction task, the authors find that the MD, LS and Visual systems are able to rank significantly above change. However, the LS and visual systems do not beat a random token embedding baseline. Aside from CodeBERTa, the MD system is also not significantly above the random baseline.", "summary_of_the_review": "While the proposition of understanding what aspects of code processing are represented in different brain regions is interesting, in its current form, I think the paper doesn’t successfully disentangle this. Several of the effects reported are not significant above chance or random baselines and further, not significantly different across brain regions. The features considered are also confounded with low-level properties of code comprehension that haven’t been controlled for. To this end, it is hard to infer what properties different systems are encoding and if there is a take away beyond prior work that identifies regions robustly responding to code.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-x6QYCVI5bI", "reviewer": "Reviewer_ELAX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThis paper is very clear and well written. It investigates an interesting question of what kinds of information is decodable from brain representations of computer code. Experiments are well thought out and carefully controlled with a decent set of properties of interest and model representation results are compared with relatively extensive sets of models. \n\nWeakness:\nI am not sure whether this paper actually provides a lot of new insights for both audiences from the neuroscience community and machine learning community. \n\nIt is not surprising to me that the Multiple Demand system in the brain is able to provide above chance decoding performance of the selected properties of the computer code. The Multiple Demand system includes a large part of the prefrontal cortex that is generally responsible for any executive control and cognitive processing human do. It would be more interesting, for example, to run a searchlight algorithm over the brain to pin down specific locations where different properties of computer code are represented.\n\nFor experiment 2, other than model complexity, another useful control would be the information integration window (or context window) of different models. The models in comparison have very different mechanism to integrate information across the time during which a computer program is presented. Therefore it is unclear to me how the integration of information over time or length of program is going to affect the brain mapping to these representations. The mapping from brain representations to context window is also potentially helpful for machine learning researchers in designing better models for code representations. However, the analysis in the paper so far, largely limited to decoding based methods and only compared with limited number of controls, are not very informative.\n\n", "summary_of_the_paper": "This paper investigates encoding of computer code in the human brain. The author build decoding models for fMRI responses to predict 1) various properties of python code and, 2) representations of python code derived from different machine learning models. The main conclusion is that the responses from the Multiple Demand system in the brain is capable of provide significance decoding performance of properties and model representations of computer code, such as runtime information.", "summary_of_the_review": "The paper has done thoughtful analysis and comparisons to investigate how the brain represent computer code and has great potential to be accepted but more in-depth analysis are needed to make it a more informative paper for both the neuroscience and/or machine learning audience. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims to relate brain activity (of people reading computer
code) to properties of the computer code. They relate the found
representations to those obtained from ML computational language 
models applied to the same programs.  The paper is clearly written and
an interesting idea.

There was a lot of discussion and the author(s) updated their paper a
lot.  Program length as a potential confound was raised and 
successfully rebutted.  The extent of novelty from Ivanova et al 2020
was also discussed and successfully rebutted.  In the end, the main
issues the reviewers had were 1) that the paper had been updated
substantially since submission (and would therefore benefit from a
thorough re-review) and 2) whether the results provide enough new  
insights about the brain or about ML language models.  

To summarize, the authors spent a lot of time addressing issues in the 
rebuttal phase and the paper got a lot better with the reviewers'
suggestions, but reviewers agreed it would benefit from more work and
further review before acceptance.  I agree with this assessment.
Decision: Reject