Extracting Meaningful Attention on Source Code: An Empirical Study of Developer and Neural Model Code Exploration
Authors: Matteo Paltenghi
Rahul Pandita
Austin Henley
Albert Ziegler
Venue: Submitted to ICLR 2023 (2023)
Keywords: eye-tracking
transformers
self-attention
code exploration
source code
neural models of code

Abstract: The high effectiveness of neural models of code, such as OpenAI Codex and AlphaCode, suggests coding capabilities of models that are at least comparable to those of humans. However, previous work has only used these models for their raw completion, ignoring how the model reasoning, in the form of attention weights, can be used for other downstream tasks. Disregarding the attention weights means discarding a considerable portion of what those models compute when queried. To profit more from the knowledge embedded in these large pre-trained models, this work compares multiple approaches to post-process these valuable attention weights for supporting code exploration. Specifically, we compare to which extent the transformed attention signal of CodeGen, a large and publicly available pre-trained neural model, agrees with how developers look at and explore code when each answering the same sense-making questions about code. At the core of our experimental evaluation, we collect, manually annotate, and open-source a novel eye-tracking dataset comprising 25 developers answering sense-making questions on code over 92 sessions. We empirically evaluate five attention-agnostic heuristics and ten attention-based post processing approaches of the attention signal against our ground truth of developers exploring code, including the novel concept of follow-up attention which exhibits the highest agreement. Beyond the dataset contribution and the empirical study, we also introduce a novel practical application of the attention signal of pre-trained models with completely analytical solutions, going beyond how neural models’ attention mechanisms have traditionally been used.


Under review as a conference paper at ICLR 2023
EXTRACTING MEANINGFUL ATTENTION ON SOURCE CODE: AN EMPIRICAL STUDY OF DEVELOPER AND NEURAL MODEL CODE EXPLORATION
Anonymous authors Paper under double-bling blind review
ABSTRACT
The high effectiveness neural models of code, such as OpenAl Codex and AI- phaCode, suggests coding capabilities of models that are at least comparable those of humans. However, previous work has only used these models for their raw completion, ignoring how the model reasoning, in the form attention weights, can be used for other downstream tasks. Disregarding the attention weights means discarding considerable portion of what those models compute when queried. To profit more from the knowledge embedded in these large pre-trained models, this work compares multiple approaches to post-process, these valuable attention weights for supporting code exploration. Specifically, we compare which extent the transformed attention signal of CodeGen, a large and publicly available pre- trained neural model, agrees with how developers look at and explore code when each answering the same sense- making questions about code. At the core of our experimental evaluation, eye-tracking dataset comprising 25 developers answering sense-making questions code over 92 sessions. We empirically evaluate five attention-agnostic heuris- post processing approaches of the attention signal
collect, manually annotate, and open-source novel
tics and ten attention-basee against our ground truth of developers exploring code, including the novel con- cept of follow-up attention which exhibits the highest agreement. Beyond the dataset contribution and the empirical study, we also introduce a novel practical application of the attention signal of pre-trained models with completely analytical solutions, going beyond how neural models' attention mechanisms have tradition- ally been used.
INTRODUCTION
Recent large neural source code models such as Codex (Chen et al. (021). CodeGen NNijkamp et al.) 2022) and AlphaCode (Li al. 2022) remarkably effective program synthesis and competitive programming tasks respectively. Yet our understanding of why they produce particular solution is limited. In practice, the models are mostly used for their prediction alone, i.e., as generative models, and the way they reason about code internally largely remains untapped.
These models are often based the attention mechanism (Bahdanau et al.) 2016). key component of the transformer architecture (Vaswani et al. 2017). Besides providing substantial performance benefit, attention weights have been used provide interpretability of neural models (Lin et al., 2017 Vashishth et al. 2019 Paltenghi & Pradel 202T). In particular, Wan et al. (2022) and Vig Belinkov (2019) have shown how the attention weights contain important syntactic information on both the Abstract Syntax Tree (AST) of source code and Part of Speech (POS) tags in natural language. Moreover, WWan et al. (2022) showed how using attention weights to infer the distance between two tokens outperformed techniques using hidden representations. In similar direction, Zhang et al. (2022) has shown how a novel graph representation of source code derived solely from attention weights achieved comparable performance on the VarMisuse dataset (Allamanis et al. 2018) to that of hand crafted graph representation based on control flow and data flow (Hellen-
doom et al. 2020).

---

Under review as a conference paper at ICLR 2023
The work cited above suggests the attention mechanism reflects encodes objective properties of the source code processed by the model. We argue, that just as software developers consider different locations in the code individually and follow precise connections between them, so the self-attention of transformers connects and creates information flow between similar and linked code locations. If those relations are indeed comparable, this raises the possibility: can the knowledge about source code conveyed by the attention weights of neural models be leveraged to support human code exploration?
There are datasets tracking developers' visual attention while looking at code, but they do not seem suitable to this task. The largest ones either put the developers in an unnatural (and thus possibly biasing) environment where most of the vision is blurred (Paltenghi & Pradel (021). or they contain few and very specific code comprehension tasks (Bednarik et al. 2020) on code snippets too short to exhibit any interesting code navigation pattern.
This work. To address these limitations and stimulate developers and the neural model not only glance at code, but also deeply reason about it, we prepare ad-hoc code understanding assignment called sense-muking task. This involves questions on code including mental code execution, side effects detection, algorithmic complexity, and deadlock detection. We collect an eye-tracking dataset of 92 valid sessions with developers. The sense-makin cense-making task is additionally designed to be machine friendly with a specific prompt to trigger completion from the model which hopefully stimulates the reasoning of the model. Then we query CodeGin on the same making task and compare its attention signa I to the attention of developers. They turn out to be positively correlated (1=+0.23), motivating the use of raw and processed versions of the attention signal for code exploration. To that end, we experimentally evaluate how well existing and novel attention post-processing methods align with the code exploration patterns derived from the chronological sequence of eye-fixttion events of our dataset. To the best of our knowledge, this work is the first to investigate the attention signal of these pre-trained models to support code exploration, specific code-related task.
We empirically demonstrate that post-processing methods based on the attention signal can be well aligned with the way developers explore code. In particular, using the novel concept of follow-up attention we achieve the highest overlap with the top-3 developers' ground truth on which line to explore next.
Main contributions. Our key contributions are:
A novel dataset of eye tracking data, comprising 92 visual attention sessions of 25 devel- opers engaged in sense-making tasks while using a common code editor with code written in three popular programming languages (Python, C++ and C#).
The first experimental comparison of both effectiveness and visual attention of GPT-like models and developers when reasoning on sense- mansin questions.
Demonstrating a connection between the neural attention signal and the temporal sequence of location shifts regarding developer focus.
The analytical formula for follow-up attention, novel post-processing approach derived solely from the attention signal, which aligns well with the developer interaction of which line to look at next when exploring code.
An empirical evaluation comprising ten post-processing approaches of the attention signal, five heuristics, and an ablation study of the follow-up attention against the collected ground truth of developers exploring code.
RELATION TO EXISTING WORK
Attention as explanation. Previous work (Jain & Wallace 2019) on studying attention weights of recurrent neural models has found that the attention weights do not always agree with other expla- nation methods and that alternative weights can be adversarially constructed while still preserving the same model prediction. However, in response, successive study (Wiegreffe & Pinter] 2019) showed how the alternative attention weights can be constructed only per single instance predic- tion, whereas obtaining a model which is consistently wrong in its explanations is very unlikely
Attention signal refers the attention weights produced during forward pass by the transformer blocks.
2

---

Under review as a conference paper at ICLR 2023
to happen. On the same line, Tutek & Snajder (020) has proposed four regularization methods to mitigate the adversarial exploitation of attention weights for recurrent models, including the use of residual connections which natively embedded into transformers (Vaswani et al. 2017). the building blocks of GPT-like models Radford et al. studied in this work.
Alternatives to attention signal. There observable signals other than attention that might cap- ture the concept of relevance, such as gradients-ban al. (2019). In particular, |Yuan et al. (2021) and [hefer et al.) (202I) are promising examples of approaches based, partially or entirely, on gradient information. However, we focus on approaches using only the attention signal for two reasons: (1) almost all state-of-thee
or layer-wise wise relevance propagation Montavon
models of code are based on the transformer block, and the attention mechanism is ultimately its fundamental com- ponent (Vaswani et al., 2017), so we expect the corresponding attention weights to carry directly meaningful information about the decision process of these transformer-basee models; (2) unlike the gradients, the attention weights can be extracted almost for free during generation with little runtime overhead, crucial condition in many practical scenarios.
Attention studies of neural models of code. Paltenghi & Pradel 2022I) have compared the at- tention weights of neural models of code and developers' visual attention when performing code summarization task, and found a strong positive correlation on the copy attention mechanism for an instance of a pointer network (Vinyals et al. 2015). Wan et al. (2022) and Zhang et al. (2022) have then shown how the attention weights of pre-trained models on source code capture important prop- erties of the abstract syntax tree of the program. However, none of them considered the use of the attention signal for a code related task, such as code exploration. Moreover, they are limited to rela- tively small self-attention transformer models, whereas we study the attention of CodeGen Nijkamp et al. (2022). large generative model using masked self-attention.
We point the interested reader to Appendix for a additional connections with existing literature.
CODE UNDERSTANDING TASK: SENSE-MAKING QUESTIONS
To study developers' and models' attention, we pre- pare code understanding task called the sense- making task. One sense-making task is contained in single source code file p composed of four sec- tions: (1) a brief description of the context of the main code snippet (e.g. The following code rea- sons about triangles in the geometrical sense.), (2) the main code snippet, either sourced on the in- ternet or written from scratch by the authors, (3) a sense- making question to stimulate the reasoning (i.e., question:: and (4) a final prompt to trigger the model's answer (i.c., Answer:).
follewing roser triengtes the gemetrical
<lass paint: del 2111 (self. salf.y
aqursce):
return
#el arcerte. (a. e)
copy (0r51)
return (1910). oey/1) opt171
patrceO, E)
poincet, 1)
:lausitytringippp pi, al]
Questio:: seat could hagpen "wnderty" were owitted
"classitytringgee"
Note that all the sections except the main snippet are in the form of code comments. The questions in- clude a variety of topies such mental code execu- tion, side effects detection, algorithmic complexity, and deadlock detection; Figure [ shows an example task, whereas the full list of questions can be seen in the Appendix [A] We prepare five main snippets and create three unique questions for each of them, then we translate the same task in three programming languages: Python, C++ and C#. In total, we have 45 unique tasks.
Aroeer
Figure 1: Example of sense- sense-making task with code and question be answered in the bottom comment.
Neural Model's Task. We feed the entire source file corresponding to single task as input of the 0.2 to generate three different answers in the
generative model, and query it with temperature form of text completion. We fix the maximum number of generated tokens to 100. The model used CodeGen (Nijkamp et al. 2022), in its largest language agnostic variant
We use CodeGen- 16 6-multi from https::/github..ct http:///c///llleeeee////////////
3

---

Under review as a conference paper at ICLR 2023
Developers' Task. We recruit 25 software developers via direct contacts at large software com- pany, without any constraint on prior programming experience, We track the eye gaze of each participant during session of maximum of 45 minutes while they answer many questions as possible, typically three or four. We ensure that they see each main code snippet only once to avoid bias in answering a question on snippet they have already explored in a previous task. To ensure consistent data collection, the eye tracking setup is calibrated at the beginning of each task.
4 PROBLEM FORMULATION, ATTENTION
The large majority of large language models is based on generative pre-trained transformers (GPT) (Radford et al... This is true also for models of code with notable examples such as Codex (Chen al. 2001I) CodeGen (Nijkamp et al. 2022), and AlphaCode (Li al., 2022). GPT- like models such as these are comprised only of decoder layers rather than of both encoder and decoder layers, and use masked self-attention to prevent token from attending tokens which come after it. In GPT-like models the self-attention given by token to the other tokens in the sequence can be represented by a vector: a; (a.,1; a; ..., ai 01,4+1...., Og where n is the maximum allowed sequence length. Note that token i has one such vector for each combination of transformer layer and attention head. For each layer and head, the attention weights at,, form triangular matrix.
4.1 VIEWS OF ATTENTION
Given a GPT-like neural model f and and a sense-making task whenever the model ingests p, its tokenizer splits it in n prompt tokens p = .1,.... ln and outputs both the original input sequence followed by newly generated tokens and the attention tensor corresponding to the masked self- attention weights:
f()) = (t1,..., En, tn+ t A)
(1)
where m is the number of generated tokens and A is the 4-dimensional attention tensor of shape (1, h, n + m. n + m) where is the number of layers, h is the number of attention heads, n is the number of tokens in the prompt. Note that unlike the predicted tokens, which are randomly sampled with temperature parameter, the attention weights stay the same across runs provided that the input prompt remains the same. In particular, when comparing developers' and the model's attention, we focus on studying the attention weights referring to the prompt tokens only, thus the dimension of Aprompt are (1, h, n, n) even if some proc-ssro approach may use the entire tensor.
Each post-processing approach of the attention signal can be represented by extraction function g which transforms the attention tensor A into one of two views: visual attention or interaction matrix.
Visual Attention. The visual attention is static view which tells us what part of the input is important for the model when solving the sense-making task. We define the visual attention of a vector
model or a developer each Og intuitively tells us the how much "attention" was given to that the i-th character when solving the task. We represent the corresponding extraction function as gesz (A), which a function that takes as input the attention tensor A and returns the visual attention a.
(a1, -e., ae) over the single characters of the prompt, where
Interaction Matrix. The interaction matrix is a dynamic view which tells us, given a position in the prompt, which other position of the prompt should likely be attended soon after the current position by the model. Intuitively, the interaction matrix can be directly used to support the code navigation of a developer since it could recommend to the developer the likely next interesting piece of code any point in time. We define the interaction matrix S as a right stochastic matrix with size X p where is the number of tokens in the prompt and is the number of admissible target positions in the prompt. We define two interaction matrices depending on the granularity of the target position p: (I) token-level, where S has size n X where n. is the number of tokens in the prompt; (2) live-level, where S has size n X ny where n; is the number of lines in the prompt. We represent the corresponding extraction function Ginter (A), which is function that takes input the attention tensor A and returns the interaction matrix S. In particular, we distinguish between the two granularity levels using two functions Otoken and gtine leading to Stoken and Stine respectively.
4

---

Under review as a conference paper at ICLR 2023
5 METHODOLOGY: EXTRACTION FUNCTIONS
For the empirical evaluation, we investigate two concrete extraction functions for the the visual attention and four for the interaction matrix.
5.1 EXTRACTION FUNCTIONS FOR VISUAL ATTENTION
To condense the attention tensor A vector a, introduce straightforward approaches: attention mean and attention max.
Attention mean. simply takes the average attention weight for each position along the two dimen- sions of different layers and attention heads to compensate the extreme values with each other and extract smoother signal. This step, outputs a matrix A with shape (n, n) where each element A., is the average attention weight of the i-th token in the prompt to the j-th token in the prompt. Note that is a lower triangular matrix due to the masked nature of the self-attention. Then, since the positions above the main diagonal are all zeros, we compute the mean of each column excluding the zeros to avoid penalizing more recent tokens with less followers. Since each column can be seen as the attention given to j-th token by its "followers" we call the step mean of followers, which produces a token-level visual attention a vector. Finally, to convert to a character level vector, we divide the attention weight on single token in equal shares among all its characters.
Attention max. This approach differs from the previous one in the way it condenses layers and heads replacing the mean with the max function to favour the extreme positive signals appearing only in one or few layers and heads, the rest is unchanged.
5.2 EXTRACTION FUNCTIONS FOR INTERACTION MATRIX
We study four approaches: max, mean, rollout and follow-up attention. Apart from the rollout attention, which has been introduced by Abnar & Zuidema (2020), the other three are either inspired by the work of Paltenghi & Pradel 2021 or novel contribution of this work, such as the follow-up attention.
Attention mean. It computes the mean among all the T. layers and H attention heads:
Algorithm 1 Follow-up Attention
Imean 1hh ΣL 1
1: Input: A (Dim: (L, H, m,n + m)]
2: Output: S
Attention max. It computes the max among
3: consec.pairs < -0 H
all the L layers and H attention heads:
Imax max max/ !! AI,.-
4: + h A (Layer-wise attention]
5: for f- 1 to L do
Rollout attention. It propagates the informa-
6: S(x) <
tion contained in the attention weight layer by layer from input to output, by multiplying the attention weights along multiple paths start-
7:
for i + to do
8:
for j + 1 to do
9:
m mix
ing and ending to the same input-output pair. Since it does not model the attention head di- mension, we simply condense the that dimen- sion via simple sum. Moreover, it does not prescribe which attention layer should be used in the end, thus we the newly computed rollout values of all the layers.
f(() + Lg,sit
10:
ffa+1) Lit1,s:
11:
f())(((t)))))
s/3 [l p
12:
13: end for
14: end for
15: consec_pairs < consec_pairs U S())
Follow-up attention. It is a novel approach that we introduce in this work. Similarly to the rollout attention. we aggregate the attention weights over the h attention heads by sum- ming the weights of along the attention head
16: end for
2= S(=) 17: S + -111
dimension and obtaining the layer-wise attention L L= > A We leave the exploration of head- level aggregation strategies to future work, and rather focus modelling the information flow between subsequent layers. We refer to this condensed 3-dimensional tensor as layer-wise attention represented by L. The follow-up attention explicitly models the temporal relationship between the
5

---

Under review as a conference paper at ICLR 2023
attention weights computed at different layers, since the attention weights in the later layers de- pend on the earlier ones. The intuition is that the layer-after-layer transformation reflects the way in which the models explores code through time, similarly to multiple successive fixations of de- veloper when navigating and exploring source code. Instead of looking at how the a token gives attention to other tokens in the same layer, the follow-up attention adopts differential approach which compares the attention received by token i at layer with the attention received by token j at layer - 1. To represent this received attention we define the follower score f(()) of token at layer 2, as the vector of the attention quota that each other token gives to token i at the same layer. Note that, similarly
the self-attention vector, the follower score is also vector of real numbers and it has the same length corresponding to the input sequence length, thus representing a complementary viewpoint. To realize agreement between follower scores two consecutive layers, we use the co- sine similarity as soft version of the intersection between the set of followers of the two tokens. Then we compute the follow-up attention for each ordered pair of tokens (i + j for i prompt or generated token) and for each pair of consecutive layers, and condense all layer pairs into a single matrix via sum. We aggregate attention over multiple layers, since Brunner et al. (2020) have empir- ically shown how token identifiability is retained over layers, thus generic embedding at position e, in any layer is traceable to the input embedding æ; in the input sequence. Their finding can be partially explained by the residual connections of transformers, which were studied by (Abnar & Zuidema 2020) Algorithm summarizes the overall algorithm. An optimized vectorized version available here
6 EXPERIMENTAL EVALUATION
In this section, we compare the visual attention and interaction matrix extracted from the attention tensor of neural models against the ground truth computed from the develop- ers. All our code is publicly available at and the dataset is available here
wwwp:/oooo.........ooo.ooo /
Eye Tracking Dataset. We derive the ground truth from eye tracking dataset comprising 25 participants over 92 valid sessions (see Appendix[]aa for details). Each session consists of a sequence of eye fixation events cuteze, each represented a tuple (t, æ. y, where is the timestamp in milliseconds, I and are the coordinates of the fixation point in pixels and d is the duration of the fixation in milliseconds. Each session is recorded in Visual Studio Code to have a natural coding environment. Based on the size of the parafoveal region (Schotter et al. 2012). each eye fixation event is converted column and line coordinates: evtssher) I is the line of the original source file (more detail in Appendix D).
(t. c, d) where is the column and
Ground truth visual attention. Here we borrow from the concept of human attention proposed by Paltenghi & Pradel (2021) and define the analogous developer attention as the total time that a specific char was visible to the participant: d (di, -..; de) where is the number of characters in the prompt and d; is the total time that the i-th character was visible to the participant according to the eye tracking data. In difference Paltenghi & Pradel (20TT), we consider the char-level instead of the token level, which is the granularity in many contexts, and in particular for eye tracking data.
Ground truth interaction matrix. From each developer session we derive a ground truth interac- tion matrix S. For fair comparisons of neural models with developers, we take into account the tokenization used by the neural model, namely we use the CodeGen tokenizer which is based on byte-level byte-pair-encoding (Sennrich et al. (016)).
To convert char-level events into token level ones, for each timestamp, least one character of a given token is visible, then the token is considered visible as well and we count the corresponding the token index and d is the event duration.
event euteye (token) (t, i, d) where is the timestamp. Based on the pairs of events involving token i and token j, we quantify how likely it is that the developer looks at token j after having looked at token
Raw eye htw.gwawww....aaaaooo....../o.aa sion Twa.l//sssEcceaacccccooo///eeee.
and clean ver-

---

Under review as a conference paper at ICLR 2023
Traversin
Trieeinn
Stronghh tomporal connection
0.015
Sted Wabul observed
0.010-
0.005
800
4000
400
800
tokens
Figure 3: The strength of the connection St depends significantly on the difference i -- As detailed Appendix both cases > j and j > can well modelled using a Weibull distribution.
449
Intuitively, we want to have stronger connection when fixation token is shortly followed by fixation on token j. and if this second fixation has a significant duration. Thus, we define the strength of the temporal connection between token and token j as:
0.75
0.50
4+4
0.00
10
Time (sec)
Example of two events where the yellow area corresponds to their contri- callt-(h+dd) (2) bution to the connection strength between from token i to token j.
Figure 2:
lytd
Stj
E
eut,,
where Pi-tj is the set with all the pairs of events where token i is seen before token j. and a is parameter that controls the decay of the connection the more the two events are far apart time, For our experiments we empirically set 0.1, accounting for observed behaviour where developers often spend several seconds scrolling and presumably shallow searching the code. In Figure we show example of the integral connecting two consecutive events.
We noticed a strong neighboring effect across the whole dataset, where the connection between closer tokens tends to be relatively stronger irrespective of context and content (see Figure B and Appendix F To extract relevance beyond mere closeness, we normalize each row of the interaction matrix S by dividing by the average empirical ground truth distribution where the probability to go to token constantly decreases the further away the target token is.
To measure the agreement between the visual attention of developers and the neural model, we regard them as vectors with meaningful ordinal content, and compute their Spear- man rank correlation coefficient (Spearman) 1987). In contrast, we need to compute the agreement between the interaction matrices row by row, using either the Spearman rank correlation coefficient or the top-3 overlap, defined as the number of top-3 target positions shared between the ground truth and the model-derived interaction matrix. We focus on the line-level interaction matrix Sinee since we argue that the line-level is perhaps the most useful granularity from an hypothetical user perspective. To obtain S we simply sum the probabilities to go tokens belonging to the same line. Moreover, balance the fact that some potential starting tokens might be rarely (or only very transiently) looked at by the developers, seconds spent by the developer on the corresponding starting token. We also fix a maximum for this weight to 10 sec prevent long-observed tokens dominating the comparison. We only ever directly compare tasks related to the same code snippet, and when we have more than one developer ground truth on the same task, we consider all the possible developer-model combinations independently.
Agreement Metrics.
weight each comparison based on the total number of
6.1 DEVELOPERS VS NEURAL MODELS
Fried Our initial investigation considers three neural models: CodeGen GPT-J InCoder Fried et al. (022). For the in-depth study of the interaction matrix we focus on the more recent and more performant CodeGen model.
hhtps://huggingffccccm htt.://gggggnaaaooo////e/eeoeean
7

---

Under review as a conference paper at ICLR 2023
Answers correctness. We manually grade developers and neural model answers as either correct, partially correct or wrong, with high inter-rater reliability indicating a reproducible annotation pro- cess (see Appendix [ The developer's 's answers correct partially correct almost 70% of cases, while the neural models show a promising and non-trivial performance (Figure 4 Appendix ( We note the possibility that were the neural models given a more sophisticated prompt than ours, i.c., #Answer: their performance might have been even stronger, considering a multitude of re-
sults prompt engineering research (Prenner & Robbes 2021 Shrivastava et al.
2022 Reynolds
& McDonell 2021 Bareiß et al. 2022).
Correct
Partial
Wrong
Developers
45.7%
23.9%
30.4%6
CodeGen
15.6%
60.7%
Gpt:)
23:2%
12.6%
63,755
Incoder
17.0%
20.7%
62.2%
%
20 %
40
60 %
%
100
Percentage of answers
Figure 4: Percentage of correct, wrong and partially correct answer for developers and model.
Agreement visual attention. In Figure 5 we
port the Spearman rank correlation Spearman (1987) between the developer attention vector and the model attention vector, for completeness we also report the comparisons among developers. We assess the differ- ence among programming languages with the Mann Whitney U Test (Mann & Whitney 1947) applied to the distributions of Spearman coefficients, and we could only spot significant difference between devel- opers and models agreement on Python and C# code (p-value < 0.05). with higher agreement obtained on C# (more in the Appendix Hj. This exceeds the agree- ment observed in previous work Paltenghi & Pradel (02TT) which we hypothesize to be due to combina- tion of us using a more advanced model Ahmaa et al. (2000) and more natural data collection setup (eye tracking vs a deblurring interface).
25
Median: 9.23
-1.00 -0.50
0.00 0.25
1:00
Speorman Rank Cor:. Developers vs Model (CodeGen)) Max
Count
50
Mean:
1.00
0.75
0.50
0.25 0.00 0.25
0.50
0.75 1:00
Spearman Bark Corr. Developers Model (Gpig) Max
Median:
Mean: 0.19
-1.00 -0.75 -0.50 -0.25
0.00
0.28
a.50
0.75 1.00
Sp Spcarman Rank Corr. Developers Model (inCoder) Max
Mediam 0.53
Mean: 0.51
1.00 -0.75 -0.50 -025 0.00
0.25
0,50
0.75 1:00
Speaeman Rark Con, Developers Developers
Figure 5:
Agreement between develop- ers and models' visual attention (extraction function: max).
Considering CodeGien's slightly better performance and significantly higher developer agreement, we re- strict our subsequent investigation this model.
Agreement on interaction matrix. We run the different extraction functions on the model attention signal to obtain interaction matrices Stine, which compare developer derived ground truth.
We distinguish between: (1) attention-based code traversal predictions, which are those introduced in Section [.2]] and (2) attention-agnostic code traversal predictions. The iitention-based meth- ods comprise raw attention in first and last layer, max, and mean, with their respective symmetric versions where the triangular matrix is mirrored and added replace the zero values, the rollout and follow-up up attention. The attention attention-annostic methods comprise: copy-cat recommending all the positions containing the same starting token, uniform recommending all the position preceding the current token, and position recommending the neighboring positions of the current token with Gaussian distribution centered on the current token.
We find that attention-based methods do carry predictive power, and in particular that follow-up attention performs best among all methods for both Spearman rank and top-3 overlap (Figure(6). We note that purely position-basee approach performs better than the copy-cat method despite being completely content-agnostic. We attribute this to developers' tendency to often read source code in (piecewise) linear order described by (Blascheck & Sharif 2019). Regarding raw attention, (itention,Zhangg cet al.) (2022) demonstrated deeper semantic information being concentrated later layers. Yet for both the triangular and symmetric versions respectively, higher levels appear inferior at predicting
8

---

Under review as a conference paper at ICLR 2023
eye-movement to earlier levels, possibly because such deeper semantic information is not directly apparent to developers at a glance.
Median
Size overlap Tep-3
Davs vs devs 0..00
0.56
Tollow-up att. *
Follow-r up att
53.0%
13:0%
Position
Max att. tsym. 1
55.
22.3%
15:85am
Comparison name
Raw art. (last) [sym.) )* 0.39
Meen att. (sym.) 3..4.
att. (1 [sym.) 0.47
Max att. Esym.)*-
10ym.)-0.000
0.442
0.37
0.41
0.40
Companson
Raw att (1st) toym. 14
Raw 13st layer
Devi devs
Max att.
Posittion
57.75
57.7%
59.715
59.59
60:216
22.8%
20:20
16.256
14.0k
13662
10:356
Copy-cat
Mean Isymm
63.77
26:5%
att. layer)"
Uniform
65:27%
26.5%
0.02
Mean
65.45%
Rollout att
0.03
0.02
att.
05:41
Uniform
0.01
0.02
05.95
Mean att.*
0.03
0.02
layerf*
layar"
Copy
-0.5
0.0
0.5
20
40 %
60
1%
100 %
Spearman
Distribution
Parcentage noxt-line predictions
(a) Spearman Rank Correlation
(b) Top-3 Overlap
Figure 6: Agreement between developers and CodeGen interaction matrix ordered from weak (top
lines) to strong (bottom lines). dicettion-base methods are marked with
Ablation study of follow-up attention. We investigate two key design choices for follow-up atten- tion: (I) the selection of layers to use, (2) the number of generated tokens, i.e., observers (Figure ( There is significant agreement with the ground truth even for smaller numbers of layers, particu- larly the very first one. This suggests little processing, and maybe even only the token embedding, might be needed to extract valuable information. Additionally, a higher number of observers of the follow-up has positive impact on the agreement of the follow-up attention with the ground truth.
Size overlap for Top- next lnes
Size of overlsp for T.p-3 next lines
53.0%
32
55.35
11:45
53.7%
13,8%
tokens
53.0%
Compartion neme
57.7%
56.2%
55.9%
14:00
Layer
24.0%
tokens
12.6%
Layer
54.0%
Layer
54.0%
Layer
52.5%
54.0%
33.85pm
10 tokens
54.8%
31.5%
11.6%
Layer
30:2N
15.90%
%
20
60
80
100 15
100
Percentage predictions
Percentage next-line predictions
(a) Effect layer pair
(b) Effect number of generated tokens
Figure 7: Ablation study of Follow-up Attention. Left: Restriction to few layers still retains signifi- cant predictive power. Right: Generating more tokens improves the prediction.
CONCLUSION
We presented and shared novel dataset eye tracking data, comprising 92 visual attention sessions of 25 developers when answering sense-making questions in three popular programming languages (Python, C++ and C#). We confirmed that neural models provide promising, but less accurate an- swers than developers formalized a new code exploration task of predicting developer code traversal and confirmed the attention signal's 's relevance for this task by evaluating multiple processing approaches. Besides existing post-processing, we contributed the concept of follow-up attention, which shows the best agreement with the developer attention data.
these questions, while paying attention to similar parts of the code. We

---

Under review as a conference paper at ICLR 2023
8 ETHICS STATEMENT
The study involved human subjects who decided to participate voluntarily. Each of them gave ex- plicit consent to use their data for scientific publication. Each eye-tracking session was limited to 45 minutes and included breaks to limit the cognitive stress on the subject. All the shared data have been anonymized for privacy by removing any reference to personal details of the participants, such as names. The eye-tracking data collection was carried out in an ergonomic setup with a non-invasive method, but with eye-tracking camera pointing the face of the participant. Each participant received a 10 USD company cafeteria food coupon for participating in the study, The authors have no conflict of interest to report.
REPRODUCIBILITY STATEMENT
All our code is publicly available at the following repository: sci wewceer/aatteeeeee itw.:eettttoootttttuttt....t -atactss-7722.
4oper.
The eye tracking dataset is available both in its raw version with all the eye tracking sessions https: hat:://figghaare com/ 8 /5 6a556fff55 6 5bc2 9e6f620 and also in its processed version with interaction matrices computed as described in the main text https: / /figsshare... com/s. / The results of the empirical comparisons are also present in the last mentioned dataset. The repository contains Readme and Python notebook to reproduce the main figures of the paper starting from the shared data.
c20e04332097125
In addition, since all the code used is publicly available, including the snippets used in the study and the code to process the eye tracking data, the entire study can be entirely replicated with new participants and new neural models well. Indeed, the code supports the huggingface API of generative models.
REFERENCES
Samira Abnar and Willem Zuidema. Quantifying Attention Flow in Transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4190-4197 Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/v/2022.ac/ main. 385.
Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. A Transformer-base Approach for Source Code Summarization. arXiv:2005.0066 00653 [es, stat], May 2020.
Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. Learning to Represent Programs with Graphs. In International Conference on Learning Representations, March 2018.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. arXiv:1409.044 [cs, stat], May 2016.
Patrick Bareiß, Beatriz Souza, Marcelo d' Amorim, and Michael Pradel. Code Generation Tools (Almost) for Free? A Study of Few-Shot, Pre-Trained Language Models on Code, June 2022.
Roman Bednarik, Teresa Busjahn, Agostino Gihaldi, Alireza Ahadi, Maria Bielikova, Martha Crosby, Kai Essig, Fabian Fagerholm, Ahmad Jbara, Raymond Lister, Pavel Orlov, James Pa- terson, Bonita Sharif, Teemu Sirkiä, Jan Stelovsky, Jozef Tvarozek. Hana Vrzakova, and Ian van der Linde. EMIP: The eye movements in programming dataset. Science of Computer Program- 102520.
ming, 198: 102520, October 2020. ISSN 01676423. doi: 10.1016/j.ssico.
Tanya Beelders. Eye-tracking analysis of source code reading line-by-line basis. In 2022 IEEE/ACM 10th International Workshop on Eye Movements in Programming (EMIP), pp. 1-7,
May 2022.
Joshua Bensemann, Alex Peng, Diana Benavides-Prache Yang Chen, Neset Tan, Paul Michael Cor- ballis, Patricia Riddle, and Michael Witbrock. Eye Gaze and Self-attention: How Humans and Transformers Attend Words in Sentences. In Proceedings of the Workshop Cognitive Modeling and Computational Linguistics, pp. 75-87, Dublin, Ireland, May 2022. Association for Compu- tational Linguistics. doi: 10. 12..8666/1120222cmee-l1
10

---

Under review as a conference paper at ICLR 2023
Tanja Blascheck and Bonita Sharif. Visually analyzing eye movements on natural language texts and source code snippets. In Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications, ETRA 19. pp. 1-9, New York. NY, USA. June 2019. Association for Computing doi: 10.1 10.134/333111 1.3319917.
Machinery. ISBN 978-1-430 1-4503-6709-7
Gino Brunner, Yang Liu, Damian Pascual, Oliver Richter, Massimiliano Ciaramita, and Roger Wat- tenhofer. On Identifiability Transformers. ICLR, 2020.
Teresa Busjahn, Roman Bednarik, Andrew Begel, Martha Crosby, James H. Paterson, Carsten Schulte, Bonita Sharif, and Sascha Tamm. Eye Movements in Code Reading: Relaxing the Linear Order. In 2015 IEEE 23rd International Conference on Program Comprehension, pp. 255-265, May 2015. doi: 10.1 109/CCC..015..66
Hila Chefer, Shir Gur, and Lior Wolf. Transformer Interpretability Beyond Attention Visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 782-791, 2021.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Broekman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such. Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage. Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating Large Language Models Trained on Code. arXiv:2107.03372 [cs], July 2021.
Jacob Devlin, Ming-Wos Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lan- guage Technologies, Volume (Long and Short Papers), pp. 4171-4188 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi:
Oliver Eberle, Stephanie Brandl, Jonas Pilot, and Anders Søgaard. Do Transformer Models Show Similar Attention Patterns to Task-Specifi Human Gaze? In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 4295- 4309, Dublin, Ireland, May 2022. Association for Computational Linguistics. 1022.acl-1o...296
LinguttiiinnnnLnniiiiiiiiieei doi: 10.18653/v1/
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. InCoder: A Generative Model for Code Infill- ing and Synthesis, April 2022.
Vincent J. Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. Global Relational Models of Source Code. In International Conference on Learning Representations, February 2020.
Sarthak Jain and Byron C. Wallace. Attention not Explanation. arXiv:1902.1011 [cs], May 2019.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbi, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Push- meet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition- Level Code Generation with AlphaCode. arXivv2203.07814 [cs], February 2022.
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A Structured Self-AAtentive Sentence Embedding. In International Conference Learning Representations, July 2017.
11

---

Under review as a conference paper at ICLR 2023
H. B. Mann and D. R. Whitney. On Test of Whether one of Two Random Variables is Stochastically Larger than the Other. The Annals of Mathematical Statistics, 18(1):50-60, March 1947. ISSN 0003-4851, 2168-89960
doi: 10.1214/amms11177733000
Mary L. McHugh. Interrater reliability: The kappa statistic. Biochemia Medica, 22(3)276-222,
October 2012. ISSN 1330-0962.
Roberto Minelli, Andrea Mocci, Michele Lanza, and Lorenzo Baracchi. Visualizing Developer Interactions. In 2014 Second IEEE Working Conference Software Visualization, pp. 147-156, September 2014, doi: 10.1 1.110//VSSSOFF:204...11
Grégoire Montavon, Alexander Binder, Sebastian Lapuschkin, Wojeiech Samek, and Klaus-Robert Müller. Layer-Wise Relevance Propagation: An Overview. Wojciech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller (eds.), Expluinable Al: Interpreting, Explaining and Visualizing Deep Learning, Lecture Notes in Computer Science, pp. 193-209. Springer International Publishing, Cham, 2019. ISBN 978-3-030-28954-6. doi:
0-17799788-033--8855---6
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. A Conversational Paradigm for Program Synthesis, March 2022.
Matteo Paltenghi and Michael Pradel. Thinking Like Developer? Comparing the Attention of Humans with Neural Models of Code. In 202/ 36th IEEF/ACM International Conference on Automated Software Engineering (ASE), pp. 867-879, November 2021. doi: 10.1109//5551544
2021.9678712.
Norman Peitek, Annabelle Bergum, Maurice Rekrut, Jonas Mucke, Matthias Nadig, Chris Parnin, Janet Siegmund, and Sven Apel. Correlates of Programmer Efficacy and Their Link Experi- ence: A Combined EEG and Eye-Tracking Study. In Proceedings of the 30th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, pp- 12. Association for Computing Machinery, 2022.
Julian Aron Prenner and Romain Robbes. Automatic Program Repair with OpenAl's Codex: Eval- [cs], November 2021.
uating QuixBugs.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving Language Under- standing by Generative Pre- Traininnn pp. 12.
Laria Reynolds and Kyle McDonell. Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, CHI EA '21, pp. 1-7, New York, NY, USA, May 2021. Association for Computing Machinery. ISBN 978-
doi: 10.1145/341 176.334160.
Elizabeth R. Schotter, Bernhard Angele, and Keith Rayner. Parafoveal processing in reading. Attention, Perception, & Psychophysics, 74(1):5-35, January 2012. ISSN 1943-393X. doi: 10.37588113414--11-00224
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguisties (Volume 1: Long Papers). pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: (0.186633v1/P16-111
Zohreh Sharafi, Ian Bertram, Michael Flanagan, and Westley Weimer. Eyes on Code: A Study on Developers' Code Navigation Strategies. IEEE Transactions Software Engineering, 48(5): 1692-1704, May 2022. ISSN 1939-3520. doi: 10.11099T9E.222033330666
Disha Shrivastava, Hugo Larochelle, and Daniel Tarlow. Repository- Prompt Generation for Large Language Models of Code, June 2022.
Ekta Sood, Simon Tannert, Diego Frassinelli, Andreas Bulling, and Ngoc Thang Vu. Interpreting Attention Models with Human Visual Attention in Machine Reading Comprchension. In ACL SIGNLL Conference Computational Natural Language Learning (CoNLL.), October 2020.
12

---

Under review as a conference paper at ICLR 2023
C. Spearman. The Proof and Measurement of Association between Two Things. The American Journal of Psychology. 100(3/4):441-44 4711, 1987. ISSN
doi: 10.2307/1422689.
Rachel Turner, Michael Falcone, Bonita Sharif, and Alina Lazar. An eye-tracking study assessing the comprehension of c++ and Python source code. In Proceedings of the Symposium on Eye Tracking Research and Applications, pp. 231-234, Safery Harbor Florida, March 2014. ACM.
ISBN 978- -550--275--.. doi: 10.1145/257815325582215
Martin Tutek and Jan Snajder. Staying True to Your Word: (How) Can Attention Become Explana- tion?, May 2020.
Shikhar Vashishth, Shyam Upadhyay, Gaurav Singh Tomar, and Manaal Faruqui. Attention Inter- pretability Across NLP Tasks, September 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention All you Need. Advances in Neural Information Processing Systems, 30:5998-6008, 2017.
Jesse Vig and Yonatan Belinkov. Analyzing the Structure of Attention in a Transformer Language Model. In Proceedings of the 2019 ACI. Workshop BlackboxNI.P: Analyzing and Interpreting Neural Networks for NLP, pp. 63-76, Florence, Italy, August 2019. Association for Computa- tional Linguistics. doi: 10.18653/v1/V 1/11//////1191000 19-4808.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer Networks. Advances in Neural Infor- mation Processing Systems, 28, 2015.
Yao Wan, Wei Zhao, Hongyu Zhang, Yulei Sui, Guandong Xu, and Hai Jin. What do they capture? structural analysis of pre-trained language models for source code. In Proceedings of the 44th International Conference on Software Engineering, ICSE '22, pp. 2377-2388, New York, NY, USA, May 2022. Association for Computing Machinery. ISBN 978- 978-1-45033-221-1. doi: 10.
1145/3510003.33 10003.35000.0000
Sarah Wiegreffe and Yuval Pinter. Attention is not not Explanation. arXiv:1908.0462. [cs], Septem- ber 2019.
Tingyi Yuan, Xuhong Li, Haoyi Xiong, Hui Cao, and Dejing Dou. Explaining Information Flow Inside Vision Transformers Using Markov Chain. In eXplainable Al Approaches for Debugging and Diagnosis., October 2021.
Kechi Zhang, Ge Li, and Zhi Jin. What does Transformer learn about source code?, July 2022.
13

---

Under review as a conference paper at ICLR 2023
A SENSE-MAKING QUESTIONS
In Table we report the sense-making questions used in the study. Note that each pair of code snippet and question has variable length depending the program- ming languages it is written with. The source code snippet can be found here: attps://anonymuus. 4open. eye sve_trrakiingsoureee
Snippet Name
Content Tower of Hanoi problem
LoC
Question
hannoi_QI
28-49
How does the algorithm moves disks from the starting rod to the ending rod?
hannoi_Q2 Q2
26-47
Which is the base of the algorithm?
hannoi Q3
28-50
Which is the name of the auxiliary rod in the call TowerOfHanoi(n, 'Mark', 'Mat', 'Luke')?
multithread Q1
Consumer-
106-
possible that consumer and producers threads end in a deadlock state, namely they both wait for each other to finish, but none of them is doing anything?
producer threads
116
multithread.00 QQ2
104-
there any line of code the consumer or pro- ducer code which will never be executed? If yes, report it below.
112
multithread Q3
104-
Will the queue object ever raise exception in this program? If yes, which condition(s) should be met for the exception be raised?
113
nqueens Q1
N queens problem
78-100
What does "solveNQ(-13) return?
nqueens_ _2
78-101
What are valid dimensions and values for the array 'board"?
nqueens.QQ Q3
78-100
How would you expect the run time of "sol- veNQ(n)" to scale with 'n'?
tree_Ql
Recursive tree con- struction
87-99
How many calls 'construucTreeUtii will "con- structTree(11,2 2, 3], [ 1, 2, 3), 2) make?
tree Q2
87-99
Under
which conditions could the check ;= 'constructTreeeUii! false?
tree_Q3
89-101
A part of the code you don't have direct access to has called "constructTree" with unknown parame- ters. What you find out about those parame- ters?
triangle QI
Triangle classifi- cation
66-112
Which of the functions have side effects (namely it modifies some state variable value outside its local environment?
triangle Q2
66-113
Questions: Which output will you get for the three points [1, 2). [1, 3], and [1, 4]?
triangle_Q2
66-112 What could happen if the call to 'order()* were omitted from classifyTrianglee
Table 1: Code snippets and related question for each sense-making task.
B DETAILS: EXTENDED RELATED WORK
Here give a comprehensive description of the related work.
14

---

Under review as a conference paper at ICLR 2023
B.1 ATTENTION ON NATURAL LANGUAGE MODELS
Previous work @Jiin & Wallace 2019) on studying attention weights of recurrent neural network models has found that the attention weights do not agree with other gradient based explanation methods and that alternative weights can be adversarially constructed while still preserving the same model prediction. Following work from Wiegreffe & Pinter (2019) showed how the alternative attention weights can be constructed only per single instance prediction, whereas obtaining model which is consistently wrong in its explanations is very unlikely to happen.
On the same line, Tutek Snajder (2020) has proposed four regularization methods to mitigate the adversarial exploitation of attention weights for recurrent models, resulting in attention weights that when changed are more likely to produce change in model prediction behavior. Among the proposed techniques, the authors mention the use of residual connections, between token embeddings and output hidden state, and a masked language model objective. Notably, both techniques are naturally used by BERT (Devlin et al., 2019). model based solely transformers.
This observation agrees with the techniques proposed by Abnar Zuidema 2020 as post- processing for attention weights. The authors explicitly account for residual connections when in- terpreting attention of transformer-based models.
Vig & Belinkov (2019) have studied the attention of GPT-2 model and observed that some attention heads are specialized in paying attention to specific part-of-speech tags. However, they focus on the smallest version of the model and they only consider natural language setting of Wikipedia articles.
Multiple previous works Sood et al. (020); [ennsemann et al. (2022); ; Eberle et al. (2022) have compared model and human attention on several natural language understanding tasks showing how the two are aligned. In particular, on reading comprehension tasks, Sood et al. (2020) show that more accurate model does not necessarily have stronger correlation with the humans.
B.2 ATTENTION STUDY ON MODELS OF CODE
Wan et al. (2022) have studied the self-attention of pre-trained models of code to see how the the attention of those model can be used to induce the abstract syntax tree of the source code. How- ever they limit to relatively small transformer models for code understanding which use only self- attention, whereas we study the first to study the largest version of CodeGen Nijkamp et al. (2022). generative model which pays attention only to preceding tokens. In the same vein, Zhang et al. (2022) have shown how the attention weights of pre-trained model on source code capture impor- tant properties of the abstract syntax tree of the program pointing towards the idea that the attention weights of pre-trained model on source code, which we now disregard, might contain valuable in- formation.
B.3 EYE- TRACKING STUDIES
eye-tracking study involving 38 students fixing or describing five simple Python and C++ programs (5-13 LoC) has shown that the fixation duration is comparable between the two lan- guages (Turner al.] 2014). [Beelders (2022) qualitatively observe the eye movement of 36 stu- dents and four lecturers when reading and mentally executing a short C# program (12 LoC). An eye tracking dataset with 216 participants was collected by [Bddarikk et al.] (2020), however they only consider two short snippets (11-22 LoC) of code, since they do not support scrolling. Blascheckk & Sharif (2019) and Busjaahn et al. (2015) instead studied the reading order in C++ and Java code comprehension task focusing six small programs that could fit into single screen, whereas consider longer snippets and a much larger dataset of 45 unique tasks. [Sharafi et al. (2022) have recently studied code navigation strategies Java code with eye tracking involving 36 participants focussing on the bug fixing process, whose specific goal and might elicit a different kind of reasoning compared our sense-making tasks.
B.4 NEURAL MODELS vs HUMANS
Paltenghi & Pradel (202I) have compared the attention weights of particular neural models of code and developers' visual attention when performing code summarization task, spotting positive
15

---

Under review as a conference paper at ICLR 2023
correlation between the two. They found stronger correlation the copy attention mechanism, which an instance of a pointer network (Vinyals et al. 2015) used by the model to copy verbatim the code from the input to the output. However, they deblurring interface which might bias the cognitive process of the developer, wehreas we use eye tracking data which are collected in the more natural IDE setting.
B.5 ALTERNATIVES TO ATTENTION
There other observable signals than attention that might capture the concept of relevance. In par- ticular, there have been some approaches based, partially or entirely, on gradient information, such as GradCam Yuan et al. 2021 Chefer al. (202T). These are promising avenues, however, focus on attention for two reasons: (1) almost all state-of-the-art models of code are based on the trans- former block, and the attention mechanism is ultimately its fundamental component (Vaswani al., 2017). for which reason we expect the corresponding attention weights to carry directly meaningful information about the decision process of these transformer-bes -bsed models; (2) unlike the gradients, the attention weights can be extracted almost for free during generation with little runtime overhead, a crucial condition many practical scenarios.
B.6 IMPORTANCE OF CODE NAVIGATION
recent study by Peitek et al. 2022) has found that higher program comprehension efficacy is correlated to lower coverage of code elements, namely developers with better code comprehen- sion also navigate the code with more focused approach. Their results shows how an accurate code navigation is crucial for the efficacy of code comprehension and further motivates our work. Minelli jet al. (2014) studies and visualizes how developers interact with a real IDE, however they only use the events of the IDE without any eye tracking date, thus the granularity of our dataset is higher.
C DETAILS: ANNOTATION PROCEDURE
We annotate each answer generated by either the developer the model involving four annotators in the process. We use scale of three values of correctness: (1) correct, when the answer touches all the expected correct points, (2) partial, if least part of the correct answer is present or if the answer is wrong but in the same style of the correct solution (c.g. the Big-O notation), (3) wrong, when the answer does not contain any correct part of the answer. Note that, especially for the model, the model generates extra text beyond the correct or partial answer we ignore the rest if it is incorrect. Moreover, whenever the question is under-specified we accept multiple correct answers as long as they are compatible with the question.
To ensure a reproducible annotation process, all the authors collectively come up with a shared set of gold standard answers for each question. Then, two of the authors independently annotates more than 20% of the answers generated by the model and the developer, and within two rounds of annotation followed by discussion, the final set of gold standard answers is agreed upon. The final agreement on the 20% of data led to a Cohen's 's Kappa of 0.898 and 0.711, 1, which is considered a very high agreement McHugg (2012). Finally, the remaining 80% of the data is split in half and annotated only by one of the two authors individually.
D DETAILS: EYE TRACKING DATA POST- POST-PROCSSIING
We use eye tracker from GazePoint, which is placed below the monitor thus not requiring the user to wear any additional device. The participants can see between to 26 lines of code. The screen size is 52.7 mm x 29.6 mm with resolution of 1920x1080 pixels. The participant seats at a fixed distance of ca 30cm from the screen.. We do not enforce this via a resting support for the chin, since the eye tracker specification claims to be robust this kind of movements. We collect eye fixation events euteye as a tuple (t, x. y. d) where f is the timestamp in milliseconds, x and y are the coordinates of the fixation point in pixels and d is the duration of the fixation in milliseconds.
Beside the eye data, we also continuously log the visible text in VSCode by using a custom plugin, so that know what was visible at given x, y coordinate at each point time. A visible text
16

---

Under review as a conference paper at ICLR 2023
event ceteat corresponds to tuple (t, tzt, f. 1) where is the timestamp in milliseconds, trt is the visible text, f is the file name shown in the code area, is the line number of the first visible line with respect to the given file. Note that this data source is crucial since we study long code snippets and allow also screen scrolling. To ensure that we have a consistent grid mapping between pixel positions and char positions in the text, where only half a line is scrolled). Thus via our plugin we allow only scrolling of a discrete number of lines.
monospace font and prevent partial scrolling (i.e.
For additional sanity check we also collect a screen recording along with the eye tracking recording. At any point time the video shows where is the fixation of the participant.
Note that our setup is as close as possible to normal coding session, without any invasive method.
To derive the developer attention maps from the eye tracking data, we first synchronize the two data sources, i.e. from the VSCode plugin and the eye tracker, via their timestamps.
Then we derive the and y coordinate of the code area displayed on the screen for given video frame by using two visual markers, which delimit the top-left and bottom-right corners respectively. Note that, since the plugin prevents any resizing of the code area during the experiment, any frame would be suitable, for our case we use the frame in the middle of the video screen recording. Then, knowing the fixed number of lines and columns displayed in the code area, we divide the code area in small rectangular areas, each of which corresponds to specific character position identified by line and column.
Then, we convert the fixation point of gaze x and y coordinates of each evleye to the correspond- ing character position in the code area, in the form of line and column coordinates the relative coordinate system of the code area. And knowing the line number we can attribute the fixation to a specific character position in the original file. In this way, we convert each euteye its equivalent
event in character coordinates evllhhr) (char) e,, I, d) where is the timestamp, and I are the col- umn and line coordinates with respect to the original file of the fixation point and d is the duration of the fixation.
Since it is hard to tell whether a participant is looking at specific character or a group of neigh- boring characters during single fixation points, we attribute the attention of the developer also to neighboring characters, namely, if the developer looks at position (c. () in the original file, we augment our data by introducing new events which point to all the neighboring characters within a vertical offset Voff and an horizontal offset hoff from our coordinate (c, (). As result, we replace each basic (chha)))) = (t, I,c,,,dd d) with the set of derived events (t, Cnew y knew; d) where c-hoff S Cnew c hoss and - Voff S Inew 51+ Doff-
Indeed, as reported by by(Schotter et al. (2012) our fovea region, which is responsible for a sharp central vision, accounts for 2 of the visual field, whereas the parafoveal region, which is used for visual search and scene perception, accounts for 50 of the visual field. Thus considering 50 visual region and our screen size (527mm 296mm) the developer characters vertically. Rounding those quantities we set ooff = and hoff = 4. This approach also contributes to mitigate eventual small and y errors in the eye tracking data collection.
see 7.16 characters horizontally and 2.92
Note that that our offset allows to give attention also to blank spaces which are not present in the original line (e.g. after the end of line), thus we do not assign any weight to those characters for better comparison with the neural models which cannot see them.
E EYE TRACKING DATASET
The collected eye tracking dataset contains data from 25 participants over 92 valid sessions. We discarded the data of the sessions where two participants who added comments in the code since it was unclear how to treat the fixations this newly created text; especially when computing the connection strength between two interaction events with an interlcaving fixation event on this newly inserted comment. Figure 8 shows the number of participants per task and programming language.
17

---

Under review as a conference paper at ICLR 2023
10
cpp
8
Numbor
Nqueens, 02
Nqueens, Q3
hanrol 02
hannol .a
multihheadd Q1
mutthhnad, 02
multihreed, 03
tree, Q1
tree 02
tree 03
triangle, 01
triangle, 02
triangle. Q3
Nqucons
hannol
Task
Figure 8: Total number of participants per task divided by programming languages.
F OBSERVED DEVELOPER CODE TRAVERSAL
Developers do not jump randomly between likely locations within the code: they have significant bias for staying close their current position. We therefore attempt to predict the observed strength of the temporal connection between tokens solely on the basis their relative position.
We suggest two-tiered approach: first consider whether the developer is traversing forwards or backwards, then use a relative model for how far they will move in that direction.
We expect the ratio of forwards backwards traversal to be dependent the exact task, and in fact in our data set its average by task ranged from 5:6 (0. 846) to 7:2 (3.58). For each task indivicually, however, well some extent in general, the best simple predicting feature appears to be the current token position divided by the total number of tokens, i.e. the ratio of the document still in front of the developer. Fitting individual linear regressions for going forward and backward (which do not sum to because of the chance of returning to the token itself) gives the predictions of
Σ St.5 RJ 0.940639 - 0.745988 * i/max (R²2 0.56)
(3)
Σ Sij 0.052612 0.74777 (R² 0.56)
(4)
JZl
We expect, and find, the distribution for the distance to depend less on the task. Of num- ber of standard distributions tested against (normal, poisson, lognormal, exponential, Pareto, Weibull), it is by far best modelled using a fitted Weibull distribution, with best fit of shape 89323333 scale 98. 1442010 tckens going forward and shape 0.8831494, scale 105. 6159039 tokens going backward (see Figure
G TASK CORRECTNESS
Figure 9 shows the correctness the answers for each task, for the three models and for the devel- opers.
18

---

Under review as a conference paper at ICLR 2023
Carre
Whon
Davelopers
CodaGan
Nesesv.cc C1
Neuoraa0 01
Mouera, 24
NoseersCa C3
faxesm..5 C2
Novore 01-
1005
Mosere, 32
Roacer.
Naseinu 52 400m
Nqveen, 00
30%
Noueen, 02
Noweens C3
5037
fores 01 61-48h
52% 1000
saved 01 30%
Novel
42%
Tanrs 07
savvl 07
lerne
tandd
mukihrrad
multinad
ratroviad
muttorrad
mukiivred
multhradd
trangl.C
10
0.3
3.0
0.5
3.0
Figure 9: Percentage of correct, wrong and partially correct answer for each tas.
Count
20
Median: 0.17
Count
20
Median: 0,04
Mean: 0.17
Mean: 0.09
0
1.0
-0.5
0.0
0.5
1.0
-1.0
0.5
0.0
0.5
1.0
Spearman Rank Corr. Developers CodeGen (Python)
Spearman Rank Corr. Developers Gpt-j (Python)
Count
20
Median: 0.28
Count
20
Median: 0.03
Moan: 0.28
Mean: 0.05
0
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
Spearman Rank Corr. Developers CodeGen (C#)
Spearman Rank Corr. Developers Gpt-j (C#)
Count
Median: 0.16
Count
10
Median: 0.07
Mean: 0.18
Mean: 0.11
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
Spearman Rank Corr. Developers CodeGen (C+++)
Spearman Rank Corr. Developers vs Gpt-j (C++)
Figure 10:
Agreement between developers and CodeGen's visual attention (extraction function:
max) while performing code understanding tasks in different programming languages.
H PROGRAMMING LANGUAGE ANALYSIS
Is there any difference between programming languages? To assess the difference between lan- guages, we group the datapoints three groups, one for each language and we report mean and median for each group in Figure [0 and we compare the distribution of the Spearman rank correla- tion coefficients via the Mann Whitney Test (Mann Whitney 1947). The test reveals that the agreement between developer and models is significantly different between C# and Python snippets (see test results in Table 2). Although, visual inspection shows how the two models do not have the same tendency, with CodeGen agreeing more on C# and Gpt-J agreeing more on Python.
In Table we report the results of the Mann Whitney U statistical tests when comparing the distri- butions of developer-model agreement (Spearman rank coefficients) tasks written in two different languages.
I AGREEMENT ON INTERACTION MATRIX: GPT-J AND INCODER
In Figure II and 12 we report the results for the agreement on the interaction matrix between human and GPT-J and InCoder-6B.
19

---

Under review as a conference paper at ICLR 2023
Neural Model
Lang.
Lang.
Statistic
p-value
Result
Gpt-j
py
cpp
2613.0
9.37c-01
Probably same distrib.
Codegen
py
cpp
2499.0
7.22e-01
Probably same distrib.
Gpt-j
py cs
7637.0
2.00e-02
Probably diff. distrib.
Codegen
py
2626.0
9.29e-15
Probably diff. distrib.
Gpt-j
cpp cs
3404.0
6.61e-02
Probably same distrib.
Codegen
cpp
1486.0
9.95e-07
Probably diff. distrib.
Table 2: Results of the Mann Whitney U statistical tests when comparing the distributions of developer-model agreement (Spearman rank coefficients) on tasks two different languages.
Median
Mean
overlap for Top-3
up
Position
Follow-up
48.1%
50.6%
13:55
Comparioon name
Raw [sym.)*
Raw att. (1st) (sym.)
Mean
Max att. (sym.)"
)*
0.46
-0448
0.53
0.43
0.50
0.46
0.45
Compantean
Raw
Max Isym.)
Copy-cat
57.2%%
59.88%
61.5%
66.75
67.595
2422
14.00
15:8%
15:55a
Copy
att. tsym.
69.0%
24:
Raw att (2st layer)
70.6%
21:226
Uniform .0.04
.0.01
Raw layer!
71.6%
.18.26
Max att. 0.03
.0.02
Mean .0.03
+0.02
-0.03
Rollout att.
layor*
0..04
Rew att (last layert
%
Spearman Rank Com Coeff, Distribution
Percentage of next-line preeeppee predictions
(a) Spearman Rank Correlation
(b) Top-3 Overlap
Figure 11: Agreement between developers and GPT-J interaction matrix ordered from weak (top lines) to strong (bottom lines). Attention-based methods are marked with
J AGREEMENT TOKEN-LEVEL
In Figure 13 report the agreement computed at the token-level for the three models. Since they have different tokenizers, especially regarding how they treat spaces, we compute the correlation on the tokens that contain some meaning, namely we approximate this concept by considering only tokens with at least one alphabetic character. Indeed, eye tracking data cannot pinpoint with absolute precision when a user was looking single space char rather than its surrounding tokens, thus excluding them avoids the result being biased by this irrelevant tokens, from a human perspective.
K AGREEMENT VS ACCURACY
In Figure 14 we report the human-vs-modee agreement for different pairs of accuracy buckets mea- sured with wrong, partial or wrong. For example, the cell wrong-wrong is contains the correla- tion when both models and human produce wrong answer to the same question, whereas the cell correct-correct contains the correlation when the produce two correct answers.
20

---

Under review as a conference paper at ICLR 2023
Median
Mean
Size of everlap for Top-3 next lihas
vs 10.
Mean att. (sym.)*
Follow-up att. 0.31
0.32
56.6%
13
Comparison name
Raw
att. [oym.)"
Max att. (sym.)* 1oym.)*-0..x.
copy car
Position 0.30
[sym. [sym.)*-00
0.32
0.31
8.16
0.32
Raw (1st) tsym.
(last) laym.)
Max
Isym.
57.07%
58:19%
58.0%
60.0%
61.0%
33.5%
Mean att. 0.01
0.01
Moan
Max att, 0.00
-0.00
(last layert*
(Ist layer" 0.00
-0.00
Raw att. 11st layer!
Uniform
Uniferm
71.0%
Raw layer"
Copy cat
71.18%
Rollout att.
40.05
Rollout att.
14.00%
Spearman Rank Con: Coeft. Distribution
Parcentage of next line predictions
(a) Spearman Rank Correlation
(b) Top-3 Overlap
Figure 12: Agreement between developers and InCoder interaction matrix ordered from weak (top lines) to strong (bottom lines). Attention-based methods are marked with *
Median: 0.20
-1.00 -0.75
-0.25
0.50
0.75
Court
Median: 0.05
-1.00 -0.75
-0.25
0,00
0.50
0.75
Spearann Developers
(Gpt))
Coart
Median: 0.05
-0.35
0.5 0.75
Spcarman Developers (eCoden
Count
25
Median: 0.57
-6.35
Spcorman Develspers Developers
Figure 13: Agreement between developers and models" visual attention (extraction function: max).
Wrong
0.6
Portal Wrong
Human ng
0.6
0:20
6.11
0.05
0.14
0:58
0.5
0.5
6.5
Humen Score
5.4
Humon Score
0.4
Human
Purtal
0.04
9.9
0.21
0.31
5.65
6.3
62
62
Correct
Correct
8.1
0.18
0.50
6.1
8.0
0.0
Wrong Partal Correct
Wrong
Partial
Correct
Wrong
Partial
Correct
Wheng Portill Correct hunaan other Score
codegen Score
Score
incodor Score
Figure 14: Human-vs-Model agreement for different accuracy pairs.
21


Reviews:
[{"review_id": "lZmC94FVLP", "reviewer": "Reviewer_q7Ss", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an eye tracking dataset for code understanding task in three programming languages (Python, C++, C#). The dataset contained 92 visual attention sessions of 25 developers. Using this eye tracking dataset, authors study the correlation between model attention and developer’s attention. Authors empirically evaluate different attention-based post processing approaches of the model’s attention signal against the ground truth of developers exploring code.  \t", "summary_of_the_review": "While I don't see any issue with the study design and findings, I have reservations about the usability of this work. In particular, I don't think that this work advances our understanding of the relationship between model attention and developer attention in a significant way.   ", "strength_and_weaknesses": "### Strengths: \n \n- The proposed dataset will be a nice contribution to the community as it can be used for additional code understanding tasks. \n- Proposed “follow-up attention” is very intuitive and is shown to have better correlation with human attention than other approaches.\n\n### Weaknesses: \n\n- Authors study only a single code model Codegen and a single generic language model GPT-J. It’s not clear how model accuracy, model size play a role on the correlation between human and model attention. In particular, do we observe better correlation for larger and/or more accurate models ? On reading comprehension tasks, [3] show that a more accurate model does not necessarily have a stronger correlation so it's important to study such correlation for different models. \n- I have reservations about the usefulness of this study. Paltenghi & Pradel (2021) have also studied correlation between human and model attention. Further, multiple prior works [1, 2, 3] have shown that human attention is aligned with several natural language understanding tasks. Authors do not discuss how their work is different from the existing works and in what ways it advances our understanding. \n- Some of the design choices are not clear. In particular, authors say that unlike  Paltenghi & Pradel (2021) they study attention at char level. Why? I think developers focus on the whole variable name instead of a single char in a variable name so it’s not obvious why char level analysis is better than token level analysis. \n\nReferences: \n- [1] Bensemann, J., Peng, A., Prado, D., Chen, Y., Tan, N., Corballis, P. M., ... & Witbrock, M. J. (2022, May). Eye Gaze and Self-attention: How Humans and Transformers Attend Words in Sentences. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics (pp. 75-87).\n- [2]  Eberle, O., Brandl, S., Pilot, J., & Søgaard, A. (2022, May). Do Transformer Models Show Similar Attention Patterns to Task-Specific Human Gaze?. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 4295-4309).\n- [3] Sood, E., Tannert, S., Frassinelli, D., Bulling, A., & Vu, N. T. (2020). Interpreting attention models with human visual attention in machine reading comprehension. arXiv preprint arXiv:2010.06396.\n- [4] Paltenghi, M., & Pradel, M. (2021, November). Thinking Like a Developer? Comparing the Attention of Humans with Neural Models of Code. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) (pp. 867-879). IEEE.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "f7Pf66pOSho", "reviewer": "Reviewer_oGLs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper uses eye tracking studies on developers and correlates the patterns with the attention from code-related deep learning models (CodeGen and Code-J). CodeGen has higher agreement with developers (median = 0.23). The paper also reports interaction patterns. While attention-agnostic schemes (such as position) do well, the proposed “follow-up attention” has the best agreement.", "summary_of_the_review": "The paper addresses an important topic of correlating human attention with model attention. The authors have done a commendable job of using non-attention based baselines and reporting the performance. A deeper interpretation of the reported results, especially the good performance of non-attention baselines, is required to benefit other researchers.", "strength_and_weaknesses": "(+) Correlating model’s attention with human attention is an interesting area of research. The eye tracking setup used in the paper is more natural compared to other studies.\n\n(+) The paper proposes “follow-up attention” which is marginally better.\n\n(-) The paper needs to discuss how even attention-agnostic baselines perform well.\n\n(-) Figures 6 and 7 need to be reconciled. Raw attention in layer 1 performs poorly (Figure 6) while follow-up attention with only layer 0-1 is competitive.  The reasons need to be discussed.\n\n(-) In Figure 6(a), it might appear higher top-3 overlap should indicate better performance. Follow-up attention has lowest score. Explain.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ju_jjmJGV9", "reviewer": "Reviewer_6UaZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on attention activations in conventional transformer models: what they learn, and what information can be extracted out of them for other use-cases. Models are usually trained directly for another objective (e.g. completion), but the attentions themselves can be used with no or minimal processing for other tasks, because they capture important features of the input data. Attentions have been shown in previous work to be useful for various tasks, including saliency and interpretability.\n\nThe authors focused on code models, and in particular tracking what developers pay attention to when looking at code. They collected/released a dataset that tracked what programmers looked at when solving various problems, and compared this to what several different transformations (including one they introduced) of the attention activations would have predicted. They compared both how often each token was perceived and what token was looked at after each token, and found that attention can be useful for predicting these. Using attention in this way has promising implications for developer tools, such as suggesting which lines developers should look at when exploring an unfamiliar file.", "summary_of_the_review": "This paper proposes using attention for code exploration, and examines how well attention performs at predicting how developers explore code. The paper as a whole is well-written and presents its results clearly, but based on the evidence presented the significance of the work is somewhat questionable. There is no strong evidence presented of how this could lead to a useful development tool, or that attention activations are better for code exploration than relatively simple baselines.", "strength_and_weaknesses": "# Strengths:\n- The paper is clear and well-written, explaining everything necessary to understand the paper.\n- This is an interesting approach that could lead to a useful developer tool based on helping people understand code faster.\n- The code and dataset were released, which is useful for follow-up work.\n\n\n# Weaknesses:\n- The results seem not that impressive. \n  - The authors defined their own methodology and metrics, and then reported results on them, which makes it hard to understand the significance of the results (in particular, the follow-up attention method introduced in this paper).\n  - For the interaction matrix results, the attention-agnostic code retrieval patterns seem to perform as well or better than the attention mechanisms?\n  - There are no baselines for visual attention, so it's hard to extract meaning from the values given (beyond that the correlation is positive). Also, based on the results on the interaction matrix, I would guess that attention-agnostic predictions could also outperform attention based ones.\n  - There are no real measures of usefulness, or investigation into how this could be useful. Based on the results presented in this paper, it's hard to judge how useful a next step would be.\n- As part of the results seeming not impressive, it's hard to judge how significant or impactful this work is. It proposes an idea for how to use attention to create a developer tool, but not a compelling case of why attention is better than relatively simple alternatives, or why this tool is useful.\n\nSome ways of improving this would be to see if the predictions produced by this model are helpful for developers in practice, split the dataset into train/test/validation and train some baselines on this (e.g. freeze a transformer and train an additional layer on top of the last attention block to predict visual attention), or add some basic baselines to visual attention (e.g. favor earlier tokens, or favor later tokens, or generally look at the developer patterns and try to come up with simple rules to explain them).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JGkWBeRqrjx", "reviewer": "Reviewer_ZiWC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tries to measure the correlation between attention weight patterns for CodeGen and GPT-J on code understanding questions and compare that with eye tracking based attention patterns for humans when answering those same questions.\n\nThey use 4 different attention weight metrics -- \n\n(1) Attention mean across all layers and heads\n(2) Attention max across all layers and heads\n(3) Rollout attention\n(4) Follow-up attention - a novel metric where they compute what they call follower score which models how the attention flows between tokens across layers. The hypothesis is that this can be similar to how humans focus on one set of tokens and then decide to focus on another set where the second set depends on the first set\n\nThey evaluate two different views --\n\n(1) Visual attention -- which is the aggregate attention across time \n(2) Interaction matrix -- which is the chance of looking at token i after token j\n\nThey find non-trivial correlation coefficients between the different types of attentions and attention metrics derived from humans for visual attention (for mean and max). They also find high correlation as well as predictive power as to what line will be attended next for interaction matrix view -- especially for follow up attention\n\nThe correlation is generally a lot less for GPT-J vs CodeGen\n\nThey also measure correlation difference between languages and find higher agreement for C# vs Python", "summary_of_the_review": "The paper provides a new eye tracking dataset as well as a thorough analysis of how it relates to attention in code generation models. Multiple aspects of the attention matrix are evaluated and a novel Transformer attention matrix based metric is developed which shows higher correlation with human attention.", "strength_and_weaknesses": "Strengths:\n\n(1) The method of eye tracking is well conceived and executed \n\n(2) There is a thorough exploration of various attention metrics and their relation to eye-tracking\n\n(3) Care is taken to properly define what interaction means in context of eye tracking -- as users can casually skim through code without attention to one piece of code necessarily being prompted after reading another piece of code\n\n(4) Follow-up attention is a novel metric that shows much improved correlation with eye tracking data\n\nWeaknesses;\n\n(1) Only CodeGen and GPT-J (which is not a code generation oriented model) are tested. It would be better to test on other models like Incoder and others that have comparable performance to CodeGen and see if the findings can be replicated.\n\n(2) Do we see higher attention matrix correlation when both the model and the developer have the right answers vs. when they don't?\n\n(3) It would be nice to see how much the attention matrix is correlated with the data flow or call graph -- and if edges between tokens in the interaction matrix that correspond to data flow are removed -- whether we still obtain a correlation between the attention matrix and the eye tracking dataset. This would help understand what aspects of the code induce the Transformer attention matrix to have similarity to the human attention matrix.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1C-8a6Mey2", "reviewer": "Reviewer_AZZZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a study which compares developers' attention to input context for sense-making tasks (e.g., mental code execution, side effects detection, algorithmic complexity, deadlock detection) with that of large autoregressive pretrained language models (CodeGen, GPT-J). For this, they collect a dataset through eye-tracking experiments involving 25 developers. For the model, they encode the prompt and analyze how tokens in the prompt attend to one another, across heads and layers. They consider many different post-processing techniques for attention, including mean, max, rollout, and a newly developed technique called follow-up attention that is designed to explicitly model the temporal relationship between attention weights. With respect to visual attention at the character-level, there is a weak positive correlation between model attention weights and developers' attention. For interaction attention at the line-level (given a position in the prompt, which part of the prompt is likely to be attended to next), there is a more moderate positive correlation. They consider the task of predicting the next line that the developer is likely to look at next, and using follow-up attention yields the best performance for this.\n\nContributions:\n- A large eye-tracking dataset corresponding to different sense-making code tasks\n- Empirical results for how the attention employed by models compare to developers\n- The follow-up attention processing technique which provides more signal on what line a developer is likely to look at next, which could be useful for supporting end tasks related to code navigation\n", "summary_of_the_review": "Overall, the study presented in this work draws some interesting connections between how developers inspect code and the attention weights of large language models. The authors also present a novel algorithm for post-processing attention which incorporates the temporal aspect. However, I have some concerns about whether the attention weights which were extracted actually align with how a model is solving the underlying task and whether the low model performance suggests that it might not be attending to appropriate tokens. Additionally, it seems like there is a lot of variability in eye-tracking data and attention weights. So, I find some of the comparisons and claims made in this paper to be not as convincing.", "strength_and_weaknesses": "Strengths:\n- With growing work in the space of understanding the reasoning capabilities of large language models, I find this to be a very interesting piece of work which tries to relate human reasoning with model reasoning through proxies (eye-tracking and attention).\n- The idea of leveraging the temporal relationship through layers to process attention weights is quite novel, and the impact that it has on predicting the next line the developer is going to look at is impressive.\n- The authors have carefully controlled for various side effects (e.g., neighboring effect, temporal connect) when running experiments and presenting for results.\n- Extensive empirical analysis of attention post-processing techniques, which could be useful for future research which employs attention weights of models for different end tasks.\n\nWeaknesses:\n- The underlying motivation for this work is comparing the alignment between developer attention and model attention when solving the task. Since the eye-tracking data is collected while the developer is writing the answer to the question in the prompt, I can see how this can be considered an appropriate proxy for what the developer is looking at \"when solving the task.\" However, my understanding is that the model's attention weights are gathered from the teacher-forced prompt tokens only, and not from when new tokens are being generated to actually solve the task. Therefore, I question whether this is in fact an appropriate proxy.\n- From Figure 4, the performance of the models are quite low, with them only achieving correctness 23.7% of the time. This suggests that these models are not performing well on the sense-making tasks. Since they are not predicting correct answers, isn't it possible that their reasoning that led to these incorrect answers is also incorrect? So, does it really make sense to analyze and make claims based on these attention weights?\n- In Figure 4, CodeGen and GPT-J achieve very similar performance. However, in Figure 5, their correlation with developers is very different, with GPT-J achieving very low correlation. This seems to suggest that models may attend to the input context very differently from one another and still achieve similar results. Additionally, the developers vs developers correlation is also only moderately positive, suggesting that developers may inspect the input context differently and still provide the similar answers to the sense-making questions. Therefore, it seems like there is a lot of variability in proxies like eye-tracking and attention which make it difficult to compare reasoning of models and humans.\n- While authors compare to a number of attention-agnostic baselines like cpy-cat, uniform, and position in Figure 6b, I feel that a simple rule-based baseline which always just predicts the next token/next line (which mimics a left-to-right reading paradigm) is missing. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject