Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an eye tracking dataset for code understanding task in three programming languages (Python, C++, C#). The dataset contained 92 visual attention sessions of 25 developers. Using this eye tracking dataset, authors study the correlation between model attention and developer’s attention. Authors empirically evaluate different attention-based post processing approaches of the model’s attention signal against the ground truth of developers exploring code.  	
strength_and_weaknesses: ### Strengths: 
 
- The proposed dataset will be a nice contribution to the community as it can be used for additional code understanding tasks. 
- Proposed “follow-up attention” is very intuitive and is shown to have better correlation with human attention than other approaches.

### Weaknesses: 

- Authors study only a single code model Codegen and a single generic language model GPT-J. It’s not clear how model accuracy, model size play a role on the correlation between human and model attention. In particular, do we observe better correlation for larger and/or more accurate models ? On reading comprehension tasks, [3] show that a more accurate model does not necessarily have a stronger correlation so it's important to study such correlation for different models. 
- I have reservations about the usefulness of this study. Paltenghi & Pradel (2021) have also studied correlation between human and model attention. Further, multiple prior works [1, 2, 3] have shown that human attention is aligned with several natural language understanding tasks. Authors do not discuss how their work is different from the existing works and in what ways it advances our understanding. 
- Some of the design choices are not clear. In particular, authors say that unlike  Paltenghi & Pradel (2021) they study attention at char level. Why? I think developers focus on the whole variable name instead of a single char in a variable name so it’s not obvious why char level analysis is better than token level analysis. 

References: 
- [1] Bensemann, J., Peng, A., Prado, D., Chen, Y., Tan, N., Corballis, P. M., ... & Witbrock, M. J. (2022, May). Eye Gaze and Self-attention: How Humans and Transformers Attend Words in Sentences. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics (pp. 75-87).
- [2]  Eberle, O., Brandl, S., Pilot, J., & Søgaard, A. (2022, May). Do Transformer Models Show Similar Attention Patterns to Task-Specific Human Gaze?. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 4295-4309).
- [3] Sood, E., Tannert, S., Frassinelli, D., Bulling, A., & Vu, N. T. (2020). Interpreting attention models with human visual attention in machine reading comprehension. arXiv preprint arXiv:2010.06396.
- [4] Paltenghi, M., & Pradel, M. (2021, November). Thinking Like a Developer? Comparing the Attention of Humans with Neural Models of Code. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) (pp. 867-879). IEEE.

clarity,_quality,_novelty_and_reproducibility: ### Clarity, Quality:
- Paper is well written and is easy to follow. Experimental design is discussed in detail. While the paper cites the relevant literature in the code analysis domain, I think authors should also cite relevant NLP papers studying correlation between human and model attention.

### Novelty:
- The proposed dataset is novel. 
- To the best of my knowledge, the proposed “follow-up attention” approach is also novel.  

### Reproducibility:
- Authors plan to release the dataset so results presented in the paper should be reproducible. 



summary_of_the_review: While I don't see any issue with the study design and findings, I have reservations about the usability of this work. In particular, I don't think that this work advances our understanding of the relationship between model attention and developer attention in a significant way.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper tries to measure the correlation between attention weight patterns for CodeGen and GPT-J on code understanding questions and compare that with eye tracking based attention patterns for humans when answering those same questions.

They use 4 different attention weight metrics -- 

(1) Attention mean across all layers and heads
(2) Attention max across all layers and heads
(3) Rollout attention
(4) Follow-up attention - a novel metric where they compute what they call follower score which models how the attention flows between tokens across layers. The hypothesis is that this can be similar to how humans focus on one set of tokens and then decide to focus on another set where the second set depends on the first set

They evaluate two different views --

(1) Visual attention -- which is the aggregate attention across time 
(2) Interaction matrix -- which is the chance of looking at token i after token j

They find non-trivial correlation coefficients between the different types of attentions and attention metrics derived from humans for visual attention (for mean and max). They also find high correlation as well as predictive power as to what line will be attended next for interaction matrix view -- especially for follow up attention

The correlation is generally a lot less for GPT-J vs CodeGen

They also measure correlation difference between languages and find higher agreement for C# vs Python
strength_and_weaknesses: Strengths:

(1) The method of eye tracking is well conceived and executed 

(2) There is a thorough exploration of various attention metrics and their relation to eye-tracking

(3) Care is taken to properly define what interaction means in context of eye tracking -- as users can casually skim through code without attention to one piece of code necessarily being prompted after reading another piece of code

(4) Follow-up attention is a novel metric that shows much improved correlation with eye tracking data

Weaknesses;

(1) Only CodeGen and GPT-J (which is not a code generation oriented model) are tested. It would be better to test on other models like Incoder and others that have comparable performance to CodeGen and see if the findings can be replicated.

(2) Do we see higher attention matrix correlation when both the model and the developer have the right answers vs. when they don't?

(3) It would be nice to see how much the attention matrix is correlated with the data flow or call graph -- and if edges between tokens in the interaction matrix that correspond to data flow are removed -- whether we still obtain a correlation between the attention matrix and the eye tracking dataset. This would help understand what aspects of the code induce the Transformer attention matrix to have similarity to the human attention matrix.
clarity,_quality,_novelty_and_reproducibility: Clarity:

The paper is well written

Quality:

It would be good to have p-values for the correlation coefficients for the sake of completeness. That said, visually looking at the distributions, the results do seem statistically significant

Novelty:

The eye tracking dataset is novel. The new attention metric of followup attention is novel as well.
summary_of_the_review: The paper provides a new eye tracking dataset as well as a thorough analysis of how it relates to attention in code generation models. Multiple aspects of the attention matrix are evaluated and a novel Transformer attention matrix based metric is developed which shows higher correlation with human attention.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on attention activations in conventional transformer models: what they learn, and what information can be extracted out of them for other use-cases. Models are usually trained directly for another objective (e.g. completion), but the attentions themselves can be used with no or minimal processing for other tasks, because they capture important features of the input data. Attentions have been shown in previous work to be useful for various tasks, including saliency and interpretability.

The authors focused on code models, and in particular tracking what developers pay attention to when looking at code. They collected/released a dataset that tracked what programmers looked at when solving various problems, and compared this to what several different transformations (including one they introduced) of the attention activations would have predicted. They compared both how often each token was perceived and what token was looked at after each token, and found that attention can be useful for predicting these. Using attention in this way has promising implications for developer tools, such as suggesting which lines developers should look at when exploring an unfamiliar file.
strength_and_weaknesses: # Strengths:
- The paper is clear and well-written, explaining everything necessary to understand the paper.
- This is an interesting approach that could lead to a useful developer tool based on helping people understand code faster.
- The code and dataset were released, which is useful for follow-up work.


# Weaknesses:
- The results seem not that impressive. 
  - The authors defined their own methodology and metrics, and then reported results on them, which makes it hard to understand the significance of the results (in particular, the follow-up attention method introduced in this paper).
  - For the interaction matrix results, the attention-agnostic code retrieval patterns seem to perform as well or better than the attention mechanisms?
  - There are no baselines for visual attention, so it's hard to extract meaning from the values given (beyond that the correlation is positive). Also, based on the results on the interaction matrix, I would guess that attention-agnostic predictions could also outperform attention based ones.
  - There are no real measures of usefulness, or investigation into how this could be useful. Based on the results presented in this paper, it's hard to judge how useful a next step would be.
- As part of the results seeming not impressive, it's hard to judge how significant or impactful this work is. It proposes an idea for how to use attention to create a developer tool, but not a compelling case of why attention is better than relatively simple alternatives, or why this tool is useful.

Some ways of improving this would be to see if the predictions produced by this model are helpful for developers in practice, split the dataset into train/test/validation and train some baselines on this (e.g. freeze a transformer and train an additional layer on top of the last attention block to predict visual attention), or add some basic baselines to visual attention (e.g. favor earlier tokens, or favor later tokens, or generally look at the developer patterns and try to come up with simple rules to explain them).
clarity,_quality,_novelty_and_reproducibility: # Clarity & Quality
The paper was generally well-written, and the results were clear and presented well. There are grammatical mistakes throughout, but they don't significantly impede understanding. There are also several small mistakes (e.g. "which are those introduced in Section 6.1" on page 8 should probably refer to 5.2 I imagine? And "Figure ??" at the bottom of page 18 should probably be corrected), but generally the paper is clear.

# Novelty
The three main points of novelty in this paper are:
- An eye-tracking dataset that differs from previous one in terms of the lengths of programs and the tasks examined (though it's not clear to me why this distinction is significant for the particular problem examined in this paper).
- Framing the examination of attention in terms of code exploration (though since no impact of this is measured and the results are not compelling, it's hard to know how significant this is).
- Follow-up attention as an extraction function to make an interaction matrix (though since this isn't benchmarked on an existing task, it's hard to tell how this performs in practice).

# Reproducibility
Since the code and dataset have been released, the paper should be reproducible, though I haven't verified this.
summary_of_the_review: This paper proposes using attention for code exploration, and examines how well attention performs at predicting how developers explore code. The paper as a whole is well-written and presents its results clearly, but based on the evidence presented the significance of the work is somewhat questionable. There is no strong evidence presented of how this could lead to a useful development tool, or that attention activations are better for code exploration than relatively simple baselines.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper uses eye tracking studies on developers and correlates the patterns with the attention from code-related deep learning models (CodeGen and Code-J). CodeGen has higher agreement with developers (median = 0.23). The paper also reports interaction patterns. While attention-agnostic schemes (such as position) do well, the proposed “follow-up attention” has the best agreement.
strength_and_weaknesses: (+) Correlating model’s attention with human attention is an interesting area of research. The eye tracking setup used in the paper is more natural compared to other studies.

(+) The paper proposes “follow-up attention” which is marginally better.

(-) The paper needs to discuss how even attention-agnostic baselines perform well.

(-) Figures 6 and 7 need to be reconciled. Raw attention in layer 1 performs poorly (Figure 6) while follow-up attention with only layer 0-1 is competitive.  The reasons need to be discussed.

(-) In Figure 6(a), it might appear higher top-3 overlap should indicate better performance. Follow-up attention has lowest score. Explain.

clarity,_quality,_novelty_and_reproducibility: The presentation quality is good. It is not clear if the experiments are repeatable in this nascent area as the experiments have lot of moving parts (task, expertise of subjects, equipment used, etc.). The same experiment in a different organization can lead to somewhat different results.

This aspect is not used to penalize the paper in any sense.

summary_of_the_review: The paper addresses an important topic of correlating human attention with model attention. The authors have done a commendable job of using non-attention based baselines and reporting the performance. A deeper interpretation of the reported results, especially the good performance of non-attention baselines, is required to benefit other researchers.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a study which compares developers' attention to input context for sense-making tasks (e.g., mental code execution, side effects detection, algorithmic complexity, deadlock detection) with that of large autoregressive pretrained language models (CodeGen, GPT-J). For this, they collect a dataset through eye-tracking experiments involving 25 developers. For the model, they encode the prompt and analyze how tokens in the prompt attend to one another, across heads and layers. They consider many different post-processing techniques for attention, including mean, max, rollout, and a newly developed technique called follow-up attention that is designed to explicitly model the temporal relationship between attention weights. With respect to visual attention at the character-level, there is a weak positive correlation between model attention weights and developers' attention. For interaction attention at the line-level (given a position in the prompt, which part of the prompt is likely to be attended to next), there is a more moderate positive correlation. They consider the task of predicting the next line that the developer is likely to look at next, and using follow-up attention yields the best performance for this.

Contributions:
- A large eye-tracking dataset corresponding to different sense-making code tasks
- Empirical results for how the attention employed by models compare to developers
- The follow-up attention processing technique which provides more signal on what line a developer is likely to look at next, which could be useful for supporting end tasks related to code navigation

strength_and_weaknesses: Strengths:
- With growing work in the space of understanding the reasoning capabilities of large language models, I find this to be a very interesting piece of work which tries to relate human reasoning with model reasoning through proxies (eye-tracking and attention).
- The idea of leveraging the temporal relationship through layers to process attention weights is quite novel, and the impact that it has on predicting the next line the developer is going to look at is impressive.
- The authors have carefully controlled for various side effects (e.g., neighboring effect, temporal connect) when running experiments and presenting for results.
- Extensive empirical analysis of attention post-processing techniques, which could be useful for future research which employs attention weights of models for different end tasks.

Weaknesses:
- The underlying motivation for this work is comparing the alignment between developer attention and model attention when solving the task. Since the eye-tracking data is collected while the developer is writing the answer to the question in the prompt, I can see how this can be considered an appropriate proxy for what the developer is looking at "when solving the task." However, my understanding is that the model's attention weights are gathered from the teacher-forced prompt tokens only, and not from when new tokens are being generated to actually solve the task. Therefore, I question whether this is in fact an appropriate proxy.
- From Figure 4, the performance of the models are quite low, with them only achieving correctness 23.7% of the time. This suggests that these models are not performing well on the sense-making tasks. Since they are not predicting correct answers, isn't it possible that their reasoning that led to these incorrect answers is also incorrect? So, does it really make sense to analyze and make claims based on these attention weights?
- In Figure 4, CodeGen and GPT-J achieve very similar performance. However, in Figure 5, their correlation with developers is very different, with GPT-J achieving very low correlation. This seems to suggest that models may attend to the input context very differently from one another and still achieve similar results. Additionally, the developers vs developers correlation is also only moderately positive, suggesting that developers may inspect the input context differently and still provide the similar answers to the sense-making questions. Therefore, it seems like there is a lot of variability in proxies like eye-tracking and attention which make it difficult to compare reasoning of models and humans.
- While authors compare to a number of attention-agnostic baselines like cpy-cat, uniform, and position in Figure 6b, I feel that a simple rule-based baseline which always just predicts the next token/next line (which mimics a left-to-right reading paradigm) is missing. 
clarity,_quality,_novelty_and_reproducibility: Clarity:
- In Figure 4, in addition to presenting aggregate results for different question types, it would have been nice to see a breakdown at the question-level to make comparisons at a more fine-grained level.
- In Section 3, it is written that some questions were sourced from the Internet and some were written from scratch by the authors. More details are needed about exactly how these questions were collected from the Internet/written by authors.
- Figure 6 is not referenced in the paper, and it is not clear whether GPT-J or CodeGen is the underlying model here.
- On Page 8, "which are those introduced in Section 6.1" seems to be a wrong reference. 

Novelty:
- There is some empirical novelty in the newly collected eye-tracking dataset, though eye-tracking datasets have been previously collected for code-related tasks.
- There is technical novelty in the follow-up attention algorithm which combines the temporal dimension with attention layers and some fine-grained analysis is done with this in Figure 7.

Reproducibility:
The authors have provided a link to the data and they use publicly available models, so it would not too difficult to replicate the results.


summary_of_the_review: Overall, the study presented in this work draws some interesting connections between how developers inspect code and the attention weights of large language models. The authors also present a novel algorithm for post-processing attention which incorporates the temporal aspect. However, I have some concerns about whether the attention weights which were extracted actually align with how a model is solving the underlying task and whether the low model performance suggests that it might not be attending to appropriate tokens. Additionally, it seems like there is a lot of variability in eye-tracking data and attention weights. So, I find some of the comparisons and claims made in this paper to be not as convincing.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a study of the attention matrices in large language models for code and developed a few different ways of visualizing the attention and how that correlates with human developer’s eye tracking data.  This line of work is potentially interesting and the new data could be useful for future studies, but at the current state this paper didn’t excite the reviewers enough to pass the bar for acceptance.  Common issues include concerns over the usefulness and significance of the proposed approach, as prior work has studied very similar problems for language models, and for a future tool for developers, there could be simpler alternatives that are also viable.
justification_for_why_not_higher_score: The proposed method and the presented results are not surprising, similar work exist in the past.

Usefulness of the approach is not significantly demonstrated.
justification_for_why_not_lower_score: Can't be lower.
Decision: Reject