/* ====================================================================================TOWARDS GENERATION OF VISUALATTENTION MAP FOR SOURCE CODE==================================================================================== */Authors = { Takeshi D. Itoh1, Takatomi Kubo1*, Kiyoka Ikeda1,2, Yuki Maruno2,11
1
1
Yoshiharu Ikutani , Hideaki Hata , Kenichi Matsumoto , Kazushi Ikeda };Affiliation_1 = Division of Information Science, Nara Institute of Science and Technology;
Affiliation_2 = Department of Contemporary Society, Kyoto Women's University;
Correspondence = takatomi-k@is.naist.jp;

Project Page

Objective

INTRODUCTION

Our objective is to develop a gaze analysis framework to compare programmers’ gaze attention toML attention mechanism.

Attention in Program Comprehension
Human programmers focus their gaze on important components in the presented source code [1].
Gaze focuses get sparser with higher expertise.
Such attentional gaze behavior might help expert
programmers to capture context and composition
in source code much eﬃciently.

Expected Resultsa) If human gaze and ML attention are consistent:→ ML model might explain human attention.
b) If they are inconsistent:
If humans outperform ML, programmers’ attention
strategy can be used to improve ML attention [3].
If ML outperforms humans, ML attention can beused to guide programmers’ gaze [4].

Towards modeling the mechanism of human programmers’ attention, we focus on testing biological plausibility of ML attention mechanisms.

METHODSCompareattention

Human attention

Source codeclassiﬁcation taks

Original source code

ML attention

Comparing biological attention to artiﬁcial attention

PRELIMINARY RESULTS & DISCUSSION

Visual Attention Map

convert code to AST

We conducted ROC analysis between the gaze distributions and the VAM.
The expert showed a higher AUC value than the novice.

softmaxcode
vectorcontext
vectorprediction

FC layercode2vec architecture (from [3]; modiﬁed)
path attention

extract path contexts
AST

Original code

Preliminary Results

attentionweights

Code2vec [3] learns a distributed representation of program functions to discriminate their names.
Attention mechanism quantiﬁes components’ importance for discrimination.
The visual attention map is computedby mapping code2vec’s path attention
into the original source code image.
Attention prediction
by code2vec

Author Page

Path context

Attention value

MethodDecl

int f (int arg) {if (arg > 0) return arg;
else

intf

Parameter

IfStmtreturn 0;
int}GreaterThanargarg

Attention map
generation

Original code0,GreaterThan↑IfStmt↓ReturnStmt,0ReturnStmt

ReturnStmt

arg

00

AST

spatial attention ← node attention

Path context

0.19

Attention value

node attention ← path attention

Subjects (1 expert + 1 novice) engaged
an algorithm classiﬁcation task.
Gaze behavior was recorded with Tobii
Pro TX 300.
Recorded gaze distribution was compared to the visual attention map toevaluate consistency.

ﬁxation+source code
public class Main {public static void main(String[] args) {Scanner in = new Scanner(System.in);
String word = in.next();
int count = 0;answerword = word.toLowerCase();while (true) {String str = in.next();

time

Expert gaze distribution (AUC = 0.87)

Novice gaze distribution (AUC = 0.77)

Discussion
The expert’s gaze was more consistent with the VAM than the novice.
The VAM and the expert showed much sparser attention than the novice.
There might be a common importance metric, shared among attentional
ML models and expert human programmers.

Procedure for generating visual attention map

Preliminary Gaze Experiment

Visual attention map (code2vec)

String(1)Math(2)

Sort(3)[]Experimental procedure

Search(4)We will continue experimenting to quantify the relationship between programmers’ attention and ML attention mechanisms, with more humansubjects, varying source code snippets, and performance evaluation.
The current result also supported that our VAM method is feasible.
We will test the biological plausibility of other ML models and seek forcommon attention representation for program comprehension.

PROSPECTS: PROGRAM COMPREHENSION IN BIOLOGICAL & ARTIFICIAL SYSTEMS
Context and Compositionality in Gaze
Gaze focuses, or ﬁxation points, move sequentially on presented source code.
[Hypothesis]
Each ﬁxation grabs local spatial composition.
A sequence of ﬁxations forms a global context.
Experts can form an accurate global context withfewer ﬁxation points, and can determine where tolook at in the next ﬁxation.

Composition-Aware ML Model

Neural Representation of Expertise

Code2vec decomposes programs into a set of“paths,” and possibly it cannot capture the composition in source code.
We are now developing a new ML algorithm forprogram comprehension, that is designed withconsidering the hierarchical compositionality insource code.
By considering compositionality, we expect that...
a) Model performance on program comprehension task will be improved.
b) The VAM will be much consistent with humanexperts’ gaze behavior.
c) The model can be used in a model-based fMRI
decoding study to explain neural representationsin the programmers’ brain.

Our fMRI study (in prep.) conﬁrmed that semantic
categories of source code could be decoded fromprogrammers’ brain.
Several brain regions contribute to the better categorization performance of expert programmers.
These brain regions might process context andcomposition of source code.

FIGURE NOT AVAILABLE[1] Uwano et al., ACM Symposium on Eye Tracking Research & Applications, 2006.

This work was originally published in APSIPA ASC 2019 (18-21 Nov, Lanzhou, China).

[2] Crosby et al., 14th Workshop of the Psychology of Programming Interest Group, 2002.

We would like to thank Dr. Nishanth Koganti and Ms. Louise Bautista for their valuable[3] Alon et al., ACM on Programming Languages, 2019.

advices and comments. This work was supported by JSPS KAKENHI Grant Numbers[4] Ikutani et al., 6th International Workshop on Eye Movements in Programming, 2019.

JP19J20669, JP18J22957, JP18K18108, JP16H06569, and JP16H05857.