Quantifying Programmers’ Mental Workload during
Program Comprehension Based on Cerebral Blood Flow
Measurement: A Controlled Experiment
Takao Nakagawa

Yasutaka Kamei

Hidetake Uwano

Nara Institute of Science andTechnology
Nara, Japan

Kyushu University
Fukuoka, Japan

Nara National College ofTechnology
Nara, Japan

Akito Monden

kamei@ait.kyushuu.ac.jp
Kenichi Matsumoto

Nara Institute of Science andTechnology
Nara, Japan

Nara Institute of Science andTechnology
Nara, Japanakito-m@is.naist.jp

matumoto@is.naist.jp

takao-n@is.naist.jp

ABSTRACT
Program comprehension is a fundamental activity in software development that cannot be easily measured, as it isperformed inside the human brain. Using a wearable Near
Infra-red Spectroscopy (NIRS) device to measure cerebral
blood flow, this paper tries to answer the question: Can
the measurement of brain blood-flow quantify programmers’
mental workload during program comprehension activities?
We performed a controlled experiment with 10 subjects; 8 ofthem showed high cerebral blood flow while understanding
strongly obfuscated programs (requiring high mental workload). This suggests the possibility of using NIRS to measure
the mental workload of a person during software development activities.

Categories and Subject Descriptors
D.2.5 [Software Engineering]: Testing and Debugging;
D.2.8 [Software Engineering]: Metrics

General Terms
Measurement

Keywords
Program comprehension, mental workload, cerebral bloodflow measurement

1. INTRODUCTION
Program comprehension is a fundamental activity required
in today’s software development processes such as coding,code review, debugging, code reuse and maintenance. Its

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.
ICSE ’14, May 31 âĂŞ June 7, 2014, Hyderabad, India
Copyright 2014 ACM 978-1-4503-2768-8/14/05 ...$15.00.

uwano@info.nara-k.ac.jp
Daniel M. German
University of Victoria
BC, Canada

dmg@uvic.ca

measurement is difficult as it is a mental (cognitive) process
performed inside the human brain.
To measure such mental activities, recent neuroscienceand cognitive science studies try to directly measure brain
activity using sensors such as EEG, fMRI and NIRS [1].
Also in the software engineering domain, Siegmund et al.
[6] pointed out (at the FSE2012 New Idea Track) the necessity of analysis of brain activities in program comprehension.
They proposed an experiment design using fMRI (functionalmagnetic resonance imaging) measurement; however, no result has been reported so far, and research progress in thisarea is strongly demanded.
In this paper, we focus on the measurement of programmers’ mental workload during program comprehension toanswer the question: Can brain measurement quantify programmers’ metal workload in program comprehension? If themeasurement could identify programmer’s very high workload, which may imply the work is beyond his/her capacity,
timely help by an expert or a manager needs to be considered.
This paper presents an experiment design using a wearable NIRS (Near Infra-red Spectroscopy) to observe the cerebral blood flow of the prefrontal cortex (PFC), which has
been considered to govern planning of complex cognitive behaviour and decision making [7]; therefore, we believe PFC
activity is vital in program comprehension. In the experiment, we asked each subject to perform two tasks: 1) non
obfuscated C programs, and 2) strongly obfuscated C programs that should require higher mental workload of PFC.
As a result of a controlled experiment with 10 graduate
students in computer science, 8 students showed higher cerebral blood flow during reading of obfuscated versions. Thissuggests the possibility of measuring mental workload in program comprehension using NIRS, while we also came up
with several improvements needed in future experiments toclarify the feasibility and limitation of our approach.

2.

RELATED WORK

To understand the human aspect of program comprehension (such as understanding level, developers’ behaviour,Figure 1: The sample of the program text and preconditions given to subjects
comprehension strategy), researchers have used indirect measurement such as interview, questionnaire or ‘think-aloud’
protocol (let subjects speak their content of thinking duringexperiment [2]).
Parnin [5] analysed both short-term and long-term memory retention of developers who are working in parallel programming tasks from the viewpoint of cognitive neuroscience.
Also, Nakamura et al. [4] focused on the remembering, recalling and forgetting of variables in source code to develop amodel of program comprehension. Their experiment showedthat the time required to complete a comprehension task
well matched the difficulty of recalling a variable.
Siegmund et al. [6] applied a neuroscientific approachfor the program comprehension process and proposed anexperiment plan for identifying cortical regions related tothe program comprehension in the FSE’12. They pointed
out the necessity of analysis of brain activities to answer
the question such as What distinguishes good programmers
from bad programmers? or What makes a good programmer?
However, they only mentioned about their interim report ofprogress of the experiment and its design, thus, they havenot published results yet. Siegmund et al. predict that prefrontal cortex (related to the memory operation or complex
intellectual activity) will be activated when developers tryto understand the program.
These studies suggest that cognitive process related tothe human memory exist during program comprehension.
However, there are no experimental results about brain activation during program comprehension, or programming.
Thus, little is known about how actually brain works during
program comprehension tasks.

3. EXPERIMENTTo prepare two difficulty-level programs for each algorithm, we use an obfuscation technique to make a ‘hard’version program from an ‘easy’ (non-obfuscated) version.
We used a loop obfuscation so that loop counters and sentinel values are updated frequently and irregularly withoutchanging the functionality of a program [3]. Figure 1 showsa non-obfuscated program that seeks the minimum value inan array.
Two different level/functionality tasks are assigned to 1
subject. To reduce learning effect, half of the subjects perform the easy task first, and the others perform the hard one
first. All subjects perform an exercise task before the main
experimental task. The exercise task has two complexity
levels similar to the main experimental task.

3.3

Task

To standardize the strategy of program comprehensionamong all subjects, they read and simulate the executionof a program using mental simulation strategy (also known
as hand simulation). It is one of the bottom-up program
comprehension strategy to simulate the program’s execution process (e.g. control flow and variable assignment).
”
To properly trace the program, subjects have to remember
the current position of loop-flow and variables name/value
in their short-term memory during mental simulation.
During the mental simulation, when the subjects reach toa checkpoint marked in the program like (1) of Line 4 and(2) of Line 8 in Figure 1, they write down the value of eachvariable at the checkpoint to an answer sheet. After writing
down these values, they raise their hand. An experimenter(one of the authors) checks the values on their answer sheet
and tell the subjects whether or not the answer is correct.
If their answer is correct, the subjects continue to perform the comprehension from the current checkpoint to the3.1 Subjects
Ten students of Nara Institute of Science and Technology
participated in the experiment as subjects. All subjects are
male, 22-26 years old, and have experience using C-language
for at least 3 years.

3.2 Programs and Assignment
Six programs (three algorithms and two difficulty levels)of 17-32 lines of code, all written in C language, are used. 3algorithms are searching a keyword, calculating total values,
and seeking the maximum value in an array. One of the algorithms is used in an exercise task before a main experimenttask.

Figure 2: WOT-200(Hitachi medical Co.)

1.00

easy

Normalized oxy-Hb

hard
0.75

0.50

0.250.00

ABCDEFGHIJSubjects
Figure 3: Distribution of normalized oxy-Hb

3.4 Equipment and environment
We use the NIRS (Wearable Hikari Topography WOT200, made by HITACHI MEDICO). Figure 2 shows the appearance of the device.
NIRS assumes that higher brain activity requires more
oxygen to be transported by the blood flow. Therefore,
to quantify the brain activity, NIRS measures the amountof oxygenated haemoglobin (oxy-Hb) in the cerebral bloodflow.
We consider this device suitable for measurement of program comprehension under the condition similar to the realenvironment. Because it is lightweight, can be easily set on
the subject’s head, and does not keep subjects’ body in a
fixed position during an experiment in contrast to the fMRI,
MEG and PET that has a finer and wider spatial resolution
than NIRS.
Subjects sit down during the experiment. Experiments
are performed in a quiet room where only the subject andthe experimenter are.
To avoid the noise in the measurements, an experimenter(one of the authors) asked the subjects not to lower and raise
their head. The subjects adjust the position and the heightof the chair before beginning of the experiment. Program
text and an answer sheet are put in front of the subjects.

3.5 Metrics
Since the amount of oxy-Hb can be measured only as arelative value from the beginning of the measurement, weused a normalized value based on the following equation:
N ormalized oxyHb =oxyHb − min(s)max(s) − min(s)where max(s) and min(s) are the maximum and minimum
value through all tasks of each subject s. The range of thenormalized oxy-Hb is [0,1]. We measured the normalized
oxy-Hb every 200ms.

Subject A
Normalized oxy-Hb

next one. If not, they go back to a previous checkpoint
and restart the comprehension task. When they correctly
answer the last checkpoint marked at the return statementin the program (like (3) of Line 12 in Figure 1), they havecompleted the task.

Time [s]Figure 4: Chronological changes of brain activation

4.

RESULTS AND DISCUSSION

Figure 3 shows the distribution of normalized oxy-Hb ofeach subject/task. Labels A to J represents each subject(the left box shows the distribution of ‘easy’, and right oneshows the ‘hard’). The y-axis corresponds to the normalized
oxy-Hb (i.e., how much the brain works actively).
We found that the normalized oxy-Hb of hard tasks islarger than easy tasks among all subjects except E and G.
This result suggests that the complexity of the program induces the activation of the prefrontal cortex, thus, we consider that mental workload could be quantified using cerebral blood flow measurement.
Another finding is that the variance of normalized oxy-Hbof hard tasks is larger than easy tasks among all subjects
except E. This suggests that even in a hard task, mental
workload is often very low. Figure 4 shows the time-coursechanges of subject A’s data during performing ‘hard’ task,
which indicates the amount of oxy-Hb continues to change
throughout the experiment. Therefore, additional measurement, such as PC operation history and eye-gaze tracking,is needed in future study to observe subjects’ external behaviours.
The result also indicates that some subjects (E and G inour case) may show the counter-trend tendency to others.
This could happen by several reasons, e.g., 1) measurement

0.800.75
0.60

Normalized oxy-Hb

Normalized oxy-Hb

1.00

0.50

0.25

0.40

0.20

0.00

EGSubjects
0.00

AFigure 5: Normalized oxy-Hb in an exercise taskerror (the sensor may not fit well to some subjects’ forehead), 2) subject’s skill (high skill subjects may not feel any
difficulty in hard tasks), 3) subject’s natural property (somesubjects may not require high oxy-Hb in mental simulation),
etc.
For further analysis, Figure 5 shows the distribution of
normalized oxy-Hb of E and G in the exercise tasks. Interestingly, subject E showed the same counter-trend reaction(oxy-Hb during ‘easy’ higher than ‘hard’) in the exercise task
(Figure 5, left graph). Further experiments are required infuture to analyse why and how often this would happen. At
least, an interview to subjects after the experiment is needed
to clarify if all subjects felt the ‘hard’ task more difficult thanthe ‘easy’ task.
Figure 6 shows the result of time-series analysis. We
equally divided the task completion time into three parts,
the early stage, the middle stage, and the final stage. Eachbar in Figure 6 shows the median of the normalized oxy-Hbof each stage.
Figure 6 indicates that normalized oxy-Hb is higher inthe middle stage than the early stage (8 out of 10 subjects)
and higher in the middle stage than the final stage (7 outof 10 subjects). This may happened because most wrong
answers occurred in the middle stage, which implies thathigh workload is required to correct the answers. This result
suggests the possibility to quantify the time-course changeof mental workload using NIRS.
Threats to validity: To generalize our result, we need toconsider top-down comprehension strategy and its difficulty
because our experiment lets subjects use only bottom-up
strategy (i.e., mental simulation). However, we believe that
our method can be applied to another strategy if difficultylevels of the program are well defined, because PFC has astrong relation with complicated intellectual activity and the
top-down strategy is as complicated as bottom-up strategy.

5. CONCLUSIONSIn this paper, we aimed to investigate whether or not developers’ workload can be quantified using cerebral bloodflow measurement of the prefrontal cortex. In our experiment, we measured the amount of oxygenated haemoglobin(oxy-Hb) during comprehension of two different types of programs, ‘hard’ (high complexity) and ‘easy’ (low complexity).
The result showed the tendency that oxy-Hb becomes higher
in ‘hard’ programs than ‘easy’ programs, which suggests theBCDEFGHIJ

Subjects

Figure 6: Chronological changes of brain activation
possibility of measuring mental workload by oxy-Hb.
In the future, we are planning to compare program comprehension tasks with other cognitive tasks such as readinga natural language text or doing a mathematical calculation.
Also, we are planning to conduct very-easy/very-hard tasks
as baseline tasks, e.g., doing (nothing) with eye-closed asvery-easy, and doing extremely-difficult mathematical calculation as very-hard. We also plan to use other measurement sources, e.g., history of PC operations, eye-tracking
and interview to subjects.

6.

REFERENCES[1] R. Cabeza and L. Nyberg. Imaging cognition II: An
empirical review of 275 PET and fMRI studies. Journalof cognitive neuroscience, 12(1):1–47, 2000.
[2] K. A. Ericsson and H. A. Simon. Verbal reports asdata. Psychological review, 87(3):215, 1980.
[3] A. Monden, Y. Takada, and K. Torii. Method forscrambling programs containing loops. IEICE Trans.
on Information and Systems, 80(7):644–652, 1997. (inJapanese).
[4] M. Nakamura, A. Monden, T. Itoh, K. Matsumoto,Y. Kanzaki, and H. Satoh. Queue-based cost evaluationof mental simulation process in programcomprehension. In Proc. of 9th IEEE International
Software Metrics Symposium (METRICS’03), pages
351–360, 2003.
[5] C. Parnin. A cognitive neuroscience perspective onmemory for programming tasks. In Proc. of 22nd
Annual Meeting of the Psychology of Programming
Interest Group (PPIG), 2010.
[6] J. Siegmund, A. Brechmann, S. Apel, C. Kästner,J. Liebig, T. Leich, and G. Saake. Toward measuring
program comprehension with functional magnetic
resonance imaging. In Proc. of the ACM SIGSOFT
20th International Symposium on the Foundations ofSoftware Engineering, (FSE ’12), pages 24:1–24:4, 2012.
[7] Y. Yang and A. Raine. Prefrontal structural andfunctional brain imaging findings in antisocial, violent,and psychopathic individuals: A meta-analysis.
Psychiatry Research: Neuroimaging, 174(2):81 – 88,
2009.