Analyzing the Decision Criteria of Software
Developers Based on Prospect Theory
Kanako Kina, Masateru Tsunoda

Hideaki Hata

Department of Informatics
Kindai University
Higashiosaka, Japantsunoda@info.kindai.ac.jp

Graduate School of Information Science
Nara Institute of Science and Technology
Ikoma, Japan
hata@is.naist.jp

Haruaki Tamada

Hiroshi Igaki

Faculty of Computer Science and Engineering
Kyoto Sangyo University
Kyoto, Japan
tamada@cse.kyoto-su.ac.jp

Faculty of Information Science and Technology
Osaka Institute of Technology
Hirakata, Japanhiroshi.igaki@oit.ac.jp

Abstract—To enhance the quality of software, many softwaredevelopment support tools and software development
methodologies have been proposed. However, not all proposed
tools and methodologies are widely used in software development.
We assume that the evaluation of tools and methodologies bydevelopers is different from the evaluation by researchers, andthat this is one of the reasons why the tools and methodologies
are not widely used. We analyzed the decision criteria of softwaredevelopers as applied to the tools and methodologies, to clarify
whether the difference exists or not. In behavioral economics,
there are theories which assume people have biases, and they do
not always act reasonably. In the experiment, we made aquestionnaire based on behavioral economics, and collected
answers from open source software developers. The results
suggest that developers do not always act to maximize expected
profit because of the certainty effect and ambiguity aversion.
Therefore, we should reconsider the evaluation criteria of tools
such as the f-measure or AUC, which mainly focus on the
expected profit.

applying the tools and methodologies sometimes requires
increased costs, and it is not always easy to apply such tools
and methodologies to development. Also, they are not always
useful for some developers, or developers may not be familiar
with them. For example, static code analysis tools are notwidely used [2], and even UML is not used widely [13]. We
assume that the developers' evaluation of the tools andmethodologies is different from the evaluation by researchers.

Keywords—evaluation criteria; fault prediction; behavioral
economics; human factor

I.

INTRODUCTION

Recently, software development projects have becomelarger and more complex, and the deadlines for such projects
are often tight. In this situation, many software development
support tools and software development methodologies havebeen proposed to suppresss the failure of a software project(e.g., low quality of software and delay of delivery). For
example, many prediction models of fault-prone modules havebeen proposed [12]. These models predict software modules
which have faults, and support makingtesting plans. Also,
static code analysis tools have been proposed to support
finding faults [5]. These tools analyze software codes, andwarn developers of spots which may violate coding standards.
However, not all proposed tools and methodologies arewidely used in software development. This may be because

As an example, we use the performance evaluation of faultprone module prediction. The prediction of the model includes
false positives and false negatives. False positives mean themodel indicates that a module includes faults, but it does not
actually. False negatives mean the model indicates that a
module does not include faults, but it actually does.
Researchers evaluate the performance of the prediction modelby the f-measure or AUC (area under the curve) [7], which arecalculated based on the false positives and the false negatives.
However, if developers only focus on the false positives, theevaluation by developers is different from researchers. For
instance, a prediction model A is better than a model B basedon the f-measure, but the false positives of model B are better
than in model A. In this case, developers will highly evaluate
model B. However, focusing on the f-measure is morereasonable than focusing on the false positives when the cost
due to the false positives is the same as (or smaller than) the
false negatives. Therefore, the sum of the false positives andthe false negatives (i.e., the sum of the cost) is smaller when
the f-measure is better.
Some papers have suggested that the false positives affectthe usability of the static code analysis tools. Johnson et al.
[10] investigated the reasons why static code analysis tools are
not widely used. For example, tool output, supporting
teamwork, user input, and customizability are the reasons. Tool
output represents a false positive (i.e., although the tool
suggests a spot is problematic, actually it is not). Another study[3] also focused on false positives in static code analysis tools.
Similarly, developers often focus on the false positives or thefalse negatives, and not on the f-measure, when evaluating thefault-prone module prediction.

A2-1. A tool which always reduces 10 hours ofworking
time.

In this paper, we analyzed the decision criteria of softwaredevelopers as applied to the tools and methodologies, to clarify
whether a biased evaluation is observed or not. In behavioral
economics, there are theories which assume people have biases,
and do not always act reasonably. Our analysis is based onprospect theory [11]. Prospect theory is used in behavioral
economics, and explains how people make decisions. In thesoftware engineering domain, there are a few studies whichhave applied behavioral economics [8]. However, these studies
did not use prospect theory in the analysis.

A2-2. A tool which reduces 25 hours of working time witha 50% possibility, and does not reduce working witha 50% possibility.

In this analysis, we conducted a survey e-mailed to open
source software developers, and tried to clarify whether
developers have the biases illustrated in prospect theory . The
result will be useful in reconsidering the evaluation criteria ofdevelopment support tools and methodologies.
II.

PROSPECT THEORY

Our analysis is based on the prospect theory [11], which isone of the theories in behavioral economics. Behavioral
economics clarifies the behavior of people, based on
experiments and observations. Traditional economics assumes
people act rationally. In contrast, behavioral economics
assumes people have biased thinking, and they do not always
act rationally. Prospect theory explains how people makedecisions when there are options which bring profits withdifferent probabilities. The assumptions of prospect theory are
different from expected utility theory [14] in traditional
economics.
Allais paradox: Prospect theory is proposed to solve theparadoxes of expected utility theory. We explain twoparadoxes which we analyzed in our study. The first one is the
Allais paradox [1]. Simply speaking, the paradox can be
explained by the following example.
Q1.

Which lottery would you choose?

A1-1. You get $1,000 with a 100% possibility.
A1-2. You get $2,000 with a 50% possibility, and get $0
with a 50% possibility.
Many people select lottery A1-1 although the expected
profit (i.e., the multiplication of the profit and the possibility)of A1-1 is the same as A1-2. People choose lottery A1-1
because it brings profit certainly. People tend to avoid the riskof losing the profit by selecting A1-2. That is, people make adecision, considering not only the expected profit, but also thecertainty. This observation is called the certainty effect [15].
We focus on the certainty effect, and analyze whether it isobserved or not in the developers’ decision. If the certainty
effect is observed, the evaluation criteria of the development
support tools and methodologies may have to be reconsidered.
For example, in the two development support tools shown in
the following, developers select a tool from between them.

Considering the expected profit of the tools, tool A2-2 isbetter than tool A2-1. As explained in the Introduction,
researchers often evaluate the effects of the tools andmethodologies, especially the prediction models, based on the
expected profits such as in the f-measure or AUC. Based on
their criteria, however, researchers would select tool A1-1
whilemany developers, on the other hand, may select tool A2-1,
if there is a certainty effect.
Ellsberg paradox: The other paradox of expected utility
theory is the Ellsberg paradox [4]. Intuitively, the paradox canbe explained by the following example.
Q3.

Which jar would you choose, if you get $1,000 when
you draw a red ball?

A3-1. An jar which includes 50 red balls and 50 black balls.
A3-2. An jar which includes red balls and black balls. The
total number of balls is 100, and the number of redballs and black balls are unknown.
In the example, the profit is not certain, despite choosing
A3-1 or A3-2. However, the probability of A3-1 is known, andthat of A3-2 is unknown. A3-2 is called an ambiguity. Many
people choose A3-1 to avoid the ambiguity. This observation iscalled as the ambiguity aversion.
We focus on the aversion, and analyze whether it isobserved or not in the developers’ decision. If the ambiguity
aversion is observed, the promotion of the tools and themethodologies may have to be reconsidered by thedevelopers.
For example, there are two development support
methodologies shown in the following, and developers mustselect a methodology from between them.
A4-1. A methodology which reduces 2 hours of working
time with a 80% possibility, and reduces 1 hour with
a 20% possibility.
A4-2. A methodology which reduces 4 hours of working atbest. The probability is unknown.
Suppose that A4-1 is the methodology already applied bydevelopers. In contrast, A4-2 is a new methodology evaluatedby a certain development project. However, most developers
have not applied the methodology to their project before. The
effect of the new methodology is often unknown, and hence, it
is regarded as containing the ambiguity. In this case, themaximum effect of A4-2 is larger than A4-1. However, many
developers may select methodology A4-1, due to ambiguity
aversion.

TABLE I.
Question

Related to

1
2

Certainty
effect3
4
5
6

DETAIL OF THE QUESTIONNAIRE

Itema) A tool which always reduces 2 hours of working time.
b) A tool which reduces 5 hours of working time at 50% probability, and does not affect working time at 50% probability.
a) A tool which always reduces 2 hours of working time.
b) A tool which reduces 4 hours of working time at 70% probability, and increases 1 hour at 30% probability.
a) A tool which always reduces 2 hours of working time.
b) A tool which reduces 6 hours of working time at 90% probability, and increases 4 hours at 10% probability.
a) A tool which reduces 4 hours of working time at 70% probability, and increases 1 hour at 30% probability.
b) A tool which reduces 2 hours of working time at best, and does not affect working time at the worst. The probability is
unknown.

Ambiguity
aversiona) A tool which always reduces 2 hours of working time.
b) A tool which reduces 6 hours of working time at best, and increases 1 hour at the worst. The probability is unknown.
a) A tool which always reduces 2 hours of working time.
b) A tool which reduces 5 hours of working time at best, and does not affect working time at the worst. The probability is
unknown.

III.

EXPERIMENT

A. Research Questions
In the experiment, we made a questionnaire and collected
answers from open source software developers, to clarify threeresearch questions.


RQ1: Are there certainty effects among softwaredevelopers when selecting software development
support tools?
RQ2: Are there ambiguity aversions among softwaredevelopers when selecting software development
support tools?
RQ3: Is the answer chosen by a developer different
from the answer chosen as a manager?

If the answer of RQ1 is “yes,” the evaluation criteria of thetools and methodologies should be reconsidered by researchers,
especially criteria such as the f-measure and AUC. Also, if theanswer of RQ2 is “yes,” then how to promote the tools and themethodologies to developers must be reconsidered. For
example, if there is some ambiguity in applying the tools, thenthe effects of the tools are large enough to ignore the ambiguity.
In questions relating to RQ1 and RQ2, we assume that the
profit of the tools for developers is the reduction of effort(time). In questions relating to RQ3, the subjects assume that
their role is as a project manager, and the profit is an evaluationof themselves from other project’s members. This is because
the tools and the methodologies are evaluated not only bydevelopers, but also by project managers.
B. Experimental Setup
We e-mailed the survey to open source software developers.
Their addresses were collected from websites [6][9]. The
answers to the questionnaire were collected via a web form.
The number of responses was twenty, and the average numberof years of their experience in software development was 21.2

(The median was 20.0). Therefore, the subjects wereconsidered to understand software development very well.
The questionnaire consists of six questions. Details of thequestionnaire are shown in Table I. Subjects selected coding
support tool ‘a’ or ‘b,’ assuming the following.


10 hours are needed to make a module.
Making a module is not hobby programming.
Conditions such as the time needed to learn a tool are
the same for each tool.
The tools are prediction tools such as a static codeanalysis tool.

The last item was selected to emphasize that the questionnaire
does not focus on the functions of the tools, but on theprediction performance of the tools.
The questions 1 to 3 relate to the certainty effect, i.e., RQ1.
In the questions, tool ‘a’ certainly reduces working time.
Although the effect of tool ‘b’ is not certain, the expected profit(i.e., reduced time multiplied by the probability) is higher thantool ‘a.’ To analyze the decision criteria of developers in detail,we specified the characteristics of tool ‘b’ as follows:

Question one: There is no risk if the tool does not work
well.
Question two: There is a small risk if the tool does notwork well.
Question three: There is a high risk if the tool does notwork well although the probability is low.

Questions 4-6 relate to ambiguity aversion, i.e., RQ2. In thequestions, the probability of the effect of tool ‘a’ is known, butthat of the tool ‘b’ is unknown. We specified the characteristics
of tool ‘a’ and ‘b’ as follows:

Question 4:Tool ‘a’ - There is a small risk if the tool does notwork well.
Tool ‘b’ - There is no risk if the tool does not work
well.
Question 5:
Tool ‘a’ - There is no risk if the tool does not work
well.
Tool ‘b’ - Developers receive a high benefit if the tool
works well, but there is small risk if it does not work
well.
Question 6:
Tool ‘a’ - There is no risk if the tool does not work
well.
Tool ‘b’ - Developers receive a high benefit if the tool
works well, and there is no risk if it does not work well.

Also, subjects selected coding support tool ‘a’ or ‘b’ fromquestions 1-6, assuming the following. The questions relate to
RQ3.



You are a manager, and give a tool to a programmer.
If his/her work time is more than 10 hours, theevaluation of your team is down.

Therefore, the number of answers by each subject is 12. That is,
we asked the same set of questions twice. One set is for therole of the developer, and the other one is for the role of the
manager. The former set is denoted by Q1-1 to Q1-6, and thelatter set is denoted by Q2-1 to Q2-6.
C. Results
The ratio of answers (a) and (b) is shown in Table II. In thefollowing, we explain the analysis results on each researchquestion.
The result related to RQ1: On Q1-1 to Q1-3, the ratio ofanswer (a) was higher than answer (b). As explained before,
the expected profit (i.e., the reduced time multiplied by theprobability) of answer (b) is higher than that of answer (a).
Therefore, the result means developers do not always act tomaximize the expected payoff, and the certainty effect exists
when developers select software development support tools.
That is, the answer to the RQ1 is “yes.”
In Q1-1, many developers did not select the tool with arelatively high expected profit and no risk, if it did not work
well. Also, in Q1-3, the developers did not select the tool witha high expected profit where the probability of the risk was low.
Consequently, we must reconsider the evaluation of the tools,
if the tools have such characteristics.
The result related to RQ2: On Q1-4 and Q1-5, the ratio ofanswer (a) was higher than answer (b), but on Q1-6, the ratioof answer (a) was lower than answer (b). On Q1-4 especially,
tool ‘a’ was selected by many developers, although tool ‘a’ hasrisk and tool ‘b’ has no risk. Thus, the result suggests that the
ambiguity aversion exists when developers select softwaredevelopment support tools.

TABLE II.

THE RATE OF EACH ANSWER ON THE QUESTIONNAIRE
Questions

Answer (a)Answer (b)

Q1-1

65%

35%

Q1-2

70%

30%

Q1-3

65%

35%

Q1-4

55%

45%

Q1-5

65%

35%

Q1-6

45%

55%

Q2-1

60%

40%

Q2-2

70%

30%

Q2-3

70%

30%

Q2-4

50%

50%

Q2-5

60%

40%

Q2-6

45%

55%

In contrast, tool ‘b’ was selected although the probability ofthe effect is unknown on Q1-6. This means that when the toolis very effective and has no risk, it will be accepted bydevelopers, even if the probability of the effect is unknown.
Note that although the effect of tool ‘b’ on Q1-5 is similar to
the effect on tool ‘b’ on Q1-6, many developers did not selecttool ‘b’ on Q1-5. This is because tool ‘b’ on Q1-5 has a small
risk and hence, developers avoided selecting it. Consequently,
the answer to RQ2 is “yes, except for when the tool is very
effective and has no risk.”
The result related to RQ3: Overall, the ratio of answers(a) and (b) on Q2-1 to Q2-6 was similar to Q1-1 to Q1-6.
Although the ratio of answers (a) and (b) was same on Q2-4,
the ratio of answer (a) was higher than that of answer (b) on
Q2-5 ( tool ‘b’ on Q2-4 has no risk, and tool ‘b’ on Q2-5 has asmall risk). Hence, the ambiguity aversion still exists when atool has a small risk. Based on this result, we concluded thatthe answer to RQ3 is “No.” That is, the decision criteria ofproject managers is almost the same as the developers.
IV.

CONCLUSION

This paper analyzed the decision criteria of softwaredevelopers to software development support tools andmethodologies in order to reconsider the evaluation criteria. In
the survey, we analyzed whether the certainty effect and theambiguity aversion exist or not on the developers’ decisions.
Based on the analysis results, we suggest the following:

Developers do not always act to maximize expected
profit because of the certainty effect. Therefore, weshould reconsider the evaluation criteria of tools suchas the f-measure or AUC, which mainly focus on the
expected profit.
Developers avoid selecting tools if the probability ofthe effect of the tools is unknown, and the tools havesome risks. This result suggests that it is not easy toapply new tools to actual software development
projects since the probability of the effect is generally

unknown before the application. To promote
development support tools, we have to suppress therisk of the tools.
The decision tendency of the project managers wasalmost the same as the developers. Therefore, the roleof the person making the decision may not be very
important when promoting the tools andmethodologies.

We believe that the main contribution of our paper is its
quantitative clarification of the decisions of the softwaredevelopers described above.
ACKNOWLEDGMENTS
This research was partially supported by the Japan Ministryof Education, Science, Sports, and Culture [Grant-in-Aid forchallenging Exploratory Research (No. 26540029), Grant-inAid for Scientific Research (C) (No. 25330090), and Grant-inAid for Young Scientists (B) (No. 24700030)].
REFERENCES
[1]

[2]

[3]

M. Allais, “Rational man's behavior in the presence of risk: critique ofthe postulates and axioms of the American school,” Econometrica,
vol.21, pp.503-546, 1953.
N. Ayewah, D. Hovemeyer, J. Morgenthaler, J. Penix, and W. Pugh,"Using Static Analysis to Find Bugs," IEEE Software, vol.25, no.5,
pp.22-29, 2008.
A. Bessey, K. Block, B. Chelf, A. Chou, B. Fulton, S. Hallem, C. HenriGros, A. Kamsky, S. McPeak, and D. Engler, “A few billion lines ofcode later: using static analysis to find bugs in the real world,”
Communications of the ACM, vol.53, no.2, pp.66-75, 2010.
[4] D. Ellsberg, “Risk, Ambiguity, and the Savage Axioms,” Quarterly
Journal of Economics, vol.75, no.4, pp.643-669, 1961.
[5] FindBugs, http://findbugs.sourceforge.net.
[6] The "Fossies" Software Archive, http://fossies.org/.
[7] T. Hall, S. Beecham, D Bowes, D. Gray, and S. Counsell, “A Systematic
Literature Review on Fault Prediction Performance in Software
Engineering,” IEEE Transactions on Software Engineering, vol.38, no.6,
pp.1276-1304, 2012.
[8] R. Hofman, “Behavioral economics in software quality engineering,”
Empirical Software Engineering, vol.16, no.2, pp.278-293, 2011.
[9] J. Howison, M. Conklin, and K. Crowston, “FLOSSmole: Acollaborative repository for FLOSS research data and analyses,”
International Journal of Information Technology and Web Engineering,vol.1, no.3, pp.17–26, 2006.
[10] B. Johnson, Y. Song, E. Murphy-Hill, and R. Bowdidge, “Why don't
software developers use static analysis tools to find bugs?” In Proc. of
the International Conference on Software Engineering (ICSE), pp.672681, 2013.
[11] D. Kahneman and A. Tversky, “Prospect Theory: An Analysis ofDecision under Risk,” Econometrica, vol.47, no.2, pp.263-91,1979.
[12] O. Mizuno and T. Kikuno, “Training on errors experiment to detect
fault-prone software modules by spam filter,” In Proc. of the joint
meeting of the European software engineering conference and thesymposium on The foundations of software engineering (ESEC-FSE),
pp.405-414, 2007.
[13] M. Petre, “UML in practice,” In Proc. of the International Conference
on Software Engineering (ICSE), pp.722-731, 2013.
[14] P. Schoemaker, Experiments on Decisions under Risk: The Expected
Utility Hypothesis, Springer Netherlands, 1980.
[15] A. Tversky and D. Kahneman, “Rational Choice and the Framing ofDecisions,” Journal of Business, vol.59, no.4, pp.S251-S278, 1986.