Does Return Null Matter?
Shuhei Kimura, Keisuke Hotta, Yoshiki Higo, Hiroshi Igaki, and Shinji Kusumoto
Graduate School of Information Science and Technology, Osaka University
1-5, Yamadaoka, Suita, Osaka 565-0871, Japan{s-kimura, k-hotta, higo, igaki, kusumoto}@ist.osaka-u.ac.jp
Abstract—Developers often use null references for the returned
values of methods (return null) in object-oriented languages.
Although developers often use return null to indicate that a
program does not satisfy some necessary conditions, it is generally
felt that a method returning null is costly to maintain. One of
the reasons for is that when a method receives a value returned
from a method invocation whose code includes return null, it
is necessary to check whether the returned value is null or not(null check). As developers often forget to write null checks, null
dereferences occur frequently. However, it has not been clarified
to what degree return null affects software maintenance duringsoftware evolution. This paper shows the influences of return nullby investigating return null and null check in the evolution ofsource code. Experiments conducted on 14 open source projects
showed that developers modify return null more frequently thanreturn statements that do not include null. This result indicatesthat return null has a negative effect on software maintenance.
It was also found that the size and the development phases ofprojects have no effect on the frequency of modifications on
return null and null check. In addition, we found that all the
projects in this experiment had from one to four null checks per
100 lines.
Index Terms—software evolution; return null; software maintenance

I. I NTRODUCTION
A null reference is a common mechanism in object-oriented
languages, such as Java and C++. Developers use null invarious roles, for instance, as an initializer or sentinel. In particular, it is general practice that a method returns null values
when some necessary conditions are not satisfied (hereafter werefer to return statements whose operand is null as “returnnull”).
Developers often use a return null rather than error constants
or exceptions for the following reasons:• As null is a unique value, we know some errors haveoccurred, even if there are no error messages.
• Returning null is easier to write than handling exceptions.
Although return null supports developers for the above
reasons, it is generally felt that return null is one of the factorsthat increase software maintenance efforts [1].
A null dereference, a bug by which a program dereferences
a null value, is the main bug caused by a return null. To
prevent a null dereference from occurring, returned values
from a method that includes a return null need to be checked(hereafter we refer to such checking as “null check”). Accordingly, when a developer adds return null to a method, anull check needs to be added for all of its invocations. If a

1
2
3

final RevCommit base = walk.next();
if(base == null)
return null;Fig. 1: Example of passing return nulldeveloper forgets to add null check, the program will try todereference null, and NullPointerException will be thrown. In
addition, null contains no information about an error; whereas
an exception contains information on errors that have occurred.
For this reason, it is very difficult for caller methods thathave received null to know what errors have occurred. This
characteristic sometimes hides the root causes of errors thathave occurred, because a null dereference will occur at places
remote from the point of origin of null if a program has
methods that include return null in a null check (e.g., Figure 1).
For these reasons, return null is regarded as a factor that
increases the cost of software maintenance.
However, it has not been clarified to what extent a returnnull affects software maintenance during software evolution.
Therefore, in this research, we conducted an experimental
study on 14 open source projects to ascertain whether, and towhat extent, the presence of return null and null check affectsoftware maintenance.
We found that the presence of return null and null checkincreases the cost of software maintenance, and that the sourcecode modifications that are related to them have the following
characteristics:
Developers modified return nulls more frequently thanreturn statements that do not include null.
• The size and the development phases of projects did notaffect the frequency of modifications on return nulls andnull checks.
• The density of null checks in source code was between
one and four in 100 lines, in any project. If the densityof null checks in a project exceeds 0.04, the project ispresumed to have a problem. None of the projects had
a density less than 0.01, and none of the projects had
special measures in place for return nulls and null checks.
• There were modifications that replaced return null withexception handling or an erased null check. These phenomena indicate that programmers considered that thepresence of return null and null check were harmful tosoftware evolution.
•

The contributions of this paper include the following:This paper revealed that return null affect software maintenance during software evolution.
• The result indicated that developers should avoid writing
return null without given a great deal of thought to doingso.
• Measurement of return null and null check showed that
return null and null check are costly to maintain throughout the entire development periods.
• The result showed the average density of null checks.
This value can be used as a criterion (e.g., when thedensity of null check in a project exceeds the average,
developers should apply refactoring to return null andnull check).
A developer could replace return null with other mechanisms, for example, exception handling, a proper object suchas an empty array, or a NullObject pattern. However, this paper
presents whether return null matter. Providing how to replace
it is our future work.
The rest of this paper is organized as follows. Section II
presents previous work by way of introducing our research.
Section III defines our research questions and describes theexperimental design. Section IV shows the result of theexperiment and answers the research questions. In Section V,
we discuss our answers to the research questions. Section VI
discusses threats posed to validity. Section VII provides various approaches to resolving problems related to null, andSection VIII concludes this paper.
•

1

2
3
4
5
6
7
8
9
10
11

public Ref call() throws GitAPIException,
RefAlreadyExistsException, RefNotFoundException,
InvalidRefNameException, CheckoutConflictException{
checkCallable();processOptions();try {if (checkoutAllPaths || !paths.isEmpty()) {checkoutPaths();
status = new CheckoutResult(Status.OK, paths);
setCallable(false);
return null;}...

(a) Method including return null
1
2
3
4
5
6
7
8

...
try {
String oldBranch = db.getBranch();Ref ref = command.call();if (Repository.shortenRefName(ref.getName()).equals(
oldBranch)) {outw.println(MessageFormat.format(
CLIText.get().alreadyOnBranch,...

(b) Invokes the method shown in (a)
1
2
3
4
5
6
7
8
9
10

...
try {
String oldBranch = db.getBranch();Ref ref = command.call();+ if (ref == null)+
return;if (Repository.shortenRefName(ref.getName()).equals(
oldBranch)) {outw.println(MessageFormat.format(
CLIText.get().alreadyOnBranch,...

II. BACKGROUND
Null dereference often occurs in programs written in objectoriented programming languages. For this reason, many researchers have proposed techniques to detect null dereference[2]–[7], and many tools have been released [8], such as
FindBugs [9], SALSA [6], JLint, and ESC/Java [10]. However,previous studies did not mention whether, and to what extent,
the presence of return null and null check affected software
maintenance during software evolution.
A factor of null dereference is that a program unintentionally refers null when programmers forget to write null check,despite the callee method possibly returning null. Figure 2(b)is a code fragment that appears in JGit. command.call(),in the
fourth line, invokes the method shown in Figure 2(a). Thismethod may return null, as written in the ninth line. The
variable named “ref,” defined in the fourth line in Figure 2(b),may be assigned null by calling command.call(). As a result,ref.getName(), in the fifth line, possibly causes null dereference. Figure 2(c) is a bug fix for this null dereference. Prefix“+” in a line means that the line was added in this commit. Thisfix added null check in order to avoid the occurrences of null
dereference. The commit message, “Do not fail when checking
out HEAD,” shows the purpose of this commit is to fix the null
dereference. As mentioned above, return null, null check, andnull dereference are closely related to each other. Thus, we
can show how null dereference affects software maintenanceby investigating the effect of return null and null check on
software maintenance.

(c) After adding null check
Fig. 2: Example code fragments of return null, null dereference, and nullcheck

To reveal the effect of return null, the experiment in thispaper covers not only return null but also null check. Error
constants, such as -1, are used as having the same meaning as
return null. However, mainly in Java, as all the variables —except primitive ones — can be assigned null, null probably
has a greater negative effect on software maintenance thanerror constants. For this reason, this research focuses on returnnull and null check.
In the remainder of this paper, we use the phrase “astatement is costly to maintain” to indicate “developers modify
the statement many times” and “the statement has a trend thatnew bugs are likely to be introduced into it.”III. E XPERIMENTAL D ESIGNIn this section, we explain the design of our experiment on
open source projects. Herein, we use the abbreviations listed inTable I for convenience. In addition, |S| means the number of
elements in a given set S. Let c be a commit between revisionr and r + 1, and then the set of added/deleted retnull , retnot ,condnull , and condnot in c are defined as Retcnull , Retcnot ,Condcnull , and Condcnot .

TABLE I: Abbreviations
Abbreviation

Explanation of Abbreviation

retnullretnotcondnull
condnot

return statements whose operands are null
return statements whose operands are NOT nullconditional expressions having comparison with nullconditional expressions NOT having comparison with null

RetrnullRetrnot
Retr
Condrnull
CondrnotCondr

a set of retnull in revision ra set of retnot in revision r
Retrnull [ Retrnot
a set of condnull in revision ra set of condnot in revision r
Condrnull [ Condrnot

Descrnull
Descrnot
Descr

Retrnull [ Condrnull
Retrnot [ Condrnot
Descrnull [ Descrnot

CR

a set of all commits in a target projecta set of all revisions in a target project

locr
latestlines of code in revision r
the latest revision in the specified term

A. Research Questions
We investigated the following research questions.
RQ1: Were retnull and condnull modified more frequently
than retnot and condnot ?
RQ2: Did the size of projects affect the frequency ofmodifications to retnull and condnull ?
RQ3: Did the development phases of the projects affect thefrequency of modifications to retnull and condnull ?
RQ4: Did the density of condnull increase as projects
proceeded?
RQ1 is the question on whether statements including null
are, in fact, costly to maintain. In this research, “a statement
is costly to maintain” means that the statement is modified
frequently. RQ2 and RQ3 are questions on whether the frequency of modifications depends on the characteristics anddevelopment periods of projects. RQ4 is the question on how
the density of condnull changes during software evolution. If
the density of condnull increases as a project proceeds, theoccurrences of condnull , which is not the functionality wewant to realize, are so many that condnull and retnull arecostly to maintain. retnull is also costly because it is the causeof writing condnull .
B. Target Projects
Table II shows the list of target projects. All the target
projects are Java systems managed using git, and all had ahigh number of revisions and the size of each revision was
not low. They were used in previous research [2], [3], [8], [9].
C. Experimental Method
In this experiment, the input was a target repository, andthe outputs were the following:
r
r
r
r
r
• Retnull , Retnot , Condnull , Condnot , loc , for 8r 2 Rccc•
Retnull , Retnot , Condnull , Condcnull , for 8c 2
C
In order to collect the necessary data, we proceeded asfollows. Figure 3 is an overview of the steps.

Step 1) We analyzed the source code in each revision to
obtain the number of instances of retnull/not and condnull/not
for each method. retnull is only the literal sequence of
tokens “return null,” and condnull are conditional predicatescomparing a variable to null, such as “a == null,” and “null!= a.”
Step 2) By using the data obtained in Step 1, we
found additions/deletions of retnull/not and condnull/not . We
regarded the changed number of instances of retnull/not
and condnull/not in a method between two revisions to be
additions/deletions, respectively. As we used the number of
instances of retnull/not and condnull/not , some changes wereignored in a case where some retnull/not were modified, but
the number of instances of retnull/not in the method was notchanged.
Step 3) We filtered out unnecessary additions/deletions.
If retnull/not and condnull/not were added/deleted as a partof module additions/deletions, they were filtered out. Thiswas done because the changes in numbers arising fromadding/deleting modules are not commonly caused by bug
fixes. In this experiment, when a method was added/deleted,
TABLE II: Target projects and their size
Project

loclatest

ant
commons-io
eclipse.jdt.core
egit
jEdit
jboss-as
jetty
JGit
log4j
lucene-soir
mavenorg.eclipse.cdt
org.eclipse.hudson.core
tomcat131,265
25,031
1,155,484
92,305
115,842
551,426
207,517
124,662
30,010537,150
72,201
1,029,497
81,876240,086

12,783
1,526
19,1403,126
6,221
10,764
6,082
2,321
3,226
8,026
9,312
21,157
1,008
9,172

Total

4,394,352

122,116|R|we considered it to be an addition/deletion of the modules.
In summary, our filtering omitted the additions/deletions ofretnull/not and condnull/not from the experimental target if
they satisfied any of the following conditions: their numberin their owner method was not changed, their owner method
disappeared, or their owner method appeared anew at a given
commit.
We obtained the remains of this filtering as necessary
modifications. In the example shown in Figure 3, we obtained
only modifications where the numbers were changed “0 to 1,”“2 to 1,” and “1 to 0.”
D. Calculating the Frequency of Modifications

X

fretnull (C) =c2C

fretnot (C) =c2C

fcondnull (C) =c2C

fcondnot (C) =c2C| Retcnull ||Retlatest |X null| Retcnot ||Retlatest |X not| Condcnull ||Condlatest
null |X| Condcnot ||Condlatest
not |



    !    

  

! 

Latest+revision

"





Step+1  3

  0  

2

3

3

1

1

3
Step+2

1

Step+3  

1

1

1

1

0

0



To answer the research questions, it is necessary to compare the frequencies of modifications between retnull/not andcondnull/not .
fretnull (C), fretnot (C), fcondnull (C), and fcondnot (C) are
the frequency of modifications to retnull , retnot , condnull ,and condnot . They were calculated as follows:n 

2

Number+of+targets+per+methodFig. 3: Overview of steps(1)
version ver.1.0.0

(2)

(3)

revision'

n

n+

k
2

ver.1.1.0'

n+k

n+k+

l
2

ver.1.2.0'

n+k +lanterior'half' posterior'half' anterior'half' posterior'half'
between'between'between'between'ver.1.0.0'and' ver.1.0.0'and' ver.1.1.0'and' ver.1.1.0'and'ver.1.1.0
ver.1.1.0
ver.1.2.0
ver.1.2.0

Fig. 4: Overview of division(4)

For example, in Eq. 1, the denominator is the total numberof retnull in the latest revision, and the numerator is thenumber of added/deleted retnull throughout all the commits. The result is the frequency of modifications to one
latestlatestretnull . By dividing |Retlatest
null |, |Retnot |, |Condnull |, andlatest|Condnut |, we can reduce the influence arising from thedifference in the number of modifications simply caused by
the difference of their number.
E. Unit of Analysis
In this experiment, by using the caller/callee relationshipsof methods, we obtained the modifications to condnull in acaller method when retnull was added to a callee method. As
this research focuses on methods, | Descc | for 8c 2 C were
calculated for each method.
F. Definition of Development Phases
We could not divide the development phases clearly because
the experimental targets were open source projects. In thisexperiment, we considered a period between major version
releases as one software development period, and we divided
each of the development periods into an anterior half anda posterior half. Figure 4 shows the division of the periods
between version 1.0.0 and version 1.2.0. We considered thatmodules had been added in the anterior half and bugs had
been fixed in the posterior half. We performed such divisions,and calculated the frequency of modifications in the anterior
half and the posterior half.
Division and statistical testing were conducted on
12 projects listed in Table II except for tomcat andeclipse.jdt.core. As the borders among major version releases
were vague, we excepted those projects.
G. Statistical Testing Methodology
In order to answer the research questions, we need tocheck whether two samples have a significant difference ora statistical correlation.
We used the Wilcoxon signed rank test [11] to determine
whether two paired-samples had a significant difference. If
the obtained p value is low, there is a low probability that theobserved differences are accidental. In this experiment, thesignificance level is 1%. This means that there is a significant
difference if p  0.01, and there is no significant difference if
0.01 < p.

Similarly, we calculated the Spearman’s rank correlation
coefficient (⇢ value) to determine whether two paired-samples
had a statistical correlation. After that, we calculated the pvalue corresponding to the ⇢ value. The two samples have apositive correlation if the ⇢ value is positive, and a negative
correlation if the ⇢ value is negative. The meaning of the pvalue is the same as for the Wilcoxon signed rank test. Two
samples have no correlation if |⇢| < 0.5, and they have acorrelation if 0.5  |⇢|.
IV. A NSWERS TO R ESEARCH Q UESTIONS
A. Answer to RQ1To ascertain whether retnull and condnull were modified
more frequently than retnot and condnot , we tested the difference between them. Figure 5 shows a comparison between
fretnull (C) and fretnot (C), fcondnull (C) and fcondnot (C). In
the figure, black regions are Descnull , and gray regions are
Descnot . In addition, the figure does not show a value justas it is, but the percentage for the value. For example, thepercentage of fretnull (C) is calculated as follows: fretnot (C),fcondnull (C), and fcondnot (C) are the same as fretnull (C).
fretnull (C)fretnull (C) + fretnot (C)C. Answer to RQ3There were 109 major versions among the target
projects. After dividing them, we calculated fretnull (C) andfcondnull (C) for the divided 218 periods. Figure 7 shows thecalculated values.
We tested fretnull (C) and fcondnull (C) for differences
between the anterior half and the posterior half. The result
from the Wilcoxon signed rank test between fretnull (C) in
the anterior half and the posterior half was p = 0.126(Figure 7(a)). As 0.01 < p, there was no significant difference.
Similarly, the result between fcondnull (C) in the anterior half
and the posterior half was p = 0.383 (Figure 7(b)). There wasalso no significant difference.
As a result, our answer to RQ3 is that the development
phases did not have a significant effect on the frequency ofmodifications on retnull and condnull .
D. Answer to RQ4
We calculated density(Condrnull ), representing the numberof condnull per line of code, for each revision r in the target
projects. The value was calculated as follows:|Condrnull |(6)
locr
For each project, we tested whether there was a positive correlation between the revision number (r) anddensitycondnull (r). A positive correlation means that the density of condnull is increasing as a project proceeds. Table III
shows the test result. The ⇢ value indicates the strength ofthe correlation, and the p value indicates whether there is asignificant correlation or not.
As a result, four projects showed significant positive correlations, five projects showed significant negative ones, andthree projects had no correlation. Two projects did not havesignificant correlation because the p value of two projects is
p > 0.01. This shows there was no regular pattern in the
results.
Consequently, our answer to the RQ4 is that the density ofcondnull did not increase as the projects proceeded.
densitycondnull (r) =(5)

The p value obtained from the Wilcoxon signed rank test between fretnull (C) and fretnot (C) is 2.44⇥10 4 (Figure 5(a)).
That means fretnull (C) fretnot (C) in almost all the projects
are positive values. As p  0.01, retnull was modified more
frequently than retnot . On the other hand, the p value between
fcondnull (C) and fcondnot (C) is 0.714 (Figure 5(b)). Thus,
condnull were not modified more frequently than condnot .
Therefore, our answer to RQ1 is that retnull was modified
more frequently than retnot , and condnull was not modified
more frequently than condnot .
B. Answer to RQ2
To ascertain whether the size of projects and the frequency
of modifications have any correlation, we hypothesized thatloclatest indicated the size of projects, and tested the correlation between them.
Figure 6 shows the scattergram between fretnull (C),fcondnull (C) and loclatest . The x-axis is loclatest , and the yaxis is fretnull (C) and fcondnull (C). We calculated that theSpearman’s rank correlation coefficient (⇢ value) and the pvalue corresponds to the ⇢ value.
The result from the Spearman’s rank correlation test between fretnull (C) and loclatest was ⇢ =0.0330, andp = 0.916 (Figure 6(a)). As 0.01 < p, there was no significant
correlation. Similarly, the result between fcondnull (C) andloclatest was ⇢ = 0.169, and p = 0.563 (Figure 6(b)). As
it was also 0.01 < p, there was no significant correlation.
As a result, our answer to RQ2 is that the size of the
projects did not have a significant effect on the frequency ofmodifications to retnull and condnull .

TABLE III: Spearman’s rank correlation coefficient (⇢ value) and p value forrevision number r and densitycondnull (r)Software⇢ valuep value

ant
commons-io
jdt.coreegit
jEdit
jboss-as
jetty
jgit
log4j
lucene-solr
maven
cdt
hudson.core
tomcat0.432
0.733-0.9740.223
0.668-0.765-0.9920.598
0.006-0.9470.934
0.156-0.151-0.986

under 2.2 ⇥ 10 16 aunder 2.2 ⇥ 10 16
under 2.2 ⇥ 10 16
under 2.2 ⇥ 10 16
under 2.2 ⇥ 10 16
under 2.2 ⇥ 10 16
under 2.2 ⇥ 10 16
under 2.2 ⇥ 10 16
0.740
under 2.2 ⇥ 10 16
under 2.2 ⇥ 10 16
under 2.2 ⇥ 10 16
2.898 ⇥ 10 4under 2.2 ⇥ 10 16

a This indicates the p value is too small to calculate a precise value.

ant

ant

commons−io

commons−io

jdt.core

jdt.core

egit

egit

jEdit

jEdit

jboss−asjboss−as

jetty

jetty

jgit

jgit

log4j

log4j

lucene−solrlucene−solr

maven

maven

cdt

cdt

hudson.core

hudson.core

tomcattomcat
0

20

40

60

80

100

(a) Comparison between fretnull (C)(black) and fretnot (C)(gray)

0

20

40

60

80

100

(b) Comparison between fcondnull (C)(black) and fcondnot (C)(gray)●ρ = −0.0330
p = 0.916

ρ = 0.169
p = 0.563

1.0

fcondnull

1.0

●

●

0.6

fretnull

●
●

●

●

●
●

●

●
●
●

●

●

●

●
●

●

0

●●

0.2

●
●

●

0.0

●
●

1.4

3.0

●

2.0

Fig. 5: Comparison between fDescnull (C)(black) and fDescnot (C)(gray)

●

200

600

LOC (1,000)

1000

0

(a) fretnull (C)

200

600

LOC (1,000)

1000(b) fcondnull (C)

0.30

●

0.30

Fig. 6: Scattergrams of loclatest and fretnull (C), fcondnull (C)

●

●
●
●

●
●

●
●

●
●

●
●
●
●
●
●

●
●
●
●
●

0.10

●
●

fcondnull

●

●
●
●

●
●
●
●

0.00

0.00

●
●

0.20

●
●

0.10

●
●

0.20

answers to RQ2 and RQ3 are that the size and the development
phases of projects did not affect the frequency of modifications
to retnull and condnull . This means a developer modified
retnull and condnull throughout the entire development periodof a project. Our answers above show that retnull affects software maintenance throughout the entire development period ofa project.

anterior half posterior half

anterior half posterior half(a) fretnull (C)(b) fcondnull (C)Fig. 7: Box plot showing fretnull (C) and fcondnull (C) of the anterior half
and the posterior half

V. D ISCUSSION
A. Discussion on the Results of RQ1, RQ2, and RQ3
Our answer to RQ1 is that retnull was modified more
frequently than retnot . As retnull was frequently modified, we
can say that their presence is probably costly to maintain. Our

B. Discussion on RQ4
In our answer to RQ4 (see IV-D), we showed that many
projects had positive or negative correlations between revision
number r and densitycondnull (r). We conducted additional
experiments to find common characteristics in the density on
condnull .
Figure 8 shows the relationship between revision numbers
and densitycondnull (r) in four projects. In Figure 8, the xaxis represents the revision number, and the y-axis represents
density(Condxnull ). These projects show that the continuation of the change has been maintained. This means thatthe density will have been decreasing/increasing, and it will
become a very low/high value. However, inspecting the overall
densitycondnull (r) gives us another idea.
Figure 9 shows densitycondnull (r) of 8r 2 R in all the
projects. Position x in the x-axis means the revision in the
top x% of revisions sorted by ascending order of revision
numbers. The y-axis is densitycondnull (x). The thick barshows the median of all the projects. This graph shows thatdensitycondnull (r) of all the projects is in the range 0.01
to 0.04, and the median is stable regardless of the progressof their projects. In other words, the number of condnull isfrom one to four lines per 100 lines, and this density will
be kept in the future of any project. This result also meansthat countermeasures to reduce retnull and condnull have notbeen taken in all the projects. After we see Figure 9, we canrecognize that the density reaches the area between 0.01 and(a) ant(b) egit

Fig. 9: densitycondnull (r) 8r 2 R of all projects. This figure shows thatdensitycondnull (latest) of all projects are between 0.01 and 0.04(c) maven(d) jetty

Fig. 8: Change in frequency of modifications to condnull in some projects.
Tendencies of condnull differ in each project0.04 in all projects as shown in Figure 8. Once a density
reaches the area, it is unlikely that the density will go out
from the area. This is presumably a common characteristic ofthe density in condnull .
In addition, if densitycondnull (latest) in a project is high(> 0.04), the project presumably has some problems. Thismeans that the number of condnull , which is not a functionality that we want to realize, has increased too much. Thus, if
densitycondnull (latest) is high, we suggest that the developer
should refactor (e.g., check the existence of unnecessary
condnull , returning suitable values instead of null, and remove
condnull by using a NullObject pattern).
C. Discussion on Modifications to retnull and condnull
We select modifications from Descc in 8c 2 C and
we discuss the effect on the density of condnull . In thisstudy, we discuss the modifications to retnull and condnull
in JGit. The reason we targeted JGit is that the size was not
too large, meaning we could check all the modifications, inaddition the JGit project recorded information on bugs using
a bug control system, Bugzilla [12]. We show the characteristic
modifications and discuss them.
Figure 10 shows an example of replacing retnull into anerror constant. This is a representative example of replacing
retnull with other mechanisms. Such a replacement oftencauses software design problems because the return value ischanged. It is difficult to perform such a replacement withoutcausing a problem in a fully automated way.

1
2
3
4
5
6
7

} else {return null;
if (merger.failedAbnormally())
return new CherryPickResult(merger.getFailingPaths());+// merge conflicts+
return CherryPickResult.CONFLICT;}++

Fig. 10: Example of replacing retnull into an error constant

Although such a replacement is necessary to identify theerrors that occurred in the same method, performing such a
replacement is difficult. Therefore, the presence of retnull iscostly to maintain.
Figure 11 shows the modification occurred in a commit
whose commit message is “[findBugs] Don’t pass null for
non-null parameter in RebaseCommand.” This commit message shows that developers detected a null dereference byusing FindBugs, and fixed the null dereference by adding
condnull . Many instances of null dereference occurred because
developers had forgotten to write condnull , but developers
could easily detect this by using null dereference detection
tools.
In Figure 12, as there was no condnull for the variable“objectID,” developers added condnull and the error handlingwhen “objectID” was null. The same problem was fixed in
two places, but the developers forgot to fix all of them. In
such cases, null dereference detection tools are effective. In
addition, we need to fix bugs in multiple places not only
when a developer forgets to write condnull , but also whena developer adds retnull to existing methods.
As described above, developers can detect null dereferencesby using null dereference detection tools and fix them. On the

1
DirCacheCheckout dco;
2 - RevCommit commit = walk.parseCommit(repo.resolve(
commitId));
3 + if (commitId == null)4 +throw new JGitInternalException(
5 +JGitText.get().abortingRebaseFailedNoOrigHead);
6 + ObjectId id = repo.resolve(commitId);
7 + RevCommit commit = walk.parseCommit(id);8
if (result.getStatus().equals(Status.FAILED)) {9
RevCommit head = walk.parseCommit(repo.resolve(
Constants.HEAD));10dco = new DirCacheCheckout(repo, head.getTree(),Fig. 11: Example showing lack of condnull
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

@@ -108,7 +110,10 @@ObjectId objectId = repository.resolve(revstr);- tree = new RevWalk(repository).parseTree(objectId);+ if (objectId != null)+tree = new RevWalk(repository).parseTree(objectId);+ else+tree = null;this.initialWorkingTreeIterator = workingTreeIterator;@@ -125,9 +130,13 @@this.repository = repository;- tree = new RevWalk(repository).parseTree(objectId);+ if (objectId != null)+tree = new RevWalk(repository).parseTree(objectId);+ else+tree = null;this.initialWorkingTreeIterator = workingTreeIterator;

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

Fig. 13: Example of removing condnull by modifying the default value

1
2
3
4
5
6

Fig. 12: Example of condnull being fixed in many places

other hand, sometimes we can remove null instead of simply
adding condnull . Figure 13 shows refactoring that removes
condnull by assigning a suitable value “repo.lockDirCache()”to variable “dc” as an initialization. However, it is difficult
to perform such refactorings because we need to analyze all
expressions that include a variable so as to be able to assign a
suitable value to a variable. In addition, there is no popular toolthat assigns suitable values to variables automatically. Therefore, the number of instances of condnull rarely decreased. We
can regard such a change as a factor of maintenance stem fromnull. Our experiment took such a change into consideration
because our experiment did not focus on only addition of
condnull .
In summary, there were retnull and condnull replaced withother mechanisms in order to avoid bad effects of retnull .
We can detect null dereference automatically by using null
dereference detection tools, and developers add condnull whennull dereference is detected. However, as it requires complex
analyses, reducing the number of instances of retnull andcondnull is difficult both when using manual methods andwhen using automatic methods.
D. Replacing Return Null with Other Mechanisms
Exception handling is one of the mechanisms for replacing
retnull . We measured the frequency of modifications on throwstatements and try statements. Then, we compared them withfretnull (C) and fcondnull (C). Figure 15 shows the comparison
result. The result of the Wilcoxon signed rank test between
fretnull (C) and throw statements was p = 0.0245. As 0.01 <p, there was no significant difference. Similarly, the result

private void resetIndex(RevCommit commit) throws
IOException {DirCache dc = null;+
DirCache dc = repo.lockDirCache();try {dc = repo.lockDirCache();dc.clear();
DirCacheBuilder dcb = dc.builder();dcb.addTree(new byte[0], 0, repo.newObjectReader(), commit.getTree());
dcb.commit();} catch (IOException e) {throw e;} finally {if (dc != null)dc.unlock();+
dc.unlock();}}

7

Object referencedObject = getTaskContext().getDataValue(reference);
if (referencedObject == null) {throw new ExecutionException("Unable to locate thereference specified by refid ’" + getReference() +"’");}if (!this.getClass().isAssignableFrom(referencedObject.
getClass())) {throw new ExecutionException("The object referenced byrefid ’" + getReference() + "’ is not compatible
with this element ");}Fig. 14: Example of throw statements being written in multiple places

between fretnull (C) and try statements was p = 0.390. As
it was also 0.01 < p, there was no significant difference.
However, developers sometimes write multiple throw statements, all of which correspond to a single retnull . This isbecause throw statements have error messages and exception
types. Developers can change error messages and exception
types depending on the reasons of errors (Figure 14 shows anexample of such a situation). For this reason, the frequencyof the modifications on throw statements became a larger
value than the actual one. Therefore, it is not fair to compare
the frequencies of the modifications between retnull andexception handling. Exception handling is better than retnull
from the qualitative points of view, such as the examples in
Section V-C, and the characteristic that retnull have no error
message.
Replacing retnull and condnull is related to research thatis intended to detect null dereference, because detecting null
dereference is the same as detecting passing null from one
method to another method and a variable can be assigned null.
The information as to which variables can be assigned null isvery useful when it comes to replacing retnull and condnull .
Therefore, the results of this research reinforce the motivationof those studies, e.g., “As null dereference occurs frequently,
it is useful to detect them,” with the motivation “Supporting
replacing retnull and condnull that affect software maintenance.”

ant

ant

commons−io

commons−io

jdt.core

jdt.core

egit

egit

jEdit

jEdit

jboss−asjboss−as

jetty

jetty

jgit

jgit

log4j

log4j

lucene−solrlucene−solr

maven

maven

cdt

cdt

hudson.core

hudson.core

tomcattomcat
0

20

40

60

80

100

0

20

40

60

80

100

(a) Comparison between fretnull (C)(black) and the frequency of modifica- (b) Comparison between fcondnull (C)(black) and the frequency of modifitions on throw statements(gray)cations on try statements(gray)Fig. 15: Comparison between fDescnull (C)(black) and the frequency of modifications on exception handling mechanisms(gray)VI. T HREATS TO VALIDITY
The fairness of comparison on RQ1: As stated in
Section V-D, it is not appropriate to compare retnull withthrow statements, because the frequency of modifications
on throw statements was larger than the actual one. The
unfairness also happened to retnot . Although fretnot (C) was
presumably larger than the actual one, fretnull (C) was larger
than fretnot (C) under adverse conditions (as shown in Section IV-A). Thus, our experimental results were not altered by
the unfairness.
Code outside the methods: In this experiment, Descc
for 8c 2 C were calculated for all methods. This calculation
had the issue that retnull/not and condnull/not were ignored
if retnull/not and condnull/not were outside the methods.
However, retnull/not and condnull/not were rarely written
outside the methods because retnull/not and condnull/not
were return statements or conditional predicates. Thus, we
excluded from consideration retnull/not and condnull/not thatwere outside the methods.
Changed/Moved code elements:
Descc for c 2 C
consisted only of additions/deletions. Thus, we did not detect“Change” modifications, for example, a variable compared tonull had been changed to another variable. This may havean effect on the experimental result and the discussion. In
addition, as our experimental steps could not detect a code
move of retnull/not and condnull/not , a code move causedby refactoring, such as Extract Method, was recognized as
modifications of retnull/not and condnull/not . A code movecaused by refactoring should be filtered out, similarly to an
addition/deletion of modules. However, in this experiment,
we considered that the density of code moves was low, andignored them.
Variables whose values are null: retnull is only a returnstatement whose operand is null. If a return statement has avariable whose value is null, the statement is not counted. The
situation is the same for condnull . For this reason, the numberof retnull and condnull could differ from the actual number.
The division of the development phases: In the comparison of the development phases, we divided the periods
between major versions into two periods, and tested whether
they had a significant difference. However, as this division
was performed based on the number of revisions, we could
not obtain a precise division of the development phases. As
a result, the experimental result could differ from the result
when the periods were divided more precisely.
VII. R ELATED W ORK
Many previous studies have tackled null dereference problems from many viewpoints. Currently, there are many tools to
detect null dereference automatically, which were introducedin Section II. Those tools target existing null dereference.
By contrast, there are techniques that focus on preventing
null dereference from occurring in compile time. One suchtechnique is that developers add the constraint “null value isnever assigned to this variable” (non-null) to some variables.
This technique enables compilers to verify whether the variables could be null. As a result, these techniques can reduce
mistakes in which a null value is unintentionally assigned
to some variables. In formal methods, developers represent a
program as a formal specification using formal specification
languages, such as JML (Java Modeling Language) [13],and check whether the constraints are violated or not byperforming verifications. Developers can obtain proof that thevariable will never become null by using these methods.
However, writing formal specifications needs a great deal ofeffort on the part of developers. Therefore, researchers tend
to implement non-null features as the preferred type system
and annotations. Fähndrich et al. proposed a NonNull type ofsystem and implemented it on C# as an annotation [14]. Papi

et al. extended the Java-type system and enabled checks to bemade as to whether variables annotated @N onN ull can benull or not [15]. There are many researchers that are trying
to make an environment in which non-null can be used in apractical environment.
There also are methods that enforce null check onvariables that can be null. Hovemeyer et al. developed@CheckF orN ull annotation, and proposed a method thatenforces null check on variables on Java [5]. Haskell [16]and Scala [17] have variables of Maybe type or Option typethat represent variables that can be null. Variables of thesetypes contain actual values or Nothing values, which areequivalent to a null value. When developers use such variables, they spontaneously handle cases in which the variables
have Nothing values. Such types of systems make developers
aware of exception handling, and can detect the lack of a nullcheck in compile time. As a result, such systems prevent null
dereference from occurring.
In addition, null as an initializer is a subject of research,
because such an initializer is another factor in the occurringof null dereference. By using initialization analysis, whichanalyzes whether variable initializations have been done or not,developers can know which variables are uninitialized [18].
This decreases the possibility of null dereference occurring.
In addition, there are techniques that track the origins of nullvalues [19]. As described above, many approaches have been
proposed.
VIII. C ONCLUSION
In this paper, we investigated whether, and to what extent, the presence of return null and null check is costly tomaintain by mining modifications during software evolution.
This research focused on return null and null check, and an
experiment conducted on 14 projects showed that return null
were modified more frequently than statements that do nothave null. In addition, we suggested that developers should
avoid writing return null by reason of our qualitative analysis.
Moreover, it was found that the frequencies of modifications
on return null and null check bore no relation to the size andthe development phases of projects. The density of null checks
was from one to four in 100 lines, in each project. If thedensity of null check in a project exceeds 0.04, we can saythat developers should refactor in order to decrease the density.
A developer could replace return null with other mechanisms,
for example, exception handling, a proper object such as an
empty array, or a NullObject pattern.
We plan to research the reasons for writing return nulland null check, and see to what extent we can lower thefrequency of modifications. We can focus our experiment on“addition of null check.” Results of such an experiment will
be worth discussing. In addition, it is our future work to

support developers by replacing return null and null checkautomatically. We also plan to find a method in order tocompare return null with other error handling mechanisms
quantitatively, and conduct the same experiment for otherprogramming languages such as C.
ACKNOWLEDGMENT
This study was supported by Grants-in-Aid for Scientific
Research (S) (25220003), Grant-in-Aid for Exploratory Research (24650011) from the Japan Society for the Promotion of Science, and Grant-in-Aid for Young Scientists (A)(24680002) from the Ministry of Education, Culture, Sports,
Science and Technology.
R EFERENCES[1] T. Hoare, “Historically Bad Ideas: ”Null References: The Billion Dollar
Mistake,” in QCon, 2009.
[2] M. G. Nanda and S. Sinha, “Accurate Interprocedural Null-Dereference
Analysis for Java,” in ICSE, May 2009, pp. 16–24.
[3] N. Ayewah and W. Pugh, “Null Dereference Analysis in Practice,” in
PASTE, Jun. 2010, pp. 65–72.
[4] W. R. Bush, J. D. Pincus, and D. J. Sielaff, “A static analyzer for finding dynamic programming errors,” Software: Practice and Experience,
vol. 30, no. 7, pp. 775–802, Jun. 2000.
[5] D. Hovemeyer, J. Spacco, and W. Pugh, “Evaluating and tuning a static
analysis to find null pointer bugs,” in PASTE, Sep. 2005, pp. 13–19.
[6] A. Loginov, E. Yahav, S. Chandra, S. Fink, N. Rinetzky, and M. Nanda,“Verifying Dereference Safety via Expanding-Scope Analysis,” in ISSTA,
Jul. 2008, pp. 213–224.
[7] D. Hovemeyer and W. Pugh, “Finding More Null Pointer Bugs, But Not
Too Many,” in PASTE, Jun. 2007, pp. 9–14.
[8] M. G. Nanda, M. Gupta, S. Sinha, S. Chandra, D. Schmidt, and
P. Balachandran, “Making Defect-Finding Tools Work for You,” in ICSE,May 2010, pp. 99–108.
[9] D. Hovemeyer and W. Pugh, “Finding Bugs is Easy,” in OOPSLA, Oct.
2004, pp. 24–28.
[10] C. Flanagan, K. R. M. Leino, M. Lillibridge, G. Nelson, J. B. Saxe, and
R. Stata, “Extended Static Checking for Java,” in PLDI, Jun. 2002, pp.
17–19.
[11] M. Hollander and D. A. Wolfe, Nonparametric Statistical Methods, 2nd
Edition. Wiley-Interscience, 1999.
[12] Bugzilla, “http://www.bugzilla.org/.”[13] G. T. Leavens, A. L. Baker, and C. Ruby, “Preliminary Design of JML:
A Behavioral Interface Specification Language for Java,” SIGSOFT
Software Engineering Notes, vol. 31, no. 3, pp. 1–38, May 2006.
[14] M. Fähndrich and K. R. M. Leino, “Declaring and checking nonnull types in an object-oriented language,” in OOPSLA, Oct. 2003, pp.
302–312.
[15] M. M. Papi and M. D. Ernst, “Compile-Time Type-Checking for Custom
Type Qualifiers in Java,” in OOPSLA, Oct. 2007, pp. 809–810.
[16] S. P. Jones, J. Hughes, L. Augustsson, D. Barton, B. Boutel, W. Burton,
J. Fasel, K. Hammond, R. Hinze, P. Hudak, T. Johnsson, M. Jones,
J. Launchbury, E. Meijer, J. Peterson, A. Reid, C. Runciman, andP. Wadler, “The Haskell 98 Language Report,” Dec. 2002, http://www.
haskell.org/onlinereport/.
[17] M. Odersky, “The Scala Language Specication Version 2.9,” May 2011,
http://www.scala-lang.org/docu/files/ScalaReference.pdf.
[18] F. Spoto and M. D. Ernst, “Inference of Field Initialization,” in ICSE,May 2011, pp. 231–240.
[19] M. D. Bond, N. Nethercote, S. W. Kent, S. Z. Guyer, and K. S.
McKinley, “Tracking Bad Apples: Reporting the Origin of Null andUndefined Value Errors,” in OOPSLA, Oct. 2007, pp. 405–422.