Statistical Analysis of Time Series Data on the Number of Faults Detected by
Software Testing
Sousuke AMASAKI, Takashi YOSHITOMI, Osamu MIZUNO, Tohru KIKUNO, and Yasunari TAKAGI
Graduate School of Information Science and Technology, Osaka University, Japan
E-mail: amasaki@ics.es.osaka-u.ac.jp, {o-mizuno,kikuno,y-yakagi}@ist.osaka-u.ac.jp

Abstract

examined successively. Therefore, we can get time series
data from these successive testing activities. However, as
far as we know, these data have been analyzed individually(that is, data for each unit test, integration test, function test,
or validation test were analyzed), and the relationship or thetrend of time series data has not yet been studied [7, 10, 12].
In this paper, we try to analyze the trends of the timeseries data on the number of faults detected by unit test, integration test, function test, and validation test, and clarify
the relationship between the trends and the field quality ofsoftware products. This result will make it possible to estimate field quality during testing, and make a remarkableprogress in the software process improvement [1, 9, 13].
First, we introduce the Kendall’s rank correlation coefficient τ [6] in order to discuss the trends. Based on theexperimental evaluation of actual project data using the coefficient τ , we define four types of trends: strict decreasing,almost decreasing, almost increasing, and strict increasing.
Then, we investigate, for each type of trend, the field
quality of software products developed by the corresponding software projects. As a result of statistical analyses,
we showed that software projects having trend of almostor strict decreasing in the number of faults detected by thesoftware testing could produce the software products withhigh quality.
The rest of this paper is organized as follows: Section
2 describes preliminaries of this research. Section 3 shows
the definition of the metrics and the overview of collected
time series data to be used for the analysis. In Section 4,
we classify the projects into four groups based on the trendof the number of the faults detected in the testing process.
We then show the relationship between the trends and field
quality in Section 5. Finally, Section 6 concludes this paperand show some future works.

According to a progress of the software process improvement, the time series data on the number of faults detectedby the software testing are collected extensively. In thispaper, we perform statistical analyzes of relationships between the time series data and the field quality of software
products.
At first, we apply the rank correlation coefficient τ to thetime series data collected from actual software testing in acertain company, and classify these data into four types oftrends: strict increasing, almost increasing, almost decreasing, and strict decreasing. We then investigate, for each typeof trend, the field quality of software products developed by
the corresponding software projects. As a result of statistical analyses, we showed that software projects having trendof almost or strict decreasing in the number of faults detected by the software testing could produce the software
products with high quality.
Keywords: software testing, software quality, statisticalanalysis.

1 Introduction
Reducing the cost for software maintenance is recognized as a serious problem, since the lifetime of software
becomes long and software is updated many times according to new requirements. It is well known that one of thefundamental causes for serious excess of cost is the remaining faults in the software. Thus, reducing the number of theremaining faults at the development is very important [1,7].
A straightforward approach is to perform sufficient testing during the development. In order to manage the testing
effectively, a software reliability growth model (SRGM) is
proposed and really applied to many actual projects [10].
Since the SRGM needs the data on the number of faults detected by the testing activities, software metrics are defined
and these quality data are extensively collected.
In the typical software development projects, the software testing consists of the unit test, integration test, function test, and validation test. Additionally, these tests are

2 Target Projects
2.1 Process ModelIn the target projects, the products are developed undera development process as shown in Figure 1. The development process is an ordinal waterfall model. This process1certain company. The software products developed by the
projects have the following common characteristics. The
systems are related to retail systems, and thus embedded
software implements rather complex functions dealing withmany sensors, actuators and control signals including various kinds of interrupts. Furthermore, since it is delivered in
the form of LSI chips, modification of faults after delivery
is very expensive. Thus, high quality is especially required
for the embedded software.
We use the actual project data of 134 projects, whichhave already finished their development. Each project was
carried out between 1995 and 1998. Additionally, we select these projects under a condition that the total numberof faults detected during the test & debug phase exceeds
more than a certain number. It implies that the size of target
project is relatively large.

Concept Design & Review
Function Design & Review
Structure Design & Review
Module Design & Review

Design &Coding phase

Coding & Review (CR)

Unit Test & Debug (UT)
Integration Test & Debug (IT)Function Test & Debug (FT)

Test &
Debug phase

Validation Test & Debug (VT)

3 Time Series Data

Figure 1. Development process
consists of two successive phases, design & coding phaseand test & debug phase. The design & coding phase is divided into five activities: Concept design, Function design,
Structure design, Module design, and Coding(CR). The
test & debug phase is divided into four activities: Unit test
& debug(U T ), Integration test & debug(IT ), Function test
& debug(F T ), and Validation test & debug(V T ).
One characteristic of the design & coding phase is that
review activity is introduced after each design activity. Review activity enables the detection and correction of faults
in software artifacts as soon as these artifacts are created.
The review activity not only improves the quality of the artifacts but also helps software development organizations to
reduce their cost of producing software [2, 3]. In the review
activity, the documents should be distributed to the persons
concerned in the company, and then review results should
be returned to developers via manager (this review activity
is called peer review [2]). They established several guidelines for the review activity. One of them directs that at least
15% of the total efforts for design & coding phase should
be assigned to review activities [9].
The test & debug phase consists of the repetition of apair of test activity and debug activity. Test activity is theprocess of analyzing a software item to detect the differences between existing and required conditions and to evaluate the features of the software items. Debug activity is theprocess to detect, locate, and correct faults [4]. The persons
engaged in the test and debug phase are directed to record
all faults that are detected by the test activity and removedby the debug activity [13]. In order to improve and assure
the quality of the software product, the product needs to betested sufficiently.

3.1 Software Metrics
In this subsection, we define several software metrics
used in this paper.
(1) Frequency of detected faults: DF
The frequency of detected faults (DF ) is the number ofdetected faults normalized by the effort needed for an
activity for detection. This metric represents the abilityof developer and density of faults indirectly. In order todefine DF , we introduce the following two parameters
where α is an activity out of five activities CR, U T ,IT , F T and V T .
Dα : the number of detected faults in an activity α.
Eα : the effort needed for an activity α (measured byperson-day).
For example, the number of detected faults in the Unit
test & debug activity (U T ) is described as D UT . Then
DF for the activity α is defined as follows:
DFα =

Dα
Eα

(1)

According to this definition, we define the frequency ofdetected faults DFCR , DFUT , DFIT , DFF T , DFV T
for activities CR, U T , IT , F T , and V T , respectively.
(2) Size of product: S
Here, we introduce the parameter LOC that counts thetotal lines of source codes except for comment lines.
Using LOC, the size of product S is defined as follows:
LOC
(2)
S=
1000
The unit of S is usually represented by KLOC.

2.2 Characteristics of Projects
The projects targeted in this paper are the developmentof computer control systems with embedded software in a
2

d

Co

st
st
st
st
Te
Te
Te
Te
it
on tion
on
i
i
t
t
Un
ra
nc
da
Fu Vali
teg
In

ew

vi

e
eR

d

Co

ss

st
st
st
st
Te
Te
Te
Te
it
on tion
on
i
i
t
t
Un
ra
nc
da
Fu Vali
teg
In

ew

vi

e
eR

ce

2.9
6.3
0.8
0.3

Pr
o

...

1.0
3.3
0.7
0.5

g

...

0.2
15.2
1.4
1.1

tin

...

1.8
5.3
2.2
1.1

Te
s

...

11.6
3.3
7.3
5.2

value of DF

...

ss

...
133
134
Average
Median

ce

DF VT
0.5
0.0

Pr
o

DF FT
1.8
1.2

g

DF IT
0.7
1.7

tin

DF UT
0.6
0.0

Te
s

DF CR
2.1
6.7

value of DF

Table 1. Actual project data
No.
12(a) Project No.5(b) Project No.20

oc

Pr

Te
s

Te
s

Co

es

s
d

tin

d

Co

g

g

In this paper, we assume that D f ield is the number offaults detected during six months after delivery.

st
st
st
st
Te
Te
Te
Te
it
on tion
on
i
i
t
Un
t
c
a
a
r
n
d
Fu Vali
teg
In

ew

vi

e
eR

tin

t
t
st
st
es
es
Te
Te
n T ion
it
nT
o
n
i
t
t
tio
U
c
a
a
r
n
d
Fu Vali
teg
In

ew

vi

e
eR

es

s

(3)
oc

Df ield
S

Pr

FFD =

value of DF

Field fault density F F D is a metric to measure thequality of final products. It is defined as follows:

value of DF(3) Field fault density: F F D(c) Project No.41(d) Project No.47

es

Te
s

tin

g

Pr

oc

es
oc

Pr

(e) Project No.63

Co

d

e
eR

t
t
st
st
es
es
Te
Te
n T ion
it
nT
o
n
i
t
tio
U
c
at
a
r
n
d
Fu Vali
teg
In

ew

vi

g

tin

Te
s

d

Co

t
t
st
st
es
es
Te
Te
n T ion
it
nT
o
n
i
t
tio
U
c
at
a
r
n
d
Fu Vali
teg
In

ew

vi

e
eR

s

s

The effort data and faults data are recorded manually andstored in workstations by each developer. Then, they arecollected by the project leader, and validated by the manager. On the other hand, field faults data are reported bya quality assurance staff, translated into a fault-based number by the project leader, and also validated by the manager.
All the validated data are sent to the software engineering
process group (SEPG), who analyzes them and reports theanalysis report back to the project team and development
organization [12].
Table 1 shows the actual project data. It includes quality data for DFCR , DFUT , DFIT , DFF T , DFV T for 134
projects. Concerning F F D, we have actual data, but wecannot show them here by the contract with the company.
In Table 1, for projects No.1, No.2, No.133, and No.134,
all metrics data are shown. However actually, for several
projects, some metric data are missing. In this paper, we
can use all of 134 projects since the analysis is applicable in
Sections 4 and 5.
Table 1 also summarizes the average and the median of
DF . As far as we observe the average and the median values, the values of DF ’s are decreasing as the testing activities proceed from CR to V T . Then, Figure 2 shows theactual values of DF ’s for six projects. From this figure, we
can find a certain trend in the successive values of DF CR ,DFUT , DFIT , DFF T , and DFV T . Intuitively speaking,projects No.5, No.20, and No.41 show the trend of decreasing in these successive DF ’s and project No.63 shows thetrend of increasing in these successive DF ’s.

value of DF

value of DF

3.2 Collected Data(f) Project No.87

Figure 2. Trend of actual data

4 Classification by Trend of Faults Detection
4.1 Key Idea
As is shown in Table 1 and Figure 2, the projects could
be classified into several groups by the trend of faults detection (for example, the trend of increasing or decreasingin the successive DF values). In order to identify the trend,
we introduce a quantitative measure of the trend. Although
there are many metrics for measuring trend, the rank correlation coefficient is one of the most popular metrics formonotonic increasing or decreasing trend [6,11]. In this paper, we adopt the Kendall’s rank correlation coefficient τ .
The following shows the definition of Kendall’s rankcorrelation coefficient τ : Here, we use two variables X
and Y for comparison, and represent i-th observation by3

Xi and Yi . For any sample of n observations, there aren(n − 1)/2 possible comparisons of points (X i , Yi ) and(Xj , Yj ) (1 ≤ i, j ≤ n). Firstly, we define C to be thenumber of pairs that are concordant, and D to be the number of pairs that are not concordant, respectively. Then, τ isdefined by the following formula:(C − D)τ=  n[(2 ) − nx ] · [(n2 ) − ny ]Table 2. Classification by fault detection trend
Types
Strict decreasing(SD)
Almost decreasing(AD)
Almost increasing(AI)
Strict increasing
(SI)

(4)

where nx and ny is the number of ties involving X and Y
respectively. Thus, the range of τ becomes −1 ≤ τ ≤ 1.
By using the Kendall’s τ calculated by Equation 4, we
can quantitatively measure the trend of faults detection. In
this paper, we define the following four types of trends
based on the value of τ .

Number of
projects (%)
51 (38.1%)
64 (47.8%)
16 (11.9%)
3 (2.2%)

Because of taking these trend for classification, the effectof the factors such as the skill level of development team,
the kind of product, etc. is mitigated and included in thetrend.

(a) Strict decreasing type(SD) (τ = −1)

4.2 Classification Result(b) Almost decreasing type(AD) (−1 < τ < 0)(c) Almost increasing type(AI) (0 ≤ τ < 1)(a) Strict decreasing: Many faults are detected and removed in the early stage, and then a few faults are detected and removed in the later stage. Thus, it is expectable that only a very few faults remain in the final
software product.

Table 2 shows the result of classification by applying
Kendall’s rank correlation coefficient τ to 134 projects. We
can see that almost 38% projects are in type SD, and almost
48% projects are in type AD. Thus, 86% projects have decreasing trends. But the rest 14% projects have increasing
trends (that is, types AI and SI).
In order to find the relationship between the F F D andfour types, we present histograms of the F F D in Figure 3.
Please note that y-axis of Figure 3 denotes the ranks of thevalues of F F D rather than the values themselves. Here,the rank of each project takes a value from 1 to 134. Thus,
rank= 1 implies the highest quality (that is, F F D = 0 in
actual data) and rank= 134 implies the lowest quality (forthis case, we cannot show actual value of F F D by the contract). Figures 3(a) and (b) show that almost half of projects
with types SD and AD have the rank= 1 and thus havevery high quality. On the other hand, Figure 3(c) and (d)show that types SI and AI tend to include projects withlower quality on the average. Figure 3 also shows the average rank of the field quality for each type.
From Table 2 and Figure 3, we can expect that the average field quality of the projects with the types SD and AD
is rather good. On the other hand, the field quality is not
so good for types SI and AI. Concerning this property, we
will investigate further in the next section.

(b) Almost decreasing: This type is similar to “strict decreasing type”. But the trend is not strict, and thus a
few remaining faults may be found.

5 Relationship between Trend and Field
Quality(d) Strict increasing type(SI) (τ = 1)
Here, we explain the motivation of the classification forτ . We firstly tried to define the two trends: decreasing(−1 ≤ τ < 0 ) and increasing (0 ≤ τ ≤ 1). However, basedon the experience of the actual developers, there may be asignificant difference between the projects with τ = −1 (orτ = 1) and −1 < τ < 0 (or 0 ≤ τ < 1). We thus use theabove four types in this paper.
Projects No.5 and No.41 with τ = −1 in Figures 2(a)and 2(c), respectively, are classified into type SD. Then,project No.20 with τ = −0.8 in Figure 2(b) and projectNo.87 with τ = −0.6 in Figure 2(f) are type AD. ProjectNo.47 with τ = 0.33 in Figure 2(d) is type AI. Finally,
project No.63 with τ = 1 in Figure 2(e) is type SI. This
result implies that the value of Kendall’s τ is adequate for
distinguishing these trends.
The followings present interpretations of four trends:(c) Strict increasing: A few faults are detected and removed
in the early stage, and then many faults are detected
and removed in the later stage. Thus, many faults still
remain in the final software product.

5.1 Outline of Analysis
In order to compare field quality of projects having fourtypes of trends, we introduce the parameter θ of the location on the rank of the F F D. In this paper, we define θ SD ,for projects with type SD, to be the average rank of F F D.
Similarly, we define θ AD , θAI , and θSI for projects with(d) Almost increasing: This type is similar to “strict increasing type”. But the trend is not so strict, and thus
many faults may remain.
4

types AD, AI, and SI, respectively. For simplicity, we call
θSD , θAD , θAI , and θSI the average F F D of projects withtypes SD, AD, AI, and SI, respectively. Then, from theinterpretations, we naturally derive the following relationship.
(5)θSD ≤ θAD ≤ θAI ≤ θSI

40

average = 44.5

35degree30
25
20

In this paper, we planned two steps of analysis for the
average F F D, θSD , θAD , θAI , and θSI of projects.

15
10
5
0
1

21

41

61

81

101

Step 1. (Jonckheere Test [8]) We check if there is a significance difference among θ SD , θAD , θAI , and θSI
and there exists the order shown in Equation 5.

121 134

Rank of FFD(a) Strict decreasing type

Step 2. (Multiple Comparison) Since the order exists
among θSD , θAD , θAI , and θSI , we perform Ryan’s
procedure [5]. Ryan’s procedure consists of the following procedure: Firstly, compare the farthermost
pair on order. Next, compare the next farthermost
pairs. By performing pairwise comparison step bystep, we finally analyze a significant difference between neighboring pairs.

40

average = 46.6

degree

35
30
25
20
15
10
5

5.2 Experimental Evaluation

0
1

21

41

61

81

101

121 134

Rank of FFD(A) Jonckheere test(b) Almost decreasing type

We define two hypotheses H 0 and H1 for the average
F F D, θSD , θAD , θAI , and θSI of projects. The hypothesis H1 is clearly alternative of the null hypothesis H 0 . In
this analysis, the level of significance α is chosen as 0.05.
H0 : There is no difference among θ SD , θAD , θAI , and θSI .
That is, the following relation holds:

40

average = 78.6

degree

35
30
25
20
15

θSD = θAD = θAI = θSI

10

(6)

5

H1 : Either of the following relations comes true:

0
1

21

41

61

81

101

121 134

Rank of FFD

θSD < θAD ≤ θAI ≤ θSI
θSD ≤ θAD < θAI ≤ θSI
θSD ≤ θAD ≤ θAI < θSI
θSD < θAD < θAI ≤ θSI
θSD < θAD ≤ θAI < θSI
θSD ≤ θAD < θAI < θSI
θSD < θAD < θAI < θSI(c) Almost increasing type40

average = 108.7

degree

35
30
25
20
15
10

By Jonckheere test, since the probability that H 0 cannot
be rejected becomes 0.02, the null hypothesis H 0 is rejected
at the 0.05 level. Thus, the alternative hypothesis H 1 is accepted. This result implies that there is a statistically significant difference in the average F F D, θ SD , θAD , θAI , andθSI of projects as a whole, and the field quality becomeslarger in the order of SD, AD, AI, and SI. However, at
this stage we cannot know which relation in the hypothesis
H1 holds.

5
0
1

21

41

61

81

101

(7)
(8)
(9)
(10)
(11)
(12)
(13)

121 134

Rank of FFD(d) Strict increasing typeFigure 3. Histogram of F F D for each type5

project has a specific trend. Especially, for the trends SI
and AI, we need a process improvement based on the reasons.

Table 3. Result of Ryan’s pairwise comparison
Step
1
2
3

NominalPair of types
significance level α'(θSD, θSI)
α'=0.017(θSD, θAI)
α'=0.025(θAD, θSI)
(θSD, θAD)
(θAD, θAI)
α'=0.05(θAI, θSI)

p-value

References

0.017
0.016
0.024
0.430
0.026
0.089

[1] V. Basili and J. Musa. The future engineering of software: A management perspective. IEEE Computer,24(9):90–96, 1991.
[2] D. B. Bisant and J. R. Lyle. A two-person inspection method to improve programming productivity. IEEE Trans. Software Engineering, 15(10):1294–
1304, 1989.

(B) Ryan’s multiple comparison
Next, the result of Ryan’s pairwise comparison is summarized in Table 3. Step 1 shows there is somewhat significant
difference between θ SD and θSI . Similarly Step 2 shows
there is a significance difference between θ SD and θAI , andalso between θAD and θSI . Step 3 implies that there is significant difference between θ AD and θAI . But, there is not
significance difference between θ SD and θAD , and betweenθAI and θSI . One of the reasons why there is no difference
between type AI and SI is that the number of projects included in the class SI was rather small. We therefore needmore project data and have to perform further analysis as a
future work.
From these facts, Equation 8 holds for the average F F D
of projects. This implies that the average rank of the field
fault density F F D becomes larger according to the order ofθSD , θAD , θAI , and θSI , and that especially large difference
exists between θAD and θAI .
In subsections 4.1 and 4.2, we estimated that two types
SD and AD behave the same way with respect to (the rankof) F F D. Similarly two types SI and AI behave the sameway. But, there is a large difference between two types AI
and AD. The estimation agrees with this result.

[3] M. E. Fagan. Advances in software inspections. IEEE
Trans. Software Engineering, 12(7):744–751, 1986.
[4] International Standard Organization. IEEE standard
glossary of software engineering terminology,. IEEE
Std 610.12-1990, 1990.
[5] S. Iwahara. Psycological statistics: Non-parametricmethod.
Nihon Bunka Kagakusha Co.,Ltd. (inJapanese), 2 edition, 1968.
[6] M. Kendall and J. D. Gibbons. Rank correlation methods. Edward Arnold, 5 edition, 1990.
[7] M. S. Krishnan and M. I. Kellner. Measuring process
consistency: Implications for reducing software defects. IEEE Trans. Software Engineering, 25(6):800–
815, 1999.
[8] E. L. Lehmann. Nonparametrics: Statistical methods
based on ranks. Holden-Day, Inc., 1975.
[9] O. Mizuno and T. Kikuno. Empirical evaluation of review process improvement activities with respect topost-release failure. In Empirical Studies on Software Development Engineering (ICSE’99 Workshop),
pages 50–53, 1999.

6 Conclusion
In this paper, we have analyzed time series data on thenumber of faults detected by successive code review andtesting activities. By applying the rank correlation coefficient to actual project data, we have successfully classified
the data into four types of trends: SI, AI, AD and SD.
Then, we have investigated the relationships between trends
and field quality, and showed that software project having
trend AD or SD would produce high quality products.
However, since we used only the data from embedded
software projects, we have to analyze the data from otherkinds of projects. On the other hand, construction of amodel, which calculates the number of remaining faults
based on time series quality data, is also an important future work.
The goal of this research is to estimate field quality basedon the time series data on the number of faults detected byvarious review activities and testing activities. In order toattain the goal, we must analyze the reason why the target[10] J. D. Musa, A. Iannino, and K. Okumoto. Software reliability: Measurement, prediction, application. McGraw-Hill, 1987.
[11] S. Muto. Statistical analysis handbook.
Books (in Japanese), 1995.

Asakura[12] Y. Takagi, T. Tanaka, N. Niihara, K. Sakamoto,
S. Kusumoto, and T. Kikuno. Analysis of review’s
effectiveness based on software metrics. In Proc. of
6th Int’l Symposium on Software Reliability Engineering(ISSRE’95), pages 34–39, 1995.
[13] T. Tanaka, K. Sakamoto, S. Kusumoto, and K. Matsumoto. Improvement of software process by process description and benefit estimation. In Proc. of
17th International Conference on Software Engineering(ICSE’95), pages 123–132, 1995.

6