On Estimating Testing Eﬀort
Needed to Assure Field Quality in Software Development
Osamu Mizuno†, Eijiro Shigematsu†, Yasunari Takagi†‡ , and Tohru Kikuno††
Graduated School of Information Science and Technology, Osaka University, Japan.
E-mail: {o-mizuno,kikuno}@ist.osaka-u.ac.jp‡
OMRON Corporation, Japan.
E-mail: taka@biwa.kusatsu.omron.co.jp

Abstract
In the practical software development, software quality
is generally evaluated by the number of residual defects. To
keep the number of residual defects within a permissible
value, too much eﬀort is often assigned to software testing.
In this paper, we try to develop a statistical model to determine the amount of testing eﬀort which is needed to assure the field quality. The model explicitly includes design,
review, and test (including debug) activities.
Firstly, we construct a linear multiple regression modelthat can clarify the relationship among the number of residual defects and the eﬀorts assigned to design, review, andtest activities. We then confirm the applicability of themodel by statistical analysis using actual project data.
Next, we obtain an equation based on the model to determine the test eﬀort. As parameters in the equation, thepermissible number of residual defects, the design eﬀort,
and the review eﬀort are included. Then, the equation determines the test eﬀort that is needed to assure the permissible
residual defects. Finally, we conduct an experimental evaluation using actual project data and show the usefulness ofthe equation.

1 Introduction
The importance of the software quality has been increasing for the last decade. In order to measure the software
quality, many metrics and methodologies have been proposed up to now[5, 7, 14]. Among them, the number ofresidual defects is frequently used since it is easily understandable and deeply concerned with the needs in the software development organization [7].
On the other hand, various approaches have been proposed to decrease the number of residual defects [4, 7]. The
most direct approach is to perform enough testing. However, improving the overall development process is moredesirable to achieve high software quality [2, 6, 9]. For
instance, constructing rigid specification, introducing review activity, and determining feasible development plans
are such improvement activities.
In the actual industry, the software testing eﬀort has been
usually determined by using the past experience. In orderto improve such a situation, many studies have been proposed to estimate appropriate testing eﬀort. For instance,
the software reliability growth model(SRGM) is one of thebest-known approach to determine optimal release point[8].
However, since the SRGM uses the whole activity data fromthe beginning to the end of defect detection, it cannot be
used before the testing begins. On the other hand, Takahashi et al. showed some metrics and a linear correlation
model to predict the number of errors[12]. Though they estimated the number of errors successfully, their metrics aremainly product metrics. Moreover, their aim was not to estimate the testing eﬀort, but predict the number of residual
faults in software.
We present an estimating method of the test eﬀort to assure field quality. Here, assuring the field quality means tokeep the number of residual defects below the permissible
number determined a priori.
We focus on a timing when the development goes to theend of its coding activity. We thus assume that the eﬀorts
needed for design and review activities have already been
known, since these activities were over. Under such a condition, we construct a model to estimate the testing eﬀort
considering the field quality. The model makes it possible
to estimate the testing eﬀort dynamically before the test activities begin. Additionally, the testing eﬀort estimated by
the model is the one necessary to assure field quality. As aresult, we can expect even the improvement of productivity
for testing activities. Furthermore, since the model itself isquite simple, the developers can understand the model easily.
In order to establish the proposed method, we firstly de-velop a linear multiple regression model that explains thenumber of residual defects by a linear function of design,
review, and test eﬀorts. To do so, as a preliminary investigation, we statistically confirm that software development
eﬀorts such as design, review, and test eﬀorts have mucheﬀect on the number of residual defects by using actual
project data.
Next, we obtain an equation, based on the linear multiple regression model, to determine the test eﬀort. Here, we
assume that the number of permissible residual defects isgiven, and the design and the review eﬀorts are specified.
We then conduct an experimental evaluation using actual
project data and showed the usefulness of the equation.
The rest of this paper is organized as follows. Section
2 describes the preliminaries of this study, such as processmodel and software metrics. In Section 3, we explain an approach, which consists of three phases. Then, in Section 4,
we investigate the relationship between the eﬀort assigned
to each activity and the number of residual defects. Next in
Section 5, we construct a model that can clarify the relationship among the number of residual defects and the eﬀorts
assignment. In Section 6, we obtain an equation to determine the test eﬀorts that is needed to assure the permissible
residual defects and conduct an experimental evaluation using actual project data. Finally, Section 7 summarizes thispaper.

These projects can be considered very similar ones, since
their objective is to produce almost the same software systems to diﬀerent customers and thus their development process is almost the same.
In the target projects, many kinds of computer systems
with embedded software are developed mainly using C language. The products are developed under a development
process as shown in Figure 1. The development process isan ordinal waterfall model.
Figure 1. Development process
Concept design & review
Function design & review
Structure design & review
Module design & review
Coding & review

Unit test & debug
Integration test & debug

2 Preliminaries

Function test & debug

2.1 Process model

Verification test & debug

The projects targeted in this paper are the developmentof computer control systems with embedded software ina certain company. The systems are related to retail systems. Such embedded software implements rather complex
functions dealing with many sensors, actuators, and control
signals including various kinds of interrupts. Furthermore,since it is delivered in the form of LSI chips, modification of
defects after delivery is very expensive. Thus, high quality
is especially required for the embedded software.
We use the actual project data of 38 projects, which havealready finished their development. The characteristics ofthese projects are as follows:
1. The projects started their development in 1996 and1997.
2. The development eﬀorts of these 38 projects range
from 7 to 35 person-months. The average eﬀort isabout 14 person-months.
3. At least 15% of the total eﬀorts for design and coding
activities was assigned to review activities.

Design &Coding phase

Test &
Debug phase

The development process consists of two successive
phases, design & coding phase and test & debug phase. The
design & coding phase is divided into five stages: Concept
design, Function design, Structure design, Module design,and Coding. The test & debug phase is divided into fourstages: Unit test & debug, Integration test & debug, Function test & debug, and Verification test & debug.
One characteristic of the design & coding phase is that
review activity is introduced after each design activity. Review activity is an approach that enables the detection andcorrection of defects in software artifacts as soon as these
artifacts are created. The review activity not only improves
the quality of the artifacts but also helps software development organizations reduce their cost of producing software[1, 3]. In the review activity, the documents should be distributed to the persons concerned in the company, and thenreview results should be returned to developers via managers (this review activity is called peer review [1]). They
established several guidelines for the review activity. Oneof them suggests that at least 15% of the total eﬀorts for de-sign & coding phase should be assigned to review activities[11].
The test & debug phase consists of the repetition of a pairof test activity and debug activity. Test activity is the process of analyzing a software item to detect the diﬀerences
between existing and required conditions and to evaluate thefeatures of the software items. Debug activity is the process
to detect, locate, and correct faults [10]. Developers are
directed to record all defects that are detected by the test
activity and removed by the debug activity [13]. In order toimprove and assure the quality of the software product, theproduct needs to be tested suﬃciently.

A metric related to the software product is defined asfollows:

2.2 Software metrics

In each project, eﬀort data and defect data are recorded
manually and stored in workstations by each staﬀ member. Then, they are collected and summed up by the project
leader, and validated by the manager. Field defect data arereported by a quality assurance staﬀ member, translated into
a defect-based number by the project leader, and also validated by the manager. All the validated data are sent to thesoftware engineering process group (SEPG), who analyzes
it and reports back to the project team and development organization [11].
Tables 1 and 2 show the actual eﬀort data in 1996 and1997, respectively. Table 3 summarizes the average andthe standard deviation of E design , E review , and E test . Notethat these projects were selected according to some criteria(such as the product size, cost, and so on) defined in thecompany.

In this subsection, we define some software metrics[5]used in this paper. Five software process metrics related toeﬀort of development phases are defined as follows:(1) E design (design eﬀort) : the eﬀort spent on the design and coding activities in design & coding phase(person-day).
We define the design eﬀort as the sum of the eﬀort
spent on the design and coding activities.
(2) E review (review eﬀort) : the eﬀort spent on the review
activities in design & coding phase (person-day).
We define the review eﬀort as the sum of the eﬀort
spent on each review activity.
(3) E test (test eﬀort) : the eﬀort spent on the test and debug
activities in test & debug phase (person-day).
We define the test eﬀort as the sum of the eﬀort spenton test activity and the eﬀort spent on debug activity.
(4) Rdesign/total (ratio of the design eﬀort to the total development eﬀort):(6) D f ield (number of field defects) : the number of field
defects after six months after delivery. Here, a field
defect means a failure found in the customer’s site.
We define D f ield as a measure of software quality. Although the number of defects is not a suﬃcient metricfor software quality, D f ield has been considered to be aquality metric in this company.

2.3 Actual project data

Table 1. Actual efforts of the projects in 1996
No.
E design Ereview
Etest
Y96-1
81.0
22.9
73.1
Y96-2
122.4
28.6 190.0
..
..
..
..
.
.
.
.
Y96-19

428.0

86.0

130.0

EdesignRdesign/total =Edesign + Ereview + Etest
This metric Rdesign/total stands for the ratio of the design
eﬀort to the sum of the eﬀorts assigned to the design
activity, the review activity, and the test & debug activity.
(5) Rtest/total (ratio of the test eﬀort to the total development
eﬀort):

Table 2. Actual efforts of the projects in 1997
No.
E design Ereview
Etest
Y97-1
138.5
32.5
77.9
Y97-2
81.2
22.5 128.3
..
..
..
..
.
.
.
.
Y97-19

150.0

28.0

43.0

Etest
Rtest/total =Edesign + Ereview + Etest
This metric Rtest/total stands for the ratio of the test effort to the sum of the eﬀorts assigned to the design
activity, the review activity, and the test & debug activity.

3 An Approach to Estimation
In this paper, we suggest a timing when the design &coding phase in Figure 1 is finished. We thus assume that

Table 3. Fundamental statistics for E design ,Ereview , and E test
Variable Average Std. Dev.
Edesign120
77.0
Ereview
28
17.5
Etest
108
58.7

the eﬀorts needed for design and review activities have already been known. Under such a condition, we construct amodel to estimate the testing eﬀort needed to assure the permissible number of field defects. Straightforwardly speaking, we estimate E test from E design and E review according to
the field quality.
In order to establish the method, we perform the following three phases:

during the design and coding activities. On the contrary,
those defects are detected and removed by review, test, anddebug activities. The remaining defects are delivered to customers as field defects.

Figure 2. Defects injected and removed process

Defects injected by design and coding activities

Defects removed
by review activities

Phase 1: (Investigating the eﬀorts’ eﬀect on the number offield defects.)
We clarify the relationship between the eﬀort assigned
to each activity and the number of field defects. Additionally, we prove the correctness of the correlation by
the statistical tests.
Phase 2: (Construction of a regression model for the number of field defects.)
According to the result obtained in Phase 1, we construct a multiple regression model. This model clarifies the multivariate relationship between eﬀorts for
activities and the number of field defects. Then, we
prove the correctness of the model by the statistical
test.
Phase 3: (Estimation of testing eﬀort to assure field quality.)
By transforming the regression model, we construct anequation to calculate the testing eﬀorts when the design and review eﬀorts, and the permissible number offield defects are given.
For the verification of Phase 1 and 2, we use the data ofprojects performed in 1996 and 1997 (that is, all projects
shown in Tables 1 and 2). But in Phase 3, we apply the dataof projects in 1996 shown in Table 1 for model construction,and for an experimental application of the model, we apply
the data of 1997 shown in Table 2.

4 Investigating Eﬀects of Eﬀorts (Phase 1)

Defects removed
by test & debug activities

Design &Codingphase

Test &
Debug
phase

Field defects

In this scheme, we can use three kinds of eﬀort data
Edesign , E review , and E test and a field defect data D f ield . In
the next subsection, we analyze their relationship.

4.2 Correlation between metrics
Firstly, we analyze the eﬀect of software development
eﬀorts such as design, review, and test eﬀorts on the numberof field defects. We make the following three hypotheseswith respect to the relationship between the number of field
defects and eﬀort assignment.
H1 : An increase in the eﬀort assigned to the design and thecoding activities increases the number of field defects.

4.1 Scheme of the proposed method

H2 : An increase in the eﬀort assigned to the review activity
decreases the number of field defects.

Figure 2 shows a scheme for defect injection and defectremoval. In this paper, we assume that defects are injected

H3 : An increase in the eﬀort assigned to the test and the
debug activities decreases the number of field defects.

4.2.1 D f ield and Rdesign/total
We investigate the correlation coeﬃcient between E design
and D f ield to confirm the hypothesis H 1 . Since the date ofdelivery is promised and the developers have to finish the
project within a given duration in the target company, theincrease in the eﬀort assigned to the design activity can decrease the eﬀorts assigned to the review activity and test &debug activity. Therefore, we analyzed the relationship between the ratio of eﬀort assigned to the design activity andthe number of field defects.
The value of Spearman’s rank correlation coeﬃcient between Rdesign/total and D f ield is 0.535 (p < 0.01). It implies
that the correlation between R design/total and D f ield is rather
strong. We thus confirm the hypothesis H 1 .
4.2.2 D f ield and Rreview/design
The review process improvement activity was established in
the company and the software quality changes caused by thereview improvement were evaluated [9, 11]. For the target
projects in this paper, we consider the relationship between
the ratio of eﬀort assigned to the review activity and thenumber of field defects in order to confirm the hypothesis
H2 .
We analyzed the value of Spearman’s rank correlationcoeﬃcient between R review/design and D f ield . The coeﬃcientis −0.385 (p < 0.05). It implies that there is a certain degree of correlation between R review/design and D f ield , since
the value of correlation coeﬃcient is not so large.
4.2.3 D f ield and Rtest/total

in subsection 4.2, among the number of field defects andthe eﬀorts assigned to design, review, and test activities.
D f ield = b0 Edesign − b1 Ereview − b2 Etest(1)(b0 > 0, b1 > 0, b2 > 0)
Since the design activities can be viewed as the process
in which defects are injected, the regression coeﬃcient of
Edesign is plus. On the contrary, since the review activities
and the test & debug activities can be viewed as the process
in which defects are removed, both regression coeﬃcientsof E review and E test are minus.

5.2 Estimation of coefficients
By applying the ordinary least squares method to the 38
project data in Tables 1 and 2, we estimated the parametersin Equation (1). The result of estimation is summarized inTable 4.
Table 4. Estimated values of parameters
Independent
variable
Edesign
Ereview
Etest

Estimatedcoeﬃcient
0.045
0.091
0.012

Standard
coeﬃcient
1.29
0.59
0.27

p-value< 0.0001
0.048
0.004

The estimated coeﬃcients are as follows:b0 = 0.045, b1 = 0.091, b2 = 0.012

We consider the relationship between the ratio of eﬀort assigned to the test activity and the number of field defects inorder to confirm the hypothesis H 3 .
The value of correlation coeﬃcient between R test/total and
D f ield is −0.491 (p < 0.01). It implies that the negative correlation between R test/total and D f ield is rather strong.
This result shows that the increase in the eﬀort assigned
to the test activity is still related to the decrease in the number of field defects. Therefore, it is valuable to analyze
the relationship between eﬀorts assigned and the number offield defects, and determine the test eﬀort needed to assure
permissible software quality.

It is shown that signs of b 0 , b1 , and b2 are plus. We thus
confirm that each of these three coeﬃcients is significant
for Equation (1).
Coeﬃcient of determination (R 2 ) is a measure of how
well the model explains the variance in the dependent variable, and it takes a value from 0 to 1. Statistical analysis
revealed R2 of 0.722, which implies that the model explains
72.2 percent of the variation observed in D f ield . When R2 ismore than 0.4, we can generally consider the value as highcoeﬃcient. Since the value of R 2 is very high, we confirm
that the constructed Equation (1) explains the variance ofthe number of field defects very well.

5 Construction of Model (Phase 2)

6 Estimation of Test Eﬀort (Phase 3)

5.1 Regression model for D f ield

6.1 Proposed method

In order to determine the amount of testing eﬀort which
is needed to assure the field quality, we construct a linear multiple regression model, described by Equation (1).
Equation (1) can clarify the relationship, which is derived

From Equation (1), we obtain the following Equation(2):
Etest = (D f ield − b0 Edesign + b1 Ereview )/(−b2 )(2)Equation (2) expresses the test eﬀort (E test ) as a functionof the number of field defects (D f ield ), the design eﬀort(E design), and the review eﬀort (E review ).
We propose a method for determining the test eﬀort
needed to assure permissible software quality by using
Equation (2) and past project data. It consists of the following four steps:
Step 1: Calculation of b 0 , b1 , and b2 in (2).
In this step, past project data are prepared as training
data for the regression model. Then, b 0 , b1 , and b2
are estimated using the ordinary least squares method.
The coeﬃcients are tested statistically.
Step 2: Assignment of a given constant to D f ield in (2).
In this step, we determine D f ield based on the requirements for the product or the development plan so thatthe quality of the product can be assured. We call thedetermined D f ield as the permissible number of field
defects and denote it by D̂ f ield .
Step 3: Assignment of actual values to E desgin and E review
in (2).
In this step, the design eﬀort and the review eﬀort of
the target project are set as E design and E review respectively. According to the progress of a project, E design
and E review are changed in the following way, if necessary.
• If the design & coding phase has been completed,then the actual data of the design eﬀort and thereview eﬀort are E design and E review .
• If the design & coding phase has not been
completed or the actual data are not measured,then estimated values are assigned to E design andEreview .
Step 4: Calculation of E test by (2).
We calculate the value of E test by Equation (2). We
represent this calculated value by Êtest and call the test
eﬀort needed to assure the permissible software quality. Then, Êtest is defined by the following Equation(3).
Êtest = (D̂ f ield − b0 Edesign + b1 Ereview )/(−b2)(3)

As mentioned before, in this paper, assuring the field
quality means to keep the number of field defects
below the permissible number determined a priori.
Therefore, if the test eﬀort of the project is more thanÊtest , then the number of field defects is expected to beless than D̂ f ield .

6.2 Experimental evaluation
In Step 1, we calculated the parameters b 0 , b1 , and b2 inmodel (1) by using project data in 1996. The values of E test ,Edesign , and E review in 1996 are shown in Table 1. Here, wecannot publish the actual values of D f ield , since these values
are confidential.
The coeﬃcients calculated were as follows:b0 = 0.039, b1 = 0.069, b2 = 0.011
Since the coeﬃcient of determination (R 2 ) is 0.743, we cansay that the estimated coeﬃcients explain the small variancein the number of field defects very well.
In Step 2, we determined the permissible number of field
defects. As the company intends to make lower the number of field defects as much as possible, we determined thetest eﬀort to keep the number of field defects less than 1.
Therefore, we set the permissible number of field defects
D̂ f ield = 1.
As a result, we constructed Equation (4) from Equation(3).
Êtest = (1 − 0.039E design + 0.069E review)/(−0.011)(4)

In Step 3, we determined to assign the actual values ofthe design and the review eﬀorts of each project in 1997
(shown in Table 2) to E design and E review in Equation (4),respectively.
Then in Step 4, we were able to calculate Êtest for the 19
projects in 1997 from Equation (4). The calculated valuesof Êtest are shown in Table 5.
In this table, Êtest represents the test eﬀort calculated byproposed method. “Comparison” shows the comparison result of actual E test (please note that actual E test is given in
Table 2) and Êtest . In this column, “” indicates that E test ismore than Êtest , and “×” indicates that E test is less than Êtest .
Again, please note that projects in Tables 1 and 2 were obtained according to some criteria defined in the company.
By summarizing Table 5, we classified the projects according to the following two conditions:C1 : E test is more than Êtest .
C2 : No field defect was found.
The projects were classified as shown in Table 6. Table
6 provides the following two observations:• There exist eight projects in which the test eﬀort ismore than calculated Êtest . For seven projects out of
the eight projects, no field defect is detected.
• There exists eleven projects in which the test eﬀort isless than calculated Êtest . For eight projects out of theeleven projects, some field defects are detected.

7 Conclusion

Table 5. Comparison of Êtest
No.
Êtest Comparison
Y97-1
196.3
×
Y97-2
55.8

Y97-3
73.1
×
Y97-4
85.6
×
Y97-5
29.5

Y97-6
4.0

Y97-7
43.5

Y97-8
40.2

Y97-9
79.7

Y97-1059.2
Y97-11 198.4
×Y97-1242.5
Y97-13 190.0
×Y97-14 251.3
×Y97-15 105.7
×Y97-16 124.6
×Y97-17 145.3
×Y97-18 440.1
×Y97-19 265.3
×

We proposed a method to determine eﬀorts that should
be assigned to the test & debug phase to assure the permissible software quality. To do so, firstly, we constructed a linear multiple regression model that explains the relationshipbetween the number of field defects and the development
eﬀorts. Several statistical analyses using the actual project
data confirmed that the constructed model nicely explains
the data of target projects. Then, in order to calculate thetest eﬀort needed to assure permissible quality, we got an
equation by transforming the constructed regression model.
As a result of experimental evaluation using actual
project data, we confirmed that the following two results:(1) As for projects having larger test eﬀort than calculated,
no field defect was detected, and (2) As for projects having test eﬀort less than calculated eﬀort, some field defects
remained.
However, this result only shows that our proposed
method is applicable to the targeted projects in a certain
company. To generalize it, we need to apply the proposed
method to more data of other software projects. It is one ofour most important future works.

References
Table 6. Classification of experimental results
Testingeﬀort Etest

More than Êtest
Less than Êtest
Total

Field defects D f ield
not found
found
7
1
3
8
10
9

Total
8
11
19

[1] D. B. Bisant and J. R. Lyle. A two-person inspection
method to improve programming productivity. IEEE
Trans. on Software Engineering, 15(10):1294–1304,
1989.
[2] M. Diaz and J. Sligo. How software process improvement helped motorola. IEEE Software, 14(5):75–81,
1997.

In order to confirm the statistical significance of these observations, we analyzed the relationship between C 1 and C 2
by using Fisher’s exact probability test. Fisher’s exact probability test calculates an exact probability value for the relationship between two dichotomous variables, as found in a
two by two cross table like Table 6. For Table 6, the Fisher’s
test would determine whether the two groups, E test is morethan Êtest or less than Êtest , diﬀer significantly in the proportion of “not found” and “found”. The null hypothesis isthat C1 and C 2 are independent. The level of significance ischosen as 0.05. The calculated p-value of Fisher’s exact test
for the table becomes 0.0198. Then, the null hypothesis isrejected at the 0.05 level.
As a result, we can say that there are some correlation
between the estimated test eﬀorts and field defects. That is,the calculated testing eﬀorts Êtest can be a threshold of thetesting eﬀorts to keep the number of field defects less thanD̂ f ield = 1.

[3] M. E. Fagan. Advances in software inspections.
IEEE Trans. on Software Engineering, 12(7):744–
751, 1986.
[4] N. E. Fenton and M. Neil. A critique of software defect prediction models. IEEE Trans. on Software Engineering, 25(5):675–689, 1999.
[5] N. E. Fenton and S. L. Pfleeger. Software Metrics :A Rigorous & Practical Approach. PWS Publishing,
1997.
[6] W. S. Humphrey. A Discipline for Software Engineering. Addison-Wesley, MA, 1995.
[7] M. S. Krishnan and M. I. Kellner. Measuring process consistency: Implications for reducing software defects. IEEE Trans. on Software Engineering,25(6):800–815, 1999.

[8] J. D. Musa, A. Iannino, and K. Okumoto. Software reliability: measurement, prediction, application. McGraw-Hill, 1987.
[9] K. Sakamoto. A study of software process improvement and quality control based on analyses of actual project data. PhD thesis, Department of Information Systems, Graduate School of Information Science, Nara Institute of Science and Technology, 2000.
NAIST-IS-DT9861009.
[10] Standards Coordinating Committee of the IEEE Computer Society. IEEE Std 610.12-1990. The Institute ofElectrical and Electronics Engineers, Inc., 1990.
[11] Y. Takagi, T. Tanaka, N. Niihara, K. Sakamoto,
S. Kusumoto, and T. Kikuno. Analysis of review’s
eﬀectiveness based on software metrics. In Proc. of
5th International Symposium on Software Reliability
Engineering, pages 34–39, 1995.
[12] M. Takahashi and Y. Kamayachi. An empirical studyof a model for program error prediction. In Proc. of
8th International Conference on Software Engineering, pages 330–336, 1985.
[13] T. Tanaka, K. Sakamoto, S. Kusumoto, K. Matsumoto,and T. Kikuno. Improvement of software process byprocess description and benefit estimation. In Proc. of
17th International Conference on Software Engineering, pages 123–132, 1995.
[14] S. Yamada and M. Takahashi. Introduction to software
management model. Kyoritsu books (in Japanese),
1993.