Empirical Evaluation of Review Process Improvement Activities
with respect to Post-Release Failure
Osamu Mizuno and Tohru Kikuno
Graduate School of Engineering Science, Osaka University
o-mizuno@ics.es.osaka-u.ac.jp

Abstract
In this paper, we analyze empirically software qualitychanges caused by long-term improvement activities for software review process in a certain company.
The objective of our position paper is to present following
three points: (1) the review process improvement activity introduced in a company, (2) confirmation that review improvement is fixed in the company (3) evaluation of the effects withrespect to the software quality changes caused by the review
improvement.
Generally, the effectiveness of review process is evaluatedfrom the viewpoint of high productivity of the software process. But truely speaking, customers really concern about
the quality of final software product rather than productivity.
Thus we analyze the quality of the final product(post-release
failure), and show that the effectiveness of review process
improvement acticities based on the analysis result.
1 Process Improvement
1.1 Overview
In the company to be investigated in this paper, the software engineering process group(SEPG) was established in
1992. Since then the SEPG has conducted two main process
improvement activities as follows: establishment and introduction of the standards for managing the software process,and improvement of the review process using the software
metrics toward high quality.
At first, the SEPG tried to establish several standards for
managing software project, and to put it into practice from1993. The main purposes of the standards include guiding
developers in each project to create the well-formed plan andmanagers to conduct the project successfully according to awell-formed plan. It is believed (without any proof) in thecompany that the well-formed plan will derive the accurate
estimation for the project cost. Furthermore, it is also believed that the accurate cost estimation will lead the projectto the high quality of the product and high productivity ofdevelopment team. We have already applied the statisticalanalysis to this improvement activity and shown the correctness of these facts by the test of statistical hypothesis(t-test)with a level of significance 5% in [5].
Next, the SEPG has extensively engaged in the improvementof review process. Generally, the cost of removing faults inthe later development phases, such as debug or test phasebecomes higher than that in the earlier phase such as design
review or coding review phase. So it is strongly recommended to remove faults in the earlier phase. Concerning thisfact, the number of faults detected in the review activity is
believed to be greater by increasing the amount of review
activity. Based on the similar experience and knowledge as
mentioned above, the SEPG started the improvement of review process in 1995. The key point of the improvement
is implementing effective review process by increasing theamount of efforts for review(especially, code review) and by
introducing good guidelines. We will discuss only the second
review process improvement in this paper.
1.2 Review process improvement
As mentioned before, reviews include the document review
in the design activity and the code review in the coding activity. Generally, it is difficult to derive concrete guidelines
or numerical target values for the document review[2, 4]. On
the other hand, it is relatively easy to derive them for the code
review. The situation is also true for the SEPG’s activities,
as shown in the past analysis result of the review’s effect[7].
Based on the analysis result of the past project data, the SEPG
has derived the following guidelines G1–G6 for the review
activities:

G1 At least the 15% of the total efforts for design and coding
activities should be assigned to reviews(the document
review and the code review).
G2 Reviewers must report the progress using the standard
review form at regular intervals.
G3 In the design review, the documents should be distributed to all the experts in the company, and then review results should be returned to developers via manager(Thisdesign review is called peer review[6]).
G4 The coding review should be performed by two or threepeople, including one person who wrote the code.
G5 The review coverage rate for the code review should be
about 200 lines of code per hour.
G6 All new codes and changed codes should be reviewed.
(Concerning reuse of old codes, reviews are not necessary required.)Among the guidelines above, G1 and G2 are general requirements for reviews(including both the document review
and the code review), G3 is specific for the design review,
and G4, G5 and G6 are only for the code review. As for the
G1, it is difficult to determine the criteria for controlling thereview activity.
For example, the number of residual faults estimated by thecapture-recapture model is used in [1] as a criteria. However,taking our previous work[7] into consideration, we adopt thenumerical value of the review effort as a criteria. The criteria
15% of total effort is also shown in [3].
Before the review process improvement began, the same activities as the guideline G2, G3 and G4 were required to be
performed in the review. However, since the guidelines were
not yet established, the review did not work effectively in thepractical developments. Actually the average review effort
was less than 10% of the total design efforts. This value 10%
is not sufficient according to the experience in the company.
Then, the SEPG started the improvement of review process
in 1995 according to these guidelines. Intuitively speaking,recently these guidelines have been followed in the company
and no serious failure reports reach to the SEPG. However
any formal or statistical discussions on the efforts by thereview process improvement are not yet done. So, we try toanalyze empirically its effectiveness using actual 23 projectdata.
2 Fixing Process Improvement
2.1 Software metrics and assertion
Figure 1 shows a simplified process model and a part offundamental data set collected at each phase of development.

The metric review =design stands for the ratio of all review
effort to design and coding efforts.
We confirm that the review improvement activities are accepted in the development teams. In other words, we investigate
the direct changes in the software development process. In
more detail, we analyze the changes in the amounts of theeffort for review activities. As mentioned in subsection 1.2,
the guideline G1 requires at least 15% of the total efforts ondesign and coding activities should be spent on review. Thus,
we can expect that the effort of review activity is increasedin the projects guided by the SEPG as the direct effect by
the guideline G1. Based on these considerations, we try toprove the following assertion A1 by statistical analysis:

A1 The ratio of effort for review on the total efforts fordesign and coding activities increases in the projects
guided by the SEPG.
2.2 Ratio of review effort
First, concerning the assertion A1, we try to investigate how
the review improvement activities by the SEPG are acceptedby the organizations. Here in order to evaluate the process
improvement activities, we focus on the project groups rather
than individual projects. There exist three project groups:
Vending System, Checking System and Retail System.
As mentioned in Section 1, the guideline G1 recommends
that review =design (the ratio of review effort to effort in design phase) should be greater than 15%. According to thisguideline, we define a project with review =design  15%
to be a faithful project. Table 1 shows the mean values ofsoftware metric review =design for three project groups.
Table 1: Comparison of ratio of review effort

Design Phase

Debug & Test Phase

Monitoring Phase

Vending system Checking system(1992-1994)(1992-1996)- Efforts neededfor design phase- Efforts needed fordebug & test phase- No. of faults detected
after shipping- No. of faults detectedby review- No. of faults detectedby debug & test

- Seriousnessof failures detected

Figure 1: Fundamental data set
As for the efforts spent in the development process, the SEPG
collects data of several fundamental metrics. The metrics
Edesign and Ereview represent the total efforts spent for all
activities in design phase and all review activities in designphase, respectively. From the debug & test phase, the SEPG
collects data of fundamental metric Etest . The metric Etest
represents the total efforts spent for all activities in the debug& test phase.
In order to evaluate the amount of efforts, we define a metricas follows:
review =design

Ereview
 100
design + Ereview=Eg review/design

8.9%

11.8%

Retail system(1995-1996)

20.6%

In Table 1, the project group Retail System seems to succeed in following the guideline G1 faithfully. However,the project group Checking System seems to fail to follow the guideline G1. Thus, we execute the test of statistical hypothesis(t-test) with 5% level of significance toreview =design ’s of Retail System and Checking System. As
a result, we can prove that there exists a significant difference
between them.
We discuss the characteristics of organizations to show thereason why there exists a distinct difference between Retail
System and Checking System. The project group Checking
System started in 1992 and the members of the organization for Checking System had already established their own
ways for software development when the SEPG started thereview process improvement. Thus it is hard for most of themto change the process instantly according to the guidelinesspecified by the SEPG group. On the contrary, the projectgroup Retail System started after the SEPG had determined
the guidelines. Thus the members of the organization for Retail System tend to accept the guideline without difficulties.
As mentioned above, we should discuss the properties oforganizations rather than that of individual projects. Thus forconvenience we define a set of faithful projects as a faithfulproject group. According to this definition, we refer
Retail System (1995–1996): faithful project group
Checking System (1992–1996): unfaithful project groupin the following. To tell the truth, Checking System includes
four faithful projects. But, it is also clear the organization thatdeveloped Checking System failed to follow the guideline
G1. Thus we interpret that the organization happened to 15% for some projects in Checking
have the result
System, and assume that Checking System is an unfaithfulproject group. For convenience, we also refer
Vending System (1992–1994): unfaithful project groupsince this project group contains only unfaithful projects.
3 Effectiveness Analysis
3.1 Software metrics and assertion
At first, we explain the fundamental metrics related to thequality collected in the development process shown in Figure 1. The metric Freview represents the total number offaults detected by all review activities in design phase. The
metric Ftest represents the total number of faults detected in
the debug & test phase. During six months after the codeshipping, the SEPG provides the monitoring phase and collects all data concerning the failures detected by customers,
as shown in Figure 1. We call these failures post-released
failures, and use the metric Fmonitor to represent the numberof post-released failures.
Using these data collected from the projects, we define software metrics  and  to evaluate the quality of software
product.
(1) Ratio of detected faults (’s: %)In order to evaluate the ratio of detected faults in a
specific phase to all the faults detected, we define three
metrics as follows:review =total =test =total =Freview
 100
Freview + Ftest + Fmonitor

Freview
 100
Freview + Ftest + Fmonitor

monitor =total =Freview
 100
Freview + Ftest + Fmonitor

The metric review =total stands for the ratio of faults detected in the review to all faults detected, and test =total
stands for the ratio of faults detected in the debug & test
phases. Finally, monitor =total stands for the ratio of
post-release failure detected in the monitor phase.
(2) Seriousness of failure ( : level)
For each post-release failure, the maintenance operator
and the SEPG jointly decide the seriousness . The
values of seriousness  are as follows: destructive,
confusing and mild. The level  =destructive meansthat the failure can lead to the system down. Thus thefailure must be removed immediately. Then the level
 =confusing means that only a part of system may
be down by the failure and other parts may keep working. Thus it should be removed immediately if possible.
Finally the level  =mild means that the failure neveraffects the essential part of the system, and thus it may
be negligible for a while.
We evaluate the effectiveness of review process improvement
quantitatively. Thus, we investigate the changes in the number of detected faults and the changes in the quality of the
final product. They are the indirect effects, but are the mostessentially expected effects of the review process improvement.
We try to prove the following assertions A2 and A3 bystatistical analysis:

A2 The number of faults detected by the review increases
in each project of the faithful project group. Similarly,
the number of faults detected in the debug & test phasedecreases.
A3 As a result of the review process improvement, the quality of the final code is also improved.
3.2 Ratio of detected faults 
Now we investigate the effect of review process improvement
concerning the assertion A2. Table 2 shows the mean valuesof software metrics review =total (the ratio of faults detected
in review phase to all the faults detected), test =total (theratio of faults detected in debug & test phase to all the faults
detected) and monitor =total (the ratio of post-release failures
to all the faults). In Table 2, we classify the projects into
faithful project group (that is, Retail System) and unfaithfulproject group(Vending System and Checking System).
By the test of statistical hypothesis with 5% level of significance, all of the values review =total and test =total confirmed the significant difference between the faithful projectgroup and the unfaithful project group. Thus, we can say
the correctness of the assertion A2 is confirmed by statisticalanalysis.
However, the values of monitor =total shown in Table 2 seems
to indicate that there are no difference between the ratio ofTable 2: Comparison of ratio of detected faults Faithful
project group

Unfaithfulproject grouprreview/total

78.4%

38.8%

rtest/total

21.1%

60.7%

rmonitor/total

0.5%

0.5%

post-release failures in faithful and unfaithful projects. Then
we have to perform another analysis to investigate the postrelease failure.
3.3 Post-release failure
As mentioned before, for each post-release failure the SEPG
or the maintenance operator decide the seriousness  andassign its value to the failure. Table 3 summarizes the total
number of post-release failures and the distributions of ’s
assigned to post-release failures. Because the values of these
metrics are confidential, we cannot publish the values themselves in this paper. Thus Table 3 shows only the relativevalues by assuming all the values for the unfaithful projectgroup to be one.
Table 3: Comparison of post-release failures
Faithful
project group

Unfaithfulproject group

0.55

1

destructive

0.44

1

confusing

0.50

1

mild

0

1

Fmonitor# of projectsc

Since the original values of these metrics are so small, we
cannot apply the statistical analysis. However, in Table 3
we can observe some interesting properties. The total number of post-release failures(Fmonitor /# of projects) of thefaithful project group is smaller than that of the unfaithfulproject group. Especially, the number of the failures with
 =destructive is smaller drastically. Thus we can guess thecorrectness of the assertion A3.
4 Conclusion
We have analyzed the effectiveness of the review process
improvement activities by the SEPG during these six years ina certain company. According to the guidelines determinedby the SEPG, we have investigated the ratio of the review
effort to the total effort for design and coding activities. As
a result we found that a newly started project group followed
the guidelines faithfully. Similarly, we have investigated
the ratio  of faults detected in review to the total numberof detected faults. The result showed that the ratio  is
improved drastically in a faithful project group. Finally, we
have confirmed the number of post-release failures during six
months after code release is also decreased by the SEPG’sprocess improvement activities. Furthermore, we confirmed
the number of critical failures are also decreased.

Acknowledgements
Authors would like to thank Dr. Shinji Kusumoto of Osaka
University and Mr. Naoki Niihara, Mr. Yasunari Takagi
and Mr. Keishi Sakamoto of OMRON Corporation for their
discussions and advice to our analysis in this paper.
REFERENCES[1] L. C. Briand, K. E. Emam, B. Freimut and O. Laitenberger: “Quantitative evaluation of capture-recapture
models to control software inspections,” Proc. of 8th
Int’l Symposium on Software Reliability Engineering,
pp.234–244, 1997.
[2] R. G. Ebenau and S. H. Strauss: Software Inspection
Process, McGraw-Hill, 1993.
[3] M. E. Fagan: “Advances in software inspections,”
IEEE Trans. on Software Engineering, Vol.12, No.7,
pp.744–751, 1986.
[4] W. S. Humphrey: Managing the Software Process,
Addison Wesley, Reading, MA, 1989.
[5] O. Mizuno, T. Kikuno, K. Inagaki, Y. Takagi and
K. Sakamoto: “Analyzing effects of cost estimation
accuracy on quality and productivity,” Proc. of 20thInt’l Conference on Software Engineering(ICSE’98),
pp.410–419, 1998.
[6] M. C. Paulk, C. V. Weber, S. M. Garcia, M. B. Chrissis
and M. Bush: “Key practice of the capability maturity
model, version 1.1,” Technical Report CMU/SEI-93TR-25, Software Engineering Institute, 1993.
[7] Y. Takagi, T. Tanaka, N. Niihara, K. Sakamoto, S.
Kusumoto and T. Kikuno: “Analysis of review’s effectiveness based on software metrics,” Proc. of 6th
Int’l Symposium on Software Reliability Engineering(ISSRE’95), pp.34–39, 1995.