ある企業におけるソフトウェアプロセス改善の効果に対する統計的分析
水野 修y，二木 俊樹z，新原 直樹z，高木 徳生z，菊野 亨yy大阪大学 大学院基礎工学研究科 情報数理系専攻〒 560-8531 大阪府豊中市待兼山町 1-3

Phone: 06-850-6567 Fax: 06-850-6569
e-mail: o-mizuno@ics.es.osaka-u.ac.jp
zオムロン株式会社

あらまし : 本研究ではある企業におけるレビュープロセス改善活動の効果について統計的分析を行う．この企業
では 1995 年よりソフトウェアプロセスグループ (SEPG) を中心にこの改善活動が進められてきており，分析に当たっては 23 件のプロジェクトから収集されたデータを用いる．まず，各プロジェクトにおいてレビュー作業工数の全体工数に対する比率に注目して，開発組織が SEPG からのプロセス改善の指示をどの程度忠実に実施できたかを分析する．次に，レビュープロセス改善の品質への影響について調べる．その結果，プロセス改善を忠実に実施できた組織とそうでない組織の間ではレビュー作業で検出するフォールト数に有意水準 5%の検定で差があることが示された．更に，出荷後のいわゆるフィールド 品質についても相当の改善が見られることが確認できた．
キーワード : ソフトウェアプロセス，レビュープロセス改善，ソフトウェア品質，統計的分析

E ectiveness Analysis of Review Process Improvement for Embedded
Software System Development at Certain Company

Osamu Mizunoy, Toshiki Nikiz, Naoki Niiharaz, Yasunari Takagizand Tohru Kikunoy
Department of Informatics and Mathematical Sciences,
Graduate School of Engineering Science, Osaka University
1-3 Machikaneyama, Toyonaka, Osaka 560-8531, Japan
Phone: +81-6-850-6567 Fax: +81-6-850-6569
e-mail : o-mizuno@ics.es.osaka-u.ac.jp
zOMRON CorporationyAbstract: In this paper, we analyze the e ectiveness of actual review process improvement activities conducted by the software engineering process group(SEPG) in a certain company that develops embedded softwaresystems.
The analysis consists of two steps: investigating the ratio of the e orts for review on the total e orts fordesign and coding(Step 1), and clarifying the e ects caused by the review process improvement(Step 2). The
Step 1 classi es projects into two categories: faithful and unfaithful projects, where  15% holds for faithfulprojects. Then Step 2 proves statistically that the number of faults detected during the review process in faithfulprojects is signi cantly greater than that of unfaithful projects. Additionally, it also shows that the number ofserious failures in faithful projects is slightly smaller compared with unfaithful projects.
Keywords: software process, review process improvement, quality of product, statistical analysis

1 Introduction
In order to achieve high quality and productivity ofthe software development, a lot of general knowledge
or experiences such as “no silver bullet[3]” and “deathmarch project[18]” have been presented. The process
improvement[7] is well known as one of the most attractive and practical methodologies. Concerning the process improvement, theoretical investigations on the formal
framework[4, 15] as well as applications to actual organizations [8, 17] have been performed. Especially, the process improvement of software review has been considered
as a cost-effective way to improve quality of software[5].
Actually, several quantitative evaluations of software review process have proved its validity using simulation
results[2, 9].
In this paper, we analyze the effectiveness of review process improvement activities during six years in a certain
company that develops embedded software systems. In
the company, software engineering process group(SEPG)
was established in 1992, then the activities for constructing well-formed project plans were introduced in 1993,
and finally the activities for improving review process
have been conducted since 1995. Concerning the casestudies of software process improvement using generalized stochastic Petri-net model and the statistical analysisof the relationship between the well-formed project planand the final product quality, we have already summarizedsome reports[11, 16, 17]. But the discussions on review
process improvement are not yet done, and so we try toanalyze empirically the effectiveness of review process
improvement using actual twenty three project data from1992 to 1996. In this study, we assume review includes
both the document review in the design activity and thecode review in the coding activity.
The objective of this study is to investigate carefully thelong-term changes in the software development process
which are introduced by the review process improvementin the company. In this paper, we try to investigate thefollowing two changes on the projects: changes in review
effort and changes in software quality. At first, we trace
direct effects on the development process caused by thereview process improvement, such as the change of review
efforts actually spent for each project, and checks if theprocess improvement is truly incorporate into the actual
development. Next, we trace indirect effects caused by
the review process improvement, such as the changes in
the number of faults detected in the review and test phases,
and verify if the quality of software products is essentially
improved.
The rest of the paper is organized as follows: In Section 2, we present target projects and process model. In
Section 3 we explain the process improvement performed
from 1992 to 1998 at a certain company. In Section 4, weshow the outline of collected data and clarify the objectiveof our study by presenting three assertions. In Section 5,
we show the result of evaluations with respect to review
effort and software quality. Finally, we conclude this pa-

per with a brief summary and some future research works
in Section 6.

2

Development at Certain Company

2.1

Target projects

In order to derive as general results as possible, we select23 projects such that development effort of each projectis larger than 15 person-months. These selected projects
are classified into three kinds of categories based on theirproducts: Vending System, Checking System, and Retail
System as follows:
Vending System: The developments of user-friendly
vending machines, that accept cash or prepaid cards, for the railroad system. All of the selected fiveprojects began in the period from 1992 to 1994. Typically, it takes from 15.3 to 37.2 person-months fordeveloping the vending machine with 9.4 to 38.8 Kstep.
Checking System: The developments of checking machines for the fare in the train stations and airports.
All of the selected twelve projects began in the period
from 1992 to 1996. Typically, it takes 16.3 to 62.3
person-months for developing the checking machinewith 7.2 to 123.1 Kstep.
Retail System: The developments of retail systems allowing consumers to use credit and prepaid cards.
All of the projects began in the period from 1995 to
1996. Typically, it takes 15.0 to 29.2 person-months
for developing retail system with 3.5 to 20.5 Kstep.
The one important point for later analysis is that theVending System and Checking System are developed by
the same organization in the company. On the other hand,
the Retail System is developed by another organization.
Figure 1 summarizes the categories of 23 projects andperiod of their developments. Please note that activitiesof the SEPG for improving review process have been conducted since 1995, to be described in Section 3.
Vending System(5 projects)

Checking System(12 projects)
Retail System(6 projects)
'92

'93

'94

'95

'96

'97

'98

Figure 1: Target projects

2.2

Process model

Figure 2 shows the software development process model
adopted in the company. The process model shown inConcept Design& its Review(CD & CDR)

Unit Test& Debug(UT & UDB)

Function Design& its Review(FD & FDR)

Integration Test
& Debug(IT & IDB)

Structure Design& its Review(SD & SDR)

Coding& its Review(PG& PGR)

Module Design& its Review(MD & MDR)

Acceptance Test
(AT)

Function Test
& Debug(FT & FDB)
Verification Test
& Debug(VT & VDB)Design Phase

Debug & Test Phase

Figure 2: Process model
Figure 2 is a standard waterfall model. Strictly speaking,some irregular control flows (such as backwards flow toprevious activity or concurrent executions between previous and current activities) do rarely happen. But these are
not explicitly described in Figure 2.
In our discussion, we classify the development process
into two successive phases: design phase and debug &test phase, as shown in Figure 2. One important characteristic of the design phase is that the review activity is
introduced after each design activity and coding activity.
The design phase is divided into five stages: Concept Design, Function Design, Structure Design, Module Design,
Coding and their corresponding Reviews. On the other
hand, the debug & test phase consists of the repetition ofa pair of test and debug activities and is divided into fourstages: Unit Test, Integration Test, Function Test, Verification Test and their corresponding Debugs. At the end of
the debug & test phase, there is Acceptance Test, where
final test before code release is performed.

3 Process Improvement
3.1 Overview
In the certain company to be investigated in this paper,
the software engineering process group(SEPG) was established in 1992. Since then the SEPG conducted two
main process improvement activities as follows: establishment and introduction of the standards for managing thesoftware process, and improvement of the review processusing the software metrics toward high quality. Figure 3
shows a brief summary of the process improvement activities at the company.
At first, the SEPG tried to establish several standards for
managing software project, and to put it into practice from1993. The main purposes of the standards include guiding developers in each project to create the well-formed
plan and managers to conduct the project successfully according to the well-formed plan. It is believed (without

SEPG was esablished in 1992.

Activities for constructing project plan
started in 1993.
Activities for improving
review process have been
conducted since 1995.
'92

'93

'94

'95

'96

'97

'98

Figure 3: Process improvement activities

any proof) in the company that the well-formed plan will
derive the accurate estimation for the project cost. Furthermore, it is also believed that the accurate cost estimation
will lead the project to the high quality of the product andhigh productivity of development team. We have alreadyapplied the statistical analysis to this improvement activity and shown the correctness of these facts by the test of
statistical hypothesis with level of significance 5% in [11].
Next, the SEPG has extensively engaged in the improvement of review process. Generally, the cost of removing faults in the later development phases, such as
debug or test phase becomes higher than that in the earlier
phase such as design review or coding review phase. So
it is strongly recommended to remove faults in the earlier
phase. Concerning this fact, it is believed that the numberof faults detected in the review activity become greater byincreasing the amount of review activity. Based on thesimilar experience and knowledge as mentioned above,
the SEPG started the improvement of review process in
1995. The key point of the improvement is implementing effective review process by increasing the amount ofefforts for review(especially, code review) and by introducing good guidelines. We will discuss only the second
review process improvement in this paper.

3.2 Review process improvement

4

Objective of Our Study

As mentioned before, reviews include the document review in the design activity and the code review in thecoding activity. Generally, it is difficult to derive concrete guideline or numerical target value for the document
review[5, 7]. On the other hands, it is relatively easy toderive them for the code review. The situation is also true
for the SEPG’s activities, as shown in the past analysis
result of the review’s effect[16].
Based on the analysis result of the past project data, theSEPG has derived the following guidelines G1–G6 forthe review activities:

4.1

Software metrics

G1 At least the 15% of the total efforts for design andcoding activities should be assigned to reviews(thedocument review and the coding review).

G2 Reviewers must report the progress using the standard review form at regular intervals.

G3 In the design review, the documents should be distributed to all the experts in the company, andthen review results should be returned to developers via manager(This design review is called peer
review[13]).

G4 The coding review should be performed by two or
three persons, including one person who develops
the code.

G5 The review coverage rate for the code review shouldbe about 200 lines of code per hour.

G6 All of new codes and changed codes should be reviewed. (Concerning reuse of old codes, reviews are
not necessary required.)Among them, G1 and G2 are general requirements forreviews(including both the document review and the code
review), G3 is specific for the design review, and G4, G5
and G6 are only for the code review.
Before the review improvement began, the same activities as the guideline G2, G3 and G4 were required to be
performed in the review. However, since the guidelines
were not yet established, the review was not work effectively in the practical developments. Actually the average
review effort was less than 10% of the total design efforts.
This value 10% is not sufficient from the experience in thecompany.
Then, the SEPG started the improvement of review process in 1995 according to these guidelines. Intuitively
speaking, recently these guidelines are truly followed in
the company and no serious failure reports reach to the
SEPG. However any formal or statistical discussions on
the efforts by the review process improvement are not yet
done. So, we try to analyze empirically its effectivenessusing actual 23 project data.

Here, we explain software metrics: ,  and  to bemeasured in each project. Figure 4 shows a simplified
process model and a part of fundamental data set collected
at each phase of development.
From the design phase, the SEPG collects data ofsix fundamental metrics: Edesign , Ereview , Ecoding ,Ec:review for the efforts spent and Freview , Fc:review for
the number of detected faults. The metrics Edesign andEreview represent the total efforts spent for all activities
in design phase and all review activities in design phase,respectively. Similarly, the metrics Ecoding and Ec:review
represent the effort spent for coding and coding review,
respectively. Next the metric Freview represents the total
number of faults detected by all review activities in designphase, and the metric Fc:review represents the number offaults detected by coding review activity.
From the debug & test phase, the SEPG collects dataof two fundamental metrics: Etest and Ftest . The metric
Etest represents the total efforts spent for all activities inthe debug & test phase. Next, the metric Ftest represents
the total number of faults detected in the debug & test
phase.
During six months after the code shipping, the SEPG
provides the monitoring phase and collects all data concerning the failure detected by customers, as shown in
Figure 4. We call these failures post-released failures,and use the metric Fmonitor to represent the number ofpost-released failures.
Using these data collected from the projects, we define
three kinds of software metrics ,  and  to analyze andevaluate the software development process.
(1) Ratio of review effort ( ’s: %)In order to evaluate the amount of efforts, we define
two metrics as follows:

review=design=E

Ereview+ Ereview 2 100=E

Ec:review+ Ec:review 2 100

designc:review=coding

coding

The metric review=design evaluates the ratio of all review effort to design and coding efforts, andc:review=coding evaluates the ratio of code review
effort to coding effort.
(2) Ratio of detected faults (’s: %)In order to evaluate the ratio of detected faults in a
specific phase to all the faults detected, we define
three metrics as follows:review=total =Freview
2 100
Freview + Ftest + Fmonitor

Design Phase

Debug & Test Phase

Monitoring Phase- Efforts neededfor design phase- Efforts needed fordebug & test phase- No. of faults detected
after shipping- No. of faults detectedby review- No. of faults detectedby debug & test

- Seriousnessof failures detected

Figure 4: Fundamental data set

c:review=total =test=total =Fc:review
2100
Freview + Ftest + Fmonitor

Ftest
2 100
Freview + Ftest + Fmonitor

The metric review=total evaluates the ratio of faults detected in the review to all faults detected, andc:review=total evaluates the ratio of faults detectedin the code review to all faults detected. Finally,test=total evaluate the ratio of faults detected in the
debug & test phase to all faults detected.
(3) Seriousness of the failure ( : level)
For each post-release failure, the maintenance operator and the SEPG jointly decide the seriousness
. The values of seriousness  is classified as follows: destructive, confusing and mild. The level
 =destructive denotes that the failure can lead tothe system down. Thus the failure must be removed
immediately. Then the level  =confusing denotesthat only a part of system may be down by the failure
and other part may keep working. Thus it should be
removed immediately if possible. Finally the level =mild denotes that the failure never affects the essential part of system, and thus it may be negligible
for a while.

4.2 Assertions
The objective of our study is to investigate the effectiveness of review process improvement conducted by the
SEPG. We try to apply statistical analysis to the data collected in the actual projects to clarify the changes in thereview effort and the changes in the software quality. The
analysis consists of two step: investigating the review effort ratio (Step 1) and investigating the software quality(Step 2).
(1) Review efforts (Step 1 of analysis)
Here we try to confirm that the review improvement
activities are accepted in the development team. In

other words, we investigate the direct changes in thesoftware development process. In more detail, we
analyze the changes in the amounts of the effort forreview activity. As mentioned in subsection 3.2, theguideline G1 requires at least 15% of the total efforts
on design and coding activities should be spent on
review. Thus, we can expect that the effort of review
activity is increased in the projects guided by the
SEPG as the direct affect by the guideline G1. Basedon these considerations, we try to prove the following
assertion A1 by statistical analysis:

A1 The ratio of effort for review on the total efforts
for design and coding activities increases in the
projects guided by the SEPG.
Intuitively speaking, the assertion A1 means that theeffort for review activity increases as the result of the
SEPG’s guidance. We call a project that satisfies theguideline G1 (that is, the 15% of the total efforts ondesign and coding activities is assigned to the review
activity) a faithful project group. In the evaluationof process improvement we should discuss the properties of organization rather than that of individual
projects. Then intuitively speaking, we define a setof faithful project as a faithful project group.
(2) Software quality (Step 2 of analysis)
Here we try to evaluate the effectiveness of review
process improvement quantitatively. Thus, we investigate the changes in the number of detected faults andthe changes in the quality of the final product. They
are the indirect affect, but are the most essentially
expected affect of the review process improvement.
In Step 2 of analysis, we try to prove the following
assertions A2 and A3 by statistical analysis:

A2 The number of faults detected by the review
increases in each project of the faithful projectgroup. Similarly, the number of faults detectedin the debug and test phase decreases.

A3 As the result of the review process improvement, the quality of the final code is also improved.

Table 1: Comparison of ratio of faithful projects
Vending System

Checking System

Retail System(1992-1994)(1992-1994) (1995-1996)(1995-1996)

No. of projects

5

7

5

6

No. of faithfulprojects

0

2

2

6

Table 2: Comparison of ratio of review effort

greview/design
gc.review/coding

Vending System(1992-1994)

Checking System(1992-1996)

Retail System(1995-1996)

8.9%
6.6%

11.8%
10.5%

20.6%
21.6%

Intuitively speaking, the assertion A2 means that byincreasing the ratio of review effort to the total efforts
on design and coding activities, we can change thesoftware process essentially. That is, we can increase
the number of faults detected in the review, and at
the same time we can reduce the number of faults
detected in the debug and test activities.
The assertion A3 means that even the improving thequality of the final product will be partly attained by
the review process improvement. In the analysis in
Section 5, we apply data on the post-release failurethat are collected during six months after shipping,to evaluate the quality of the final product.

5 Effectiveness Analysis
5.1 Ratio of review effort
At first, concerning the assertion A1, we try to investigate
how the review improvement activities by the SEPG are
accepted by the organizations. Here in order to evaluate
the process improvement activities, we consider projectgroup rather than an individual project. As explained
already in subsection 2.1, there exist three project groups:
Vending System, Checking System and Retail System.
As mentioned already in subsection 3.2, the guideline
G1 recommends that review=design (the ratio of review
effort to effort in design phase) should be greater than
15%. According to this guideline, we define a project withreview=design  15% to be a faithful project. Then Table 1 shows the number of faithful projects in each projectgroup. In Table 1, in order to investigate Checking System(1992-1996) in detail, we divide it into Checking System(1992–1994) and Checking System (1995–1996) basedon the year 1995 when the SEPG started the review pro-

cess improvement. (Please note that the review process
improvement activities started in 1995, and thus Vending System is out of scope.) Next, Tables 2 and 3 show
the mean values of software metrics review=design andc:review=coding (the ratio of code review effort to coding
effort) for three project groups.
From Tables 1 and 2, the project group Retail Systemseems to succeed to follow the guideline G1 faithfully.
However, the project group Checking System seems to
fail to follow the guideline G1. Thus, we execute the testof statistical hypothesis with 5% level of significance toreview=design ’s of Retail System and Checking System.
As the result, we can prove that there exists a significant
difference between them. Similarly, for c:review=coding ,we can also prove that there exists a significant difference
between that of Retail System and Checking System.
We discuss the characteristics of organizations to show
the reason why there exists a big difference between Retail
System and Checking System. The project group Checking System started in 1992 and the members of the organization for Checking System had already established theirown ways for software development when the SEPG started the review process improvement. Thus it is generally
hard for them to change the process instantly according
to the guidelines specified by the SEPG group. On thecontrary, the project group Retail System started after the
SEPG had determined the guidelines. Thus the membersof the organization for Retail System tend to accept theguideline beyond all question.
Next, from Table 3 we can see the mean values ofreview=design and c:review=coding in 1995–1996 become
greater than that in 1992–1994. Thus, the review improvement seems to be accepted by the organization little bylittle. However, from the test of statistical hypothesis with
5% level of significance, we cannot see a significant difference for review=design and c:review=coding between

Table 4: Comparison of ratio of detected faults 

Table 3: Comparison of review effort in Checking System

greview/design
gc.review/coding

Checking System(1992-1994)

Checking System(1995-1996)

11.0%
9.6%

12.9%
11.8%

1992–1994 and 1995–1996.
As already mentioned, we should discuss the propertiesof organizations rather than that of individual projects.
Thus for convenience we define a set of faithful projects
as a faithful project group. According to this definition,
we refer
Retail System (1995–1996): faithful project group
Checking System (1992–1996): unfaithful project groupin the following. Clearly by this definition, Checking
System includes four faithful projects. But, it is also clear
the organization that developed Checking System failed
to follow the guideline G1. Thus we interpret that theorganization happened to have the result  15% forfour projects in Checking System, and take a view pointthat Checking System is an unfaithful project group. For
convenience, we also refer

Faithful
project group

Unfaithfulproject group

78.4%
21.7%
21.1%

38.8%
12.7%
60.7%

rreview/total
rc.review/total
rtest/total

the values of these metrics are confidential, we cannot
publish the values themselves in this paper. Thus Table 5shows only the relative values by assuming all the values
for unfaithful project group to be one, and the symbol 3
denotes its representation.
Since the original value of these metrics are so small,
we cannot apply the statistical analysis. But from Table
5 we can observe some interesting properties. The total
number of post-release failure (Fmonitor /# of projects) offaithful project group is smaller than that of unfaithfulproject group. Especially, the number of the failures with
 =destructive is smaller drastically. Thus we can guessthe correctness of the assertion A3.

Table 5: Comparison of post-release failures

Vending System (1992–1994): unfaithful project group
Faithful
project group

Unfaithfulproject group

Fmonitor / # of projects

0.55*

1*

destructive
confusing
mild

0.44*
0.50*
0*

1*
1*
1*

since this project group contains only unfaithful projects.

5.2 Ratio of detected faults 
Now we investigate the effect of review process improvement concerning the assertion A2. Table 4 shows themean values of software metrics review=total (the ratioof faults detected in review phase to all the faults detected), c:review=total (the ratio of faults detected in code
review to all the faults detected) and test=total (the ratioof faults detected in debug & test phase to all the faults
detected). In Table 4, we classify the projects into faithful project group (that is, Retail System) and unfaithfulproject (Vending System and Checking System).
By the test of statistical hypothesis with 5% level of significance, all of the values review=total , c:review=totaland test=total confirmed the significant difference between faithful project group and unfaithful project group.
Thus, we can say the correctness of the assertion A2 isproved by statistical analysis.

5.3 Post-release failure
As mentioned before, for each post-release failure the
SEPG or the maintenance operator decide the seriousness
 and assign its value to the failure. Table 5 summarizes
the total number of post-release failures and the distributions of ’s assigned to post-release failures. However,c6Conclusion

We have analyzed the effectiveness of the review process
improvement activities by the SEPG during these six years
in a certain company. According to the guidelines determined by the SEPG, we have investigated the ratio of thereview effort to the total effort for design and coding activities. As a result we found that a newly started projectgroup followed the guidelines faithfully. Similarly, we
have investigated the ratio  of faults detected in review
to the total number of detected faults. The result showedthat the ratio  is improved drastically in a faithful projectgroup. Finally, we have confirmed the number of postrelease failure during six months after code release is also
decreased by the SEPG’s process improvement activities.

References[1] L.C. Briand, K.E. Emam, B. Freimut and O.
Laitenberger: “Quantitative evaluation of capturerecapture models to control software inspections,”
Proc. 8th International Symposium on Software
Reliability Engineering, pp.234–244, 1997.
[2] L.C. Briand, K.E. Emam, O. Laitenberger and T.
Fussbroich: “Using simulation to build inspection
efficiency benchmarks for development projects,”
Proc. 20th International Conference on Software
Engineering(ICSE’98), pp.340–349, 1998.
[3] F.P. Brooks Jr.: “The Mythical Man-Month,” Addison Wesley, 1975.
[4] A. Cimitile and G. Visaggio : “A formalism forstructured planning of a software project,” International Journal of Software Engineering and Knowledge Engineering, Vol.4, No.2, pp.277–300, 1994.
[5] R.G. Ebenau and S.H. Strauss: “Software inspection process,” McGraw-Hill, 1993.
[6] N.E. Fenton and S.L. Pfleeger: “Software Metrics: A Rigorous & Practical Approach,” PWS Publishing, 1997.
[7] W.S. Humphrey: “Managing the Software Process,” Addison Wesley, Reading, MA, 1989.
[8] W.S. Humphrey, T. Snyder and R. Willis: “Software process improvement at Hughes Aircraft,”IEEE Software, Vol.8, No.4, pp.11–23, 1991.
[9] S. Kusumoto: “Quantitative evaluation of software
reviews and testing process,” PhD. Dissertation,
Osaka University, 1993.
[10] S. Kusumoto, O. Mizuno, Y. Hirayama, T. Kikuno,
Y. Takagi and K. Sakamoto: “A new project simulator based on generalized stochastic Petri-Net,”
Proc. 19th International Conference on Software
Engineering(ICSE’97), pp.293–303, 1997.
[11] O. Mizuno, T. Kikuno, K. Inagaki, Y. Takagi and
K. Sakamoto: “Analyzing effects of cost estimation accuracy on quality and productivity,” Proc.
20th International Conference on Software Engineering(ICSE’98), pp.410–419, 1998.
[12] K.H. Möller and D.J. Paulish: “Software Metrics: A Practitioner’s Guide to Improved Product Development,” IEEE Press (Chapman & Hall Computing), 1993.
[13] M.C. Paulk, C.V. Weber, S.M. Garcia, M.B. Chrissis and M. Bush: “Key practice of the capability maturity model, version 1.1,” Technical Report
CMU/SEI-93-TR-25, Software Engineering Institute, 1993.

[14] A.A. Porter, H.P. Siy, C.A. Toman and L.G. Votta:“An experiment to assess the cost-benefits of code
inspections in large scale software development,”
IEEE Transactions on Software Engineering, Vol.
23, No. 6, pp.329–346, 1997[15] R.M. Podorozhny and L.J. Osterweil: “The criticality of modeling formalisms in software design method comparison,” Proc. 19th International Conference on Software Engineering(ICSE’97),
pp.303–313, 1997.
[16] Y. Takagi, T. Tanaka, N. Niihara, K. Sakamoto, S.
Kusumoto and T. Kikuno: “Analysis of review’s
effectiveness based on software metrics,” Proc. 6th
International Symposium on Software Reliability
Engineering(ISSRE’95), pp.34–39, 1995.
[17] T. Tanaka, K. Sakamoto, S. Kusumoto and T.
Kikuno: “Improvement of software process by process visualization and benefit estimation,” Proc.
17th International Conference on Software Engineering(ICSE’95), pp.123–132, 1995.
[18] E. Yourdon: “Death March : The Complete Software Developer’s Guide to Surviving ‘Mission Impossible’ Projects,” Prentice Hall Computer Books,
1997.