IEICE TRANS. FUNDAMENTALS/COMMUN./ELECTRON./INF. & SYST., VOL. E85-A/B/C/D, No. 1 JANUARY 2002

1
LETTER Special Section/Issue on *****

Extended Metrics to Evaluate Cost Effectiveness of Software Inspections
Giedre Sabaliauskaite†, Shinji Kusumoto†† and Katsuro Inoue††, members
Summary
Software inspection is one of the most effective methods todetect defects. However, inspections are not always
worthwhile. This letter proposes an inspection cost model
to describe inspections-related costs and extended metrics
to evaluate the cost effectiveness of software inspections.
Key words:
Software metrics, cost effectiveness, software inspection.

1. Introduction
For more than twenty-five years software inspections havebeen considered an effective and efficient method forsoftware quality improvement [3,4,9]. The goal of
inspections is to detect defect before the testing stage
begins. According to the data published in the literature,
software design inspections save on average 44% of thedefect detection costs, and code inspections save on
average 39% of the defect detection costs [1]. However,are the software inspections always worthwhile?
In order to evaluate the effectiveness of the inspections
with respect to software development cost, several metrics
have been previously proposed [2,3,6]. Collofello et al.
have taken into account of the costs consumed and saved
by inspections and proposed a metric, called Cost
Effectiveness [2]. Kusumoto et al. proposed a metric Mk
for evaluating cost effectiveness of inspections, which isbased on the degree to which costs to detect all faults fromthe software in a project are reduced by inspections [6].
Yet, none of those metrics includes the information about
false positives (the issues that require no repair) introduced
during inspection, although the rework of false positives iscostly and can introduce new defects [8,12].
This letter presents a) an inspection cost model that
describes all costs related to inspections, b) extension of
the metric Mk to evaluate the cost effectiveness as well as
the losses of inspections. Proposed cost model and metrics
can be useful for project managers to decide whether
inspections are worthwhile.
The structure of the letter is as follows. Section 2
describes the inspection process. Section 3 describes theinspection cost model. In Section 4, we review the existing
metrics to evaluate software inspections. Section 5 presents
the extended metrics for software inspection evaluation. In
Section 6, we compare extended metrics to Collofello’s
Manuscript received
Manuscript revised†
The author is with Department of Informatics and Mathematical
Science, Osaka University, Toyonaka-shi, Osaka, 560-8531 Japan,††
The author is with Department of Computer Science, Osaka
University, Toyonaka-shi, Osaka 560-8531, Japan.

metric and Mk. Section 7 concludes the letter with an
overall summary and directions for future research.

2. Inspection Process
Software inspection as structured process was firstdescribed by Fagan [3]. It consists of the following stages:
Planning (moderator ensures that the entry criteria aremet); Overview (author of the product presents the review
team an overview of the product); Preparation (individual
reviewers analyze the work product with the goal ofunderstanding it thoroughly); Inspection meeting(reviewers meet face-to-face in an inspection team to
scrutinize the work product for defects); Rework (author
performs rework to correct the identified defects); Followup (moderator verifies that all the defect have beencorrected). Recently, the goals of preparation andinspection meeting stages have been modified. The
common modification is to include defect detection as an
explicit goal of preparation [7,10,11]. Furthermore, several
researchers [5,10,13] discuss if the inspection meetings arereally necessary, because they increase the cost of software
inspection and usually do not detect more defects than it
was already detected during preparation stage.
We will introduce two diagrams (Fig. 1 and Fig. 2) toexplain the life-cycles of defects and false positives
respectively. These diagrams show in what stages defects
and false positives are being introduced, detected andremoved.
The life-cycle of a defect is depicted in Fig. 1. It shows
four cases of defect’s life-cycle: d1, d2, d3 and d4. In all
the cases, defects are introduced before inspection process
begins, for instance during designing or coding. According
to Fagan’s model [3], defects are detected and confirmed
during inspection meeting stage, and removed by author
during rework (case d1). However, defects are usuallydetected by individual reviewers during preparation,
confirmed by inspection team during inspection meeting,and removed by author during rework (case d2). In somecases, defects are detected during preparation, however notconfirmed as defects during inspection meeting (case d3).
Some defects are not detected during inspection at all (cased4). In cases d3 and d4, defects are detected and removed
only during testing.

[*author name] SAKURADA :. INSTRUCTIONS FOR THE IEICE TRANS. AUTHOR’S TEMPLATE

2
d1
Designing
Codingd2

d3

3. Inspection Cost Modeld4

Introduced Introduced Introduced Introduced

3.1 Traditional Cost Model

Planning

Inspection

Overview
Preparation
Inspection
meetingRework

Detected

Detected

Detected
Confirmed

Confirmed

Notconfirmed

RemovedRemoved

The traditional inspection cost model consists of thefollowing components [6] (Fig. 3): Cr – cost spent for
inspection; Ct – cost needed for testing; ∆Ct – testing costsaved by inspection; Virtual testing cost – testing cost if noinspections are executed. By spending cost Cr during
inspection, the cost ∆Ct is being saved during testing.

Follow-up

Cr
Detected
RemovedTesting

Detected
Removed∆Ct

Reviews
Testing

Fig. 1 Life-cycle of a defect.

Cost

Ct

The life-cycle of a false positive is shown in Fig. 2. It
identifies five cases of false positive’s life-cycle: f1, f2, f3,
f4 and f5. False positives can be introduced during
preparation or inspection meeting stages. In cases whenfalse positives are introduced during preparation stage,
they can be detected and excluded from defect list byinspection team during inspection meeting (case f1) or byauthor during rework (case f2). False positives introduced
during inspection meeting can be detected and excluded
from defect list by author during rework (case f4).
However, if false positive is not excluded from the defectlist, the rework will be done and consequently a defect may
be introduced, which will be detected and removed only
during testing (cases f3 and f5).
The goal of the inspection is to ensure that the minimum
number of defects and false positives reaches testing.

Virtual testing cost

Fig. 3 Traditional cost model.

3.2 Extended Cost Model for Preparation Stage of
Inspection
In order to evaluate the influence of the false positives
introduced during preparation stage of inspection over thetesting cost, we extended the traditional cost model. The
following additional costs were defined (Fig. 4): CrDEF –cost spent to detect actual defects during preparation; CrFP
– cost spent to detect false positives during preparation;
CtDEF – cost needed for testing to detect remaining defects;
CtFP – cost needed for testing to detect defects introducedby false positives during preparation.

Designing
Coding

Cr
CrDEF

CrFP

Planning
Overview

f1

f2

f3

f4

f5

∆Ct

Preparation

Inspection

Testing
Preparation

Introduced Introduced Introduced

Inspection
meeting

Detected
ExcludedReworkCtDEFCt
Detected
DefectDetected
Excluded introduced Excluded- Preparation losses

Defectintroduced

Fig. 4 Extended cost model for preparation stage.

Follow-up
Testing

Cost

CtFP

Introduced Introduced

Detected
RemovedFig. 2 Life-cycle of a false positive.

Detected
RemovedIn this model, by spending cost CrDEF during inspection,
the testing cost ∆Ct is being saved. However, by spending
the cost CrFP during inspection, the cost CtFP is being added
to the testing cost. Therefore, the costs CrFP and CtFP are
the preparation stage losses (Fig. 4).

IEICE TRANS. FUNDAMENTALS/COMMUN./ELECTRON./INF. & SYST., VOL. E85-A/B/C/D, No. 1 JANUARY 2002

3

3.3 Extended Cost Model for Preparation andInspection Meeting Stages of Inspection
Inspection meetings are usually carried out after
preparation stage is completed. However, several authors
question the usefulness of the meetings [5,10,13]. To
enable the evaluation of the benefits and the losses ofinspection meetings, we propose an extended cost model
for preparation and inspection meeting stages (Fig. 5).
The following are the costs (Fig. 5): CmDEF – cost spent
to confirm actual defects detected during preparation stage;
CmFP – cost spent to confirm false positives detected during
preparation stage; CmADD_DEF – cost spent to detect
additional defects during inspection meeting stage;
CmADD_FP – cost spent to detect additional false positives
during inspection meeting stage; CmLOST_DEF – cost spent toeliminate actual defects detected during preparation stage;
CmELIM_FP – cost spent to eliminate false positives detectedduring preparation stage; CtADD_FP – cost needed for testing
to detect defects introduced by additional false positives
detected during inspection meeting stage; CtLOST_DEF –testing cost needed to detect defects lost during inspection
meeting stage; ∆CtADD_DEF – testing cost saved byadditional defects detected during inspection meetingstage; ∆CtELIM_FP – testing cost saved by false positives
eliminated during inspection meeting stage; ∆CtDEF –testing cost saved by defects detected during preparationand confirmed during inspection meeting stages.
Cr
CrDEF

CrFP
∆Ct
CmELIM_FP

CmLOST_DEF

CmADD_FPCmADD_DEF

CmFP

Inspection
meeting

CmDEF

Preparation

CtFP

4.1 Fagan’s Metric
Fagan [3] introduced the Error Detection Efficiency metricMf for measuring inspection efficiency. Mf is defined as thenumber of defects found during inspection over the total
number of defects in the product existing before inspection.
We define the total number of defects in the productexisting before inspection as DEFtotal, and the number ofdefects found during inspection as DEFr. Then we get thefollowing equation:
Mf =

Cost
CtLOST_DEF∆CtDEF

DEFr
DEFtotal(1)

As we can see from Eq. (1), metric Mf does not accountfor the cost expended for inspection.

4.2 Collofello’s Metric
Collofello and Woodfied [2] proposed Cost Effectivenessmetric Mc, which is defined as a ratio of the “cost saved by
the process” to the “cost consumed by the process”. Using
the notation described in Section 3.1, we get the following
equation:∆CtELIM_FP

CtADD_FPCt

4. Metrics to Evaluate Software Inspections

Mc =∆CtADD_DEF

Testing
CtDEF

Inspection meeting costs CmDEF and CmFP, spent toconfirm defects and false positives detected during
preparation stage, do not increase or reduce testing costs;
however, they increase the overall inspection cost.

∆Ct
Cr

(2)

Although metric Mc takes into account the costs
consumed and saved by inspections, it does not take into
account the total cost to detect all defects in the software
product by inspection and testing.

- Inspection meeting losses

4.3 Kusumoto’s MetricFig. 5 Extended cost model for preparation and inspection meeting stages.

In this model, cost CmELIM_FP spent to eliminate false
positives, reduces the testing cost by ∆CtELIM_FP; and cost
CmLOST_DEF, spent to eliminate actual defects, increases
testing cost by CtLOST_DEF. Inspection meeting cost spent todetect additional defects, CmADD_DEF, reduces the testing
cost by ∆CtADD_DEF; and inspection meeting cost spent todetect additional false positives, CmADD_FP, increases thetesting cost by CtADD_FP. In other words, the costs
CmLOST_DEF , CmADD_FP, CtLOST_DEF and CtADD_FP are the costs
lost by inspection meeting (Fig. 5).

Kusumoto et al. [6] proposed a metric Mk to evaluate thecost effectiveness of software inspections in terms of
reduction of cost to detect and remove all defects fromsoftware. Using the notation described in Section 3.1, we
get the following equation:
Mk =∆C t − C r
C t + ∆C t

(3)

Mk is a ratio of the reduction of the total costs to detect
and remove all defects from documents using inspections
in a project to the virtual testing cost. The testing cost is[*author name] SAKURADA :. INSTRUCTIONS FOR THE IEICE TRANS. AUTHOR’S TEMPLATE

4
reduced by ∆Ct compared to the virtual testing cost (Ct +∆Ct) if no inspection is executed (see Fig. 3).

5. Extended metrics
5.1 Need for New Metrics
Among metrics Mf, Mc and Mk, metric Mk is the mostpractical one to evaluate the cost effectiveness ofinspections. However, it includes only the total cost spenton inspection, not taking into consideration thecomposition of the inspection cost, which was described in
Section 3.
We decided to extend metric Mk to conform to theextended cost model. Section 5.2 presents metrics for theextended cost model for preparation stage, and Section 5.3
presents metrics for the extended cost model of preparationand inspection meeting stages.

5.2 Extension of Metric Mk for Preparation StageIn accordance with extended cost model for preparation
stage (see Section 3.2), cost ∆Ct is a testing cost saved by
preparation, and costs CrFP and CtFP are the costs lost byinspections, since additional effort is being spent during
inspection for the detection of false positives, and those
false positives cause additional cost during testing (Fig. 4).
We want to introduce a new metric Ml_IDV to evaluate thePreparation Losses, which is the ratio of “cost lost byinspections” by “cost saved by inspections”:

M l _ IDV =C rFP + CtFP
∆Ct

(4)

Not only the costs CrFP and CrDEF, but also CtFP is thecost caused by inspection, since additional effort is being
spent during testing to remove the defects introduced byfalse positives during preparation stage. Therefore, wewant to introduce a new metric Mg_IDV, which is anextension of metric Mk (Eq. 3), to evaluate the Extended
Cost Effectiveness of Preparation Stage of Inspection. It
can be expressed using the following formula:M g _ IDV =∆Ct − C r − CtFP ∆Ct − C rDEF − C rFP − CtFP=
CtDEF + ∆Ct
CtDEF + ∆Ct(5)

In metric Mk (see Section 4.3), the virtual testing cost(the testing cost if no inspection is executed) is defined as(Ct + ∆Ct). However, inspection increases the testing costby CtFP if false positives have been detected (see Fig. 4).
Therefore, if no inspection is executed, the cost CtFP should
not be included into the virtual testing cost. In Eq. (5), we

exclude this cost from the virtual testing cost, and define
virtual testing cost as (CtDEF + ∆Ct). In addition, since thetesting cost CtFP is the cost caused by inspection as well,
we add it to inspection costs in Eq. (5).
In case if no false positives have been introduced during
preparation stage Mg_IDV= Mk, otherwise Mg_IDV< Mk.

5.3. Extension of Metric Mk for Preparation andInspection Meeting Stages
According to the extended cost model for preparation andinspection meeting stages (see Section 3.3, Fig. 5), two
additional testing costs are introduced by inspection
meeting: CtLOST_DEF and CtADD_FP. CtLOST_DEF is a testing cost
spent to detect defects lost during inspection meeting, and
CtADD_FP is a cost spent to eliminate additional false
positives, introduced during inspection meeting. Inspection
meeting may save testing cost by finding additional defects
detected during inspection meeting stage ∆CtADD_DEF, andby eliminating false positives detected during preparation
stage ∆CtELIM_FP.
Inspection meeting costs CmDEF and CmFP, spent toconfirm actual defects and false positive defects detectedduring preparation stage, do not have influence overtesting cost, however they increase the overall inspection
cost.
Costs CmADD_DEF and CmELIM_FP, spent to detect additional
defects and to eliminate false positives, reduce testing
costs by ∆CtADD_DEF and ∆CtELIM_FP respectively. Costs
CmADD_FP and CmLOST_DEF spent to detect additional false
positives and to eliminate actual defects, increase testing
costs by CtLOST_DEF and CtADD_FP respectively. The purposeof the inspection meeting should be to minimise the costs
CmADD_FP and CmLOST_DEF, and to maximise the costs
CmADD_DEF and CmELIM_FP.
Similarly as the metric Ml_IDV (Eq. 4) to evaluate thelosses of preparation stage, we introduce a new metric
Ml_MEET to evaluate the Inspection Meeting Losses, whichis the ratio of “cost lost by inspection meeting” by “costsaved by inspection meeting” (see Fig. 5):
M l _ MEET =C mADD _ FP + C mLOST _ DEF + C tADD _ FP + C tLOST _ DEF∆C tADD _ DEF + ∆C tELIM _ FP

(6)

Although the costs CmFP and CtFP are the additional costs
caused by inspection, we do not include them into themetric Ml_MEET (Eq. 6), because they depend on bothpreparation and inspection meeting stages.
Similarly as the metric Mg_IDV (Eq. 5), we want topropose a metric Extended Cost Effectiveness of
Preparation and Inspection Meeting Stages, Mg_MEET, toevaluate cost effectiveness of software inspections, whenboth preparation and inspection meeting are performed:

IEICE TRANS. FUNDAMENTALS/COMMUN./ELECTRON./INF. & SYST., VOL. E85-A/B/C/D, No. 1 JANUARY 2002

5
M g _ MEET =

=

−

−

∆Ct − C r − C m − CtFP − CtADD _ FP − CtLOST _ DEF
CtDEF + ∆Ct=

cost than in the Cases I and II, therefore Case III would be
expected to be more cost effective.
10
CrDEF CrFP
Preparation
6
4∆CtADD _ DEF + ∆CtELIM _ FP + ∆CtDEF − CrDEF − CrFPCtDEF + ∆CtADD _ DEF + ∆CtELIM _ FP + ∆CtDEF

CtDEF
700

Testing

CtFP
200

CmDEF + CmFP + CmADD _ DEF + CmADD _ FP + CmLOST _ DEF
CtDEF + ∆CtADD _ DEF + ∆CtELIM _ FP + ∆CtDEF
C mELIM _ FP + CtFP + CtADD _ FP + CtLOST _ DEF
CtDEF + ∆CtADD _ DEF + ∆CtELIM _ FP + ∆CtDEF∆Ct100
900 1000

Cost(a) Case I (Testing cost reduced by 100)
10(7)Metric Mg_MEET (Eq. 7) is a modification of metric Mk(Eq. 3). In Eq. (7), the virtual testing cost is defined as(CtDEF +∆CtADD_DEF +∆CtELIM_FP +∆CtDEF), and the testing
costs CtFP, CtADD_FP and CtLOST_DEF, caused by inspection,
are added to the inspection cost.

Preparation CrDEF CrFP
1
9
CtFP ∆Ct
30
100
900 1000

CtDEF
870

Testing

Cost(b) Case II (Testing cost reduced by 100)60
CrDEF CrFP
40
20
CtDEF
Testing300

Preparation

∆Ct
600

CtFP
100
400

1000

Cost

(c) Case III (Testing cost reduced by 600)

CmELIM_FP

5 5

5

5

CtFP

CtLOST_DEF∆CtADD_DEF∆CtELIM_FP∆CtDEF

900

CtDEF

CtADD_FPCmADD_FP

10

CmLOST_DEF

CmADD_DEF

10

40
CmFP

Inspection
meeting

CmDEF

700

100

50

50

20 20

60

Testing1000 Cost(d) Case IV (Testing cost reduced by 100)

8

2 2

900

8

20 10 10 20 20

60∆CtADD_DEF

860

Testing

CtADD_FPCtFPCtLOST_DEF

CtDEF∆CtDEF

10∆CtELIM_FP

10

CmELIM_FP

40
CmADD_FPCmLOST_DEF

Inspection
meeting

10
CrDEF CrFP
91
CmADD_DEF

Preparation

CmFP

The differences among Mc, Mk and proposed metrics canbe demonstrated with reference to the five imaginary
projects (Fig. 6). In all projects, if no inspections had been
executed, the testing cost would be 1000 units.
In cases Case I – Case III of Fig. 6, the preparation stageof inspection has been performed. In cases Case IV and
Case V of Fig. 6, preparation and inspection meetingstages have been performed. The notation of Fig. 6 is takenfrom the extended cost models (Fig. 4 and Fig. 5).
In Case I of Fig. 6 (a), inspection consumes 10 units ofcost (6 units to detect actual defects, and 4 units to detect
false positives), saves 100 units of testing cost, and thetesting cost is 900 (700 to detect remaining defects insoftware product, and 200 to detect defects introduced byfalse positives). Therefore, the total cost is 910. Case II issimilar to Case I, because it consumes the same costs on
inspection and testing as in Case I, however the
distribution of inspection and testing costs is different inCase II (during inspection, 9 units of cost are spent todetect actual defects and 1 unit to detect false positives;
during testing, 870 units of cost are spent to detect
remaining defects in software product and 30 to detect
defects introduced by false positives). In Case III of Fig. 6(c), inspection costs 60 units (40 units to detect actual
defects, and 20 units to detect false positives), saves 600,
testing cost is 400 (300 to detect remaining defects insoftware product, and 100 to detect defects introduced byfalse positives), and the total cost is 460.
If we apply Collofello’s metric Mc to Cases I, II and III,the value of Mc is 10 in all cases. However, in the Case III,
inspection saved much more of the total defect detection

10
Preparation CrDEF CrFP
6
4

CmDEF

6. Comparison of Proposed Metrics to Mc and
Mk Metrics

1000 Cost(e) Case V (Testing cost reduced by 100)
Fig. 6 Comparing five different cases of inspection.

The value of metric Mk for Case I and II is 0.09, and forCase III is 0.54. Thus, Mk identifies that the inspection inCase III is more effective than in Cases I and II. However,it does not show the difference between Cases I and II,
although the inspection losses due to the false positives are
greater in Case I. If we apply metric Ml_IDV to evaluate
inspection losses, the value of Ml_IDV is 2.04 in Case I, 0.31
in Case II, and 0.2 in Case III. Thus, Ml_IDV identifies that
inspection losses are the greatest in Case I.
The values of the extended cost effectiveness metric
Mg_IDV, which takes into consideration inspection losses,

[*author name] SAKURADA :. INSTRUCTIONS FOR THE IEICE TRANS. AUTHOR’S TEMPLATE

6
are -0.14 in Case I, 0.06 in Case II, and 0.49 in Case III.
As we can see from those results, metric Mg_IDV is more
precise than Mk, because it shows that inspection in Case II
is more effective than in Case I.
Case IV and Case V of Fig. 6 demonstrate the projects
in which preparation and inspection meeting stages are
performed. In both cases, preparation consumes 10 units ofcost, inspection meeting consumes 40 units of cost, thetesting cost is 900, and 100 units of cost are saved byinspection. However, these cases differ in the distributionof costs. During inspection meeting, in Case IV 5 units ofcost are spent to detect additional defects (Case V: 8 units),
5 units to detect additional false positives (Case V: 2 units),
5 units to eliminate actual defects (Case V: 2 units), and 5
units to eliminate false positives (Case V: 8 units). During
testing, in Case IV 700 units of cost are spent to detect
defects (Case V: 860), 100 units to detect defects
introduced by false positives (Case V: 20), 50 units todetect defects introduced by additional false positives
detected during inspection meeting (Case V: 10), and 50
units to detect defects lost during inspection meeting (Case
V: 10). The total cost in both cases is 950.
The value of metric Mc in both cases Case IV and Case
V is 2, and the value of metric Mk in both cases is 0.05.
The value of metric Ml_MEET in Case IV is 2.75, and in Case
V is 0.6. Therefore, Ml_MEET identifies that inspection
losses are greater in Case IV as compared to Case V. The
value of metric Mg_MEET is -0.188 in Case IV, and 0.0104 in
Case V. Consequently, metric Mg_MEET is more precise thanMc and Mk since it shows that inspection is more effective
in Case V as compared to Case IV.

7. Summary and Conclusions
In this letter, we have proposed two extended cost models:a model to describe the costs spent during preparation
stage, and a model to describe the costs spent during
preparation and inspection meeting stages. Those models
can be useful for other researchers to have a greater
understanding of all the costs related to inspections.
In addition, we have proposed two new metrics Ml_IDV(Eq. 4) and Ml_MEET (Eq. 6) to evaluate the Preparation
Losses and the Inspection Meeting Losses respectively.
Also, we have proposed two metrics Mg_IDV (Eq. 5) and
Mg_MEET (Eq. 7), which are the modifications ofKusumoto’s metric (Eq. 3), to evaluate the Extended Cost
Effectiveness of Preparation and the Extended Cost
Effectiveness of Preparation and Inspection Meeting
respectively. All those metrics enable more precise
evaluation of software inspections as compared to theconventional metrics.

Future research will be directed to the experimental
evaluation of proposed metrics.
References[1] L. Briand, K. El Emam, O. Laitenberger, T. Fussbroich,“Using Simulation to Build Inspection Efficiency
Benchmarks for Development Projects”, Proc. 20th
International Conference on Software Engineering, Kyoto,
Japan, pp.340-349, 1998.
[2] J.S. Collofello, S.N. Woodfield, “Evaluating theeffectiveness of reliability-assurance techniques”, Journal of
Systems and Software, vol.9, no.3, pp.191-195, 1989.
[3] M. Fagan, “Design and code inspections to reduce errors inprogram development”, IBM Systems Journal, vol.15, no.3,
pp.182-211, 1976.
[4] T. Gilb, D. Graham, Software inspection, Addison-Wesley,
1993.
[5] P.M. Johnson, D. Tjahjono, “Does every inspection reallyneed a meeting?”, Empirical Software Engineering: An
International Journal, vol.3, no.1, 1998.
[6] S. Kusumoto, K. Matsumoto, T. Kikuno, K. Torii, “A new
metrics for cost effectiveness of software reviews”, IEICE
Transactions on Information and Systems, vol.E75-D, no.5,
pp.674-680, September 1992.
[7] O. Laitenberger, J.M. DeBaud, “An encompassing life cycle
centric survey of software inspection”, The Journal of
Systems and Software, vol.50, no.1, pp.5-31, 2000.
[8] L.P.W. Land, R. Jeffery, C. Sauer, “Validating the DefectDetection Performance Advantage of Group Designs for
Software Reviews: Report of Replicated Experiment”, Proc.
of the Australian Software Engineering Conference, Sydney,
Australia, pp. 17-26, 1997.
[9] O. Mizuno, T. Kikuno, “Empirical Evaluation of Review
Process Improvement Activities with respect to PostRelease Failure”, Proc. Empirical Studies on Software
Development Engineering Workshop, Los Angeles, CA,
USA, pp. 50-53, May 1999.
[10] A.A. Porter, P.M. Johnson, “Assessing Software Review
Meetings: Results of a Comparative Analysis of Two
Experimental Studies”, IEEE Transactions on Software
Engineering, vol.23, no.3, pp.129-145, March 1997.
[11] G. Sabaliauskaite, F. Matsukawa, S. Kusumoto, K. Inoue,“Further Investigations of Reading Techniques for ObjectOriented Design Inspection”, Information and Software
Technology, vol.45, no.9, pp. 571-585, 2003.
[12] C. Sauer, R. Jeffery, L. Land, P. Yetton, “The effectivenessof software development technical reviews: A behaviorally
motivated program of research”, IEEE Transactions on
Software Engineering, vol.26, no.1, pp.1-14, January 2000.
[13] L.G. Votta Jr, “Does Every Inspection Need a Meeting?”,
Proc. ACM SIGSOFT Symposium on Foundations ofSoftware Engineering, ACM Software Engineering Notes,
vol.18, no.5, pp.107-114, December 1993.