CR-189409

SOFTWARE ENGINEERING LABORATORY SERIES

COLLECTED SOFTWARE

ENGINEERING PAPERS:

VOLUME XII

NOVEMBER 1994

SE_ _

(NASA-CR-Ic9409) CCLLECTFO

SOFTWARE ENGINEERING PAPERS, VOLUME

12 (NASA. Goddard Space Flight

Center) 117 p

N95-28713

Unclas

NASA.

National Aeronautics and

Space Administration

G]/61 0053148

SOFTWARE ENGINEERING LABORATORY SERIES SEL-94-004

COLLECTED SOFTWARE

ENGINEERING PAPERS:

VOLUME Xll

NOVEMBER 1994

National Aeronautics and

Space Administration

Goddard Space Flight Center

Greenbelt, Maryland 20771

Foreword

The Software Engineering Laboratory (SEL) is an organization sponsored by the National

Aeronautics and Space Administration/Goddard Space Flight Center (NASA/GSFC) and

created to investigate the effectiveness of software engineering technologies when applied to

the development of applications software. The SEL was created in 1976 and has three

primary organizational members:

NASA/GSFC, Software Engineering Branch

University of Maryland, Department of Computer Science

Computer Sciences Corporation, Software Engineering Operation

The goals of the SEL are (1) to understand the software development process in the GSFC

environment; (2) to measure the effect of various methodologies, tools, and models on this

process; and (3) to identify and then to apply successful development practices. The

activities, findings, and recommendations of the SEL are recorded in the Software

Engineering Laboratory Series, a continuing series of reports that includes this document.

Single copies of this document can be obtained by writing to

Software Engineering Branch

Code 552

Goddard Space Flight Center

Greenbelt, Maryland 20771

10022514L

°..

111

_G, P_GE BL'&I_K NOT FIL_,_ED

TABLE OF CONTENTS

Section 1--Introduction .............................................

Section 2--Software Measurement ....................................

"A Change Analysis Process to Characterize Software Maintenance Projects,"

L. C. Briand, V. R. Basili, Y. Kim, and D. R. Squier ...............

Defining and Validating High-Level Design Metrics, L. Briand, S. Morasca,

and V. R. Basili ...........................................

Section 3---Technology Evaluations ....................................

Comparing Detection Methods for Software Requirements Inspections: A

Replicated Experiment, A. A. Porter, L. G. Votta Jr., and

V. R. Basili ................................................

"Software Process Evolution at the SEL," V. Basili and S. Green ..............

Section 4---Ada Technology ...........................................

"Genericity Versus Inheritance Reconsidered: Self-Reference Using Generics,"

E. Seidewitz ...............................................

Standard Bibliography of SEL Literature

1-1

2-1

2-3

2-15

3-1

3-3

3-33

4-1

4-3

10022514L V

• .; _-,,, F_itbW_#t_

P,:_f_.... _-_-...._i_,_-(f_h"' "....... ,tJI,_',_L_ _."!K

SECTION 1reINTRODUCTION

This document is a coUection of selected technical papers produced by participants in the

Software Engineering Laboratory (SEL) from November 1993 through October 1994. The

purpose of the document is to make available, in one reference, some results of SEL research

that originally appeared in a number of different forums. This is the 12th such volume of

technical papers produced by the SEL. Although these papers cover several topics related to

software engineering, they do not encompass the entire scope of SEL activities and interests.

Additional information about the SEL and its research efforts may be obtained from the

sources listed in the bibliography at the end of this document.

For the convenience of this presentation, the five papers contained here are grouped into

three major sections:

• Software Measurement

• Technology Evaluations

• Ada Technology

The first section (Section 2) includes a study on the analysis of software maintenance

changes to understand the flaws in the change process and a study on the comparison of four

strategies for defining high-level design metrics. Section 3 presents studies on software

inspection techniques and the SEL's Quality Improvement Paradigm. A study on simulating

inheritance in an object-oriented environment appears in Section 4.

The SEL is actively working to understand and improve the software development process at

Goddard Space Flight Center (GSFC). Future efforts will be documented in additional

volumes of the Collected Software Engineering Papers and other SEL publications.

I00?_..2514L 1-1

SECTION 2---SOFTWARE MEASUREMENT

The technical papers included in this section were originally prepared as indicated below.

"A Change Analysis Process to Characterize Software Maintenance Projects,'"

L. C. Briand, V. R. Basili, Y. Kim, and D. R. Squier, Proceedings of the International Conference on Software Maintenance, September 1994

Defining and Validating High-Level Design Metrics, L. Briand, S. Morasca, and

V. R. Basili, University of Maryland, Technical Report TR-3301, June 1994

10022514L 2-1

A Change Analysis Process to Characterize Software Maintenance Projects

Lionel C. Briand, Victor R. Basili, Yong-Mi Kim

Computer Science Department and Institute for Advanced Computer Studies

University of Maryland, College Park, MD, 20742

Donald R. Squier

Computer Sciences Corporation

System Sciences Division

Lanham-Seabrook, MD, 20706

Abstract

In order to improve software maintenance processes, we

need to be able to first characterize and assess them.

This task needs to be performed in depth and with

objectivity since the problems are complex. One

approach is to set up a measurement program

specifically aimed at maintenance. However,

establishing a measurement program requires that one

understands the issues and is able to characterize the

maintenance environment and processes in order to

collect suitable and cost-effective data. Also, enacting

such a program and getting usable data sets takes time.

A shorttermsubstituteisneeded

We proposeinthispaper a characterization process

aimed specificallyat maintenanceand based on a

generalqualitative analysismethodology.Thisprocess

isrigorouslydefinedinordertoberepeatableandusable

bypeoplewho arenotacquaintedwithsuch analysis

procedures.A basicfeatureofour approachisthat

maintenancechangesareanalyzedinordertounderstand

thefiawsinthechangeprocess.Guidelinesareprovided

and a case study isshown thatdemonstratesthe

usefulnessoftheapproach.

1 Introduction

As described in [HV92], numerous factors can affect

soRware maintenance quality and productivity, e.g., the

maintenance personnel experience profde and training,

the way knowledge about the maintained systems is

managed and conveyed to the maintainers and users, the

maintenance organization, processes and standards in

use, the initial quality of the software source code and

its documentation. This last factor involves concepts

such as self-descriptiveness, modularity, simplicity,

consistency, expandability, and testability.

Because of the complexity of the phenomena studied, it

is difficult for maintenance organizations to identify and

assess the issues they have to address in order to

improve the quality and productivity of their

maintenance projects. Each project may encounter

specific difficulties and situations that are not

necessarily alike across all the organization's

This work was supported in part by NASA grant NSG5123

maintenance projects. This may be due in part to

variations in application domain, size, change

frequency, and/or schedule/budget constraints. As a

consequence, each project has first to be analyzed as a

separate entity even if, later on, commonalities across

projects may require similar solutions for

improvement. Informally interviewing the people

involved in the maintenance process would be unlfl_ly

to help determine accurately the real issues.

Maintainers, users and owners would likely each give

verydifferent, and oftencontradictory, insightson the

issuesdue to a somewhat incompleteand biased

p_spective.

Establishing ameasurementprogramintegratedintothe

maintenanceprocessisI_elytohelpany organization

achievean in-depthunderstandingof itsspecific

maintenanceissuesand therebylaya solidfoundation

for maintenanceprocess improvement [RUV92].

However,definingandenactinga measurementprogram

may taketimeand a shortterm,quicklyoperational

substituteisneeded in ordertoobtaina firstquick

insight,atlow cost,intotheissuestobe addressed.

Furthermore,definingefficient andusefulmeasurement

proceduresfirstrequiresa characterization of the

maintenanceenvironmentinwhichmeasurementtakes

place,i.e.,organizationstructures, processes,issues,

risks, etc. [BR88].

This paper presents a qualitative and inductive analysis

methodology for performing objective project

characterizations and thereby identifying their specific

problems and needs. It is an implementation of the

general qualitative analysis methodology defined in

[SS92]. It encompasses a set of procedures which

allows the determination of causal links between

maintenance problems and flaws of the maintenance

organization and process. Thus, a set of concrete steps

for maintenance quality and productivity improvement

can be taken based on a tangible understanding of the

relevant maintenance issues. Moreover, this

understanding provides a solid basis on which to define

relevant software maintenance models and metrics.

Section 2 describes the phases, techniques and

guidelines composing the methodology. Section 3

presents a case study of an orbit determination system

maintained by the Flight Dynamics Division (FDD) of

the NASA Goddard Space Flight Center for the last 26

years and still used daily for most operating satellites

10022.514L 2-3

PlilEOE_NG. PAGE BLANK NOT FILMED
/

• Intm"vimmwith _

. S_tom and rekmo docammD

.Fioiddktz

.CbnCsn_umt¢ar_

I_luetlw _

Duhet_e_

•lk,httlou (e.g., _vlUm)

•'Ib.zmomim (e,4., took)

•Wm'ktnZb,vpothem_ (L$, error

mmcbuimm, lZ'omm flaws)

Figure 1: Qualitative Analysis

(GTDS: Goddard Trajectory Detexmination System).

This study takes place in the framework of the NASA

Software Engineering Laboratory (NASA-SEL), an

organization aimed at improving FDD software

development processes based on measurement and

empirical analysis. Recently, responding to the

growing cost of software maintenance, the SEL has

initiated a program aimed at characterizing, evaluating

and improving its maintenance processes. This paper is

a first step in this dLrection. Section 4 outlines the

main conclusions of the case study and the future

research directions.

2 Causal Analysis of Maintenance

Problems

In this section, we present a (mainly) qualitative

methodology that allows for an in-depth

characterization of maintenance projects at a relatively

low cost. However, this approach could be easily

augmented to integrate data collection and analysis and

could thus provide more quantitative information (but

at a higher cost).

2.1 A Qualitative Analysis Process

This characterization process is essentially an

instantiation of the generic qualitative analysis process

defined in [SS92]. Figure 1 illustrates at a high level

our maintenance specific analysis process. It can be

seen that it is a combination of both inductive and

deductive inferences. Inductive inferences are based on

the collected information, and deductive inferences occur

when experimentally validating and refining our

Process for Software Maintenance

taxonomies, process models, organizational models and

working hypotheses. These deductive inferences then

serve to tel'me the data collection process, which leads

to refined and revised inductive inferences. The process

continues in an iterative fashion.

We present below a general description of the process

involved in preparing and performing characterizations

of maintenance projects. Maintenance is defined he_'e as

any kind of enhancement, adaptation or correction

performed on the software system once in operation. At

the highest level of abstraction, parts of this process do

not appear specific to maintenance and could also be

used for development. However, the taxonomies and

guidelines developed to support this process and

presented in Section 2.2 are specifically aimed at

maintenance.

Step 1 focuses on defining the organizational

structures, i.e., organization entities, their

communication channels and information flows. The

process of producing a new release is then described and

modeled in Step 2. It is important to note that we do

not address here the issues related to emergency bug

fining procedures but only those relevant to regular

product releases that go into configuration

management. Step 3 maps generic activities into the

release process in order to specify the type of work

performed at each stage of the process. Then, a release

(or several) has to be selected in order to define the set

of changes on which the analysis will be performed

(Step 4). In Step 5, relying on the work performed in

Steps 1-3, information about the changes is collected

and analyzed. Step 6 summarizes and abstracts from the

results obtained in Step 5.

10022514L 2-4
Although the steps are defined sequentially, they are

really iterated within and across steps. As we learn

more about the organization, we continue to refine the

chamcteriz_on models. Tlaeorganizational and process

models produced should include enough detail to allow

Step 5 to be performed, but should not be so detailed as

to obscure the maintenace proce_ itself. We now define

the steps in more detail:

1 Identify the organizational entities with which

the maintenance team interacts and the organizational

structure in which maintainers operate.

1.1 Identify distinct organizational entities, i.e.,

what are the distinct teams involved in the maintenance

project? Usually, besides the maintainers themselves,

tile following entities are encountered: users, owners,

QA team, configuration control team, change control

board. However, their roles and prerogatives can differ

significantly.

1.2 Characterize the working environment of each

entity, i.e., support tools (see tool taxonomy in

Section 2.2), internal organizational structure.

1.3 Characterize information flows between

entities, i.e., what is the type (and amount when data

available) of information, documentation, source code

and other software artifacts flowing between

organizational entities?

2 Identify the phases involved in the creation of

a new system release.

2.1 Identify the phases as defined in the

environment studied. At this stage, it is important not

to map an a priori external/generic maintenance process

model and vocabulary.

2.2 Each artifact (e.g., document, source code)

which is input or output of each phase has to be

determined and its content carefully described (see

document taxonomy in Section 2.2).

2.3 The personnel in charge of producing and

validating the output artifacts of each phase have to be

identified and located in the organizational structure

defined in Step 1.

3 Identify the generic activities involved in each

phase.

3.1 Select (from the literature [C88, BC91]) or

def'me a taxonomy of generic activities based on widely

accepted definitions and used in the maintenance

process. As a guideline, such a taxonomy is proposed

in the next section.

3.2 Map these activities into each phase by

reading the technical documents produced and

interviewing the technical project leaders and

maintainers about their real work habits. If possible,

collect effort data for each activity so that the

importance of each activity in each phase can be

assessed somewhat quantitatively.

Select one or several past releases for analysis.

We need to select releases on which we can analyze

problems as they are occuring and thereby better

understand process and organization flaws. However,

because of time constraints, it is sometimes more

practical to work on past releases. We present below a

set of guidelines for selecting them:

• Recent releases are preferable since

maintenance processes and organizational structure

might have changed and this would make one's analysis

somewhat irrelevant.

• Some releases may contain more complete

documentationthanothers.Documentationhasa very

important role in detecting problems and cross_g

the information provided by the maintainers.

• The technical leader(s) of a release may have

left the company whereas another relense's technical

leader may still be contacted. This is a crucial element

since, as we will see, the causal analysis process will

involve project technical leader(s) and, depending on

Ms/her/theirlevelof controland knowledge,poss_ly

themaintainersthemselves.

5 Analysis of the problems that occurred while

performing the changes of the selected releases.

For each change (i.e., error correction,

enhancement, adaptation) in the selected release(s), the

following information should be acquired by

interviewing the maintainers and/or technical leaders

and by reading the related documentation (e.g., release

documents):

I1. Determine the difficulty or error-proneness of the

change.

I2. Determine whether the change difficulty could have

been alleviated or the error(s) resulting from the change

avoided and how?

I3. Evaluate the size of the change (e.g., # components,

LOCs changed, added, removed).

14. Assess discrepancies between initial & intermediate

planning and actual effort / time.

15. Determine the human flaw(s)(ifany)that originated

the error(s) or increased the difficulty related to the

change. A taxonomy of human errors is proposed in

Section 2.2.

I6. Determine the maintenance process flaws that led to

the identified human errors (if any). A taxonomy of

maintenance process flaws is proposed in Section 2.2.

17. Try to quantify the wasted effort and/or delay

generated by the maintenance process flaws (if any).

10022514L 2-5
The knowledge and understanding acquired

through steps 1-3 is necessary in order to understand,

interpret and formalize the information of type I2, I5 or

I6. As a guidance in conducting interviews, templates

of questions will be provided in Section 2.2•

6 Establish the frequency and consequences of

problems due to flaws in the organizational structure

and the maintenance process by analyzing the

information gathered in Step 5.

Based on these results, further complementary

investigations (e.g., measurement based) related to

specific issues that have not been fully resolved by the

qualitative analysis process, should be identified.

Moreover, a fast set of suggestions for maintenance

process improvement should be devised.

For those steps which are iterative, we map

the appropriate step back into the q,mUtative analysis

process (Figure 1). Thereby, we show how our

characterization process fits into the more general

qualitative analysis methodology presented above. In

this context, a step usually corresponds to a set of

iterations of the qualitative analysis process. Thus for

each step we have the input to that step which defines

the Observational Database (ODB), the output of each

step which contains the resulting characterization

models that go into the Interpretative Knowledge Base

(IKB), and a validation procedure which helps verify

that the characterization models are correct. The pieces

of information which compose the ODB are given in

decreasing order of importance at each step. The order

and content of the ODB varies at each step since the

analysis focus is progressively shifting [SS92].

Step 1: Model organizational structures

Input: maintenance standards definition document,

interviews, sample of release documents, organization

chart

Output: organizational model (roles, agents, teams,

information flow, etc.)

ValMat/on:

• Are all the standard documents and artifacts

included in the modeled information flow?

• Do we know who produces, validates, and

_es _ standarddocumentsand artifacts?

• Are all the people referenced in the release

documentsa partoftheorganizationscheme?

Steps 2, 3: Model process and map activities into

process phases

Input: maintenance standards definition document,

interviews, release documents

Output: process model

ValMatio_-

Are all the people in the process model a part

of the organization scheme?

Do the documents and artifacts included in the

process model match thoseof the information flow of

the organization model?

Is the mapping between activities and phases

complete, i.e., exhaustive set of activities, complete

mapping?

Ate the taxonomies of maintenance tools,

methods, and activities adequate, i.e., unambiguous,

disjoint and exhaustive classes?

Step 5: Perform causal analysis

Input: interviews, change request forms, release

documents, organization model, process model,

maintenance standardsdefinition document

Output: causal analysis

Validation:

Are the taxonomies of errors and

maintenance process flaws adequate, i.e., unambiguous,

disjoint and exhaustive classes? This is checkedagainst

actual change data and validated during interviews with

maintainers.

2.2 Guidelines and Taxonomies

This section presents a set of guidelines aimed at

facilitating the characterization process described in the

previous section. These guidelines are mainly

composed of taxonomiesdistinguishing maintenance

activities, errorsand maintenanceprocessflaws.In

addition,a setofquestionswhichcan be usedduring

maintainers' interviewsandforeachchangeisprovided.

Step 1: Identify organizational entities

Taxonomies of Maintenance Tools and Methods (Step

1.2)

The maintenance tools and methods available to

maintainers can be used to understand the maintenance

process, and identify potential sources of problems. The

following paragraphs represent the first level of

abstraction of environment characteristics' taxonomies

that should be used to characterize the change

fzamewcrk:

• Maintenance Tools: Impact analysis &

planning tool ; Tools for automated extraction &

reprosentation ofcontroland dataflows;Debugger;

Cross-referew.er ; Regression testing environment (data

generation, execution, and analysis of results) ;

Information system linking documentation and code.

• Maintenance Methods are characterized by the

following taxonomy: rigorous impact analysis,

planning, and scheduling procedures ; Systematic and

disciplined update procedures of the user and system

10022514L 2-6
documentation ; Communication channels and

procedures with the users ;

A Taxonomy of Maintenance Documentation (Step

1.3)

The type of docmnentation related to a software system,

which may be available to maintainers, can be defined

by a generic taxonomy as shown below.

Documentation has been described as one of the most

importantfactorsaffectingthemaintainability of a

softwaresystem [HA93, P94].This is why itis

importanttodefinepreciselywhat shouldbecontained

ina completesetofdocumentation(eitheron-lineor

off-line) formaintenance.Sucha taxonomycanbeused

as a guideline tOdefine the maintenance organization.

Also, when some of these documents appear to be

missing, potential sources of maintenance problems

may be identified. Based on the literatnre [BC91] on the

subject and our own experience, we propose the

following taxonomy:

• Product-related: Software requirements specifications ;

Software design specifications ; Software product

specifications

• Process-related: Test plans ; Configuration

management plan ; Quality assurance plan ; Software

development plan

• Support-related: Software user's manual ; Computer

systems operator's manual ; Software maintenance

manual ; Firmware support manual

Step 3: Identify the generic activities involved in each

phase.

Generic Description of Maintenance Activities (Step

3.1)

Acronym Activity

DET

SUB

UND

IA

CBA

AR

SC

CD

CC

UT

IT

Detennination of the need for a

change

Submission of change request

Understanding requirements of

changes: localization, change

design prototype

Impact analysis

Cost/benefit analysis

Approval/Rejection/priority

assignment ofchange request

Scheduling of task

Change design

Code changes

Unit testing of modifiedpans

i.e., has the change been

implemented?

integration testing,

RT

AT

USD

SA

IS

PIR

EDU

i.e., does the changed

part interface correcdy with the

system?

Regression testing,

i.e., does the change have any

unwanted side effects?

Acceptance testing

i.e., does the new release fulfill

the system requirements?

Update system & us_

documentation

Standards characterizations;

quality assurance procedures

Installation

Post-installation review of

changes

Education/training regarding the

application domain/system

All these activities usually contain an overhead of

commtmication (meeting + release document writing)

with owner/users, management hierarchy and other

maintainers which should be estimated. This is

possible, through data collection or by interviewing

maintainers (e.g., Delphi method).

Step 5: Perform causal analysis

Questions asked for each change in selected release(s)

(Items 11.14)

The following list describes a set of questions for

which answers can be provided by maintainers and/or

release standard documents. These questions attempt to

capturetheinformationnecessaryfortheidentification

of maintenance process flaws.

1 - Description of the change

1.1 Localization

subsystem(s) affected

medulc(s) affected

inputs/outputs affected

1.2 Size

LOCs deleted, changed,added

Modules examined, deleted, changed,

added

1.3 Type of change

Preventive changes: improvement of clarity,

maintainability or documentation.

Enhancement changes: add functionalities,

optimization of space/time/accuracy

•Adaptive changes: adapt system to change of hardware

and/or platform

• Corrective changes: corrections of development errors.

10022514L 2-7
2 - Description of the change process

2.1 effort, elapr,ed time

22 maintainers expertise and experience

How long has the person been working

on the system

How long has the person been working

in this application domain?

2.3 Did the change generate a change in any document?

Which doctmaent(s)?

3 - Description of the problem

3.1 Were some errors committed?

Description of the errors (see

taxonomiesbelow)

Perceivedcauseoftheerrors:

maintenanceIm:zessflaw(s)

3.2 Difficulty

What made the change _t?

What was the most difficult activity

associated with the change?

3.3 How much effort was wasted (if any) as a result of

maintenam_ process flaws?

3A What couldhave been done toavoidsome ofthe

difficulty, errors(ifany)?

Taxonomies of human errors (Item 15)

Note that we are exclusively refering to errors occuring

during the maintenance process, not errors resulting

from the development.

• Error Origin: when did the misunderstanding occur?

Change requirements analysis

Change localization analysis

Change design analysis

Coding

• Error domain: what caused it?

Lack of application domain knowledge:

operational constraints (user interface, performance),

mathematical model

imowledgLacke:of system designor implementation data structure or process dependencies,

performance or memory constraints, module interface

inconsistency

Ambiguous orincompleterequirements

Language misunderstanding <semantic,

syntax>

Schedule pressure

Existing uncovereA fault

Oversight.

Demmiming the origin and cause of the errors will help

determine their possible causal relationships to

maintenance process flaws in the taxonomy presented

below.

Taxonomy of Maintenance Process flaws (Item 16)

• Organizational flaws:

communication: Interface problems,

information flow "bottlenecks" in the communication

between the maintainers and the

management hierarchy

quality assurance (QA) team

configuration management team

roles:

prerogativesandresponsa_ilities are notfully

definedorexplicit

incompatibleresponsibilities, e.g.,

developmentandQA

processconformance:noeffectivestructmefor

enforcingstandardsandprocesses

• Maintenance methodological flaws

Inadequate change selection & priority

assignment process

Inaccurate methodology for planning of effort,

schedule,personnel

Itmconatemethodologyforimpactanalysis

Incomplete,ambiguousprotocolsfortransfer,

preservation and maintenance of system

knowledge

Incomplete, ambiguous definitions of

change requirements

Lack of rigor in configuration (versions,

variations) management and control

Undefined / unclear regression testing

success criteria.

• Resource shortages

Lack of financial resources allocated, e.g.,

necessary for preventive maintenance,

unexpected problems unforseen during

impact analysis.

Lack of tools providing technical support

(see previous tool taxonomy)

La_k of tools providing management

support (i.e., impact analysis, planning)

• Low quality product(s)

Looselydefinedsystemrequirements

Poorqualitydesign,codeofmaintained

system

Poor quality system documentation

Poor quality user docamentation

I0022514L 2-8
• Personnel-related issues

Lack of experience and/or ttaining with

respect to the application domain

Lack of experience and/or training with

respecttothesystemrequirements

O_dware, _omance) and deign

Lack ofexperienceand/ortrainingwith

respecttotheusers'operationalneedsand

constraints

In order to demons_ate the feasibility and usefulness of

the above approach, we present the following case

study.

3 A Case Study

This case study is intended to provide actual examples

and results of the change causal analysis process

described in previous sections. We first present the

maintained system used as a case study. Then, the

specific maintenance organization and process are

described in detail according to the template provided in

Section 2.1. Examples of change causal analyses are

shown and the lessons learned resulting from this

analysis l_ocess are presented.

3.1 System History and Description

GTDS is a 26 year old, 250 KLOC, FORTRAN orbit

determination system. It is public domain software and,

as a consequence, has a very large group of users all

over the world. Usually, 1 or 2 releases are produced

every year in addition to mission _c versions that

do not go intoconfigurationmanagement rightaway

(butare integratedlateron to a new versionby going

through the standard release process).Like most

maintainedsoftwaresystems,very few of the original

developersarc stillpresentin theorganization,but the

turnoverof the maintenance team islow compared to

othermaintenance organizations.However, turnover

stillremainsa crucialissueIn thiscnviromenC

3.2 Modeling of the

Organization and Processes

Maintenance

During the process of building a new release of GTDS,

different organizational entities interact in different

ways. By performing Step 1 of the characterization

processdescribedin Section2.1, two typesof entities

and five types of interactions(i.e.,differentiated

according to the purpose of tlae information flow) were

identified.

The entities, teams and groups, arc represented

in Figure 2 by boxes and ellipses, respectively. Teams

are persistent organizational structures; groups are

composed of members of several different teams, and

are dynamic entities in the. sense that they only exist

when group members meet. These groups have been

designedtofacilitate communication between teamsand

decisionmaking.

In the five interaction types identified,

information was used for the following purposes:

decision - decisionbased on information provided;

review -review of documents; approval - approvalof

documents or plans; transformation - supplied

information product is transformed into another

informationproduct; and information - disseminationof

information.

Teams."

• Testers: they present acceptance test plans,

perform acceptance test and provide change requests to

themaintainerswhen necessary.

Owners / Users: they suggest, control and

approveperformedchanges.

ProductAssurance Organization(PAO): They

control maintainers' work, e.g., conformance to

standards,attend release meetings, audit deLivery

packages.They have a differentmanagement from the

maintenanceteam.

ConfigurationManagement (FDCM): They

integrateupdates into the system. Coordinate the

production and releaseof versions of the system.

Providetrackingof changerequests.

Maintenance management: They grant

preliminaryapprovalsof maintenance change requests

and releasedefinitions.

Maintainers: They analyze changes, make

recommendations, perform changes, perform unit and

change validation testing after linking the modified

units to the existing system, perform validation and

regression testing after they get back the recompiled

system from the FDCM team.

Groups:

Software Management Planning Board

iSMPB): Their main goal is to address management

issues that run across maintenance projects. For

example, they help resolve conflicts between owners

and maintainers and review release planning docmnenB.

Also, they allow task leaders and higher level managers

to exchange relevant information about the evolution of

their respective systems. However, SMPB has no

official function• The board is composed of the task

leader, section manager, department manager, and

operationnmnager.

10022514L 2-9
P_qu_t= for

CCB/CMO _ in

11

SMPB

15

16

Audit rtlxrts

Tedmiml

LSSlli4_ Release

definition 116

document 15

7 All Release

.d

Reviews

New

112

FDCM }

Recompfled

New Rele_e

Infoamtion Flow Purpoe_

II: aplXoval 15: review 19: transiormadon 113: review 117: decisk_

12: infom'mion I_ informltion 110:. transformation 114: h'anslon'natio.r, 118: inlo_r, ation

13: rovk_ 17."_ II1: transformation 115: review

14: infom'zMion 18: _ IlO" Iransfom'ation 116: infon'xzation

Figure 2: Information Flow within

• Configuration Control Board and

Configuration Management Office ( CCB/CMO): They

arc officially responsible for all changes to configured

software and the allocated budget. Their goal is to

ensure that the production of new releases is consistent

with the long-term goals of the organization. It is

composed of high-level managers.

• GTDS user's group: It is a forum for

discussion of technical issues but has no official

function• It is composed of users, maintainers, and

testers.

The process described below represents our

understanding of the working process for a release of

GTDS and the mapping into standard generic activities.

This combines the in[otmation gained from Steps 2 and

3 of the characterization process. Phases, their

associated inputs/outputs and activities are presented

below. Activity acronyms are used as det-med in Section

2.2. In this case, each phase milestone in a release is

the Maintenance Organization

represented by the discussion, approval and distribution

of a specific release documcnL

I. Change analysis

Input: change requests from software owner + priority

list

Output: Release Content Review (RCR) document

which contains change design analysis, prototyping,

and cost/benefit analysis that may result in a priority

change to be discussed with the software owner/user.

Activities: UNDR, IA, CBA, CD, some CC, UT and

IT (for prototyping)

2. RCR meeting

Inpuu Release Content Review document proposed by

maintainers is discussed, i.e., change priority, content

of release.

Output: Updated Release Content Review document

Activities: AR, SA (QA engineers are reviewing the

release documents and attending the mee_g)

10022514L 2-10
3. Solution analysis

Input: Updated Release Content Review document

Output: devise technical solutions based on prototyping

analysis they performed in Step I, Release Design

Review (RDR)

Activities: SC, CD, CC, LIT, (preparation of test

strategy for) IT (based mainly on equivalence

partitioning)

4. RDR meeting

Input: RDR documentation

Output: approved (and possibly modified) RDR

documentation

Activities: review and discuss CC, UT, (plan for) IT,

SA

5. Change implementation and test

Input: RDR + prototype solutions (phases 1, 3)

Output: changes are completed ; change validation test

is performed with new compiled components linked to

unchanged components of the current system version ;

regression testing is performed on the system

recompiled from scratch (provided by the _ team)

; a report with the purpose of demonstrating that the

system is ready for acceptance test is produced:

Acceptance test readiness review document (ATRR)

Activities: IT, RT, USD

6. ATRR meeting

Input: Acceptance test readiness review document

Output: The changes are cU_ and vafidated and the

used testing strategy is discussed. The acceptance test

team presents its acceptance testing plan.

Activities: review the current output of IT, SA

7. Acceptance test

Input: the new GTDS release and all release

documentation

Outputs: A list of Software Change requests (SCRs) is

provided to the maintainers. These changes correspond

to inconsistencies between the new release and the

general system requirements.

Activities: AT

Step 1, 2, and 3 required several iterations before there

was sufficient validation of the resulting

characterization of the organization, phases and

documents. As part of Step 2, for each of the standard

documents generated during the releases of GTDS

studied, we determine who produces it, who approves

it, and what additional relevant information and data

they contain. When doing so, we have to look for

possible inconsistencies between the organization

model (Step 1) and the identified producers/approvers of

the documents.

• Document 1: Release Content Review (RCR):

Producer: maintenance team

Approvers: users, maintenance management, CCB

Content:

• change requirement description

• description of error (if any) that originated the change

•design of a prototype solution

• schedule, effort plans

• impact analysis assessment

• Document 2: Release Design Review (RDR):

Producer: maintenance team

Approvers: users, CCB

Content:

•identification of modified units

• a definitive solution is proposed

• rough cost/schedule estimates

• testing guidelines: mainly equivalence partitioning

classes

• definition of the test success criterion

* Document 3: Acceptance Test Readiness Review

(ATRR):

Producers: m,qmtenance team, acceptance test team (test

plan)

Approvers: CCB, testers

Content:

• results of test cases and benchmarks (regression

testing)

• screen printouts, short reports

• Acceptance test plan

•Document 4: Delivery package:

Producer: maintenance team

Approvers: CCB

Content:

• cause of error (if any)

. effort breakdown: analysis, design, code, test

. # components examined, modified, added, deleted.

• #Locs modified, added, deleted

As specified in Step 4 of our process, we selected a

release for analysis. This release was quite recent, most

of the documentation identified in Step 2 was available,

and most importantly, the technical leader of the release

was available for additional insights and information.

Step 5 involved a causal analysis of the problems

observed during maintenance and acceptance test of the

releases studied. These problems were linked back to a

precise set of issues belonging to taxonomies presented

in Section 2.2. Figure 3 summarizes Step 5 as

instantiated for this ease study.

10022514L 2-11
Inputs Outputs

•Organization

• Process

• Resources

• Products

•Personnel

Causallink

Maintenance

process

execution

Proble_

Causal Analysis

S/W

User's guide

Reported

errors

testing

Figure 3: Causal Analysis in GTDS

In orderto illustrateStep 5,we provide below

an example of causalanalysisforone of thechanges in

the selected release. Implementation of this change

resulted in 11 errors that were found by the acceptance

test team, 8 of which had to be corrected before final

delivery could be made. In addition, a substantial

amount of rework was necessary. Typically, changes do

not generate so many subsequent errors, but the flaws

that were present in this change are representative of

maintenance problems in GTDS. In the following

paragraphs, we discuss only two of the errors generated

by the change studied.

Increaseddifficultyrelatedtochange(rework)

Description: Initially, users requested an

enhancement to existing GTDS capabilities (change

642). The enhancement involved vector computations

performed over a given timespan. This enhancement

was considered quite significant by the maintainers, but

usersfailedto supplyadequaterequirementsand didnot

attendthe RCR meeting. Users did not reporttheir

dissatisfactionwith the design untilATRR meeting

time, at which time requirements were rewritten and

maintainers had to perform rework on their

implementation. This change took a total of 3 months

to implement, of which at least 1 month was attributed

to several flaws in the process.

Maintenance process flaw(s): organizational: a

lack of cleat defmitious of the prerogatives/duties of

users with respect to release document reviews and

meetings (roles), and a lack of enforcement of the

release procedure (processconformance);maintenance

methodological flaw:incomplete, ambiguous

det-mitious of change requirements.

• Errors caused by change 642

The implementation of the change itself resulted in an

error (A1044) found at the acceptance test phase. When

the correction to A1044 was tested, an error (A1062)

was found that could be _ back to both 642 and

A1044.

A1044

. Description: Vector computations at the endpoints of

the timespan were not handled correctly. But in the

requirements it was not clear whether the endpoints

should be considered when implementing the solution.

. Error origin: change requirement analysis

• Error domain: ambiguous and incomplete

requirements

• Maintenance process flaw(s): organizational:

communication between users and maintainers, due in

part to a lack of def'med standards for writing change

requirements; maintenance methodological flaw:

incomplete, ambiguous definitions of change

requirements.

A1062

• Description: One of the system modules in which the

enhancement change was implemented has two

processing modes for data. These two modes are listed

in the user manuai. When run in one of the two

possible processing modes, the enhancement generated

a set of errors, which were put under the heading

A1062. At the phase these errors were found, the

enhancement had already successfully passed the tests

fortheotherprocessingmode. The maintainershould

have designed a solution to handle both modes

corre_y.

•Error origin." change design analysis.

•Error domain." lack of application domain knowledge.

10022514L 2-12
•Maintenance process flaw(s): pe_onnel-related: lack of

experience and/or training with respect to the

application domain.

The next section presents in detail the results of

performing Step 6.

3.3 Lessons Learned about the Studied

Maintenance Project

The lessons learned are classified according to the

taxonomy of maintenance flaws defined in Section 22..

By performing an overall analysis of the change causal

analysis results (Step 6), we abstracted a set of issues

classified as follows:

Organization

• There is a large communication cost overhead

between maintainers and users, e.g., release standard

documentation, meetings, management forms. In an

effort to improve the communication between all the

participants of the maintenance process, non-technical,

communication-oriented activities have been

emphasized. At first glance, this seems to represent

about 40% (rough approximation) of the maintenance

effort. This figure seems excessive, especially when

considering the apparent communication problems

(next paragraph).

• Despite the number of release meetings and

documents, disagreements and misunderstandings seem

to disturb the maintenance process until late in the

release cycle. For example, design issues that should be

settled at the end of the RDR meeting keep emerging

until acceptance testing is completed.

As a result, it seems that the administrative

process and organization scheme should be investigated

in order to optimize communication and sign-off

procedures, especially between users and maintainers.

Process

• The tools and methodologies used have been

developed by maintainers themselves and do not belong

to a standard package provided by the organization.

Some ad hoc technology transfer seems to take place in

order to compensate for the lack of a global, commonly

agreed upon strategy.

• The task leader has been involved in the

maintenance of GTDS for a number of years. His

expertise seems to compensate for the lack of system

documentation. He is also in charge of the training of

new personnel (some of the easy changes are used as an

opportunity for training). Thus, the process relies

heavily on the expertise of one or two persons.

• The fact that no historical database of changes

exists makes some changes very difficult. Maintainers

very often do not understand the semantics of a piece of

code added in a previous correction. This seems to be

partly due to emergency patching (during a mission)

which was not controlled and cleaned up afterwards (this

has recently been addressed), a high turnover of

personnel and a lack of written requirements with

respect to performance, precision and platform

configuration constraints.

• For many of the complex changes,

requirements are often ambiguous and incomplete, from

a maintainer's perspective. As a consequence,

requirements are often unstable until very late in the

release process. While prototyping might be necessary

for some of them, it is not recognized as such by the

users and maintainers. Moreover, there is no well

defined standardfor expressing change requirements in a

way suitable to both maintainers and users.

Praa_s

• System documentation (besides the user's

guide) is not fully maintained and not trusted by

maintainers. Source code is currently the only reliable

source of information used by maintainers.

• GTDS has a large number of users. As a

consequence, the requirements of this system are

complex with respect to the hardware configurations on

which the system must be able to run, the performance

and precision needs, etc. However, no requirement

analysis document is available and maintained in order

to help the maintainers devise optimal change

solutions.

• Because of budget constraints, there is no

document reliably defining the hardware and precision

requirements of the system. Considering the large

number of users and platforms on which the system

runs, and the rapid evolution of users' needs, this would

appear necessary in order to avoid confusion while

implementing changes.

People

• There is a lack of understanding of operational

needs and constraints by maintainers. Release meetings

were supposed to address such issues but they seem to

be inadequate in their era'rant form.

• Users are mainly driven by short term

objectives which are aimed at satisfying particular

mission requirements. As a consequence, there is a very

limited long term strategy and budget for preventive

maintenance. Moreover, the long term evolution of the

system is not driven by a well defined strategy and

maintenance priorities are not clearly identified.

AS a general set of recommendations and based

on the analysis presented in this paper, we suggest the

following set of actions:

• A standard (that may simply contain

guidelines and checklists) should be set up for change

10022514L 2-13
requirements. Both users and maintainers should give

their input with respect to the content of this standard.

• The conformancetothedefinedreleaseprocess

should be improved,e.g.,throughteam building,

training.Inotherwords,thereleasedocumentsand

meetingsshouldmore effectivelyplaytheirspecified

rolein the process,e.g.,the RDR meetingshould

alldesigndisagreementsand inconsismncies.

• The partsofthesystemthatareoftenchanged

and highly convoluted(as a resultof numerous

modifications) shouldberedesignedanddocumentedfor

more productiveand reliablemaintenance.Technical

taskleadershouldbe abletopointoutthesensitive

systemunits.

4 Conclusion

Characterizing and understanding software maintenance

processes and organizations are necessary, if effective

management decisions are to be made and if adequate

resource allocation is to be provided. Also, in order to

plan and efficiently organize a _nt program--

a necessary step towards process improvement

[BR88] n, we need to better characterize the

maintenance environment and its related issues. The

difficuRy of performing such a characterization stems

from the fact that the people involved in the

maintenance process, who have the necessary

information and knowledge, cannot perform it because

of their inherently biased perspective on the issues.

Therefore, a well defined characterization process, which

is cost-effective, objective, and applicable by outsiders,

needs to be devised.

In this paper, we have presented such an

empirically refined characterization process which has

allowed us to gain an in-depth understanding of the

maintenance issues involved in a particular project, the

GTDS project. We have been able to gather objective

information on which we can base management and

technical decisions about the maintenance process and

organization. Moreover, this process is general enough

to be followed in most of the maintenance

organizations.

However, such a qualitative analysis is a priori

limited since it does not allow us to quantify precisely

the impact of various organizational, technical, and

process related factors on maintenance cost and quality.

Thus, the planning of the release is sometimes

arbitrary, monitoring its progress is extremely difficult,

and its evaluation remains subjective.

Hence, there is a need for a data collection

program for GTDS and across all the maintenance

projects of our organization. In order to reach such an

objective, we will base the design of such a

measurement program on the results provided by this

study. In addition, we need to model more rigorously

the maintenance organization and processes so that

precise evaluation criteria can be defined [SB94].

This approach will be used to analyze several

other maintenance projects in the NASA-SEL in order

to better understand project similarities and differences

in this environment. Thus, we will be able to build

better models of the various classes of maintenance

projects.

Acknowledgments

We are grateful to Steve Condon, Walcelio Melo,

Carolyn Seaman, Barbara Swain and Ion Valett for

reviewing early drafts of this paper. We also would like

to thank Amy Bleich for helping us to analyze the

release documents.

References

['BC91] K. Bennett, B. Cornelius, M. Munro, D.

Robson, "Software Maintenance", Software

Engineering Reference Book, Chapter 20, ButterworthHeinemann Lid, 1991

[BR88] V. Basili and H. Rombach,"The TAME

Project: Towards Improvement-Oriented Software

Environments', IEEE Trans. Software Eng., 14 (6),

June, 1988.

[(288] N. Chapin, "The Software Maintenance LifeCycle", CSM'88, Phoenix, Arizona, 1988.

[HA93] C. Hartzman, C. Austin, "Maintenance

Productivity: Observations Based on an Experience in a

Large System Environment", CASCON'93, Toronto,

Canada, 1993

[HV92] M. Hariza, J.F. Voidrot, E. Minor, L.

Pofelski, and S. Blazy, "Software Maintenance: An

analysis of Industrial Needs and Constraints", CSM_)2,

Orlando, Florida.

[P94] D. Parnas, "Software Aging", ICSE 16th,

May 1994, Sorrento, Italy.

[RUV92] D. Rombach, B. Ulery and J. Valett'

"Toward Full Cycle Control: Adding Maintenance

Measurement to the SEL", Journal of systems and

software, May 1992.

[SB94] C. Seaman, V. Basili, "OFF: An Approach to

Organizational and Process Improvement", AAAI 1994

Spring Symposium Series, Stanford University, March

1994.

[SS92] A, Shelly, E. Sibert, "Qualitative Analysis: A

Cyclical Process Assisted by Computer", Qualitative

Analysis, pp 71-114, Oldenbourg Verlag, Munich,

Vienna, 1992

10022514L 2-14
Defining and Validating High-Level Design

Metrics1

Lionel Briand*,sandro Morasca**, VictorR. Basili*

* Computer Science Department

University of Maryland, College Park, MD, 20742

{lionel, basili} @cs.umd.edu

** Dipartimento di Elettronica • Informazione

Politeenico di Milano

Piazza Leonardo da Vinci 32, 1-20133 Milano, Italy

morasca@elet.polimi.it

Abstract

The availability of significant metrics in the early phases of the software development

process allows for a better management of the later phases, and a more effective quality

assessment when software quality can still be easily affected by preventive or corrective

actions. In this paper, we introduce and compare four strategies for defining high-level

design metrics. They are based on different sets of assumptions (about the design process)

related to a well defined experimental goal they help reach." identify error-prone software

parts. In particular, we define ratio-scale metrics for cohesion and coupling that show

interesting properties. An in-depth experimental validation, conducted on large scale

projects demonstrates the usefulness of the metrics we define.

1 Introduction

Software metrics can help address the most criticalissuesin software development and

provide support for planning, predicting,monitoring, controlling,and evaluating the

qualityof both software products and processes [BR88, F91]. Most existingsoftware

metricsattemptto capturecharacmristicsofsoftwarecode ['b-x) 1];however, softwarecode is

justone of the artifactsproduced during software development, and, moreover, itisonly

availableat a latestage.Itiswidely recognized that the production of bettersoftware

requiresthe improvement of theearlydevelopment phases and the artifactsthey produce:

1 This work was supported in part by NASA grant NSG--5123, UMIACS, and NSF grant 01-5-24845.

Sandm Morasca was also supported by grants from MURST and CNR.

10022514L 2-15
The production of better specifications and better designs reduces the need for extensive

review, modification, and rewriting not only of code, but of specifications and designs as

well. As a result, this allows the software organization to save time, cut production costs,

and raise the final product's quality.

Early availability of metrics is a key factor to a successful management of software

development, since it allows for

• early detection of problems in the artifacts produced in the initial phases of the lifecycle (specification and design documents) and, therefore, reduction of the cost of

change---late identification and correction of problems are much more cosily than

early ones;

• better software quality monitoring from the early phases of the life-cycle;

• quantitative comparison of techniques and empirical refinement of the processes to

which they are applied;

• more accurate planning of resource allocation, based upon the predicted errorproneness of the system and its constituent parts.

In this paper, we will focus on high-level design metrics for software systems. A number

of studies have been published on software design metrics in recent years. It has been

shown that system architecture has an impact on maintainability and error-proneness

[HK84, G86, R87, R90, $90, SB91, Z91, AE92, BTH93, BBH93]. These studies have

attempted to capture the design characteristics affecting the ease of mainminlng and

debugging a software system. Most of the design metrics are based on information flow

between subroutines or declaration counts. We think that, even though it provides an

interesting insight into the program structure, this should not be the only strategy to be

investigated, since many other types of program features and relationships are a priori

worth studying. Moreover, there is a need for comparison between strategies in order to

identify worthwhile research directions and build aceur_ prediction mode, Is.

Besides this focus on information flow, most of the existing approaches share two

common characteristics. (1) They define mettles without making clear assumptions about

the contexts (i.e., processes, problem domain, environmental factors, etc.) in which they

can be applied (with the exception of [AE92], where this issue was partially addressed).

This implies they should have general validity, and be applicable to different environments

and problem domains. (2) There are not fully explicit goals, for whose achievement the

metrics are defined. This may cause problems in their application, since they may be

defined based on implicit assumptions which the context may not satisfy; interpretation,

10022514L 2-16
since their meaning is not clear; and validation [IS88, K88], since their relevance with

respect to a clearly stated goal is not established.

The definition of universal metrics (like in physical sciences) is an acceptable longterm goal, which, however, is only achievable after we gain better insights into specific

processes from specific perspectives in the short term. It is our opinion that the definition

of a metric should be driven by both the characteristics of the context or family of contexts

in which it is used, and one or more clearly stated goals that it helps reach. In other words,

the assumptions underlying the defined metrics should rely on a deep knowledge of the

context and should be precisely related to a stated goal. After this, the defined metrics must

undergo a thorough experimental validation, to assess their significance and usefulness

with respect to the stated goals. Last, based on the experimental evidence, metrics may be

refined and modified, to better achieve the goals and comply with the process

characteristics.

The goal of the research documented in this paper is to define and validate a set of

high-level design metrics to evaluate the quality of the high-level design of a software

system with respect to its error-proneness, understand what high-level design

characteristics are likely to make software error-prone, and predict the error-proneness of

the code produced.

We introduce four families of metrics, which are based on different types of

mathematical abstractions of program designs [MGBB90]. In particular, we introduce a

family of metrics based on data declaration dependency _ (Section 2.2.4). This strategy

allows us to introduce metrics for cohesion (Section 2.2.4.1) and coupling (Section

2.2.4.2) [F91] that are characterized by interesting properties and are based on consistent

principles. Such a consistency is important because it should facilitate furore research on

quantitative tradeoff mechanisms between coupling and cohesion, i.e., variations can be

expressed using consistent measurement units. Other metric families include: metrics based

on declaration counts (Section 2.2.1), metrics based on the USES relationships between

modules [GJM92] (Section 2.2.2), and metrics based on the IS_COMPONENT_OF

relationships [GJM92] (Section 2.2.3).

In addition, we experimentally compare and validate the metrics introduced in

Section 2 on three NASA projects. The results are shown in Section 3. In Section 4, we

summarize the lessons we have learned, and outline directions for future research activities

based on these lessons.

10022514L 2-17
2 Defining Metrics for High-level Design

In this section, we first introduce the basic concepts of high-level design and the

terminology we wiLl use in the paper (Section 2.1). We then define, based on the goals

stated in Section 1 and context assumptions, four families of high-level design metrics

(Section 2.2).

2.1 Basic Definitions

Our object of study is the high-level design of a software system. To define it, we will start

from its elementary constituents: software modules.

In the literature, there are two commonly accepted definitions of modules. The first

one sees a module as a routine, either procedural or functional, and has been used in most

of the design measurement publications [M77, CY79, HK84, R87, $90]. The second

definition, which takes an object-oriented perspective, sees a module as a collection of

type, data, and subroutine definitions, i.e., a provider of computational services [BO87,

G_d92]. In this view, a module is the implementation of an Abstract Data Type / Object. In

this paper, unless otherwise specified, we will use the term subroutine for the first

category, and reserve the term module for the second category. Modules are composed of

two parts: interface and body (which may be empty). The interface contains the

computational resources that the module makes visible for use to other modules. The body

contains the implementation details that are not to be exported.

At a higher level of abstraction, modules can be seen as the components of higher

level aggregations, as defined below.

Definition 1: Library Module Hierarchy (LMH).

A library module hierarchy is a hierarchy where nodes are modules and subroutines, arcs

between modules are IS_COMPONENT_OF [G31VI92] relationships, and there is just one

top level node, which is a module.

0

In the remainder of this paper, we will define concepts and metrics that can be applied to

both modules and LMHs, which are the most significant syntactic aggregation levels below

the subsystem level. For short, we will use the term software part (sp) to denote either a

module or an LMH.

In the high-level design phase of a software system, only module and subroutine

interfaces and their relationships am defmed--module body and subroutine detail de.sign is

I0022514L 2-18
carried out at low-level design time. Therefore, we define the high-level design of a

software system as follows.

Definition 2: High-level Design

The high-level design of a software system is a collection of module and subroutine

interfaces related to each other by means of USES [GJM92] and IS_COMPONENT_OF

relationships. No body information is yet available at this stage.

0

2.2 Strategies to Define High-level Design Metrics

In this section, we investigate several strategies for defining high-level design metrics. This

appears necessary at this stage of knowledge, where we can only rely on very limited

theoretical and empirical ground to help us identify interesting concepts, relationships and

objects of study. One of the results of this investigation is to provide directions to focus our

research on a smaller set of strategies and concepts.

Some of the concepts introduced in this section cannot be directly mapped onto all

imperative languages, because not all of them allow the implementation of Abstract Data

Types/Objects. However, these concepts are shared by many modern programming

languages.

As we said in the Introduction, context assumptions are necessary to define metrics

that are applicable and useful. Therefore, we list a context assumption for each of the

metrics of the four strategies we introduce below. We do not assume that all of these

process assumptions axe equally important, i.e., not all of the process characteristics we

take into account have an equal impact on software error-proneness.

2.2.1 Declaration Counts

These metrics are counts of data declarations, associated with a software part, that are

imported, exported or declared locally.

I

Metric 1: Local [

Local(sp) will denote the number of locally defined data declarations of a software part sp. I

lO022S14L 2-19
Assumption A-LO.

The count of declarations of a software part may be seen as a measure of size, which is

known to be associated with errors, i.e, the larger the set of declarations, the more likely

the errors.

Metric Global(sp)2: Global. will denote !

the number of external data declarations visible from a software part

sp.

Assumption A-GL

The larger the number of external declarations visible in a software part, the larger the

number of external concepts to be understood and used consistently, the higher the

likelihood of error.

0

!

Metric 3: Scope. I

I

Scope(sp) will denote the number of external data declarations for which the data

declarations of a software part sp are visible.

Assumption A-SC.

The larger the number of data declarations in the scope of the software part, the larger the

number of contexts of use, the more likely it is to be inadequate to fulfill the needs of the

declarations in the scope.

0

2.2.2 Metrics Based on the USES Relationships

These metrics capture the dependencies between software parts based on the USES

relationships of the system.

Metric 4: Imported Software Parts. [

ISP(sp) will denote the number of software parts imported and used by a software part sp.

10022514L 2-20
Assumption A-ISP.

The larger the number of used external software parts, the larger the context to be

understood, the more likely the occurrenceof an error.

0

Metric 5: E,vorted Parts.

Software

ESP(sp) willdenote thenumber of softwarepartsthatuse a softwarepartsp.

Assumption A-ESP.

The larger the number of contexts of use of a software part, the larger the number of

services it provides, the more flexible it must be, and, as a consequence, the more likely the

occurrence of error.

0

2.2.3 Metrics Based on the IS_COMPONENT_OF Relationships

These metrics capture information about the stmctme of the IS_COMPONENT_OF graph.

Metric 6: Maximum/Average Depth.

Max_Depth(sp) IAvg_Depth(sp) will denote the maximum/average depth of the nodes

composin_ a softwarepartsp.

Assumption A-MIA.

The larger the depth of a hierarchy, the larger the context information to be known in the

lower nodes, the more likely the occurrence of error.

0

Metric 7: Number of paths. !

No Paths(sp) will denote the number of complete paths (from root to leaf) within a a

software part sp.

Assumption A-NOP.

The larger the number of paths, the larger the number of parent, sibling, and child

relationships to be dealt with, the larger the complexity of the hierarchy, the higher the

l_lihood of error occamence.

0

10o22s14L 2-21
2.2.4 Interaction-Based Metrics

In this section, we focus specifically on the dependencies that can propagate inconsistencies

from data declarations to data declarations or subroutines when a new software part is

integrated in a system. Those relationships will be called interactions and will be used to

define metrics capturing cohesion and coupling within and between software parts,

respectively. (Interactions linking subroutines to subroutines or data declarations will not

be considered because they are, in the vast majority of cases, encapsulated in module or

routine bodies and are therefore not detectable in our framework, which only takes into

account high-level design.)

Definition 3: Data declaration-Data declaration (DD ) Interaction.

A data declaration A DD-interacts with another data declaration B if a change in A's

declaration or use may cause the need for a change in B's declaration or use.

0

The DD-interaction relationship is transitive. If A DD-interacts with B, and B DD-interacts

with C, then a change in A may cause a change in C, i.e., A DD-interacts with C. Data

declarations can DD-interact with each other regardless of their location in the designed

system. Therefore, the DD-interaction relationship can link data declarations belonging to

the same software part or to different software parts.

The DD-interaction relationships can be defined in terms of the basic relationships

between data declarations allowed by the language, which represent direct DD-interactions

(i.e., not obtained by virtue of the transitivity of interaction relationships). Dam declaration

A directly DD-interacts with data declaration B if A is used in B's declaration or in a

statement where B is assigned a value. As a consequence, as bodies are not available at

high-level design time, we wRl only consider interactions detectable from the interfaces.

DD-interactions provide a means m representthedependency relationships between

individual data declarations. Yet, DD-interactions per se are not able to capture the

relationships between individual data declarations and subroutines, which are useful to

understand whether data declarationsand subroutinesare rela_ed to each otherand therefore

should be encapsulatedintothe same module (seeSection2.2.4.1on cohesion).

Definition 4: Data declaration-Subroutine (DS) lnteractiom

A data declaration DS-interacts with a subroutine if it DD-interacts with at least one of its

data declarations.

0

I0022514L 2-22
Wbenevea" a data declaration DD-in_racts with at least one of the data declarations contained

in a subroutine interface, the DS-interaction relationship between the data declaration and

the subroutine can be detected by examining the high-level design. For instance, from the

Ada-lke code fragment in Fignm 1, it is apparent that both type I"1 and object OBJECTll

DS-interact with procedure SRll, since they both DD-interact with parameter PARll,

procedure SR/I's interface data declaration.

package M 1 is

oo.

type T1 Is ...;

OBJECTll, OB_2: T1;

procedure $Rll(PARll: In TI:--OBJECTll);

oo.

package MY. is

OBJECT13: "1"1;

type T2 Is array (1..100) of T1;

OBJECT21: T2;

procedure SR21(PAR21: in out "/'2);

end M2";

OBJEC'T22: M2.T2;

end MI;

Figure 1. Program fragment

For graphical convenience, both sets of interaction relationships will be represented by

directed graphs, the DD-interaction graph, and the DS-interaction graph, respectively. In

both graphs (see Figure 2, which shows DD- and DS-interaction graphs for the code

fragment of Figure 1), data declarations are represented by rounded nodes, subroutines by

thick lined boxes, modules by thin lined boxes, and interactions by ares.

Next, we will define high-level design metrics for cohesion and coupling, based on the

above definitions. It is generally acknowledged that system architecture should have low

coupling and high cohesion [CY79]. This is assumed to improve the capability of a system

to be decomposed in highly independent and easy to understand pieces. However, the

reader should bear in mind that high cohesion and low coupling may be conflicting goals,

i.e., a trade-off between the two may exist. For instance, a software system can be made of

small modules with a high degree of internal cohesion but very closely related to each other

and, therefore, with a high level of coupling. Conversely, a software system can be

composed of few large modules, representing its subsystems, loosely related to one

another, i.e., with low coupling, but with a low degree of internal cohesion as well.

I0022514L 2-23
(a) fb)

Figure 2. DD-interaction (a) and DS-interaction Co) graphs for the program fragment in

Hgure 1

Moreover, high cohesion and low coupling are not the only factors to be taken into account

when designing a software system. Other issues (e.g., potential muse) must be taken into

account as well.

2.2.4.1 Cohesion

Cohesion captures the extent to which, in a software part, each group of data declarations

and subroutines that are conceptually related belong to the same module. Based on

• an assumption (A-CH), which provides the rationale to &fine cohesion metrics;

• the concept of cohesive interactions, Le., those interactions which contribute to

cohesion;

• a set of properties (Properties 1-3) that cohesion metrics must have in order to

measme cohesion

we now introduce a set of metrics fMetrics 8-11) to measure the degree of cohesion of a

software part.

Assumption A-CH:

A high degree of cohesion is desirable because information related to declaration and

subroutine dependencies should not be scattered across the system and among irrelevant

10022.514L 2-24
information.Data declarationsand subroutineswhich are not relatedto each other should

be encapsulatod to the extent possible into different modules. As a result of such a strategy,

we expect the software parts to be less error-prone.

0

Consistently with the definition of Abstract Data Type/Object, data declarations and

subroutines should show some kind of interaction between them, if they are conceptually

related. Therefore, we are interested in evaluating the tightness of the interactions between

the data declarations and data declarations or subroutines declared in a module interface.

We will capture this by means of cohesive interactions.

Definition 5: Cohesive Interaction.

The set of cohesive interactions in a module m, denoted by el(m), is the union of the sets

of DS-interactions and DD-interaetions, with the exception of those DD-interactions

between a data declaration and a subroutine formal parameter.

0

We do not consider the DD-interactions linking a data declaration to a subrouline parameter

as relevant to cohesion, since they are already accounted for by DS-interactions and we are

interested in evaluating the degree of cohesion between data declarations and routines seen

as a whole. Furthermore, eobesive interactions occur between data declarations and

subroutines belonging to the same module. Interactions across different modules are not

considered cohesive, since cohesion is the extent to which data declarations and

subroutines that are conceptually related belong to the same module. Interactions across

different modules contribute to coupling. Therefore, given a software part sp, the sets of

cohesive interactions of its constituent modules (if any) are disjoint.

Remark.

It is worth reminding the reader that those relationships that cannot be detected by

inspecting the interfaces,i.e.,global variablesinteractingwith subroutine bodies, can

actually be quite relevant to cohesion evaluation, because they often represent the

connections between an objectand the subroutinesthatmanipulam it.This issuewill be

further discussed later in this section.

0

10022514L 2-25
We base our cohesion metrics for software parts on cohesive interactions. Before defining

them, we introduce the following three properties that they must satisfy in order to match

Our assumptions 1.

Property 1: No_tion.

Given a software part sp, the metric cohesion(sp) belongs to a specified interval [O, MaxI,

and cohesion(sp)= 0 ifand only ffCl(ap)isempty, and cohesion(sp)= Max ffand only if

Cl(sp)includesallpossiblecohesive interactions.

0

Normalization allows meaningful comparisons between the cohesions ofdifferentsoftware

parts,since they allbelong to the same interval,and the extreme values of the cohesion

range must represent the extreme cases. We will denote by M(sp) the maximal set of

cohesive interactionsof the software partsp, Le.,the setthatincludes allof sp's possible

cohesive interactions,obtained by linking every data declaration to every other data

declarationand subroutinewith which itcan interact.Some care must be used in defining

M(sp) for languages thatallow circulartype defmitions,such as the ones used to definethe

nodes of a linked list.In thiscase,thedeclarationsof two typesTI and T2 are builtinsuch

a way thatTI interactswith T2 and T2 interactswith TI. We choose to count only one

interaction between them. This is explained by the fact that a single interaction between two

datadeclarationsjustifiestheirencapsulationina singlemodule/AbstractData Type.

Property 2: Monotonicity.

Let spl be a software partand Cl(spl)itssetof cohesive interactions.Ifsp2 isa modified

version of spl with the same sets of data and subroutine declarationsand one more

cohesive interaction so that Cl(sp2) includes Cl(sp]), then cohesion(sp2) >_cohesion(spl).

0

Adding cohesive interactions to a a software part cannot decrease its cohesion.

Property 3: Cohesive Modules.

Let sp be a software part, and let ml and m2 be two of its modules. Let sp' be the software

part obtained from sp by merging the declarations belonging to ml and m2 into a new

module m. If no cohesive interactions exist between the declarations belonging to ml and

m2 when they are grouped in m, then cohesion(sp) >_cohesion(spg.

0

1properties and metrics can be defined for module sets more general than software parts. However, for

simplicity, we wR1provide them only for software parts.

10022.514L 2-26
Splitting two sets of declarations which are not related to each other via cohesive

interactions into two separate modules cannot decrease the cohesion of the software part.

Based on the properties defined above, we introduce a cohesion metric for software

parts.

Metric 8: Ratio of Cohesive Interactions (RC1) for a Software Part.

The Ratio of Cohesive Interactions for sp is

RCI" _lCI(sp)l

tsp)- _l(sp)l (*)

It is straightforward to prove that RCI(sp) satisfies the above properties 1-3, and that,

based on properties 1-3, it is def'med on a ratio scale IF91]. Furthermore, RCI(sp) can also

be computed as a weighted sum of the RCI(m)'s of the single modules m belonging to sp.

From Formula (*), since cohesive interactions only occur within modules, but not across

modules

ICI(sp)l = XL'ICm)l

me sp

IM(sp)l = _[M(n)l

n_ sp

SO

m¢_spn ICI(m)l

RCI(sp)= XlM(n)l

E sp

By multiplying and dividing each term in the summation by IM(m)l, we obtain

m_stlllM(m)l ICI(m)l m_sl9 IM(m)l RCI(m)

RCI(sp)= EIM(n)I _ - _iMCn)l

sp ¢ sp

The weights represent the potential contribution of each module m belonging to the

software part sp to the cotmsion of the whole sp.

10022.514L 2-27
Figure 3 shows an example of cohesion computation for a singlemodule. T denotes

a type declaration, C a variable declaration,and SRI, SR2, and SR3 subroutine

declarations.

RCI = 4/7= 0.571

Figure 3.Cohesion example

Based on the above cohesion metric, we can define a threshold for deciding whether a set

of data and subroutines should be kept in one single module or divided into two or more

modules. For simplicity, we will show here only the case in which we have to decide

whether the declarationsbelonging toa module m should be splitintotwo modules mI and

m2. This should be the case ffthe cohesion of the software part consistingof the two

modules m] and m2 isgreaterthan thecohesion of module m, i.e.,

ICI(m 1)l+lCI(m2)l

IM(ml)_gCl(m2)l >

ICI(m 1)I+ICI(m2)I+ICI121

_¢l(m)l

where ICI121 is the number of cohesive interactions between the declarations belonging to

modules ml and m2 when they are in module m. Based on the above inequality, we can

define a threshold on _I121, as follows

(IM(m)t-lM(ml)i-IM(m2)O (_I(ml)l+lCI(m2)D

IM(ml)h-iM(m2)l > ICI121

We want to emphasize, however, that, since cohesion is not the only characteristic relevant

to software design, its increase should not be used as the only criterion on which to base

such a decision.

10022514L 2-28
The Role of Additional Information

Additional information to what is visible in the interfaces may be available at the end of

high-level design. For instance, given the interface of a module m, the designers have at

least a rough idea of which objects declared in m will be manipulated by a subroutine in

m's interface. It will be left to the person responsible for the metric program to decide

whether or not it is worth collecting this kind of information, thus making the designer

describe which objects will be accessed by which subroutines. Formatted comments may

be a convenient way of conveying this information through module interfaces and therefore

of automating the collection of this type of information.

For instance, from the code fragment in Figure 1, we cannot tell whether

OBJECT12 DS-interacts (as a global variable) with subroutine SRll. In this case,

designers can answer in three different ways:

(1)

(2)

(3)

OBJECT12 DS-interact with Pl l

OBJECT12 does not DS-interact with Pll

the information they have is not sufficient

It is worth saying that answers of kind (2) provide valuable, though negative, information

on the DS-interactions present in a system. For instance, in the code fragment on Figure 1,

the designer may indicate the existence of a DD-interacfion between object OBJECT12 and

PAR11 and the lack of interaction between OBJECT13 and PAR21. As a consequence, the

computation of cohesion is affected. If we take into account this additional information,

other alternative cohesion metrics can be defined.

Given a software part sp, and a pair <A,B>, where A is a data declaration and B is

either a data declaration or a subroutine, we will say that the interaction between them is

known if it is detectable from the high-level design or is signaled by the designers (they

provide an answer similar to answer (1) above); we will say that the interaction between

them is unknown if it is not detectable from the high-level design and is not signaled by the

designers (they provide an answer similar to answer (1) above).

The set of known interactions of a software part sp will be denoted by K(sp), and

the set of unknown interactions by U(sp). In general, IM(sp)l >_ IK(sp)l + i U(sp)l, since

some interactions are not detectable from the high-level design and the designers explicitly

exclude their existence.

100225'14L 2-29
Metric 9: Neutral Ratio of Cohesive Interactions (NRCI).

Unknown CIs are not taken into account

IK(sp)l

NRCI(sp)-_(sp)l.nJ(sp)l

Metric 10: Pessimistic Ratio of Cohesive Interactions (PRCI).

Unknown CIs are considered as ff they were known not to be actual interactions.

_ IK sp)l

PRCI(sp)- nVICsp)l

(This is equivalent to RCI(sp).)

Metric 11: Optimistic Ratio of Cohesive Interactions (OR CI).

Unknown CIs are considered as ff they where known to be actual interactions

O .... IK(sp)l + IU(sp)l

Kt_l(sp)-- [M(sp)l

The above three metrics satisfy Properties 1-3, where CI(sp) is replaced by

K(sp) t.:(sp).

IfPRCI(sp), NRCI(sp), and ORCI(sp) areallnot undefined, itcan be shown that,

for allsoftwarepartssp,

O_RCI(sp) < NRCI(sp) -< ORCI(sp)__l

ORCI(sp) and PRCI(sp) provide the bounds of the admissible range for cohesion, and

NRCI(sp) takes a value in between. It can also be shown that the smaller the number of

unknown interactions, the smaller the interval [PRCL ORCI], Le., the more complete the

information, the narrower the uncertainty interval It should be noted that, once the low-

"100_5"14L 2-30
level design is completed, accurate and complete information about cohesive interactions

should be available.

Remark.

NRCI(sp) is tmdefmed if and only if all interactions are unknown, Le., no information is

available on cohesive interactions. It is interesting to notice that in this case, and only in this

case, PRCI(sp) = 0 and ORCI(sp) = 1, Le., PRCI(sp) and ORCI(sp) do not provide

stricter bounds than the ones provided by the interval for cohesion. The fact that NRCI(sp)

is undefined can be interpreted as the possibility that NRCI(sp) can take any value in the

interval [0,1].

2.2.4.2 Coupling

In this section, we first give general definitions and assumptions on coupling, then, we

present a set of metrics, and discuss the issue of genericity in the context of coupling. To

address the particular issue of coupling, we will refer to the/report/nterac_ns of a module

m as all interactions going from a declaration outside m to a declaration inside m. S'mailarly,

we define export interactions as going from a declaration located inside m to a declaration

outsidem.

Assumption A-IC:

The more dependent a software part on external data declarations, the more external

information needs to be known in order to make the software part consistent with the rest

of the system. In other words, the larger the amount of external data declarations, the more

incomplete the local description of the software part interface, the more spread the

information necessary to integrate a software part in a system. Thus, the software part

becomes more error-prone.

0

Definition 6" Import Coupling of a software part (1C).

Import Coupling is the extent to which a software part depends on imported external data

declarations.

0

10022514L 2-31
Assumption A-EC:

Export coupling is related to how a software part is used in the system. The more often the

software part is used, the larger the number of services it has to provide, the more flexible

it needs to be, e.g., genetic module. This may lead to errors.

0

Definition 7: Earn Coupling of a software part (EC).

Export coupling is the extent to which the data declarations of a software part affect the data

declarations of the other software parts in the system.

0

Import and export coupling of a software part will be expressed in terms of the actual DDinteractions involving imported external data declarations and the internal data declarations

of the software part. We now provide properties that must be satisfied by both import and

export coupling metrics.

Property 4: Non negativity

Given a software part sp, the metric import_coupling(sp) >_0 (resp. export_coupling(sp) >__

0). import_coupling(sp) = 0 (resp. export coupling(sp) = O) if and only if sp does not have

import (resp. export) interactions with other software parts.

0

Property 5: Monotonicity

Let ml be a module and H(ml) (resp. EI(ml )) its set of import (resp. export) interactions.

If m2 is a modified version of ml with the same sets of data and subroutine declarations

and one more import (resp. export) interaction so that I1(m2) (resp. El(m2)) includes

I1(m2) (resp. El(m2)), then import_coupling(m2) >_ import_coupling(m1) (resp.

export_coupling(m2) >_export_coupling(m1)).

0

Adding import (resp. export) interactions to a module cannot decrease its import (resp.

export) coupling.

Property 6: Merging of Modules

The sum of the couplings of two modules is no less than the coupling of the module which

is composed of the data declarations of the two modules.

0

'100"22514L 2-32
This stems from the fact that two modules may contain interactions between each other's

declarations, which are no longer import or export interactions for the module resulting

from merging the original modules.

It should be noted that, as opposed to cohesion, coupling is not a normalized

metric. This comes from assumptions A-CH, A-IC, and A-EC (see Sections 2.2.4.1 and

2.2.4.2), where we state that cohesion is a degree of interdependence within a software

part, whereas coupling is a quantity of dependencies between a software part and the rest of

the system.

We will now introduce interaction-based coupling metrics. The issue will be first

addressed by ignoring generic modules for the sake of simplification. Generic modules and

their impact on the defined metrics will be treated later in this section.

Metric 12: Import Coupling

Given a software part sp, Import Coupling of sp (denoted by IC(sp)) is the number of DDinteractions between data declarations external to sp and the data declarations within sp.

I

IMern'c 13: Export Coupling

Given a software part sp, Export Coupling of sp (denoted by EC(sp)) is the number of DDinteractions between the data declarations within sp and the dam declarations external to sp.

It is straightforward to prove that IC(sp) and EC(sp) satisfy the above properties 4-6, and

that, based on properties 4-6, these metrics are defined on a ratio scale [F91].

Each box in Figure 4 represents a module interface. Module interfaces m2 and m3

are located in their parent's interface ml. m2 is assumed to be declared before m3 and

therefore visible to m3. Tij and OBJEC15j data declarations represent respectively types and

objects in module mi FP3 represents a subroutine formal parameter. The IC and EC values

for the modules in Figure 4 are computed as follows.

_C(ml) = 0 EC(ml) = ]0

IC(m2) = 4 EC(m2) = ]

IC(m3) =5 EC(nd) =0

IC(m4) = 2 EC(m4) = 0

In the example of Figure 4, we see that ml expeetedly shows the largest export coupling.

10022514L 2-33
ml

m3

Figure 4. Calculation of IC and EC with non-generic modules only

Based on the definitions of IC(sp) and EC(sp), we derive four related metrics, DIC(sp)

(Direct Import Coupling), TIC(sp) (Transitive Import Coupling), DEC(sp) (Direct Export

Coupling), TEC(sp) (Transitive Export Coupling). DIC(sp) and DEC(sp) only take into

account direct interactions, whereas TIC(sp) and TEC(sp) only take into account transitive

interactions. By their definitions, IC(sp)= DIC(sp) + TIC(sp), and

EC(sp) = DEC(sp) + TEC(sp). This allows us to separately evaluate the impact of direct

and u'ansifive interactions on error-proneness, as we show in the experimental validation.

In practice, the number of transitive interactions turns out to be much bigger than that of

direct interactions, so IC(sp) = TIC(sp) and EC(sp) = TEC(sp).

The Treatment of Generic Modules

There are two possible ways of taking into account generics when calculating coupling.

Either each instance can be seen as a different module or a genexic can be seen as any other

module whose scope/global data declarations is/are the union of the scope/global data

declarations of its instances. The second solution does not consider instances as

independent modules and appears to be more suitable to our specific perspective, since

errors are to be found in generics and, only as a consequence, in instances.

The import coupling of a generic module is the cardinality of the union of the sets of

DD-intemctions between the data declarations in the software system and those of each of

its instances. When calculating export coupling, we take into account the DD-interactions

between the data declarations of each of its instances and those of the software system.

Consistent with the definition of DD-interaction, generic formal parameters DD-interact

10022514L 2-34
with their particular genetic actual parameters (i.e. type, object) when the genetic module is

instantiated, since a change in the former may imply a change in the latter.

This is what the example in Figure 5 illustrates. Gen_m is the interface of a genetic

module, with a generic formal parameter GenFP and a generic type GenT. The export

coupling of module Gen_m is given by the sum of three parts

• two interactions from Gen_m to ml, due to the two instantiadom, Gen_m(1) and

Gen_m(2), of Genre in ml,

• the in_raction from the _tantiation Gen_m(I)

• the two interactions from the instantiation Gen_m(2).

IC(ml) = 2

IC(m2) = 3

IC(m3) = 4

IC(Oen_m) = 0

EC(ml) = 4

EC(m2)= 0

EC(m3)= 0

EC(Gen m) - 5

Figure 5. Genetics when calculating coupling

10022514L 2-35
3 Experimental Validation

The experimental validation has two main goals.

Goa/]

We want to find out which of the metrics defined above have a significant impact on the

error-proneness of software parts. This allows us to

a. prove that high-level design information can be used to build significant indicators

of software error-proneness

b. determine which of our assumptions about the development process (Section 2) are

experimentally supported

c. compare the four strategies for defining high-level design metrics

d. identify themost promisingresearch directions.

Goa/2

We need toinvestigatedependenciesbetweenmetrics,inordertodeterminewhich onesare

complementary, and can be used in combination,and which ones capturesimilar

phenomena.

0

Section3.1presentstheexperimentaldesignoftheanalysis,theprojectdatasetsused and

thetoolbuilttocapturethediscussedesignmetrics.Section3.2providesand discusses

the resultsof a univariateanalysisof the metrics.The significanceof the metricsas

predictorsof error-pronesoftwarepartsisassessedand thedifferencesbetween systems

areinvestigated.Section3.3investigatestheresultsobtainedwhen buildingmultivariate

classification models fordetectingerror-proneLMHs basedon significantdesignmetrics.

The model resultsareassessedandthemodel ftmctionalstructureisinvestigated.

3.1 Experiment Design

Experiment Layout

In order to validate software measurement assumptions experimentally, one can adopt two

main strategies: (1) small-scale controlled experiments, (2) real-scale industrial case

studies. In this research project, we chose the second alternative since we thought the

10022514L 2-36
phenomena we are studying would be even more visible and significant on software

systems of realistic size and complexity. Also, we thought that (2) should be a more

relevant and convincing validation for the software industry practitioners.

However, the problem in such studies is that it becomes difficult to study the

phenomena of inmrestinisolation,without having to deal with othersources of variation.

In our case,we thought that,ffthesemetricswere tobe interesting,they should explaina

significantpercentage ofthevariationindividuallyorincombination,despiteothersources

of variation.However, we expectsome degree ofvariationacrossprojects.

Environment

The first system studied is an attitude ground support software for satellites (GOAl)A)

developed at the NASA Goddard Space Flight Center. The second one (GOESIM) is a

dynamic simulator for a geostationary enviromenml satellite. These systems are composed

of 525 and 676 Ada units, 90 Klocs and 170 Klocs, respectively, and have a fairly small

reuse rate (around 5% of source code lines). The third system we studied (TONS) is an

onboard navigation system for satellite that has been developed in the same environment

and is about 180 Ada units and 50 Klocs large, with an extremely small rate of reuse (2%

of source code lines). We selected projects with lower rates of reuse in order to make our

analysis of design factors more straightforward by removing what we think is a major

source of noiseinthiscontext.

Tool

A tool analyzing the interfacepartsof Ada source code has been developed in order to

capture the design characteristicsof these systems. This tool isbased on LEX&YACC

[LY92] and extracts generic high-level design information about the visibilityand

interactionsof the system declarations.This informationisconsequently used to compute

themetricspresentedinSection2.2,and othersthat might be defined.

Analytical Model

The response variable we use to validate the design metrics is binary, i.e., Did an error not

occur in an LMH? In order to analyze the impact of software metrics on the error-proneness

of software parts, we used logistic regression, a classification technique [HL89] used in

many experimental sciences, based on maximum likelihood estimation, and presented

below. In this case, a careful outlier analysis must be performed in order to make sure that

10022514L 2-37
the observed trend is not the result of few observations [DG_412. In particular, we fu'st

used univ_ logistic regression, to evaluate the impact of each of the metrics in isolation

on error-proneness. Then, we performed multivariate logistic regression, to evaluate the

relative impact of those metrics that had been assessed sufficiently significant in the

univariate analysis (e.g., a < 0.20 is a reasonable heuristic). This modeling process is

further described in [HL89].

A multivariate logistic regression model is based on the following relationship

equation (the univariate logistic regression model is a special case of this, where only one

variable appears):

log(1--_p) = CO + ClXl + C2X2 + ... + CnXn

where p is the probability that no errors were found in a software part during the validation

phase, and the Xi's are the design metrics included as predictors in the model (called

covariates of the logistic regression equation). In the two extreme cases, i.e., when a

variable is either non-significant or entirely differentiates error-prone software parts, the

curve (between p and any single X i, i.e., assuming that all other Xj's are constant)

approximates a horizontal line and a vertical Iine respectively. In between, the curve takes a

flexible S shape. However, since p is unknown, the coefficients C i will be estimated

through a likelihood function optimization. This procedure assumes that all observations

are statistically independent. When building the regression equations, each observation was

weighted according to the number of errors detected in each software part. The rationale is

that each (non) detection of error is considered as an independent event. As a consequence,

software parts where no errors were detected were weighted 1.

Goodness-of-fit for such a model is assessed via a statistic called R 2 (because

similar in concept to the least-square regression coefficient of determination), belonging to

the interval [0,1]. The higher R 2, the more accurate the model However, as opposed to the

R 2 of least-square regression, high R2s are rare for logistic regression, for reasons whose

explanation is well beyond the scope of this text. The interested reader may refer to [HL89]

for a detailed introduction to logistic regression.

Tables 1 and 2 contain the results we obtained through, respectively, univariate and

multivariate logistic regression on the three systems. We report those related to the metrics

2In addition, in order to confh-m the obtained results, we used non-parametric tests for rank distn'butioas

suchas the Mann-Whimey U test [CAP88]. Re.mlts appeared to be consistent across techniques and, in

order to limit the amount of statistics provided to the reader and preserve the clarity of the text, we only

show the remits obtained with logistic regression.

10022514L 2-38
that turned out to be the most significant ones across all three projects. For each metric, we

provide the following statistics:

C (appearing in both tables), the estimated regression coefficient. The larger the

coefficient, the stronger the impact of the explanatory variable on the probability p.

A_ (appearing in Table 1 only), which is based on the notion of odd ratio [HL89],

and provides an evaluation of the impact of the metric on the dependent variable.

More specifically, the odd ratio _s(X) repents the ratio between the probability of

not having an error and the probability of having an error when the value of the

metric is X. As an example, if, for a given value X, _(X) is 2, then it is twice more

likely that thesoftware part does not contain errors than that it does contain errors.

The value of A_/is computed by means of the following formula

Therefore, AV represents the reduction/increase in the odd ratio when the value X

increases by 1 unit. This provides a more intuitive insight than regression

coefficients into the impact of explanatory variables. (Since the whole range of RCI

is [0,1], we used one-tenth as the quantum for RCI increase with respect to which

A¥ is computed.)

¢z (appearing in both tables), the level of significance, which provides an insight

into the accuracy of the coefficient estimates. The significance (u) of the logistic

regression coefficients tells the reader about the probability for the coefficient to be

different from zero by chance. Also, the larger the level of significance, the larger

the standard deviation of the estimated coefficients, the less believable the calculated

impact of the coefficient. The significance test is based on a likelihood ratio test

[[-IL89] commonly used in the framework of logistic regression.

10022514L 2-39
3.2 Univariate Analysis

Results

As Table 1 shows, all strategies presented in Section 2.2 provide significant metrics, but

the strategy based on declaration counts. Therefore, these metrics, although based on

simple and appealing concepts, do not appear to be significant predictors.

All the metrics based on exported declarations, Le., Local(sp), ESP(sp), EC(sp),

DEC(sp), and TEC(sp), are not significant. Our explanation is that when an inconsistency

exists between an exporting module E and an importing module I, I is more lilmly to be

corrected, since E may export to other modules. Changing E is likeIy to require changing

those other modules. Alternatively, a large amount of exports sometimes translates into a

need for genericity but, for many declarations, just results into additional fields and

dimensions. Therefore, the assumption underlying the export interactions metric appears

somewhat questionable.

All the metrics based on the IS_ COMPONENT_OF relation appear significant in

the univariate analysis. However, they show a strong multicolinearity (i.e., the linear

correlations are strong between metrics). Since Avg_depth is the best predictor in its

category and in order to minimize the size of Table 1, only the Avg_depth results are

shown.

A dose analysis of the correlation matrix of the studied metrics shows that these

results are not due to strong correlations between factors, e.g., when all factors are size

predictors. Therefore, all the metrics in Table 1 seem to capture not only significant but

different trends. This shows that most of the strategies are likely to be complementary and

useful. This is confirmed by the multivariate results presented in Section 3.3.

Strateg_

USES

I_C_O

Inter.

Inter.

Inter.

Project

f_

Metrics

ISP

Avg_Depth

lgCI

TIC

DIC

GOADA

C A¥ m

-0.8 45% 0.000

-2.27 11% 0.000

0.63 188% 0.000

-0.016 98.5% 0.001

-0.23 79% 0.000

GOESIM

c Aw

-0.717 49%

-2.4 9%

0.215 124%

-0.017 98.3%

-0.19 83%

TONS

a C 4¥

O.OO2 -0.96 38%

0.000 -3.9 2%

0.047 0.34 141%

0.002 -0.02 98%

0.001 -0.04 96%

tX

0.00C

0.000

0.001

0.15

0.19

10022514L

Table 1. Univariate Analysis

2-40
DetailedDiscussion

TIC and DIC do not appear tobe significantin TONS (or= 0.19 and 0.15,respectively),

whereas they are very significant in the two other systems. The analysis of the distribution

of these factors in all three systems, respectively, Shows that their standard deviation (o')

and median (m) arc much smaller in TONS, i.e., with respect to T/C, or = 10, m = 2.5 for

TONS versus or = 32.74, m = 25.5 for GOADA, or = 32.18, ra = 59 for GOESIM. As a

consequence, any trend related to either DIC or T/C is very likely not to be visible in the

TONS datas_L When considering that TONS is a significantly smaller system than the two

other ones, results may be interpreted as follows: the distribution of import interactions is

strongly dependent on the size of the system and input interaction metrics are likely to be

mediocre predictors for small sysmms.

Comparing Models

From a more general perspective, variations across models (i.e., univariate regression

equations) should be expected due to differences in project characteristics, i.e., size,

application domain. However, it is worth noticing that, despite the fact that these projects

belong to different application domains (within the context of satellite support systems) and

have been developed at different times, most of the models are surprisingly stable across

projects. Because of the functional shape of logistic models, coefficients that may appear

significantly different actually generate very similar models, e.g., In Table 1, coetticients

-2.27 and -3.9 yield A_s of 11% and 2%, respectively. As a consequence, to evaluate the

stability of the models, the reader should rather look at the Avcolumn in Table 1. When

doing so, only RCI appears to have a noticeable model instability even though the trends

are consistenL

3.3 Multivariate Models

In this section, we present the results obtained by/x:rforming a stepwise multivariate

logistic regression. Table 2 provides the estimated regression coefficients (C) and their

significance (a) based on a Wald test [HL89], which is obtained by comparing the

maximum likelihoodestimateof a parameter to itsestimatedstandarddeviation. Regression

coefficients are not shown when their level of significance is above 0.2 (substituted by a

*).

10022514L 2-41
Strategy

USES

LC_O

InCr.

Incr.

InCr.

Pr_e_s

Metn'cs

GOADA

C a

GOESIM

C ¢X

TONS

C a

RCI

TIC

DIC

-0.9 0.04

-L8 0.003

* -1.18

0.000 -5.62

0.4 0.006

-0.023 0.000

0.23 0.04

0.3

-0.02

-0.13

0.07 0,2

0.005 [ *

0.04 -0.11

0.000

0.000

0.16

.

0.002

Table 2. Coefficientsof Multivariate Models

Results

The very low levels of significance in Table 2 suggest that these metrics may be used in

combination as indicators of error-prone LMHs. Indeed, when used in a multivariate

model, many of these metrics ate still significant and produce models that are more accurate

than univariate models (Table 2). The best uaivariate R2s are 0.115, 0.20 and 0.16 for

GOADA, GOESIM, and TONS, respectively. In the same order, the multivariate R2s are

0.21, 0.24, and 0.43. We can see that the results improved very significantly for GOADA

and TONS.

Interaction-baseA metrics are more complex but worth collecting, since they are the

only metrics defined at the declaration level that appeared significant. In addition, the

average LMH depth was consistently selected as a very good indicator. This is likely to be

an early measure of "size" of the LMI-I and is expectedly significant. Also, ISP, a metric

similar to the notion of fan-in shows to be significant across projects (except in the

multivariate GOESIM model for reasons explained below), while ESP, the equivalent

measure for exports (based on the fan-out of LMHs) is not significant. As a consequence, a

metric of the form (fan_in. fan_out) 2, suggested in numerous occasions in the literature

[HK84, IS88, $90, Z91], does not appear to be significant. From a more general

perspective, metrics based on imports, regardless of the associated concepts, appear to

predict more accurately the error-proneness of software parts.

Comparing Models

Some variability in the estimated regression coefficients can be observed across

projects in Table 2 and requires some discussion. In multivariate models, coefficients have

10022514L 2-42
a tendency to adjust, statistically, for other variables [HL89]. Sometimes, variables are

weak predictorsof the response variablewhen taken individually,and become more

significantwhen integramd in a multivariatemodel. In Table 1,DIC showed, for TONS, a

mediocre level of significance, whereas it appears to be a significant predictor in Table 2.

Moreover, its coefficient is very unstable across projects and the trend is reversed (positive)

for GOADA and TONS. When looking more carefully at the associations (not only the

narrower concept of linear correlation) between metrics, it can be determined that this is the

results of strong association between DIC and ISP in GOADA and TONS. These

associations are a typical source of coefficient instability [DG84], e.g., the coefficient of

ISP in GOADA varies from -0.9 to -0.39 when DIC is removed from the equatiom

T/C remains non-significant because of its strong linear correlation (R 2 = 0.76)

with DIC in the TONS dataset. Similarly, ISP does not appear significant in the GOESIM

dataset because of a strong correlation with DIC (R 2 = 0.50). RCI in TONS shows a

weaker sigt2ficance (¢z = 0.16) than in the univariate results and no strong linear correlation

can be observed with the other metrics included in the multivariate equation. However,

LMHs with large numbers of imported interactions are all located in the low part of the

cohesion range. Such an association (likely to be spurious since it is not the case in the

other datasets) with DIC is likely to affect the significance of RCI in a multivariate

equation.

It is important to note that a different set of systems showing different distributions

might show very different trends. This points out a need for large scale investigation across

various development environments and application domains.

4 Conclusion

This study has shown that statistical models of extremely good significance can be built

based on high-level design information. In particular, we have determined accurate early

predictors for error-prone software. Moreover, the results suggested that, at this stage of

understanding, several strategies were worth investigating because none of them showed

dominant trends, while most of them appeared to be complementary. In order to provide

the practitioner with usable, well understood and validated models, software engineering

researchers will have to keep refining and validating the existing metrics. There is still

substantial room for improvement.

The stability of the impact of these metrics across projects allows us to draw

optimistic conclusions about the use of such quality indicators. Using early quality

10022514L 2-43
indicators based on objective empirical evidence appears possible. However, it is very

Likely that this kind of indicators win behave differently across various domains of

application and development environments.

Therefore, the use of such indicators should always be preceded by a careful

empirical analysis of local error patterns and a thorough comparison across projects.

Our future work will be three-fold:

• Analyze more systems

• Validate further and refine the metrics we defined in this paper. The variations

across environments and the study/comparison of different architecunes is likely to

give us interesting insights.

• Consistent with our current objectives, we will address the issues related to

building metric based empirical models earlier in the life cycle. In particular, the

next stage of this _h will focus on defufing and validating metrics for formal

specifications.

Acknowledgments

We thank Giuseppe Calavaro and Chris Lott for their helpful comments on the earlier drafts

of this paper.

REFERENCES

[AE9'2] W. Agresti and W. Evanco, "Projecting Software Defects from Analyzing

Ada Designs," IEEE Trans. Software Eng., 18 (11), November, 1992.

['BBH93] L. Briand, V. Basili and C. Hetmanski, "Developing Interpretable Models

with Optimized Set Reduction for Identifying High Risk Software Components," IEEE

Trans. Software Eng., 19 (11), November, 1993.

[]3087] G. Booch, "Software Engineering with Aria," Benjamin/Cumming

Publishing Company, Inc., Menlo Park, California, 1987.

[BR88] V. Basili and H. Rombach,"l'he TAME Project: Towards ImprovementOriented Software Environments," IEEE Trans. Software Eng., 14 (6), June, 1988.

10022514L 2-44
[BTH93] L. Briand, W. Thomas and C. Hetmanski, "Modeling and Managing Risk

early in Software Development," International Conference on Software Engineering,

Maryland, May 1993

[CAP88] J. Capon, "Statistics for the Social Sciences", Wadworth publishing

company, 1988

[CY79] L. Constantine, E. Yourdon, "Structured Design," Prentice Hall, 1979

[DG84] W. Dillon and M. Goldstein, Multivariate Analysis: Methods and

Applications, Wiley and Sons, 1984.

[DoD83] ANSI/MIL-STD-1815A- 1983, Reference Manual of the Ada Programming

Languages, U.S. Department of Defense, 1983

[5"91] Norman Fenton, "Software Metrics, A Rigorous Approach," Chapman&Hall,

1991.

[G86] J. Gannon, E. Katz, V. Basili, "Metrics for Ada Packages: an Initial

Study," Communications of the ACM, Vol. 29, N. 7, July 1986.

[GJM92] C. Ghezzi, M. Jazayeri, D. Mandrioli, "Fundamentals of Software

Engineering," Prentice Hall, Englewood Cliffs, NJ, 1992

[I-IK84] S. Henry, D. Kafura, "The Evaluation of Systems' Structure Using

Quantitative Metrics," Software Practice and Experience, 14 (6), June, 1984.

[IS88] D. Ince, M. Shepperd, "System Design Metrics: a Review and Perspective," Proc.

Software Engineering 88, pages 23-27, 1988

[K88] B. Kitchenham, "An Evaluation of Software Structure Metrics," Proc.

COMPSAC 88, 1988

['LY92] J. Levine, T. Mason, D. Brown, "lex & yacc," O_Reilly & Associates,

Inc., 1992

[M77] J. Myers, "An Extension to the Cyclomatic Measure of Program Complexity,"

SIGPLAN Notices, 12(10):61-64, 1977

[MGBB90] A. Melton, D. Gustafson, J. Bieman, A. Baker, "A Mathematical

Perspective for Software Measures Research," Software Engineering Journal, September

1990.

[R87] H.D. Rombach, "A Controlled Experiment on the Impact of Software

Structure and Maintainability:," IEEE Trans. Software Eng., 13 (5), May, 1987.

[R.90] H.D. Rombach, "Design Measurement: Some Lessons Learned," IEEE

Software, March 1990.

[$90] M. Shepperd, "Design Metrics: An Empirical Analysis," Software

Engineering Journal, January 1990.

10022514L 2-45
[SB91] R. Sclby and V. Basil/, "Analyzing Error-Prone System S_ucturc," IEEE

Trans. Software Eng., 17 (2), February, 1991.

[Z91] W. Zagc, D. Zagc, P. McDanicl, L Khan, "Evaluating De,sign Metrics on

Large-Scale Software," SERC-TR-106-P, September 1991.

10022514L 2-46
SECTION 3mTECHNOLOGY EVALUATIONS

The technical papers included in this section were originally prepared as indicated below.

Comparing Detection Methods for Software Requirements Inspections: A Replicated Experiment, A. A. Porter, L. G. Votta Jr., and V. R. Basili, University of

Maryland, Technical Report TR-3327, July 1994

"Software Process Evolution at the SEL," V. Basili, S. Green, IEEE Software,

July 1994

10022.514L 3-1

Comparing Detection Methods For Software Requirements

Inspections: A Replicated Experiment

Adam A. Porter Lawrence .G. Votta, Jr. Victor R. Basili"

Abstract

Software requirements specifications (SP_) are _s'ually validaSed by inspections, in which several reviewers

independently analyze all or part of _ie specification and search for defects. These defects are _hen collected at a

meeting of the reviewers and au_or(s).

Usually, reviewers use Ad Hoe or Chexklist methods _o uncover defects. These methods force all reviewers to

rely on nonsystematic _echniques to search for a wide variety of defects. We hypothesize fha¢ a Scenario-based

method, in which each reviewer uses different, systematic _echniques to search for different, specific classes of

defects, will have a significan13_t higher success rate.

We evaluated this hypothesis using a 3 x 24 partial factorial, randomized experimental design. Forty eight

graduate students in computer science participated in the experiment. They were assembled into sirteen, _hreeperson teams. Each team inspected two St_S using some combination of Ad 2_oc, Checklist or Scenario methods.

For each inspection we performed four measurements: (1) individual defect detection rate, (g) team defect

de_ection rate, (3) percentage of dejects first identified a_ the collection meeting (meeting gain rate), and (4)

percentage of defects first identified by an individual, bu_ never reported at the collection meeting (meeting loss

ra_e).

The ezTerimental re.val_ show that (1) _he Scenario method has a higher defect detection rate than either Ad

Hoc or Checklist methods, (_) Scenario reviewers are more effective at detecting the defects their scenarios are

designed to nncover, and are no less effective at detecting other defects, (3) Checklist reviewers were no more

effective _han Ad Hoc reviewers, and (4) Collection meetings produce no ne_ improvement in the defect detection

ra_e - meeting gains are offset by meeting losses.

A prdimmary version of this article entitled, =An Expeximent to Assess DitTezent Defect Detection Methods For

Software Requirements Inspections', has been selected to appea_ in the proceedings of the 16 th Internationa] Con:[erence

on Softwaze Engimeelfng. This article expands on our previous work in several ways:

1. We have replicated the initial expemnemt - doubling the number of data points.

2. We have. expamded the descziption of the Scemarlo detection methods and included appendices cont_i_ing the fuji

text of the Ad Hoc, Checklist, and Sceaaxio defect detection aids that were used dmdng the _ent.

3. Our original analysis analyzed the effect of diiTerent detection methods on team performance. With the increased

number of data points, we are now able to extend the amalysis to determine how these methods influence individual

performance. This _ us to reject seve_ral threats to the experiment's internal validity.

4. We have added a new section aJaa]yzing the how inspection meetings affect inspection performamce. Our results

show that meetings contribute nothing to defect detection effectiveness.

"This work is supported ha part by the National Aeroxxautics mad Space A_=ation under grant NSG--5123. Porter and Basili

are with the Department of C_mputer Science, University of Ma.wlaud, C_lege Park, Marylmxd 20472. Votta is with the Software

Production Research Department, AT&T Bell Laboratories Naperville, IL 60566

1007_2514L 3-3
1 Introduction

One of the most common ways of validating a software requirements specification (SRS) is to submit it to an

inspection by a team of reviewers. Many organizations use a three-step inspection procedure for eliminating

defects 1 : detection, collection, and repair 2. [8, 17] A team of reviewers reads the SRS, identifying as many

defects as possible. Newly identified defects are collected, usually at a team meeting, and then sent to the

document's authors for repair.

We are focusing on the methods used to perform the first step in this process, defect detection. For this

article, we define a defect detection method to be a set of defect detection techniques coupled with an assignment

of responsibilities to individual reviewers.

Defect detection techniques may range in prescriptiveness from intuitive, nonsystematic procedures, such as

Ad Hoc or Checklist techniques, to explicit and highly systematic procedures, such as formal proofs of correctness.

A reviewer's individual responsibility may be general - to identify as many defects as possible - or specific -

to focus on a limited set of issues such as ensuring appropriate use of hardware interfaces, identifying untestable

requirements,or checking conformity to cod_g standards.

These individualresponsibilitiesmay be coordinated among the members of a review team. When they are

not coordinated,allreviewershave identicalresp.onsibilities. In contrast,the reviewersin coordinated teams may

have separateand distinctresponsibilities.

In practice,reviewers often use Ad Hoc or Checklist detectiontechniques to discharge identical,general

responsibilities.Some authors, notably Pamas and Weiss[13],have argued that inspections would be more

effectiveifeach reviewer used a differentset of systematic detectiontechniques to discharge different,specific

responsibilities.

Until now, however, there have been no reproducible,quantitativestudies comparing alternativedetection

methods forsoftwareinspections:We have conducted such an experiment and our resultsdemonstrate that the

choiceof defectdetection method significantlyaffectsinspectionperformance. Furthermore, our experimental

designmay be easilyreplicatedby interestedresearchers.

ZWeuse theword deject Lusteadof the word/a_l/even though this doesnot a_]Jacre to the TR._.II_, Standardso_ So.'¢tware_n_eez_g

Terminolo_D, [9]. We feel the word fault has a code-specific co_notafion - only one of the many places where inspections are used.

ZDepending on the exact form of the inspection, they are sometimes called reviews or waJkthroughs. For a more thorough

description of the taxonomy see [8] pp. 1Tl_ and [10].

10022514L 3-4
Below we describe the relevant literature, several alternative defect detection methods which motivated our

study,our researchhypothesis,and our experimental observations,analysisand conclusions.

I.I Inspection Literature

A summary of the originsand the currentpracticeof inspectionsmay be found in Humphrey [8].Consequently,

we willdiscussonly work directlyrelatedto oar currentefforts.

Fagan[6] defined the basic software inspection process. W_ni]e most writers have endorsed his approach[3,

8], Parnas and Weiss are more critical [13]. In part, they argue that effectiveness suffers because individual

reviewers are not assigned specific responsibilities and because they lack systematic techniques for meeting those

responsibilities.

Some might argue that Checklists are systematic because they help define each reviewer's responsibilities and

suggest ways to identify defects. Certainly, Checklists often pose questions that help reviewers discover defects.

However, we argue that the generality of these questions and the lack of concrete strategies for answering them

makes the approach nonsystematic.

To address these concerns - at least for software designs - Parnas and Weiss introduced the idea of active

design reviews. The principal characteristic of an active design review is that each individual reviewer reads for a

specific purpose, using specialized questionnaires. This proposal forms the motivation for the detection method

proposed in Section 2.2.2.

1.2 Detection Methods

Ad Hoc and Chec_]_._tmethods are the two most frequentlyused defect detection methods. With Ad Hoc

detectionmethods, allreviewersuse nonsystematic techniquesand are assignedthe same general responsibilities.

Checklist methods are similar to Ad Hoc, but each reviewer receives a checklist. Checklist items capture

important lessons learned from previous inspections within an environment or application. Individual checklist

items may enumerate characteristic defects, prioritize different defects, or pose questions that help reviewers

discover defects," such as _Are all interfaces dearly defined?" or _f input is received at a faster rate than can

be processed, how is this handled. _ The purpose of these items is to focus reviewer responsibilities and suggest

ways for reviewers to identify defects.

100225_4L 3-5
I.o -°] O*

o t O O •

-/ \--

................. I_ () - o

Figure 1: Systematic Inspection Research Hypothesis. This figure representsa software requirements

specification before and a_er a no_utema_ic technique, genera/and identical responsibility inspection and a

sysfema_i¢ technique, spec/j_c and d/a_in_ responsibility inspection. The points and holes represent various

defects. The line-filled regions indicate the cover_e achieved by different members of the inspection team.

Our hypothesis is that systematic technique, specific and coordinated responsibility inspections achieve broader

coverage and _e reviewer overlap, resulting in higher defect detection rates and greater cost benefits than

nonsystematic methods.

1.3 Hypothesis

We believe that an alternative approach which gives individual reviewers specific, orthogonal detection responsibilities and specialized techniques for meeting them will result in more effective inspections.

To explore this alternative we developed a set of defect-specific techniques called Scenarios - collections of

procedures for detecting particular classes of defects. Each reviewer executes a single scenario and multiple

reviewers are coordinated to achieve broad coverage of the document.

Our underlying hypothesis is depicted in Figure 1: that nonsystematic techniques with general reviewer

responsibility and no reviewer coordination, lead to overlap and gaps, thereby lowering the overall inspection effectiveness; while systematic approaches with specific, coordinated responsibilities reduce gaps, thereby increasing

the overall effectiveness of the inspection.

2 The Experiment

To evaluate our systematic inspection hypothesis we designed and conducted a multi-trial experiment. The goals

of this experiment were twofold: to character_e the behavior of existing approaches and to assess the potential

benefits of Scenario-based methods. We ran the experiment twice; once in the Spring of 1993, and once the

following Fall. Both runs used 24 subjects - students taking a graduate course in formal methods who acted

10022514L 3-6
as reviewers. Each complete run consisted of (1) a training phase in whida the subjects were taught inspection

methods and the experimental procedures, and in which they inspected a sample SRS, and (2) an experimental

phase in which the subjects conducted two monitored inspections.

2.1 Experimental Design

The design of the experiment is somewhat unusual. To avoid misinterpreting the data it is important to understand the experiment and the reasons for certain elements of its design s.

2.1.1 "variables

The experiment manipulates five independent variables:

1. the detection method used by a reviewer (Ad Hoc, Checklist, or Scenario);

2. the experimental replication (we conducted two separate replications);

3. the inspection round (each reviewer participates in two inspections during the experiment);

4. the specification to be inspected (two are used during the experiment).

5. the order in which the specifications are inspected (either specification can be inspected first).

The detection method is our treatment variable. The other variables allow us to assess several potential

threats to the experiment's internal validity.

For each inspection we measure four dependent variables:

1. the individual defect detection rate,

2. the team defect detection rate 4,

3. the percentage of defects first identified at the collection meeting (meeting gain rate), and

4. the percentage of defects first identified by an individual, but never reported at the collection meeting

(meeting loss rate).

SSee Judd, et al. [11], chapter 4 for an excellent discussion of randomized social experimental designs.

4The team and individual defect detection razes are the number of defects detected by a team or individual divided by the total

number of defects known to be in the speciiicazion. The closer that value is to 1, the more effeczive the detection method. No defecm

were inte_ationally seeded into the specifications. All defects are _tura_y occarring.

10022514L 3-'7
Detection

Method

Round/Specification

Round 1 Round 2

ad hoc

checklist

scenarios

WLMS CRUISE

1B, 1D, 1G

1H, 2A

2B

2C, 2F

1A, lC; 1E

1F, 2D

2E, 2G

2H

WLMS

1A

1E, 2D, 2G

1F, 1C, 2E

2H

CRUISE

1D, 2B

1B, 1H

1G, 2A, 2C

2F

Table 1: This table shows the settings of the independent variables. Each team inspects two documents, the

WLMS and CRUISE, one per round, using one of the thIee detection methods. Teams from the Rrst replication

are denoted 1A-1H, teams from the second replication are denoted 2A-2H.

2.1.2 Design

The purpose of this experiment is to compare the Ad Hoc, Checklist, and Scenario detection methods for inspecting software requirements specifications.

When comparingmultipletreatments,experimentersfrequentlyusefractionalfactorialdesigns.Thesedesigns

systematicallyexploreallcombinationsofthe independentvariables,allowingextraneousfactorssuch as team

ability,specification quality,and learningto be measured and eliminatedfrom theexperimentalanalysis.

Had we used such a designeachteam would have participatedinthreeinspectionrounds,reviewingeach of

threespecifications and usingeach ofthreemethods exactlyonce.The orderinwhichthemethods areapplied

and thespecifications areinspectedwouldhave been dictatedby theexperimentaldesign.

Such designsareunacceptableforthisstudybecausetheyrequiresome teamstousetheAd Hoc orChecklist

method aftertheyhave usedtheScenario.method.SincetheAd Hoc and Checklistmethodsarenonsystematic,it

isimpossibletodefine,monitorand enforcetheiruse.Therefore,we were concernedthattheuseoftheScenario

method inan earlyround might imperceptiblydistortthe useoftheothermethodsinlaterounds.

Consequently,we chosea partialfactorialdesigninwhich notallcombinationsoftheindependentvariables

are present.With thisdesign,each team participatesintwo inspections,usingsome combinationofthe three

detectionmethods,but teams usingtheScenariomethod inthefirstround must continuetouse itinthesecond

round.TableI shows thesettingsoftheindependentvariables.

2.1.3 Threats to Internal Validity

A potential problem in any experiment is that some factor may affect the dependent variable without the researcher's knowledge. This possibility must be minimized. We considered five such threats: (1) selection affects,

10022514L 3-8
(2) maturation effects, (3) replica_on effects, (4) instrumentation effects, and (5) presentation effects.

Selection effects are due to natural variation in human performance. For example, random assignment of

subjects.may accidentally create an elite team. Therefore, the di_erence in this team's natural ability will mask

di_erences in the detection method performance. Two approaches are often taken to limit this effect:

1. Create teams with equal skills. For example, rate each participant's bsckground knowledge and experience

as either low, medium, or high and then form teams of three by selecting one individual at random from

each experience category. Detection methods are then assigned to fit the needs of the experiment.

2. Compose teams randomly, but require each team to use all three methods. In this way, differences in team

skill are spread across all treatments.

Neither approach is entirely appropriate. Although, we used the first approach in our initial replication, the

approach is unacceptable for multiple replications, because even if teams within a given replication have equal

skills, teams from different replications will not.

As discussed in the previous section, the second approach is also unsuitable because using the Scenarios in

the first inspection l_und will certainly bias the application of the Ad Hoc or Che_t methods in the second

inspection Round.

Our strategy for the second replication _acl future replications is to randomly assign teams and detection

methods. However, teams that used Scenarios in the first round were constrained to use them again in the

second round. This compromise efficiently employs the subjects without biasing the performance of any teams.

Maturation effects are due to subjects learning as the experiment proceeds. We have manipulated the detection

method used and the order in which the documents are inspected so that the presence of this effect can be

discovered and taken int_ account.

Replication effects are caused by differences in the materials, participants, or execution of multiple replications. We limit this effect by using only first and second year graduate students as subjects - rather than

both undergraduate and graduate students. Also, we maintain consistency in our experimental procedures by

packaging the experimental procedures as a classroom laboratory exercise. This helps us to ensure that similar

steps are followed for all replications.

As will be shown in Section 3, variation in the defect detection rate is not explained by selection, maturation,

10022514L 3-9
or replication effects.

Finally, instrumentation effects may result from differences in the specification documents. Such variation is

impossib.le to avoid, but we controlled for it by having each team inspect both documents.

2.1.4 Threats to External Valldi_

Threats to external validity limit our ability to generalize the results of our experiment to industrial practice.

We identified three such threats:

1. the reviewers in the first run of our experiment may not be representative of software programming professionals;

2. the specification documents may not be representative of real programming problems;

3. the inspection process in our experimental design may not be representative of software development practice.

The first two threats are real. To surmount them we are currently replicating our experiment using software

programming professionals to inspect industrial work products. Nevertheless, laboratory experimentation is a

necessary first step because it greatly reduces the risk of transferring immature technology.

We avoided the third threat by modeling the experiment's inspection process after the design inspection

process described in Eick, et al. [5], which is used by several development organizations at AT&T; therefore, we

know that at least one professional software development organization practices inspections in this manner.

2.1.5 Analysis Strategy

Our analysis strategy had two steps. The first step was to find those independent variables that individually

explain a signi/icant amount of the variation in the team detection rate. This was done by using an analysis of

variance technique as discussed in Box, et al.([4], pp. 165_).

The second step was to evaluate the combined effect of the variables shown to be significant in the initial

analysis. Again, we followed Box, et al. closely ([4], pp. 210_).

Once these relationships were discovered and their magnitude estimated, we examined other data, such as

correlations between the categories of defects detected and the detection methods used that would confirm or

10022514L 3-10
reject (if possible) a causal relationship between detection methods and inspection performance.

2.2 Experiment Instrumentation

We developed several instruments for this experiment: three small software requirements specifications (SRS),

instructions and aids for each detection method, and a data collection form.

2.2.1 Software Requirements Specifications

The SRS we used describe three event-driven process control systems: an elevator control system, a water level

monitoringsy_zn, and an automobile cruise control system. Each specification has four sections: Overview, Specific Functional Requirements, External Interfaces, and a Glossary. The overview is written in natural language,

while the other three sections are specified using the SCR tabular requirements notation [7].

For this experiznent, all three documents were adapted to adhere to the I_EE suggested format [10]. All

defects present in these SRS appear in the original documents or were generated during the adaptation process;

no defects were intentionally seeded into the document. The authors discovered 42 defects in the WLMS SRS;

and 26 in the CRUISE SRS. The authors did not inspect the ELEVATOR SRS since it was only used for training

exercises.

Elevator Control System (ELEVATOR) [18] describes the functional and performance requirements of a

system for monitoring the operation of a bank of devators (16 pages).

Water Level Monitoring System (WLMS) [16] describes the functional and performance requirements of

a system for monitoring the operation of a steam generating system (24 pages).

Automobile Cruise Control System (CRUISE) [121 describes the functional and performance requirements for an automobile cruise control system (31 pages).

2.2.2 Defect Detection Methods

"To make a fair assessment of the three detection methods (Ad Hoc, Checklist, and Scenario) each method should

search for a well-defined population of defects. To accomplish this, we used a general defect taxonomy to define

the responsibilities of Ad Hoc reviewers.

10022514L 3-11
Oml_n

MP

MP

MI

Commimon

AT

11

EF

ws

Omission

ON_D_mOn

_ kkum_m

b=mmmd _- _m.a IPu_-

Ad Hoc Checklist Sc_ario

Figure 2: Relationship Between Defect Detection Methods. The figure depicts the relationship between

the defect det_on methods used in this study. The vertical extent represents the coverage. The horizontal axis

labels the method and represents the degree of detail (the greater the horizontal extent the greater the detail).

Moving from Ad Hoc to Checklist to Scenario there is more detail and less coverage. The gaps in the Scenario

and Checklist columns indicate that the Checklist is a subset of the Ad Hoc and the Scenarios are a subset of

the Che_t.

The checklistused in thisstudy is a refinementof the taxonomy. Consequently,Checklistresponsibilitiesare

a subset of the Ad Hoc responsibilities.

The Scenarios are derived f:om the checklistby replacingindividualChecklistitems with procedures designed to implement them. As a result,Scenario responsibilitiesare distinctsubsets of Checklistand Ad Hoc

responsibilities.The relationshipbetween the three methods isdepicted in Figure 2.

The taxonomy is a composite of two schemes developed by Schneider,et al.[14]and Basiliand Weiss [2].Defectsare dividedintotwo broad types:omission - in which important informationisleftunstated and commission

- in which incorrect,redundant, or ambiguous information isput into the SRS by the #mthor. Omission defects

were furthersubdivided into four categories:Missing Functionality,Missing Performance, Missing Environment,

and Missing InterfaceCommission defectswere also divided into four categories:Ambiguous Information, InconsistentInformation, Incorrector Extra Functionality,and Wrong Section. (See Appendix A for complete

taxonomy.) We provided a copy of the taxonomy to each reviewer.

Ad Hoc reviewersreceivedno furtherassistance.

Checklistreviewersreceiveda singlechecklistderivedfrom the defecttaxonomy. To generatethe checkl_._twe

populated the defecttaxonomy with detailedquestionsculledfrom severalindustrialchecklists.Thus, they are

very similarto che_li_tsused in practice.All Checklistreviewersused the same checklist.(See Appendix B for

the complete che_]r_t.)

10022514L 3-12
]Defect Report Form

s,,._=_ _ _. _ '_" _-'_"_

,r,wm c_ m_.,,.m ,_- "z_,o,,t,.,_._

• .

_NL

Figure 3: Reviewer Defect Report Form. This is a small sample of the defect report form completed during

each reviewer's defect detection. Defects number 10 and 11, found by reviewer 12 of team C for the WLMS

sp_/fica_onare shown.

Finally, we developed three groups of Scenarios. Each group of Scenarios was designed for a specific subset

of the Checklist items:

1. Data Type Inconsistencies (DF),

2. Incorrect Functionalities (IF),

3. Missing or Ambiguous Functionalities (MF).

After the experiment was finished we applied the Scenarios to estimate how broadly they covered the WLMS

and CRUISE defects. We estimated that the Scenarios address about half of the defects that are covered by the

Che_t. Appendix C contains the complete list of Scenarios.

2.2.3 Defect Report Forms

We also developed a Defect Report Form. Whenever a potential defect was discovered - during either the

defect detection or the collection activities - an entry was made on the form. The entry included four kinds

of information: Inspection Activity (Detection, Collection); Defect Location (Page and Line Numbers); Defect

Disposition, (Defects can be True Defects or False Positives); and a prose Defect Description.

A smallsampleofa DefectReportappearsinFigure 3.

10022514L 3-13
2.3 Experiment Preparation

The participants were given a series of lectures on software requirements specifications, the SCR tabular requirements notation, inspection procedures, the defect classification scheme, and the filling out of data collection

forms. The references for these lectures were Fagan [6], Parnas [13], and the IEEE Guide to Software Requirements Specifications [i].

The participants were then assembled into three-person teams - see Section 2.1.3 for details. Within each

team, members were randomly assigned to act as the moderator, the recorder, or the reader during the collection

meeting.

2.4 Conducting the Experiment

2.4.1 Tr_;_;_g

For the training exercise, each team inspected the ELEVATOR SRS. Individual team members read the specification and recorded all defects they found on a Defect Report Form. Their efforts were restricted to two hours.

Later we met with the participants and answered questions about the experimental procedures. Afterwards, each

team conducted a supervised collection meeting and filled out a master Defect Report Form for the entre team.

The ELEVATOR SRS was not used in the remainder of the experiment.

2.4.2 Experimental Phase

This phase involved two inspection rounds. The instruments used were the _VLMS and CRUISE specifications

discussed in Section 2.2.1, a checklist, three groups of defect-based scenarios, and the Defect Report Form. The

development of the checklist and scenarios is described in Section 2.2.2. The same checklist and scenarios were

used for both documents.

During the firstRound, fourofthe eightteams were asked toinspect the CRUISE specification;the remaining

four teams inspectedthe WLMS specification.The detectionmethods used by each team are shown in Table I.

Defect detectionwas limitedto two hours, and allpotentialdefectswere reported on the Defect Report Form.

After defectdetection,allmaterialswere collected?

SFor each round, we set aside 14 two-hc_r _ne slots dmdng wI_ch inspection tasks could be done. Participants performed each

task within _ single two-hour session and were not allowed to wm-k at other times.

I0022514L 3-14
m

Rev Method Sum 1 2 2._._1 3:2 41 42

42 Data inc_istency 9 0 0 0 0 0 0

43 Inca'rect functionality 6 0 1 " 0 0 0 0

000 -- 000 _ 000

44 M ugfunc cm 18 o 0 __!_l __0__0 0 0

Team 23 0 1 . 0 0

Key AH DT MA AH DT AH

Figure 4: Data Collection for each Vv'LMS inspections. This figure shows the data collected from one

team's WLMS inspection.The firstthree rows identifythe review team members, the detectionmethods they

used, the number of defectsthey found, and shows theirindividualdefectsummaries. The fourth row contains

the team defectsummary. The defectsummaries show a I (0)where the team or individualfound (did not find)a

defect.The fifthrow cont_ the defectkey which identifiesthose reviewerswho were responsiblefor the defect

(AH for Ad Hoc only;CH for Cherkli_tor Ad Hoc; DT for data type inconsistencies,Checklist,and Ad Hoc; IF

forincorrectfunctionality,Checklistand Ad Hoc; and MA formissin." g or ambiguous functionality,Checklist and

Ad Hoc). Meeting gain and lossratescan be calculatedby comparing the individualand team defectsummaries.

For instance,defect21 is an example of meeIin# loss.It was found by reviewer44 during the defectdetection

activity,but the team did not reportitat the collectionmeeting. Defect 32 isan example of meeling gain;itis

firstdiscoveredat the collectionmeeting.

Once all team members had finished defect detection, the team's moderator arranged for the collection

meeting. At the collection meeting, the documents were reread and defects discussed. The team's recorder

maintained the team's master Defect Report Form. Collection was also limited to 2 hours. The entire Round

was completed in one week.

The second Round was similar to the first except that teams who had inspected the WLMS during Round 1

inspected the CRUISE in Round 2 and vice versa.

3 Data and Analysis

3.1 Data

Three setsof data are important to our study: the defectkey, the team defectsummaries, and the individual

defectsummaries.

The defectkey encodes which reviewersare responsibleforeach defect.In thisstudy,reviewerresponsibilities

are definedby the detectiontechniquesa revieweruses.Ad Hoc reviewersareresponsible(askedto search for)for

alldefects.Checklistreviewersareresponsiblefora largesubsetof the Ad Hoc defects. Since each Scenario isa

refinement of severalCheckli_titems,each Scenariorevieweris responsiblefor a distinctsubsetof the Checkli._t

ei.e., defects for which an Ad Hoc reviewer is responsible.

10022514L 3-15
Rev Method

42 Ad Hoc

43 Ad Hoc

44 Ad Hoc

Team Ad Hoc

Key

Sum 1 2

7 0 1

6 0 1

ooo

4 0 0

I0 0 1

AH NIT

14 17 I 25 26

0 0 I 1 0

0 0 1 1 0

QO0 _ OOql,

0 0 1 0 0

1 0 I 1 0

AI"I AI'-I I A.I-I DT

Figure 5: Individual and Team Defect Summaries (CRUISE). This figureshows the data collectedfrom

one team's CRUISE inspection.The data isidenticalto that of the WLMS inspectionsexceptthat the CRUISE

has fewer defects- 26 versus42 for the WLMS - and the defectkey is different.

defects.

The team defect summary shows whether or not a team discovered a particular defect. This data is gathered

from the defect _eport forms filled out at the collection meetings and is used to assess the effectiveness of each

defect detection method.

The individual defect summary shows whether or not a reviewer discovered a particular defect. This data is

gathered from the defect report forms each reviewer completed during their defect detection activity. Together

with the defect key it is used to assess whether or not each detection technique improves the reviewer's ability

to identify specific classes of defects.

We measuze tl_e value of collection meetings by comparing the team and individual defect summaries to

determine the meeting gain and lossrates.

One team's individualand team defect summaries, and the defectkey axe represented in Figures 4 and

Figure 5.

3.2 Analysis of Team Performance

Figure 6 summarizes the team performance data. As depicted,the Scenario detectionmethod resulted in the

highest defectdetectionrates,followedby the Ad Hoc detectionmethod, and finallyby Checklist.the detection

method.

Table 2 presents a statisticalanalysisof the team performance data as outlinedin Section 2.1.5.The independent variablesaze listedfrom the most to the leastsignifi_nt.The Detection method and Specificationare

significant,but the Round, Replication,and Order are not.

Next, we analyzed the combined Instrumentation and Treatment effects.Table 3 shows the input to this

10022514L 3-16
O

oiO

CRUSE

=,J _*j

i i I i _ I

*_I _ Method _ci_Jon Round l=_o_on O_ler

V,_IABL,E .

Figure 6: Defect Detection Rates by Independent Variable. The dashes in the far left column show each

team's defect detection rate for the WLMS and CRUISE. The horizontal line is the average defect detection rate.

The plot demonstrates the ability of each variable to explain variation in the defect detection rates. For the

Specification variable, the vertical location of WLMS (CRUISE) is determined by av_g the defect detection

rates for all teams inspecting _ArLMS (CRUISE). The vertical bracket, ], to the right of each variable shows one

standard error of the difference between two settings of the variable. The plot indicates that both the Method

and Specification are significant; but Round, Replication, and Order are not.

Independent S_r Pr SSI_ Tie

Variable

Detection Method - treatment .200 2 .359 29

Specification- instrumentation .163 I .396 30

Inspection round- maturation .007 1 .551 30

.007 1 .551 30

.003 1 .003 30

1.28915I .268I 16I

Erperimental run - replication

Order- presentation

Te_m composition - selection

Si_.ca,lBce

Level

8.064 < .01

12.338 < .01

.391 .54

.391 .54

.141 .71

1.151 .39 I

Table 2: Analysis of Variance for F__achIndependent Variable. The analysisof varianceshows that only

the choiceof detectionmethod and specificationsignificantlyexplainvariationin the defectdetectionrate.Team

composition isalsonot significant.

analysis.Sixofthe cellscontainthe average detectionrateforteams usingeach detectionmethod and specification

(3 detectionmethods apphed to 2 specifications).The resultsof thisanalysis,shown in Table 4, indicatethat the

interactionbetween Specificationand Method isnot significant.This means that although the average detection

rates varied for the two specifications,the effectof the detection methods is not linkedto these differences.

Therefore,we rejectthe nullhypothesis that the detectionmethods have no effecton inspectionperformance.

I0022514L 3-17
Table3:

teams.

Specification

WLMS

(average)

Cruise

(average)

Detection Method

Ad Hoc

.5 .38.29.5.48.45"

.43

.46.27.27.23.38.23.35

.31

Checklist Scenario

.29 .'52.5.33 .74.57.55.4.62.55

.41 :57

.19.31.23.23 .5.42.42.54.35

.24 .45

Team Defect Detection Rate Data. The nominal and average defect detection rates for all 16

_ect SST _z SSR v_ (SSr/_z)(_a/SSR)

Detection' Method .200 2 .212 26

Specification 143 i .212 26

MethxSpec .004 2 .212 26

S_carlce

Levd

12.235 < .01

"17.556 < .01

.217 .806

Table 4: Analysis of Variance of Detection Method and Specification. This table displays the results of

an analysis of the variance of the average detection rates given in Table 3.

3.3 Effect of Scenarios on Individual Performance

We initiallyhypothesized that increasing the specializationand coordinationof each reviewer'sresponsibilities

would impzove team performance. We proposed that the Scenario would be one way to achieve this.We have

shown above that the teams using Scenarios were the most effective.However, thisdid not establishthat the

improvement was due to increasesin specializationand coordination,and not to some other factor.

Consequently, our concern isto determine exactlyhow theuse of Scenariosaffectedthe reviewer'sperformance.

To examine this,we formulated two hypothesis schemas.

• HI: Method X reviewers do not find any more X defects than do method Y reviewers.

• H2: Method X reviewers find either a greater or s_naller number of non X defects than do

method Y reviewers.

Alternativeexplanationsfor the observed improvement could be (1) the Scenarioreviewersresponded to some

perceived expectation that their performance should improve; or (2)the Scenarioapproach improves individual

performance regardlessof Scenario content.

3.3.1 Rejecting the Perceived Expectation Argument

IfScenario reviewersperformed betterthan Checklistand Ad Hoc reviewerson both scenario-targetedand nonscenario-targeteddefects,then we must considerthe possibilitythat theirimprovement was caused by something

10022514L 3-18
Reviewers Using Method Finding Defects of TypeDetection Number Defect .Number

Method Reviewers Population Present

DT 6 DT 14

MF 6 MY 5

IF 6 IF 5

CH 12 CH 38

AH 18 AH 42

Compared with Reviewers using Method

DT MF IF CH AH

- .02 .06 .01 .02

(6.5) (3) (4.5) (4) (4)

.07 .12 .02 .04

(0.5) (2) (1) (0) (1)

.01 .01 - .04 .01

(o) (1) (z.5) (1) (1)

.95 .86 .89 - .51

(10.5) (11) (12.5) (8) (10)

.91 .84 .75 .37 -

(12) (12.5) (13) (9.5) (11)

Table 5: Significance Table for H1 hypotheses: WLMS inspections. This table teststhe HI hypothesis

- Method X reviewers do not find any more X defectsthan do method Y reviewers- for allpairsof detection

methods. Each row in the tablecorresponds to a population of reviewers and the population of defectsforwhich

they were responsible,i.e.,method X reviewers and X defects. The lastfivecolumns correspond to a second

reviewerpopulation, i.e.,method Y reviewers.Each cellin the lastfivecolumns containstwo values.The first

value isthe probabilitythat H1 istrue,using the one-sided Wilcoxon-Mann-Whitney test.The second value -

in parentheses - isthe median number of defectsfound by the method Y reviewers.

Reviewers Using Method Finding Defects of Type

Detection Number Defect Number

Method Reviewers Population Present

DT 5 DT 10

MF MF 1

IF

CH 12

AH 21

IF 3

CH 24

AH 26

Compared with Reviewers using Method

DT MF IF CH AH

- .05 .03 <.01 .02

(6) (3) (2) (i) (3)

NA NA NA NA

(0) (0) (0) (0) (0)

NA NA - NA NA

(0) (0) (0) (0) (0)

> .99 .95 .93 .98

(8) (5) (4) (2.5) (5)

.96 .50 .41 .02

(s) (5) (5) (3) (5)

Table 6: Significance Table for HI hypotheses: CRUISE inspections. This analysisis identicalto that

performed for WLMS inspections.However, we chose not to perform any statisticalanalysisfor the Missing

Functionalityand IncorrectFunctionalitydefectsbecause there are too few defectsof those types.

10022514L 3-19
Reviewers Using Method

Detection Number

Method Reviewers

DT 6

Finding Defects of Typ e

Defect Number

Population Present

DT c 28

M_F 6 MF ¢ 37

IF 6 l}_ 37

C£I 12 CH ¢

AH 18 AH ¢ 0

Compared with Reviewers using Method

DT MF IF CH

- .92 .82 .50

(4.5) (9) (7.5) (5.5)

.87 .83 .56

(11) (9.5) (12.5) (8.5)

.66 .53 .24

(11) (12) (11.5) (8.5)

.12 .28 .35

(0.5) (1) (1) (1)

NA NA NA NA

(0) (0) (0) (0)

AH

.64

(6)

.64

(10)

.27

(10)

.07

(1)

(o)

Table 7: Significance Table for H2 hypothesis: WLMS inspections. This table tests the H2 hypothesis

- Method X reviewers find a greater or smaller number of non X defects than do method Y reviewers - for all

pairs of detection methods. Each row in the table corresponds to a population of reviewers and the population of

defects for which they were not _ponsible - i.e., method X reviewers and non X defects (the complement of the

set of X defects). The last five columns correspond to a second reviewer population, i.e., method Y reviewers.

Each cell in the last five columns contains two v-Mum. The first value is the probability that H2 is true, using the

two-sided Wilcoxon-Mann-Whitney test. The second value is the median number of defects found by the method

Y reviewers.

other than the scenarios themselves.

One possibility was that the Scenario reviewers were merely reacting to the novelty of using a clearly different

approach, or to a perceived expectation on our part that their performance should improve. To examine this

we analyzed the individual defect summaries to see how Scenario reviewers differed from other reviewers.

The detection rates of Scenario reviewersrare compared with those of all other reviewers in Tables 5, 6, 7

and 8. Using the one and two-sided Wilcoxon-Mann-Whitney tests [15], we found that in most cases Scenario

reviewers were more effective than Checklist or Ad Hoc reviewers at finding the defects the scenario was designed

to uncover. At the same time, all reviewers, regardless of which detection method each used, were equally effective

at finding those defects not targeted by any of the Scenarios.

Since Scenario reviewers could not have known the defect classifications, it is unlikely that their reporting could

have been biased. Therefore these results suggest that the detection rate of Scenario reviewers shows improvement

only with regard to those defects for which they are explicitly responsible. Consequently, the argument that the

Scenario reviewers' improved performance was primarily due to raised expectations or unknown motivational

factors is not supported by the data.

ri.e., reviewe_ using Scenarios.

10022514L 3-20
Reviewers Using Method Finding Defects of Type

Detection Number Defect Number

Method Reviewers Population Present

DT 5 DT ¢ 16

MF 5 MF c 25

IF 5 IF ¢ 23

CH 12 CH ¢ 2

AH 21 AH ¢ 0

Compared with Reviewers using Method

DT MF IF CH

.59 .86 .37

(2) (2) (3) (2)

.96 - .33 .06

(8) (5) (4) (3)

.96 .41 - .44

(S) (4) (5) (2.5)

NA NA NA -

(0) (t) (0) (0)

NA NA NA NA

(0) (0) (0) (0)

AH

.46

(2)

.62

(5)

.57

(5)

NA

(0)

(0)

Table 8: Significance Table for H2 hypothesis: CRUISE inspections. This analysisisidentical to that

performed forWLMS inspections.However, we chose not to perform statisticalanalysisforthe non non Checklist

defectsbecause thereare too few defectsof that type.

3.3.2 Rejecting the General Improvement Argument

Another possibility is that the Scenario approach rather than the content of the Scenarios was responsible for

the improvement.

Each Scenario targets a specific set of defects. If the reviewers using a type X Scenario had been no more

effective at finding type X defects than had reviewers using non-X Scenarios, then the content of the Scenarios

did not significantly influence reviewer performance. If the reviewers using a type X Scenario had been more

effective at finding non-X defects than had reviewers using other Scenarios, then some factor beyond content

caused the improvement.

To explore these possibilities we compared the Scenario reviewers' individual defect summaries with each

other.

Looking again at Tables 5, 6, 7, and 8 we see that each group of Scenario reviewers were the most effective

at finding the defects their scenarios were designed to detect, but were generally no more effective than other

Scenario reviewers at Ruding defects their Scenarios were not designed to detect.

Since Scenario reviewers showed improvement only in finding the defects for which they were explicitly responsible, we conclude that the content of the Scenario was primarily responsible for the improved reviewer

performance.

10022514L 3-21
O

t'q

|

,,° o

g.

O

O

i

0

,..,..,...,.._,..._,...,........... .._.,....,..,.._

5 10 15

Re_w Number

Figure 7: Meeting Gains for WLMS Inspections. Each point represents the meeting gain rate for a single

inspection, i.e., the number of defects first identified at a collection meeting divided by the total number of defects

in the specification. Each rate is marked with symbol indicating the inspection method used. The vertical line

segment through each symbol indicates one standard deviation in the estimate (assuming each defect was a

Bernoulli trial). This information helps in assessing the significance of any one rate. The average meeting gain

rate is 4.7_: 1.3% for the WLMS. (3.14" 1.1% for the CRUISE.)

3.4 Analysis of Checklists on Individual Performance

The scenarios used in this study were derived from the checklist. Although this che_t targeted a large number

of existing defects, our analysis shows that the performance of Checklist teams were no more effective than Ad

Hoc teams. One explanation for this is that nonsystematic techniques axe di_cuJtfor reviewers to implement.

To study this explanation we again tested the H1 hypothesis that Checklist reviewers were no more effective

than Ad Hoc reviewers at finding Checklist defects.

From Tables 5 and 6 we see that even though the Checklist targets a large number of defects, it does not

actually improve a reviewer's ability to fiud those defects.

3.5 Analysis of Collection Meetings

In his origin_ paper on software inspections Fagan [6] asserts that

Sometimes flagrant errors are found during ... [defect detection], but in general, the number of errors

found is not nearly as high as in the... [collection meeting] operation.

10022514L 3-22
From a study ofover 50 inspections,Votta [17]collecteddata that stronglycontradictsthisassertion.In this

Section,we measure the benefitsof collectionmeetings by comparing the team and individualdefectsummaries

to determine the meeting gain and meeting lossrates.(See Figure 4 and Figure 5).

A _meeting gain_ occurswhen a defectisfound for the fast time at the collectionmeeting. A "meeting loss"

occurs when a defectis firstfound during an individual'sdefectdetectionactivity,but itis subsequently not

recorded during the collectionmeeting. Meeting gains may thus be offsetby meeting lossesand the difference

between meeting gainsand meeting lossesisthe net improvement due to collectionmeetings.

Our resultsindicatethat collectionmeetings produce no net improvement.

3.5.1 Meeting Gains

The meeting gain ratesreported by Votta were a negligible3.9 ± .7%. Our data tellsa similarstory. (Figure 7

displaysthe meeting gain ratesforWLMS inspections.)The mean gain rateis4.7± 1.3% forWLMS inspections

and 3.1± 1.1% forCRUISE inspections.The ratesare not significantlydifferent.

It is interestingto note that these resultsare consistentwith Votta's earlierstudy even though Votta's

reviewerswere professionalsoftware developersand not students.

3.5.2 Meeting Losses

The average meeting loss rates were 6.8 + 1.6% and 7.7 ± 1.7% for the WLMS and CRUISE respectively. (See

Figure 8.)

One cause of meeting lossmight be that reviewersare talkedout of the beliefthat something is a defect.

Another cause may be that during the meeting reviewersforgetor can not reconstructa defectfound earlier.

This effecthas not been previouslyreportedinthe literature.However, sincethe intervalbetween the detection

and collectionactivitiesisusuallylongerin practicethan itwas in our experiment (one to two days in our study

versusone or two weeks in practice),thiseffectmay be quitesignificant.

3.5.3 Net Meeting Improvement

The average net meeting improvement is -.9±2.2 for WLMS inspections and -1.2± 1.7 for CRUISE inspections.

(Figure 9 displays the net meeting improvement for WLMS inspections.) We found no correlations between the

loss, gain, or net improvement rates and any of our experiment's independent variables.

0022514L 3-23
LO

E

br_

o

o

2

O

I.......

i i 1

5 10 15

Te_n

Figure 8: Meeting Loss Rate for WLMS Inspections. Each point repr--_nts the meeting loss rate for a

single inspection. The meeting loss rate is the number of defects first detected by an individual reviewer divided

by the total number of defects in the specification. Each rate is marked with a symbol indicating the inspection

method used. The vertical line segment through each symbol indicates one standard deviation i_ the estimate

of the rate (assuming each fault was a Bernoulli trial). This information helps in determining the significance of

any one rate. The average team loss rate is 6.8 ± 1.6% for the WLMS. (7.7 ± 1.7% for CRUISE).

4 Summary and Conclusions

Our experimental design for comparing defect detection methods is flexible and economical, and allows the

experimenter to assess several potential threats to the experiment's internal validity. In particular, we determined

that neither maturation, replication, selection, or presentation effects had any significant influence on inspection

pefformazce. However, differences in the SRS did.

From our analysis of the experimentzl data we drew several conclusions.

1. The defect detection rate when using Scenarios is superior to that obtained with Ad Hoc or

Checklist methods - an improvement of roughly 35%.

2. Scenarios help reviewers focus on specific defect classes. Furthermore, in comparison to Ad Hoc

or Checklist methods, their ability to detect other classes of defects is not compromised. (It should be

noted however, that the scenarios appeared'to be better suited to the defect profile of the WLMS than the

CI_UISE. This indicates that poorly designed scenarios may lead to poor inspection performance.)

3. The Checklist method - the industry standard, was no more effective than the Ad Hoc

10022514L 3-24
O

o

.J

_c

o .... ....

0 5 10 15

Te_rl

Figure 9: Net Meeting Improvement for WLMS. Each symbol indicates the net meeting improvement for

a single inspection. The average net meeting improvement rate is -.9 ± 2.2 for the WLMS. (-1.2 + 1.7 for the

CRUISE). These rates are not significantly different from 0.

detection method.

4. On the average, collection meetings contribute nothing to defect detection effectiveness.

The resultsof thiswork have important implicationsfor software practitioners.The indicationsare that

overallinspectionperformance can be improved when individualreviewersuse systematicprocedures to address

a small set of specificissues.This contrastswith the usual practice,in which reviewershave neithersystematic

procedures nor clearlydefinedresponsibilities.

Economical experimental designs are necessary to allow replicationin other environments with different

populations. For software researchers,this work demonstrates the feasibilityof constructingand executing

inexpensive experiments to validatefundamental researchrecommendations.

5 Future Work

The experimental data raise many interestingquestionsforfuturestudy.

• In many instancesa singlereviewer found a defect,but the defectwas not subsequently recorded at the

collectionmeeting. Are singlereviewers sometimes forgettingto mention defectsthey observed, or is

10022514L 3-25
the reviewer being talked out of the defect at the team meeting? What are the significant suppression

mechanisms affecting collection meetings?

• Very few defects are initially discovered during coliection meetings. Therefore, in view of their impact on

production interval, are these meetings worth holding?

• More than half of the defects are not addressed by the Scenarios used in this study. What other Scenarios

are necessary to achieve a broader defect coverage?

• There are several threats to this experiment's external validity. These threats can only be addressed by

replicating and reproducing these studies. Each new run reduces the probability that our results can be

explained by human variation or experimental error. Consequently, we are creating s laboratory kit (i.e.,

a package containing all the experimental materials, data, and analysis) to facilitate replication. The kit

should be publicly available by June, 1994.

• Finally, we are using the lab kit to reproduce the experiments with other uaiversity researchers in Japan,

Germany, Italy, and Australia and with industrial developers at AT&T Bell Laboratories and Motorola

Inc. These studies will allow us to evaluate our hypotheses with different populations of programmers and

different software art/facts.

Acknowledgments

We would like to recognize the efforts of the experimental participants - an excellent job was done by all. Our

thanks to Mark Ardis, John Kelly, and David Weiss, who helped us to identify sample requirements specifications

and inspection checklists, and to John Gannon, Richard Gerber, Clive Loader, Eric Slud and Scott VanderWeil

for their valuable technical comments. Finally, Art Caso's editing is greatly appreciated.

I0022514L 3-26
References

[1] IEEE Gnide to Software Requiremenl.s Specifications. Soft. Eng. Tech. Comm. of the IEEE Computer Society,

1984. IEEE Std 830-1984.

[2] V. tL Basili and D. M. Weiss. Evaluation of a software requixements document by analysis of change data.

In Proceedings of the Fifth International Conference on Software Engineering, pages 314-323, San Diego,

CA, March 1981.

[3] Barry W. Boehm. Software Engineering Economics. Prentice Hall, Engl.ewood Cliffs, NJ, 1981.

[4] G. E. P. Box, W. G. Hunter, and J. S. Hunter. Statistics for Ezperimenters. John Wiley & Sons, New York,

1978.

[5] Stephen G. Eick, Clive R. Loader, M. David Long, Scott A. Vander Wiel, and Lawrence G. Votta. Estimating software fault content before coding. In Proceedings of the l_th International Conference on Software

Engineering, pages 59-65, May 1992.

[6] M. E. Fagan. Design and code inspections to reduce errors in program development. IBM Systems ]ournal,

15(3):182-211, 1976.

[7] Kathryn L. Heninger. Specifying Software Requirements for Complex Systems: New Techniques and their

Application. IEEE Transactions on Software Engineering, SE-6(1):2-13, 3anuary 1980.

[8] Watts S. Humphery. Managing the Software Process. Addison-Wesley Publishing Co., 1989. Reading,

Massachusetts.

[9] IEEE Guide for the Use of IEEE Standard Dictionary of Measures to Produce Reliable Software. Soft. Eng.

Tech. Comm. of the IEEE Computer Society, 1989. IEEE Std 982.2-1988.

[10] IEEE Standard for software reviews and audits. Soft. Eng. Tech. Comm. of the IEEE Computer Society,

1989. IEEE Std 1028-1988.

[11] Charles M. Judd, Eliot R. Smith, and Louise H. Kidder. Research Methods in Social Relations. Holt,

Rinehart and Winston, Inc., Fort Worth, TX, sixth edition, 1991.

[12] J. Kirby. Example NRL/SCR software requirements for an automobile cruise control and monitoring system.

Technical Report TR-87-07, Wang Institute of Graduate Studies, July 1984.

[13] Dave L. Parnas and David M. Weiss. Active design reviews: principles and practices. In Proceedings of the

8th International Conference on Software Engineering, pages 215-222, Aug. 1985.,

[14] G. Michael Schnieder, Johnny Martin, and W. T. Tsai. An experimental study of fault detection in user

requirements. ACM Transactions on Software Engineering ang Mcthodology, 1(2):188-204, April 1992.

[15] S. Siegel and N.J. Castellan, 3r. Nonparametric Statistics For the Behavioral Sciences. McGraw-Hill Book

Company, New York, NY, second edition, 1988.

[16] J. vanSchouwen. The A-7 requirements model: Re-examination for real-time systems and an application

to monitoring systems. Technical Report TR-90-276, Queen's University, Kingston, Ontario, Canada, May

1990.

[17] Lawrence G. Votta. Does every inspection need a meeting? In Proceedings ofACM SIGSOFT '93 Symposium

on Foundations of Software Engineering. Association for Computing Machinery, December 1993.

[18] William G. Wood. Temporal logic case study. Technical Report CMU/SEI-89-TR-24, Software Engineering

Institute, Pittsburgh, PA, August 1989.

10022514L 3-27
A Ad Hoc Detection

The defect taxonomy is due to the work of Schneider, et al., and Basili and Weiss.

• Omission

- MissingFunctionality:Informationdescribingthe desiredinternaloperationalbehaviorofthesystem

has been omittedfrom the SRS.

- MissingPerformance:Informationdescribingthe desiredperformancespec/fications has eitherbeen

omittedor describedin a way thatisunacceptableforacceptancetesting.

- MissingInterface:Informationdescribinghow the proposedsystem willinterfaceand communicate

with objectsoutsidethe thescopeofthesystem has been omittedfrom theSITS.

- MissingEnvironment:Informationdescribingtherequiredhardware,software,database,orpersonnel

enviroment inwhich the systemwillrun has been omittedfrom the SRS

• Commission

- Ambiguous Information:An important term,phraseor sentenceessentialto the understandingof

system behaviorhas eitherbeen leftundefinedor definedin a way that can causeconfusionand

misunderstanding.

- InconsistentInformation:Two sentencescontainedintheSRS directlycontradicteachotherorexpress

actionsthatcannotboth be correctorcannot both be carriedout.

- IncorrectFact:Some sentencecontainedin the SRS assertsa factsthat cannotbe trueunder the

conditionsspecifiedinthe SRS.

- Wrong Section:Essentialinformationismisplacedwithinthe SRS

10022514L 3-28
B Checklist Method

• General

- Are the goals of the system defined?

- Are the requirements clear and unambiguous7

- Is a functional overview of the system provided7

- Is an overview of the operational modes provided?

- Have the software and hardware environments been specified?

- If assumptions that affect implementation have been made, are they stated?

- Have the requirements been stated in terms of inputs, outputs, and processing for each function?

- Are all functions, devices, constraints traced to requirements and vice versa?

- Are the required attributes, assumptions and constraints of the system completely listed?

• Omission

- Missing Functionality

• Are the d_ribed functions sufficient to meet the system objectives?

• Are all inputs to a function sufficient to perform the required function?

• Axe undesired events considered and their requked responses specified?

• Are the initial and special gates considered (e.g., system initiation, abnormal termination)?

- Missing Performance

• Can the system be tested, demonstrated, analyzed, or inspected to show that it satisfies the

requirements?

• Have the data type, rate, units, accuracy, resolution, limits, range and critical values

• for all internal data items been specified?

• Have the accuracy, precision, range, type, rate, units, frequency, and volume of inputs and outputs

been specified for each function?

- Missing Interface

• Are the inputs and outputs for all interfaces sufficient?

• Are the interface requizements between hardware, software, personnel, and procedures included?

- Missing Environment

• Have the functionality of hardware or software interacting with the system been properly specified?.

• Commission

- Ambiguous Information

• Are the individual requirements stated so that they are discrete, unambiguous, and testable?

• Are all mode transitions specified determJnisticly?

- Inconsistent Information

• Are the requirements mutually consistent?

• Are the functional requizements consistent with the overview?

• _re the functionalrequirements consistentwith the actualoperating environment?

- Incorrect or Extra Functionality '

• Are all the describedfunctionsnecessaryto meet the system objectives?

• Are allinputs to a function necessaryto perform the requiredfunction?

• Are the inputs and outputs forallinterfacesnecessary?

• Are allthe outputs produced by a function used by another function or transferredacross an

external interface?

- Wrong Section

• Are all the requirements,-interfaces, constraints,etc.listed in the appropriatesections.

10022514L 3-29
C Scenarios

(a)

(b)

(c)

C.1 Data Type Consistency Scenario

I. Identifyalldata objectsmentioned in the overview (e.g.,hardware component, applicationvariable,abbredated term or function)

(a) Are alldata objectsmentioned in the overview listedin the externalinterfacesection?

2. For each data object appearing in the externalinterfacesectiondetermine the followinginformation:

Object name:

Class: (e.g.,input port,output port,applicationvariable,abbreviated term, function)

Data type: (e.g.,integer, time,boolean, enumeration)

Acceptable values:Are there any constraints,ranges, limitsforthe values of thisobject

Failurevalue:Does the objecthave a specialfailurevalue?

Units or rates:

Initial value:

.

Isthe object'sspeci_cationconsistentwith itsdescriptionin the overview?

Ifobjectrepresentsa physicalquantity,are itsunits properly specified?

Ifthe object'svalueis computed, can that computation generate a non-acceptable value?

For each functionalrequirement identifyalldata object references:

(a) Do alldata objectreferencesobey formatting conventions?

(b) Are alldata objectsreferencedin thisrequirement listedin the input or output sections?

(c) Can any data objectuse be inconsistentwith the data object'stype, acceptablevalues,failurevalue,

etc.?

(d) Can any data objectdefinitionbe inconsistentwith the data object'stype, acceptablevalues,failure

value,etc.?

C.2 Incorrect Functionality Scenario

1. For each functional requirement identify all input/output data objects:

(a) Are allvalueswrittento each output data objectconsistentwith itsintended function?

(b) Identifyat leastone functionthat uses each output data object.

2. For each functionalrequirement identifyallspecifiedsystem events:

(a) Is the specificationofthese eventsconsistentwith theirintended interpretation?

3. Develop an invariantfor each system mode (i.e.Under what conditionsmust the system exitor remain in

a given mode)?

(a) Can the system'sinitialconditionsfailto satisfythe initialmode's invariant?

(b) Identifya sequence of events that allows the system to enter a mode without satisfyingthe mode's

invaxiant.

(c) Identifya sequence of eventsthat allowsthe system to enter a mode, but never leave (deadlock).

10022514L 3-30
C.3 Ambiguities Or Missing Functionality Scenario

1. Identify the required precision, response time, etc. for each functional requirement.

(a) Are all required precisions indicated?

2. For each requirement, identify all monitored events.

(a) Does a sequence of events exist for which multiple output values can be computed?

(b) Does a sequence of events exist for which no output value will be computed?

3. For each system mode, identify all monitored events.

(a) Does a sequence of events exist for which transitions into two or more system modes is a_lowed?

10022.514L 3-31

Software

Process

Evolution

at the SEL

VICTOR BASIH, University of Maryland

SCOTT GREEN, NASA Goddard Space Flight Center

0 The Soft'ware En_neering

Laborato77 has been adapting,

analyzing, and evolving

sofr'c:areprocessesfor the last

18years. Their apt.'oacb is based

on the Quality Improvement

Paradigm, which is used to

_,atuate process _Cfectson both

product and people. The authors

ewplain this aptn_ach as it was

applied to reduce d_Cectsin code.

Since 1976, the Software

Engineering Laboratory of the

National Aeronautics and Space

Administration's Goddard Space

Flight Center has been en_ged in a

program of understanding, assessing,

and packa_ng soft_-are exTperience.

Topics of study include process, product, resource, and defect models, as

well as specific technolo_es and tools.

The approach of the SEL _ a consortium of the Software Engineering

Branch of NASA Goddard's Flight

Dynamics Division, the Computer

Science Department of the University

of Maryland, and the Soft_'are

En_neering Operation of Computer

Sciences Corp. -- has been to _in an

in-depth understanding of project and

environment characteristics using

process models and baselines. A

process is evaluated for study., applied

experimentally to a project, analyzed

with respect to baselines and process

model, and evaluated in terms of the

experiment's goals. Then on the basis

of the experiment's conclusions,

results are packaged and the process is

tailored for improvement, applied

again, and reevaluated.

In this article, we describe our

improvement approach, the Quality"

Improvement Paradigm, as the SEL

applied it to reduce code defects by

emphasizing reading techniques. The

box on p. 63 describes the Quality."

Improvement Paradigm in detail. In

examining and adapting reading techniques, we go through a systematic

process of evaluating the candidate

10022514L

0"_40-74.T_194,/S04 00 G lq_ E._

3-33

PRECF.DING-PA E I LP,, IK,ROT

J U LY 199,4
process and refining its implementation through lessons learned from previons experiments and studies.

As a result of this continuous, evolutionary process, we determined that

we could successfully apply key elements of the Cleanroom development method in the SEL environmerit, espedally for projects involving

fewer than 50,000 lines of code (all

references to lines of code refer to

developed, not delivered, lines of

code). We saw indications of lower

error rates, higher productivity., a

more complete and consistent set of

code comments, and a redistribution

of developer effort Although we have

not seen similar reliability, and cost

gains for larger efforts, we continue to

invesd_te the Cleanroom method's

effect on them.

EVALUATINGCANDIDATEFROCESSES

To enhance the possibility of

improvement in a particular environwent, the SEL introduces and evaluates new technology _ithin that environment. This involves experimentation with the new technolo_-, recording fmdin_ in the context of lessons

learned, and adjusting the associated

processes on the basis of this experience. When the technology is notably"

riskT -- substantially different from

what is familiar to the em-ironment m

or requires more detailed e_aluation

validity and credibility" through the use

of typical development systems and

professional staff. In analyzing both

controlled experiments and case studies, the Goal/Question/Metric paradigm, described in the box on p. 63,

provides an important framework for

focusing the analysis.

On the basis of experimental

results, the SEL packages a set of

lessons learned and makes them a_ilable in an experience base for future

analysis and application of the technology.

Experiment1: Readingversns testing.

Although the SEL had historically

been a test-driven organization, we

decided to experiment with introducing reading techniques. We were particularly interested in how reading

would compare with testing for fault

detection. The goals of the first offline, controlled experiment_ were to

analyze and compare code reading,

functional testing, and structural testing, and to evaluate them with respect

to fanlt-detection effectiveness, cost,

and classes of faults detected.

We needed an analysis from the

viewpoint of quality assurance as well

as a comparison of performance with

respect to soft_are _-pe and programmet experience. L'sing the GQM paradigm, we generated specific questions

on the basis of these goals.

We had subjects use reading by

stepwise abstraction,: equivalence-parthan would normally be e.-qw.nded, the titioning boundary-value testing, and

SEL conducts experimentation offline from the project environment.

Off-line experiments may take the

form of either controlled experiments

,, or case studies. Controlled experiments are warranted when the SEL

,i needs a detailed analysis with statistical

i,iJ assurance in the results. One problem

!; uith controlled experiments is that the

;i project must be small enough to repli-

!: cute the experiment several times. The

i SEL then performs a case study to validate the results on a project of credible size that is representative of the

environment. The case study adds

statement-coverage structural testing.

We conducted the experiment

twice at the University of ;_hryland on

graduate students (42 subjects) and

once at N.'LSA Goddard (32 subjects).

The experiment structure was a fractional factorial design, in which eyeDsubject applied each technique on a

different program. The programs

included a text formatter, a plotter, an

abstract data _'pe, and a database, and

they ranged from 145 to 365 lines of

code. VCe seeded each program with

faults. The reading performed was at

the unit level.

Although the results from both

experiments support the emphasis on

reading techniques, we report only the

results of the controlled experiment on

the NASA Goddard subjects because it

involved professional developers in the

target environment.

Figure 1 shows the fault-detection

effectiveness and rate for each

approach for the NASA Goddard

experiment. Reading by stepwise

abstraction proved superior to testing

Figure I. Remdts of the reading-versus-testing controlled experiment, in

which reading was compared with

fmlcrional and st-t'u,Tural testing. (A)

Mean number of faults detected for

eack technique and (B) number of

filulrs detected po" hour of use for each

tect,niqm,, i

:EEE SO =TWo' ¢IE 59

10022514L 3-34

•_,_ .....

pooe,., jz.n:V,
techniques in both the effectiveness

and cost of fault detection, while obviously using fewer computer resources.

Even more interesting was that the

subjects did a better job of estimating

the code quality using reading than

they did using testing. Readers

thought they had found only about

half the faults (which was nominally

correct), while functional testers felt

that had found essentially all the faults

(which was never correct).

Furthermore, after completing the

experiment, more than 90 percent of

the participants thought functional

testing had been the most effective

technique, although the results dearly

showed otherwise. This gave us some

insight into the psychological effects of

reading versus testing. Perhaps one

reason testing appeared more satisfying was that the successful execution of

multiple test cases generated a greater

comfort level with the product quality,

actually providing the tester with a

false sense of confidence.

Reading was also more effective in

uncovering most classes of faults,

including interface faults. This told us

that perhaps reading might scale up

well on larger projects.

On the basis of these results, we

decided to emphasize reading techniques in the SEL environment.

However, we saw little improvement

in overall reliability of the development systems. Part of the reason may

have been that SEL project personnel

had developed such faith in testing

that the quality of their reading was

relaxed, with the assumption that testing would ultimately uncover the

same faults. We conducted a small

off-line experiment at the University

of Maryland to test this hypothesis;

the results supported our assumption.

(We did this on a small scale just to

verify our hypothesis before continuing with the Cleam'oom experiment.)

_y the _nr00m method?.The Cleanroom method emphasizes human discipline in the development process,

using a mathematically based design

approach and a statistical testing

approach based on antidpated operaFigure 2. Sample measures, baselines, and expectations for _he case studies ircoestigating _be Cleanroom method.

tional use.3 Development and testing

teams are independent, and all development-team activities are performed

without on-line testing.

Techniques associated with the

method are the use of box structures

and state machines, reading by stepwise abstraction, formal correctness

demonstrations, and peer review.

System development is performed

through a pipeline of small increments

to enhance concentration and permit

testing and development to occur in

parallel.

Because the Cleanroom method

removes developer testing and relies

on human discipline, we felt it would

overcome the psy.cholo_cal barrier of

reliance on testing.

_l_l_ _ OIP.The first step of the

Quality Improvement Paradigm is to

characterize the project and its environment. The removal of developer

unit testing made the Cleanroom

method a high-risk technology. Again,

we used off-line experimentation at

the University of Maryland as a mitigating approach. 4 The environment

was a laboratory course at the university, and the project involved an electronic message system of about 1,500

LOC. The experiment structure was a

simple replicated design, in which

control and experiment teams are

defined. We assigned 10 three-person

experiment teams to use the

Cleanroom method. We gave five

three-person control teams the same

development methodology, but

allowed them to test their systems.

Each team was allowed five independent test submissions of their programs. We collected data on programmer background and attitude, computer-resource activity, and actual

testing results.

The second step in the Quality

Improvement Paradigm is to set goals.

The goal here was to analyze the

effects of the Cleanroom approach and

evaluate it with respect to process,

product,and participants,ascompared

with the non-Cleanroom approach.

J U LY 1994

10022514L 3-35

oF.
We generated questions corresponding to this goal, focusing on the

method's effect on each aspect being

studied.

The next step of the Quality Improvement Paradigm involves selecting an appropriate process model. The

process model selected for this experiment was the Cleanroom approach as

defined by Harlan Mills at IBM's

Federal Systems Division, but modified for our environment. For example, the graduate-student assistant for

the course served as each group's independent test team.._so, because we

used a lang_aage unfamiliar to the subjects to prevent bias, there was a risk of

errors due solely to ignorance about

the langlaage. We therefore allowed

teams to cleanly compile their code

before submit_g it to the tester.

Because of the nature of controlled

experimentation, we made _w modifications during the experiment.

CAeanroom's effect on the softwaredevelopment process resulted in the

Cleam-oom developers more effectively applying the off-line reading techniques; the non-Cleanroom teams

focused their efforts more on functional testing than reading. The

Cleanroom teams spent less time onLineand were more successful in making scheduled deliveries. Further

analysis revealed that the Cleanxoom

products had less dense complexity, a

higher percentage of assignment statements, more global data, and more

code comments. These products also

more completely met the system

requirements and had a higher percent.age of successful independent test

_es.

The Cleanroom de, elopers indicated that they modified their normal

software-development activities by

doing a more effective job of reading,

though they missed the satisfaction of

actual program execution. Almost all

said they would be willing to use

Cleanroom on another development

assignment.

Through observation, it was also

clear that the Cleanroom developers

IEEE SOFTWARE

did not apply the formal methods

associated with Cleanroom very rigorously. Furthermore, we did not have

enough failure data or experience with

Cleanroom testing to apply a reliability model. However, general analysis

did indicate that the

Cleanroom approach had

potential payoff, and that

additional investigation

was warranted.

You can also view this

experiment from the following perspective: We

applied two development

approaches. The only

real difference between

them was that the control teams had one extra

piece of technology

(developer testing), yet

they did not perform as well as the

experiment teams. One explanation

might be that the control group did

not.use the available nontesting techuiques as effectively because they knew

they could rely on testing to detect

faults. This supports our earlier findings associated with the reading-versus-testing experiment.

EVOLVINGSELECTEDPROCESS

The positive results gathered from

these two experiments gave us the justification we needed to explore the

Cleanroom method in case studies,

using typical development systems as

data points. Vee conduct.ed two case

studies to examine the method, again

following the steps of the Quality

Improvement Paradigm. A third case

study wasalsorecently begun.

First case study. The project we

selected, Project 1, involved two subsystems from a typical attitude

ground-support system. The system

performs ground processing to determine a spacecraft's attitude, receiving

and processing spacecraft telemetry

data to meet the requirements of a

particular mission.

The subsystems we chose are an

integral part of attitude determination and are highly algorithmic.

Both are interactive programs that

together contain approximately

40,000 LOC, represen_ng about 12

ALMOST

ALLTHE

CLEANROOM

TEAMSAID

THEY'DUSE

THEMETHOD

AGAIN.

percent of the entire

attitude ground-support

system. The rest of the

ground-support system

was developed using

the standard SEL development methodology.

The project was

staffed principally by five

people from the Flight

Dynamics Division,

which houses the SEL.

All five were also workhag on other projects, so

only part of their time

was allocated to the two subsystems.

Their other responsibilities often took

time and attention away from the case

study, but tiffs partial allocation represents typical staffing in this environment. All other projects with which

the Project I salt were involved were

non-Cleanroom efforts, so staff members would often be required to use

multiple develop-meat methodologies during the same workday.

The primary goal of the first ease

study was to increase software quality

and reliability without increasing cost.

We also wanted to compare the characteristics of the Cleanroom method

with those typical of the FDD environment. A well-calibrated baseline

was available for comparison that

described a variety of process characteristics, including effort distribution,

change rates, error rates, and producfivky. The baseline represents the history of many earlier SEL studies.

Figure 2 shows a sample of the expected variations from the SEL baselines

for a set of process characteristics.

Choosingand tailoring processes. The

process models available for examination were the standard SEL model,_

which represents a reuse-oriented

waterfall life-cycle model; the

I0022514L 3-36
IBM/FSD Cleanroom model, which

appeared in the literature and was

available through training; and the

experimental University of Maryland

Cleanroom model, which was used in

the earlier controlled experiment.*

W'e examined the lessons learned

from applying the IBM and University

of Maryland models. The results from

the IBM mode] were notably positive,

showing Bat the basic process, methods, and techniques were effecuve for

that particular environment. However,

the process model had been applied by.

the actual developers of the methodology, in the environment for which it

was developed. The University of

Mary-land model also had specific

lessons, including the effects of not

allowing developers to test their code,

the effectiveness of the process on a

small project, and the conclusion that

formal methods appeared particularly difficult to apply and required specific _.

On the basis of these lessons and the

characteristics of our

environment, we selected a Cleanroom process model with four

key elements:

, separation of development and test teams,

* reliance on peer

review instead of unitlevel testing as the primary developer verification tedmique,

, use of informal

state machines and

functions to define the

syst_ design, and

, a statistical approach to testing

basedon operationalscenafio_

_Arealso provided trainingfor the

subjects,consistentwith a Universityof

Maryland course on the Cleanroom

process model, methods, and techniques, with emphasis on reading

through step,rise abstraction. "tArealso

stressed code reading by multiple

reviewers because stepwise abstraction

was new to many subjects. Michael

Dyer and Terry Baker of IBM/FSD

PROJECT

RESULTS

LEDUSTO

EMPHASIZE

PEERREVIEWS

ANDUSEOF

INDEPENDENT

TESTING.

i provided additional training and motivation by describing IBM's use of

Cleanroom.

To mitigate risk and address the

developers' concerns, we examined

backout options for the experiment.

For example, because the subsystems

were highly mathematical, we were

aft-aid it would be difficult to find and

correct mathematical errors without

any developer testing. Because the project was part of an operational system

with mission deadlines, we discussed

options that ranged from allowing

developer unit testing to discontinuing

Cleanroom altogether. These discussions helped allay the primary apprehension of NASA Goddard management in using the new methodolo_-.

When we could not get information

about process application, we followed

standardSEL process-modelactivities.

We also noted other management

and project-team concerns.

Requirements and specifications change

frequently during the development

cycle in the FDD environment- This instability was of particular concern because the Cleanroom method is builton

the precept of developing software fight

the first time. Another

concern was that, _ven

the difficulties encountered in the University.

of Marylan. d experiment

about applying formal

methods, how successfullycould a classical

Cleanroom approach be

applied? Finally,there was concern

aboutthe psycholo#caleffectsof separatingdevelopment and testing,specificallythe inabilityof the developersm

execute theircode. We targeted all

these concerns for our postprojcct

analy_$.

Projectl lastedfrom January 1988

@trough September 1990.We separated the five team members into a threeperson development team and a twoperson test team. The development

team broke the total effort into six

incremental builds of approximately

6,500 LOC each..Ma experimenter

team consisting of NASA Goddard

managers, SEL representatives, a technolo_ advocate familiar with the IBM

model, and the project leader monitored the overall process.

We modified the process in real

time, as needed. For example, when we

merged Cleartroom products into the

standard FDD formal review and documentation activities, we had to modify.

both. We altered the desi_ process to

combine the use of state machines and

traditional structured design. V_re also

collected data for the momtoring team

at various points throughout the project, although we n-ied to do this with

as little disturbance as possible to the

project team.

_r_ly_ andI_a#_ rm_. The final

steps in the QIP involve analyzing and

packaging the process results. We

found significant differences in effort

distribution during development

between the Cleanroom project and

the baseline. Approximately six percent

of the total project effort shiftedfi_om

coding to design activities in the

Cleanroom effort..Mso, the baseline

development teams traditionally spent

approximately 85 percent of their coding effort writing code, 15 percent

reading it. The Cleanroom team spent

about 50 percent in each activity.

The primary goal of the first case

study had been to improve reliability

without increasing cost. Analysis

showed a reduction in change rate of

nearly 50 percent and a reduction in

error rate of greater than a third.

Although the ex-pectation ,_as for productiviry equivalent to the baseline, the

Cleanroom effort also improved in that

area by approximately 50 percent. _Are

also saw a decrease in rework, as

defined by the amount of time spent

correcting errors. Additional analysis of

code reading revealed that three

fourths of all errors uncovered were

found bv only one reader. This

prompted a renewed emphasis on mulJ U LY 1994

10022514L 3-37
QUALITY IMPROVEMENT PAUDIGM: FOUNDATION FOR IMPROVEMENT •

The Quality Improvement Paradigm is an effecfiveframework for conducting experimentsand studies

like those described in the

main text. It is an experimental but evolutionary

concept for learning and

improvement.

The QIP has six steps:

I. Characterizetheprojectand itsenvironment2.Set quantifiable g_ls_[,

forsuccessfulprojectperforrazace and improvement.

3. Choose the appropriate proc_ model_,supportingmethods,_ i_olsfur

the prote=- " °'

_ .,myr__e produc_,eof

l_'and_li_tethepred_ _ia _ the

.dam toprovidereal-time

feed_ck forcon_ctive

action.

The QIP uses two tools:

the Goal/Question/Metric

paradigm and the

Experience FactoD"

Organization.

TheC,O, f

.p_a-adigmisa mechanism

ttsedinthep_ phS_,i_

of the Qu_ity Improvement

Paradigm for defining ar/d

evaluating a set of operationllgodsusingme_surement.2 It provides a systematic approach for tailoring

and integraa_ggoalswith

models of thesohware

ofin -

est,according m thespeci6c

needs oftheI_eCt and

or_on.

You de6ne gods inan

opemioml,_c=blew_ by

_ them into a set of

quesficats that exa-act aplm >-

ix'late informationfrom the

models.The questions,in

to detect defects), the purpose of the study (like assessment or prediction), the

point of view from which

the study is performed (like

customer's or manager's),

and the context in which the

sty- isperfo_d 0ike people-orientedor problem-oriented factors).

For example, two goals

associated with theapplication of the Cleanmom

method in the SEL were

analysis of the Cleanmom

process to characteaize

resource a/location from the

project mamgces point d

view, and analysis of the

Cleanroom producttocharacteri_ dofects from the

customer's point of view.

men. The Ezp_ence

Factory _tie,. is an

organizational smxcaa_ that

supports theactivities sl_citeaming and packaging of

reusableexperiences.It

packages experiences by

building informal, schematired, formal, and automated

models and measures of

software processes, products,

and othitr forms of kngwledge, and distributes them

dm-oughcon._-qmdon,documenm_on, and anmn_ted

suppo_

Wh_e pro_to_-

tion follows an evolutionary

process model that reuses

pae.t_ged experiences, the..

Experience Factm'y provides

th_ set of procemes needecl.. =

l/i- mg, p :hsi , and

sto_[ g,the project organizltion'sexperlenceforfeud.

T_e ;Experience Factory

_on represents the

integr_a of these two

ftmction._ :

tiplereadersd-a'oughoutthe SEL env/-

ronn_ent.

W'e alsoe.x_minedthe earlierconcernsexpressed by.managers and the

project team. The resuhs showed

increasedeffor_in earlyrequirementsanalysisand design activitiesand a

dearer setof in-linecomments. This

led to a better understanding of the

whole .system and enabled the project

team to understand and accommodate

changes with greater ease than was

typical for that environment.

We reviewed the application of

classical Cleanroom and noted successes and difficulties• The structure of

independent teams and the emphasis

on peer review during development

was ea_" to apply. However, the development team did have difticulty using

the associated formal methods. Also,

unlike the scheme in the classical

Clea_oom method, the test team followed an approach that combined statistical testing with traditional functional tes_g.

Finally-,the psy.vahologicll effectsof

independent testing appeared to be

negligible. All team members indicated

high job satisfaction as well as a willingness to apply the method in future

projects.

_Ve packaged these early results in

various reports and presentations,

including some at the SEL's 1990

Software En_neering Workshop. As a

reference for future SEL Cleanroom

projects, we also be_n efforts to produce a document describing the SEL

Cleam-oom process model, induding

details on specific activities: (The

completed document is now avaihble

to current Cleam-oom projects.)

5mada_ sm_.The firstcasestudy

showed us thatwe needed betternaining in the use of formal methods and

more guidance in applyingthe testing

approach. We also realized that ex'periences from the initial project team had

to be disseminatedand used.

Again, we followed the Quality

Improvement Paradigm. We selected

two projects: one similar to the initial

Cleanroom project, Project 2A, and

one more representative of the q_pical

FDD contractor-support environment,

_EEE SOFTWARE 63

100P.2514L 3-38
;!if!i: i:ii ii :: ""

_0

10

0

[A]

4O

J

-Figure 3. _,_4easm-ement comparisons for two case szudies investigating Cleanroom.

The first case smuty in'aolved one pro_ect, Project I. The second case study invofi, ed

t-wo proje_, Projects 2A and 2B. (,4) Percentage of total d_'elopmenr effort for

various de'_'elopment actA,ities, and (B) productivity in lines of code per day, change

rate in changes per thousand lines of code, and reliability in errors per thousand

lines of code.

Project 2B.

Project 2A involved a different subsystem of mother amtude ground-support _-stem. This subsystem focused

on the processing of telemetry data,

comprising 22,000 LOC. The project

was staffed with four developers and

two testers. Project 2B involved an

entire mission a_tude ground-support

system, consisting of approximate]y

160,000 LOC. At its peak, it was

staffed with 14 developers and four

testers.

Setting goals and choosing processes. The

second case study had two goals. One

was to yetiS- measures from the first

study by applying the Cleanroom

method to Project 2A, a project of

similar size and scope. The second was

to verify the applicabili_- of

Cleaaroom on Project 2B, a substantially larger project but one more representative'of the _'pical environment.

We also wanted to further tailor the

process model to the environment by

using results from the first ease study

and applying more formal techniques.

Paek2ges from the SEL Experience

Factory. (described in the box on p. 63)

were available to support project

development. These included an

evolved training program, a more

knowledgeable experimenter team to

morutor the projects, and several inprocess interactive sessions with the

project teams. Mthough we had beg'an

producing a handbook detailing the

SEL Cleam'oom process model, it was

not ready in time to _ve to the teams

at the start of these projects.

The project leader for the initial

Cleanroom project participated as a

member of the experimenter team,

served as the process modeler for the

handbook, and acted as a consultant to

the current projects.

We modified the process according

to the experiences of the Cleanroom

team in the firs_ study. Project l's team

had had difficult).- using state machines

in system desiwn, so we changed the

emphasis to Mills' box-structure algorithm.= We also added a more ex'rensive .:

10022514L 3-39

JULY 1994
IT,_m _ze

Controlledexperiments

Readingvs.testing aeanroom

32 pardd1_m_

Chonroomcasestudies

Projed1 ProjeO2A Project2B

Fourteen-person

I

Project size

and application

l_s-ults

Small (145-365

sample Fortran

programs

" techni

inOl'¢ _-

t_ than tes_g

mchaiques for fault

detection

Three-person develop- iThree-person Four-person "

raemmarm(lOeaI_i" i devebpamattemn; development ream;

mere trams;fr_ amax_lj two-person test team _o-person test teara

tewns_coaanminde1,500LOC, Fortran,

electronic messa_

.system for graduate

laboratory course

Cleam-oom teams

me ft_ C_im=

make higher peremtageof sc_m_

(Mfivefies

development team;

four-person test

temm

I

40,000LOC, Foman, 22,000LOC, Fortran, 160,000LOC,

fLi._ht-dynamics flight-dymamics Fortran,flightgr_tmd2support ground- support dy.mamiesfound-

_ern _xtem support s?_tem

Project spem_ highez Project _a_ames Project reliability

percemageofeffdct txendinbetterreIia-th_n_l_@e_.; in design, rues fewea bi_tywhile mainmincompum-resources, ing baseline Droduc- Drodue_tv falls

and achieves l)_a:a¢ fivity belowbasd/ne

productivity and tallability than environment

training program focusing on Cleanroom techniques, experiences from the

: initial Cleanroom team, and the relationship between the Cleam-oom studies and the SEL's general goals. The

instruction team included representatives from the 5EL, members of-,.he

initial team, and Mills. Mills gave talks

on various aspects of the methodology,

as well as mom-ational remarks on the

potential benefits of the Cleanroom

method in the software community.

Project 2A ran from March 1990

through January 1992. Project 2B

ran from February 1990 through

December 1992. A_in, we examined

reliability., producdviq-, and process

characteristics, comparing them to

Project I results and the SEL baseline.

Aml_ and _a#_ rea/ts. As Figure

3 shov,% there were significant differences between the two projects. Error

and change rates for Project 2A continued to be favorable. Productivi_." rate,

however, renamed to the SEL baseline

value. Error and change rates for

Project 2B increased t_omProject 1_-alues. although they remained lower than

SEL baseline numbers. Productivi_',

however, dropped below the baseline.

When we examined the effort distribution among the baseline and

Projects I, 2A, and 2B. we fi)und a

continuing upward trend in the percentage of desig'n effort, and a corresponding decrease in coding effort.

Additional analysis indicated that

although the overall error rates were

below the baseline, the percentage of

system components found to contain

errors during testing was still representative of baseline projects de,'eloped in

this environment. This suggests that

the breadth of error distribution did

not change with the Cleanroom

method.

In addition to evaluating objectiv e

data for these two projects, we gathered subjective input through written

and verbal feedback from project participants. In general, input from

Project 2A team members, the smaller

of the two projects, _xs very favorable,

while Project 2B members, the larger

contractor team, had sitmificant reservations about the method's application.

Interestingly, though, specific shortcomings were remarkably similar for

both teams. Four areas were generally

cited in the comments. Participants

were dissatisfied _ith the use of design

abstractions and box structures, did not

fully accept the rationale for having no

developer compilation, had problems

coordinating information between

developers and testers, and cited the

need for a reference to the SEL Cleanroom process model.

Again, we packaged these results

into various reports and presentations,

which formed the basis for additional

process tailoring.

Thirdcasestudy.We have recently

begun a third case study to examine

difficulties in scaling up the Cleanxoom

method in the typical contractorsupport environment and to verify previous trends and anaR.-zeadditional tailoring of the SEL process model. VCe

expect the study to complete in

September.

In keeping with this goal, we again

selected a project representative of the

FDD cona'actor-support environment,

but one that was estimated at 110,000

LOC, somewhat smaller than Project

2B. The project involves development

of another entire mission attitude

g'round-support system. Several team

members have prior experience _th

the Cleanroom method _rough previous SEL studies.

Experience Factory packages available to this project include training in

the Cleanroom method, an experienced

experimenter team, and the SEL

Cleam'ooml:b'oce_"3,Iodd (the completed

-handbook). In addition to modi_'ing the

process model according to the results

from the first t_o case studies, we are

3-40

65

_L ORIG_t,_L P,_.... iS

POOR UIUt' 
providing regularb, scheduled sessionsin

which the ream members and experimenters can interact. These sessions

_ve team members the opporturfity to

communicate problems the), are having

in apph-ing the method, ask for darificadon, and get feedback on their acth'ities.

This activi_" is aimed at dosing a communication _p that the contractor team

felt crested in Project 2B.

Ihe concepts assodated with the QIP and its use of measurement have

p'en us an evolutionary framework for

understanding, assessing, and packaging

the SEL's e_x'periences.

Table 1 shows how the evolution of

our Cleanroom stud), pro_essed as we

used measurements from each experiment and case stud)" to define the next

exT_enment or study. The SEL Cleanroom process model has evolved on the

basis of results packaged through earlier

evaluations. Some aspects of the target

methodolo_ continue to evolve: Experimentation _a_th formal methods has

transitioned from functional decomposition and state machines to box-su'ucmre

desi_ and ag'ain to box-su'ucmre application as a way to abstract requirements.

Testing has shifted from a combined

statistical/functional approach, to a

purely statistical approach based on

operational scenarios. Our current case

study is examining the effect of allo_ng

developer compilation.

Mong the way, we have eliminated

some aspects of the candidate process;

we have not ex,nmined reliability models,

for example, since the environment does

not currendv have sufficient dam to seed

them. We l_ave also emphasized some

aspects. For example, we are conducting

studies that focus on the effect of peer

reviews and independent test teams for

non-Cleanroom projects. We are also

studying how to improve reading by

developing reading techniques through

off-line _x'perimentation.

The SEL baseline used for comparison is undergoing continual evolution.

Promising techniques are filtered into

the development organization as general

process improvements, and corresponding measures of the modified process

(effort distribution, rel.iability, cost) indicate the effect on the baseline.

The SEL Cleanroom process

model has evoh-ed to a point where it

appears applicable to smaller projects

(fewer than 50,000 LOC), but additional understanding and tailoring is

still required for larger scale efforts.

The model will continue to evolve as

we gain more data from development

projects. Measurement will provide

baselines for comparison, identify

areas of concern and improvement,

and provide insight into the effects of

ACKNOWLEDGMENTS

This work has been supported by

NASMGSFC contract NSG-5123. We

thank all the members of the SEL team

who have been part of d_e(3eanroom experimenter teams, the Cleanroom training

teams, and the various Cleanroom project

teams. We especially thank Frank

MeGarry, Rose Pajers "kLSally Godfrey,

Ara Kouchadjian, Sharon _.Valigora,

Harlan Mills, Michael D_'er, and Terry.

Baker for their efforts.

REFERENCES

1. V. Bmili and R. Selby. -Comparing the

F.ff_veness of Software Tesung Swat eg_es,"

Tram. Safra-.am En_., Dec. 1987, pp.

1278-1296.

2. R. I./mg_, H. ._1_ and B. XVr_ S_d

w_ey., R=u_ng,.Mass.."079.

3.H.Milk M. 1_._, andR. Linger,

"Ctemn0oraSofr_-areL-.umeenng,"/EF_.E

S_,'-w_, Sept. 1987. pp. lO-24.

4. R. S¢lby.,,Jr. V, Basili. _d T. Bake*,

"Cleanroom Soft_lwe Dex'etopmen_ .-_

_pirica] E_-alua-lon." I_-.E_ Trcm.

E_g., Sel_ 1987. pp. 102--1037.

L L Landis et aL, -Recommended .-_proach to

Sof't_re Devdopmem: R._':_on 3," Tech.

Report 5EL-81-305. So.',-_-are Ea_neering

Laboratory, Greenbelt..',ld. 1992.

6. S. Green, S_._e F_Lcm,',','mg La_

(SF_d.) _n l_,'oc_ M'dd, Tech. Report

SE.L-9I-O(N., Software En_neering

Lal_)ratory.. Greenbelt, Md.. 1991.

7. H. Mills, "Step_'ise Refin,-mem and

Verification in Box-SL-a_art:d Sxmems,"

IF_2E 5ofm_rt.June I_. pp. 23-36.

! process modifications. In this way,

i we can set quantitative expectations

! and evaluate the degree to which

goals have been achieved.

By adhering to the Qualit-y Improvement Paradig'm, we can refine

the process model from study to

study, assessing strengxhs and weaknesses, experiences, and goals.

However, our investi_tion into the

Cleanroom method illustrates that

the evolutionaq" infusion of technologT is not trivial and that process

improvement depends on a structured approach of understanding,

assessment, and packa_ng. •

Victor Bmili is a professor

of computer science at the

In_rute for A&lnced

Computer Studies at _e

Universiq" of Maryland at

College Park. One of the

founders and principals of

r.he Sofrw'are En_neering

Laboratory his interests

include qtmnota6ve

approaches for soft-'are

manlg_ent, en_neering, and qu_li_." assurance.

He is on the editorial board of_t_Sys_ems

B_ll receivedaBS in mathematicsfrom

FordharnCollege.an Y.ISin mathematicsfrom

SyracuseUnivershy.andaPhD in computersciencefrom the Univemi'_."of Texas at Austin.He is

an IEEE fellowanda member of_e IE.E.E

ComputerSocie_.".

,_o_ Gr_m is a senior software en_neer in

NASA Goddard's Right DTnamics DixSsion, where

he is involved in the proiec_ mana_ement of

_ro_d-_upport _'s_erns and m leading sofv_'_reengineering studies ar the Sofr_'_re Engineenng

Laborato_'.

Green received a BS in computer science from

Lo._la College.

Address quest'ions about this article to Basili at

CS Dept., University ,,f Ma_-Iand. C,ollc_ Park.

MD 20742: basili@e*.umd.cdu: or t(, Green at

NAS.-VGFSC. Code 5:,2.1, Gn._.nl_lt. MD 2077I;

se.._recn@gsfcmail.nasa._)v.

10022514L

oF R)OR 

3-41

JULY "t994

SECTION 4--ADA TECHNOLOGY

The technical paper included in this section was originally prepared as indicated below.

• "Genericity Versus Inheritance Reconsidered: Self-Reference Using Generics,"

E. Seidewitz, Proceedings of the Conference on Object-Oriented Programming

Systems, Languages, and Applications, October 1994

10022514L 4-1

GENERICITY VERSUS INHERITANCE RECONSIDERED:

SELF-REFERENCE USING GENERICS

OOPSLA '94

Ed Seidewitz

NASA Goddard Space Flight Center

Code 552.3

Greenbelt MD 20771

(301)286-7631

eseidewitz@ gsfcmail.gsfc.nasa.gov

Abstract

As shown by the work of Bertrand Meyer, it is possible to simulate genericity using inheritance, but not

vice-versa. This is because genericity is a parameterization mechanism with no way to deal with the

polymorphic typing introduced using inheritance.

Nevertheless, if we focus on the use of inheritance

as an implementation technique, its key feature is

the dynamic binding of self-referential operation

calls. This turns out to be basically a parameterization mechanism that can in fact be simulated using

generics and static binding. And for some applications this approach may actually be of more than

academic interest.

Introduction

In his classic paper on "Genericity versus

Inheritance", Bertrand Meyer concludes that inheritance cannot be simulated using genericity because

genericity provides no mechanism for achieving the

polymorphism of inheritance [Meyer 86]. This is, of

course, true, since genericity is a parameterization

mechanism, not a typing mechanism. However, as

an implementation technique, rather than as a typing

mechanism, the polymorphism of inheritance is primarily used to achieve the dynamic binding of selfreferential calls to object operations (e.g., messages

to sel_' in Smalltalk).

This is not a minor point. Wegner and Zdonik

state that "'In a world without self-reference, inheritance reduces to invocation and inheritance hierarchies are simply tree-structured resource sharing hierarchies. However, recursive definitions are just as

fundamental for objects as for functions and procedures." [Wegner 88]. In effect, inheritance is not inheritance without self-reference. In this paper I will

show that this crucial self-reference property of inheritance can, in fact, be simulated using genericity.

Cook and Palsberg define a denotational semantics of self-referential inheritance equivalent to the

traditional operational semantics using dynamic

binding [Cook 89]. They use a "wrapper" function

to parameterize the super- and self-references of a

class. These parameters are then "statically bound"

using a fixed-point operation. Thus, self-reference

becomes basically a parameterization problem,

which can be handled quite well by generics.

The following three sections show in detail how

this is done. The first section reviews the general

issues of self-reference in the traditional inheritance

mechanism. The next section shows how generics

can be used to parameterize this self-reference.

Finally, the third section extends this approach to

also parameterize superclass reference.

The examples in this paper are written in Ada

9X, the proposed revision to the Ada language

[Ada9X 94a] (likely to be approved in 1994). Ada

9X has powerful features for both genericity and object-oriented inheritance and is therefore an excellent real-world vehicle for the discussion here. I will

introduce and describe the Ada 9X mechanisms for

inheritance and genericity as necessary in the following. This should be sufficient for a self-contained reading of this paper, but it is by no means a

complete overview of Ada 9X, or even its object-oriented features. For fuller discussions of Ada 9X, I

refer the reader to the references [Ada9X 94a],

[Ada9X 94b] and [Taft 93].

10022514L 4-3
Inheritance

Hauk uses an instructive example to discuss the issues involved in inheritance and self-reference

[Hauck 93]. This example is based on a class of objects that service hardware ports. One can output

characters and lines to such ports, with the output of

lines defined in terms of the output of characters.

We define this class in Aria 9X using the following

package specification:

package Port £a

tYl_e object £m tagged pri_to;

p_o©odure Put(O= in out Object; C= in ehaz_aotor);

procedure Put L£no(O: in out Object; L= in 8trlng);

privato

type O]a.4eot hi _gged record _ end _ooz_;

end Port;

In Add 9X, encapsulation is achieved by defining abstract data types called private types. The type

Port .Object is defined as a private type in the visible part of the package specification above, with its

full definition given in the private part. Public

primitive operations on this private type are also declared in the visible part of the package specification. The implementations of these operations are

given in the corresponding package body, which we

will get to in a moment.

The use of the keyword tagged in the definition

of Port .Object signals the availability of the object-oriented features of type extension and dispatching for this type. For example, suppose we wish to

define a subclass of ports that buffer their output.

We can define this as an extension of Port.Object:

with Port;

I_¢.ge l_tffered Port is

tYPe Ob::leot(Siso= Pos£tivo) L=

Port.Oh:leer with prLv=to;

prooe_tu=_, Plush(O= in out Object);

private

t21Pe Ol)_ect(Size= PosLtive) is now Port.Object with

_ooz_t

Last: Natural == O;

l_tffer= 8tring(1..S£ze);

end re¢oz_;

end ]klfferod Port,

The type Buffered_Port.Object iS a derived

typeofPort.ObJect extendedwith thecomponents

required to implement a buffer. The discriminant

Size is used to set the maximum number of characters stored in the buffer. A derived type inherits the

primitive operations of its parent type. In this case,

Buffered_Port. Object inherits the operations Put

and Put_Line/Tom Port .Object. An addltiODal operation, Flush, is defined solely on the type

Buffered_Port. ObJect.

Derivedtypesaredistincttypesfrom thek parenttypes. Thus, given the declarations:

P= Poz'T..Ob_eet;

B: Buffered_Port.Object;

the following assignment is illegal:

P == B; -- Type ",,_t:_hl

even though Buffered_Port .Object is derived

from Port. ObJect. The following explicit conversion is legal:

-- £= obffeQC of type Buffered_Port. Objeot oen abe

-- co.vetted to type Poz_.Objec_t

P = m Port. Object (B)

but the converted value is of type Port. ObJect, and

the extension components in B are lost.

Add 9X separates polymorphism from the basic

tagged type construct through the concept of classwide types. For example, there is a class-wide type

denoted Port.Object'Class rooted in the tagged

type Port.Object. A class-wide type includes all

values of all types in the derivation class of its root

tagged type. The derivation class of a tagged type

includes the type and all descendant types derived

from it.

Due to the availability of type extension, the size

of a value of a class-wide type cannot generally be

determined at compile time. Therefore, polymorphic

variables in Ada 9X generally contain pointers to

class-wide types. Pointer types in Aria are known as

access types. Thus, given the following declarations:

t_ClPe Port_Polnter is iooess Port. ObJect'Cltssi

type Bufferc_l_Port_Polnter is aooeas

Bur feted_Port. O]_jec t "Clas s ;

PP= PortPointer;

BP: Buffered_Port_Pointer

10022514L 4-4
the following assignment is legal:

-- Po/n_er to Por_. Object "Claas cJm poi,_t to

-- Buffared _ore'C1aaa object.

PP := BPI

because the derivation class of Buffered_Port. OhSect is contained in the derivation class of Port .obJect.

In addition to allowing polymorphic variables,

class-wide types also provide the mechanism for

polymorphic dynamic binding of operations. Each

value of a tagged type has a tag that identifies the

dynamic type of that value. When a primitive operation of a tagged type is passed a value of the corresponding class-wide type (which may actually be a

value of any type derived from the root tagged type),

the operation dispatches to the implementation identiffed by the tag of the value. For example, given the

above declarations:

-- No¢_ _)_at ".all" derefere_cea poin_a_a

Port.Put(PP.sll,C); -- _ Co PUt Im_lemantation

-- in body of Poz_

PP :,_ BP;

port. Put (PP. all,C) I -- Bound to .l_t i_leuwentation

-- in body of Buffe_e4 Po_t

The second c_]l to Port. Put is dynan_cally bound

t0 Buffered_Port.Put, because the tag of the object pointed to by PP after the assignment indicates

Buffered_Port. ObJ act.

Now let'sturn to the body of package Port.

This body contains the implementations of the two

operations on type Port. ObJ ect:

package body Port i,

procedure Put(O: in out Object; C: in ChaEactez) is --

end Put t

procedure Put_Line(Or in mat Object; L: In String) is

begin

for I in L'Rm_ga loop

Put (ObJ_t'Cla,e (O) ,L(I)) ; -- Redi_atching ¢a11

and loop;

end Put_Line/

end Port/

Note theuse of theconversionObJact ' Class (O) in

the call above to the procedure Put. This conversion

causes the call to Put to be dynamically bound, depending on the dynamic tag of the argument o. This

is known as a redispatching call in Aria 9X, and it

has the same effect as the use of seat in Smalltalk

[Goldberg 83] or this in C++ (for a virtual function) [Stroustrup 91].

The use of redispatching in the implementation

of.Put_Line makes the implementation of type

Port .object self-referential. This self-reference is

very important for the implementation of the operationsof the derived type Buffered_Port .Object.

The body of package Buffered_Port must, Of

course, include the implementation of the new operation Plu,,h. In addition, the implementation of procedure Put inherited from Port .Object must be

overridden with a new implementation that handles

the buffering required for a s u f -

feted_Port. ObJect:

package body Buffered Port is

procedure Put(O_ in out Object; C: in (_haracter) is

begin

O.La_t := O.Lalt + I;

O.Buffmr(O.Lamt) := C;

if O.Last • O.Size then

PluJh (ObJect'ClaJJ (0}) ; -- Redlap_tching ¢alJ

end if;

Ind Put;

procedure Flush(O: in out Object) is

_gin

for I in I..O.Laat loop

Po_. Put (Port. Object (O), O. Buffer (I) ) ;

-- Statically-bound call

end loop;

O.La_t := 0;

end Plush!

and Bur fared_Port;

Note the conversion Port .Object (O) in the sLq_-

caily bound call to the parent operation Port .put in

the implementation of Flush.

Since the procedure Put_L±ne is not overridden

in the body of package Buffered_Port, its implementation is inherited without change. This is shown

diagrammatically in Figure 1, where the shading indicates that there is no implementation for _t r.ine

paeka0e Port

I .......

Figure I

package Bulfered_Port

-I

I00_514L 4-5
physically included in package Buffered_Port.

Figure 1 also shows that the actual implementation

for Put_Line in package Port makes a call on

Port. Put. However, when this implementation is

inherited in package _auffe red_Port, the redispatching call to Port. Put, when passed a value of type

Buffered__Port .Object, will now be dynamically

bound to the overriding implementation of Busferea__Port .Put. Thus, the characters in a line are

all properly bttffered, even though the implementation of procedure Put_Line has not changed.

Genericity

Consider again the hardware port example from the

last section. We wish to implement the same

Port .object private type, with the same visible

operations, but without the use of rexfispatcl_ng.

Nevertheless, we wish to retain the ability to redirect

the binding of self-referential calls in operations inherited by a descendant of Port .Object. TO do this

we make this binding explicit using a generic package nested within the specification of package Port:

package Poz_ IB

t_o Ob_Joct 4s taggod p¢_top

pcocOdklrO Put(Ox in out Self; C: in Character);

procedux_ Put__I_na(O: in out Solf; L: in String);

gonoric

type 8elf(<>) 4j nm_ Objoct with pri_to;

package Opont£ozm 4s

pr_oduro Do_Put(Oz in out Sale; C: in Character);

proc_lura Do_Put_/_no(O: In out BelfTLt in String);

end OlPont_ona;

private

t_a Ob_act 4m tagged r_ord _ on¢1 rsc_oz_t;

e_d Poz_7

While the type Port. Object retains its operations Put and Put_Line, the actual implementation

of these operations are moved to the inner generic

package Port. operat ±one. This generic package is

parameterized by the type self, which must be a descendant of Port.Object (or Port.Object itself).

As a descendantof Port.Object, any actualtype

bound to the parameter Self will have Put and

Put_Line operations. This binding of the parameter

self will be used in the implementation 0fthe operations Do_Put and Do_Put_Line to replace any selfreferential redispatching calls.

The generic package operations is nested inside Port so that its body has visibility to the full

definition of the private type Port. obJect. This allows the subprogram Do_Put to be implemented the

same way as Port. Put would have been in the last

section (if we had actually shown it!). The implementation of Do_Put_Line is alSO similar to the implementation of Port. Put_Line in the last section,

but with a crucial difference:

pachga body Port iJ

package body OperatLons is

px_eedux_ DoPut

(0: in out SeZf; C: in Charsctar) iJ

_ snd Put;

px_cedure Do_Put_Line(O=in out Belf;L:£n StrLng) is

bog_

for Z 'In L'P.a/IRo loop

Put(O,L(Z) ) ; -- 8tattoO/y-bound o_lZ

end loop;

ODd Put_L:Lna;

and Oporat£onl;

package Sel¢ Opox_tions £s

now OperattonJ (Port.Object) t

procedure Put(O: 4n out 8elf; C_ in ¢l_ractar)

renaaes Be1 f_OperationJ. Do_]Put;

procedure Put L_no(O_ in out 8o1£; L_ in String)

rameses SaZ f_Oparat £ona. Do_Pu t_l_L ne 7

an4 Port;

In place of the redispatching call in the implementation of Put_Line there is now a statically-bound call

in procedure Do_Put_r.ine to the operation Put on

type self. Rather than using redispatching, self-reference is achieved by instantiating the generic package Operations inthebody ofpackage Port. This

instantiation effectivelyprovidesthefixed-pointoperationofCook and Palsbcrg.

The package Self_Operatlons is an instantiationof thegenericpackage operations with

Port. Object USed forthe parameterSelf. The procedures Put and Put_Line ale then simply renamings of the real implementationsfrom self operatlons (which have the correctargument type

profiles, since Self iS Port.Object fOE Self_Operations!). The implementation of Self_Operations .Do_Put_Line contains a call to the operation

10022514L 4-6
packa_ Poe

/

I..l_ retlal7 _,._ :::::::::::::::::::::::::::::::::::::::::::::::::::::: I

...................... ::::::::::::::::::::::::::::::::

i _Wa,.tiares

}'

_. operations _-:.:_ ........

Fil_ure2

Put for the generic type parameter sezf. Since the

type parameter sel_ is bound to Port.Object for

this instantiation, its Put operation is simply

tort. Put, which is a renaming of sel__Operations .Do_Put. Thus Port:. Put_Line self-referentially calls Self Operatlons.Do Put,as shown in

Figure 2.

Note, however, that Port. Put_T.ine now makes

a statically-bound call to Port. Put. Thus this call

will not be automatically redirected to Butfeted Port.Put in the inherited operation ,auffered_Port.Put_Line. Instead, we must instantiate

the generic package Port. operations differently

for the implementation of the Buffered_Port operations, so as to achieve the correct bindings. To

see how this is done, let's turn next to the implementation ofBufferedPort. ObJect usingtheour new

approach.

As we did with package Port, we include a

nested generic package within package rauffared Port:

with Port;

package Suffered_Port is

t_pa Object(Size: Positive) is

new Port.Object with private;

proc_ura Plush(O: in out Self);

generic

type Self (,<>) is new Object with private/

pael_e B_ered_Poa

::::::::::::::::::::: :::::::::::::::::::::: ¢I_B/I'_

•|i::i_::!i!::_i:::_| ..................

:::::::::::::::::::::::::::::::::::::::::::::: /_I,*/_?/TI_S

packa,De Self_Operations J

_oI-_S,,_LO_,,._,_ I l i

packet O_eretlon_ is

procedure Do PUt(O: _n o_t Self; C: in Characte¢);

procedure Do PUt_LiDe(O_ ll_ out SelflL_ in String);

procedure Do_Plush(O: in out Self)l

end Operation_ l

pEivete

t_ obJe=t(Size: Positive) £s new Port.Object with

r_ord

L_stz _tuE_l :_ 0;

E_Iffer_ Strlng(l..Siza)_

en_ E_ord;

e_ Bur fare_L_Port;

Note that inner" generic package Buffered_Pore. Operations contains implementations

for the inherited operations Put and _,ut_r.ine as

well as the new buffered port operation Flush:

package bod.y _uffe=_t_Po=t is

pa.ckage hoc_ Operations is

package Su_er_Operatlon_ is

new port.operation(Self);

p=oce_ure Do_Put(O_ in out SelflC:.in Character) is

O.Last :. O.Lmst + l;

O._R1ffer(O.L_kst) |. C;

if O.La_t = O.S£Ze then

Flush(O}; -- _eti¢ally-_uad ¢_Z1

_nd if;

a_ Put;

proce_ureDo_put_Line(Oz in out Self; L: in String)

renames Supe=Ope=_ticne.Do_Put_L_ne;

_ 0022514L 4-7
procedure Do_Flush(O: in out Object) is

begin

for Z in I..O.Last loop

Super Operationa. Do_Put (O, O.Buf£er (1)) ;

-- Seaclea11y-bound call

end Xoop;

O.Laat t= O;

_d Plush;

one10porationJ /

package llelf__orationl ia

new But£fered Po_t. 0peEat£onJ (Buffered Port.Object) ;

pz'oce_ro Put(at in out Object; C: in Character)

z_nLs_m _1£ Oi_zations. Do Put;

p_oco_bat_ Put_Line(Or in out Ob_Ja¢t; L: £n String)

leenAmnm S@1 f _Opera tlonm. Do_Put _Line;

p_codu_ Flush(at _n out Object)

x_na_s _l£_Operation_. Do_FluJh;

end _affared Patti

Note the nestedinstanUafi0nof Port. Operations

within Buffered_Port. Operations, passing along

the co.act binding for self.

shown in Figure 2, the in_anfiation Buffered__Port. Self_Operations appropriately redirects the self-referenlial calls to Put and FXush to the

implementations as required. The nested instantiation of Port.olperations within Buffered_Port.Self_operations assures that even

references to Put in Buffered_Port. Super_Operations .Do_Put_Line now call B u ffered_Port. Sel f_operat ions. Do_Put.

Mixins

The wrapper functions of Cook and Palsberg parameterize both the super- and self-references of a class

[Cook 89]. In the last section we used generics to

parameterize the self-references. An extension of

this approach can be used to parameterize superclass

references as well.

To do this, we first turn the package defining the

subclass type into a generic package with the superclass type as a generic parameter. Such a generic

package provides an independent increment of functionality that can be added on to any appropriate superclass type. We will call such a package a mixin,

since its functionality can be "mixed into" the superclass. The term "mixin" comes originally from the

LISP-based Flavors system [Moon 86] and is usually

used in conjunction with multiple inheritance. The

mixins we will define here are closer in spirit to the

generalized concept proposed by Bracha and Cook

[Bracha 90]. (See also the Ada 9X Rationale

[Ada9X 94b] for a discussion of using generics as

mixins in Ada 9X; I have also previously described

how mixins can even be created in non-object-oriented Ada 83 [Seidewitz 92].)

For example, consider the buffered pert class.

We can turn this class into a mixin by replacing its

superclass dependency on the port class with a

generic parameter:

generic

type Elmnt Is private;

type Super(<>) is a_stract tagged private;

package Buffor_.Mixin is

type Object(Sizes Poeitive) ie a_tract now _hq_r

withprivata;

genorlc

type self(<>) le now Object with private;

with procedure SuavE_Put

(O: in out BolE; R: in EloJent);

with procedure Solf_11ush(Ot in out SoIZ);

pe,¢kago Operatlono is

p_ocedure Do_hi(Or in out blf; Et in Element);

p_oouduro Do_Plu_h(O: in out 8all);

_d Oporatlolul;

private

type Iloment_AErayie

azTay (Positlvo z_ngo <_) o£ WAement;

type ObJect(Slzet Poeitive) is

abstract now Super with

rocox_

Laat: Natuzal t= O;

Buffer: Element_Arra¥(l..Size) ;

end record;

end Buffered_Port;

The type parameter Super provides the required parameterization of the superclass type. The type

'augfer_mx±n.obJect is then derived from this

generic parameter. Since we needed to make this a

generic package anyway, the buffer mixin is further

generalized above by using the generic type parameter ,_1ement (which does not need to be tagged) in

place of Caaraeter.

Note that the type parameter super is declared

to be abstract. This means that the actual type used

for this parameter may be an abstract type (though it

may also be non-abstract). It is illegal to create ob10022514L 4-8
jects of an abstract type, though there may be objects

of non-abstract descendants of the abstract type.

Further, an abstract type may have abstract operations that have no implementations (these are

equivalent to pure virtual functions in C++

[Stroustrup 91] or deferred routines in Eiffel

[Meyer 88]). Non-abstract descendants of an abstract

type must override all abstract operations with nonabstract implementations. The type ,eat f e r Mixin.Object is alSO declared to be abstract, since it

may inherit abstract operations from super.

¢ The type parameter super is also not constrained to be a descendant of any known type.

Therefore, within the body of the generic package,

there are no primitive operations guaranteed to be

available for super (except for some basic operations like equality, but that's a technicality). Since

there are no known operations to be inherited from

super, and no other operations are defined for it, the

type Buffer_Mixln. ObJeet also has no known

primitive operations. Instead, this type only provides

a basis for defining the implementations of the operations given in the generic package Suffer_rexin. Ope rat ions.

As before, the generic package Operations is

parameterized by the derived type parameter serf.

NOW, however, there are no known primitive operations to be inherited from Buffer_Mixin.ObJeet.

Instead, the only operations on self are those that

are explicitly given as generic parameters, in this

case super_Put and Self_Flush. As the names indicate, the super_Put parameter is intended to provide the superclass Put operation, while the

Self_Flush parameter provides the self-referential

FZueh operation. Thus, this generic clause defines

the complete inheritance interface for the buffer

mixin. (As will become clearer in a moment, the

Super_Put operation is defined on the type Self

rather than Super to ensure the correct binding of

"any .self-referential calls it may make.)

Calls to the operations given by super_Put and

Self_Flush are now the only external calls that can

be made on type Self in the implementations of

Do_Put and Do_Flush"

package b_ly Buffer_Mixin is

package _d_ Operations is

procedure Do_Put(O: in out Self; E: in Rlament) is

O.Last :. O.Last * II

O.Buffsr(O.Last) == E;

if O.Last . O. Size than

Self Plush(O)l -- e_a_ically-bound call

end if ;

end Put;

p_cedure Do_Plush(O: in out Self) is

for I in I..O.Last loop

Super_Put (O,O.Buffar{I)) ;

-- etatioally-bo_u_ call

end looP1

O.Last :_ O;

end Plush/

ena ODarationm /

end Bur far_Mixin;

Note that this buffer mixin does not define a

Do_Put_Line operation. This is because a mixin

should represent a discrete increment of functionality, and the ability to put a line is not really part of

the buffering functionality as defined here.

As defined in the previous sections, the port

class does not have any superclass. However, for

consistency, we can also turn this class into a mixin:

_eneric

tYPe Super(<>) is id_0tract tsgged private;

package Pcrt_Jlixln is

type Object is abstract new Super with private;

ge_rlc

type Self (<_) is new Object with privatat

with procedure Self PUt

(0: ila out: Self; C: in Character);

package O_eratlonm is

procedure Do_Put(O: in out Self; C: in Character)l

procedure Do_Put_LI_(O: in out Self;L: in String}l

dud Oparstlon_;

private

type Object is al_tract new Super with

record _ _ recoz_l;

end Port;

package body Port_Mixin is

package hod_ Operetlons is

procedure Do_Put(O: in out Self,C: in character) is

_ end Putt

procedure Do_Put_L_ne(O:in out SelflL:in Stril_) is

_in

for I in L'Rluage l_p

10022514L 4-9
Bel£_l_t(O,L(Z)); -- 8eae_aZly-_u_d call

_d loopt

e_d Put_/_Lne;

end Operat£o_-;

Even though the hardware port functionality does

not require any superclass operations, this mixin allows such functionality to be freely mixed in as part

of any class implementation.

Note that there is no typing relationship at all

between the port and buffer mixJns. Mixins provide

incremental implementation completely independently of problem-domain typing relationships. As a

complement to these mixins, we can define a set of

abstract types that capture typing relaflo_l_ps completely independentlyofimplementationdetails.

For example, we can use two abstracttypesto

definethe supertype/subtype relationship between

ports and buffered ports:

paclmge _'oz't Type_ _s

t:ype Poz*l: £a abstract tagg_l null _-ecord;

proco4uz_ Put(O: in out Po_; C: £n Character) £s

abmtz_ct;

pz_oodurm Put_J_no(O: £n a_t Port; Ls 4n _cr£ng) £s

abatraot 7

typ4 Duffo_ed Port 4s

abltz_ct new Port with null z_coz_l;

prooodurm FZush(O: In out Buffo_e__Poz_) _a _bst_ct;

For simplicity, this one package defines both abstract types, though they could equally well have

been defined in separate packages.

To. actually implement the port and buffered port

classes, we need to bring together the functionality

implemented in the port and buffer mixins with the

type hierarchy defined by the port and buffered port

abstract types. The following shows how this is clone

for the buffered port class:

wLth Por__Typem, I?orC_M£x_n, l_ffor__Lx£n;

package ]_z££ez_d_l?oz_c 4m

new _o_t__l_.iluffore_t_Po_t with p_£v_te;

new ]?o_t_Mi_Ln(Po_c:_T2_es .]h_ffez__Poz_) ;

Package ]_u££eted_po:t_Z_lemon_tlon is

new _f fo__NAs_n

(Clmr_cto_, lotto Zm_$_ntat£on.Ob:lect) I

LT]_ Ob_Jeot ta

new Duffoz_dJ_o_ Z_em_ntttLoa.Ob_ect with

n_I$ z_c_z_t;

The instantiations of the two mixins incrementally

builds the implementation of the type _u_'-

f ered_Port. ObJect.

As shown in Figure 3, the instantiation

Port_zmpl_aencation adds port-related components to the type Pore_Types .Buffered_Port

descendant

overrides

overriG_s

derived from

package Port_Implementation

Figure 3

tr_antitss ! instantiaies i

/" ................................. "-'.x. e.".................................. "N

i tenedc package _ _ Oimerlc pactmge

_uffet' Mixln ,_ _ Pact_.M_:in J

"_................................... / _ ................................

10022514L 4-10
(which hasno componentsitself), producingthe

type Port_Implementation.Object (this iS als0 all

example of why we need to allow mixin generics to

be instantiated with abstract types). The instantiation

Buffered_Port_Implementer los then extends the

type Port_Implementation. ObJect with buffer-related components, producing the type B uffered__Port_Implementation .Object. Tile full definition of Buffered_Port. ObJ e c t is a null extension of Buff ered_Port_implementat ion. ObJect.

The partial view of Buffered_Port .Object

giveninthevisiblepartofpackage BufferedPort

declares this type to be a descendant of

Port_Types .Port. The fulldefinitionof Buffered_Port .Object given in the private part of the

package isindeed a descendantofthe abstracttype

Port_Types. Port via the type extensions resulting

from the two mixin instantiations and the final null

extension. As such, it inherits the three abstract operations Put, Put_Line and Plush. However, Suffered Port.Object is not declared to be abstract

and so must provide implementations for these inberited operations.

The implementations of the Bur fered_Port. ObJect operations are, of course, given in the body of

package Bur fered_Port, using the Operations

generic packages from the port and buffer mixins:

package bodlv Dtlffored_Port is

pa=ke.go Port_Operation_ is

new Port_Xllplomontatlon. Oporatiomm

(BufferedPort.Object, Put) 1

pla=kege Buffered_Port Oporetions is

now Bu f f trod_Per t I_Iomenta tlon. Operations

( Bur foz_d_Port. ObJ _t,

Per t_Oporationa .DoPut,

Flush) ;

pro=edure Put(O= in out Ol_joct; C= in Character)

renames Bu f f ored_Pcr t_Opor&tion_. DO_PUt;

procedure Put_Line(Or in out Object; Lx in String}

renames Pc rt_Opora tlone. Do_Put_Line l

procedure Flush(Oz in out Obj_t)

ren4usns Bur f orod_Pcr t_Oporetionm. Do Plushl

end BufferedPort;

Since the buffer implementation is now independent

of the port implementation, both operations

generic packages must be instantiated here. The instantiation of Port_Implementation.Operations

uses operationBuffered_Port. Put for theself-referentialSelf_Put generic parameter. The instantiatJ0n of B u ffered_Port_Implementation.Operations uses operation BufferedPort. Flush for the

self-referential Sol f_Plu sh parameter. However, it

uses the operation Port_Operations.Do_Put, not

Buffered_Port .Put, for the superclass operation

super_Put. (This also shows why superclass operations must be parameters of the inner generic package operations in a mixin.)

The actual Buffered_Port. Object operations

are once again defined as renamings of subprograms

from the instantiated operations packages. Note,

however, that Put_,.tne is taken from Port_operat tons, not Buffered_Port_Operat ions, since the

buffer mixin does not implement a Put_Line operation. Nevertheless, the generic instantiations insure

that BufferedPort. Put_Line is implemented with

a proper self-referential call to Buffered_Port_operations.Do Put (the reader can trace how this

happens using Figure 4).

Conclusion

The use of generics for the static-binding of self-referential calls is at least of academic interest in the

comparison of inheritance and genericity. However,

since the generic approach can be a bit cumbersome,

one may ask if it has any practical application. In

fact, there are some good reasons to consider this

approach:

1. Experience has shown that the common use of

self-reference with inheritance can make an object-oriented program difficult to understand and

change (see, for example, [Taenzer 89],

[Leijter 92], [Wild 92] and [Wild 93]). The

generic approach gives the programmer much

more precise control about when and where

these self-referential bindings are made and thus

makes the use and intent of self-reference more

apparent to the maintainer.

2. In many safety-critical applications (such as

avionics software), any "dynamic" construct

(dynamic memory allocation, dynamics binding,

etc.) is regarded with suspicion. This is because

10022514L 4-11
Iiii! i !!iiiiiiiiiiiiiiiit

:::::::::::::::::::::::::::::: _re/_a3'r_$ "

._r_"18/?/o$

p_kage Buffered_Porl_Opera_on= "

pro¢ Do_Put

pro¢ Do_Rush

pa_k_e Port_Oper_o_

fl proc DoPut !1

package Bulter_l_ Port_lmplementatio_

- ......................

pac_e Port_lrnp_ementation

Figure 4

such features make it much harder to verify that

a program meets stringent safety requirements.

The generic approach provides self-reference

and deferred operation implementation with

fully static binding.

3. For a generic mixin, the generic clause of the inner or, erat±o,,8 generic package effectively

gives a complete "'typing" of the inheritance interface. That is, it explicitly lists all operations

required from the superclass and all operations

called self-referentially. As described by Haul

such a complete typing allows the type-safe replacement of a superclass implementation during class library maintenance [I-Iauk 93] (see

also [Gibbs 90] on the issues of modifying class

hierarchies). For example, in the a ug-

_'erod Port implementation given in the last

section, the use of the Port Mixin could be

easily replaced with a different implementation

of the hardware port functionality, so long as it

provided the Put operation needed by Buffer_Mtxin. Such a replacement requires no

changes to the implementation of the buffering

functionality, nor any changes to the clients of

nufferecl I_orl:. For that matter, it would be

equally easy to replace the nu_er__ixtn with a

different implementation of the buffering functionality.

Meyer was indeed correct in concluding that

genericity cannot be used to fully simulate inheritance. However, inheritance is a much more expansive mechanism than genedcity, and thus the comparison with genericity is not entirely fair. We can

decompose the inheritance mechanism as type extension plus polymorphic typing plus self-reference.

Genericity is only comparable to the parameterizat.ion-oriented effect of self-reference in the inheritance mechanism. As we have seen in this paper, inheritance actually can be simulated by type extension plus polymorphic typing plus genericity, and

that the generic approach actually has some potential

advantages.

I0022514L 4-12
References

[Ada9X 94a] Ada 9X Reference Manual (Draft

Version 5.0), ANSI/ISO/IEC DIS 8652, Ada Mapping/Revision Team, Intermetrics, June 1994

[Ada9X 94b] Ada 9X Rationale (Draft Version 5.0),

Ada Mapping/Revision Team, Intermetrics, June

1994

[Bracha 90] G. Bracha and W. Cook, "Mixin-based

Inheritance", Proceedings of the Conference on

Object-Oriented Programming Systems, Languages

and Applications / European Conference on ObjectOriented Programming, SIGPLAN Notices, October

1990

[Cook 89] W. Cook and L. Palsberg, "A Denotational Semantics of Inheritance and its Correctness",

Proceedings of the Conference on Object-Oriented

Programming, Systems, Languages and Applications, SIGPLAN Notices, October 1989

[Gibbs 90] S. Gibbs, D. Tsichritzis, E. Casals,

O. Nierstrasz and X. Pintado, "Class Management

for Software Communities", Communications of the

ACM, September 1990

[Goldberg 93] A. Goldberg and D. Robson, Smalltalk-80: The Language and Its Implementation,

Addison-Wesley, 1983

[Hauck 93] F. Hauck, "Inheritance Modeled with

Explicit Bindings: An Approach to Typed Inheritance", Proceedings of the Conference on ObjectOriented Programming, Systems, Languages and

Applications, SIGPLAN Notices, September/October 1993

[Lejter 92] M. Leijter, S. Meyers and S. P. Reiss,

"Support for Maintaining Object-Oriented Programs", IEEE Transactions on Software Engineering, December 1992

[Meyer 86] B. Meyer, "Genericity versus Inheritance", Proceedings of the Conference on ObjectOriented Programming, Systems, Languages and

Applications, SIGPLAN Notices, November 1986

[Meyer 88] B. Meyer, Object-Oriented Software

Construction, Prentice-Hail, 1988

[Moon 86] D. A. Moon, "Object-Oriented Programrning with Flavors", Proceedings of the Conference

on Object-Oriented Programming, Systems, Languages and Applications, SIGPLAN Notices, November 1986

[Seidewitz 92] E. Seidewitz, "Object-Oriented Programming with Mixins in Ada", Ada Letters,

March/April 1992

[Stroustrup 91] B. Stroustrup, The C++ Programming Language (2nd ed.), Addison-Wesley, 1991

[Taenzer 89] D. Taenzer, M. Ganti and S. Podar,

"Object-Oriented Software Reuse: The Yoyo Problem", Journal of Object-Oriented Programming,

September/October 1989

[Taft 93] T. Taft, "Ada 9X: From Abstraction -Oriented to Object-Oriented", Proceedings of the Coference on Object-Oriented Programming, Systems,

Languages and Applications, SIGPLAN Notices,

October 1993.

[Wegner 88] P. Wegner and S. B. Zdonik, "Inheritance as an Incremental Modification Mechanism or

What Like Is and Isn't Like," Proceedings of the

European Conference on Object-Oriented Programming, LNCS 322, Springer-Verlag, August 1988

[Wilde 92] N. Wilde and R. Huitt, "Maintenance

Support for Object-Oriented Programs", IEEE

Transactions on Software Engineering, December

1992

[Wilde 93] N. Wilde, P. Matthews and R. Huitt,

"Maintaining Object-Oriented Software", IEEE

Software, January 1993

10022514L 4-13

STANDARD BIBLIOGRAPHY OF SEL LITERATURE

The technical papers, memorandums, and documents listed in this bibliography are organized into two groups. The first group is composed of documents issued by the Software Engineering Laboratory (SEL) during its research and development activities.

The second group includes materials that were published elsewhere but pertain to SEL

activities.

SEL-ORIGINATED DOCUMENTS

SEL-76-001, Proceedings From the First Summer Software Engineering Workshop,

August 1976

SEL-77-002, Proceedings From the Second Summer Software Engineering Workshop,

September 1977

SEL-78-005, Proceedings From the Third Summer Software Engineering Workshop,

September 1978

SEL-78-006, GSFC Software Engineering Research Requirements Analysis Study,

E A. Scheffer and C. E. Velez, November 1978

SEL-78-007, Applicability of the Rayleigh Curve to the SEL Environment, T. E. Mapp,

December 1978

SEL-78-302, FORTRAN Static Source Code Analyzer Program (SAP) User's Guide

(Revision 3), W. J. Decker, W. A. Taylor, et al., July 1986

SEL-79-002, The Software Engineering Laboratory: Relationship Equations,

K. Freburger and V. R. Basili, May 1979

SEL-79-004, Evaluation of the Caine, Farber, and Gordon Program Design Language

(PDL) in the Goddard Space Flight Center (GSFC) Code 580 Software Design Environment, C. E. Goorevich, A. L. Green, and W. J. Decker, September 1979

SEL-79-005, Proceedings From the Fourth Summer Software Engineering Workshop,

November 1979

SEL-80-002, Multi-Level Expression Design Language-Requirement Level (MEDL-R)

System Evaluation, W. J. Decker and C. E. Goorevich, May 1980

SEL-80-005, A Study of the Musa Reliability Model, A. M. Miller, November 1980

SEL-80-006, Proceedings From the Fifth Annual Software Engineering Workshop,

November 1980

SEL-80-007, An Appraisal of Selected CostResource Estimation Models for Software

Systems, J. E Cook and E E. McGarry, December 1980

BI-1
SEL-80-008, Tutorial on Models and Metrics for Software Management and Engineering,

V. R. Basili, 1980

SEL-81-011, Evaluating Software Development by Analysis of Change Data,

D. M. Weiss, November 1981

SEL-81-012, The Rayleigh Curve as a Model for Effort Distribution Over the Life of

Medium Scale Software Systems, G. O. Picasso, December 1981

SEL-81-013, Proceedings of the Sixth Annual Software Engineering Workshop, December 1981

SEL-81-014,Automated Collection of Software Engineering Data in the Software Engineering Laboratory (SEL), A. L. Green, W. J. Decker, and E E. McGarry, September

1981

SEL-81-101, Guide toData Collection, V. E. Church, D. N. Card, E E. McGarry, et al.,

August 1982

SEL-81-104, The Software Engineering Laboratory, D.N. Card, E E. McGarry,

G. Page, et al., February 1982

SEL-81-110, Evaluation of an Independent Verification and Validation (IV& IO Methodology for Flight Dynamics, G. Page, E E. McGarry, and D. N. Card, June 1985

SEL-81-305,RecommendedApproach to Software Development, L. Landis, S. Waligora,

E E. McGarry, et al., June 1992

SEL-81-305SP1,Ada Developers' Supplement to the Recommended Approach, R. Kester and L. Landis, November 1993

SEL-82-001, Evaluation of Management Measures of Software Development, G. Page,

D. N. Card, and E E. McGarry, September 1982, vols. 1 and 2

SEL-82-004, Collected Software Engineering Papers: Volume 1, July 1982

SEL-82-007, Proceedings of the Seventh Annual Software Engineering Workshop,

December 1982

SEL-82-008, Evaluating Software Development by Analysis of Changes: The Data From

the Software Engineering Laboratory, V. R. Basili and D. M. Weiss, December 1982

SEL-82-102, FOR TRAN Static Source CodeAnalyzer Program (SAP) System Description

(Revision 1), W. A. Taylor and W. J. Decker, April 1985

SEL-82-105, Glossary of Software Engineering Laboratory Terms, T. A. Babst,

M. G. Rohleder, and E E. McGarry, October 1983

SEL-82-1306, Annotated Bibliography of Software Engineering Laboratory Literature,

L. Morusiewicz and J. Valett, November 1994

1000G_28

11/16_4

BI-2
SEL-83-001, An Approach to Software Cost Estimation, E E. McGarry, G. Page,

D. N. Card, et al., February 1984

SEL-83-002, Measures and Metrics for Soft'ware Development, D.N. Card,

E E. McGarry, G. Page, et al., March 1984

SEL-83-003, Collected Software Engineering Papers: Volume I1, November 1983

SEL-83-007, Proceedings of the Eighth Annual Software Engineering Workshop,

November 1983

SEL-83-106, Monitoring Software Development Through Dynamic Variables (Revision 1), C. W. Doerflinger, November 1989

SEL-84-003, Investigation of Specification Measures for the Software Engineering Laboratory (SEL), W. W. Agresti, V. E. Church, and E E. McGarry, December 1984

SEL-84-004, Proceedings of the Ninth Annual Software Engineering Workshop,

November 1984

SEL-84-101, Manager's Handbook for Software Development (Revision 1), L. Landis,

E E. McGarry, S. Waligora, et al., November 1990

SEL-85-001, A Comparison of Software Verification Techniques, D.N. Card,

R. W. Selby, Jr., F. E. McGarry, et al., April 1985

SEL-85-002, Ada Training Evaluation and Recommendations From the Gamma Ray

Observatory Ada Development Team, R. Murphy and M. Stark, October 1985

SEL-85-003, Collected Software Engineering Papers: Volume 111, November 1985

SEL-85-004, Evaluations of Software Technologies: Testing, CLEANROOM, and

Metrics, R. W. Selby, Jr., and V. R. Basili, May 1985

SEL-85-005, Software Verification and Testing, D.N. Card, E. Edwards, E McGarry,

and C. Antle, December 1985

SEL-85-006, Proceedings of the Tenth Annual Software Engineering Workshop,

December 1985

SEL-86-001, Programmer's Handbook for Flight Dynamics Software Development,

R. Wood and E. Edwards, March 1986

SEL-86-002, General Object-Oriented Software Development, E. Seidewitz and

M. Stark, August 1986

SEL-86-003, Flight Dynamics System Software Development Environment (FDS/SDE)

Tutorial, J. Buell and P. Myers, July 1986

SEL-86-004, Collected Software Engineering Papers: Volume 1_, November 1986

10000229

11115/94

BI-3
SEL-86-005, Measuring Software Design, D. N. Card et al., November 1986

SEL-86-006, Proceedings of the Eleventh Annual Software Engineering Workshop,

December 1986

SEL-87-001, Product Assurance Policies and Procedures for Flight Dynamics Software

Development, S. Perry et al., March 1987

SEL-87-002,Ada ® Style Guide (Version 1.1), E. Seidewitz et al., May 1987

SEL-87-003, Guidelines for Applying the Composite Specification Model (CSM),

W. W. Agresti, June 1987

SEL-87-004, Assessing the Ada ® Design Process and Its Implications: A Case Study,

S. Godfrey, C. Brophy, et al., July 1987

SEL-87-009, Collected Software Engineering Papers: Volume V, November 1987

SEL-87-010, Proceedings of the Twelfth Annual Software Engineering Workshop,

December 1987

SEL-88-001, System Testing of a Production Ada Project: The GRODY Study, J. Seigle,

L. Esker, and Y. Shi, November 1988

SEL-88-002, Collected Software Engineering Papers: Volume 1/7, November 1988

SEL-88-003, Evolution of Ada Technology in the Flight Dynamics Area: Design Phase

Analysis, K. Quimby and L. Esker, December 1988

SEL-88-004, Proceedings of the Thirteenth Annual Software Engineering Workshop,

November 1988

SEL-88-005, Proceedings of the First NASA Ada User's Symposium, December 1988

SEL-89-002, Implementation of a Production Ada Project: The GRODY Study,

S. Godfrey and C. Brophy, September 1989

SEL-89-004, Evolution of Ada Technology in the Flight Dynamics Area: Implementation/

Testing Phase Analysis, K. Quimby, L. Esker, L. Smith, M. Stark, and E McGarry,

November 1989

SEL-89-005, Lessons Learned in the Transition to Ada From FORTRAN at NASA/

Goddard, C. Brophy, November 1989

SEL-89-006, Collected Software Engineering Papers: Volume VII, November 1989

SEL-89-007, Proceedings of the Fourteenth Annual Software Engineering Workshop,

November 1989

SEL-89-008, Proceedings of the Second NASA Ada Users' Symposium, November 1989

1OOOO229

11115/94

BI-4
SEL-89-103, Software Management Environment (SME) Concepts and Architecture

(Revision 1), R. Hendrick, D. Kistler, and J. Valett, September 1992

SEL-89-301, Software Engineering Laborary (SEL) Database Organization and User's

Guide (Revision 3), L. Morusiewicz, December 1993

SEL-90-001, Database Access Manager for the Software Engineering Laboratory

(DAMSEL) User's Guide, M. Buhler, K. Pumphrey, and D. Spiegel, March 1990

SEL-90-002, The Cleanroom Case Study in the Software Engineering Laboratory: Project

Description and Early Analysis, S. Green et al., March 1990

SEL-90-003,A Study of the Portability of an Ada System in the Software Engineering Laboratory (SEL), L. O. Jun and S. R. Valett, June 1990

SEL-90-004, Gamma Ray Observatory Dynamics Simulator in Ada (GRODY) Experiment Summary, T. McDermott and M. Stark, September 1990

SEL-90-005, Collected Software Engineering Papers: Volume VIII, November 1990

SEL-90-006, Proceedings of the Fifteenth Annual Software Engineering Workshop,

November 1990

SEL-91-001, Software Engineering Laboratory (SEL) Relationships, Models, and Management Rules, W. Decker, R. Hendrick, and J. Valett, February 1991

SEL-91-003, Software Engineering Laboratory (SEL) Ada Performance Study Report,

E. W. Booth and M. E. Stark, July 1991

SEL-91-004, Software Engineering Laboratory (SEL) Cleanroom Process Model,

S. Green, November 1991

SEL-91-005, Collected Software Engineering Papers: Volume IX, November 1991

SEL-91-006, Proceedings of the Sixteenth Annual Software Engineering Workshop,

December 1991

SEL-91-102, Software Engineering Laboratory (SEL ) Data and Information Policy (Revision 1), E McGarry, August 1991

SEL-92-001, Software Management Environment (SME) Installation Guide, D. Kistler

and K. Jeletic, January 1992

SEL-92-002, Data Collection Procedures for the Software Engineering Laboratory (SEL)

Database, G. HeUer, J. Valett, and M. Wild, March 1992

SEL-92-003, Collected Software Engineering Papers: Volume X, November 1992

SEL-92-004, Proceedings of the Seventeenth Annual Software Engineering Workshop,

December 1992

"11115/9,4

BI-5
SEL-93-001, Collected Software Engineering Papers: Volume X/, November 1993

SEL-93-002, Cost and Schedule Estimation Study Report, S. Condon, M. Regardie,

M. Stark, et al., November 1993

SEL-93-003, Proceedings of the Eighteenth Annual Software Engineering Workshop,

December 1993

SEL-94-001, Software Management Environment (SME) Components and Algorithms,

R. Hendrick, D. Kistler, and J. Valett, February 1994

SEL-94-002, Software Measurement Guidebook, M. Bassman, E McGarry, R. Pajerski,

July 1994

SEL-94-003, C Style Guide, J. Doland and J. Valett, August 1994

SEL-94-004, Collected Software Engineering Papers: Volume XI1, November 1994

SEL-RELATED LITERATURE

l°Abd-E1-Hafiz, S. K., V. R. Basili, and G. Caldiera, "Towards Automated Support for

Extraction of Reusable Components," Proceedings of the 1EEE Conference on Software

Maintenance-1991 (CSM 91), October 1991

4Agresti, W. W., V. E. Church, D. N. Card, and P. L. Lo, "Designing With Ada for Satellite Simulation: A Case Study," Proceedings of the First International Symposium on

Ada for the NASA Space Station, June 1986

2Agresti, W. W., E E. McGarry, D. N. Card, et al., "Measuring Software Technology,"

Program Transformation and Programming Environments. New York: Springer-Verlag,

1984

1Bailey, J. W., and V. R. Basili, 'A Meta-Model for Software Development Resource

Expenditures," Proceedings of the Fifth International Conference on Software Engineering. New York: IEEE Computer Society Press, 1981

8Bailey, J. W., and V. R. Basili, "Software Reclamation: Improving Post-Development

Reusability," Proceedings of the Eighth Annual National Conference on Ada Technology,

March 1990

1°Bailey, J. W., and V. R. Basili, "The Software-Cycle Model for Re-Engineering and

Reuse," Proceedings of the ACM Tri-Ada 91 Conference, October 1991

1Basili, V. R., "Models and Metrics for Software Management and Engineering,"

ASME Advances in Computer Technology, January 1980, vol. 1

Basili, V. R., Tutorial on Models and Metrics for Software Management and Engineering.

New York: IEEE Computer Society Press, 1980 (also designated SEL-80-008)

10000_

11/15/94

BI-6
3Basili, V. R., "Quantitative Evaluation of Software Methodology," Proceedings of the

First Pan-Pacific Computer Conference, September 1985

7Basili, V. R., Maintenance = Reuse-Oriented Software Development, University of

Maryland, Technical Report TR-2244, May 1989

7Basili, V. R., Software Development: A Paradigm for the Future, University of Maryland,

Technical Report TR-2263, June 1989

8Basili, V. R., "Viewing Maintenance of Reuse-Oriented Software Development,"

IEEE Software, January 1990

1Basili, V. R., and J. Beane, "Can the Parr Curve Help With Manpower Distribution

and Resource Estimation Problems?," Journal of Systems and Software, February 1981,

vol. 2, no. 1

9Basili, V. R., G. Caldiera, and G. Cantone, '9, Reference Architecture for the Component Factory,'_4CM Transactions on Software Engineering and Methodology, January

1992

10Basili, V., G. Caldiera, E McGarry, et al., "The Software Engineering Laboratory--

An Operational Software Experience Factory," Proceedings of the Fourteenth International Conference on Software Engineering (ICSE 92), May 1992

1Basili, V. R., and K. Freburger, "Programming Measurement and Estimation in the

Software Engineering Laboratory," Journal of Systems and Software, February 1981,

vol. 2, no. 1

12Basili, V., and S. Green, "Software Process Evolution at the SEL," IEEE Software,

July 1994

3Basili, V. R., and N. M. Panlilio-Yap, "Finding Relationships Between Effort and

Other Variables in the SEL," Proceedings of the International Computer Software and

Applications Conference, October 1985

4Basili, V. R., and D. Patnaik, A Study on Fault Prediction and Reliability Assessment in

the SEL Environment, University of Maryland, Technical Report TR-1699, August 1986

2Basili, V. R., and B. T. Perricone, "Software Errors and Complexity: An Empirical

Investigation," Communications oftheACM, January 1984, vol. 27, no. 1

1Basili, V. R., and T. Phillips, "Evaluating and Comparing Software Metrics in the Software Engineering Laboratory," Proceedings of the ACM SIGMETRICS Symposium/

Workshop: Quality Metrics, March 1981

3Basili, V. R., and C. L. Ramsey, '_ARROWSMITH-P--A Prototype Expert System for

Software Engineering Management," Proceedings of the IEEE/MITRE Expert Systems

in Government Symposium, October 1985

10000229

11115/94

BI-7
Basili, V. R., and J. Ramsey, Structural Coverage of Functional Testing, University of

Maryland, Technical Report TR-1442, September 1984

Basili, V. R., and R. Reiter, "Evaluating Automatabie Measures for Software Development," Proceedings of the Workshop on Quantitative Software Models for Reliability,

Complexity, and Cost. New York: IEEE Computer Society Press, 1979

5Basili, V. R., and H. D. Rombach, "Tailoring the Software Process to Project Goals

and Environments," Proceedings of the 9th International Conference on Software Engineering, March 1987

5Basili, V. R., and H. D. Rombach, "T A M E: Tailoring an Ada Measurement Environment," Proceedings of the Joint Ada Conference, March 1987

5Basili, V. R., and H. D. Rombach, "TAM E: Integrating Measurement Into Software

Environments," University of Maryland, Technical Report TR-1764, June 1987

6Basili, V. R., and H. D. Rombach, "The TAME Project: Towards ImprovementOriented Software Environments," IEEE Transactions on Software Engineering, June

1988

7Basili, V. R., and H. D. Rombach, TowardsA Comprehensive Framework for Reuse: A

Reuse-Enabling Software Evolution Environment, University of Maryland, Technical

Report TR-2158, December 1988

8Basili, V. R., and H. D. Rombach, Towards A Comprehensive Framework for Reuse:

Model-Based Reuse Characterization Schemes, University of Maryland, Technical

Report TR-2446, April 1990

913asili, V. R., and H. D. Rombach, "Support for Comprehensive Reuse," Software Engineering Journal, September 1991

3Basili, V. R., and R. W. Selby, Jr., "Calculation and Use of an Environment's Characteristic Software Metric Set," Proceedings of the Eighth International Conference on Software Engineering. New York: IEEE Computer Society Press, 1985

Basili, V. R., and R. W. Selby, "Comparing the Effectiveness of Software Testing Strategies," IEEE Transactions on Software Engineering, December 1987

3Basili, V. R., and R. W. Selby, Jr., "Four Applications of a Software Data Collection

and Analysis Methodology," Proceedings of the NA TO Advanced Study Institute, August

1985

5Basili, V. R., and R. Selby, "Comparing the Effectiveness of Software Testing Strategies," IEEE Transactions on Software Engineering, December 1987

9Basili, V. R., and R. W. Selby, "Paradigms for Experimentation and Empirical Studies

in Software Engineering," Reliability Engineering and System Safety, January 1991

I0000229

1111,_g4

BI-8
4Basili, V. R., R. W. Selby, Jr., and D. H. Hutchens, "Experimentation in Software

Engineering," IEEE Transactions on Software Engineering, July 1986

2Basili, V. R., R. W. Selby, and T. Phillips, "Metric Analysis and Data Validation Across

FORTRAN Projects," IEEE Transactions on Software Engineering, November 1983

2Basili, V. R., and D. M. Weiss, A Methodology for Collecting Valid Software Engineering

Data, University of Maryland, Technical Report TR-1235, December 1982

3Basili, V. R., and D. M. Weiss, 'A Methodology for Collecting Valid Software Engineering Data," IEEE Transactions on Software Engineering, November 1984

1Basili, V. R., and M. V. Zelkowitz, "The Software Engineering Laboratory: Objectives," Proceedings of the Fifteenth Annual Conference on Computer Personnel Research,

August 1977

Basili, V. R., and M. V. Zelkowitz, "Designing a Software Measurement Experiment,"

Proceedings of the Software Life Cycle Management Workshop, September 1977

1Basili, V. R., and M. V. Zelkowitz, "Operation of the Software Engineering Laboratory," Proceedings of the Second Software Life Cycle Management Workshop, August

1978

1Basili, V. R., and M. V. Zelkowitz, "Measuring Software Development Characteristics

in the Local Environment," Computers and Structures, August 1978, vol. 10

Basili, V. R., and M. V. Zelkowitz, "Analyzing Medium Scale Software Development,"

Proceedings of the Third International Conference on Software Engineering. New York:

IEEE Computer Society Press, 1978

Bassman, M. J., E McGarry, and R. Pajerski, Software Measurement Guidebook,

NASA-GB-001-04, Software Engineering Program, July 1994

9Booth, E. W., and M. E. Stark, "Designing Configurable Software: COMPASS Implementation Concepts," Proceedings of Tri-Ada 1991, October 1991

1°Booth, E. W., and M. E. Stark, "Software Engineering Laboratory Ada Performance

Study--Results and Implications," Proceedings of the Fourth Annual NASA Ada User's

Symposium, April 1992

1°Briand, L. C., and V. R. Basili, "A Classification Procedure for the Effective Management of Changes During the Maintenance Process," Proceedings of the 1992 IEEE Conference on Software Maintenance (CSM 92), November 1992

1°Briand, L. C., V. R. Basili, and C. J. Hetmanski, "Providing an Empirical Basis for

Optimizing the Verification and Testing Phases of Software Development," Proceedings of the Third IEEE International Symposium on Software Reliability Engineering

(ISSRE 92), October 1992

1111_

BI-9
nBriand, L. C., V. R. Basili, and C. J. Hetmanski, Developing Interpretable Models with

Optimized Set Reduction for Identifying High Risk Software Components, TR-3048,

University of Maryland, Technical Report, March !993

12Briand, L. C., V. R. Basili, Y. Kim, and D. R. Squier, '_. Change Analysis Process to

Characterize Software Maintenance Projects," Proceedings of the International Conference on Software Maintenance, September 1994

9Briand, L. C., V. R. Basili, and W. M. Thomas,A Pattern Recognition Approach for Software Engineering Data Analysis, University of Maryland, Technical Report TR-2672,

May 1991

11Briand, L. C., S. Morasca, and V. R. Basili, "Measuring and Assessing Maintainability

at the End of High Level Design," Proceedings ofthe 1993 IEEE Conference on Software

Maintenance (CSM 93), November 1993

12Briand, L., S. Morasca, and V. R. Basili, Defining and Validating High-Level Design

Metrics, University of Maryland, Technical Report TR-3301, June 1994

nBriand, L. C., W. M. Thomas, and C. J. Hetmanski, "Modeling and Managing Risk

Early in Software Development," Proceedings of the Fifteenth International Conference

on Software Engineering (ICSE 93), May 1993

5Brophy, C.E., W. W. Agresti, and V. R. Basili, "Lessons Learned in Use of AdaOriented Design Methods," Proceedings of the Joint Ada Conference, March 1987

6Brophy, C. E., S. Godfrey, W. W. Agresti, and V. R. Basili, "Lessons Learned in the

Implementation Phase of a Large Ada Project," Proceedings of the Washington Ada

Technical Conference, March 1988

2Card, D. N., "Early Estimation of Resource Expenditures and Program Size,"

Computer Sciences Corporation, Technical Memorandum, June 1982

2Card, D. N., "Comparison of Regression Modeling Techniques for Resource Estimation," Computer Sciences Corporation, Technical Memorandum, November 1982

3Card, D.N., '_ Software Technology Evaluation Program," Annais do XVIII

Congresso Nacional de Informatica, October 1985

5Card, D. N., and W. W. Agresti, "Resolving the Software Science Anomaly," Journal

of Systems and Software, 1987

6Card, D. N., and W. W. Agresti, "Measuring Software Design Complexity," Journal of

Systems and Software, June 1988

4Card, D. N., V. E. Church, and W. W. Agresti, '_An Empirical Study of Software Design

Practices," 1EEE Transactions on Software Engineering, February 1986

Card, D. N., V. E. Church, W. W. Agresti, and Q. L. Jordan, 'A Software Engineering

View of Flight Dynamics Analysis System," Parts I and II, Computer Sciences Corporation, Technical Memorandum, February 1984

looo_

Vll"Z,_a4

BI-IO
Card, D. N., Q. L. Jordan, and V. E. Church, "Characteristics of FORTRAN Modules,"

Computer Sciences Corporation, Technical Memorandum, June 1984

5Card, D.N., E E. McGarry, and G. T Page, "Evaluating Software Engineering

Technologies," IEEE Transactions on Software Engineering, July 1987

3Card, D. N., G. T. Page, and E E. McGarry, "Criteria for Software Modularization,"

Proceedings of the Eighth International Conference on Software Engineering. New York:

IEEE Computer Society Press, 1985

1Chen, E., and M. V. Zelkowitz, "Use of Cluster Analysis To Evaluate Software Engineering Methodologies," Proceedings of the Fifth International Conference on Software

Engineering. New York: IEEE Computer Society Press, 1981

4Church, V.E., D. N. Card, W.W. Agresti, and Q. L. Jordan, '_,n Approach for

Assessing Software Prototypes," ACM Software Engineering Notes, July 1986

2Doerflinger, C. W., and V. R. Basili, "Monitoring Software Development Through

Dynamic Variables," Proceedings of the Seventh International Computer Software and

Applications Conference. New York: IEEE Computer Society Press, 1983

Doubleday, D., ASAP: An Ada Static Source Code Analyzer Program, University of

Maryland, Technical Report TR-1895, August 1987 (NOTE: 100 pages long)

6Godfrey, S., and C. Brophy, "Experiences in the Implementation of a Large Ada

Project," Proceedings of the 1988 Washington Ada Symposium, June 1988

5Jeffery, D. R., and V. Basili, Characterizing Resource Data: A Model for Logical

Association of Software Data, University of Maryland, Technical Report TR-1848, May

1987

6jeffery, D. R., and V. R. Basili, "Validating the TAME Resource Data Model," Proceedings of the Tenth International Conference on Software Engineering, April 1988

llLi, N. R., and M. V. Zelkowitz, '_m Information Model for Use in Software Management Estimation and Prediction,"Proceedings of the Second lntemational Conference on

Information Knowledge Management, November 1993

5Mark, L., and H. D. Rombach, A Meta Information Base for Software Engineering,

University of Maryland, Technical Report TR-1765, July 1987

6Mark, L., and H. D. Rombach, "Generating Customized Software Engineering

Information Bases From Software Process and Product Specifications," Proceedings of

the 22nd Annual Hawaii International Conference on System Sciences, January 1989

5McGarry, E E., and W. W. Agresti, "Measuring Ada for Software Development in the

Software Engineering Laboratory (SEL)," Proceedings of the 21st Annual Hawaii

International Conference on System Sciences, January 1988

1OOOO229

11/15/94

BI-11
7McGarry, E, L. Esker, and K. Quimby, "Evolution of Ada Technology in a Production

Software Environment," Proceedings of the Sixth Washington Ada Symposium

(WADAS), June 1989

3McGarry, E E., J. Valett, and D. Hall, "Measuring the Impact of Computer Resource

Quality on the Software Development Process and Product," Proceedings of the

Hawaiian International Conference on System Sciences, January 1985

3page, G., E E. McGarry, and D. N. Card, '_ Practical Experience With Independent

Verification and Validation," Proceedings of the Eighth International Computer Software

and Applications Conference, November 1984

12porter, A. A., L. G. Votta, Jr., and V. R. Basili, Comparing Detection Methods for Software Requirements Inspections: A Replicated Experiment, University of Maryland, Technical Report TR-3327, July 1994

5Ramsey, C. L., and V. R. Basili, '_ Evaluation of Expert Systems for Software Engineering Management," IEEE Transactions on Software Engineering, June 1989

3Ramsey, J., and V. R. Basili, '_knalyzing the Test Process Using Structural Coverage,"

Proceedings of the Eighth International Conference on Software Engineering. New York:

IEEE Computer Society Press, 1985

5Rombach, H. D., '_ Controlled Experiment on the Impact of Software Structure on

Maintainability," IEEE Transactions on Software Engineering, March 1987

8Rombach, H. D., "Design Measurement: Some Lessons Learned," IEEE Software,

March 1990

9Rombach, H. D., "Software Reuse: A Key to the Maintenance Problem," Butterworth

Journal of Information and Software Technology, January/February 1991

6Rombach, H. D., and V. R. Basili, "Quantitative Assessment of Maintenance: An

Industrial Case Study," Proceedings From the Conference on Software Maintenance,

September 1987

6Rombach, H. D., and L. Mark, "Software Process and Product Specifications: A Basis

for Generating Customized SE Information Bases," Proceedings of the 22nd Annual

Hawaii International Conference on System Sciences, January 1989

7Rombach, H. D., and B. T. Ulery, Establishing a Measurement Based Maintenance

Improvement Program: Lessons Learned in the SEL, University of Maryland, Technical

Report TR-2252, May 1989

l°Rombach, H. D., B. T. Ulery, and J. D. Valett, "Toward Full Life Cycle Control:

Adding Maintenance Measurement to the SEL," Journal of Systems and Software,

May 1992

11/15/lN,

BI-12
6Seidewitz, E., "Object-Oriented Programming in Smalltalk and Ada," Proceedings

of the 1987 Conference on Object-Oriented Programming Systems, Languages, and

Applications, October 1987

5Seidewitz, E., "General Object-Oriented Software Development: Background and

Experience," Proceedings of the 21st Hawaii International Conference on System

Sciences, January 1988

6Seidewitz, E., "General Object-Oriented Software Development with Ada: A Life

Cycle Approach," Proceedings of the CASE Technology Conference, April 1988

9Seidewitz, E., "Object-Oriented Programming Through Type Extension in Ada 9X,"

Ada Letters, March/April 1991

l°Seidewitz, E., "Object-Oriented Programming With Mixins in Ada," Ada Letters,

March/April 1992

12Seidewitz, E., "Genericity Versus Inheritance Reconsidered: Self-Reference Using

Generics," Proceedings of the Conference on Object-Oriented Programming Systems,

Languages, and Applications, October 1994

4Seidewitz, E., and M. Stark, "Towards a General Object-Oriented Software Development Methodology," Proceedings of the First International Symposium on Ada for the

NASA Space Station, June 1986

9Seidewitz, E., and M. Stark, '_kn Object-Oriented Approach to Parameterized Software in Ada," Proceedings of the Eighth Washington Ada Symposium, June 1991

8Stark, M., "On Designing Parametrized Systems Using Ada," Proceedings of the

Seventh Washington Ada Symposium, June 1990

nstark, M., "Impacts of Object-Oriented Technologies: Seven Years of SEL Studies,"

Proceedings of the Conference on Object-Oriented Programming Systems, Languages, and

Applications, September 1993

7Stark, M. E. and E. W. Booth, "Using Ada to Maximize Verbatim Software Reuse,"

Proceedings of TRI-Ada 1989, October 1989

5Stark, M., and E. Seidewitz, "Towards a General Object-Oriented Ada Lifecycle,"

Proceedings of the Joint Ada Conference, March 1987

10Straub, P. A., and M. V. Zelkowitz, "On the Nature of Bias and Defects in the Software Specification Process," Proceedings of the Sixteenth International Computer Software and Applications Conference (COMPSAC 92), September 1992

8Straub, P. A., and M. V. Zelkowitz, "PUC: A Functional Specification Language for

Ada," Proceedings of the Tenth International Conference of the Chilean Computer Science

Society, July 1990

BI-13

10000Q29

11115/94
7Sunazuka, T., and V. R. Basili, lntegrating Automated Support for a Software Management Cycle Into the TAME System, University of Maryland, Technical Report TR-2289,

July 1989

l°Tian, J., A. Porter, and M. V. Zelkowitz, '_m Improved Classification Tree Analysis of

High Cost Modules Based Upon an Axiomatic Definition of Complexity," Proceedings

of the Third 1EEE International Symposium on Software Reliability Engineering

(ISSRE 92), October 1992

Turner, C., and G. Caron, A Comparison of RADC and NASA/SEL Software Development Data, Data and Analysis Center for Software, Special Publication, May 1981

l°Valett, J. D., '_kutomated Support for Experience-Based Software Management,"

Proceedings of the Second Irvine Software Symposium (ISS '92), March 1992

5Valett, J. D., and E E. McGarry, 'A Summary of Software Measurement Experiences

in the Software Engineering Laboratory," Proceedings of the 21st Annual Hawaii

International Conference on System Sciences, January 1988

3Weiss, D. M., and V. R. Basili, "Evaluating Software Development by Analysis of

Changes: Some Data From the Software Engineering Laboratory," IEEE Transactions

on Software Engineering, February 1985

5Wu, L., V. R. Basili, and IC Reed, 'A Structure Coverage Tool for Ada Software Systems," Proceedings of the Joint Ada Conference, March 1987

1Zelkowitz, M. V., "Resource Estimation for Medium-Scale Software Projects," Proceedings of the Twelfth Conference on the Interface of Statistics and Computer Science.

New York: IEEE Computer Society Press, 1979

2Zelkowitz, M.V., "Data Collection and Evaluation for Experimental Computer

Science Research,"EmpiricalFoundationsfor Computer and Information Science (Proceedings), November 1982

6Zelkowitz, M. V., "The Effectiveness of Software Prototyping: A Case Study," Proceedings of the 26th Annual Technical Symposium of the Washington, D. C., Chapter of the

ACM, June 1987

6Zelkowitz, M. V., "Resource Utilization During Software Development," Journal of

Systems and Software, 1988

8Zelkowitz, M. V., "Evolution Towards Specifications Environment: Experiences With

Syntax Editors," Information and Software Technology, April 1990

IOOOO_

11115_,4

BI-14
NOTES:

1This article also appears

Volume 1, July 1982.

2This article also appears

Volume I1, November 1983.

m SEL-82-004, Collected Software Engmeenng

m SEL-83-003, Collected Software Engmeenng

3This article also appears m SEL-85-003, Collected Software Engmeenng

Volume III, November 1985.

4This. article also appears

Volume 1_, November 1986.

5This article also appears

Volume 1_,November 1987.

m SEL-86-004, Collected Software Engmeenng

m SEL-87-009, Collected Software Engmeenng

6This article also appears m SEL-88-002, Collected Software Engmeenng

Volume VI, November 1988.

7This article also appears m SEL-89-006, Collected Software Engmeenng

Volume VII, November 1989.

8This article also appears m SEL-90-005, Collected Software Engmeenng

Volume I,'111, November 1990.

9This article also appears m SEL-91-005, Collected Software Engmeenng

Volume IX, November 1991.

1°This article also appears m SEL-92-003, Collected Software Engmeenng

Volume X, November 1992.

llThis article also appears m SEL-93-001, Collected Software Engmeenng

Volume XI, November 1993.

12This article also appears m SEL-94-004, Collected Software Engmeenng

Volume XII, November 1994.

Papers:

Papers:

Papers:

Papers:

Papers:

Papers:

Papers:

Papers:

Papers:

Papers:

Papers:

Papers:

10001_29

11115/94

BI-15

Form Approved REPORT DOCUMENT/, iON PAGE oMeNo,o7o4-o188

Public reporting burden for this collection of information is estimated to average 1 )'.our per response, including the time for reviewing instructions, searching existing data sources, gathering

and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any o(her aspect of this collection of

information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite

12C,4, Arlington, VA 22202-4302, and to the Office of Mana_lement and Budget, Paperwork Reduction Propel 10704-01881, Washin_lton, DC 20503,

1. AGENCY USE ONLY (Leave blank) 2. REPORT DATE 3. REPORT TYPE AND DATES COVERED

November 1994 Contractor Report

4. TITLE AND SUBTITLE

Collected Software Engineering Papers: Volume XII

6. AUTHOR(S)

Software Engineering Laboratory

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

Software Engineering Branch

Code 552

Goddard Space Flight Center

Greenbelt, Maryland ,,

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)

NASA Aeronautics and Space Administration

Washington, D.C. 20546-0001

5. FUNDING NUMBERS

552

f_

8. PERFORMING ORGANIZATION

REPORT NUMBER

SEL-94-004

10. SPONSORING/MONITORING

AGENCY REPORT NUMBER

CR-189409

11. SUPPLEMENTARY NOTES

12a. DISTRIBUTION/AVAILABILI'rY STATEMENT

Unclassified-Unlimited

Subject Category: 61

Report is available from the NASA Center for AeroSpace Information,

800 Elkridge Landing Road, Linthicum Hei_;hts_ MD 21090; (301) 621-0390.

13. ABSTRACT (Maximum 200 words)

12b. DISTRIBUTION CODE

This document is a collection of selected technical papers produced by participants in the Software Engineering

Laboratory (SEL) from November 1993 through October 1994. The purpose of the document is to make available, in one

reference, some results of SEL research that originally appeared in a number of different forums. This is the 12th such

volume of technical papers produced by the SEL. Although these papers cover several topics related to software engineering,

they do not encompass the entire scope of SEL activities and interests. Additional information about the SEL and its research

efforts may be obtained from the sources listed in the bibliography at the end of this document.

14. SUBJECT TERMS

Software Engineering, Software Measurement, Ada Technology, Bibliography

17. SECURITY CLASStRCATION 18.SECURITY CLASSIRCATION

OF REPORT OF THIS PAGE

Unclassified Unclassified

NSN 7540-01-280-5500

19. SECURITY CLASSlRCATION

OF ABSTRACT

Unclassified

15. NUMBER ?(F/oPAGES

16. PRICE CODE

20. LIMITATION OF ABSTRACT

Unlimited

Standard Form 298 (Rev. 2-89)


..... _ "i_ _ f. _7. k _-- •

'i

• _ r _.

7t k