I

I

I

I

I

I

I

I

i

I

I

|

I

i

I

I

I

I

i

iN "- :_' "

SOFTWARE ENGINEERING LABORATORY SERIES

|- .

SEL-88-003

/'- /.st. _,_

COLLECTED SOFTWARE

ENGINEERING PAPERS: VOLUME ii

NOVEMBER 1983

National Aeronautics and

Goddard Space Flight Center

Greenbelt. Maryland 20771
I

i

I

I

I

I

!

I

I

i

I

I

I

I

I

I

I

I

I

SOFTWARE ENGINEERING LABORATORY SERIES SEL-83-003

COLLECTED SOFTWARE,

ENGINEERING PAPERS: VOLUME II

NOVEMBER 1983

National Aeronautics and

Space Administration

Goddard Space Flight Center

Greenbelt, Maryland 20771
l

I

I

i

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

FOREWORD

The Software Engineering Laboratory (SEL) is an organization

sponsored by the National Aeronautics and Space

Administration/Goddard Space Flight Center (NASA/GSFC) and

created for the purpose of investigating the effectiveness

of software engineering technologies when applied to the

development of applications software. The SEL was created

in 1977 and has three primary organizational members:

NASA/GSFC (Systems Development and Analysis Branch)

The University of Maryland (Computer Sciences Department)

Computer Sciences Corporation (Flight Systems Operation)

The goals of the SEL are (i) to understand the software development process in the GSFC environment; (2) to measure

the effect of various methodologies, tools, and models on

this process; and (3) to identify and then to apply successful development practices. The activities, findings, and

recommendations of the SEL are recorded in the Software Engineering Laboratory Series, a continuing series of reports

that includes this document. The papers contained in this

document appeared previously as indicated in each section.

Single copies of this document can be obtained by writing to

Frank E. McGarry

Code 582

NASA/GSFC

Greenbelt, Maryland 20771

_--_C,:2DiNG PAGE BLANK NOT FILIIEI)

IA_IN,TENltONALLt 'BLANK
|

I

I

I

I

!

i

I

!

I

I

I

I

i

I

I

I

i

TABLE OF CONTENTS

Section 1 - Introduction ................

Section 2 - The Software En@ineerin_ Laboratory ....

"Measuring Software Technology," W. W. Agresti,

F. E. McGarry, D. N. Card, et al ..........

"Technical Summary 1982: Report to the National

Aeronautics and Space Administration," V. R. Basili

Section 3 - Resource Models ..............

"Comparison of Regression Modeling Techniques for

Resource Estimation," D. N. Card ..........

"Early Estimation of Resource Expenditures and

Program Size " D N Card

Section 4 - Software Measures .............

1-1

2-1

2-3 --/

3-25-

4-1-- _1, r

"Metric Analysis and Data Validation Across

FORTRAN Projects," V. R. Basili, R. W. Selby, /i

and To Phillips • • • • • • • • • • • • • • • • • • 4i 3-5

"Monitoring Software Development Through Dynamic

Variables, " C ....W Doerflinger and V. R. Basili.

"Software Errors and Complexity: An Empirical

Investigation," V. R. Basili and B. T. Perricone. .

Section 5 - Data Collection ..............

"A Methodology for Collecting Valid Software

Engineering Data," V. R. Basili and D. M. Weiss . .

"Data Collection and Evaluation for Experimental

Computer Science Research, M. V. Zelkowitz ....

Biblio@raphy of SEL Literature

4-45 "_

5-3 _/

5-27

:_:_C_D_N_ pEtE BLANK HOT FILMED

v

I_L_,iN;TENTiONALL_ BLANK
|IIIIIIIiIIiIIII

SECTION1- INTRODUCTION
|

I

!

I

!

i

I

I

I

l

I

I

I

I

I

I

l

l

SECTION 1 - INTRODUCTION

This document is a collection of technical papers produced

by participants in the Software Engineering Laboratory (SEL)

during the period January i, 1982, through November 30, 1983.

The purpose of the document is to make available, in one reference, some results of SEL research that originally appeared

in a number of different forums. This is the second such

volume of technical papers produced by the SEL. Although

these papers cover several topics related to software engineering, they do not encompass the entire scope of SEL activities and interests. Additional information about the SEL

and its research efforts may be obtained from the sources

listed in the bibliography at the end of this document.

For the convenience of this presentation, the nine papers

contained here are grouped into four major categories:

• The Software Engineering Laboratory

• Resource Models

• Software Measures

• Data Collection

The first category presents summaries of the SEL organization, operation, and research activities. The second and

third categories include papers describing the results of

specific research projects in the areas of resource models

and software measures, respectively. The last category

presents papers describing strategies for data collection

for software engineerlng research.

The SEL is actively working to increase its understanding and

to improve the software development process at Goddard Space

Flight Center. Future efforts will be documented in additional volumes of the Collected Software Engineering Papers

and other SEL publications.

l-1

I
SECTION 2 - THE SOFTWARE ENGINEERING LABORATORY
l

I

I

I

I

I

I

I

I

I

I

I

I

I

l

I

l

l

l

SECTION 2 - THE SOFTWARE ENGINEERING LABORATORY

The technical papers included in this section were originally prepared as indicated below.

• Agresti, W. W., F. E. McGarry, D. N. Card, et al.,

"Measuring Software Technology," Computer Sciences

Corporation, Technical Memorandum, November 1983

(reprinted by permission of the authors)

A version of this paper will appear in Pro@ram

Transformation and Pro@rammer Environments.

New York: Springer-Verlag, 1984.

• Basili, V. R., "Technical Summary - 1982: Report

to the National Aeronautics and Space Administration," University of Maryland, Technical Memorandum, December 1982 (reprinted by permission of the

author)

2-1
I

i

I

I

I

I

I

l

I

I

l

I

I

I

I

I

l

I

I

1_87- ,?,4896

MEASURING SOFTWARE TECHNOLOGY

Wo Wo Agresti, Do N. Card, V. E. Church, and G. Page

Computer Sciences Corporation

System Sciences Division

8728 Colesville Road

Silver Spring, Maryland 20910

F. E. McGarry

National Aeronautics and Space Administration

Goddard Space Flight Center

Code 582

Greenbelt, Maryland 20771

ABSTRACT

Results are reported from a series of investigations into the effectiveness of various methods and tools used in a software production

environment. The basis for the analysis is a project data base,

built through extensive data collection and process instrumentation.

The project profiles become an organizational memory, serving as a

reference point for an active program of measurement and experimentation on software technology.

INTRODUCTION

Many proposals aimed at improving the software development process

have emerged during the past several years. Such approaches as

structured design, automated development tools, software metrics,

resource estimation models, and special management techniques have

been directed at building, maintaining, and estimating the software

process and product.

Although the software development community has been presented with

these new tools and methods, it is not clear which of them will prove

effective in particular environments. When this question is approached from the user's perspective, the issue is to associate with

each programming environment a set of enabling conditions and "win"

predicates to signal when methods can be applied and which ones will

improve performance. LacKing such guidelines, organizations are left

to introduce new procedures with little understanding of their likely

effect.

Assessing methods and tools for potential application is a central

activity of the Software Engineering Laboratory (SEL) [i, 2]. The

SEL was established in 1977 by the National Aeronautics and Space

,_,,:.__,_ P_,GE E_L.ANI{NOT. F._

2-3

__iI__ENI_JONALLZ BLAN[
Administration (NASA)/Goddard Space Flight Center (GSFC) in conjunction with Computer Sciences Corporation and the University of

Maryland. The SEL's approach is to understand and measure the software development prdcess, measure the effects of new methods through

experimentation, and apply those methods and tools that offer improvement. The environment of interest supports flight dynamics applications at NASA/GSFC. This scientific software consists primarily

of FORTRAN, with some assembler code, and involves interactive

graphics. The average size of a project is 60,000 to 70,000 source

lines of code.

SEL investigations demonstrate the advantages of building and maintaining an organizational memory on which to base a program of experimentation and evaluation. Over 4_ projects, involving

1.8 million source lines of code, have been monitored since 1977.

Project data have been collected from five sources:

• Activity and change forms completed by programmers and managers

• Automated computer accounting information

• Automated tools such as code analyzers

• Subjective evaluations by managers

• Personal interviews

The resulting data base contains over 25 megabytes of profile information on completed projects.

Some highlights of SEL investigations using the project history data

base are presented here, organized into three sections:

• Programmer Productivity

• Cost Models

• Technology Evaluations

PROGRAMMER PRODUCTIVITY

The least understood element of the software development process is

the behavior of the programmer. One SEL study examined the distribution of programmer time spent on various activities. When specific

dates were used to mark the end of one phase and the beginning of the

next, 22 percent of the totai hours were attributed to the design

phase, with 48 percent for coding, and 30 percent for testing. "However, if the programmers' completed forms were used to identify actual time spent on various activities, the breakdown was

2-4

II

il

I

I

I

II

II

I

I

l

I

i

II

I

I

II

II

II
I

i

i

I

l

I

I

I

I

I

l

I

I

I

I

I

I

I

approximately equal for the four categories of designing, coding,

testing, and "other" (activities such as travel, training, and

unknown) [3]. Although an attractive target for raising productivity

was t6 eliminate the "other" category, the SEL found that this was

not easily done.

Regarding individual programmer productivity, the SEL found differences as great as l0 to i, where productivity was measured in lines

of code per unit of effort [4]. This result was consistent with

similar studies in other organizations [5].

COST MODELS

Cost is often expressed in terms of the effort required to develop

software. In the effort equation,

E = aI b

where E equals effort in staff time and I equals size in lines of

code, some studies reported a value of b greater than one, indicating

that effort must be increased at a higher rate than the increase in

system size. The SEL analysis of projects in its data base did not

support this result, finding instead a nearly linear relationship

between effort and size [6]. This conclusion may be due to the SEL

projects being smaller than those that would require more than a

linear increase in effort.

In a separate study, the SEL used cost data from projects to evaluate

the performance of various resource estimation models. One study,

using a subset of completed projects, compared the predictive ability

of five models: Dory, SEL, PRICE S, Tecolote, and COCOMO [7]. The

objective was to determine which model best characterized the SEL

environment. The results showed that some models worked well on some

projects, but no model emerged as a single source on which to base a

program of estimation [8]. In the SEL environment, cost models have

value as a supplementary tool to flag extreme cases and to reinforce

the estimates of experienced managers.

TECHNOLOGY EVALUATIONS

Several SEL experiments have been conducted to assess the effectiveness of different process technologies. One study focused on the use

of an independent verification and validation (IV&V) team. The

2-5

I
premise for introducing an IV&V team into the software development

process is that any added cost will be offset by the early discovery

of errors. The expected benefit is a software product of greater

quality and reliability. In experimenting with an IV&V team in the

SEL environment, the benefits were not completely realized [9]. The

record on early error detection was better with IV&V than without it,

but the reliability of the final product was not improved. Also, the

productivity of the development team was comparatively low, due in

part to the necessary interaction with the IV&V team. The conclusion

was that an IV&V team was not effective in the SEL environment, but

may be effective where there are larger projects or higher reliability requirements.

A recent SEL investigation measured the effect of seven specific

techniques on productivity and reliability. From the project data

base, indices were developed to capture the degree of use of quality

assurance procedures, development tools, documentation, structured

code, top-down development, code reading, and chief programmer team

organization. The results showed that the greatest productivity and

reliability improvements due to methodology use lie only in the range

of 15 to 30 percent. Significant factors within this range are the

positive effect of structured code on productivity and the positive

effects of quality assurance, documentation, and code reading on reliability [10].

Figure i summarizes the perceived effectiveness of various practices

in the the SEL environment [4]. The placement of the models and

methods is based on the overhead cost of applying the model or method

and the benefit of its use. This summary must be interpreted in the

following context:

• The placement reflects subjective evaluations as well as

experimental results.

• The chart is indicative of experiences in the SEL environment only.

• The dynamic nature of the situation is not apparent. The

evaluation may reflect on an earlier and less effective example of the model or method.

2-6

I

I

i

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I
II

II

l

II

II

I

II

I

I

I

II

I

II

i

I

II

i

e

t,m

U4

0

BENERT _,_._

Figure I. What Has Been Successful in Our Environment?

CONCLUSIONS

The experiences of the SEL demonstrate that statistically valid evaluation is possible in the software development environment, but only

if the prerequisite quantitative characterization of the process has

been obtained. Through its program of assessing and applying new

methods and tools, the SEL is actively pursuing the creation of a

more productive software development environment.

2-7

I
REFERENCES

I. V. R. Basili and M. V. Zelkowitz, "Operation of the Software

Engineering Laboratory," Proceedings of the Second U.S. Army/IEEE

Software Life C_cle Management Workshop. New York: Computer

Societies Press, 1978

2. D. N. Card, F. E. McGarry, G. Page, et al., SEL-81-104, The Softwar 9 En_ineerin_ Laboratory, Software Engineering Laboratory, 1982

3. E. Chen and M. V. Zelkowitz, "Use of Cluster Analysis To Evaluate

Software Engineering Methodologies," Proceedin@s of the Fifth

International Conference on Software Engineering. New York:

Computer Societies Press, 1981

4. F. E. McGarry, "What Have We Learned in the Last Six Years Measuring Software Development Technology," SEL-82-007, Proceedin@s

of the Seventh Annual Software Engineering Workshop, Software

Engineering Laboratory, 1982

5. H. Sackman, W. J. Erikson, and E. E. Grant, "Exploratory Experimental Studies Comparing Online and Offline Program Performance,"

Communications of the ACM, January 1968, vol. 11, no.l, pp. 3-11

6. J. W. Bailey and V. R. Basili, "A Meta-Model for Software Development Resource Expenditures," Proceedings of the Fifth International Conference on Software Engineering. New York: Computer

Societies Press, 1981

7. IIT Research Institute, Quantitative Software Models, Rome Air

Development Center, New York, 1979

8. J. Cook and F. E. McGarry, SEL-80-007, An Appraisal of Selected

Cost/Resource Estimation Models for Software Systems, Software

Engineering Laboratory, 1980

9. G. Page, "Methodology Evaluation: Effects of Independent Verification and Integration on One Class of Application," SEL-81-013,

Proceedings of the Sixth Annual Software En_ineerin@ Workshop,

Software Engineering Laboratory, 1981

i0. D. N. Card, F. E. McGarry, and G. Page, "Evaluating Software

Engineering Methodologies in the SEL" (paper presented at Sixth

Minnowbrook Workshop on Software Performance Evaluation,

Minnowbrook, New York, 1983)

2-8

!

i

I

II

i

I

I

I

I

l

I

I

I

I

I

I

i

I
-;_-_I

1_87- 24897

I

l

I

I

l

i

I

I

I

I

I

I

I

i

I

i

TECHNICAL SUMMARY

1982

REPORT TO THE NATIONAL AERONAUTICS

AND SPACE ADMINISTRATION

Grant 01-526104

Department of Computer Science

University of Maryland

College Park, MD 20742

Principal Investigator:

Dr. Victor Basill

2-9
Overview

During 1982, in conjunction with NASA/GSFC Software Engineering

Laboratory (SEL), research was conducted in 4 areas: Software Development Predictors, Error Analysis, Reliability Models and Software Metric

Analysis. Summaries of the projects follow below.

_. Software Development Predictors

A study is being done on the use of dynamic characteristics as

predictors for software development. It is hoped that by examining a

set of readily available characteristics, the project manager may be

able to determine such things as when a project is in trouble and evaluate the quality of the product as it is being designed.

Project DEB was selected as the control for the project since it

was considered fairly successful and is well documented. Information

found in the history files and resource summary files was initially

utilized. These files were chosen because the information they contain

is readily accessible to the managey (ie. number of lines of code, manpower, computer time) etc.). Several profiles of project DEB were then

made using this information, Project DEA's profiles were then compared

with these results. This project was chosen because it was very similar

to DEB but was considered less successful.

The history file was first examined to see if any growth pattern

existed for the lines of code. The initial look at DEA and DEB looked

hopeful but further investigation of other projects showed no discernible pattern. Other examinations of this file yielded similar results.

2-10

i

l

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I
l

I

I

I

i

I

I

l

l

I

l

I

I

I

I

I

I

l

When a comparison of the information in the history and resource

summary files was made some differences did appear. Initial plots used

accumulative totals versus different time factors. These plots did

demonstrate visible differences between the two projects. Further

investigation using weekly totals instead of accumulative totals showed

an even larger difference between the projects.

Project DEA had a higher frequency of changes at the beginning

of the project, while at the same time, the number of hours of manpower

reported for the interval was less. The number of computer runs made

was higher for DEB in the part of the project where DEA was experiencing

the higher number of changes per manpower. In all, project DEA appears

to have had less effort placed during the early phase of the project

which may of led to the problems in the end. Another important aspect

of project DEA was that several thousand lines of code appear to have

been transported. Adaptation of this code may explain the high number

of changes initially seen in DEA.

From this examination the following general goals and

hypothesis have been generated:

A) The manpower usage in the SEL environment is a discernible pattern

and may be used as a predictor.

I) The ideal staffing for a successful project is a two hump curve

with the second hump beginning roughly 2/3 into the project.

2) The two humps mentioned in hypothesis I should peak at approximately the same height.

3) The maximum peak height of the first hump is proportional to the

final size of the project. This also hold for the second hump based

on hypothesis two.

4) The location of the two peaks is constant with relation to the

amount of manpower utilized.

5) The amount of manpower expended between the two peaks is constant.

2-11

I
6) Projects deemed less successful by subjective analysis have

sharp changes in the amount of manpower spent per change.

B) The pattern of changes in relation to manpower, computer runs, lines

of code, etc. may be used as a predictor in the SEL environment.

1) The amount of manpower to make a change should increase toward

the end of a project and be stable at the beginning.

2) The manpower per change should be lower in the beginning of the

project. See also goal D.

3) Projects deemed less successful by subjective analysis have

sharp changes in the amount of manpower spent per change.

4) The ratio of changes to computer run should decrease as the proJect evolves.

5) The amount of computer time spent on detecting and correcting a

given change will remain constant.

C) The number of computer runs is closely related to the development of

a project and may be used to Judge project development.

I) The number of computer runs remains constant during the initial

hump of the staffing curve. The number of computer runs will drop

during the second hump of the staffing curve.

2) The ratio of changes to computer runs should decrease as the

project evolves.

D) A close examination of the types of changes and the pattern they make

over time should be a good indication of the success of a given project.

I) Time consuming changes that occur late in the project more often

appear in modified code.

2) Unit testing is not as extensive on modules with modified code.

Undetected errors may cause major problems latter in development.

3) The types of changes vary across the development of a project.

4) The number of changes per hour of manpower is related to the

type of changes being done.

5) The types of change that require more time to correct occur during the second staffing hump.

Several projects will now examined to test the validity of these

finds. The change report forms will also be examined to see if the

information in them yields any useful predictors.

To conclude, the study has completed its initial analysis of the

two projects. It appears there are some significant factors that could

be useful as predictors. Further analysis may yield some information

2-12

!

I

i

i

I

g

I

I

I

i

!

i

I

i

I

I

I

g

I
II

i

I

II

II

i

I

II

I

II

II

I

i

I

i

II

I

II

I

that would be useful to a project manager.

2. Error Analysis

A). Publication of existing results -- Three papers are being prepared

from earlier work on error analysis conducted by the SEL laboratory.

One is on the data collection methodology and the validation of the

accuracy of the data, the second one is on the analysis of the SEL proJects directly and the third one is a comparison of the SEL projects

with projects of the Naval Research Laboratory. These papers are

currently being submitted for publication and will be published as

University of Maryland Technical Reports in the interim.

B). A study on software errors and complexity -- The distribution and

relationships derived from the change data collected during the development of the medium scale satellite project shows that meaningful results

can be obtained which allow insight into software traits and the

environment in which it is developed. The project studied in this case

was GMAS. Modified and new modules were shown to behave similarly. An

dabstract classification scheme for errors which allows a better understanding of the overall traits, of a software project was also provided.

Finally, various size and complexity metrics are examined with respect

to errors detected within the software yielding some interesting

results. A University of Maryland Technical Report describing these

results was published [Bas82]. This paper has been submitted for publication.

C). A further examination of the error characteristics of the DE_A and

DE B projects is currently being undertaken. This error anaiys[s is

2-13
being conducted using the techniques developed and documented in [Wei81]

and [Per82]. The focal point of this research effort is to characterize

errors in the NASA/GSFC software development environment.

A preliminary review of a sample of the Change ReportForms from

both DE A and DE B has been conducted. The sample included only those

CRF°s for which an error change was reported. The purpose of this

review was to "get a flavor" for the data collected and to preliminarily

assess the consistency of that data with the results found to date by

SEL personnel.

The sample included 98 CRF's from DE A and 90 CRF's from DE B. Of

the 98 CRF's from DE A, 63 (64.3%) of the errors were classified as an

"error in the design or implementation of a single component." Of the

90 CRF's from DE B, 16 errors were reported as "clerical errors." Of the

remaining 74 DE B errors (non-clerical errors), 61 (84.2%) of the errors

were also classified as "errors in the design or implementation of a

single component."

Although the percentage classi_fied as "errors in a single component" for DE B was higher than the other studies, these preliminary

results appear to follow the results of previous analyses [Wei81]. As i_

that previous work, the distribution of errors in other categories does

not neatly fit a pattern. In fact, there are too few events in the

other categories to draw any initial conclusions. It will be interesting to explore the reason(s) DE B experienced a substantially larger

number of "clerical errors."

There are marked differences in the remaining DE A and DE B error

reports. This may be attributable to the reported differences in the

2-14

I

I

l

I

I

I

I

I

I

I
I

I

I

i

l

l

I

I

I

I

l

I

I

I

I

I

I

I

two projects. It is not possible at this time to conjecture on more

tangible causes for the differences. The full set of error change

reports will have to be examined, for both projects.

It is worth noting here that for DE A, 31 of 98 error reports

(31.8%) examined were classified as being an "error in the design or

implementation of more than one component." Based on previous results

cited above, this is an unusually high percentage. Only 4 components

(4.1%) had errors reported that were not in the design or implementation

of component(s) categories.

As part of the preliminary work toward the above goal, the related

literature released by SEL was reviewed. A conclusion reached was that

the definitions of several critical terms were not necessarily con-

_istent, and often times the technical reports make too great an assumption about the uniformity of use of software engineering terms.

"Interface" provides a good example of an ill-defined yet oft used

term. Using the definition from [Wei81] (the same definition is used in

[Bas8Ob] and [Gio79]) it is arguable that interface errors can be captured five ways from the CRF:

-an error involving more than one component;

-an error involving a common routine;

-from textual comments in the CRF (eg: a CRF for which the error

was entered as having affected one component but the text indicated

that the error was in a subroutine call statement);

-an error reported as having been located in one component but the

change required to repair the error affected more than one component; and

-a change that caused an error because either the change invalidated an assumption made elsewhere in the software or an assumption

made about the rest of the software in the design of the change was

incorrect (contingent on ability to capture supporting text and

ability to distinguish from erroneous assumptions made about a single component).

2-15

I
An effort is currently underway to develop a more restrictive set

of definitions for software engineering terms, specifically those that

apply to error analysis. The basis of this effort is the set of definitions published in [Bas80] and [Gio79] and will be modified, as necessary, in consultation with those persons associated with SEL in the past

and present, whose work is or was related to the error analysis effort.

_. Reliability Models

A study is being performed in the area of reliability models. This

research includes the field of program testing because the validity of

some reliability models depends on the answers to some unanswered questions about testing.

• The eventual goal of this research is to understand how and when to

use reliability models. We are investigating the use of functional

testing because some reliability models make assumptions about the way

program testing is accomplished [Musa]. It is not known if functional

testing satisfies the random testing assumptions made by the reliability

models. The validity of reliability models that use data generated by

functional testing is uncertaih until this question is answered.

We are using structural coverage metrics to gain further insight

into the effects of functional testing. A structural coverage metric is

a measure of how much of a program was executed for given input data.

Studying the coverage metric may allow us to develop other measures of

reliability.

An additional bonus of this research is that it allows us to compare functional testing and structural testing. It is not known how

2-16

I

I

I

I

i

I

l

I

I

I

I

I

I

I

I

I

I

I
I

I

I

l

I

i

I

I

I

I

I

I

I

I

I

I

I

I

these two methods of testing are related• The results of this investigation may answer that question.

Since January background material has been studied with regard to

reliability models, and functional and structural testing [Mueller]. A

FORTRAN preprocessor has been written to calculate the structural coverage metrics of GSFC FORTRAN source code.

The preprocessor calculates the simplest metric, the percent of

executable code that is executed• There are several ways to measure

coverage [Auerbach]. One method uses interpretation of the source code.

The interpreter records which statements are executed• At the end of

interpretation, it writes a list of executed statements.

The second method uses "switches", small sections of code that are

inserted into the source program text wherever the flow of control

diverges or converges. The switch has 2 values: 0 if it was not executed, I if it was executed• The value of the switches is output after

execution•

An example:

INTEGER SWITCH ( N )

FOR I = I, N

SWITCH (I) = 0

READ ( J );

IF ( even ( J ))

THEN

SWITCH ( I ) : I;

ELSE

SWITCH ( 2 ) = I;

2-17

I
ENDIF

FOR I = I, N

WRITE ( SWITCH ( I ));

END

When this program is executed, one of the two branches of the if

statement will be executed. By examining the values of the array

SWITCH, we can determine what code was executed. By analyzing the code

and counting statements, the number of statements executed can be determined• In practice, the amount of data generated will be large•

Software tools are needed to help analyze the data.

The switches can be inserted by a preprocessor (before compilation)

or by a compiler (during compilation). The switches may be in-line code

(as in the example) or a call to a switch subroutine that records the

flow of control.

This latter approach was taken and a preprocessor was developed

that runs on VAX/Unix at UMCP. The preprocessor takes a copy of the

input source code, and modifies it. This modified copy will be returned

to the source computer (at GSFC) where it will be compiled and executed.

The execution produces the desired coverage data. The coverage data

will be returned to the University for analysis.

Many things remain to be done before we reach our goal of understanding how and when to use reliability models. The immediate goal is

to try to answer the functional testing / reliability model question.

The project RADMAS has been chosen as an experimental system [CSC]. The

preprocessor must be used to modify the RADMAS source code. (The RADMAS

2-18

I

I

I

I

I

I

1

I

I

I

I

I

I

I

I

I

I

I

I
I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

i

I

project and its functionally-generated acceptance tests have been made

available for the coverage experiment.) The modified RADMAS code must be

executed at GSFC using the functionally-generated acceptance tests.

This experiment should answer these questions about functional

testing and reliability models:

-What is the percent coverage of functional testing?

-Does functional testing meet the randomness requirements

of the MTTF models? If not, can it be made to?

-Do the structural metrics show any useful patterns in

the way that functional testing tests programs? How

does the coverage set grow? At what rate does the coverage set

grow?

-How independent are individual tests from a coverage

point of view?

The results of this experiment will raise further questions about

functional testing and reliability models. This will require more experimentation. If these questions are answered, there is more work to do

concerning how and when to use reliability models.

4. Software Metrics.

The attraction of the ability to predict the effort in developing

or explain the quality of software has led to the proposal of several

theories and metrics [Hal77, McC76, Gaf, Che78, Cur79]. In the Software

Engineering Laboratory, the Halstead metrics, McCabe's cyclomatic complexity and various standard metrics have been analyzed for their relation to effort, development errors and one another [Bas82a]. This study

examined data collected from seven SEL (FORTRAN) projects and applied

three effort reporting accuracy checks to demonstrate the need to validate a database.

2-19
The investigation examined the correlations of the various metrics

with effort (functional specifications through acceptance testing) and

development errors (both discrete and weighted according to amount of

time to locate and fix) across several projects at once, within individual projects and for individual programmers across projects.

In order to remove the dependency of the distribution of the correlation coefficients on the actual measures of effort and errors, the

non-parametric Spearman rank-order correlation coefficients were examined [Ken79]. The metrics" correlations with actual effort seem to be

strongest when modules developed entirely by individual programmers or

taken from certain validated projects are considered. When examining

modules developed totally by individual programmers, two averages formed

from the proposed validity ratios induce a statistically significant

ordering of the magnitude of several of the metrics" correlations. The

systematic application of one of the data reliability checks (the frequency of effort reporting) substantially improves either all or several

of the projects" effort correlations with the metrics. In addition to

these relationships, the Halst%ad metrics seem to possess reasonable

correspondence with their estimators, although some of them have size

dependent properties. In comparing the strongest correlations, neither

Halstead's E metric, McCabes" cyclomatic complexity nor source lines of

code relates convincingly better with effort than the others.

The metrics examined in this study were calculated from primitive

measures derived from a source analyzing program (SAP -- Revision I)

[Dec82]. An earlier version of this static analyzer implemented a less

comprehensive definition of Halstead operators and operands[O'Ne78].

2-20

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I
I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

Some work has been done comparing the metrics" correlations when they

have been determined from the different interpretations of the primitive

measures.

This investigation has been submitted for publication to the Transactions on Software Engineering and will appear as a University of

Maryland Technlcal Report.

2-21
5. References

[Auerbach] Auerbach Publishers Inc., "Practical Measures for Program

Testing Thoroughness", 1977.

[BasS0] V. Basili, Tutorial on Models and Metrics for Software Management and Engineering, p. 340, IEEE 1980

[Bas82a] V. Basili, R. Seiby and T. Phillips, "Data Validation in a

Software Metric Analysis of FORTRAN Modules," -- to appear

IEEE Transactions on Software Engineering, July 1982.

[Bas82b] V. Basili and B. Perricone, "Software Errors and Complexity:

An Empirical Investigation," The Software Engineering

Laboratory, University of Maryland Technical Report TR-1195,

August 1982

[Bas82c] V. Basili, "An Assessment of Software Measures in the Software

Engineering Laboratory," presented at Goddard Space Flight

Center, January 1982.

[Card82] Card, D., F.McGarry and J. Page, "Evaluation of Management

Measures of Software Development," Vol I & II, Software

Engineering Laboratory Series, SEL - 82 - 001, Goddard Space

" Flight Center, September 1982.

[Chen 78] E. T. Chen, "Program Complexity and Programmer Productivity,"

IEEE Transactions on Software En_ineerins, Vol. SE-4, No. 3,

pp---_187-194 (May 1978).

[CSC] Computer Sciences Corporation, RADMAS User's Guide., September

1981.

[Curtis et al 79] Curtis, Sheppard and Milliman, "Third Time Charm:

Stronger Prediction of Programmer Performance by Software

ComPlexity Metrics," Proceedings of the Fourth International

on Software En_ineerins, pp. 356-360 (1979).

[Decker & Taylor 82] W. J Decker and W. A. Taylor, "FORTRAN Static

Source Code Analyzer Program (SAP) User's Guide (Revision

I)," SEL-78-I02, Software Engineering Laboratory, (May

1982).

[Oaffney & Heller ] J. Gaffney and O. L. Heller, "Macro Variable

Software Models for Application to Improved Software

Development Management," Proceedinss of Workshop on Quantitative Software Models for Reliability, Complexitz and Cost,

IEEE Computer Society.

[Gio79] S. Gloss-Soler, The DACS Glossary -- A Bibliography of Software

En_ineerin_ Terms, Data and Analysis Center for Software, p.

56, October 1979

[Halstead 77] M. Halstead, Elements of Software Science, Elsevier NorthHolland, New York (1977).

[Kendall & Stuart 79] M. Kendall and A. Stuart, The Advanced Theory of

Statistics, Vol. 2, Fourth Ed., MacMillian, New York, 1979,

pp. 503-508.

2-22

I

I

I

I

I

I

!

I

I

I

I

I

I

I

I

I

I

I
!

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

[McCabe 76] T. J. McCabe, "A Complexity Measure," IEEE Transactions on

Software Engineering, Vol. SE-2, pp. 308-320 (December

1976).

[Mueller] Muellerp Barbara, "Test Data Selection: A Comparison of Structural and Functional Testing", April 1980, private paper.

[Musa] Musa, John, D., "Software Reliability Management", Software Life

Cycle Management Workshop, August 1977.

[O'Neill et al 78] E. M: O'Neill, S. R. Waligora and C. E. Goorevich,

"FORTRAN Static Source Code Analyzer (SAP) User's Guide,"

SEL-78-002, Software Engineering Laboratory (February 1978).

[Pic82] G Picasso, "The Rayleigh Curve as a Model for Effort Distribution Over the Life of Medium Scale Software Systems,"

Department of Computer Science, University of Maryland

Technical Report TR-1186, July 1982.

[Wei81] D. Weiss, "Evaluating Software Development by Analysis of

Change Data," The Software Engineering Laboratory, University of Maryland Technical Report TR-1120, November 1981

2-23
IIIIIIIIiIIIIIII

SECTION3- RESOURCE MODELS

ii

HI
I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

SECTION 3 - RESOURCE MODELS

The technical papers included in this section were originally prepared as indicated below.

• Card, D. N., "Comparison of Regression Modeling

Techniques for Resource Estimation," Computer

Sciences Corporation, Technical Memorandum,

November 1982 (reprinted by permission of the

author)

• Card, D. N., "Early Estimation of Resource Expenditures and Program Size," Computer Sciences Corporation, Technical Memorandum, June 1982 (reprinted

by permission of the author)

3-1
I

I

I

I

i

I

I

I

I

I

I

I

I

I

I

I

I

I

i

COMPARISON OF REGRESSION MODELING

TECHNIQUES FOR RESOURCE ESTIMATION

N87-24898

Prepared by

COMPUTER SCIENCES CORPORATION

D_ N. Card

FoE

GODDARD SPACE FLIGHT CENTER

Under

Contract NAS 5-27555

Task Assignment 85100

November 1982

:__;_ZC_D!NG PAGE BLANK NOT Fll

3-3

_|N,T[NTtONALLY BLANK
INTRODUCTION

The development and validation of resource utilization models has

been an active area of software engineering research. Regression

analysis is the principal to01 employed in these studies. However, little attention has been given to determining which of the

various regression methods avail-=ble is the most appropriate.

The objective of the study presented in this memorandum is to colapare three alternative regression procedures by examining the results of their application to one commonly acceFted equation for

resource estimation. This memorandum summarizes the data studied,

describes the resource estimation equation, explains the regression procedures, and compares the results obtained from the procedures.

DATA SUMMARY

This study is based on data collected from 22 flight dynamics software projects studied by the Software Engineering Laboratory (SEL).

The general class of flight dynamics software includes applications

tosupport attitude determination, attitude control, maneuver planning, orbit adjustment, and mission analysis (Reference i). The

specific projects selected for this analysis were developed in

FORTRAN for operation on the same computer system. The range of

system size (developed lines of source code) and development effort

(staff-months) for e_hese 22 projects is indicated in Table I.

THE RESOURCE ESTIMATION EQUATION

Variations of one basic equation have been incorporated in many resource estimation models (Reference 2). This equation relates project size to development effort. Additive and/or multiplicative factors based on experience, complexity, software tape, etc. are added

to form more sensitive models. The SEL also has developed a model

based on this equation (Reference 3). The general form of the estimating equation is

H = AL B (i)

where

H = staff-months of effort

L = lines of source code

A, B are constants

I

I

I

3-4
IIIIIII

i

IIIIIIIIII

.rn

4} m

4} XO

4J

_ ,.-i _,D .i.c1_ _,oor_

__ ,_, ,_P ,-i ,..4 ,-4 ,-i

r_

cr_

r___o ,-Q u"1_ o', O

AOI

I

r__ _.1o•

3-5

oi o

_._o •u _j

o_ ,o _J

•_•o i _o _ .,-t

... ,-.1

_I_O

000:> I I

!I

1
Because the projects studied by the SEL include a substantial proportion of reused code, a "developed" lines of source code measure

was devised to account for the higher productivity due to reusing

code (Reference 3). The equatlon for computing developed lines of

source code is

L = N + E + 0.2S + 0.2U (2)

Where

L = developed lines of source code

N = newly coded lines

E = extensively modified lines

•S = slightly modified lines

U = lines reused unchanged.

This software product measure (L) can be related to three measures

of developmenteffort(H). These measures,, as they are defined for

the subsequent analysis, are the following:

• HP - programmer staff-months of effort

• HPM - programmer and manager staff-months of effort

• HPMO - programmer, manager, and other (total) staff-months of

effort

ALTERNATIVE REGRESSION PROCEDURES

Three alternative regression procedures are availabe for deriving

values for the constants in Equation i. These are the following:

• Non-linear regression of original data

• Linear regression of original data

• Linear regression of logarithmically transformed data

A non-linear regression procedure can find aleast-squares solution

for the constants in Equation 1 without requiring either a manipulation of the equation or a transformation of the data. Several such

algorithms have been implemented. However, the calculation of nonlinear solutions is computationally intensive. Thus, it consumes a

substantial amount of computer resources. Reference 4 describes the

derivative-free algorithm used in this study.

Equation 1 can be reduced to a linear form by fixing the value of

the exponent (B) at 1.0. The resulting equation is the following:

H = AL (3)

Then ordinary linear least-squares regression can be applied to the

untransfo_ed data. Unfortunately, this simple solution ignores the

conceptual importance of a potential exponential relationship between

software size and development effort.

3-6

!
!

!

!

!

!

I

I

I

t

I

!

!

!

II

!

I

!

!

This relationship can be captured by performing a logarithmic transformation of Equation 1 and the data. The resulting equation is

Log (H) = Log (A) + B Log (L) (4)

Solutions for A and B in this equation can be derived by ordinary

linear least-squares regression. Although this procedure is computationally less intensive than the non-linear procedure, it requires a prior transformation of the data. The range of the logarithmically transformed data is shown in Table 1.

COMPARISON OF RESULTING MODELS

Each of the regression procedures described in the previous section

were applied to the data for each measure of effort. These analyses

were performed with the Statistical Analysis System software package

(Reference 5). Table 2 summarizes the results. The goodness-offit obtained by any regression model is measured by the mean square

error (MSE) and correlation coefficient (R). Unfortunately, as shown

in Table 2, these values are not directly comparable for all the regression models considered here.

However, it is clear from Table 2 that for all measures of effort

the results provided by the linear and log-linear procedures are

very similar. The estimates of A and B for the log-linear model

(Equation 4) are close to those of the linear model (Equation 3);

sllght decreases in S in the log-linear case are compensated by increases in A. Furthermore, the correlation coefficients obtained

by the two procedures are nearly identical in all three cases.

Therefore, the linear regression procedure produces a model as good

as that of the log-linear procedure in a considerably more straightforward manner.

The model produced by the non-linear procedure differs considerably

from those produced by the linear and log-linear procedures (see

Table 2). The values of B (Equation i) depart significantly from

1.0; the relationship defined is clearly exponential. Furthermore,

the mean square error of the non-linear model is substantially less

than that of the linear model. Although a direct comparison between

the non-linear and log-linear models (in terms of MSE or R) is not

possible, the log-linear model is so close to the linear model that

we can safely conclude that the non-linear model is the most accurate

of the three.

Figures 1 through 3 illustrate the relationships between system size

and development effort defined by the linear and non-linear models.

(The log-linear model is not shown because it is so similar to the

linear model). A cursory examination of these figures indicates

that the linear model fits the data at the low end of the range better while the non-linear model fits the data at the high end of the

range better.

3-7
{/1

4J

P-t

oi 000

.,-I

t_° _l

III III III

OU

00

°l

aOI X

I

_1 _1 _1

ol

°1•M t_-,-t _ _,._

0._0 0-_0

,<

=

0

3-8

,<

(U=

+

1.4

• :ll II II

0

_J .. t__

_• .-= P-_:>=

,.-_= l._-,-__

•,-4 .,-_ t___ 'E_ ",_

IIIII

IIiIiIII
!I!IiII1I!IIIiIiIIi

Z•

-JZ

•Z ..I

==.-,_

Z_

Z ,.I•

B-9

r.oI

_1.

4-)

1-1

l.i 00

-¢1

'13000

1=0

,4
I

I

:- |

4J4J ,.

o--4 0 ,'

dh

, r._ il

•_ z " .io X

t

"o _

' |

: o

t

• * Z =J • O •

• z.._ _ ii

I

I

eO

:" |

,,

z " " :|o

Z ..1,

:_' •,, __ I

z - :o

z _. ,, =

m

•_* .=1 t

z_, : I

*0

0 0 0 " 0 0 _ 0 0 0 0 0 0 0 0 0 0 '"

I

3-10
IiIIiiIIIIIIlIlIlI

-i Z,

.,lZ

..IZ•

ORiG_._-_ALPAGE IS

OF POOR QUALITY

Zl

eZ.i

Z..J

®.ll •i.l .ii

Oil in

RI-,-I

,ia.,Ia_

m{n_

,,-i_ .,,,4

f.)..,-I0

>1 ii il il ,B_Z Z .I • Z --le P .J o '_ tl Ill

oioiit

0

4_

ii

",2

'8!wii!

*0i

-to

i

:oiii* IIn iii

r-i 0

_o_

ii

1oI!i

*0III

+l"_ 1

+0iII | ." :oI

"o

3-11

z

w%0

.=0

:ZI

q,,I

ll,,i

.t.l

,,,,,-4

,,&,l

01

,--400C0m

..=4

Q.

O- A

I
This phenomenon suggests an explanation for the closeness of the

log-linear model to the linear model. The effect of the logarithmic transformation is to weight smaller data values relatively

higher; Table 1 shows that large data values are affected more

dramatically by the transformation. Thus, the log-linear regression procedure produces a nearly linear result because it is

weighted in favor of smaller data values where observation indicates that the relationship between system size and development

effort is most nearly linear.

CONCLUSION

The non-linear regression procedure emerges from this study as the

superior technique. The foregoing evaluation of the three alternative regression procedures is summarized in Table 3. The total rating of each procedure shown in the table would be changed if the

three elements, of which it is composed (numerical accuracy, conceptual accuracy, and computational cost), were not weighted, equally.

In addition to the implication for the choice.of statistical techniques, the results of the study suggest some other factors that

should be considered in future research. The estimate of the exponent (B) derived by each procedure is fairly constant for all

measures of effort (see Table 2). The additional effort contributed by managers and others is accounted for by an increase in the

multiplicative factor (A) for the :_M and HPMO measures, cf effort.

Furthermore, the effort contributed by managers and other nonprogrammer personnel is strongly affected by the complexity of a project, the experience of the development team, and the development

methodologies employed. This confirms that these other effects should

be represented as multiplicative factors in a comprehensive resource

estimation model. Published models generally have taken this approach.

The exponential relationship, illustrated in Figures 1 through 3

has another implication. Although the relationship between system

size and development effort is nearly linear for small systems, _he

development effort due to size alone does not increase in proportion

to size for large systems. This suggests that the influences of

factors such as methodology, experience, and complexity, may be

more important for large systems.

The results of this study allow the optimistic conclusion that the

basic relationship presented in Equation 1 provides a sufficient

framework for the construction of comprehensive resource estimation

models when the appropriate statistical techniques are applied.

3-12

I

I

I

I

i

i

I

I

I

1

I

I

I

I

i

1

i

I
IIiI _1

_1 _,_, _, I _l ._,__

qlI .o

Z

I__ ._

0 _QI

I_,U I "_="_ 8"_1=_'_

_

t _. _l i _ -_j tiI

I-.I CI

r_ .,__ .,.4

oo,o=o

r_ _10 .,_0 I

gi

.,..I

lw

ol

Ill 0

{11 0

,..4U

.,-I

_.,it

3-13

i
APPENDIX - REGRESSION ANALYSIS RESULTS

This appendix reproduces the computer generated output from which

Table 2 was compiled. The following detailed tables are included:

Table Content

A-I

A-2

A-3

A-4

A-5

A-6

A-7

A-8

A-9

Non-Linear Model for Programmer Staff-Months

Linear Model for Programmer Staff-Months

Log-Linear for Programmer Staff-Months

Non-Linear Model for Programmer and Manager Staff-Months

Linear Model for Programmer and Manager Staff-Months

Log-Linear Model for Programmer and Manager Staff-Months

Non-Linear Model for Total Staa_-Months

Linear Model for Total Staff-Months

Log-Linear'Model for Total Staff-Months

3-14

I

I
IIi!IiI 3-15

_ri

r..

_Lm

¢n

- | o

_--_I_ • • w xX _WN=X

.J

.J

t_

I
m

0

XI

q-l.

'4..I

r./3

1.4 |0

q-I

,=.4

n:l I

q)

I

• PJ

0

0

s"

6

^

!i

m

i

-e

c:_

___oZ

3-16

_" I N L'_ w 0._I

,v
!!!i

Imz

!°

imoX

11

!°I!

I"

!!!I!!

wU0

¢EL

.J03

&0

^_ _, m

f_6

'0 I'_ _ __N

o

^

.4

t_

t'q

_n0

:Z

_4

w

3-17

•_ O0 A •

.

O0

G. •.m O_

66
01

4J0XI

q4

u,=l

I-I

¢1

¢1X

-

1400

_4

.,4IoI

,-4

-r

Om

_g ,-o,et ,q,

,/I fill*..

CIC f,1 r.,

,Ill_ m_

L1g

i

f_

G._O

n. O_n Z _.0

:1_._ _.

.oo_

o= o_1=

. _'0 _'• '

n,

•.

3-18

I!i

Q. _m

iIi!I

e_

cL

el

,el

i/I Nu

c_

,s

u.i Z
IIi

:i°_"+

ii +_

tl_. "+>.

I+

.. _x

,+i

g;

o_

q}!

,=4

E"

ItII1

___o"_ o=

• --0 tW>_ OX

•g

,I

-e

,,j ,,J < _ g ,,

0w0

_'}_I

i 3-19 I
II

d=

_=o

m_J

0_

om_m

I

=a

tlIi

3-20
I

III

ORIG'!._;AL PAGE IS

OF POOR QUALITY IIIiIIIIi 3-21 I
0

X

I-

..Ia

0

E_

(3.

,=4

_3

O

Z

I"

ql

¢,1

0

!,

N

_ 0

'r

& 6

m

_ m

0

^

o

- g

" r,

^ 0

0

el,

_ m

L

=_ _

0,,_ 0

GCZ O_

r

-

^

o

3-22

_u _ m _ 0 0

_. 01 .. • 01

,- _. 8

et _
0

I

rll

,-I

III

4J

0

P,t

W

0

Z

!

I

u

0

0

X

N

.J

w

x

-I

m

sJ _ m

w

^ In

_ o 2 o

tit

w _

_, o

2 _: 0

,. r. ,. ,-

Cl r-III ,.i_ o

X

-i

i/l

.I

q

I3-23

A 0

0

_. 0

_m

>

3-

_/I - •

^

O0

ik

_ "., O0

w

N _t_

q'O

oo
REFERENCES

i. Software Engineering Laboratory, SEL-81-104, The Software Enqineerin_ Laboratory, D. N. Card, F. E. McGarry, G. Page, et al., February

1982

2. V.R. Basili, "Models and Metrics for SoftwareManagement and Engineering, _ASME Advances in Computer Technology, January 1980, Vol. 1

3. J. W. Bailey and V. R. Basili, "A Meta-Model for Software Development Resource Expenditures," Proceedings of the Fifth International

Conference on Software Engineering. New York: Computer Societies

Press, 1981

4. M. L. Ralston and R. I. Jennrich, "Dud, A Derivative-Free Algorithm

for Nonlinear Least Squares," Technometrics, February 1978, Vol. 20,

NO. i.

5. SAS Institute, Statistical Analysis system user's Guide,

J. H. Goodnight, J. P. Sall, J. T. Helwig, et al., 1979

3-24
I

I

!

I

i

i

I

i

I

I

I

I

I

I

I

I

I

i

I

N87- 24899

EARLY ESTIMATION OF RESOURCE EXPENDITURES

AND PROGRAM S I ZE

Prepared by

COMPUTER SCIENCES CORPORATION

D. Card

For

GODDARD SPACE FLIGHT CENTER

Under

Contract NAS 5-24300

June 1982

3-25
i. INTRODUCTION

A substantial amount of software engineering research effort

has been focused on the development of software cost estimation models. A concensus (of sorts) has emerged on that

topic. The following relationship is widely accepted:

H s = aL b (i)

where H s = staff-hours of effort

L = lines of code

a = a constant

b = a constant

The Software Engineering Laboratory (SEL) has devised a

measure of lines of code based on the origin of the delivered

code that is substituted in the equation above. This is

Lde v = N + E + 0.2 (S+O) (2)

where Ldev .= "developed" line_ of code

N = newly implemented lines of code

E = extensively modified lines of code

S = slightly modified lines of code

O = old (unchanged) lines of code

Equation 1 using "developed" lines of code has given good

results as an estimator of development effort. (The analyses in this document are based on a sample of 20 groundbased attitude systems). Table 13 shows a regression analysis that produced a correlation of 0.99 and an estimate of

b of i.i when the value of a was fixed at 1.0 in Equation i.

Despite these encouraging results, this model has two sigificant limitations. These are the following:

• The substantial amount of development work done in

activities other than code implementation may not be

adequately considered in the lines of code measure.

3-26
!

!

i

I

I

I

i

i

I

I

I

l

I

l

I

I

• The lines of code, whether "delivered" or "developed",

is not known accurately until late in the development

cycle when accurate estimates are less useful.

The purpose of this memorandum is to discuss these limitations and to propose some alternative estimation models that

can be used earlier in the development process, e.g., during

requirements analysis and preliminary design.

2. MODELS OF WORK

The obvious alternative to lines of code as a measure of the

work done is pages of documentation. Although only a portion of a software development team is involved in coding,

almost everyone produces some documentation. This includes

requirements, design, and operations documents. Table i compares the components of developed lines of code with pages

of documentation as estimators or programmer hours. A regession model based on the two most strongly correlated

measures is described in Table 2. This model showed the

following relationship:

Hp = 0.056 N + 4.15D (3)

where H = programmer hours

P

N = newly implemented lines of code

D = pages of documentation

A similar comparison is made in Table 3 for these measures

as estimators of staff-hours (including programmer, manager,

and other hours). A regression model based on the two most

strongly correlated measures is described in Table 4. This

model showed the following relationship:

where H

s

N

D

H = 0.051 N + 7.10D

s

= staff-hours

= newly implemented lines of code

= pages of documentation

(4)

3-27

!
The correlation coefficient (r) associated with each of the

relationships expressed in Equations 3 and 4 was 0.97, comparable to that obtained by substituting Equation 2 for L in

Equation i. These results suggest that the best measures of

work done are lines of new code and pages of documentation.

Reused lines of code do not seem to contribute directly to

resource expenditures. However, the requirements analysis

and design effort involved in reusing previously developed

code may be included in the pages of documentation measure.

Although pages of documentation appears to be an important

measure of work, it has the same limitation as lines-of-code

measures. Pages of documentation cannot be determined accurately early in the development cycle. The next sections discusses some other measures that can be used to develop models

for early estimation of resource expenditures and program

size.

3. MODELS FOR EARLY ESTIMATION

Few objective measures are available early in the software

development process. The following five measures were considered in this analysis:

• Number of subsystems - requirements analysis

• Number of data sets - preliminary design

• Complexity (PRICE-S) - preliminary design

• Number of new modules - detailed design

• Number of reused modules (extensively modified, slightly

modified, and old) - detailed design

The following sections discuss the use of these measures for

early estimation of program size and resource expenditures.

3-28

I

I

I

I

I

I
I

i

I

I

I

I

I

I

i

I

I

I

I

I

I

I

I

i

3.1 PROGRAM SIZE

The correlations of the measures described here with delivered lines of code are compared in Table 5. Three regression

models were developed (Tables 6, 7, and 8). The two most

useful of these are the following:

Lde I = 7596 S (5)

Lde I = 168N + 195R (6)

where Lde I = delivered lines of code

S = number of subsystems

N = number of new modules

R -- number of reused modules

Equation 5 (r = 0.99) defines an estimating relationship for

program size that can be used during the requirements analysis phase. Equation 6 (r = 0.98) defines an estimating relationship of comparable reliability that can be used during

the design phase.

3.2 RESOURCE EXPENDITURES

The correlations of the measures described here with staffhours of effort are compared in Table 9. Three regression

models were developed (Tables I0, II, and 12). The two most

useful of these are the following:

H s = 1634 S (7)

H s = 45 N + 28 R (8)

where H = staff-hours

s

S = number of subsystems

N -- number of new modules

R = number of reused modules

Equation 7 (r = 0.93) defines an estimating relationship for

resource expenditures that can be used during the requirements analysis phase. Equation 8 (r = 0.94) defines an

3-29

I
estimating relationship of higher reliability that can be

used during the design phase.

4. CONCLUSION

The preceding analysis has demonstrated two important points.

These are the following:

• New measures of productivity which incorporate other

development products besides lines of code must be investigated. Pages of documentation is a good candidate.

• Effective estimates of program size and resource expenditures can be made using measures that are available early in the development cycle.

3-30

I

i

I

i

!

I

i

I

I

I

I

I

I

I

I

I

I

i

I
I|!!!i!!Ii!III!

II!! 0

u_

u_O

q.4O

tA

_JOEO

_J

0J

,.-4

,O

tO

E_

In=

n_Z

Q.

U.I m

I-- ..I Z

IAI

a.Z

WJ O --J

LW ,_ 0ZZ0

),q t_l

In tv

(3 u_

t_o

z

C{u_

mO

:[Z

ZZ

zzz_

O_U

_0_0

OlZa

_0_0

0000

ZZZ_

_XOQ_

ZZZZZZ

IZZO_Z

oooooo

3-31

_UUU

ZZZZ

ZZZZ

ZIZZ•

.,.

OOOO

LW

(3U0

r_wZ

.J00z

.J0

_JZ

t_Zo
0 =

q-i w

..d

cc

0 <

0

X

.--!

n,

• ,w

u_

m

m

t

^ _ _

+. 6 2 .

_ g

04

z ,.o

0

u_

°'l

'q" .

0 _r q'

;/J ¢q ,-_

_'§ r,

=

iY

.,

^ $:

mO

_. o6

mo

g_

_o

oo

QCZ _tD

2_ o4

• oow

c4 ^ • •

QI_

O.

O_

¢k • • O[tU

mm

_m

t_{q

0

_r_

O"

Z_ _L Z_

3-32
4J

0

u_

CO

-IJ

r/l

,-'4

n_

0

E_

0

O_

0

E

0

_J

n_

E_

"t"

Z

x

m

iz

lal

0

Z

a.

W.I

0

0

It.

;.W

0

X

Z

0

t.e

lal

w.I

o

-I

;a,I

0

0

Z

uJ

.J

",'1

0

!

z

.1

w_

XX

OXOO_a

w_

r_ ol q. 0 t_ 0 _ ol t_ ll_

_ t'5 ,... co t_ w (_1 co m ol

0_ _ _)

oooo oooooo

" _" " '," el e'_ _10t e_l t'_

3-33

_UUU

0_

zz

Z_ZZ

-moo

• , . °

0000

ug

C_

U

0

r_

z

.J

r_

0

.J

,,,.I

0

6
,WW0

¢tl

,-4

_U

k_0

_J

'00x

(U

tO

E_

i

•o

_s.•

q_

II1'1_

mq"

_ b,, ow

ww0

_0

m

_ Oi P"

CI_••

_. O0

eL

t_ 0'_0_0 c_c.D

_Sq_

IIliiIIIII

3-34
Ol 0

4J

tOEN

O3

440o

Ol

tOEo

(1)

..Q

(0

E_

wZ

.I0

I-

.a

m[

9( Z

.aZ

{1.

e_

O_ 0

u_

..I

ua00XZ0

),N

It;

tn

tu

t_0

_Ni Z

.J

uJ0Z

ul

ILl

.Jm

'(!NZ

XX

XOX_

OOwwO

UZ_ZZ

_0_

O_

00000

U_z_Zzu_

__°° ZZ_0Z

0___

...., .....

0000000000

3-35

xx>_x_ox

O_ 0_0_0_0

Z_ZZZZZZ_

___0

__0_

0000000000

>x_>x

x_ox_ 0

0

O00

2ZZ_

ZZZZZ

<==3:

00000

ZZZZZ

_0_

0_

00000

>,.

i-x

,,_

_r0U

=E

uJ0Z

l,,-

_J

I,-

,<

,m0

tn

>.

tnm

tn0Z0

01

r,.6

I
0z

C_

(1) N

._

Ca

._x

cU

II

_^

a

i

6

mm

I

,gII

I1

w

UP q'

o

or

0.0

_ _r

_^ |

u__0

G.

3-36
llIIIiIIIIIIOlllIIl

,-I0

m:

.,-I

.IJ

_=

.,4

rnN

.,-I

ca

,,.-I

,=

-,-4

4a

,-4

'°'l

N

_g

_m

i=

6

m, Q

&l,,.

_o

-

3-37

cs oo

,,j•.

0--

._ I¢1

I¢l,v

e_..

?., :-

o_,

_Pm.

N m._0 _m-

-=® _l
o

X

.,-4

_J

U2

N

-,-4

C_

.,._

J,J

<

.a

0

uJ

u

o_

Q.

0

Z

W

N

^

_ o 2 _

0

o o

Z

• w 0

_ w

e

O0

O. 000

• wN,_

0_

O0

n_ , , .

g. 000

I.,-

• _ ^

O.

I'-0--

3-38

0

• ° .

_w

_ _"-0 _

_r_

_, v_ _r _r
IIIIIIIIIlIIilIlII

ol

0

E

.,-I

4-:

qJ

rJ0

_J

>,

CO00

U]0 |XX_)

_Oqm

_z

0

X_X_

_|_x_ _'_=

I

3-39

!
0XG

.,._

_J

O_0

-,_ X

.0

o0

QC

Q.

.JON

G®

_Qm

,4

pb

&6

_6m

z__

^

V_

II

_g

I

A

_-O

IIII

'4 O0

¢¢ ,.., _

-

m

.^ | Z

_d

3-40

o.0

O.Z
!!

°Z

I°

.,.4

-,.I

4.J

I°

!°q_ @ | _ III

0

G.

tn_ in

zW

_,- _s'l_

> ,_ _"

tk.

^ oo

G.

___-_? _o

o=_=. _,"

• t-.

m<

Omm

! 3-41 !
0

_J

_3

.,"4 O0

_>

_J

n3

u40N

il

,,d••-

OI

I

III

il

II

_r

_°_m

f,,,.

¢_,.,

•. OOO

moo

^•.,

OOO

___ .___ ---

°,

414

_>

lIIIIIlIII

3-42
ilIIIIIIIlIIIllIIII

.,=i

010

","I

4J

rd

Q} Nt

aJU00

,.=I

_J0X

_4

,=4

IlJ

•NZ_ _.

if/w (_^

CJ_•m

_ _,_Eo

_0

_a

o

vI ufl ¢w

ci

0m

^

o ,_

u.._0

eL

Vi

0_0m

XM

N

I-,

3-43
<00

k_

,-t

,.Q

=

D<

r_ZZ_ r__ ul

3-44 IIIIIIIIIIIIIIIIII
IIlI

III

II

II

II

IIIi

II

|-

II

III

li

N0

f_00

ua0

_fi,i

||

zn¢

zz

1.0N

,.,.t ,,.i

i,Ik_

oo

1.. e,-

oo

F,,. i,.,

QQ

o_Q,.

Z

Z

Z

Z

Z

Z

Z

Z

Z

Z

Z

Z

iIii

_Zt

_' ali

÷mi

•

:L

0

ii:I

•

::I

1o

oil) Ii

',_!i

÷_iiiiiiiI

'4.

!

÷_Iii!i

4,Ii

:ov

_

i

i

0

X

,oZ

! 3-45 I
_JN

c,r'J OE

_J

._J

al

rJ_O

c_O

_J

QJ

_o

_N

_Y

_Q

V'B_

W_

ZZ

NN•.I .../

QQ

)"m

t/.. _.

OQ

k_l"

QQ

O,,.O.

O

O

a

OO

Q

Vt_Q

Vt QO

_m ,-,

_.(.w.4. m._. _÷ _._ _,_._.4. w.4. m._. u.¢. m._. m._. _tcf/ -4. I

;oiilIiIlliIi

,,iiliiiil,

_

iiiiiii

liiiili1

,,1

T"ilil:i:

3-46

ZQZ

IIIIIIIIIII!IIIIlI
iiIiIIIIIIIIIIIIII

0Q

U)

,.-.400

-,,'40

O_

ZW

Nil"

_U4 •.I ,,J

¢30

),._.

Z_t

•

3131"

00

O0

O.a.

Z

Z

W

Z nl

Z

_r_ZZ

.,..4

Z

ZZ

Z

ZZ

_Z

-,z:

i@

÷ o0 i qP I

:oI • IIIlii

,, ÷iIi÷'8ii+iI

lo1

,,Ii

÷(I) _

0

ilIi

÷0li

*0

I.o:i

3-47

I
.Ill0

-la

fll

4a0

.la

.,_

k'-I 0

.,=I

e.

(.- 0

.,,-i

,-=i

o,,=I

_tl'D

m.i._

_Li.l

.=I..I

O0

)..::,. _ I,.,,

WI,-

F=_,-

O0

_L_L

Q

Q

0Q

0 V'I

00

3-48

lIllIil

,,

÷aiiliiiliiiiiiiiii,

_

iliii

,,iiIiiiti

T"iiliiiii

IIIIIIIllIIIIIlIIII
llIIIIIIiIIlIIII

SECTION4- SOFTWARE MEASURES
l

l

l

l

l

I

l

I

I

I

I

I

I

I

l

l

I

I

SECTION 4 - SOFTWARE MEASURES

The technical papers included in this section were originally published as indicated below.

• Basili, V. R., R. W. Selby, and T. Phillips,

"Metric Analysis and Data Validation Across FORTRAN

Projects," University of Maryland, Technical Report

TR 1228, November 1982 (reprinted by permission of

the authors)

A version of this paper also appears in IEEE Transactions on Software En@ineerin@, November 1983,

vol. 9, no. 7.

• Doerflinger, C. W., and V. R. Basili, "Monitoring

Software Development Through Dynamic Variables,"

University of Maryland, Technical Memorandum,

August 1983 (reprinted by permission of the

authors).

A version of this paper also appears in Proceedin@s

of the Seventh International Computer Software and

Applications Conference. New York: Computer

Societies Press, November 1983.

Basili, V. R., and B. T. Perricone, "Software Errors and Complexity: An Empirical Investigation,

"University of Maryland, Technical Report TR-II95,

August 1982 (reprinted by permission of the authors)

A version of this paper will appear in Communications of the ACM, January 1984, vol. 27, no. i.

4-1

r

s

i

!
i

l

HI

I

I

I

I

I

l

I

I

I

I

I

I

I

I

I

i

N87- 24900

Technical Report TR-1228 November 1982

NSG-5123

AFOSR-F49620-80-C-O01

METRIC ANALYSIS AND DATA VALIDATION

ACROSS FORTRAN PROJECTS *

Victor R. Basili, Richard W. Selby, Jr.

and Tsai-Yun Phillips

Department of Computer Science

University of Maryland

College Park, MD 20742

*Research supported in part by the National Aeronautics and Space

Administration Grant NSG-5123 and the Air Force Office of Scientific

Research Contract AFOSR-F49620-80-C-O01 to the University of Maryland.

Computer support provided in part by the facilities of NASA/Goddard

Space Flight Center and the Computer Science Center at the University

of Maryland.

•,_.._NG PAGE BLANK NOT FILI_iZ_

4-3

_i N,T][NTIONALL_BI..AN[
ABSTRACT

The desire to predict the effort in developing or explain the

quality of software has led to the proposal of several metrics in

the literature. As a step toward valldating these metrics, the

Software Engineering Laboratory has analysed the Software Science

metrics, cyclomatic complexity and various standard program measures fop their relation to 1) effort (including design through

acceptance testing), 2) development errors (both discrete and

weighted according to the amount of time to locate and fix) and

3) one another. The data investigated are collected from a production FORTRAN environment and examined across several projects

at once, within Individual projects and by individual programmers

across projects, with three effort reporting accuracy checks

demonstratlng the need to validate a database. When the data

come from individual programmers or certain validated projects,

the metrics" correlations with actual effort seem to be strongest. For modules developed entirely by indlvidual programmers,

the validity ratios induce a statistically significant ordering

of several of the metrics" correlations. When comparing the

strongest correlations, neither Software Science's E metrIQ,

cyclomatic complexity nor source lines of code appears to relate

convincingly better with effort than the others.

4-4

i

l

i

l

I

I

I

l

I

l

I

l

I

I

I

I

I

l

!
I

l

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

l

I. Introduction

Several metrics based on characteristics of the software

product have appeared in the literature. These metrics attempt

to predict the effort in developing or

that software [11], [17], [19], [23].

to data from various organizations to

and appropriateness [I], [13], [15].

explain the quality of

Studies have applied them

determine their validity

However, the question of

how well the various metrics really measure or predict effort or

quality is still an issue in need of confirmation. Since

development environments and types of software vary, individual

studies within organizations are confounded by variations in the

predictive powers of the metrics. Studies across different

environments will be needed before this question can be answered

with any degree of confidence.

Among the most pQpular metrics have been the Software Science metrics of Halstead [19] and the cyclomatic complexity

metric of McCabe [23]. The Software Science E metric attempts to

quantify the complexity of understanding an algorithm.

Cyclomatic complexity has been applied to establish quality

thresholds for programs. Whether these metrics relate to the concepts of effort and quality depends on how these factors are

defined and measured. The definition of effort employed in this

paper is the amount of time required to produce the software product (the number of man-hours programmers and managers spent from

the beginning of functional design to the end of acceptance testing). One aspect of software quality is the number of errors

4-5

I
reported during the p_oduct°s development, and this is the measure associated with quality for this study.

Regarding a metric evaluation, there are several issues that

need to be addressed. How well do the various metrics predict or

explain these measures of effort and quality? Does the correspondence increase with greater accuracy of effort and error reporting? How do these metrics compare in predictive power to simpler

and more standard metrics, such as lines of source code or the

number of executable statements? These questions deal with the

external validation of the metrics. More fundamental questions

exist dealing with the internal validation or consistency of the

metrics. How well do the estimators defined actually relate to

the Software Science metrics? How

metrics, the cyclomatic complexity

tional metrics relate to one another?

do the Software Science

metric and the more tradiIn this paper, both sets

of issues are addressed. The analysis examines whether the given

family of metrics is internally consistent and attempts to determine how well these metrics really measure the quantities that

they theoretically describe.

One goal of the Software Engineering Laboratory [6], [7],

[8], [10], a Joint venture between the University of Maryland,

NASA/Goddard Space Flight Center and Computer Sciences Corporation, has been to provide an experimental database for examining

these relationships and providing insights into the answering of

such questions.

4-6

D

II

I

I

l

l

It

,l

II

I

I

i

I

I

i

I

I

II
I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

The software comprising the database is ground support

software for satellites -. The systems analyzed consist of 51,000

to 112,000 lines of FORTRAN source code and took between 6900 and

22,300 man-hours to develop over a period of 9 to 21 months.

There are from 200 to 600 modules (e.g., subroutines) in each

system and the staff size ranges from 8 to 23 people, including

the support Personnel. While anywhere from 10 to 61 percent of

the source code is modified from previous projects, this analysis

focuses on Just the newly developed modules.

The next section discusses the data collection process and

some of the potential problems involved. The third section

defines the metrics and interprets the counting procedure used in

their calculation. In the fourth section, the Software Science

metrics are correlated with their estimators and related to more

primitive program measures° Finally, the fifth section deterL

mines how well this collection of volume and complexity metrics

corresponds to actual effort and developmental errors.

II. The Data

The Software Engineering Laboratory collects data that deal

with many aspects of the development process and product. Among

these data are the effort to design, code and test the various

modules of the systems as well as the errors committed during

their development. The collected data are analyzed to provide

insights into software development and to study the effect of

various factors on the process and product. Unlike the typical

4-7

!
controlled experiments where the projects tend to be smaller and

the data collection process dominates the development process,

the major concern here is the software development process, and

the data collectors must affect minimal interference to the

developers.

This creates potential problems with the validity of the

data. For example, suppose we are interested in the effort

expended on a particular module and one programmer forgets to

turn in his weekly effort report. This can cause erroneous data

for all modules the programmer may have worked on that week.

Another problem is how does a programmer report time on the

integration testing of three modules? Does he charge the time to

the parent module of all three, even though that module may be

Just a small driver? That is clearly easier to do than to proportion the effort between all three modules he has worked on.

Another issue is how to count errors. An error that is limited to

one module is easy to assign. What about an error that required

the analysis of ten modules to determine that it affects changes

in three modules? Does the programmer associate one error with

all ten modules, an error with Just the three modules or one

third of an error with each of the three?- The larger the system

" Efforts [18], [21] have attempted to make this assignment

scheme more precise by the explanation_ a "fault" is a specific

manifestation in the source code of a programmer "error"; due to

a misconception or document discrepancy, a programmer commits an

"error" that can result in several "faults" in the program. With

this interpretation, wha_ are referred to as errors in this study

should probably be called faults. In the interest of consistency

with previous work and clarity, however, the term error will be

used throughout the paper.

4-8

l

l

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I
I

I

I

I

I

I

i

I

I

I

I

l

I

I

l

I

I

I

the more complicated the association. All this assumes that all

the errors are reported. It is common for programmers not to

report clerical errors because the time to fill out the error

report form might take longer than the time to fix the error.

These subtleties exist in most observation processes and must be

addressed in a fashion that is consistent and appropriate for the

environment.

The data discussed in this paper are extracted from several

sources. Effort data were obtained from a Component Status

Report that is filled out weekly by each programmer on the proJect. They report the time they spend on each module in the system partitioned into the phases of design, code and test, as well

as any other time they spend on work related to the project,

e.g., documentation, meetings, etc. A module is defined as any

named object in the system; that is, a module is either a main

procedure, block data, subroutine or function. The Resource Summary Form, filled out weekly by the project management,

represents accounting data and records all time charged to the

project for the various personnel, but does not break effort down

on a module basis. Both of these effort reports are utilized in

Section V of this paper to validate the effort reporting on the

modules. The errors are collected from the Change Report Forms

that are completed by a programmer each time a change is made to

the system. While the collection of effort and error data is a

subjective process and done manually, the remainder of the

software measures are objective and their calculation is

4-9

!
automated.

A static code analyzlng program called SAP [25] automatically computes several of the metrics examined in this analysis.

On a module basis, the SAP program determines the number of

source and executable statements, the cyclomatio complexity, the

primitive Software Science metrics and various other volume and

complexity related measures. Computer Sciences Corporation

developed SAP specifically for the Software EnglneePing Laboratory and the program has been recently updated [14] to incorporate a mope consistent and thorough counting scheme of the

Software Science parameters. In an earlier study, Basili and

Phillips [3] employed the prellminary version of SAP in a related

analysis. The next section explains the revised ccuntlng procedure and defines the various metrics.

III. Metric Definition

In the application of each of the metrics, there exist various ways to count each of the entities. This section interprets

the counting procedure used by the updated version of SAP and

defines each of the metrics examined in the analysis. These

definitions are given relative to the FORTRAN language, since

that is the language used in all the projects studied here. The

counting scheme depends on the syntactic analysis performed by

SAP and is, therefore, not necessarily chosen to coincide exactly

with other definitions of the various counts.

4-10

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I
I

I

I

I

i

I

I

I

l

I

I

I

I

I

l

I

I

I

Primitive Software Science metrics Software Science

defines the vocabulary metric n as the sum of the number of

unique operators nl and the number of unique operands n2. The

operators fall into three classes.

I) Basic operators include

÷ _ • / am =

• GE. .OT. .AND.

ii) Keyword operators include

// .NE. .EQ. .LE. .LT.

• XOR. .NOT. .EOV. .NEQV.

IF() THEN

IF{) THEN ELSE

IF() , ,

IF() THEN ENDIF

IF() THEN ELSE ENDIF

IF() THEN

ELSEIF() THEN

• .. ENDIF

DO

DOWHILE

GOTO <tarEet>

GOTO (TIJ..Tn) <expr>

/e logical if m/

le logical if-then-else e/

/e arithmetic if e/

/e block if ml

/o block if-then-else m/

/n case if i/

/e do loop m/

/m while loop m/

/m unconditional goto: distinct

targets imply different operators m/

/m computed goto: different number of

tangets imply different operators m/

GOTO <ident>, (T1...Tn) /_ assigne4 goto: distinct identifiers

<subr>(, ,

END=

ERR=

ASSIGNTO

EOS

e<target>)

imply different operators e/

/e alternate return m/

/_ read/write option m/

/m read/write option m/

/m target assignment m/

/_ implicit statement delimiter m/

iii) Special operators consist of the names of subroutines,

functions and entry points.

Operands consist of the all variable names and constants. Note

that the major differences of this counting scheme from that used

by Basili and Phillips [3] are in the way goto and if statements

are counted.

The metric n I represents the potential vocabulary, and

Software Science defines it as the sum of the minimum number of

4-11

I
operators nle and the minimum number of operands n2 s. The potential operator count nl m is equal to two; that is, nl • equals one

grouping operator plus one subroutine/function designator. In

this paper, the potential operand count n2 s is equal to the sum

of the number of variables referenced from common blocks, the

number of formal parameters in the subroutine and the number of

additional arguments in entry points.

Source lines This is the total number of source lines that

appear in the module, including comments and any data statements

while excluding blank lines.

Source lines - comments This is the difference between the

number of source lines and the number of comment lines.

Executable statements This Is the number of FORTRAN executable statements that appear in the program.

Cyclomatic complexity Cyclomatic complexity is defined as

being the number of partitions of the space in a module°s

control-flow graph. For programs with unique entry and exit

nodes, this metric is equivalent to one plus the number of decisions and in this work, is equal to the one plus sum of the following constructs: logical If's, if-then-else's, block-lf*s,

block if-then-else's, do loops, whale loops, AND*s, OR's, XOR's,

EQV's, NEQV's, twice the number of arithmetic if*s, n - I decision counts for a computed Eoto with n statement labels and n

4-12

l

I

I

I

I

I

I

I

I

I

I

I

I

l

I

I

I

I

I
decision counts for a case if with n predicates.

A variation on this definition excludes the counts of AND's,

OR*s, XOR*s, EQV*s and _EQV's (later referred to as

Cyclo_cmplx_2).

Calls This is the number of subroutine and function invocations in the module.

Calls and _ This is the total

decisions as they are defined above.

number of calls and

Revisions This is the number of versions

that are generated in the program library.

of the module

Changes This is the total number of changes to the system

that affected this module. Changes are classified into the followlng types (a single change can be of more than one type)_

a. error correction

b. planned enhancement

c. implement requirements change

d. improve clarity

e. improve user service

f. debug statement insertion/deletlon

g. optimization

h. adapt to environment change

i. other

Weighted changes This is a measure of the total amount of

effort spent making changes to the module. A programmer reports

the amount of effort to actually implement a given change by

4-13
indicating either

a. less than one hour,

b. one hour to a day,

e. one day to three days or

d. over three days.

The respective means of these durations,

hours, are divided

change. The sum of

involving a given

module.

0.5, 4.5, 16 and 32

equally among all modules affected by the

these effort portions over all changes

module defines the weighted changes for the

Errors This is the total number of errors reported by programmers; i.e., the number of system changes that listed this

module as involved in an error correction. (See the footnote at

the bottom of page q regarding the usage of the term "error".)

Weighted errors This is a measure of the total amount of

effort spent isolating and fixing errors in a module. For error

corrections, a programmer also reports the amount of effort spent

Isolating the error by indicating either

a. less than one hour,

b. one hour to one day,

c. more than one day or

d. never found.

The representative amounts of time for these durations, 0._, 4.5,

16 and 32 hours, are combined with the effort to implement the

correction (as calculated earller) and divided equally among the

modules changed. The sum of these effort portions over all error

corrections involving a given module defines the weighted errors

for the module.

4-14

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I
I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

IV. Internal Validation of the Software Science Metrics

The purpose of this section is to briefly define the

Software Science metrics, to see how these metrics relate to

standard program measures and to determine if the metrics are

internally consistent. That is, Software Science hypothesizes

that certain estimators of the basic parameters, such as program

length N and program level L, can be approximated by formulas

written totally in terms of the number of unique operators and

operands. Initially, an attempt is made to find correlations

between various definitions of these quantities based on the

interpretations of operators and operands given in the previous

section. Then, the family of metrics that Software Science proposes is correlated with traditional measures of software.

Program length Program length N is defined as the sum of

the total number of operators NI and the total number of operands

_2; i.e., N = NI + N2. Software Science hypothesizes that this

can be approximated by an estimator N* that is a function of the

vocabulary, defined as

N M = nllog2(nl) ÷ n21og2(n2).

The scatter plot appearing in Figure 1 and Pearson correlation

coefficient of .899 (p < .001; 179q modules)" show the relationship between N and N" (polynomial regression rejects including a

second degree term at p = .05). Several sources [12], [16],

[26], [27] have observed that the length estimator tends to be

" The symbol p will be used to stand for significance level.

4-15

I
high for small programs and low for large programs. The correlations and significance levels for the pairwise Wilooxon statlstlo

[20], broken down by exeoutable statements and length, are

displayed in Table 1. In our environment, either measure of size

demonstrates that N* signifioantly overestimates N in the first

and seoond quartileS and underestimates it (most significantly)

in the fourth quartile. Feuer and Fowlkes [15] assert that the

aoouraoy of the relation between the natural logarithms of

estimated and observed length changes less with program size. The

soatter plot appearing in Figure 2 and ooPrelation ooeffioient

for In S vs. in _" of .g27 (p < .0011 179q modules) show moderate

improvement.

<< Figure I >>

Table 1. Observed vs. estimated length broken down by program siz__._ee.

_. N vs. H" broken down by exeoutable statments.

XQT STMTS MOD3 R" ESTIMATION WILCOXON 3IGNIF

0 - 19 _6 .601 over <<.0001

20 - 40 4_2 .511 over <<,0001

_1 - T8 q5T .478 under ,0367

79 <= _qg .751 under <<.0001

_. N vs. N* broken down by N.

Length N MODS R- ESTIMATION WILCOXO_ SIGNIF

0 - 114 449 .750 over <<.0001

115 - 2_3 445 ._47 over <<.0001

24_ - 512 _53 .348 under .0010

513 <= 447 .731 under <<.0001

- (p < .OOl)

<< Figure 2 >>

4-16

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I
I

I

I

I

I

I

l

I

I

I

I

I

I

I

I

I

I

I

Prqgram volume A program volume metric V defined as N

log2 n represents the size of an implementation, which can be

thought of as the number of bits necessary to express it. The

potential volume V t of an algorithm reflects the minimum

representation of that algorithm in a language where the required

operation is already defined or implemented. The parameter V I is

a function of the number of input and output arguments of the

algorithm and is meant to be a measure of its specification. The

metric V" is defined as

V m = (2 ÷ n2 m) log2 (2 ÷ n2m).

The correlation coefficient for V vs. V m of .670 (p < .001;

modules) shows a reasonable relationship

necessary volume and its specification.

1794

between a program's

Program level The program level L for an algorithm is

defined as the ratio of its potential volume to the size of its

implementation, expressed as

L ffi Ve/V.

Thus, the highest level for an algorithm is its program specification and there L has value unity. The larger the size of the

required implementation V, the lower the program level of the

implementation. Since L requires the calculation of V e, which is

not always readily obtainable, Software Science hypothesizes that

L can be approximated by

2 n2

nl N2

4-17

I
The correlation for L vs. L* of .531 (p < .001; 1794

modules) is disappointingly below that of .90 given in [19].

Roping for an increase in the correlations, the modules are partitioned by the number of executable statements in Table 2.

Although the upper quartiles show measured improvement over the

correlation of the whole sample, a more interesting relationship

surfaces. The level estimator significantly underestimates the

program level in the second, third and fourth quartiles, with the

hypothesis being rejected in the first quartile. The increase in

magnitude of the n2 t parameter does not appear to be totally captured by the definition of L'.

Table 2. Relationship of observed vs. estimated program level

broken down by program size.

XQT STMTS MODS R" ESTIMATION WILCOXON SIGNIF

0 - 19 _46 .484 ....

20 - 40 4_2 .672 under <<.0001

_1 - 78 _57 .597 under <<.0001

79 <= _49 .615 under <<.0001

all 1794 .531 under <<.0001

- (p < .001)

the

gram level L have an inverse relationship; D is expressed

D = 1/L •

An alternate interpretation of difficulty defines it

inverse of L M, given by

Program difficult 7 The program difficulty D is defined as

difficulty of coding an algorithm. The metric D and the proas the

4-18

I

I

I

I

I

i

I

I

I

I

I

I

I

I

I

I

I

I
I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

1 nlN2

D2 = --- = ......

L" 2 n2

Christensen, Fitsos and Smith [12] demonstrate that the unique

operator count nl tends to remain relatively constant with

respect to length for q90 PL/S programs. They propose that the

average operand usage N2/n2 is the main contributor to the program diffleulty D2. The scatter plot appearing in Figure 3 and

Pearson correlation ooefflelent of .729 (p < .001; 1794 modules)

display the relationship between N2/n2 and D2 for our FORTRAN

modules. The application of polynomial regression brings in a

second degree term (p < .001) and results in a oorrelation of

.738.

<< Figure 3 >>

However, after observing in Figure q that nl varies with program

size, it seems as if the n1"s inflation might possibly better

explain D2. The scatter plot appearing in Figure 5 and the

correlation of .865 (p < .001; 179q modules) show the relationship of D2 vs. nl. Step-wise polynomial regression brings in a

second degree term initially, followed by a linear term (p <

.001), and results in a correlation of .879. In our environment,

the unique operator count nl explains a greater proportion of the

variance of the difficulty D2 than the average operand usage

N2/n2.

<< Figure q >>

4-19

I
<< Figure 5 >>

Program effort The Software Science effort metric E

attempts to quantify the effort required to comprehend the implementation of an algorithm. It is defined as the ratio of the

volume of an implementation to its level, expressed as

V (v)e"2

L V e

The E metric increases for programs implemented with large

volumes or written at low program levels; that is, it varies with

the square of the volume. An approximation to E can be obtained

without the knowledge of the potential volume by substituting L M

for L in the above equation. The metric

V nl N2 V nl N2 N lo,g2 n

E M

L" 2 n2 2 n2

defines the product of one half the number of unique operators,

the average operand usage and the volume. In an attempt to

remove the effect of possible program impurities [9], [19], N _ is

substituted for H in the above equation, yielding

H" log2 n

L _

nl N2 (nllog2nl + n21og2n2) log2 n

2 n2

The correlation coefficients for E vs. E', E vs. E" , in E vs. In

E _ and In E vs. In E _M are given in Table 3a. A fit of a least

squares regression line to the log-log plot of E vs. E" produces

4-20

i

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I
I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

the equation

Equivalently,

in E = .830eln E: ÷ 1.357 •

E = exp(1.357) " (E')''0.830 .

Due to this non-linear relationship and the improved correlation

of In E vs. in E', the modules are partitioned by executable

statements in Table 3b. The application of polynomial regression

confirms this non-linearity by bringing in a second degree term

(p < .001), resulting in a correlation of .698. In Table 3b,

notice that the correlations seem substantially better for

modules below median size. The significant overestimation in the

upper three quartiles attributes to the relationship of L and L*

described earlier.

Table 3. Observed vs. estimated Software Science E metric.

5" Pearson Correlation (E < .00___!1;1794 modules).

R

E vs. E _ .663

In E vs. In E* .931

E vs. E'* .603

In E vs. in E "* .890

_. E vs. E" broken down by executable statements.

XQT STMTS MODS R- ESTIMATION WILCOXON SIGNIF

0 - 19 446 .708 under .0050

20 - 40 442 .709 over <<.0001

41 - 78 457 .qll over <<.0001

79 <= 449 .550 over <<.0001

" (p < .001)

Program _ Software Science defines the bugs metric B as

the total number of "delivered" bugs in a given implementation.

Not to be confused with user acceptance testing, the metric B is

4-21

I
the number

pletion of a

expressed by

of inherent errors in a system component at the comdistinct phase in its development. Bugs B is

B z

E V

Eo Eo

where Eo is theoretically equivalent to the mean number of elementary discrimlnations between potential errors in programming.

Through a calculation that employs the definitions of E, L and

lambda (lambda = LV m is referred to as the language level), this

equation becomes

(lambda)''I/3 (E)mt2/3

Eo

The derivation determines an Eo value

(lambda)lel/3 "= I and obtains

of 3000, assumes

(E)_m2/3

B _ = •

3000

The correlation for B vs. B M is •789 (p < .001; 179_ modules).

In summary, the relationship of some _f the Software Science

metrics with their estimators seems to be program size dependent.

Several observations lead to the result that the metric N" signiFicantly overestimates N for modules below _he median size and

underestimates for those above the median size• The level estimator L" seems to have a moderate correlation with L, and its sig4-22

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I
I

I

I

I

I

I

I

l

I

I

I

I

I

I

I

I

I

I

nificant underestimation of L in the upper three quartiles

reflects its failure to capture the magnitude of n2 m in the

larger modules. With respect to the g metric, the effort estimator E" correlates better over the whole sample than g "A, and

their strongest correlations are for modules below median size.

The estimator g A shows a non-linear relationship to the effort

metric g. The correlation of In g vs. in g" significantly

improves over that of g vs. g *, with the E A metric's overestimation of g for larger modules attributing to the role of L A in its

definition. With the above family of metrics, Software Science

attempts to quantify size and complexity related concepts that

have traditionally been described by a more fundamental set of

measures.

Table 4 displays the correlations of the Software Science

metrics with the classical program measures of source lines of

code, cyclomatic complexity, etc. There are several observations

worth noting. Length M and volume V have remarkably similar

correlations and correspond quite well with most of the program

measures. Several of the metrics correlate well with the number

of executable statements, especially the program "size" metrics

of MI, H2, M and V (also B). The level estimator L A and its

inverse D2 seem to be much more related to the standard size and

complexity measures than their counterparts L and DI. The

language level lambda does not seem to show a significant relationship to the standard size and complexity measures, as

expected. The g AA metric relates best with the number of execut4-23

I
able statements and the modified cyclomatic complexity, while

correlating with all the measures better than the g metric and

slightly better than g*. Mone of the Software $cienoe measures

correlate especially well with the number of revisions or the sum

Table 4. Comparison of Software Science metrics against more

traditional software measures.

Key: ?

a

otherwise

not significant at .05 level

significant at .05 level

significant at .01 level

significant at .001 level

Source_Lines Source-Cmmts Cyclo_cmplx_2

I I I I

I Execut_Stmtsl Cyclo cmplx I Revisions I

I I I I

Calls &_Jumps

Calls

nl .776 .854 .778 .796 .818 .361 .802 .542

n2 .852 .867 .853. .767 .774 .430 .809 .614

N1 .824 .964 .868 .881 .889 .328 .869 .552

_2 .826 .9_9 .871 .858 .870 .355 .870 .597

n2 _ .792 .691 .754 .635 .629 .501 .683 .541

II

V

Vl

L

• 829 .961 .873 .874 .884 .343 .874 .571

• 864 .897 .864 .800 .811 ._20 .836 .621

• 837 .962 .875 .873 .883 .343 .876 .58_

• 776 .677 .734 .618 .611 .q85 .66_ .525

-.098 -.179 -.112 -.170 -.173 ? -.158 -.083

_

DI=I/L

D2=I/L *

N2/n2

Lambda

-.383 -._11 -.39_ -.389 -.396 -.216 -.386 -.250

.067a .2_ .113 .178 .196 -.093 .134 ?

.696 .872 .7_5 .816 .839 .269 .791 ._78

• 365 .5_4 .437 .508 .517 .106 ._70 .2_1

• 136 ? .108 ? ? .134 ? .051 n

E

E"

B "

B"

• 439 .629 .500 .535 .556 .106 .506 .282

.663 .831 .711 .771 .797 .224 .748 .452

• 738 .871 .760 .799 .829 .268 .788 .501

• 831 .962 .875 .873 .883 .3_3 .876 .58_

• 5_6 .7_9 .610 .650 .670 .1_9 .620 .355

" B and V will have identical correlations since they are linear

functions of one another.

4-24

I

I

I

I

i

I

I

I

I

I

I

I

I

I

I

I

I

I

I
I

I

I

I

I

I

I

i

I

i

I

I

I

I

i

I

!

I

of procedure and function calls. The primary measures of unique

operators nl and unique operands n2 correspond reasonably well

overall with n2 being stronger with source lines and nl stronger

with the cyclomatic complexities. In the next section, an

analysis attempts to determine the relationship that these parameters really have with the quantities that they theoretically

describe.

V. External Validation of the Software Science and Related Metrics

The purpose of this section is to determine how well the

Software Science metrics and various complexity measures relate

to actual effort and errors encountered during the development of

software in a commercial environment. These objective product

metrics are compared against more primitive volume metrics, such

as lines of source code. The reservoir of development data

includes the monitoring of several projects and the analysis

examines several projects at once, individual projects and individual programmers across projects. To remove the dependency of

the distribution of the correlation coefficient on the actual

measures of effort and errors, the nonparametric Spearman rank

order correlation coefficients are examined in this section [22].

(The ability of a few data points to

deflate the Pearson product-moment

well recognized.) The analysis first

artificially inflate or

correlation coefficient is

examines how well these

measures correspond to the total effort spent in the development

of software.

4-25

I
A. Metrics" Relation to Actual Effort

Initially, a correlation across seven projects of the

Software Science E metric vs. actual effort, on a module by

module basisusing only those that are newly developed, produces

the results in Table 5. The table also displays the correlations

of some of the more standard volume metrics with actual effort.

These disappointingly low correlations create a fear that there

Table 5. Spearman rank order correlations Rs with effort for

all modules (73__!) fro____mall projects.

Key: ?

a

otherwise

not significant at .05 level

significant at .05 level

significant at .01 level

significant at .001 level

E .345

E" .445

E'" .488

Cyclo_cmplx .463

Cyclo_cmplx_2 ._67

Calls ._14

Calls_&_Jumps .494

D;=I/L .126

D2=I/L" .417

Source Lines .522

Execut Stmts .456

Source-Cmmts .460

V .448

.434

eta1 .485

eta2 .461

B .448

B" .3_5

Revisions .531

Changes .469

Weighted_Chg .468

Errors .220

Weighted Err .226

4-26
I

I

I

I

I

I

I

t

I

i

I

i

|

i

I

i

I

I

may be some modules with poor effort reporting skewing the

analysis. Since there is partial redundancy built into the effort

data collection process, there exists hope of validatinE the

effort data.

Validation of effort data The partial redundancy in the

development monitoring process is that both managers and programmers submit effort data. Individual programmers record time spent

on each module, partitioned by design, code, test and support

phases, on a weekly basis with a Component Status Report (CSR).

Managers record the amount of time every programmer spends working each week on the project they are supervising with a Resource

Summary Form (RSF). Since the latter form possesses the enforcement associated with the distribution of financial resources, it

is considered more accurate [24]. However, the Resource Summary

Form does not break effort down by module, and thus a combination

of the two forms has to be used.

Three different possible effort reporting validity checks

are proposed. All employ the idea of selecting programmers that

tend to be good effort reporters, and then using Just the modules

that only they worked on in the metric analysis. The three proposed effort reporting validity checks are:

a. Vm -

number of weekly CSR's submitted by programmer

number of weeks programmer appears on RSF's

4-27

i
b. Vt -

sum of all man-hours reported by programmer on all CSR's

sum of all man-hours reported for programmer on all RSF's

e. Vi = 1 -

number of weeks programmer's CSR effort > RSF effort

total number of weeks programmer active in project

The first validity proposal attempts to capture the frequency of

the programmer's effort reporting. It checks for massing data by

ranking the programmers according to the ratio Vm of the number

of Component Status Reports submitted over the number of weeks

that the programmer appears on Resource Summary Forms. The second

validity proposal attempts to capture the total percentage of

effort reported by the programmer. This proposal ranks the program_ers according to the ratio Vt formed by the sum of all the

man-hours reported on Component Status Reports over the sum of

all hours delegated to him on Resource Summary Forms.

Note that for a given week, the amount of tame reported on a

Component Status Report should be always less than or equal to

the amount of time reported on the corresponding Resource Summary

Form. This is not because the programmer fails to "cover" himself, but a consequence of the management's encouragement for

programmers to realisticly allocate their time rather than to

guess in an ad hoc manner. This observation defines a third valldity proposal to attempt t_ capture the frequency of a

programmer's reporting of inflated effort. This data check ranks

4-28
I

l

I

I

I

I

I

i

I

I

I

i

I

I

I

I

I

I

the programmers according to the quantity Vi equal to one minus

the ratio of the number of weeks that CSR effort reported

exceeded RSF effort over the total number of weeks that the programmer is active in the project.

Metrics" relation to validated effort data Of the given

proposals, the systems development head of the institution where

the software is being developed suggests that the first proposal,

the missing data check, would be a good initial attempt to select

modules with accurate effort reporting [24]. The missing data

ratios Vm are defined for programmers on a project by project

basis. Table 6 displays the effort correlations of the newly

developed modules worked on by only programmers with Ym >: 90_

from all projects, those with Vm

developed modules. Most of the

included in the Vm >= 90% level seem

>: 80_ and for all newly

correlations of the modules

to show improvement over

those at the Vm >: 80_ level. Although this is the desired effect

and several of the Vm >= 90_ correlations increase over the original values, a majority of the correlations with modules at the

Vm >= 80_ level are actually lower than their original coefficients. Since the effect of the ratio's screening of the data is

inconsistent and the overall magnitudes of the correlations are

low, the analysis now examines modules from different projects

separately.

4-29

i
Table 6. Spearman ran..___korde_____ correlations Rs with effort for modules

across seven projects wit_ various validity levels.

Key: ?

a

otherwise

not significant at .05 level

slKnlflcant at .05 ievel

siEnlficant at .01 level

slgnlfioant at .001 level

Validity ratio Vm (#mods)

a11(731) 80_(398) 905(215)

E .3a5 .307 .3fi7

E" ._5 .422 .467

E'* ._88 .480 .513

Cyclo_omplx ._63 ._57 ._79

Cyolo_cmplx_2 ._67 ._5_ .506

Calls ._1_ .360 ._02

Calls_&_Jumps .qgq ._75 ._79

DI=I/L .126 .0881 ?

D2=I/L" ._17 .371 ._21

Source_Lines .522 .519 .501

Exeeut_Stmts ._56 ._29 ._75

Souroe-Cmmts ._60 ._20 ._39

._q8 ._3_ ._75

M ._3_ ._16 ..60

etal ._85 .462 .493

eta2 .q61 .467 .503

B ._a8 .q3q ._75

B* .345 .307 .357

Revisions .531 .580 .565

Chanses ._69 ._95 .385

Weighted Oh8 .q68 .521 .q62

Errors .220 .381 .205

Weighted_Err .226 .382 .247

The Spearman correlations of the various metrics with effort

for three of the individual projects appear in Table 7.

4-30

I

I

i

I

1

1

I

I

i

I

l

I

I

I

I

I

I

/
|

!

I

I

i

i

I

i

I

I

I

i

|

I

!

i

i

I

Table _. Spearman rank order oorrelations Rs with effort for

various validity rankinEs of modules from individual

prqJects $I, S3 and S7.

Key: ?

a

otherwise

Z

not significant at .05 level

significant at .05 level

significant at .01 level

significant at .001 level

unavailable data

Project

Sl S3"

Validity ratio

Vm all 80_ 90_ 80_ 90_

#modules 79 29 20 132 81

$7"-

all 80_

127 49

E .613 .647 .726 .469 .419 .285 .409a

E" .665 .713 .746 .602 .585 .389 .569

E'" .700 .747 .798 .638 .640 .430 .567

Cyclo_cmplx .757 .774 .792 .583 .608 .463 .523

Cyclo_cmplx_2 .764 .785 .787 .609 .664 .491 .523

Calls .681 .698 .818 .442 .492 .404 .485

Calls &_Jumps .776 .813 .822 .594 .619 .488 .569

DI=I/L .262a ? ? .156 • ? ? ?

DZfl/L" .625 .681 .745 .507 .442 .377 .499

Source Lines .686 .672 .729 .743 .734 .486 .499

Execut Stmts .688 .709 .781 .609 .594 .408 .515

Source_Cmmts .670 .710 .778 .671 .654 .416 .471

V .657 .692 .774 .627 .637 .377 .497

• 653 .680 .755 .613 .619 .360 .484

eta1 .683 .740 .848 .553 .533 .439 .431

eta2 .667 .701 .747 .643 .698 .365 .445

B .657 .692 .774 .627 .637 .377 .497

B" .613 .643 .726 .469 .419 .285 .409a

Revisions .677 .717 .804 .655 .632 .449 .510

Changes .687 .645 .760 .672 .639 .238a .380a

Weighted_Chg .685 .629 .749 .673 .649 .238a .256 •

Errors z z z .644 .611 .253a .438

Weighted Err z z z .615 .605 .245a .2761

- All modules in project S3 were developed by programmers

with Vm >= 80_.

"- There exist fewer than a significant number of modules developed

by programmers with Vm >= 90_.

4-31

I
Although the correlation coefficients vary considerably between

and among the projects, the overall improvement in projects $1

and $3 is- apparent. Almost every metric's correlation with

development effort increases with the more reliable data in proJects $I and $7. When comparing the strongest correlations from

the seven individual projects, neither Software Science°s E

metrics, cyolomatic complexity nor source lines of code relates

convincingly better with effort than the others. Note that the

estimators of the Software Science E metric, E * and E **, appear

to show a stronger relationship to actual effort than E.

The validity screening process substantially improves the

correlations for some projects, but not all. This observation

points toward the existence of

interactions. In an attempt

effects, the analysis focuses on

project dependent factors and

to minimize these intraproJect

individual programmers across

projects.

programmer differences have a large effect on

many individuals contribute to a project.

mers

Note that Basili and Hutchens [2] also suggest that

the results when

The use of modules developed solely by individual programsignificantly reduces the number of available data points

because of the team nature of commercial work. Fortunately, however, there are five programmers who totally developed at least

fifteen modules each. The correlations for all modules developed

by them and their values of the three proposed validity ratios

are given in Table 8. The order of increasing correlation coefficients for a particular metric can be related to the order of

4-32
I

I

|

I

I

I

I

I

!

!

!

!

t

I

!

!

ii

!

t

Table _. Spearman rank orderr correlations Rs with effort for modules

_ by. fiv____e individ--ua_-p-rprogrammers--_-

Key: ?

e

a

otherwise

not significant at .05 level

significant at .05 level

significant at .01 level

significant at .001 level

Programmer (#roods)

PI(31) P2(17) P3(21) P"(2_) P5(15)

E .593 ? ? .561a ?

E" .718 .526* .375* .555a .507*

E'" .789 .570a ? .539a .511m

Cyclo_cmplz .592 ._69 e .521a .565a ?

Cyclo cmplz_2 .68_ .583a ._811 .5_6a ?

Calls .622 .787 ? .669 ?

Calls_&_Jumps .701 .60_a ._51 • .579a ?

DI=I/L .31_" ? ? ? ?

D2=I/L" .713 ._60 • ? ._97a ._67e

Source_Lines .863 .682 .605a .62_ ?

Execut Stmts .747 .5_Oe ._36 e .631 .53_"

3ource-Cmmts .826 .576a .530a .612 .509"

V .718 "5_oe -453 e .579a .451,

N .676 .526e -_61 e .556a ._71"

eta1 .811 .575a ? .536a ?

eta2 .765 .701 .527a .597 ?

B- .718 .5_Oe ._53 • .579a

B" .593 ? ? .561a

Revisions .675 .523e .777 .468e

Changes .412 e ._68e .600a ?

Welghted_Chg .q28a .527e .502a ?

Errors .386e ? .668 ?

Weighted_Err .342 • ? .62_ ?

._51"

?

?

?

?

.596a

.545"

_ALIDITT _ATIO$ (%)

Vm

Yt

Vl

Ave. Vm,Vt

Ave. Vm,Vi

92.5 96.0 87.7 83.9 7;.1

97.9 91.8 98.8 82.1 7_.1

78.6 69.5 77.6 80.0 87.5

95.2 93.9 93.25 83.0 74.1

85.5 82.75 82.65 81.95 80.8

4-33
increasing values for a Eiven validity ratio using the Spearman

rank order correlation. The significance levels of theserank

order oorrelatlons for several of the metrics appear in Table 9.

The statistlcally significant correspondence between the programmers" validity ratios Vm and the correlation coefficlents Justifies the use of the ratio Vm in the earlier analysls; possible

improvement is sugEested if Vm were combined with either of the

ether two ratios.

Table 9. Significance level____!sfor the Spearman rank order correlation

between the programmer's validit_ ratios and the correlati

coefficients for several of the metrics.

Batlo

Metric Vm Vt Vi Ave(Vm,Vt) Ave(Vm,Vi) Ave(Vt

g'" .09 .09

Cyolo_cmplx

Cyalo_emplx_2 .05 .02 .02

Calls_&_Jumps .05 .02 .02

Source_Lines .05 .02 .02

Source-Cmmts .09 .09

V (B) .09 .09

eta2 .05 .02 .02

Revisions .001 .09" .09 .09

.05

" Hegative oorrelatlon.

In summary, the strongest sets of correlations occur between

the metrics and actual effort for certain validated projects and

for modules totally developed by individual programmers. While

relationships across all projects uslng both all modules and only

validated modules produce only fair coefficients, the validation

process shows patterns of improvement. Applyin_ the validity

4-34

|

I

I

I

!

i

I

!

!

!

!

i

J

I

I

i

|

i

I
ratio screening to individual projects seems to filter out some

of the project specific interactions while not affecting others,

with the correlations improving accordingly. Two averages of the

validity ratios (Vm with Vt a.d Vm with Vi) impose a ranking on

the indlvidual programmers that statlstlcally agrees with an ordering of the improvement of several of the oorrelatlons. In all

sectors of the analysis, the ineluslon of L* in the Software Solenoe g metric in its estimators E" and E'" seems to improve the

metric correlations with actual effort. The analysis now attempts

to see how well these metrics relate to the number of errors

encountered during the development of software.

B. Metric's Relation to Errors

This section attempts to determine the correspondence of the

Software Science and related metrics both to the number of

development errors and to the weighted sum of effort required to

isolate and fix the errors. A correlation across all projects of

the Software Science bugs metric B and some of the standard

volume and complexity metrics with errors and weighted errors,

using only newly developed modules, produces the results in Table

10. Most of the correlations are very weak, with the exception

of system changes. These disappointingly low correlations attribute to the discrete nature of error reporting and that 340 of

the 652 modules (52_) have zero reported errors. Even though

these correlations show little or no correspondence, the following observations indicate potential improvement.

4-35

!
Table 10. Spearman rank order correlations Rs with errors and

weighted-ennors fen all modules (65__2) fnom six pnoJects.

Key: ?

a

otherwise

not slgnifiQant at .05 level

signlfloant at .05 level

signifloant at .01 level

signifloant at .001 level

Errors Weighted_err. i

B .083 _ .101a

E" .151 .171

E'" 163 186

cyolo_o.pIx _196 _2o5 III

Cyelo_emplx_2 .189 .200

Calls .220 .236

Calls_&_Jumps .235 .2_8

DI=I/L ? ?

D2ffil/L" .124 .1_0

Source_Lines .255 .265

Exeout_Stmts .177 .198

3ouree-Cmmts .288 .298

Y .168 .186

H .162 .180

eta1 .102a .132

eta2 .181 .199

i

l

l

i

B .168 .186

B" .083" .101a

Revisions .375 .375

Changes .677 .636

Weighted Chg .627 .677

I

!

Design Elf .219 .185

Code_Elf .285 .316

Test_Elf .1_9 .16_

Tot_EffoPt .324 .332

- ProJect 31 has no data to distinguish errors from changes.

!

I

I

Weiss [q], [5] conducted an extensive error analysis that

involred three of the projects and employed enforcement of error

reporting through programmer interviews and hand-checks. For two

4--36

I

I

I

!
!

!

!

1

|

!

I

I

I

of the more recent projects, independent validation and verification was performed. In addition, the on-site systems development

head asserts that due to the maturity of the collection environment, the accuracy of the error reporting is more reliable for

the more recent projects [24]. These developmental differences

provide the motivation for an examination of the relationships on

an individual project basis.

Table I; displays the attributes of the projects and the

correlations of all the metrics vs. errors and weighted errors

for three of the individual projects. The correlations in 37, a

project involved in the Weiss study, are fair but better than

those of project $5 (not shown) that was developed at about the

same time. Project Sq and $6 (also not shown) have very poor

overall correlations and unreasonably low relationships of revisions with errors, which point to the effect of being early proJects in the collection effort. The trend

produce is not very apparent, although

reporting enforcement do seem to have some

that the attributes

chronology and error

effect. In another

attempt to improve the correlations, the analysis applies the

Table 11. Spearman rank order correlations R s with errors and

weighted-errors for modules from three individual

projects.

Key: ? not significant at .05 level

• significant at .05 level

a significant at .01 level

otherwise significant at .001 level

Err errors

W err weighted-errors

4-37

!
Project (#Nods)

S3(132) 34(35)

Err W err Err W err

E .401 .378

E" .536 .482

E'" .579 .522

Cyclo_cmplx .542 .481

Cyclo_omplx_2 .553 .489

Calls .445 .432

Calls_&_Jumps .566 .518

DI=I/L ? ?

D2=I/L* .491 .426

S7(127)

Err W err

7 7 .397 .391

? ? .507 .503

? 7 .492 .505

7 ? .393 .368

? ? .405 .400

.300 • .316 • .423 .419

? ? .432 .412

? ? .168 m .178 t

? ? .563 .559

Source_Lines .648 .622

Exeout Stmts .538 .505

Source=Cmmts .599 .568

V .541 .495

.526 .480

eta1 .550 .500

eta2 .541 .500

.339 n ? .490 .487

? ? .478 .465

? ? .501 .483

? ? .461 .456

? ? .457 .4_9

? ? .488 .522

? ? .3_8 .367

B .5qi .495

B" .401 .378

Revisions .784 .694

Changes .939 .864

Weighted_Chg .8_0 .885

? ? .461 .456

? ? .396 .390

.686 .630 .567 .500

• 770 .761 .727 .670

.661 .757 .62q .714

Design_Elf ? ?

Code Elf .620 .632

Test Eft .473 .481

Tot Effort .6_4 .615

? ? ? ?

.413a .398a .274 .264

.312" ? ? ?

.455a ._47a .253a .245a

PROJECT ATTRIBUTES

Weiss s_udy

IV & V X

Chronology recent

X X

early middle

previous section's hypothesis of focusing on individual programmers. Table 12 gives the correlations of the metrics with errors

and weighted errors for modules that two of the individual programmers totally developed. Even though it is encouraging to see

4-38
I

i

I

i

I

i

i

I

I

I

i

I

I

I

Table 12. Spearman rank order correlatlons Rs with errors and

weighted-errors for modules totall_ developed by tw_._So

individual _rogrammers.

Key: ?

a

otherwise

not significant at .05 level

significant at .05 level

significant at .01 level

significant at .001 level

Err

W err

errors

weighted-errors

Programmer (#roods)

P2(17) P3(21)

Err W err Err W err

E .514t .447 m

E" .527 e .493*

E'" .515 u .473 •

Cyclo_cmplx .575a .558a

Cyclo_cmplx_2 .661a .616a

Calls ? .498a

Calls_&_Jumps .545" .560a

DI=I/L ? ?

D2=I/L* .558a .526"

.368e ?

.600a .563a

.666 .649

.463e .428t

.484e ._49 n

.506a .469 •

• 598a .557a

? ?

.4591 .429 •

Source Lines ? ?

Execut Stats .624a .577a

Source-Cmmts ? .436 m

V .491" .472 e

M ._94" .479 m

etal .497" .448_

eta2 ? ?

.662 .646

• 579a .533a

• 635 .594a

• 679 .655

.641 .610a

.611a .589a

• 715 .717

S .491e .472*

B" .514" .4_7 e

Revisions ? ?

Changes .716 .662a

Weighted_Chg ? .510 •

• 679 .655

.368t ?

.830 .811

• 855 .828

.863 .861

Design_Eft ? ?

Code Elf ? ._50 e

Test Elf ? ?

Tot Effort ? ?

m

._60 m .392 •

• 699 .667

.668 .644

.668 .624

4-39

i
the correspondences of the metrics B, E "" and eta2 with errors as

among the best for programmer P3, the same metrics do not relate

as well for other programmers.

In summary, partitioning an error analysis by individual

project or programmer shows improved correlations with the various metrics. Strong relationships seem to depend on the Indivldual programmer, while few high correlations show up on a project

wide basis. The correlations for the projects reflect the positive effects of reporting enforcement and collection process

maturity. Overall, the correlations with total errors are

slightly higher than those with weighted errors, while the number

of revisions appears to relate the best.

VI. Conclusions

In the Software Engineering Laboratory, the Software Science

metrlos, cyclomatic complexity and various traditional program

measures have been analyzed for their relation to effort,

development errors and one another. The major results of this

investigation are the followlng: I) _one of the metrics examined

seem to manifest a satisfactory explanation of effort spent

developing software or the errors incurred during that process;

2) neither Software Science's E metric, cyclomatic complexity nor

source lines of code relates convincingly better with effort than

the others; 3) the strongest effort correlations are derived when

modules obtained from individual programmers or certain validated

projects are considered; 4) the majority of the effort correla4-40

I

i

I

i

!

I

!

i

i

i

i

i

I

I

I

I

I

I

I
!

!

!

I

I

!

I

!

!

!

!

!

!

l

I

I

I

II

tions increase with the more reliable data; 5) the number of

revisions appears to correlate with development errors better

than either Software Science's B metric, E metric, cyclomatic

complexity or source lines of code; and 6) although some of the

Software Science metrics have size dependent properties with

their estimators, the metric family seems to possess reasonable

internal consistency. These and the other results of this study

contribute to the validation of software metrics proposed in the

literature. The validation process must continue before metrics

can be effectively used in the characterization and evaluation of

software and in the prediction of its attributes.

Acknowledgment

The authors are grateful to F. McGarry and B. Curtis for

their valuable comments on this analysis. We would also like to

thank B. Decker, W. Taylor and E. Edwards for their assistance

with the SAP program and the S.E.L. database.

[I]

[2]

[33

Bibliography

V. R. Basili, Tutorial on Models and Metrics for Software

Management and Engineering, IEEE Comput. Society, IEEE Catalog No. EHO-167-7, 1980.

V. R. Basili and D. H. Hutchens, "Analyzing a Syntactic Family of Complexity Metrics," Dept. Comput. Sci., Univ. of

Maryland, College Park, MD 20742, Tech. Rep. TR-I053, Dec.

1981 (to appear in T.S.E.).

V. R. Basili and T. Phillips, ,Evaluating and Comparing the

Software Metrics in the Software Engineering Laboratory,"

ACM Sigmetrics (1981 ACM Workshop/Symp. Measurement and

Evaluation of Software Quality), Vol. I0, pp. 95-106, Mar.

1981.

4-41

I
[4] V. R. Basili and D. M. Weiss, "A Methodology for Collecting

Valid Software Engineering Datam," Dept. Comput. Sci., Univ.

of Maryland, ColleEe Park, MD 20742, Tech. Rep. TR-1235,

Dec. 1982.

[5] V. R. Basill and D. M. Weiss, "Evaluating Software Development by Analysis of Changes: The Data from the Software

Engineering Laboratorye," Dept. Comput. Sci., Univ. of Maryland, ColleEe Park, MD 20742, Tech. Rep. TR-1238, Dec. 1982.

[6] V. R. Basili and M. V. Zelkowitz, "Analyzing

Software Developments," Proc. 3rd Int. Conf.

Atlanta, GA, May 1978, pp. 118-123.

Medium Scale

Software En_.,

[7] V.R. Basili and M. V. Zelkowitz, "Measuring Software

Development Characteristics in the Local Environment," Com______-

puters and Structures, Vol. 10, pp. 39-_3, 1979.

[8] V. R. Basili, M. V. Zelkowitz, F. E. McGarry, R. W. Reiter,

Jr., W. F. Truszkowski and D. L. Weiss, "The Software

Engineering Laboratory," Software Eng. Lab., NASA/Goddard

Space Flight Center, Greenbelt, MD 20771, Rep. SEL-77-O01,

May 1977.

[9] Bulut, Necdet and M. H. Halstead, "Impurities Found in Algorithm Implementations," ACM SIGPLAN Notices, Vol. 9, Mar.

1974.

[10] D. N. Card, F. E. McGarry, J. Page, S. Eslinger and V. R.

Basili, "The Software Engineering Laboratory," Software Eng.

Lab., NkSA/Goddard Space Flight Center, Greenbelt, MD 20771,

Rep. 3EL-81-104, Feb. 1982.

[11] E. T. Chen, "Program Complexity and Programmer Productivity," IEEE Trans. Software Eng., Vol. SE-4, pp. 187-194,

May 1978.

[12] K. Christensen, G. P. Fitsos and C. P. Smith, "A Perspective on Software Science," IBM Syst. _., Vol. a0, pp. 372-

387, 1981.

[13] B. Curtis, S. B. Sheppard and P. M. Milllman, "Third Time

Charm: Stronger Replication of the Ability of Software Complexity Metrics to Predict Programmer Performance," Proc.

4th Int. Conf. Software Eng., Sept 1979, PP. 356-360.

[lq] W. _ Decker and W. A. Taylor, "FORTRAN Static Source Code

Analyzer Program (SAP) User's Guide (Revision I)," Software

Eng. Lab., NASA/Goddard Space Flight Center, Greenbelt, MD

20771, Rep. SEL-78-I02, May 1982.

[15] A. R. Feuer and E. B. Fowlkes, "Some Results from an Empirical Study of Computer Software," Pro____c. _th Int. Conf.

4-42

I

I

I

I

I

t

I

I
!

! Software En&., Sept. 1979, pp. 351-355.

[16] G. P. Fitsos, "Vocabulary Effects in Software Science," IBM

Santa Teresa Lab., San Jose, CA 95150, Tech. Rep. TR 03.082,

Jan. 1980.

[17] J. g. Gaffney and G. L. Beller, "Macro Variable software

Models for Application to Improved Software Development

Management," __Pr°c- of Workshop on Quantitative Software

Models for Reliability, Complexity and Cost, IEEE Comput.

Society, 1980.

[18] S. A. Gloss-Soler, The DACS Glossary:

Software Engineering Terms, Data &

Software, Grlfflss Air Force Base, NY

Oct. 1979.

A Bibliography of

Analysis Center for

13_41, Rep. GLOS-1,

[19] M. H. Halstead, Elements of Software Sqience,

North- Holland, New York, 1977.

Elsevier

[20] R. V. Howe and E. A. Tanis, Probabilit 7 and Statlstical

Inference, MacMillian, New Tork, 1977, pp. 265-271.

[21] IEEE Standard Glossary of Software Engineering Terminology,

IEEE, 342 E. _Yth S_., New York, Rep. IEEE-STD-729-1983,

1983.

[22] M. Kendall and A. Stuart, The Advanced Theory of Statistics,

Vol. 2, 4th Ed., MacMillian, New York, 1979, pp. 503-508.

[23] T. O. McCabe, "A Complexity Measure," IEEE Trans.

En_.n_., Vol. SE-2, pp. 308-320, Dec. 197_.

Software

[24] F. g. McGarry, Systems Development Head, Code 582.1, NASA/

Goddard Space Flight Center, Greenbelt, MD 20771, personal

consultation, Jan.-July 1982.

[25] E. M. O'Neill, S. R. Wallgora and C. E. Goorevich, "FORTRAN

Static Source Code Analyzer (SAP) User's Guide," Software

Eng. Lab., NASA/Goddard Space Flight Center, Greenbelt, MD

20771, Rep. SEL-78-O02, Feb. 1978.

[26] V. Y. Shen and H. E. Dunssore, "A Software Science Analysis

of COBOL Programs," Dept. Comput. Sol., Purdue Univ., West

Lafayette, IN q7907, Tech. Rep. CSD-TR-348, August 1980.

[27] C. P. Smith, "A Software Science &nalysis of IBM Programming

Products," IBM Santa Teresa Lab., San Jose, CA 95150, Tech.

Rep. TR 03.081, Jan. 1980.

!

4-43

!
Monitoring Software Development

through Dynamic Variables

t

N87- 24901

ABSTRACT

This paper describes research conducted by the Software

Engineering Laboratory (SEL) on the use of dynamic variables as a

tool to monitor software development. The intent of the project

is to identify project independent measures which may be used in

a management tool for monitoring software development. This

study examines several FORTRAN projects with similar profiles.

The staff was experienced in developing these types of projects.

The projects developed serve similar functions. Because these

projects are similar we believe some underlying relationships

exist that are invariant between the projects. These relationships, once well defined, may be used to compare the development

of different projects to determine whether they are evolving the

same way previous projects in this environment evolved.

Authors:

Carl W. Doerflinger

University of Maryland

Dept. of Computer Science

College Park, MD 20742

(301) 454-4251

Victor R. Basili

University of Maryland

Dept. of Computer Science

College Park, MD 20742

(301) 454-2002

KEYWORDS

management tool, metric, measurement, predictive model

t

;_ECEDING PAGE 8L_NK NOT i_L_D

4-45

• __LTF.IW_ONALLI _I.ANg
Monitoring Software Development

through Dynamic Variables

by

Carl W. Doerflinger

and

Victor R. Basili

I. Overview

The Software Engineering Laboratory (SEL) is a Joint effort

between the National Aeronautics and Space Administration (NASA),

the Computer Sciences Corporation (CSC), and the University of

Maryland established to study the software development process.

To this end, data has been collected for the last six years. The

data was from attitude determination and control software

developed by CSC, in FORTRAN, for NASA. Additional information

on the SEL, the data collection effort, and some of the studies

that have been made may be found in papers from the Software

Engineering Laboratory Series published by the SEL [Card82],

[Church82], [SEL82].

The interest in the software development process is

motivated by a desire to predict costs and quality of projects

being planned and developed. For several years, studies have

examined the relationships between variables such as effort,

size, lines of code, and documentation [Walston77], [Basili81].

These studies, for the most part, used data collected at the end

of past projects to predict the behavior of similar projects in

the future, in 1981 the SEL concluded that many of these factors

4-46

!

!

i

i

!

i

!

!

i

I

I

I

!

I

I

I

!

i

I
were too dependent on the environment to be useful for the models

that had been developed [Bailey81]. Any model which attempts to

trace these relationships should therefore be calibrated to the

environment being examined. The meta-model proposed by the SEL

is designed for such flexibility [Bailey81].

Another way to isolate out the environment dependent factors

is by comparing two internal factors of a project, thus ignoring

all outside influences. One approach that is used to monitor

software development examines the time gap between the initial

report of software problems and the complete resolution of the

problem [Manley82]. Comparing two*variables is useful because it

also accentuates problem areas as they develop, providing relative information rather than absolute information. Relative

information is useful to the project manager because it accentuates trends as the project develops. If project environments are

similar, then similar values should be expected. Because the

project environments in the SEL are similar, it was felt that

this approach could be further extended to provide managers with

information about how a set of variables over the course of a

project differed from the same set of variables on other projects

(baselines). The managers could be alerted to potential problems

and use Other variable data and project knowledge to determine

whether the project was in trouble.

This methodology is flexible enough to respond to changing

needs. Every time a project is completed the measures collected

during its development may be added in to calculate a new

4-47
baseline. In this way, the baselines may adapt to any changes in

the environment, as they occur.

Baselines might also be developed to reflect different

attributes. For instance, several projects which had good productivity might be grouped to form a productivity baseline. Once

baselines are established, projects in progress may be compared

against them. All measures falling outside the predetermined

tolerance range are interpreted by the manager.

I__I. Methodology

The implementation of this methodology is dependent on two

factors. The first factor is the availability of measures that

are project independent and can also be collected throughout a

project's development. Variables like programmer hours and

number of computer runs are project dependent. By comparing

these variables against each other a set of relative measures may

be generated which is project independent. For instance, the

number of software changes may vary from project to project. The

project dependent features shared by each variable will cancel

out when the ratio of software changes per computer run is taken.

The resulting relative measure is project independent.

The second factor is the need for fixed time intervals common to all projects. To normalize for time, project milestones

were used. The time into a project might be twenty percent into

coding instead of ten weeks into the project, for instance.

4-48
!

!

a

i

11

!

II

II

II

i

II

It

I

i

l

g

D

ii

When computing the baselines one other factor was considered. At any given interval during development a variable may

measure either the total number of events that have occurred from

the beginning of development (cumulative) or the number of of

events that have occurred since the last measured interval

(discrete). Since these approaches may convey different information it was felt that they both should be used.

For simplicity, the baseline for each relative measure was

defined as the average and standard deviation computed for the

measure at predetermined intervals. A project's progress may now

be charted by the software manager. At each interval in a proJects development the relative measures are compared with their

respective

are flagged.

manager to

measure may indicate a project is developing exceptionally

or it may indicate a problem has been encountered.

baseline. Any measures outside a standard deviation

These measures are then interpreted by the project

determine how the project is progressing. A flagged

well

The interpretation of a set of flagged measures is a three

step process. First, the manager must determine the possible

interpretations for each flagged relative measure using lists of

possible interpretations developed and verified based on past

projects.

Second, the union of the lists of possible interpretations

of each flagged measure must be taken. The list formed by this

union contains all the possible interpretations ordered using the

4-49
number of times each interpretation is repeated in the different

lists. The larger the number of overlaps a possible interpretation has, the greater the probability it is the correct interpretation.

Third, the manager must analyze the combined list and determine if a problem exists. Interpretations with an equal number

of overlaps all have an equal probability of being the correct

interpretation. If none of the possible interpretations for a

given relative measure overlap then the relative measure should

be considered separately.

When analyzing the interpretations, three pieces of information must be considered; the measurements, the point in development, and the managers knowledge of the project. A relative

measure may indicate different things depending on the stage of

development. For instance, a large amount of computer time per

computer run early in the project may indicate not enough unit

testing is being done. Personal knowledge may also give valuable

insight.

A fundamental assumption for using this methodology is that

similar type projects evolve similarly. If a different type of

project was compared to this database, the manager would have to

decide whether the baselines were applicable. Depending on the

type of differences, the established baselines may or may not be

of any value.

4-50

I

!

i

I

I

I

I

I

i

i

I

I

I

I

I

!

I

I
1

i

I

i

I

I

I

I

I

I

1

I

I

I

I

I

I

-11

EXAMPLE I:

Forty percent into coding a software manager finds that the

lines of source code per software change is higher than normal.

A list previously developed is examined to determine what the

relative measure might indicate. The possible interpretations

for a large number of lines of source code per software change

might be:

- good code

- easily developed code

- influx of transported code

- near build or milestone date

- computer problems

- poor testing approach

If this were the only flagged measure the manager would then

investigate each of the possibilities. If the value for the

measure is close to the norm less concern is needed than if the

value is further away.

If in addition to lines of source code per software change

the number of computer runs per software change was higher than

normal, the manager would also examine this measure. The possible interpretations for a large number of computer runs per

software change might be:

- good code

- lots of testing

- change backlog

- poor testing approach

The union of the possible interpretations of these two measures

indicates that the strongest possible interpretations are I) good

code and 2) a poor testing approach. The number of possibilities

to investigate is smaller because these are the only measures

4-51
which overlap. The manager must now examine the testing plan and

decide whether either of these interpretations reflect what is

actually occurring in the project. If these two possible

interpretations do not reflect what is happening on the project,

the manager would then examine the other interpretations.

III. Baseline Development 1

To develop a baseline one must first have variables whose

measurements were taken weekly for several projects. Five variables in the SEL database were used. The lines of source code,

number of software changes, and number of computer runs were collected on the growth history form. The amount of computer time

and programmer hours were collected on the resource summary form.

Measurement of these variables started near the beginning of coding. In this study, nine separate projects were examined whose

development was documented, with sufficient data, in the SEL

database. The projects ranged

source code with an average of 75K.

the requirements or design phases.

in size from 51-112K lines of

No examination was done for

Once the variables were chosen the average and standard

deviation was computed for each baseline. Some baselines suffered from limited data points during the beginning of the coding

phase. A couple of the projects, in which problems were known to

have existed, were flagged as soon as data on these projects

appeared, but this was fifty percent of the way into coding. It

is not known how much earlier they would have appeared, if data

4-52
,3

o

t

0

o_

U

•.4 -_

Sample Baseline .

"o_.

•.0,.

.o

oO

°o

•\

.i

• .°°°°'°°; /

:

i.

-"'-_...

(I 0 a) .0 _"

= ..e q.4 6 o 6 6

_q>-

._.o_

4-53
existed at the early intervals.

IV. Interpretation of Relative Measures

Once a set of baselines are established new projects may be

compared to them and potential problems flagged. To interpret

these flagged relative measures a list should be developed with

each measures possible interpretations. Each list must consider

the possible interpretations of the relative measure when it is

either above normal or below normal. What each component variable actually measures should also be considered when the different lists are developed.

A list was developed with possible interpretations for each

relative measure being examined in the context of the SEL

environment. In another environment the interpretation of these

measures might be different. These lists are subdivided into two

categories; above and below normal. The above normal category

contains possible interpretations for the relative measure when

it is outside one standard deviation from the average in the

positive direction. The below normal category refers to

interpretations when the measure is outside one standard deviation from the mean in the negative direction.

One of the reasons this methodology works is because of the

implicit interdependencies between different relative measures.

To show these interdependencies more explicitly a cross reference

chart has also been provided for each interpretation to indicate

4-54
I

i

II

IIII

IIIlIIi

II

III

III

II

oo II

00,0

•q_g

"_00k

0 L,.O0

Uo__

go00_

e

L_

a,B0

B

'0

o

Q,

ill

n

IIC

L0

)'1 '_I0 f,,t

"dN_O 00 _ g O

_ Op.4 OP*_ U _ I .,,4 0 k 0 0 II _ L"L

0"i0O II

•-4IlI I I

_l IN 0 r.. 0

0o

e

r.e el •0

0o I ck

0"40• r-

_ _0

__ "40

_4|IlII

aS

0

aP0

o0

• _r.lU 0 0

Q II

aS

r., El ,= _,,o _1 r,.,,0_0

_l000_ 00 _,O _,

II

III!IIII!

_ ¢'d _I _.-r i..I ,.0 L,.. _0 0-_

,..4

4-55

0o

&, 0oL

310

--i0

f_0g

Q. 0

.J

...'_ o o I

c: ...._ r.

2J_

¢. o IiI

_

oo

ootl

,_°

s.

o

e_

f_Q

_t _-

_r I0

OU iI Q iI _ e-

_lI0 f,-0

¢_ I-4

_-iI! I l

0

0o

•o

_I _-•

o0

.o e- 0

ooQ

G c: gas

_-isIIIIl

.GI0

o

o ,n o

c.on

,2=°X_._o

0-40_I

,_IIII I 0 I
m_m o o I G ,.-¢ _ Q

f.m

>m

Z.,n0

0 ,,IC

Cgg

_

00BL

BL

_

g

_L80

,.1

o..-_ o o I

e- ,..,L

o

_ :-- o n o_ o o t- kooo

i-• ,,.4 = [ o _. o

...4

UI

t_0_

2

0

,oo

¢o

*Q

_o rrlm_ t_i

.u

4,_ ["G

e- ,,_ 00 0 S. 4.. O I I I ,-4 I I O m i! ¢h ¢I! . eo _1

_m

"cl,_ oo

mG

•_o

O.U

o_8

"_o

0 I i o

4-56

I0

o,O

k.IIL

b0 O

S*

E'--I0Gmo

_7

oO i

e, ,..,L

ee0

e_ .o e*

e. o

a_

e_c

¢&

oom I I o

[...o e*

4.,

t,._o

og

G_

o•o

_,,-¢•

,._':oo • 0 t,.

L

>!

0 e_

,g

,2¢

•

0 o.lo _ L O

0

lII

0B

W•

G

o

°:

-_

0_ t_O

•_IIII

8

.¢. o

_0s..

0

_ .,,40

o°_:o

o E! @ o IIIIIIlIIIIIiIIIIII
lIlIIIIIIlIilIIIIII

II

II

f.

(I00

L.

D.

e_G

_Jl

.iJoLg80

r_

a_

00 li

oo

g

t_w

>8

U L_o

CQI_

L

4.. @

&,O

e -.4 _ o _ L

L,, ,_g

O0 il • e 0 Q

o_L

¢..QO

¢OO

N

.,.) m

!! ,..*o

II0 .._

¢_ real

_O ...* il o

_- O0 G ¢,s,

•,4 Im

O .u_ ["

04,Dg•

00 ill

0 O_,G

,., .'???

>B

0n

o

,i ¢:

Q

11o0 c: : JO

o

,..4I!I!I

•I. _II 0 li

/ --T

II _.D

m0

[.. i_o

0

k. :1_0 ,11

O O.,-IJ__O

@ _I > a o [..

J_o

QoG0

e_

:gt

•..4tII

OB

4-57

ORIG!h'AL r_,_ ,._,

OF POOR QUAJ.JTY
other relative measures that can have the same interpretation. A

number in the cross reference section indicates the list number

of a relative measure that can have the same interpretation. The

position of the list number in the 4-quadrant cross reference

section indicates whether both interpretations are found with

above normal values, both with below normal values, or one with

above and the other with below normal values.

With these lists a set of flagged relative measures may be

evaluated. When a relative measure is flagged, its associated

list is examined for possible interpretations. Overlaps of this

list with the lists of other flagged relative measures form the

new list of what these relative measures together might indicate.

The more overlaps a particular interpretation has, the greater

the.chance it is the correct interpretation. Interpretations

with the same number of overlaps must be considered equally. The

more relative measures flagged the more serious the problem may

be. It is up to the manager to determine whether the deviation

is good or bad.

2- Monitorin_ a Software Project's Development

Once the baselines have been developed and the lists of possible interpretations have been put together a software manager

may monitor the actual development of a project. Example I

demonstrated how a single interval may be interpreted. The following discussion will trace the development of an actual proJect. During the actual use of this methodology, influence would

4-58

!

!

l

l

!

!

!

l

I

I

I

I

I

I

I

I

I

I
I

t

I

i

1

i

I

I

I

i

I

I

I

I

I

I

I

be exerted to correct problems as soon as they are identified.

With this study, we must be content to study a projects evolution, without hindrance, and see at what points problems could of

been detected.

Project twenty m was chosen for this examination because data

existed throughout the projects development. In most respects

project twenty was an average project. The project did have a

lower than normal productivity rate. The lower rate may be partial!y explained by the fact the management was less experienced

when compared to other projects. The project also suffered from

some delayed staffing. Changes in staffing will be noted when

the different time intervals are discussed.

The tables on the following page show which

ures were

lines for

represent

measure was from the baseline. The baseline for each

measure was calculated using all nine projects.

relative measflagged when project twenty was compared to the baseeach stage of development. The numerical values

how many standard deviations each flagged relative

relative

Start of Coding:

At the start of coding only one relative measure is flagged.

The smaller than normal number of software changes per line of

source code using the discrete approach reflects work done during

• The numbering convention used is an extension

first used by Bailey and Basili [Bailey81].

of the one

4-59

I
o

II oe

0 II N

,°00

0

_..

a.m

a

q;

fa0m0

a_0

f.. 0.

f. _I0

0 a.10

00 >_

0 a;

aJ

...-00 "0

c:O0

@0

"0

0

_ 0"0 •._ _

0

¢;0

00 "0

_"0

_ "0

3 (_0

_"0

a_

00

OO

f., O

O

_-,O

@0

•,-tO I1

,.-_L@

_]O .,4

.;=O_

L, I_ a)

b0_ e,-

;-=0

_ u'J _n

@•

I ,'-' III@IIII÷IIII÷IIrIIIIIIIII+iiI!II

I.!IIiiII

e-L50BB

_:_ 0

f..

Q.

_-_

@II!

•÷

,-III

•÷

,-IIII!I

4,III÷

•

III ¢_I ÷ • I ,- III ,- I ,- III@I!I!| ÷IIII

4,IIII÷IIII

@.IIII@IIII

.@IIII@IIII

4,IIII

.@!II|I!I

120

ID

I10

tl I.

f.. III O ,.) C fl O

f., t_

t.,

a.;

rJ _- O a,_

I..,_ _J

f,.> cO

"O "o C c D O O e_ t o

.O

O "_C

"o0

a.;B

_oO

f..

4; O

f., O

O

t.,O

•,-0 rJOO_

,.._ L. _1 rJ 4)

f.,_OO ,.-_

[.._ ,,.4 ,.4 a._

_,. :_.;= .;=O

Q.L,OOO

I

0÷••

0I ¢_I"

IIIII

4)I

"0@OIII

"0@II

0I_ "-

01II "--

¢IIII

_

0I_ ""

01II

0!

01I

0

S0 _I

t..0

• .I_ 4,÷

¢_IIII

III

III

01II

••@ .I,÷

¢_ ¢'_IIlI

III

III

III

@÷ 4.

III

III

III

__II ¢_II

••@@• .lI I I I ; I I I I

.@ 4, 4, I I I I I ' I I I I ! ! !

4"'÷@

III

III

I!I

""_III

• 4- 4. .I- I I I I I I I I I • .@ 4.. "I,'

'--III

III

III

_ ¢%tI _1II

•• 4"•@ a.

III

III

÷@ _I-

•-III

III

I_I_I

4-60

IIIiIIIiIiIIIIII
I

|

I

I

I

I

i

I

I

I

I

I

I

i

I

I

I

I

the design phase. The lists designed in the previous section

were directed towards code production and testing and do not

apply to this time interval when using the discrete approach.

This measure may indicate good specifications or lots of PDL

being generated. The manager might want to examine this measure

later if it constantly repeated. Since it is the only measure

flagged at this time it will be ignored.

20% Coding:

The flagged relative measures found using the discrete

approach at this point represent the work done from the start of

coding until twenty percent of the way through coding. The list

of possible interpretations for the flagged relative measures,

generated from the lists made previously for the individual relatire measure, would look like:

overlaps interpretation

3

3

2

2

2

1

1

bad specifications

code removed

low productivity

high complexity

error prone code

lots of testing

good testing

changes hard to isolate

changes hard to make

unit testing being done

easy errors being found

The strongest interpretations are bad specifications and code

being removed. If the actual history is examined one finds that

during this period there were a lot of specifications being

changed. This resulted in code which was to be modified being

4-61

I
discarded and new code being written. During the early period

lots of PDL was being produced but very little new executable

code. The list of possible interpretations does show that low

productivity is also a strong possibility.

40% Coding:

The flagged relative measures which appear using the cumulative approach, from this time period on, are stronger indicators

than the ones used in the first couple of intervals because the

average is computed using more data points. The use of the

discrete approach for the interval of twenty to forty percent is

still dependent on three data points. The list of possible

interpretations for this time period is:

# overlaps interpretation

I low productivity

I high complexity

I error prone code

I bad specifications

1 code being removed

changes hard to isolate

changes hard to make

lots of testing

unit testing being done

good testing

easy errors

The number of possibilities is larger with this set of possible

interpretations. Five interpretations are slightly stronger than

the others. During the actual development, the first release of

the project was made. The amount of code actually written was

also lower than normal during this period. The use of the

discrete approach gives a stronger feeling that code is not being

4-62

!

m

I

I

II

II

I

I

I

I

I

i

I

i

II

I

i

II
l

I

I

I

I

i

i

I

!

I

I

I

i

i

I

i

I

I

written. Transported code tends to be installed in large

which can be isolated using the discrete approach.

blocks

50$ Coding:

The relative measures flagged during this period are the

same as the ones flagged at the twenty percent coding interval.

The deviation from the norm for this interval is larger. The

larger deviation may indicate a more serious problem. The problem may of been Just as serious earlier but without the extra

data points, that are now available, it could not be determined.

The possible interpretations may be taken from the list developed

earlier. Bad specifications and code removal were not factors

during this period. The next three highest priority interpretations were; high complexity, error prone code, and low productivity. In addition to this the manager should be concerned with

the continued appearance of the relative measure, programmer

hours per computer run, as seen using the cumulative approach.

This may indicate a lot of testing going on. This in conjunction

with error prone code as a possible interpretation may indicate

trouble. During actual development this period was spent

developing code for the second release. The project manager felt

that code was still not being developed quickly enough during

this period.

60% Coding:

4-63

!
Only one relative measure is shown at this interval. The

number of programmer hours per computer run using the cumulative

approach is lower than normal for the third consecutive time.

This should concern the manager because when examining the list

for this measure one finds:

error prone code

lots of testing

easy errors being fixed

Since the occurrence of this measure is persistent it may indicate that the problem was corrected but not enough effort was

expended to completely compensate for the past problems. It

might also indicate the problem still exists. During the actual

project it was found that while a lot of code was written, it had

not been throughly tested. Release two was made during this

period which could explain a heavy test load. Two additional

staff members were added to the project during this phase to aid

in coding and testing.

80% Coding:

The eighty percent coding interval does not show any measures outside the normal bounds. The addition of two staff

members during the sixty percent coding phase, as well as the

addition of a senior staff member during this phase, appears to

have adjusted the project back along the lines of normal development. To fully compensate for the earlier problems one might

expect some of the measures to swing in the other direction away

4-64

I

It

I

I

I

I

I

I

I

II

I

I

I

li

II

II

I

II

II
I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

l

from the average. The fact this over correction did not occur

might explain the problems encountered in the next section.

Start of System and Integration Testing:

The flagged relative measures at this time period

the build up of effort for the third and final release.

of possible interpretations for the collective set of

measures looks like:

# overlaps interpretation

3

3

3

2

2

2

I

I

I

I

high complexity

bad specifications

code being removed

error prone code

low productivity

lots of testing

changes hard to isolate

unit testing being done

good code

poor testing

changes hard to make

good testing

compute bound algorithms

being run

easy errors being fixed

Since the code did have a past history of poor testing an

ally large build up of testing should be expected.

interpretations that apply most to this situation are

testing and error prone code.

reflect

The list

flagged

unusuThe two

lots of

50% System and Integration Testing:

Only one relative measure is flagged at this interval. This

measure was flagged using the cumulative approach. An examination of the measure at the previous interval shows a very high

4-65

I
value. A slow drop off from this high measure is to be expected

when using the cumulative approach. An examination of possible

interpretations that would apply for this period of development

include:

high complexity

lots of testing

unit testing being done

testing code being removed

A lot of testing is certainly indicated by past history.

Start Acceptance Testing:

The relative measures flagged at this interval reflects the

build up in testing before the start of acceptance testing. The

list of possible interpretations looks like:

# overlaps interpretation

bad specifications

code being removed

high complexity

low productivity

error prone code

lots of testing

changes hard to isolate

changes hard to make

unit testing being done

good testing

Since little code was being developed during the testing period,

a large amount of testing with errors being found is the most

reasonable interpretation of these flagged measures. The early

history of poor testing may be seen here with errors being

uncovered late.

4-66

I

I

I

I

l

I

I

I

I

I

I

I

I

I

I

I

I

I
I

I

l

I

I

I

I

l

I

I

I

I

i

I

!

I

I

I

I

End Acceptance Testing:

The two flagged relative measures at the end of acceptance

testing reflect the clean up effort being made on the code. An

average amount of computer time and an average number of computer

runs indicates that the acceptance testing is going well. The

project was behind schedule due to the eariier problems encountered. Clean up was done during the acceptance testing phase in

an attempt to get the project out the door as soon as possible.

As seen in this example, the problems that

projects development

the relative measures.

monitor projects.

occur during a

are reflected in the values calculated for

The methodology preposed can be used to

The number of possible interpretations

increases with each new flagged relative measure. The ordering

of the measures by the number of overlaps provides an easy method

of sorting the possible interpretations by priority. Another

method of sorting the possible interpretations could include a

factor that considers both the number of overlaps and the probability of a given interpretation being the cause at a given

interval. The weighting of interpretations for a given interval

could be calculated using the pattern of occurrence of the different interpretations which have appeared during the same interval in past projects.

V I. An Alternate A_roach

4-67
Flagged relative measures might also be interpreted using a

decision support system. The data for the various relative measures would be stored in a knowledge base along with a set of production rules. To evaluate a project the values for each relarive measure would be entered into

base would compare the relative

baselines, determine which relative

the system. The knowledge

measures to their respective

measures were outside the

norm, and interpret these relative measures using the production

rules. A list of possible interpretations ordered by probability

would be generated as a result.

The difference between a decision support system and the

approach presented in this paper is the method of interpreting

the flagged relative measures. Each production rule in the decision supportsystem is the logical disjunction of several flagged

measures which yields a given interpretation. Each production

rule is assigned a confidence rating which is then used to rate

the possible interpretations. The lists for the relative measures provided earlier in the paper may be easily converted to

production rules using the cross reference section. To develop

the production rules for an interpretation one must generate the

various combinations of relative measures which might reasonably

imply the interpretation. Some relative measures may not imply a

particular interpretation unless they are found in conjunction

with another relative measure. Once the production rules are

known and a knowledge base constructed a decision support system

may be built. For an example of a domain independent decision

4-68
I

I

I

I

I

I

I

I

I

I

l

i

l

I

I

I

i

I

support system see Reggia and Perricone [Reggia82].

VII. Summary

The methodology presented in this paper showed that invariant relationships exist for similar projects. New projects may

be compared to the baselines of these invariant relationships to

determine when projects are getting off track.

The ability of the manager to interpret the measures that

fall outside the norm is dependent on the amount of information

the underlying variables convey. The manager must decide what

attributes are to be measured (e.g. productivity) and pick variables that are closely related to them and are also measurable

throughout the project. As an example, a variable like lines of

code may be too general when measuring productivity. Measuring

the newly developed code, either source code or executable code,

would be more informative since these variables are more directly

related to effort. How applicable an interpretation is for the

period currently being examined should also be considered when

ordering the list. The variables the manager finally decides on

are then combined to form relative measures.

One method of interpreting a relative measure is by associating lists of possible interpretations with it. When a relative

measure appears outside the norm, the list of possible interpretations is considered. If more than one relative measure is outside the norm the lists are combined. The more times a possible

4-69

I
interpretation is repeated in the lists, the greater the probability it is the cause. How applicable an interpretation is for

the period being examined should also be considered when ordering

the list. The manager must investigate the suggested causes to

determine the real one.

VIII. Conclusion

The ability to monitor a projects development and detect

problems as they develop may be feasible. The methodology proposed showed favorable results when examining a past case.

The use of baselines and lists of interpretations for comparing projects provides an easy method for monitoring software

development. Both the baselines and the lists of interpretations

may be updated as new projects are developed. As more knowledge

is gleaned the accuracy of this system should improve and provide

a valuable tool for the manager.

4-70

l

I

I

I

l

I

I

I

I

l

I

I

I

I

I

I

I

I

I
!

I

I

I

I

i

I

I

I

I

I

i

!

I

I

I

I

I

Bibliography

[Bailey81]

Bailey, John W. and victor R. Basili, A Meta-Model for

Software Development Resource Expenditures, Proceedings,

Fift______hInternational Conference on Software Engineering, September 1981.

[Basili81]

Basili, Victor R. and Karl Freburger, Programming Measurement and Estimation in the Software Engineering Laboratory,

Journal of Systems and Software, 1981.

[Card82]

Card, David, Frank McGarry, Jerry Page, Suellen Eslinger,

and Victor Basili, The Software Engineering Laboratory,

SEL-81-I04, Software Engineering Laboratory Series, Goddard

SpaceFlight Center, February 1982.

[Church82]

Church, Victor, David Card, Frank McGarry, Jerry Page, and

Victor Basili, Guide To Data Collection, SEL-81-I01,

Software Engineering Laboratory Series, Goddard Space Flight

Center, August 1982.

[Manley82]

The Role of Measurements in Programming Technology, Lecture

presented at University of Maryland, November 15, 1982.

[Minsky75]

Minsky, M. L., A Framework for the Representation of

Knowledge, The Psychology of Computer Vision, pp. 211-280,

McGraw Hill, New York, 1975.

[Reggla82]

Reggla, James and Barry Perricone, KMS Manual, TR-1136,

Department of Mathematics, University of Maryland Baltimore

County, January 1982.

[SEL82]

SEL,, Collected Software Engineering Papers: Volume I, SEL82-004, Software Engineering Laboratory Series, Goddard

Space Flight Center, July 1982.

[Walston77]

Walston, C. E. and C. P. Felix, A Method of Programming

Measurement and Estimation, IB___MM Systems Journal, January

1977.

4-71

!
I

I

I

I

l

I

I

I

I

I

I

I

I

I

I

I

I

I

_7-6 :'"

N 87 - 24 90 2

Technical Report TR-II95 August 1982

NSG-5123

SOFTWARE ERRORS AND COMPLEXITY:

AN EMPIRICAL INVESTIGATION*

Victor R. Basili and Barry T. Perricone

.i

*Research supported in part by National Aeronautics and Space Administration grant NSG-5123 to the University of Maryland.. Computer time supported

in part through the facilities of the Computer Science Center of the

University of Maryland.

4-73
SOFTWARE ERRORS AND_COMPLEXITY:

AN EMPIRICAL INVESTIGATION

Victor R. Basili and Barry T. Perricone

Department of Computer Science

University of Maryland

College Park, Md.

1982

ABSTRACT

The distributions and relationships derived from the change

data collected during the development of a medium scale

satellite software project shows that meaningful results can

be obtained which allow an insight into software traits and

the environment in which it is developed. Modified and new

modules were shown to behave similarly. An abstract classification scheme for errors which allows a better understanding of the overall traits of a software project is also

shown. Finally, various size and complexity metrics are

examined with respect to errors detected within the software.

yielding some interesting results.

4-74
I

I

i

I

I

I

I

I

i

I

I

i

I

I

I

I

1.0 INTRODUCTION

The discovery and validation of fundamental relationships between the development of computer software, the

environment in which the software is developed, and the frequency and distribution of errors associated with the

software are topics of primary concern to investigators in

the field of software engineering. Knowledge of such relationships can be used to provide an insight into the characteristics of computer software and the effects that a programming environment can have on the software _roduct. In

addition, it can provide a means to improve the understanding of the terms reliability and quality with respect to

computer software. In an effort to acquire a knowledge of

these basic relationships, change data for a medium scale

software project was analyzed (e.g., change data is any

documentation which reports an alteration made to the

software for a particular reason).

In general, the overall objectives of this paper are

threefold : first, to report the results of the analyses;

second, to review the results in the context of those

reported by other researchers; and third, to draw some conclusions based on the aforementioned. The analyses

presented in this paper encompass various types of distributions based on the collected change data. The most important of which are the error distributions observed within

the software project.

In order for the reader to view the results reported in

this paper properly, it is important that the terms used

throughout this paper and the environment in which the data

was collected are clearly defined. This is pertinent since

many of the terms used within this paper have appeared in

the general literature often to denote different concepts.

Understanding the environment will allow the partitioning of

the results into two classes: those which are dependent on

and those which are independent of a particular programming

environment.

1.1 DESCRIPTION OF THE ENVIRONMENT

The software analyzed within this paper is one of a

large set of projects being analyzed in the Software

Engineering Laboratory (SEL). The particular project

analyzed in this paper is a general purpose program for

satellite planning studies. These studies include among

others: mission maneuver planning; mission lifetime; mission

launch; and mission control. The overall size of the

software project was approximately 90,000 source lines of

code. The majority of the software project was coded in FORTRAN. The system was developed and executes on an IBM 360.

4-75
The developers of the analyzed software had extensive

experience with ground support software for satellites. The

analyzed system represents a new application for the

development group, although it shares many similar algorithms with the system studied here.

It ts also true that the requirements for the system

analyzed kept growing and changing, much more so than for

the typical ground support software normally built. Due to

the commonality of algorithms from existing systems, the

developers re-used the design and code for many algorithms

needed in the new system, Hence a large number of re-used

(modified)

modules became part of the new system analyzed here.

An approximation of the analyzed software's life cycle

is displayed in Figure I . This figure only illustrates the

approximate duration in time of the various phases of the

software's life cycle. The information relating the amount

of manpower involved with each of the phases shown was not

specific enough to yield meaningful results, so it was not

included.

4-76

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I
I

I

I

I

I

l

I

I

I

I

I

l

I

I

I

I

I

.... i

I

LIFE _CLE OF _NALYZED SOFTWARE

CHANGE FO l_"Lg

DESIGN

CODING

i

ACCEPTANCE

TESTING I I

MAINTENANCE

JA_N.

1975

I I i I I 1

1976 1977 1978 1979 1980 1981

Figure I

I.2 TERMS

This section presents the definitions and associated

contexts for the terms used within this paper. A discussion

of the concepts involved with these terms is also given when

appropriate.

Module: A module is defined as a named subfunction, subroutine, or the main program of the software system. This

definition is used since only segments written in FORTRAN

which contained executable code were used for the analyses.

Change data from the segments which constituted the data

blocks, assembly segments, common segments, or utility routines were not included. However, a general overview of the

data available on these types of segments is presented in

Section 4.0 for completeness.

There are two types of modules referred to within this

paper. The first type is denoted as modified. These are

4-77
modules which were developed for previous software projects

and then modified to meet the requirements of the new project. The second type is referred to as ne___w. These are

modules which were developed specifically for the software

project under analyses.

The entire software project contained a total of 517

code segments. This quantity is comprised of 36 assembly

segments, 370 FORTRAN segments, and 111 segments that were

either common modules, block data, or utility routines. The

number of code segments which met the adopted module definition was 370 out of 517 which is 72% of the total modules

and constitutes the majority of the software project. Of

the modules found to contain errors 49% were categorized as

modified and 51% as new modules.

Number of Source and Executable Lines: The number of source

lines within a module refers to the number of lines of executable code and comment lines contained within it. The

number of executable lines within a module refers to the

number of executable statements, comment lines are not

included.

Some of the relationships presented in this paper are

based on a grouping of modules by module size in increments

of 50 lines. This means that a module containing 50 lines

of code or less was placed in the module size of 50; modules

between 51 and 100 lines of code into the module size of

100, etc. The number of modules which were contained in

each module size is given in Table I for all modules and for

modules which contained errors (i.e., a subset of all

modules) with respect to source and executable lines of

code.

4-78

l

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I
I

I

I

I

i

I

I

I

I

I

I

I

I

I

I

I

I

I

Number modules

All Modules Modules with Errors

Number

of Lines Source Exececutable Source Executable

0-50 53 258 3 49

51-100 107 70 16 25

101-150 80 26 20 13

151-200 56 13 19 7

201-250 34 1 12 I

251-300 14 I 9 0

301-350 7 1 4 I

351-400 9 0 7 0

>400 10 0 6 0

Total 370 370 96 96

Table I

Error: Something detected within the executable code which

caused the module in which it occurred to perform

incorrectly (i.e., contrary to its expected function ).

Errors were quantified from two view points in this

paper, depending upon the goals of the analysis of the error

data. The first quantification was based on a textual rather

than a conceptual viewpoint. This type of error quantification is best illustrated by an example. If a "*" was

incorrectly used in place of a "+" then all occurrences of

the "*" will be considered an error. This is the situation

even if the "*"'s appear on the same line of code or within

multiple modules. The total number of errors detected in

the 370 software modules analyzed was 215 contained within a

total of 96 modules, implying 26% of the modules analyzed

contained errors.

The second type of quantification was used to measure

the effect of an error across modules, textual errors associated with the same conceptual problem were combined to

yield one conceptual error. Thus in the example above, all

incorrectly used *'s replaced by +'s in the same formula

were combined and the total number of modules effected by

that error are listed. This is done only for the errors

reported in Figure 2. There are a total of 155 conceptual

errors. All other studies in this paper are based upoon the

4-79
first type of quantification described.

Statistical Terms and Methods: All linear regressions of the

data presented within this paper employed as a criterion of

goodness the least squares principle (i.e., "choose as the

"best fitting" line that one which minimizes the sum of

squares of the deviations of the observed values of y from

those predicted" [I]).

Pearson's product moment coefficient of correlation was

used as an index of the strength of the linear relationship

independent of the respective scales of measurement for y

and x. This index is denoted by the symbol r within this

paper. The measure for the amount of variability in y

accounted for by linear regression on x is denoted as r2

within this paper.

All of the equations and explanations for these statistics can be found in [I]. It should be noted that other

types of curve fits were conducted on the data. The results

of these fits will be mentioned later in the paper.

Now that the software's environment and the key terms

used within the paper have been defined and outlined, a discussion of the basic quantification of the data collected,

the relationships and distributions derived from this quantification, and the resulting conclusions are presented.

2.0 BASIC DATA

The change data analyzed was collected over a period of

33 months, August 1977 through May 1980. These dates

correspond in time to the software phases of coding, testing, acceptance, and maintenance (Figure I) . The data collected for the analyses is not complete since changes are

still being made to the software analyzed. However, it is

felt that enough data was viewed in order to make the conclusions drawn from the data significant.

The change data was entered on detailed report sheets

which were completed by the programmer responsible for

implementing the change. A sample of the change report form

is given in the Appendix. In general, the form required

that several short questions be answered by the programmer

implementing the change. These queries allowed a means to

document the cause of a change in addition to other characteristics and effects attributed to the change. The majority of this information was found useful in the analyses.

The key information used in the study from the form was: the

data of the change or error discovery, the description of

4-80

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I
I

I

I

I

I

I

I

I

I

i

I

I

I

I

I

I

I

I

I

the change or error, the number of components changed, the

type of change or error, and the effort needed to correct

the error.

It should be mentioned that the particular change

report form shown in the Appendix was the most current form

but was not uniformly used over the entire period of this

study. In actuality there were three different versions of

the change report form, not all of which required the same

set of questions to be answered. Therefore , for the data

that was not present on one type of form but could be

inferred, the inferred value was used. An example of such

an inference would be that of determining the error type.

Since the error description was given on all of the forms

the error type could be inferred with a reasonable degree of

reliability. Data not incorporated into a particular data

set used for an analysis was that data for which this inference was deemed unreliable. Therefore, the reader should be

alert to the cardinality of the data set used as a basis for

some of the relationships presented in this paper. There

was a total of 231 change report forms examined for the purpose of this paper.

The consistency and partial validity of the forms was

checked in the following manner. First, the supervisor of

the project looked over the change report forms and verified

them (denoted by his or her signature and the date).

Second, when the data was being reduced for analysis it was

closely examined for contradictions. It should be noted

that interviews with the individuals who filled out the

change forms were not conducted. This was the major difference between this work and other error studies performed by

the Software Engineering Laboratory, where interviews were

held with the programmers to help clarify questionable data

(8).

The review of the change data as described above

yielded an interesting result. The errors due to previous

miscorrections showed to be three times as common after the

form review process was performed, i.e. before the review

process they accounted for 2% of the errors and after the

review process they accounted for 6% of the errors. These

recording errors are probably attributable to the fact that

the corrector of an error did not know the cause was due to

a previous fix because the fix occurred several months earlier or was made by a different programmer, etc.

_.0 RELATIONSHIPS DERIVED FROM DATA

This section presents and discusses relationships derived

from the change data.

4-81
_.I CHANGE DISTRIBUTION BY TYPE

Types of changes to the software can be categorized as

error corrections or modifications (specification changes,

planned enhancements, clarity and optimization improvements). For this project, error corrections accounted for

62% of the changes and modifications 38%. In studies of

other SEL projects, errors corrections ranged from 40% to

64% of the changes.

3.2 ERROR DISTRIBUTION BY MODULES

Figure 2 shows the effects of an error in terms of the

number of modules that had to be changed. (Note that these

errors here are counted as conceptual errors.) It was found

that 89% of the errors could be corrected by changing only

one module. This is a good argument for the modularity of

the software. It also shows that there is not a large

amount of interdependence among the modules with respect to

an error.

NUMBER OF MODULES AFFECTED BY AN ERROR (data set: 211 textual errors)

174 conceptual errrors)

#ERRORS #MODULES AFFECTED

155 (89%) I

9 2

3 3

6 4

I 5

Figure 2

Figure 3 shows the number of errors found per module.

The type of module is shown in addition to the overall total

number of modules found to contain errors.

4-82

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I
I

I

I

I

I

I

I

I

I

I

I

I

I

i

I

I

I

i|

NUMBER OF ERRORS PER MODULE (data set:

#MODULES NEW MODIFIED #ERRORS/MODULE

36 17 19 I

26 13 13 2

16 10 6 3

13 7 6 4

4 I** 3* 5

1 I** 7

215 errors)

Figure 3

The largest number of errors found were 7 (located in a

single new module) and 5 (located in 3 different modified

modules and I new module). The remainder of the errors were

distributed almost equally among the two types of modules.

The effort associated wlth correcting an error is

specified on the form as being (I) I hour or less, (2) I

hour to I day, (3) I day to 3.days, (4) more than 3 days.

These categories were chosen because it was too difficult to

collect effort data to a finer granularity. To estimate the

effort for any particular error correction, an average time

was used for each category, i.e. assuming an 8 hour day, an

error correction in category (I) was assumed to take .5

hours, an error correction in category (2) was assumed to

take 4.5 hours, category (3) 16 hours, and category (4) 32

hours.

The types of errors found in the three most error prone

modified modules (* in Figure 3) and the effort needed to

correct them is shown in Table 2. If any type contained

error corrections from more than one error correction

category, the associated effort for them was averaged. The

fact that the majority of the errors detected in a module

was between one and three shows that the total number of

errors that occurred per module was on the average very

small.

4-83

I
The twelve errors contained in the two most error prone

new modules (** in Figure 3) are shown in Table 3 along with

the effort needed to correct them.

NUMBER OF ERRORS

(15 total)

AVERAGE EFFORT[

TO CORRECT

I

I

I

I

I

misunderstood

or incorrect

speclfications

incorrect design

or implementation

of a module

component

clerical error

8 24 hours

5 16 hours

2 4.5 hours

EFFORT TO CORRECT ERRORS IN THREE MOST ERROR PRONE

MODIFIED MODULES

Table 2

NUMBER OF ERRORS

(12 total)

AVERAGE EFFORT

TO CORRECT

misunderstood

i

I

i

I

I

I

I

or incorrect

requirements 8

incorrect design

or implementation

of a module

clerical error

32 hours

3 0.5 hours

I 0.5 hours

EFFORT TO CORRECT ERRORS IN THE TWO MOST ERROR PRONE

NEW MODULES

Table 3

4-84

I

I

I

I

I

I

i
I

i I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

3-3 ERROR DISTRIBUTION BY TYPE

In Figure 4 the distribution of errors are shown by type. It

can be seen that 48% of the errors were attributed to

incorrect or misinterpreted functional specifications or

requirements.

The classification for error used throughout the

Software Engineering Laboratory is given below. The person

identifying the error indicates the class for each error.

A: Requirements incorrect or misinterpreted

B: Functional specification incorrect or misinterpreted

C: Design error invloving several components

I. mistaken assumption about value or structure of

data

2. mistake in control logic or computation of an

expression

D: Error in design or implementation of single component

I. mistaken assumption about value or structure of

data

2. mistake in control logic or computation of an

expression

E: Misunderstanding of external environment

F: Error in the use of programming language/compiler

G: Clerical error

H: Error due to previous miscorrection of an error

The distribution of these errors by source is plotted

in Figure 4 with the appropriate subdistribution of new and

modified errors displayed. This distribution shows the

majority of errors were the result of the functional specification being incorrect or misinterpreted . Within this

category, the majority of the errors (24%) involved modified

modules This is most likely due to the fact that the modules

reused were taken from another system with a different

application. Thus, even though the basic algorithms were the

same, the specification was not well enough defined or

appropriately defined for the modules to be used under

slightly different circumstances.

4-85

!
4O

]O,

2O

iO

36_.

L\\\

\\\

'_ 16%

6_

8_ 12%

2_ 2X

AS C.I C.2 D.1 D.2

E

Q MODIFIED MODULES O NEW MODULES

12%

FGH A,C.I A, FtD

SOURCES OF ERRORS

Figure4

4-86

IIIIIIIIIIIIIIIIIII
I

I

I

I

!1

I

I

I

|

I

I

I

I

I

I

I

I

I

I

P

E

E

N

T

O

F

N

O

N

C

L

E

R

!

C

A

L

$

80

70

80

SO

40

30

20

I0

O

Req

i 3

72

10

Fnl Design DesiKn Lan K

Spec Multi- Sinsle

COBp Comp

Type of Error

1 1

i 1

Env Ocher

SOURCES OF ERROR ON OTHER PROJECTS

Figure 5

The distribution in Figure 4 should be compared with

the distribution of another system developed by the same

organization shown in Figure 5. Figure 5 represents a typical ground support software system and was rather typical of

the error distributions for these systems. It is different

from the distribution for the system we are discussing in

this paper however, in that the majority of the errors were

involved in the design of a single component. The reason

for the difference is that in ground support systems, the

design is well understood, the developers have had a reasonable amount of experience with the application. Any re-used

design or code comes from similar systems, and the requirements tend to be more stable. An analysis of the two distributions makes the differences in the development environments clear in a quantitative way.

4-87
Thepercent of requirements and specification errors is

consistent with the work of Endres'[1]. Endres found that

46% of the errors he viewed involved the misunderstanding of

the functional specifications of a module. Our results are

similar even though Endres" analysis was based on data

derived from a different software project and programming

environment. The software project used in.Endres ° analysis

contained considerably more lines of code per module, was

written in assembly code, and was within the problem area of

operating systems. However, both of the software systems

Endres analyzed did contain new and modified modules.

Of the errors due to the misunderstanding of a module's

specifications or requirements (48%), 20% involved new

modules while 28% involved modified modules.

Although the existence of modified modules can shrink

the cost of coding, the amount of effort needed to correct

errors in modified modules might outweigh the savings. The

effort graph (Figure 6) supports this viewpoint: 50% of the

total effort required for error correction occurred in modified modules; errors requiring one day to more than three

days to correct accounted for 45% of the total effort with

27% of this effort attributable to modified modules within

these greater effort classes. Thus, errors occurring in new

modules required less effort to correct than those occurring

in modified modules.

4-88

I

I

I

I

I

I

I

I

I

I

I

I

I

I

!

I

I

I
i

I

I

I

I

I

I

I

I

I

t

I

I

I

I

I

I

I

m¢

36_

\\ \I

'- _ \\I

\\\1

\\\ ]

30Y,

\15x\

\\\',

\\\\

27%

\iz ',N\ \ \\

\\\

,\\ \ I\ \\.

"'0.5% \

\\\

ki% _ 15% I

\\\ !

3%

EFFORT

1 2 3 4

O MODIFIED NODULES

NEW MODULES

1 - I hr. or less

2 - i hr. to I day

3 - i day co 3 days

4 - more than 3 days

EFFORT GRAPH

Figure 6

The similarity between Endres" results and those

reported here tend to support the statement that independent

of the environment and possibly the module size, the majority of errors detected within software is due to an inadequate form or interpretation of the specifications. This

seems especially true when the software contains modified

modules.

In general, these observations tend to indicate that

there are disadvantages in modifying a large number of

already existing modules to meet new specifications. The

alternative of developing a new module might be better in

some cases if there does not exist good specifications for

the existing modules.

_._ OVERALL NUMBER OF ERRORS OBSERVED

Figure 7 displays the number of errors observed in both

new and modified modules. The curve representing total

4-89
modules (new and modified) is basically 5ell-shaped. One

[nterpretat[on is that up to some point errors are detected

at a relatively steady rate. At this point at least half of

the total "detected-undetected" errors have been observed

and the rate of d£scovery thereafter decreases. It may also

imply the maintainers are not adding too many new errors as

the system evolves.

It can be seen, however, that errors occurring in

modified modules are detected earlier and at a slightly

higher rate than those of new modules. One hypothesis for

this is that the majority of the errors observed in modified

modules are due to the misinterpretation of the functional

specifications as was mentioned earlier in the paper.

Errors of this type would certainly be more obvious since

they are more blatant than those of other types and therefore, would be detected both earlier and more readily.(See

next section.)

4-90
!

I

I

I

i

i

i

!

I

I

I

i

I

I

I

I

i

I

OR;CI_"AL PA_E IS

OF POOR QUALITY

7O

,_0

,=.

c.,

.=.

30

/

\

/

%

\

\

1927 1978i 1979i

_Da I0 54 _0

MOD 10 67 11

COMB 2O 121 51

\

\,

\

\

\

,i

1980

9

23

NEW AND MODIFIED NODULES

NEW MODULES

J

,"...... i MODIFIED MODULES

NUMBER OF ERRORS OCCURRING IN MODULES

Figure 7

3.5 ABSTRACT ERROR TYPES

An abstract classification of errors was adopted by the

authors which classified errors into one of five categories

with respect to a module: (I) initialization; (2) control

structure; (3) interface; (4) data; and (5) computation•

This was done in order to see if there existed recurring

classes of errors present in all modules independent of

size. These error classes are only roughly defined so examples of these abstract error types are presented below• It

should be noted that even though the authors were consistant

with the categorization for thls project, another error

4-91

I
analyst may have interpreted the categories differently.

Failure to initialize or re-initialize a data structure

properly upon a module's entry/exit would be considered an

initialization error. Errors which caused an "incorrectpath" in a module to be taken were considered control

errors. Such a control error might be a conditional statement causing control to be passed to an incorrect path.

Interface errors were those which were associated with

structures existing outside the module's local environment

but which the module used. For example, the incorrect

declaration of a COMMON segment or an incorrect subroutine

call would be an interface error. An error in the declaration of the COMMON segment was considered an interface error

and not an initialization error since the COMMON segment was

used by the module but was not part of its" local environment. Data error would be those errors which are a result

of the incorrect use of a data structure. Examples of data

errors would be the use of incorrect subscripts for an

array, the use of the wrong variable in an equation, or the

inclusion of an incorrect declaration of a variable local to

the module. Computation errors were those which caused a

computation to erroneously evaluate a variable's value.

These errors could be equations which were incorrect not by

virtue of the incorrect use of a data structure within the

statement but rather by miscalculations. An example of this

error might be the statement A : B + I. when the statement

really needed was A = B/C + I.

These five abstract categories basically represent all

activities present in any module. The five categories were

further partitioned into errors of commission and omission.

Errors of commission were those errors present as a result

of an incorrect executable statement. For example, a commissioned computational error would be A = B * C where the

"*" should have been" "÷'. In other words, the operator was

present but was incorrect. Errors of omission were those

errors which were a result of forgetting to include some

entity within a module. For example, a computational omission error might be A = B when the statement should have

read A = B + C. A parameter required for a subroutine call

but not included in the actual call would be an example of

an interface omission error. In both of the above examples

some aspect needed for the correct execution of a module was

forgotten.

The results of this abstract classification scheme as

discussed above is given in Figure 8. Since there were

approximately an equal amount of new (49) and modified (47)

modules viewed in the analysis, the results do not need to

be normalized. Some errors and thereby modules were counted

more than once since it was not possible to associate some

errors with a single abstract error type based on the error

4-92

!

!

!

!

|

II

!

!

!

I

!

!

i

!

II

!

!
description given on the change report form.

initialization

control

interface

data

computation

commission

new modified

omission

new modified

2 9 5 9

12 2 16 6

23 31 27 6

10 17 1 3

16 21 3 3

28% 36% 23% 12%

iilJlllJlJllJJJl lJllllJJJlJJJllJil

64% 35%

initialization

control

interface

data

computation

total

new modified

7 18 ---

28 8 ---

50 37 ---

11 20 ---

19 24 ---

--m---- --m----

115 107

25 (11%)

36 (16%)

87 (39%)

31 (14%)

43 (19%)

ABSTRACT CLASSIFICATION OF ERRORS

Figure 8

According to Figure 8, interfaces appear to be the

major problem regardless of the module type. Control is more

of a problem in new modules than in modified modules. This

is probably because the algorithms in the old modules had

more test and debug time. On the other hand, initialization

and data are more of a problem in modified modules. These

facts, coupled with the small number of errors of omission

in the modified modules might imply that the basic algorithms for the modified modules were correct but needed some

adjustment with respect to data values and initialization

for the application of that algorithm to the new environment.

_.6 MODULE SIZE AND ERROR OCCURRENCE

4-93
Scatter plots for executable lines per module versus

the number of errors found in the module were plotted. It

was difficult to see any trend within these plots so the

number of errors/1000 executable lines within a module size

was calculated (Table 4).

Module Size Errors/t000 lines

50 16.0

100 12.6

150 12.4

200 7.6

>200 6.4

ERRORS/1000 EXECUTABLE LINES (INCLUDES ALL MODULES)

Table 4

The number of errors was normalized over 1000 executable

lines of code in order to determine if the number of

detected errors within a module was dependent on module

size. All modules within the software were included, even

those with no errors detected. If the number of errors/1000

exececutable lines was found to be constant over module size

this would show independence, An unexpected trend was

observed: Table 4 implies that there is a higher error rate

within smaller sized modules. Since only the executable

lines of code were considered the larger modules were not

COMMON data files. Also the larger modules will be shown to

be more complex than smaller modules in the next section.

Then how could this type of result occur?

The most plausable explanation seems to be that since

there are a large number of interface errors, these are

spread equally across all modules and so there are a larger

number of errors/1000 executable statements for smaller

modules. Some tentative explanations for this behavior are:

the majority of the modules examined were small (Table I)

causing a biased result; larger modules were coded with more

care than smaller modules because of their size; errors in

smaller modules are more apparent and there may indeed still

be numerous undetected errors present within the larger

modules since all the "paths" within the larger modules may

not yet have been fully exercised.

_.7 MODULE COMPLEXITY

Cyclomatic complexity [5] (number of decisions * I) was

correlated with module size. This was done in order to

4-94
determine whether or not larger modules were less dense or

complex than smaller modules containing errors. Scatter

plots for executable statments per module versus the

cyclomatic complexity were plotted and again, since it was

difficult to see any trend in the plots, modules were

grouped according to size. The complexity points were

obtained by calculating an average complexity measure for

each module size class. For example, all the modules which

had 50 executable lines of code or less had an average complexity of 6.0, Table 5 gives the average cyclomatic complexity for all modules within each of the size categories.

The complexity relationships for executable lines of code

within a module is shown in Figure 9. As can be seen from

the table the larger modules were more complex than smaller

modules.

Module size Average Cyclomatic Complexity

50 6.0

100 17.9

150 28.1

200 52.7

>200 60.0

AVERAGE CYCLOMATIC COMPLEXITY FOR ALL MODULES

Table 5

4-95
8O

7o

b0

50

3O

2o

in.

.....

F CO."_LEXIT'f VS. _Dt'LE SIZE

Executable lines /

R" - .94

10 I00I 150I 2(")# 2_4J| _O',_ I _5{_l 4Of)l

MODULE SIZE

i

_4UO

Figure 9

For only those modules containing errors, Table 6 gives

the number of errors/t000 executable statements and the

average cyclomatic complexity. When this data is compared

with Table 5 , one can see that the average complexity of

the error prone modules was no greater than the average complexity of the full set of modules.

4-96

1

i

I

I

I

I

I

I

l

I

i

I

i

I

I
I

i

I

i

I

i

i

I

I

I

i

I

I

!

i

l

I

I

!

Module Size Average Cyclomatic

Complexity

Errors/t000

executable lines

50 6.2 65.0

100 19.6 33.3

150 27.5 24.6

200 56.7 13.4

>200 77.5 9.7

COMPLEXITY AND ERROR RATE FOR ERRORED MODULES

Table 6

4.0 DATA NOT EXPLICITLY INCLUDED IN ANALYSES

The 147 modules not included in this study (i.e.,

assembly segments, common segments, utility routines) contained a total of six errors. These six errors were

detected within three different segments. One error

occurred in a modified assembly module and was due to the

misunderstanding or incorrect statement of the functional

specifications for the module. The effort needed to correct

this error was minimal (I hour or less).

The other five errors occurred in two separate new data

segments with the major cause of the errors also being

related to their specifications. The effort needed to

correct these errors was on the average from I hour to I day

(I day representing 8 hours).

5.0 CONCLUSIONS

m

The data contained in this paper helps explain and

characterize the environment in which the software was

developed. It is clear from the data that this was a new

application domain in an application with changing requirements.

Modified and new modules were shown to behave similarly

except in the types of errors prevalent in each and the

amount of effort required to correct an error. Both had a

high percentage of interface errors, however, new modules

had an equal number of errors of omission and commission and

a higher percentage of control errors. Modified modules had

a high percentage of errors of commission and a small percentage of errors of omission with a higher percentage of

4-97
data and initialization errors. Another difference was that

modified modules appeared to be more susceptible to errors

due to the misunderstanding of the specifications.

Misunderstanding of a module's specifications or requirements constituted the majority of errors detected. This

duplicates an earlier result of Endres which implies that

more work needs to be done on the form an6 content of the

specifications and requirements in order to enable them to

be used across applications more effectively.

There were shown tobe some disadvantages to modifying

an existing module for use instead of creating a new module.

Modifying an existing module to meet a similar but different

set of specifications reduces the developmental costs of

that module. However, the disadvantage to this is that

there exists hidden costs. Errors contained in modified

i

modules were found to require more effort to correct than

those in new modules, although the two classes contained

approximately the same number of errors. The majority of

these errors was due to incorrect or misinterpreted specifications for a module. TherefoPe, there is a tradeoff

between minimizing development time and time spent to align

a module to new specifications. However, if better specifications could be developed it might reduce the more expensive errors contained within modified modules. In this

case, the reuse of "old" modules could be more beneficial in

terms of cost and effort since the hidden costs would have

been reduced.

One surprising result was that module size did not

account for error proneness. In fact, it was quite the contrary, the larger the module the less error prone it was.

This was true even though the larger modules were more complex. Additionally, theerror prone modules were no more

complex across size grouping than the error free modules.

In general, investigations of the type presented in

this paper relating error and other change data to the

software in which they have occurred is important and

relevant. It is the only method by which our knowledge of

these types of relationships will ever increase and evolve.

4-98

I

I

i

l

I

i

i

I

I

I

l

I

I

!

i

I

I

I

!
Acknowledgments

The authors would like to thank F. McGarry, NASA Goddard,

for his cooperation in supplying the information needed for

this study and his helpful suggestions on earlier drafts of

this paper.

References

(I) Mendenhall,W. and Ramey,M., Statistics for Psycholo_,

Duxbury Press, North Seituate, Mass., 1973, pp. 280-315.

(2) Endres,A.,"An Analysis of Errors and their Causes in

System Programs", Proceedings of the International Conference on Software Engineering, April, 1975, pp. 327-336.

(3) Belady,L.A. and Lehman,M.M., "A Model of Large Program

Development", IBM Systems Journal, Voi.15, 1976, pp.225-251.

(4) Weiss,D.M., "Evaluating Software Development by Error

Analysis : The Data from the Architecture Research Facility", The Journal of Systems and Software, Vol.1, 1979, pp.

57-7O.

(5) Schneidewind,N.F., "An Experiment in Software Error

Data Collection and Analysis", IEEE Transactions on Software

Engineering , Vol. SE-5, No.3, May 1979, pp.276-286.

(6) McCabe, T.J., "A Complexity Measure", IEEE Transactions

on Software Engineering, Vol. SE-2, No. 4, Dec. 1976,

pp.308-320.

(7) Basili,V. and Freburger,K., "Programming Measurement

and Estimation in the Software Engineering Laboratory", The

Journal of Systems and Software, Vol.2, 1981, pp.47-57.

(8) Weiss, D.M.," Evaluating Software Development by

Analysis of Change Data", University of Maryland Technical

Report TR-1120, November 1981.

APPENDIX

4-99

!
OF POOR QUA_J'rY

I

I

I

&lll_j[ffO 1:X NUMII|R

CHANGE REPORT FORM

_O.IllC'r NAMIi ..... OJRIIIN'r OAI"E

1

I

REASON: why .,m _ =lmqle mom_

SIEC'nQN A - IOENTIFICATION

OF.SCRIP'NON: _ _ was

EFP(_I": _ cmxom_n_ far ao_vmm_ am c_enged/()nauae ,,wsiem

llFFOWI": Whm aom_ mnmmmmu (or momwmj m _m In 4Mmmnlml we_ ohmge m "--'-__

SECIlON II. 1"Y1_ OP CX,_qGi (14ow ia thin =_,_,qle lain _wm_hd;'t

F..'vw mmJct_

[] lmm_mm,._ of ,_m_. mmi,m_lWo, m"mm_mmwm

_] Im_mNmunt of _m' m

Oummm=,, of _,mm/.mawv

,,_mNm_ m m,_mm,m _,mw

Wu ma_ tmn om mmmm_ _ff_m_ IW _ _m_ Ym___

I

i

I

i

i

I

1

1

I

I

I

Change Report Foem

I

4-100

I

I
OF POOR QUALITY

FOR ERROR CORRECTIONI ONLY

SECTICfi 0 - VAUOATIQN AND RI_AIR

M_at acaviw* were umd W vaklav me moFam, do_ Ire w_0r, mcl flncl im cau_

J

Pm,li:motln_ roll i_lfll

i

ACCel)tm_ m_g

Pos1-a_cmnall_l ul i

inKmction of ou_gut I

Code mmmg mr wocrmtmlr

:_ mamg ov older I=erscm

Tiks wigt c_4r programman

JSO_-' m_ c_e

$vnem e_ror mem4ps

I_,oWm mectflc *,nrcw _

R._q Oa_memaoon

Trace

OumO

Crom._mti_w list

MC_f t_nniaue

O_er (Ex_ m E)

Acmnm

UgN for

.

V_i_Wman

Aemim

Euecm_l

in OMm_ng

Emir

_Uvibl

Trk_ m

Fkld

tim

Suc*m_'

FW_I

C4me

Whl! WU the wine u_d to isOill _ caum?

_one _ ar less, __one _ fo cme day, __rr_'o then Gem dmf, __hirer _f14

If newt' round, w_ • tt_NalouI¢ uld?__Yl__ No (I;vptlln in E)

Was m_t errm _amd to • :mneus cltm_

__Yes (¢,,Nmle Re_ _VIDm__) __No __Can't tell

_ff_ln _d _e erm,- Mqsf _ wflffir*?

SECTION E * AOOITIONAL INFORMATION

P!em gr_ any . . , " .intormltlo_ roll rely Oe l_lOh,ll m Clml_lllnl _ _ m' _ llml _ lw _ iI_ m

rlmlfir_tIom.

Name: Aulhorllld: Dill: _

Change Report Form

4-i01
III!IIIIiI|IIIiI

SECTION5- DATA COLLECTION
SECTION 5 - DATA COLLECTION

The technical papers included in this section were originally prepared as indicated below.

• Basili, V. R., and D. M. Weiss, "A Methodology for

Collecting Valid Software Engineering Data,"

University of Maryland, Technical Report TR-1235,

December 1982 (reprinted by permission of the

authors)

• Zel_owitz, M. V., "Data Collection and Evaluation

for Experimental Computer Science Research,"

University of Maryland, Technical Memorandum,

November 1982 (reprinted by permission of the

author)

A version of this paper also appears in Empirical

Foundations for Computer and Information Science

(Proceedings), November 1982.

5-1

!
NS?- 24903

Technical Report TR-1235 December 1982

NSG-5123

A METHODOLOGY FOR COLLECTING VALID

SOFTWARE ENGINEERING DATA *

Victor R. Basili

University of Maryland

David M. Weiss

Naval Research Laboratory

*Research supported in part by the National Aeronautics and Space

Administration Grant NSG-5123. Computer support provided in part by

the facilities of NASA/Goddard 5pace Flight Center and the Computer

Science Center at the University of Maryland.

5-3

'_RECEDiNG PAGE CLAN[( NOT FiL_:ED IrA_CzJI_.__ N_NTJONALL_ BLANK
ABSTRACT

An effective data collection method for evaluating software development

methodologies and for studying the software development process is

described. The method uses goal-directed data collection to evaluate

methodologies with respect to the claims made for them. Such claims

are used as a basis for defining the goals of the data collection,

establishin_ a list of questions of interest to be answered by data

analysis, defining a set of data categorization schemes, and designing

a data collection form.

The data to be collected are based on the changes made to the software

during development, and are obtained when the changes are made. To

insure accuracy of the data, validation is performed concurrently with

software development and data collection. Validation is based on

interviews with those people supplying the data. Results from using

the methodology show that data validation is a necessary part of change

data collection. Without it, as much as 50% of the data may be

erroneous.

Feasibility of the data collection methodology was demonstrated by

applying it to five different projects in two different environments.

The application showed that the methodology was both feasible and useful.

5-4

!

II

!

II

II

!

!

!

!

!

II

il

II

!

II

B,

II
!

!

!

!

II

!

!

!

!

i

!

i

!

!

!

!

I

i

A Methodology For Collecting Valid Software

Engineering Data

14eto_- R. Ba.v//_

University of Maryland

/)m4d M. _ei.vs

Naval Research Laboratory

I. Introduction

According to the mythology of computer science, the first computer program ever written contained an error. Error detection and error correction are

now considered to be the major cost factors in software development [I, 2, 3].

Much current and recent research is devoted to finding ways of preventing

software errors. This research includes areas such as requirements definition

[4], automatic and semi-automatic pro£rarn generation [5,6], functional

specification [7], abstract specification [8,9, 10, 11], procedural specification

12], 19, 20,code21,22,specification 23, 24], error [13,detection 14, 15], verification [25], testing [16, [26,!7,27],18],andcoding languagetechniques design

L16, 28,29, 30, 31].

One result of this research is that techniques claimed to be effective for

preventing errors are in abundance. Unfortunately, there have been few

attempts at experimental verification of such claims. The purpose of this paper

is to show how to obtain valid data that may be used both to learn more about

the software development process and to evaluate software development methodologies in a production environment. Previous [15] and companion papers [32]

present the data and evaluation results. The methodology described in this

paper was developeti as part of studies conducted by the Naval Research Laboratory and by NASA's Software Engineering Laboratory [33].

Software Kugineering Ex1_rimentation

The course of action in most sciences when faced with a question of opinion

m to obtain experimental verificatior_ Software engineering disputes are not

usually settled that way. Data from experiments exist, but rarely apply to the

question to be settled. There are a number of reasons for this state of afImrs.

Probably the two most important are the number of potential confounding factors involved in software studies and the expense of attempting to do controlled

studies tn an industrialenvironment involvingmedium or large scale systems.

Rather than attempting controlled studies,we have devised a method for

conducting accurate causal analyses in production environments. Causal analyses are efforts to discover the causes of errors and the reasons that changes

are made to software. Such analyses are designed to provide some insight into

the software development and maintenance processes, help confa'm or reject

clmms made for different methodologies, and lead to better techniques for

prevention, detection, and correction of errors. Relatively few examples of this

Rand of study exist in the literature; some examples are. [34, 35, 4, 15, 36]

To provide useful data, a data collection methodology must display certain

attributes. Since .much of the data of interest for real projects are-collected

5-5

!
during the test phase, complete analysis of the data must await project completion. Although it is important that data collection and validation proceed concurrently with development, the final analysis must be done from a historical

viewpoint, after the project ends.

Developers can provide data as they make changes during development. In

a reasonably well-controlled software development environment, documentation

and code are placed under some form of conflguration control before being

released for use by others than the author. Changes are defined as alterations

to baseIined design, code or documentation.

A key factor in the data gatherin E process is validation of the data as they

become available. Such validity checks result in corrections to the data that

cannot be captured at later times owir_ to the n;_ture of human memory. [37]

Timeliness of both data collection and data validation is quite important to the

accuracy of the analysis.

Careful validation means that the data to be collected must be carefully

specified, so that those suppiyin_ data, those validatinE data, and those performLn.E the analyses will have a consistent view of the data collected. This is especially important for the purposes of those wishLn E to repeat studies in both the

same and different environments.

Careful specification of the data requires the data collectors to have a clear

idea of the goals of the study. Specifying goals is itself an important issue,

since, without goals, one runs the risk ot collectin_ unrelated, meaninEless data.

To obtain insight into the software development process, the data collectors

need to know the kinds of errors committed and the kinds of changes made. To

identify troublesome issues, the effort needed to make each change is necessary. For greatest .usefulness, one would like to study projects from software

production envirorLrnents involvinE teams of programmers.

We may summarize the precedin E as the following six criteria:

I. the data must contain information perrmttin_ identification of the

types of errors and changes made,

2. the data must include the cost of makin_ chanEes and correctin E

errors,

3. data to be collected must be defined as a result of clear specification

of the goals of the study,

4. data should include studies of projects from production environments,

involving teams of programmers,

5. data analysis should be hlstorical, but data must be collected and validated concurrently with development

6. data classification schemes to be used must be carefully specified for

the sake of repeatability of the study in the same and different

environments.

5-6

i

i

i

i

I

I

I

I

I

i

I

i

I

I

I

I

I

i

I
1]. Schema For The Invest_ative ]&{ethodology

Our data collection methodology is goal oriented. It starts with a set.of

goals to be satisfied, uses these to generate a set of questions to be answered,

and then proceeds step-by-step through the design and implementation of a

data collection and validation mechamsm. Analysis of the data yields answers to

the questions of interest, and may also yield a new set of questions. The procedure reties heavily on an interactive data validation process: those supplying

the data are interviewed for validation purposes concurrently with the software

development process. The methodology has been used in two different environments to study five software projects developed by groups with di_erent backgrounds using very different software development methodologies. In both

environments it yielded answers to most questions of interest and some insight

into the development methodologies used.

The projects studied vary widely with respect to factors such as application,

size, development team, methodology, hardware, and support software.

Nonetheless, the same basic data collection methodology was applicable everTwhere. The schema used has six basic steps, listed in the following, with considerable feedback and iteration occurring at several different places.

1. Esta_sh the goals of the data collection

We divide goals into two categories: those that may be used to evaluate a

particular -^'+ _,_=L......... _ development methodology" relative to th= cla_-ns ma,-d= iv: _-,

and those that are common to allmethodologies to be studied.

As an example, a goal of a particular methodology, such as information hid-

[ng [88], might be to develop software that is easy to change. The corresponding

data collection goal is to evaluate the success of the developers in meeting this

goal, i.e.evaluate the ease with which the software can be changed. Goals in this

category may be of more interest to those who are involved in developir_ or

testing a particular methodology, and must be defined cooperatively with them.

A goal that is of interest regardless of the methodology being used is to

characterize changes in ways that permit comparisons across projects and

environments. Such goals may interest software engineers, programmers,

man_gers, and others more than goals that are specific to the success or failure

of a particular methodology.

Consequences of OmitUng Goals

Without goals, one is likely to obtain data in which either incomplete patterns or no patterns are discernible. As an example, one goal of an early study

[15] was to characterize errors. During data analysis, it became desirable to

discover the fraction of errors that were the result of changes made to the

software for some reason other than to correct an error. Unfortunately, none of

the goals of the study were related to this type of change, and there were no

such data available.

2. Develop a list of questions of interest

Once the goals of the study have been established, they may be used to

develop a list of questions to be answered by the study. Questions of interest

de_Ine data parameters and categorizations that permit quantitative analysis of

the data. In general, each goal will result in the generation of several different

questions of interest. As an example, if the goal is to characterize changes,

some corresponding questions of interest are: "What is the distribution of

changes accordin_ to the reason for the change?", "What is the distribution of

5-7

OF POOR QUALITY
changes across system components?", '_'hat is the distribution of effort to

design changes?"

As a second example, if the goal is to evaluate the ease with which software

can be changed, we may identify questions of interest such as: "Is it clear where

a change has to be made in the software?", "Are changes confined to single

modules?", "What was the average effort involved in making a change?"

Questions of interest form a bridge between subjectively-determined goals

of the study and the quantitative measures to be used in the study. They permit

the investigators to determine the quantities that need to be measured and the

aspects of the goals that can be measured. As an example, if one is attempting

to discover how a design document is being used, one might collect data that

show how the document was being used when the need for a change to it was

discovered. This may be the only aspect of the document's use that is measurable.

Goals for which questions of interest cannot be formulated and goals that

cannot be satisfied because adequate measures cannot be defined may be discarded. Once formulated, questions can be evaluated to determine ifthey completely cover their associated goals and if they define quantitative measures.

Finally, questions of interest have the desirable property of forcing the investigators to consider the data analyses to be performed before any data are collected.

Consequences of Omitting Questions Of Interest

Without questions of interest, there may be no quantitative basis for satisfying the goals of the study. Data distributions that are needed for evaluation purposes, such as the distribution of effort involved in making changes, may have to

be constructed in an ad hoc way, and be incomplete or inaccurate.

3. Establish data categories

Once the questions of interest have been established, categorization

schemes for the changes and errors to be examined may be constructed. Each

question generally induces a categorization scheme. If one question is, "What

was the distribution of changes according to the reason for the change?", one

will want to classify changes according to the reason they are made. A simple

categorization scheme of this sort is eTrov corrections vs. no_z-e_-ror coy"rectio'n_

(hereafter ca/led nzodifications).

Each of these categories may be further subcategorized according to reason. As an example, modifications could be subdivided into those modifications

resulting from requirements changes, those resulting from a change in the

development support environment (e.g. compiler change), planned enhancements, optimizations, and others.

Such a categorization permits characterization of the changes with respect

to the stabilityof the development environment, with respect to different kinds

of development activities, etc. When matched with another categorization such

as the difficultyof making changes, this scheme also reveals which changes are

the most difficultto make

Each categorization scheme should be complete and consistent, i.e.every

change should fit exactly one of the subcategories of the scheme. To insure

completeness, the category "Other" is usually added as a subcategory Where

some changes are not suited to the scheme, the subcategory "Not Applicable"

may be used. As an example, ifthe scheme includes subcategories for different

levels of effort in isolating error causes, then errors for which the cause need

5-8

I

I

I

I
il

II

if

II

e

II

II

I

!

!

I

I

I

II

If

II

I

H

not be isolated (e._.clericalerrors noticed when reading code) belong in the

"Not Applicable" subcategory.

Consequences Of Not Defining Data Categories Before CollectingData

Omitting the data categorization schemes may result in data that cannot

later be identifiedas fittingany particularcategorization. Each change then

tends to define itsown category, and the result isan overwhelming multiplicity

of data categories,with littledata ineach category.

4. Design and test data collection form

To provide a permanent copy of the data and to reinforce the programmers' memories, a data collectionform is used. Form design was one of the

trickiestparts of the studies conducted, primarily because forms represent a

compromise among conflictingobjectives. Typical conflictsare the desire to

collecta complete, detailedset of data that may be used to answer a wide range

of questions of interest,and the need to minimize the time and effortinvolved in

supplying the data. Satisfyingthe former leads to large, detailed forms that

require much time to fillout. The latterrequires a short form organized so that

the person supplying the data need only check offboxes.

Including the data suppliers in the form design process is quite beneficial.

Complaints by those who must use the form are resolved early (i.e.before data

collectionh_inq____ ....,,the form ,=_,y_o _l_,o_ _^ _,_,,_ ...._Ao_ __,_^' _'^___data .....'_^--

(e.g.for use as in configuration management), and the data suppliers feelthey

are a useful part of the data collectionprocess.

The forms mu::t be constructed so that the data they contain can be used to

answer the questions of interest. Several design iterationsand test periods are

generally needed before a satisfactorydesign isfound.

Our principalgoals inform design were to produce a form that:

I. fiton one piece of paper,

2. could be used in severaldifferentprogramming environments, and

3. permitted the programmer some flexibilityin describing the

change.

Figure I shows the lastversion of the form used for the SEL studies. (An

earlierversion of the form was significantlymodified as a result of experience

gained in the data collectionand analysisprocesses.) The firstsections of the

form request textual descriptions of the change and the reason it was made.

Following sections contain questions and check-off tables that reflectvarious

categorization schemes.

As an example, a categorizationof time to design changes isrequested in

the firstquestion followingthe descriptionof the change. The completer of the

form is given the choice of 4 categories (one hour or less,one hour to one day,

one day to three days, and more than three days) that cover allpossibilitiesfor

design time.

Consequences Of Not Using A Data CollectionForm

Without a data collectionform, it is necessary to rely on the developer's

memomes and on perusal of early versions of design documentation and code to

identifyand categorize the changes made. This approach leads to incomplete,

unaccurate data.

5-9

!
PROJECT NAME

CHANGE REPORT FORM

NUMBER

CURRENT DATE

I

I

I

REA.SON: Why was the change made?

SECTION A - iDENTiFICATION

DESCRIPTION: What change was made?

EFFECT: What ¢omponents (or documtmt3| Ire (:hinged? (Include ve_ioni

EFFORT: What additiomd components (or documents) were examined in demrmining what change was neKIKI?,

(Month Day Year)

Nell for change determined on ....

Change started on ............

What was the effort in p_on time required to understand and implement the change?

_1 hour or less. _1 hour to I day. _I day to 3 days. more 1:ban 3 days

SECTION B - TYPE OF CHANGE (How is this change best charac'mrized?|

C] ¢.rmr correction

C] Planned enhancement

t'l Imolemenration of requirements change

{_ Improvement of tJarity, maintainability, or documentation

I_. ImDrovement of user ,_rvices

Was more than one component affected by the chan_l? Yes

r-i Insertion/deletion of debug code

r'l Optimization of time/space/accuracy

r-I Adaptation 1:o emironment change

I"=1 Other (Explain in E]

NQ .,.

FOR ERROR CORRECTIONS ONLY

SECTION C - TYPE OF ERROR (How is this error bes¢ ¢harac':erized?}

O Recluirements incorrect or misinterpreted _ Misunderccanding of ex_rnai envlronment, except lanquage

,"; Functional soecific:tions incorre¢,: or misinterpreted r-I Error in u_m of Drogramming iangua,eje/compller

Oes,gn error, involving several comt,,onents I-I Cieric'al error

Error in the design or {rnc_iemen_3tion of a single component I_l Other (Explain in E|

FOR DESIGN OR iMPLEMENTATiON ERRORS ONLY

If the error was in design or imDlementation:

The error was a mista_;:en ass,JmOtlon GP_OUI_ the value or _tnJc_Jre of data

The error was a mistake tn control I_,c or c_mputation of an exDression

i

I

I

I

I

I

I

I

I

I

I

I

l

SlO-2 (6/7 e}

Figure I SEL Change Report Form (front)

5-i0

ORiG'.,NAL PAGe. [3

OF POOR QUALITY

I

I

I
OR_GI?_AL

OF POOR

PAGE IS

QUALITY

I

i

I

I

I

i

i

I

I

I

I

i

I

l

FOR ERROR CORRECTIONS ONLY

SECTION 0 - VALIDATION ANO REPAIR

Whll activities _m uwd to vaGdam _._e program, demct '_e error, and find its c,tuse?

Activities

Used for

Validation

Prl-lCClDtlncl teSt runs

A¢ceDtan¢ll tllstlng

Pos [.acc**D tance use

In_tiQn of o*JclDut

Code reading by _rogram.ner

P-,--_ m_lting by other person

T,dks vim other pmgrammen

System error mMsegls

Prc_ect Sl_lCifi¢ error m_tacjes

Reading docu_mtati¢_

Tr_'e

Oucrto

Ctou.m (.-r_ncelarmbum list

P.*oo. = :ec_.nique

Activities

Successful

in Oet¢,¢ dng

Erl_r Symptoms

._h** (ExWoin ;n E) z T .

What was the tim= used to isolam _e cause?

_o¢_1_ hour or lesS. _one how tO one day.

If nmmr found, was = workamund uwd?_Y_ No (Explain in E}

V;.'t,i this ar:_r n_lamd to a _rt._ious change?

_Yes ICI_rRe Rer, ort #/Date ) _No _C_n't tell

_._.e,1 did the error enter _he wstem?

_requkements _tunctional sl_.c_ _des_ln _ding and test

Activities

Tried to

Find

Cause

_more than one day. _mr, er found

_.__ot her Can't tell

Activities

Succmsfui

in Finding

Cease

SECT;ON E - AOOITIONAL INFORMATION

PI_.p. qive an'/ir_tormecion that may be hel;)tu| in ¢atecjorizing tl_e error or cnan_e, and understanding its ¢au_ and its

r4ml;i_tlonS.

_uthorlzec_: _)aTe;

I

I

I

Figure i SEL Change Report Form (back)

5-Ii
5. Collect and validate data

Data are collected by requtrir_ those people who are making software

changes to complete a change report form for each change made, as soon as the

the change is completed. VaLidation consists of checking the forms for correctness, consistency, and completeness. As part of the validation process, in cases

where such checks reveal problems the people who filled out the forms are

interviewed. Both collection and validation are concurrent with software

development; the shorter the lag between programmers completing forms and

being interviewed concerning those forms, the more accurate the data.

Perhaps the most significant problem during data collection and validation

is insurir_ that the data are complete, i.e. that every change has been described

on a form. The better controlled the development process, the easier this is to

do. At each sta_e of the process where conf_uration control is imposed, change

data may be collected. Where projects that we have studied use formal

configuration control, we have integrated the configuration control procedures

and the data collection procedures, using the same forms for both, and taking

advantage of conf_uration control procedures for validation purposes. Since all

changes must be reviewed by a conf_uration control board in such cases, we are

guaranteed capture of all changes, i.e. that our data are complete. Furthermore, the data collection overhead is absorbed into the configuration control

overhead, and is not visible as a separate source of irritation to the developers.

Consequences Of Omitting Validation

One result of concurrent development, data collection,and data validation

isthat the accuracy of the collectionprocess may be quantified. Accuracy may

be calculated by observing the number ofmistakes made in completing data collectionforms. One may then compare, for any data category, pre-vaLidationdistributions with post-validationdistributions.We call such an analysis a validation analysis. The validationanalysisof the SEL data shows that itispossiblefor

inaccuracies on the order of 50% to be introduced by ornittmg validation. To

emphasize the consequences of omitting the validationprocedures, we present

some of the resultsof the validationanalysisof the SEL data in'sectionIll.

8. Analyze Data

Data are analyzed by calculating the parameters and distributionsneeded

to answer the questions of interest. As an example, to answer the question

"What was the distributionof changes according to the reason for the change?",

a distributionsuch as that shown in figure2 might be computed from the data.

Application of the Schema

Applying the schema requires iterating among the steps several times.

Defining the goalsand establishingthe questions of interestare tightlycoupled,

as are establishin_the questions of interest,designing and testing the form(s),

and collectingand validatingthe data. Many of the considerations involved in

trnplementing and integrating the steps of the schema have been omitted here

so that the reader may have an overview of the process. The complete set of

goals, questions of interest,and data categorizations for the SEL projects are

shown in a cornpa_ntonpaper [32].

5-12

l

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

l
t

I

,m

i

°

l

!w

to.

I

I _

i

_J

I [,z

_a

OF POOR QUALF_

.- • ..

--" _ _ _ _.

• !

::-:

!•

I,

,---

I

i

_c

5-13

r.

r

u%

-n

0

_n

C'4

_2

N

©
Support Procedures and Facilities

In addition to the activitiesdirectlyinvolved in the data collectioneffort,

there are a number of support activitiesand facilitiesrequired. Included as

support activitiesare testing the forms, collection,and validationprocedures,

training the programmers, selectinga data base system to permit easy analysis

of the data, encoding and entering data into the data base, and developing

analysisprograms.

[] DetailsOf SEL Data Collection And Validation

In the SEL environment, program librarieswere used to support and control

software development. There was a full-timelibrarianassigned to support SEL

projects. Allproject library changes were routed through the librarian.In general,we define a change to be an alterationto baselined design, code, or documentation. For ZEL purposes, only changes to code, and documentation contained in the code, were studied. The program librariesprovided a convenient

mechanism foridentifyingchange s.

Each time a programmer caused a librarychange, he was required to complete a change report form (figurei). The data presented here are drawn from

studies of three differentSEL projects,denoted SELl, SEI2, and SEL3. The pro-

" cessing procedures were as follows.

I. Programmers were required to complete change report forms for all

changes made to libraryroutines.

2. Programs were kept inthe project libraryduring the entire testphase.

. After a change was made a completed change report form describing

the change was submitted. The form was firstinformally reviewed by

the project leader. It was then sent to the SEL library staffto be

logged and a unique identifierassigned to it.

° The change analyst reviewed the form and noted any inconsistencies,

omissions, or possible miscategorizations. Any questions the analyst

had were resolved in an interview with .theprogrammer. (Occasionally

the project leader or system designer was consulted rather than the

individualprogrammer.)

The change analyst revised the form as indicated by the resultsof the

programmer interview,and returned itto the librarystafffor further

processing. Revisions often involved cases where several changes were

reported on one form. In these cases, the analyst insured that there

was only one change reported per form; this often involved fillingout

new forms. Forms created inthisway are known as generated forms.

(Changes were considered to be different ff they were made for

differentreasons, ifthey were the result of differentevents, or ifthey

were made at substantiallydifferenttimes (e.g.several weeks apart).

As an example, two differentrequirements amendments would resultin

two differentchange reports, even if the changes were made at the

same time in the same subroutine.)

5-14

I

i

I

i

I

I

II

I

I

i

I

I

I

I

II

I

I

I

!
il

II

II

II

II

I

I

II

II

II

l

II

I

i

II

I

!

I

Occasionally,one change was reported on several differentforms. The

forms were then merged into one form, again to insure one and only

one change per form. Forms created in this way are known as cam-

&_ed forms

. The library staffencoded the form for entry into the (automated) SEL

data base. A preliminary, automated check of the form was made via a

set of data base support programs. This check, mostly syntactic,

ensured that the proper kinds of values were encoded into the proper

fields,e.g. that an alphabetic character was not entered where an

integer was required.

7. The encoded data were entered into the SEt data base.

8. The data were analyzed by a set of programs that computed the neces-

•sary distributionsto answer the questions of interest.

Many of the reported SEL chan_es were error corrections. We define an

error to be a discrepancy between a specificationand its implementation.

Although itwas not always possible toidentifythe exact location of an error, it

was always possible to identify exactly each error correction. As a result,we

generally use the term error to mean error correction.

For data validationpurposes, the most important parts of the data collection procedure are the review by the change analyst,and the associated programmer interview to resolve uncertaintiesabout the data.

The SEL validationprocedures afforded a good chance to discover whether

validation was really necessary; it was possible to count the number of rniscategorizations of changes and associated misinformation. These counts were

obtained by counting the number of times each question on the form was

incorrectlyanswered.

An example ismisclassificationsoferrors as clericalerrors.(Clericalerrors

were defined as errors that occur in the mechanical translationof an item from

one format to another, e.g. from one coding sheet to another, or from one

medium to another, e.g.coding sheets to cards.)For one of the SEL projects,46

errors originallyclassifiedas clericalwere actually errors of other types. (One

of these consisted of the programmer forgettingto include several linesof code

in a subroutine. Rather than clerical,this was classifiedas an error in the

design or implementation of a singlecomponent of the system.) Initially,this

project reported 238 changes, so we may say that about 19Z of the original

reports were misclassifiedas clericalerrors.

The SEL validationprocess was not good for verfiytngthe completeness of

the reported data. We cannot tellfrom the validationstudieshow many changes

were never reported. This weakness can be eliminated by integrating the data

collectionwith stronger conf_uration control procedures.

Validation Differences Among SE_ Projects

As experience was gained in collecting,validating,and analyzing data for

the SEL projects, the qualityof the data improved significantly,and the validation"procedures changed slightly. For SELl and SEL2, completed forms were

examined and programmers interviewed by a change analyst within a few weeks

(typLcally3 to 6 weeks) of the time the forms were completed. For project ZEL2,

the task leader (lead programmer forthe project) examined each form before

the change analysts saw it.

5-15
Project SEL3 was not monitored as closely as SELl and SEL2. The task

leader, who was the same as for SEL2, by then understood the data categorization schemes quite well and again examined the forms before sending them to

the SEL. The forms themselves were redesigned to be simpler but stillcapture

nearly allthe same data. Finally,several of the programmers were the same as

on project SEI2 and were experienced in completing the forms.

Est_aat.ing Inaccuracies In The Data

Although there isno completely objective way to quantify the inaccuracy in

the validated data,we believe itto be no more than 57,for SELl and SEL 2. By

this we mean that no more than 5Z of the changes and errors are misclassifled

in any of the data collection categories. For the major categories, such as

whether a change isan error or modification,the type ofchange, and the type of

error,the inaccuracy isprobably no more than 8%.

For SEL3, we attempted to quantify the resultsof the validationprocedures

more carefully. After validation,forms were categorized according to our

confidence in theiraccuracy. We used four categories:

(i) Those forms for which we had no doubt concerning the accuracy of

the data. Forms in this cateogry were estimated to have no more

than a I% chance of inaccuracy.

(2) Those forms for which there was littledoubt about the accuracy of

the data. Forms in this category were estimated to have at most a

10F,chance ot an inaccuracy.

(3) Those forms for which there was some uncertaincy about the accuracy,with an estimated inaccuracy rate of no more than 30%.

(4) Those forms for which there was considerable uncertaincy about the

accuracy, with an estimated inaccuracy rate of about 50_o.

Applying the inaccuracy rates to the number of forms tn each category gave us

an estimated inaccuracy of at most 3Z inthe validated forms for SEL3.

Prevalent Mistakes In Completing Forms

Clear patterns of mistakes and misclassiflcationsin completing forms

became evident during validation. As an example, programmers on projects

SELl and SEL2 frequently included more than one change on one form. Often

this was a result of the programmers sending the changes to the library as a

group.

Comparative ValidationResults

Figure 3 provides an overview of the resultsof the validationprocess for the

3 SEL projects The percentage of originalforms that had to be corrected as a

result of the validationprocess is shown. As an example, 3270 of the originally

completed change report forms for SEL3 were corrected as a result of validation. The percentages are based on the number of original forms reported

(since some forms were generated, and some combined, the number of changes

reported after validationis differentthan the number reported before validation). Figure 4 shows the fractionof generated forms expressed as a percentage

of totalvalidatedforms.

Figure 3 shows that pre-validationSEL3 forms were sigmficantlymore accurate than the pre-validationSELl or SEL2 forms. When the generated and combined forms are also considered, the pre-validattonSEL3 data appear to be considerably better then the pre-validationdata for either of the other projects. We

5-16

I

I

I

I

I

I

I

I

I

I

I

I

l

l

I

l

I

I

I
!

l

!

!

!

l

I

!

!

l

l

I

l

!

!

l

i

believe the reasons for this are the improved design of the form, and the farniIiarityof the task leader and programmers with the data coUection process.

(Generated forms are shown in _ure 4. Combined forms for allof the projects

represented a very small fractionof the totalvalidated forms.)

These (overall)results show that careful validation,including programmer

interviews, is essential to the accuracy of any study involving change data.

]_hermore, itappears that with weU-designed forms, and programmer trainb_, there is improvement with time inthe accuracy of the data one can obtain.

We do not believe that itwillever be possible to dispense entirelywith programmer interviews, however.

Erroneous Classifications

Table 1 shows misclassifications of error as modifications and modifications

as errors. As at/example, for SELl, 14_, of the original forms were classified as

modifications, but were actually errors. Without the validation process, considerable inaccuracy would have been introduced into the initial categorization of

changes as modifications or errors.

Table 2 is a sampling of other kinds of classification errors that could contribute significantly to inaccuracy in the data. All involve classification of an

error into the wrong subcategory. The first row shows errors that were classified

by the programmer as clerical, but were later reclassified as a result of the validation process into another category. For SELl, significant inaccuracy (19_,)

would be introduced by omitting the vatidationprocess.

Table 3 is similar to table _.,but shows misclassificationsinvolving

modifications. The firstrow shows modifications that were classifiedby the.programrner as requirements or specificationschanges, but were reclassifiedas a

result ofvalidation.

Variation In MisclassificaUon

Data on misclassifications of change and error type subcategories, such as

shown in table 2, tends to vary considerably among both projects and subcategories. (Misclasssification of clerical errors as shown in table 2 is a good

example.) This is most likely because the misclassLelcations represent biases in

the judgements of the programmers. It became clear during the validation process that certain programmers tended toward particular misclassifications.

The consistency between projects SEL2 and SEL3 in table 2 probably occurs

because both projects had the same task leader, who screened all forms before

sending them to the SEL for validatiorL

5-17

!
,"3

0 50"

o II 4g T 3_ F F

2g

S

%g

v 3g fD0RM$ Ig

55

51

32

SELl SEL_ SEL_

PROJECT

FIGURE3 CC_RECTE9 FORMS

3_

I?

IS

SEL_ SEL2 5EL3

=ROJECT

5-18

IIIIIIIIIIIlIIIllIl
l

I

I

l

I

l

I

I

I

I

l

l

I

I

I

I

I

I

I SELl SEL2 SEL3

Modificationsclassifiedas errors iZ [ 5% 'iless than I%

Errors classifiedas modifications 14% I 5% I 2%

Table I Erroneous Modificationand Error Classifications

(Percent of OriginalForms)

OriginalClassification

ClericalError

(Use of)Programming Language

Incorrect or Misinterpreted Requirements

,Design Error

SELl SEL2 ! SEL3

19% 7_ I 6Y.

O_ 5_ I 3%

0% I less than I%

_,. !%

Table 2 Typical Error Type Misclassifications

(Percent of OriginalForms)

Requirements or specificationchange

Design change

Optimization

Other

SELl

I%

8%

8%

3%

SEL8

lessthan !%

I%

less than I%

less than I%

Table 3 Erroneous ModificationClassifications

(Percent of OriginalForms)

5-19

!
Conclumona Concerning Validation

The preceding sections have shown that the validation process, particularly

the programmer interviews_ are a necessary part of the data collection methodology. Inaccuracies on the order of 50Z may be introduced without this form of

validation. Furthermore, it appears that with appropriate form design and programmer experience in completing forms, the inaccuracy rate may be substantially reduced, although it is doubtful that it can be reduced to the level where

programmer interviews may be omitted from the validation procedures.

A second significant conclusion is that the analysis performed as part of the

validation process may be used to guide the data collection project; the analysis

results show what data can be reliably and practically collected, and what data

cannot be. Data collection goals, questions of interest, and data collection forms

may have to be revised accordingly.

IV. Recommendations For Data Collectors

We believe we now have sufficient experience with change data collection to

be able to apply it successfully in a wide variety of environments. Although we

have been able to make comparisons between the data collected in the two

environments we have studied, we would like to make comparisons with a wider

variety of environments. Such comparisons will only be possible if more data

become available. To encourage the establishment of more data collction projects, we feel it ts important to describe a successful data collection methodology, as we have done in the preceding sections, to point out the pitfalls involved,

and to suggest ways of avoiding those pitfalls.

Procedural Lessons Learned

ProbLems encountered in various procedural aspects of the studies were

the most difficult to overcome. Perhaps the most important are the following.

i. Clearly understanding the working environment and specifying the

data collection procedures were a key part of conducting the investigation. Misunderstanding by the programmer of the circumstances that

require him/her to file a change report form will prejudice the entire

effort. Prevention of such misunderstandings can partly be accomplished by training procedures and good forms design, but feedback to

the development staff, i.e. those filling out the data collection forms,

must not be omitted.

2. Similarly, misunderstanding by the change analyst of the circumstances that required a Change to be made will result in

misclassiflcations and erroneous analyses. Our SEL data collection was

helped by the use of a change analyst who had previously worked in the

NASA environment and understood the application and the development procedures used.

3. Timely data validation through interviews with those responsible for

reporting errors and changes was vital, especially during the first few

projects to use the forms. Without such validation procedures, data

will be severely biased, and the developers wall not get the feedback to

correct the procedures they are using For reporting data.

4. Mimmtz/ng the overhead imposed on the people who were required to

complete change reports was an important factor in obtainmg complete and accurate data. Increased overhead brought increased reluctance to supply and discuss data. In projects where data collection has

been :ntegrated with configuration control, the visible data collection

5-20

I

I

I

I

I

I

I

I

I

I

I

l

I

l

I

l

l

l
I

I

I

I

I

I

I

I

I

I

l

I

I

I

I

I

I

I

l

.,

and validationoverhead is significantlydecreased, and is no longer an

important factor in obtaining complete data. Because configuration

control procedures for the SEL environment were informal, we believe

we did not capture allSEL changes.

In cases where an automated data base is used, data consistency and

accuracy checks at or immediately prior to analysisare vital.Errors in

encoding data for entry into the data base willotherwise bias the data.

Nonprocedural Lessons Learned

In addition to the procedural problems involved in desinging and implementing a data collectionstudy, we found several other pitfallsthat could have

strongly affected our resultsand theirinterpretation.They are listedin the following.

I.

,

,

4,

Perhaps the most significantof these pitfallswas the danger of interpreting the results without attempting to understand factors in the

environment that might affect the data. As an example, we found a

surprisinglysmall percentage of interface errors on allof the SEL projects. This result was surprising since interfaces are an often-cited

source of errors. There was also other evidence in the data that the

software was quite amenable to change. In trying to understand these

results,we discussed them with the principal designer of the SEL projects (allof which had the same application),itwas clear from the discussion that as a result of their experience with the application,the

designers had learned what changes to expect to theirsystems, organized the design so that the expected changes would be easy to make,

and then re-used the design from one project to the next. Rather than

misinterpreting the data to mean that interfaceswere not a significant

software problem, we were led to a better understanding of the

environment we were studying.

A second pitfallwas underestimating the resources needed to validate

and analyze the data. Understanding the change reports well-enough

to conduct meaningful, efficientprogrammer interviews forvalidation

purposes initiallyconsumed considerable amounts of the change

analysts'time. Verifying that the data base was internallyconsistent,

complete, and consistent with the paper copies of reports was a conttnuingsource of frustrationand sink for time and effort.

A third potential pitfallin data collectionisthe sensitivityof the data.

Programmers and designers sometimes need to be convinced that

error data will not be used against them. This did not seem to be a

significantproblem on the projects studied for a variety of reasons,

including management support, processing of the error data by people

independent of the project,identifyingerror reports in the analysis

process by number rather than name, informing newly hired project

personnel that completion of error reports was considered part of

theirjob, and high project morale. Furthermore, project management

did not need error data to evaluate performance.

One problem for which there isno simple solutionisthe Hawthorne (or

observer) effect [39], When project personnel become aware that an

aspect of theirbehavior isbeing monitored, their behavior willchange.

Iferror monitoring is a continuous, long-term aetivltythat is part of

the normal scheme of software development, not associated wlth

evaluation of programmer performance, this effect may become

tnstgmficant. We believethiswas the case with the projects studied.

5-21
5. The sensitivity of error data is enhanced in an environment where

development isdone on contract. Contractors may feelthat such data

are proprietary. Rules for data collectionmay have to be contractually

specified.

Avoiding Data Collection PR/alls

In the foregoing sections a number of potential pitfallsin the data collection process have been described. The following listincludes suggestions that

help avoid some of these pitfalls.

I. Select change analysts who are familiarwith the environment, application,project,and development team.

2. Establish the goals of the data collectionmethodology and define the

question_ of interest before attempting any data collection.Establishing goals and deflmng questions should be an iterativeprocess performed in concert with the developers. The developers' interests are

then served as well as the data collector's.

3. For initialdata collectionefforts,keep the set of data collectiongoals

small. Both the volume of data and the time consurried in gathering,

validating,and analyzing itwillbe unexpectedly large.

4. Design the data collectionform so that itmay be used for configuration

control,so that itistailoredto the project(s)being studied,so that the

data may be used for comparison purposes, and so that those filling

out the forms understand the terminology used. Conduct training sessionsin fillingout forms fornewcomers.

5. Integrate data collection and validation procedures into the

configuration control process. Data completeness and accuracy are

thereby improved, data collectionis unobtrusive, and collectionand

validationbecome a part of the normal development procedures. In

cases where configuration control is not used or is informal, allocate

considerable time to programmer interviews,and, ifpossible, documentation search and code reading.

6. Automate as much of the data analysisprocess as possible

Limitations

It has been previously noted that the main limitation of using a goaldirected data collectionapproach in a production software environment isthe

inabilityto isolatethe effectsof single factors. For a variety of reasons, controlledexperiments that may be used to test hypotheses concerning the effects

of single factors do not seem practical. Neither can one expect to use the

change data from goal-directed data collectionto test such hypotheses.

A second major limitationts that lost data cannot be accurately recaptured. The data collected as a result of these studies represent fiveyears of

data collection.During that time there was considerable and continuing considerationgiven to the appropriate goals and questions of interest. Nonetheless,

as data were analyzed, it became clear that there was information that was

never requested but that would have been useful. An example is the length of

time each error remained in the system. Programmers correcting their own

errors, which was the usual case, can supply this data easily at the time they

correct the error. Our attempts to discover error entry and removal times after

the end of development were fruitless.(Error entry times were particularly

difficultto discover.) Given such data, one could isolateerrors that were not

easilysusceptible to detection. This type of example underscores the need for

5-22

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I
!

!

I

I

I

I

I

I

I

I

I

I

i

I

!

I

i

careful planning prior to the startof data collection.

Recommendations That May Be Provided To the Software Developer

The nature of the data collectionmethodology and the environments in

which itcan be used do not generally permit isolationof the effectsofparticular

factors on the software development process. The results cannot be used to

suggest that controlling a particular factor in the development process will

reduce the quantity or cost of particularkinds o_ errors. We have found that the

patterns found in the data do suggest that certain approaches, when applied in

the environment studied,willimprove the development process.

As an example, in the SEL environment neither external problems, such as

requirements changes, nor global problems, such as interface design and

specification,were significant.Furthermore, the development environment was

quite stable. Most problems were associated with the individualprogrammer.

The data show that in the SEL environment itwould clearlypay to impose more

control on the process of composing individualroutines. Since "detectingand

correcting most errors was apparently quite easy in the overwhelming majority

of cases, more attention should be paid to preventing errors from entering the

code initially.

Conclusions Concerning Data CollectionFor Methodology Evaluation Purpolmg

The data collectior-,schema F'-_'-'-___ ......t,t,...................._--

)ects in two different environments. We have been able to draw the following

conclusions as a result of designing and implementing the data collection

processes.

i. In allcases, ithas been possible to collectdata concurrently with the

software development process in a software production environment.

2. Data collectionmay be used to evaluate the applicationof a particular

software development methodology, or simply to learn more about the

software development process. In the former case, the better defined

the methodology, the more precisely the goals of the data collection

may be stated.

3. The better controlledthe development process, the more accurate and

complete the data.

4. For all projects studied, it has been necessary to validate the data,

including interviewswith the project developers.

5. As patterns are discerned in the data collected, new questions of

interest emerge. These questions may not be answerable with the

availabledata, and may require establishingnew goals and questions of

interest.

Motivations For Conducting _mflar Studies

The difficultiesinvolved in conducting large scale controlled software

engineering experiments have as yet prevented evaluations of software development methodologies m the environments where they are often claimed to work

best. As a result,software engineers must depend on tess formal techmques

that can be used inreal workmg environments to establishlong-term trends. We

vlew charge analysts as one such technique and feelthat more techniques, and

many more resultsobtained by applying such techniques, are needed.

5-23

!
Acknowledgments

The authors thank the many people at NASA/GSFC and Computer Sciences

Corporation who filledout forms and submitted to interviews, especially Jean

Grondalski and Dr. Gerald Page, and the librarians, especially Sam DePriest.

We thank Dr. John Gannon, Dr. Richard Meftzer, Frank McGarry, Dr. Gerald

Page, Dr. David Parnas, Dr. John Shore, and Dr. Marvin Zelkowltz for their many

helpful suggestions.

Deserving of special mention is Frank McGarry, who had sufficient foresight

and confidence to sponsor much of this work and to offer his projects for study.

References

1. B. Boehrn and Others, In/o_ation processing/Data Automa2ion Iraplicat_ns .Of Air Force Comma_nd and Control Requirements in the 1980's

(CCIP-85), Space and Missile Systems Organ/zation, Los Angeles (February

1972). Technology Trends: Software

2. B. Boehrn, "Software and Its Impact: A Quantitative Assessment," Datamation 19(5) pp. 48-59 (May 1973).

3. R. Woiverton, "The Cost Of Developing Large Scale Software," IEEE Trans.

ComImzte_ 23(6) (1974).

4. T. Bell, D. Bixler, and M. Dyer, "An Extendable Approach to Computer-Aided

Software Requirments Engineering," [EEE Trm_s. Softv#-re Engineering

SE-8(1) pp. 49-60 (January 1977).

5. A Ambler, D. Good, J. Browne, and et. a/., "GYPSY: A Language for

Specification and Implementation of Verifiable Programs," t_oc. o/The ACM

Can/erence on LanFuage Design )'or Reliable Softy#ave, pp. I-I0 (March

1977).

6. Z. Manna and R. Waldinger, "Synthesis: Dreams => Programs," [EEE Trans.

Software Engineering Sg-5(4) pp. 294-329 (July 1979).

7. K Hemnger, "Specifying Requirements for Complex Systems: New Techtuques and Their Application," IEEE Trans. Soft,are _ngineering SE-6 pp.

2-13 (January 1980),

8. DI L Parnas, "A Technique For Software Module Specification With Examples," Comm. ACM 15(5) pp. 330-336 (May 1972).

9. J. Guttag, "The Specification and Application to Programming of Abstract

Data Types," CSRG-59, Umversity of Toronto Dept. of Computer Science

Computer Systems Research Group (1975).

10. J. Guttag, "Abstract data types and the development of data structures,"

Comm. ACM 20 pp. 396-404 (June 1976).

!i ]3. Liskov and S. Zilles, "Specification Techniques for Data Abstraction,"

[EEE Tra_. So/t_uare Engineering SE-I(I) pp. 7-19 (March 1975).

12. H. Mills,R. Linger, and B. Witt, StT"uctured Programming TheoT%/and Practzce, Addison-Wesley, Reading (1979).

:3 S. Came and E. Gordon, "PDL - A tool for software design," Proc. Na2. Com-

;nzter Con/, pp. 27!-276 (1975)

5-24

l

I

I

I

I

I

I

I

I

l

I

I

I

I

I

I

I

I

I
I

I

l

l

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

!4. H. Elovitz, "An Experiment In Software Engineering: The Architecture

Research FacilityAs A Case Study," Proc. Fourth IntntL Covtf.Soft_uere

e.@i_eemng, pp. 145-152 (1979),

15. D. Weiss, "Evaluating Software Development by Error Analysis: The Data

from the Architecture Research Facility,"J. S_stern._cLnd Soft--re 1 pp.

57-70 (1979).

16. E, W. Dijkstra,A D_cipl{ne of Prog'rcLnzrtti_tg, Prentice-Hall,Englewood Cliffs

(1978).

17. R.W. Floyd, "Assigning Meanings to Programs," Prsc. Svntposi_rn _t Applied

Methema_es XIX pp. 19-$2 AlxlericanMathematical Society,(1967),

18. C. A. R. Hoare, "An Axiomatic Basis for Computer Programming," Co_tn%,

ACM 12(10) p,p. 576-580 (October 1989),

19, F. Baker, "Chief Programmer Team Management of Production Programruing,"IBM Systems Jo_ II(i)pp. 56-78 (1972).

20. E. W. Dijkstra, "Notes on Structured Programming," in Structured Progr,,m.Tn._g,Academic Press, London (1972).

21. D.E. Knuth, "Structured Programming With Go To Statements," Conzpu/_n_j

S_n-uelm 8(4) pp. 281-301 (December 1974).

22. H. Mills,"Chief Programmer Teams: Principles and Procedures," FSC 71-

5108, IBM Federal Systems Division(1971).

23. H. Mills,"Mathematical Foundations for Structured Programming," FSC 72-

8012, IBM Federal Systems Division(1972).

24. N. Wirth, "Program Development by Stepwise Refinement," Comm. ACM

14(4) pp. 221-227 (April1971).

25. E. Satterthwmte, "Debugging Tools for High-Level Languages," Soft_u,,re-

}_tice and Ezper_nce _(8) pp. 197-217 (July-September 1972).

28 W Howden, "Theoretical and Empirical Studies of Program Testing," P_oc.

TA4.rdlwtntl.Conf. Soft_uare Engineemng, pp 305-310 (May 1978).

27. J. Goodenough and S. Gerhart, "Toward a theory of test data selection,"

Proc. ITttntl.Conf. Reliable Software, pp. 493-510 (1975).

28. J. Gannon, "Language Design to Enhance Programming Reliability,"CSRG47, University of Toronto Dept. of Computer Science ComputerSystems

Research Group (1975).

29. J Gannon and J. Homing, "Language Design for Programming Reliability,"

IEEE Prans. Soft_,_:re E_. SE-l(2)(June 1975).

30. C. A. R. Hoare and N. Wirth, "An Axiomatic Definition of the Programming

Language Pascal," Acta Infor'r, tatica 2 pp. 335-855 (1978).

31. K. Jensen and N. Wirth, Pascal User MaTzual and Report Second Edition,

Springer-Verlag, New York (1974).

32. V. Basiti and D. Weiss, "Evaluating Software Development By Analysis of

Changes: The Data From The Software Engineering Laboratory," , ().

33. V. Bastli, M. Zetkowitz, F. McGarry, and others, "The Software Engineering

Laboratory," Report TR-535, University of Maryland (May !977)

34 B Boehm, "An Experiment in Small-Scale Application Software Engineering," Report TRW-SS-80-01, TRW (I 980)

5-25
35 A. Endres, "Analysis and Causes of Errors in Systems Programs," Pro¢

Intntl. Con/. Reliable Soft?_are, pp. 327-336 (1975).

36. V. Basili and D. Weiss, "Evaluation of a Software Requirements Document By

Analysis of Change Data," Proc. ]_fth lntntl. Conj. Soft_uare Engineering,

pp. 314-323 (March 1981).

37. G. Miller, "The Magical Number Seven, Plus or Minus Two: Some Limits On

Our Capacity For Processing Information," TAe Ps'ychologicaZ Re_i_

88(2) pp. 81-97 (march 1956).

38. D. L. Parnas, "On the crtterta to be used in decomposing systems into

modules," CornTn. ACM 15(12) pp. 1053-1058 (December 1972).

39. J. Brown, The Social Psychology of [nd_str'y, Penguin Books, Baltimore

(1954).

5-26

I

I

I

I

I

I

I

I

I

I

I

i

I

I

I

I

I

I

l
I

I

!

I

ii

g

I

I

i

i

I

i

I

I

I

!

I

!

N87 - 24 904 _

DATA COLLECTION AND EVALUATION FOR

EXPERIMENTAL COMPUTER SCIENCE RESEARCH

Marvin V. Zelkowitz

Department of Computer Science

University of Maryland

College Park, Maryland 20742

Abstract

The Software Engineering Laboratory has been monitoring software development at

NASA Goddard Space Flight Center since 1976. This report describes the data collection activities of the Laboratory and some of the difficulties of obtaining reliable data.

In addition, the application of this data collection process to a current prototyping

experiment is reviewed.

I. INTRODUCTION

There is a significant need to collect reliable data on software development projects in order

to provide an empirical basis for making conclusions about software development methodologies,

models and tools. However, such data is usually hard to collect and even harder to evaluate.

Software is a multibillion dollar industry where 100% cost overruns are common, and maintenance activities can take up to 70/_ of the total cost of the system [11]. The availability of reliable data to evaluate competing software development techniques is crucial.

As Lord kelvin stated, "I often say that when you can measure what you are speaking

about, and express it in numbers, you can know something about it, but when you cannot express

it in numbers, your knowledge is of a meager and unsatisfactory kind." The lack of adequate

measures is certainly a problem in the software industry today.

Many of the recent analyses of the software development process are based on data that is

obtained from university experiments. Students often program special problems whose results are

subjected to analysis. This gives the researcher the l0 to 100 data points necessary for statistical

validity of the results. However, by virtue of being part of an academic program, such experiments are necessarily small and usually involve inexperienced programmers. There is a need to

5-27

!
extend the scope of these experiments to a level appropriate to the muitibillion dollar industry.

Most software development data in industry has been collected after the fact. That is, a

project is built and then a pile of documents are handed to a research group for evaluation.

Often, critical information is missing and the results are not what one would expect. Rather than

following the model of archeology - the study of dead software projects, software evaluation must

model sociology - the study of living software societies. Data must be collected from ongoing projects, but the software sociologists must not impact the objects of their study. Given the need to

finish projects on time and within budgets - a goal too often missed - it is difficult to justify

spending money on data collection and evaluation activities.

Specifically to address these problems, the Software Engineering Laboratory (SEL) was set

up within NASA Goddard Space Flight Center in 1976. The goal was to study software development activities within NASA and report on experiences that will improve the process. This report

describes the SEL and its experiences over the last six years.

11- THE SOFTWARE ENGINEERING LABORATORY

In 1976 the SEL was organized to study software development within the NASA environment. More specifically, its primary charter was to monitor the development of ground support

software for unmanned spacecraft. Each such system was typically 30,000 to 50,000 source lines of

Fortran and took from 8 to 10 programmers up to two years to build. While this environment is

not representative of all software development environments, SEL experiences are generalizable in

some respects:

a) Ground support software includes several program types such as data base functions, real

time processing, scientific calculations and control language functions. The software is largely

implemented in Fortran.

b} By looking at a relatively narrow environment, d:.ta collected from many projects can be

compared. Thus we get some of the benefits of a carefully controlled experiment without the

expense of duplicating large developments. We do not have the problem of looking at a variety of

5-28

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

l

I

I
I

I

l

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

projects, like compilers, COBOL programs, ground support software, MIS programs and then trying to say something consistent about all of these.

To date, 46 projects have been studied, containing over 1.8 million lines of code. Over 150

programmers participated in these projects, and the data base contains over 40 million bytes of

data. The general SEL strategy is t carefully monitor a project and regularly collect data during

its development. The data is then entered in the SEL data base for analysis. The purpose of this

report is not to dwell on specific research results based on this data (See, for example, [8] for a

collection of published papers about the SEL) but is concerned with the problems of collecting

data, and what we have learned from this process.

HI. DATA COLLECTION

HI.I MODEL GENERATION

In order to fully take advantage of the available data, it must be known what information is

desired. The models and measures that are to be investigated must be defined. A random data

collection activity will usually miss relevant data, and then it will be too late to try and recover

that information.

In the SEL, two classes of measures were identified for study, and the data collection activities were oriented around those areas. The initial activities included:

a) Process Measures. Evaluating personnel and computer resources over time was a clear

need. One activity was to try and validate models that others have identifie (e.g., the Putnam

Norden Rayleigh curve [I]) while another activity was to try and build new models to fit the

empirical data (e.g., the Parr curve [7]). Once models were identified, their predictive nature was

studied as a means of resource scheduling.

The generation and correction of errors is another activity that h_ important economic

consequences. However, few models are available to build upon, so there was a need to develop

new models of errors and investigate their effects upon performance.

5-29

I
b) Product Measures. The size, structure, and complexity of software are other important

economic factors to consider. The evaluation of measures such as the software science measures

of Halstead [5], the.cyclomatic complexity of McCabe [6] and other measures developed within the

SEL was another early goal.

Reliability is a critical activity in most environments. In our particular environment, the

software that was previously developed was hi_,hly reliable (typically under 10 errors in an operational _stem), so that reliability, while important, was not a primary driving force in organizing

the SEL.

III.2 FORMS GENERATION

The first process in evaluating empiric-_l data is the data collection activity. Ideally, you

would like the process to be automated and transparent to the programmer. However, this was

not possible in this situation. We were interested in the human activities of software development. Thus we needed detailed information about how programmers spend their time. Because of

this, a decision made early in the life of the SEL was that some data would be manually collected

using a series of forms.

There is a significant tradeoff consideration at this point. If we tried to collect too much

information, programmers would object to the interierence of the data collection activity on their

work. If too little information was asked, then there would be little point in collecting it.

We first developed an initial set of reporting forms. These have been revised several times

since then. Each time certain fields were clarified and the amount of information sought decreased

somewhat. At the present time, the effort required to fill out the forms is not significant. Initially

seven forms were developed. However, only three are used heavily. These seven forms axe:

a) Resource Summary. This form lists the number of hours per week spent by all personnel

on the project. This information is obtained mostly from the weekly time cards supplied by the

contractor. It is easy to obtain this data, and causes little overhead to a project. However, it is

very useful for monitoring global resource expenditures, especially in conjunction with the follow5-30

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I
I

I

I

I

I

I

I

I

I

I

I

I

I

i

I

I

I

I

ing Component Status Report.

b) Component Status Report. This form is submitted weekly by each programmer. It lists

for each component of the system (e.g., Fortran subroutine) the number of hours spent on each of

nine categories (e.g., design, code, test, review, etc.). The detail required by this form initially

caused some concern; however, in looking over past forms the average programmer worked on

only 5 to 10 components per week and only 2 or 3 activities per component. Thus the overhead

was not excessive. While the data is only approximate to the nearest hour, we believe that it is

more accurate than many other data collection procedures.

For example, many research papers give percentages for design, code, and test on a project.

However, these are usually taken from resource summary data and calendar date milestones. If a

design review occurs on a Friday, then all activities up until that date are design, with all activities the next week being code. In the SEL environment, there was approximately a 25 percent

error in using calendar dates for percent effort [4I. On four projects, approximately °,5 percent of

the design occurred during the coding phase, while almost half of the testing occurred prior to the

testing phase (Figure 1).The Component Status Report is critical for a proper view of development activities.

c} Change Report Form. This form is completed after each change to a component is completed and tested. Due to the number of changes that a component undergoes during early

development, there was no attempt to capture this data before the component was "complete"

(i.e., through unit test}. Note that we are capturing "changes" and not simply "errors." All

modifications, due to errors or other considerations such as enhancements, are tracked.

Besides identifying the type of change, this form also identifies the cause of the change -

they are not always the same, although programmers have difficulty separating the two. The form

also asks for information on the time to find and correct an error, and what tools and techniques

were used in the process.

In some environments, the introduction of this form might cause programmers to object;

however, this was not the case in our environment. A standard change monitoring procedure was

5-31

I
in place, so we simply changed the form that this branch ofNASA GSFC was using before the

SEL was created.

These three forms provide the most important data collected by the SEL. Four other forms

have been created and used with limited success. These are:

d) Component Summary. This form identifies the characteristics of each component in a

system. It gives the size, complexity and interfaces. The goal was to have this form filled out at

least twice - once when the component was first identified during design, and again when it was

completed. Our experience was that the initial form was filled out before much relevant information was known, and the data on the final form could be extracted automatically from the source

code data base.

e) Computer Run Analysis. An entry on this form is filled out for each computer run giving

characteristics of the run (execution time, purpose of run, components processed) as well as

whether the run met its objectives. This is one form that could be automated. However, the

usual range of operating system "Completion Codes" is inadequate for many purposes. For example, a debugging run that was expected to fail at a certain statement, but ran to a successful exit,

would have a satisfactory completion code, yet it was a failure as a run since the desired error did

not occur.

An interactiv job submittal system could help. Before any run, the system could prompt for

some of this information. After the run, the system could a_k what happened. Since the current

NASA environment consists primarily of interactive editing with batch processing, such an online

process would have been difficult to implement.

f) Programmer Analyst Survey. This form attempts to characterize the experiences of the

programmers on the project in order to get a general profile of the project tea However, we

immediately ran into confidentiality problems concerning personnel records. We never got the

detailed information that we desired, but have obtained general comments on each programmer -

although the goal is NOT to rate programmers. If there is any hint of any of this data being used

for any sort of personnel action, then compliance drops sharply and the value of the data becomes

5-32

I

I

I

I

I

I

I

I

I

I

I

l

I

I

I

I

I

I
I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

open to question.

g) General Project Summary. This is a form that provides a high-level description of a project. Since the software is developed by NASA and contractor personnel, the form is somewhat

superfluous and the information is entered directly into the data base.

An important consideration in forms development is consistency in collecting data. Along

with each form a detailed instruction sheet was developed, as well as a glossary of relevant terms

like "component," "line of code," and "life cycle phase." For example, we chose the name "componenC rather than "subroutine" or "module" simply because those terms were well known (with

alternative meanings) and we did not want to evoke any preconceived but wrong image in the

minds of the participants. Even so, there was a great deal of confusion about the meanings of the

various terms. During the early days of the SEL, many meetings were held to explain the process

to programmers, since each programmer worked about one year on a project, after six years there

is a large core of personnel experienced in filling out our reporting forms..

111.3 DATA PROCESSING

After being filled out, each form is entered into a data base on a PDP 11/70 computer. In

addition to the forms previously described, analyzers were run over the source programs to extract

additional information, including lines of code and other measures such as the Halstead software

science measures.

Another step in forms processing is data validation. Someone must review the forms as they

are submitted. This is expensive, but necessary. It is a quick was to catch and correct errors. In

addition, the data entry program should check for data consistency and value ranges. For example, if the program is to read in input in the format _,_vIDDYY, then a month input that is not a

number in the range from 01 to 12 must be rejected. A field requiring an input of A, B, or C

should reject any other value. Even though we manually check each form, a validation program

was more effective for catching errors.

5-33

I
All forms, especially the change report form, need to be reviewed by SEL personnel. Two

common errors in the Change report form are to turn in one change report form which actually

represented several errors, and the submission of multiple forms for the same error. From earlier

work over half of the change report forms were modified following a careful study of each form.

This is an expensive process, but needs to be done in order to have accurate data about your

environment.

Redundancy of data is another important consideration. Collecting the same or similar data

on multiple forms allows for cross validation. There should be a reasonable correlation between

the collected values. The resource summary and component status reports have been the easiest

to validate. The Computer Run Analysis form is important for validating some of the change

report data; however, limited availability of this form has handicapped some of this validation

work. Because of that, it is important to manually check each change report form for selected

projects.

IV RESEARCH ACTIVITIES

IV.1 PREVIOUS RESEARCH

Research in the SEL has centered on resource and error models and on predicting software

productivity. ([8] is a collection of relevant papers published over the last few years.) Perhaps the

most important conclusion - although obvious in hindsight - which is relevant to this current discussion is that there is no typical software development environment.

All models include parameters - factors which represent variables in that environment (Figure 2 representsa listof factorsfrom the SEL as well as two otherstudies[I0][3].)When models

b-..sedon other environments are applied to the NASA environment, they invariablyfail.Does

that mean that NASA isdifferent?unique? much betteror much worse than other environments?

For example, SEL programmers show much higher productivityin linesof code per week than in

other organizations.Does that mean that other organizationsshould pirateaway NASA's staff?.

5-34

I

I

I

I

I

I

I

I

I

i

I

I

I

I

I

I

I

I

I
I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

Perhaps, but another explanation becomes apparent when NASA's environment is studied in

detail. In the $EL, most of the projects are similar ground support software systems. Thus the top

level design for these projects are similar. Programmers are experts at this particular problem -

thus high productivity. Many factors affecting requirements and design do not apply here. On the

other hand, a contractor that bids on a variety of projects - an operating system, a compiler, a

data base management system, an attitude orbit determination program, etc. does not build an

institutional knowledge about any one particular environment. Requirements and design factors

now become significant in this environment and productivity drops.

All companies operate in a different manner. Company policy as to working conditions, computer usage (batch or interactive), leave policy and salaries, management, support tools, etc. all

affect productivity. Thus each organization (probably even separate divisions within a single

organization) has a different structure and a different set of parameters.

For this reason, one must first calibrate any model to be applied. First develop a quantitative relationship using many factors. Chose those factors relevant to your environment. Calibrate

the equations based upon previous projects, and then use the calibrated model for prediction [2].

It is this important calibration step that is missing from most models.

For example, if a baseline equation is given by:

Effort -- a * size'+ b

then one can fit a and b from historical data; and the units of size can be determined from those

relevant to your environment - such as lines of code, lines of source (including comments), number

of modules, number of output statements, etc. Thus instead of a single model, there is a class of

models t_ilored to each environment.

IV.2 PROTOTYPES

Over the past few years various methodologies have been studied by the SEL. A current

SEL activity is the development of software prototypes. Currently software fs designed, built and

delivered. Rarely is the product evaluated in advance. However, the use of engineering prototypes

5-35

I
in a preliminary evaluation is starting to be discussed by software engineering professionals [9].

While the term is appearing with increasing frequency, what does it really mean? Is it a

quick and dirty throw-a-way implementation or a carefully designed subset of a final implementation? What are the cost and reliability parameters for a prototype compared to a full implementation.

Currently data on the subject is meagre and usually based on small projects [12]. The SEL

is now investigating a larger implementation with some techniques as applied to previous SEL

projects.

Briefly, the target implementation is an integrated support system for flight dynamics

research. Currently, experimenters (NASA scientists), in trying a new spacecraft model (e.g., a

new orbit calculation) must understand the structure of the existing system, access the Fortran

source modules, modify them, rebuild the operating program, test it, and then run the experiment

- a complex and costly process. The new system is expected to _understand" several flight dynamics systems and to provide a higher level command language that guides the experimenter through

the process of building a new version of a system, even if the experimenter is not thoroughly familiar with the existing system. This system is basically a command language interpreter with a

complex data dictionary describing the underlying flight dynamics subsystems.

This program is quite different from existing software produced by NASA, so the plan is to

prototype it first. Two classes of data will be obtained from the prototype:

a) Characteristics of the process. The Computer Science world has little information available about prototyping, thus this data will add to the general knowledge about this process. What

does the life cycle of a prototype look like? How much time is spent in design? code? test? Are

errors crucial or can they be side-stepped in the prototype somewhat by "eliminating" the

offending feature ia the requirements?

Similarly, how does prototyping effect the later full implementation? Will design be easier?

Will productivity be higher? Will the overall cost of the system plus prototype be less than the

cost of just the full system? Will reliability be higher or the interface more "user friendly? _

5-36

l

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I
I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

b) Predictive nature of the prototype. Once a prototype is built, is it successful? How does

one measure success? Will the full system be successful based upon an evaluation of the prototype? A set of measures will be built into the prototype to provide some of these answers.

A baseline study will be made of how experiments are conducted - the cost of machine and

people resources will be measured. Some of these experiments will be repeated with the prototype

to derive a cost. These will be used to predict the cost of using the full system. If acceptable, then

that design will be used for the full implementation, if not, then the design will be modified to

correct the problem in the full implementation.

In addition, data will be collected on how often features are used in the prototype, and also

how often the prototype is being circumvented in order to provide features that currently do not

exist but are needed by the users.

Once the final system is built, the predictive model can be validated in order to aid ia

developing a theory of software prototypes.

V, CONCLUSIONS

The Software Engineering Laboratory has been in existence for six years and has studied

almost 40 projects. The empirical data that has been collected supports several conclusions:

(1) Data collection is hard and expensive. It must be dynamically collected during the

development of a project and not after completion.

(2) Data must be validated. Error rates on manually filled out forms are high. A lack of

standardized nomenclature for the field hurts consistency. Much effort must go in training personnel to understand the data collection methodology.

(3) Each software development environment is unique. Baseline equations must first be calibrated with past projects before a model can be used in the future.

(4) Little is known, but much is being said, about software prototypes. The SEL is

currently studying this issue as part of its ongoing activities.

5-37

I
VL ACKNO_LEDGEMENTS

This paper was supported by NASA grant NSG-5123 to the University of Maryland. The

SEL is under the direction of Frank McGarry of NASA GSFC. Dr. Victor Basili directs the

University of Maryland activities of the SEL, and Jerry Page is the coordinator for Computer Sciences Corporation. The results described in this report were developed by the author, the above

mentioned persons, as well as several graduate students at the University of Maryland and

researchers at CSC. The author is indebted to David Card of CSC for his detailed comments on

an earlier draft of this paper.

VH. BIBLIOGRAPHY

[1] Basili V. R. and M. V. Zelkowitz, Analyzing medium scale software developments, Third International Conference on Software Engineering, Atlanta GA, May 1978.

[2] Basili V. R., Models and metrics for software management and engineering, ASME Advances

in Computer Technology 1, January, 1980.

[3] Boehm B., Software Engineering Economics, Prentice Hall, 1981.

[4] Chen E. and Zelkowitz M. V., Use of cluster analysis to evaluate software engineering methodologies, Fifth International Conference on Software Engineering, San Diego CA, March, 1981.

[5] Halstead M., Elements of Software Science, American Elsevier, 1977.

[6] McCabe T., A complexity measure, IEEE Transactions on Software Engineering 2, 1976.

[7] Parr F., An alternative to the Rayleigh Curve model for software development, IEEE Transactions on software engineering 6, 1980.

[8] Collected Software Engineering Papers: Volume 1, SEL-82-004, Code 582.1, NASA GSFC,

July, 1982.

[9] ACM SIGSOFT Software Engineering Symposium: Workshop on Rapid Prototyping, Columbia, MD, April, 1982.

[10] Walston C. and C. Felix, A method of programming measurement and estimation, IBM Systems Journal 16, No. 1, 1977.

[11] Zelkowitz M. V., A. C. Shaw and J. D. Gannon, Principles of Software Engineering and

Design, Prentice Hall, 1979.

[12] Zelkowitz M. V., A c_e study in rapid prototyping Software Practice and Ezperience 10,

1037-1042, 1980.

5-38

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I
I

I

I

I

I

I

I

I

i

I

I

I

I

I

I

I

I

I

BIBLIOGRAPHY OF SEL LITERATURE

The technical papers, memorandums, and documents listed in

this bibliography are organized into two groups. The first

group is composed of documents issued by the Software Engineering Laboratory (SEL) during its research and development

activities. The second group includes materials that were

published elsewhere but pertain to SEL activities.

SEL-ORIGINATED DOCUMENTS

SEL-76-001, Proceedings From the First Summer Software Engineerin 9 Workshop, August 1976

SEL-77-001, The Software Engineerin@ Laboratory,

V. R. Basili, M. V. Zelkowitz, F. E. McGarry, et al., May

1977

SEL-77-002, Proceedings From the Second Summer Software Engineering Workshop, September 1977

SEL-77-003, Structured FORTRAN Preprocessor (SFORT), B. Chu

and D. S. Wilson, September 1977

SEL-77-004, GSFC NAVPAK Design Specifications Languages

Study, P. A. Scheffer and C. E. Velez, October 1977

SEL-78-001, FORTRAN Static Source Code Analyzer (SAP) Design

and Module Descriptions, E. M. O'Neill, S. R. Waligora, and

C. E. Goorevich, February 1978

ISEL-78-002, FORTRAN Static Source Code Analyzer (SAP)

User's Guide, E. M. O'Neill, S. R. Waligora, and

C. E. Goorevich, February 1978

SEL-78-102, FORTRAN Static Source Code Analyzer Program

(SAP) User's Guide (Revision i), W. J. Decker and

W. A. Taylor, September 1982

SEL-78-003, Evaluation of Draper NAVPAK Software Desig N ,

K. Tasaki and F. E. McGarry, June 1978

SEL-78-004, Structured FORTRAN Preprocessor (SFORT)

PDP-II/70 User's Guide, D. S. Wilson and B. Chu, September

1978

B-I

I
SEL-78-005, Proceedings From the Third Summer Software Engineering Workshop, September 1978

SEL-78-006, GSFC Software Engineering Research Requirements

Analysis Study, P. A. Scheffer and C. E. Velez, November 1978

SEL-78-007, Applicability of the Rayleigh Curve to the SEL

Environment, T. E. Mapp, December 1978

SEL-79-001, SIMPL-D Data Base Reference Manual,

M. V. Zelkowitz, July 1979

SEL-79-002, The Software Engineering Laboratory: Relationship Equations, K. Freburger and V. R. Basili, May 1979

SEL-79-003, Common Software Module Repository (CSMR) System

Description and User's Guide, C. E. Goorevich, A. L. Green,

and S. R. Waligora, August 1979

SEL-79-004, Evaluation of the Caine, Farber, and Gordon Program Design Language (PDL) in the Goddard Space Flight Center (GSFC) Code 580 Software Design Environment,

C. E. Goorevich, A. L. Green, and W. J. Decker, September

1979

SEL-79-005, Proceedings From the Fourth Summer Software Engineerin@ Workshop, November 1979

SEL-80-001, Functional Requirements/Specifications for

Code 580 Confiquration Analysis Tool (CAT), F. K. Banks,

A. L. Green, and C. E. Goorevich, February 1980

SEL-80-002, Multi-Level Expression Design Language7

Requirement Level (MEDL-R) System Evaluation, W. J. Decker

and C. E. Goorevich, May 1980

SEL-80-003, Multimission Modular Spacecraft Ground Support

Software System (MMS/GSSS) State-of-the-Art Computer Systems/

Compatibility Study, T. Welden, M. McClellan, and

P. Liebertz, May 1980

ISEL-80-004, System Description and User's Guide for Code

580 Configuration Analysis Tool (CAT), F. K. Banks,

W. J. Decker, J. G. Garrahan, et al., October 1980

SEL-80-104, Configuration Analysis Tool (CAT) System Description and User's Guide (Revision i), W. Decker and

W. Taylor, December 1982

SEL-80-005, A Study of the Musa Reliability Model,

A. M. Miller, November 1980

B-2

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I
I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

i,|

I

I

SEL-80-006, Proceedings From the Fifth Annual Software Engineerin@ Workshop, November 1980

SEL-80-007, An Appraisal of Selected Cost/Resource Estimation Models for Software Systems, J. F. Cook and

F. E. McGarry, December 1980

ISEL-81-001, Guide to Data Collection, V. E. Church,

D. N. Card, F. E. McGarry, et al., September 1981

SEL-81-101, Guide to Data Collection, V. E. Church,

D. N. Card, F. E. McGarry, et al., August 1982

ISEL-81-002, Software Engineering Laboratory (SEL) Data

Base Organization and User's Guide, D. C. Wyckoff, G. Page,

and F. E. McGarry, September 1981

SEL-81-102, Software Engineerin @ Laboratory (SEL) Data Base

Organization and User's Guide Revision i, P. Lo and

D. Wyckoff, July 1983

ISEL-81-003, Data Base Maintenance System (DBAM) User's

Guide and System Description, D. N. Card, D. C. Wyckoff, and

G. Page, September 1981

SEL-81-103, Software Engineering Laboratory (SEL) Data Base

Maintenance System (DBAM) User's Guide and System Description, P. Lo and D. Card, July 1983

ISEL-81-004, The Software Engineering Laboratory,

D. N. Card, F. E. McGarry, G. Page, et al., September 1981

SEL-81-104, The Software Engineering Laboratory, D. N. Card,

F. E. McGarry, G. Page, et al., February 1982

1SEL-81-005, Standard Approach to Software Development,

V. E. Church, F. E. McGarry, G. Page, et al., September 1981

ISEL-81-105, Recommended Approach to Software Development,

S. Eslinger, F. E. McGarry, and G. Page, May 1982

SEL-81-205, Recommended Approach to Software Development,

F. E. McGarry, G. Page, S. Eslinger, et al., April 1983

SEL-81-006, Software Engineering Laboratory (SEL) Document

Library (DOCLIB) System Description and User's Guide,

W. Taylor and W. J. Decker, December 1981

1SEL-81-007, Software Engineering Laboratory (SEL) Compendium of Tools, W. J. Decker, E. J. Smith, A. L. Green,

et al., February 1981

B-3

I
SEL-81-107, Software Engineering Laboratory (SEL) Compendium

of Tools, W. J. Decker, W. A. Taylor, and E. J. Smith,

February 1982

SEL-81-008, Cost and Reliability Estimation Models (CAREM)

User's Guide, J. F. Cook and E. Edwards, February 1981

SEL-81-009, Software Engineering Laboratory Programmer Workbench Phase 1 Evaluation, W. J. Decker and F. E. McGarry,

March 1981

lSEL-81-010, Performance and Evaluation of an Independent

Software Verification and Integration Process, G. Page and

F. E. McGarry, May 1981

SEL-81-110, Evaluation of an Independent Verification and

Validation (IV&V) Methodology for Flight Dynamics, G. Page

and F. McGarry, December 1983

SEL-81-011, Evaluating Software Development by Analysis of

Change Data, D. M. Weiss, November 1981

SEL-81-012, The Rayleigh Curve As a Model for Effort Distribution Over the Life of Medium Scale Software Systems, G. O.

Picasso, December 1981

SEL-81-013, Proceedings From the Sixth Annual Software Engineering Workshop, December 1981

SEL-81-014, Automated Collection of Software Engineering

Data in the Software Engineering Laboratory (SEL),

A. L. Green, W. J. Decker, and F. E. McGarry, September 1981

SEL-82-001, Evaluation of Management Measures of Software

Development, G. Page, D. N. Card, and F. E. McGarry,

September 1982, vols. 1 and 2

SEL-82-002, FORTRAN Static Source Code Analyzer Program

(SAP) System Description, W. A. Taylor and W. J. Decker,

August 1982

SEL-82-003, Software Engineering Laboratory (SEL) Data Base

Reporting Software User's Guide and System Description,

P. Lo, September 1982

SEL-82-004, Collected Software Engineering Papers: Volume l, July 1982

ISEL-82-005, Glossary of Software Engineering Laboratory

Terms, M. G. Rohleder, December 1982

B-4

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I
I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

SEL-82-I05, Glossary of Software Engineering Laboratory

Terms, T. A. Babst, F. E. McGarry, and M. G. Rohleder,

October 1983

lSEL-82-006, Annotated Bibliography of Software Engineering Laboratory (SEL) Literature, D. N. Card, November 1982

SEL-82-106, Annotated Bibliography of Software Engineering

Laboratory Literature, D. N. Card, T. A. Babst, and

F. E. McGarry, November 1983

SEL-82-007, Proceedings From the Seventh Annual Software

Engineering Workshop, December 1982

SEL-82-008, Evaluating Software Development by Analysis of

Changes: The Data From the Software Engineering Laboratory,

V. R. Basili and D. M. Weiss, December 1982

SEL-83-001, Software Cost Estimation Experiences,

F. E. McGarry, G. Page, D. N. Card, et al., November 1983

SEL-83-002, Measures and Metrics for Software Development,

D. N. Card, F. E. McGarry, G. Page, et al., November 1983

SEL-83-003, Collected Software Engineering Papers: Volume II, November 1983

SEL-83-004, SEL Data Base Retrieval System (DARES) User's

Guide, T. A. Babst and W. J. Decker, November 1983

SEL-83-005, SEL Data Base Retrieval System (DARES) System

Description, P. Lo and W. J. Decker, November 1983

SEL-83-006, Monitoring Software Development Through Dynamic

Variables, C. W. Doerflinger, November 1983

SEL-83-007, Proceedings From the Eighth Annual Software Engineering Workshop, November 1983

SEL-RELATED LITERATURE

2Agresti, W. W. , F. E. McGarry, D. N. Card, et al.,

"Measuring Software Technology," Program Transformation and

Programming Environments. New York: Springer-Verlag, 1984

3Bailey, J. W., and V. R. Basili, "A Meta-Model for Software Development Resource Expenditures," Proceedings of the

Fifth International Conference on Software Engineering.

New York: Computer Societies Press, 1981

B-5

I
Banks, F. K., "Configuration Analysis Tool (CAT) Design,"

Computer Sciences Corporation, Technical Memorandum, March

1980

3Basili, V. R., "Models and Metrics for Software Management and Engineering," ASME Advances in Computer Technolog[,

January 1980, vol. 1

Basili, V. R., "SEL Relationships for Programming Measurement and Estimation," University of Maryland, Technical Memorandum, October 1979

Basili, V. R., Tutorial on Models and Metrics for Software

Management and Engineering. New York: Computer Societies

Press, 1980 (also designated SEL-80-008)

3Basili, V. R., and J. Beane, "Can the Parr Curve Help

With Manpower Distribution and Resource Estimation Problems?", Journal of Systems and Software, February 1981,

vol. 2, no. 1

3Basili, V. R., and K. Freburger, "Programming Measurement

and Estimation in the Software Engineering Laboratory,"

Journal of Systems and Software, February 1981, vol. 2, no. 1

2Basili, V. R., and B. T. Perricone, Software Errors and

Complexity: An Empirical Investigation, University of

Maryland, Technical Report TR-II95, August 1982

3Basili, V. R., and T. Phillips, "Evaluating and Comparing Software Metrics in the Software Engineering Laboratory," Proceedings of the ACM SIGMETRICS Symposium/

Workshop: Quality Metrics, March 1981

2Basili, V. R., R. W. Selby, and T. Phillips, "Metric

Analysis and Data Validation Across FORTRAN Pr'ojects," IEE____EE

Transactions on Software Engineering, November 1983

Basili, V. R., and R. Reiter, "Evaluating Automatable Measures for Software Development," Proceedings of the Workshop

on Quantitative Software Models for Reliability, Complexity

and Cost, October 1979

2Basili, V.R., and D. M. Weiss, A Methodology for Collecting Valid Software Engineering Data, University of

Maryland, Technical Report TR-1235, December 1982

Basili, V. R., and M. V. Zelkowitz, "Designing a Software

Measurement Experiment," Proceedings of the Software Life

Cycle Management Workshop, September 1977

B-6

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I
I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

3Basili, V. R., and M. V. Zelkowitz, "Operation of the

Software Engineering Laboratory," Proceedings of the Second

Software Life Cycle Management Workshop, August 1978

3Basili, V. R., and M. V. Zelkowitz, "Measuring Software

Development Characteristics in the Local Environment,"

Computers and Structures, August 1978, vol. i0

Basili, V. R., and M. V. Zelkowitz, "Analyzing Medium Scale

Software Development," Proceedings of the Third International Conference on Software Engineering. New York: Computer Societies Press, 1978

3Basili, V. R., and M. V. Zelkowitz, "The Software Engineering Laboratory: Objectives," Proceedings of the

Fifteenth Annual Conference on Computer Personnel Research,

August 1977

2Card, D. N., "Early Estimation of Resource Expenditures

and Program Size," Computer Sciences Corporation, Technical Memorandum, June 1982

2Card, D. N., "Comparison of Regression Modeling Techniques for Resource Estimation," Computer Sciences Corporation, Technical Memorandum, November 1982

Card, D. N., and V. E. Church, "Analysis Software Requirements for the Data Retrieval System," Computer Sciences

Corporation Technical Memorandum, March 1983

Card, D. N., and V. E. Church, "A Plan of Analysis for

Software Engineering Laboratory Data," Computer Sciences

Corporation Technical Memorandum, March 1983

Card, D. N., and M. G. Rohleder, "Report of Data Expansion

Efforts," Computer Sciences Corporation, Technical Memorandum, September 1982

3Chen, E., and M. V. Zelkowitz, "Use of Cluster Analysis

To Evaluate Software Engineering Methodologies," Proceedings of the Fifth International Conference on Software

Engineering. New York: Computer Societies Press, 1981

2Doerflinger, C. W., and V. R. Basili, "Monitoring Software Development Through Dynamic Variables," Proceedings of

the Seventh International Computer Software and Applications

Conference. New York: Computer Societies Press, 1983

Freburger, K., "A Model of the Software Life Cycle" (paper

prepared for the University of Maryland, December 1978)

B-7

i
Higher Order Software, Inc., TR-9, A Demonstration of AXES

for NAVPAK, M. Hamilton and S. Zeldin, September 1977 (also

designated SEL-77-005)

Hislop, G., "Some Tests of Halstead Measures" (paper prepared for the University of Maryland, December 1978)

Lange, S. F., "A Child's Garden of Complexity Measures"

(paper prepared for the University of Maryland, December

1978)

McGarry, F. E., G. Page, and R. D. Werking, Software Development History of the Dynamics Explorer (DE) Attitude Ground

Support system (AGSS), June 1983

Miller, A. M., "A Survey of Several Reliability Models"

(paper prepared for the University of Maryland, December

1978)

National Aeronautics and Space Administration (NASA), NASA

Software Research Technology Workshop (proceedings), March

1980

Page, G., "Software Engineering Course Evaluation, " Computer

Sciences Corporation, Technical Memorandum, December 1977

Parr, F., and D. Weiss, "Concepts Used in the Change Report

Form," NASA, Goddard Space Flight Center, Technical Memorandum, May 1978

Reiter, R. W., "The Nature, Organization, Measurement, and

Management of Software Complexity" (paper prepared for the

University of Maryland, December 1976)

Scheffer, P. A., and C. E. Velez, "GSFC NAVPAK Design Higher

Order Languages Study: Addendum," Martin Marietta Corporation, Technical Memorandum, September 1977

Turner, C., and G. Caron, A Comparison of RADC and NASA/SEL

Software Development Data, Data and Analysis Center for

Software, Special Publication, May 1981

Turner, C., G. Caron, and G. Brement, NASA/SEL Data Compendium, Data and Analysis Center for Software, Special Publication, April 1981

Weiss, D. M., "Error and Change Analysis," Naval Research

Laboratory, Technical Memorandum, December 1977

I,

Williamson, I. M., "Resource Model Testing and Information,

Naval Research Laboratory, Tethnical Memorandum, July 1979

B-8

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I
I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

3Zelkowitz, M. V., "Resource Estimation for Medium Scale

Software Projects," Proceedings of the Twelfth Conference on

the Interface of Statistics and Computer Science.

New York: Computer Societies Press, 1979

2Zelkowitz, M. V., "Data Collection and Evaluation for Experimental Computer Science Research," Empirical Foundations

for Computer and Information Science (proceedings),

November 1982

Zelkowitz, M. V., and V. R. Basili, "Operational Aspects of

a Software Measurement Facility," Proceedings of the Software Life Cycle Management Workshop, September 1977

iThis document superseded by revised document.

2This article also appears in SEL-83-003, Collected Software Engineering Papers: Volume II, November 1983.

B-9