U.S. Department

of Commerce

National Bureau

of Standards

Computer Science

and Technology

NBS

PUBLICATIONS

'f

NATL INST OF STAND & TECH

NBS Special Publication 500-82

Final Report:

A Survey of

Software Tools Usage
NATIONAL BUREAU OF STANDARDS

The National Bureau of Standards' was established by an act of Congress on March 3, 1901. The Bureau's overall goal is to strengthen and advance the Nation's science and technology

and facilitate their effective application for public benefit. To this end, the Bureau conducts

research and provides: (1) a basis for the Nation's physical measurement system, (2) scientific and technological services for industry and government, (3) a technical basis tor equity in trade, and (4) technical services to promote public safety. The Bureau's technical work is per- formed by the National Measurement Laboratory, the National Engineering Laboratory, and

the Institute for Computer Sciences and Technology.

THE NATIONAL MEASUREMENT LABORATORY provides the national system of

physical and chemical and materials measurement; coordinates the system with measurement

systems of other nations and furnishes essential services leading to accurate and unitorm

physical and chemical measurement throughout the Nation's scientific community, industry,

and commerce: conducts materials research leading to improved methods of measurement,

standards, and data on the properties of materials needed by industry, commerce, educational

institutions, and Government; provides advisory and research services to other Government

agencies; develops, produces, and distributes Standard Reference Materials; and provides

calibration services. The Laboratory consists of the following centers:

Absolute Physical Quantities' — Radiation Research — Thermodynamics and

Molecular Science — Analytical Chemistry — Materials Science. THE NATIONAL ENGINEERING LABORATORY provides technology and technical ser- vices to the public and private .sectors to address national needs and to solve national problems; conducts research in engineering and applied science in support of these efforts; builds and maintains competence in the necessary disciplines required to carry out this research and technical service; develops engineering data and measurement capabilities;

provides engineering measurement Iraceability services; develops test methods and proposes

engineering standards and code changes; develops and proposes new engineering practices;

and develops and improves mechanisms to transfer results of its research to the ultmiate user. The Laboratory consists of the following centers:

Applied Mathematics — Electronics and Electrical Engineering- — Mechanical

Engineering and Process Technology' — Building Technology — Fire Research —

Consumer Product Technology — Field Methods.

THE INSTITUTE FOR COMPUTER SCIENCES AND TECHNOLOGY conducts

research and provides scientific and technical services to aid Federal agencies in the selection, acquisition, application, and use of computer technology to improve effectiveness and

economy in Government operations in accordance with Public Law 89-306 (40 U.S.C. 759), relevant Executive Orders, and other directives; carries out this mission by managing the Federal Information Processing Standards Program, developing Federal ADP standards

guidelines, and managing Federal participation in ADP voluntary standardization activities; provides scientific and technological advisory services and assistance to Federal agencies; and

provides the technical foundation for computer-related policies of the Federal Government.

The Institute consists of the following centers:

Programming Science and Technology — Computer Systems Engineering.

'Headquarters and Laboratories at Gailhersburg, MD, unless otherwise noted; mailing address Washington, DC 20234. •Some divisions wiitiin the center are located at Boulder, CO 80303.
Computer Science

and Technology

NBS Special Publication 500-82 '"'"p^^"^

,-. I r-« i DEC 1 1981

Final Report:

A Survey of

Software Tools Usage

Herbert Hecht

SoHaR Incorporated

1040 So. La Jolla Avenue

Los Angeles, California 90035

U.S. DEPARTMENT OF COMMERCE

Malcolm Baldrlge, Secretary

National Bureau of Standards

Ernest Ambler, Director

Issued November 1981
Reports on Computer Science and Technology

The National Bureau of Standards has a special responsibility within the Federal

Government for computer science and technology activities. The programs of the

NBS Institute for Computer Sciences and Technology are designed to provide ADP

standards, guidelines, and technical advisory services to improve the effectiveness

of computer utilization in the Federal sector, and to perform appropriate research and

development efforts as foundation for such activities and programs. This publication

series will report these NBS efforts to the Federal computer community as well as to interested specialists in the academic and private sectors. Those wishing to receive

notices of publications in this series should complete and return the form at the end of

this publication.

National Bureau of Standards Special Publication 500-82

Nat. Bur. Stand, (U.S.), Spec. Publ. 500-82, 58 pages (Nov. 1981)

CODEN: XNBSAV

Library of Congress Catalog Card Number: 81-600112

U.S. GOVERNMENT PRINTING OFFICE WASHINGTON: 1981

For sale by the Superintendent of Documents, U.S. Government Printing Office, Washington, DC 20402

Price $4.25

(Add 25 percent for other than U.S. mailing)
PAGE Hi

TABLE OF CO^fTE^^^S

PAGE

1. EXECUTIVE SUMMARY 1

1 .1 Background 1

1 .2 Extent of Tool Usage 1 1.3 User Reactions to Tools 2

1.4 Projected Tools Usage , . 3

2. INTRODUCTION 5

3. DESCRIPTION OF THE SURVEY 7

3.1 Contractual Requirements 7

3.2 Selection of Participants 8

3.3 Approach to Participants , . 9

3.4 Interview Practices 9

4. CLASSIFICATION OF ENVIRONMENTS 11 4.1 Size of the Software Organization 11 4.2 Type of Organization 13

4.3 Applications and Language 14

4.4 Program Development Environment 15

4.5 Program Running Environment 16

4.6 Computer Type 18

4.7 Involvement with Tool Development . 18

4.8 Survey Coverage 19

5. PRESENT TOOL USAGE 20

5.1 Overall Extent of Tool Usage 20

5.2 Tool Features Classification 23

5.3 User Reaction to Selected Tools 24

6. ADDITIONAL ANALYSIS OF THE SURVEY 27

6.1 Acquisition of Tools 27

6.2 Tools Experience 29

6.3 Future Tools Usage 32

6.4 User Characteristics of Interest for Tool Usage . . 34

7. INTERPRETATION OF FINDINGS 37

7.1 Factors Favorable to Tool Usage 37

7.2 Obstacles to Tools Usage 38

7.3 Requirements for Future Tools Usage 42
PAGE Iv

TABLE OF CONTENTS (CONT.)

PAGE

8. CONCLUSIONS 45

8.1 Overall Tool Usage 45

8.2 Motivation for Acquisition of Tools 46

8.3 Benefits of Tool Usage 46

8.4 Policies that Promote Tool Usage 47

REFERENCES 49

APPENDIX A - LETTER SOLICITING PARTICIPATION 50

APPENDIX B - TOOL USER INTERVIEW FORM 51

LIST OF TABLES

Table Title Page

1 - 1 Numerical Indices of Tool Usage 2

4 - 1 Organization Type and Size of Participants. . 19

5 - 1 Extent of Tool Usage 21

5-2 Numerical Indices of Tool Usage 22

5- 3 Utilization of Function Features of Key Tools 23

6 - 1 Sources of Procured Tools 27

6- 2 Authority for Tool Procurement 28

6-3 Reason for Non-Use or Abandonment of Tools. . 29

6-4 User's Reliability Rating of Tools 30

6-5 Training for Introduction of Tools 30

6-6 Extent of Tool Options Usage 31

6-7 Desired Improvements In Present Tools .... 32

6-8 Proposed Integration 33

6-9 Purpose of Contemplated Tool Acquisitions . . 34

6-10 Major Software Problems ... 35
PAGE 1

SECTION 1

EXECUTIVE SUMMARY

1 . 1 BACKGROUND

Tools have a great potential for Improving the quality of software, for

Increasing productivity In software development and maintenance, and for

facilitating more efficient utilization of computer resources. All of these

benefits are of particular significance to software operations In Federal

agencies because of the critical nature of much of the programming effort and

because of the tremendous volume of software generated and maintained for the

Government.

A study of software tool usage was contracted by the National Bureau of

Standards, Institute of Computer Sciences and Technology (NBS/ICST), as part of an effort to develop methodologies and standards for software quality control,

which Is one of the areas of responsibility of NBS/ICST under the Brooks Act.

This report describes a survey of software tool usage and the analysis of the

survey findings In terms of present and projected use of software tools.

1 .2 EXTENT OF TOOL USAGE

The survey population consisted of 23 sites of which 8 were In the private

sector, 5 were Government-support organizations, and 10 were Government

agencies. The sites represent a wide spectrum of programming environments and

size of software staff. Seven of the organizations were tool developers.

Participation in the survey was voluntary QSection 33*. Interviews based on a questionnaire were conducted at each site concerning software practices, tool

usage, and specific tools in use. When It was found that different software

practices were followed by Individual organizations at a given site, each

distinct type of environment was separately interviewed. This caused the number

of Interviews to be 29.

The extent of tool usage was classified Into three categories:

Minimal - only tools normally supplied as part

of an operating system are used.

Intermediate - special purpose tools to support the mission

of the organization are In use.

General purpose - tools for general purpose software

development and test are used.

* In this summary, square brackets, [ ], are used to direct the reader to

sections and paragraphs of the report that expand on the material that Is here

presented In a very abbreviated manner.
PAGE 2

The 29 Interviews Identified 3 minimal tool users, 7 Intermediate tool

users, and 19 general purpose tool users. The seven tool developers were all

general purpose tool users, and because these were considered a distinct

population, they have been censored from the following analysis of environmental

effects.

To obtain a better Insight Into the overall status of tool usage. It Is

convenient to assign numerical (ordinal) Indices to tool usage. Arbitrarily, 0

Is equated to minimal tool usage, 1 to Intermediate tool usage, and 2 to general

purpose tool usage. The extent of tool usage In various environments Is

represented by these Indices In Table 1-1.

TABLE 1 - 1 NUMERICAL INDICES OF TOOL USAGE

Software Organization Type

Staff Private Gov't Government

Support

Small (<15) 1 1 0

Medium (15-39) 1 1 0.7

Large (40-99) 1 1 .5

Very large 099) 2 2 2

These data Indicate that the size of the software staff has a significant effect

on tool usage. The type of organization did not seem to affect the extent of

tool usage.

In terms of major tool functions, the feature most frequently encountered was Transformation (22 times), which was followed closely by Static Analysis (21

times), while Dynamic Analysis was utilized less frequently (13 times). (A

typical tool that uses transformation Is a preprocessor, a tool that uses static

analysis is a code auditor, and a tool that uses dynamic analysis Is a test

analyzer [5.2].)

1.3 USER REACTIONS TO TOOLS

Most tools were regarded as useful and as eliminating tedious and repetitive

operations. The tools that elicited the most f avorab I e cornnients from the

I mmed late users were ed I tors, pre-processors, and (general purpose) development

tools. Because most of them address a restricted area of software development

and have few options, they are regarded as easy to use [15.3.1]. From the

management point of view, libraries, data base managers, and performance

analyzers received the most favorable comments. The reaction of programmers and

first level supervision to this group was mixed. The use of these tools

Involved some effort on their part and the payoff was not always apparent to

them C5.3.2].
PAGE 3

For several tools used primarily by large and very large software organizations,

difficulties were cited repeatedly by both the immediate user and management;

e. g., the run-time expansion and restrictions on program size or structure

Inherent In the use of some test analyzers caused problems. They were still

regarded as useful because the Information which they furnished was Impossible

to obtain by other means. Requirements analyzers were sometimes characterized

as difficult to use, but again the potential benefits were recognized [5.3.3].

A local manager made the decisions regarding tool acquisition In two-thirds of

the organizations interviewed. In most of the others a headquarters manager was the final authority [6,1.23. In only two organizations, both private, a deliberate cost-benefit study was made prior to tool procurement or In-house

development. In many cases the procurement or development of a tool Is based on

a clearly Identified need, and economic factors are not explicitly considered.

Examples are pre-processor s for structured programming, simulations, and

management tools that generate reports required by a customer. In others the

tool Is perceived to present a better way of accomplishing a necessary task, and

technical management authorizes Its procurement out of discretionary funds.

Program design languages and performance monitoring or optimization tools may be

acquired out of this motivation [5.1.33.

The reliability of tools was generally rated high. Where there were any adverse

comments they were related to failure to furnish adequate diagnostics on improper input [6.2.23. Most of the organizations used a formal training

program when tools were Introduced. About one-quarter relied on informal

training (reading of manuals, on-the-job training, etc.) exclusively; small

organizations were heavily represented In this group [6.2.3J. Tool

documentation was. In most cases, rated as good or adequate, but Improvements

were desired by several survey participants. One adverse comment related to

failure to adequately document "all the special conditions" [6.2.33. A large

majority of the organizations contacted expected to acquire additional tools In the near future. The tools that were to be procured generally had greater

capability than those currently In use [6.3.23.

1 ,4 PROJECTED TOOLS USAGE

The use of tools can be expected to Increase due to two factors: current tools

will become accepted In a broader sector of the software community (particularly

If a "friendly Interface" and better documentation can be provided) and new software engineering practices will create a demand for new tools. The

evaluation of software quality metrics (which are being accepted as contractual

requirements) can best be carried out by means of tools. Cost modeling and

reliability modeling are other areas In which large amounts of data on software

characteristics are required In order to validate or apply current research.

Software tools will be required to support the extensive data collection.

Further impetus for tool usage may arise from the adoption of naming conventions

within large programs and data bases. Adherence to conventions and verification

of uniqueness of names can best be carried out by software tools.

Increased interest at high levels of project and organizational management In gaining visibility and control of software development is expected to speed the
PAGE 4

Introduction of tools. The survey participants frequently commented that

present software practices were fragmented, but that there was perceptible

pressure, or already existing directives, toward greater uniformity L7.1.2]

Some obstacles to tool usage must also be recognized. Among these are lack of

portability (many tools are tied to specific computers and operating systems)

and the amount of training required for efficient use of some tools. A

significant organizational obstacle is that many tools provide benefits that are outside the sphere of responsibility of the Immediate user. Examples are

requirements analyzers (the primary benefits of which are felt at project stages

that are far In the future), test tools (which are applied by the development

organization but benefit operations), and management report generators (which are of greatest use to levels of management that are not directly involved with

the software operations). In all of these areas It is important to establish an appropriate motivation for the tool user [7.21].
PAGE 5

SECTION 2

INTRODUCTION

This report Is the second document generated for the National Bureau of

Standards, Institute of Computer Sciences and Technology ( NBS/ I CST)^ under

contract NB79SBCA0273 by SoHaR Incorporated. This report covers Work Assignment

No. 2, Survey of the Development and Use of Software Tools In Industry. The

overall scope of the Work Assignment Is Identified as follows:

"Determine how different programming environments affect the use of and

direct the development of tools for quality software. Report the current

state of the art regarding the development and use of tools In each of the

programming environments."

In Work Assignment No. 1 of this contract a Software Tool Taxonomy was generated

ClD, and major portions of this were Incorporated In an NBS Special Publication

C2]. This material was then utilized In the effort reported here to classify

features of software tools encountered In the user environment. The

nomenclature for software tool features throughout the present report

corresponds to that In the NBS Special Publication. As a sequel to the survey

of software tools, guidelines for the Introduction of software tools Into a programming environment will be generated under Work Assignment No. 3 of this

contract.

The Software Tools Studies contract Is one of a series of efforts undertaken by

NBS/ICST In connection with Its responsibilities under the Brooks Act (PL

89-306)whlch alms to aid Government agencies in i mprov! ng the cost-effect Iveness

In the selection, acquisition, and utilization of automatic data processing

resources. NBS efforts to satisfy its responsibilities under the Brooks Act

include research In computer science and technology and the development of

Federal Government-wide standards for data processing equipment, practices, and

software. The software standards efforts comprise six families of standards,

one of which deals with software quality control. Although it is recognized

that software tools can aid in software quality control, NBS/ICST has concluded

that there did not exist a clear body of techniques for making effective use of

tools. The present and related efforts are intended to f 1 1 I this gap.

The potential benefits of Increased software tool usage by Federal agencies are described in a recent GAO report C3j which states:

"During Its work In automatic data processing, GAO found many areas where

using better software tools and techniques might have saved a considerable

amount of money and trouble. These Include:
PAGE 6

—Management control.

--User needs, —Maintaining and modifying production computer programs... —Software conversion..."

The purpose of this report Is to describe the users and the use of software

tools, to determine how the environment affects tool usage, and to Identify

factors which might lead to Increased software tool usage In the future. In line with these primary objectives, the extent of tool use among several

classes of software developers Is studied, the features of tools encountered are analyzed, and the benefits, as well as difficulties experienced with current

tools, are related.

Section 3 of the report describes the goals of the survey, how It was planned

and conducted, and the selection of participants. Section 4 discusses the

factors In the environment which might affect tool usage and Introduces the

classifications that are used In the remainder of the report. The central

findings of the survey regarding present tool usage are contained In Section 5

which covers the extent of tool usage, the classification of the tools

encountered, and user reaction to selected tools. Additional analysis of survey

findings, primarily oriented towards future tool usage. Is presented In Section

6. An Interpretation of the findings In terms of factors that favor or Inhibit

tool usage and requirements for tools usage In the near future are presented In Section 7. Conclusions regarding the broad aspects of tool usage are summarized

In Section 8.

Appendices A and B contain, respectively, the letter soliciting participation In the survey and the survey questionnaire. The findings from each Interview are synopslzed In a companion volume to this report Il4ll. Before leaving this Introduction, the author wishes to acknowledge the

contributions of collaborators In the survey effort. H. M. "Bob" Farmer

conducted Interviews and analyzed survey results, Myron Hecht analyzed survey

results and formatted them for Inclusion In this report, and Donald J. Relfer

classified the tools encountered In the survey. Substantia! assistance in

conducting the survey. In contacting survey part 1 c 1 pants, and In providing

background material, was rendered by the technical monitor for the tools

studies, Mr. R. C. Houghton, Jr. Continued encouragement and many helpful

suggestions were furnished by Dr. Martha Branstad and Dr. Selden L. Stewart, who

supervised the effort.
PAGE 7

SECTION 3

DESCRIPTION OF THE SURVEY

This section describes the goals of the survey, how It was planned and

conducted, and the overall composition of survey participants. The following

specific topics are covered In the order Indicated:

Contractual requirements

Selection of participants

Approach to participants

Interview practices

3.1 CONTRACTUAL REQUIREMENTS

The Specific Assignments Section of the Statement of Work under which the survey

was conducted reads In part:

"The contractor shall determine the different programming environments that

affect the different types of tools used In software quality control.

Typical environments shall range over small systems developed by a single

programmer, medium systems developed by small teams, and large scale

systems. The software and systems developed may be characterized as scientific, management, business oriented, real-time command and control,

word processing, real-time transaction oriented, or some other application

area."

"The contractor shal I report the current state of the art regarding the

development and use of tools In each of the environments. The final report

shall Include the following:

* Types of tools being developed and used

* When In the software life cycle tools are used

* Software standards that the tools are attempting

to enforce

* Cost/benefit analysis of the tools (If this data

can be easily obtained)

* Compatibility of tools with respect to commonality

of representation and redundancy."

The Statement of Work required at least eight private sector organizations to be

surveyed. At the suggestion of the contractor the scope was subsequently

enlarged to Include Government and Government-support organizations as well.

Also, In the planning discussion with NBS/ICST personnel, the meaning of

'software quality control' In the first quoted sentence above was clarified to

Include all activities that can be expected to affect software quality.
PAGE 8

3.2 SELECTION OF PARTICIPANTS

The key factor In the selection of participants for the survey was to cover the

wide range of programming environments specified by the contractual requirements

within the limited resources and time available. The major emphasis was on Including organizations of a wide range of size and on obtaining approximately

equal representation of private. Government-support and Government activities.

Also, representation of scientific and business oriented programming was sought

within each organizational category. The detailed classification guidelines for

the survey are discussed In Section 4, as well as the characteristics of ttie

survey population.

It was realized at the outset that tool usage among tool developers represented

a special case that was of Interest In the overall assessment of software tool

usage. Therefore an attempt was made to Include several tool developers In the

survey population. Although tool developers constitute only a very small part

of the total software population, and thus It Is difficult to capture a representative sample In a small survey, their participation was easily

obtained. Tool developers are motivated by nature and self-interest to furnish

Information about their tools. So as not to bias the survey by the

over-representation of tool developers among the survey population, significant

findings are In most cases stated separately for tool developers and others.

In order to husband resources, a major factor In the selection of participants

was location In the vicinity of Los Angeles where the contractor's offices are

situated. Of the 23 sites visited, only seven were located outside of

California, Of these, five were Government organizations for which no

equivalent In terms of size and mission could be located near Los Angeles, One

organization each In the private and Government-support sectors was outside of

California, Thus, a strong bias In the geographic distribution of participants

had to be acknowledged. It was not believed that this would have a major

bearing on the significance of the findings.

Publicity during the conduct of the survey prompted four organizations to

volunteer their participation. One of these was a Federal agency, the others

were private companies. Due to schedule constraints It was possible to contact

only one of these organizations In person, and findings from that Interview were

Incorporated In the following analysis. The Interview questionnaire was sent to

the other volunteers, and these responded In writing. Because of possible

Inconsistencies In the Interpretation of some questions, these responses have

not been utilized In the quantitative data In Sections 5 and 6 (except where

specifically stated). Tools and software problem comments from the volunteers

have been Included In the qualitative discussion where appropriate.
PAGE 9

3.3 APPROACH TO PARTICIPANTS

Participation in the survey was entirely voluntary. After an organization had

been tentatively selected by the criteria mentioned above, an informal Inquiry was made, usually by phone, to determine willingness to participate In the

survey. The scope of the survey, the nature of the questions, and the

approximate length of the interview (one to two hours) were discussed. Of those

contacted, only two declined, both because knowledgeable personnel were fully

committed during the period over which the interviews were scheduled. In addition, one Federal district office in Los Angeles referred the request to the

headquarters office, and the latter was then selected as a survey site.

Once agreement on participation had been reached, a letter was sent by NBS/ICST

that stated the interest of the sponsor and made a formal request for an interview by SoHaR personnel. A copy of this letter Is Included as Appendix A

of this report. All recipients of these letters were. In fact. Interviewed. In some cases a request for anonymity of the Interviewee or of the organization was made. It was therefore decided to omit this Information for all Interviews.

However, no descriptive material has been omitted or disguised. It is believed

that the classifications described in the following section and used throughout

the rest of this report captured the primary detail of the user that has a bearing on software tools utilization. Additional Information Is contained In

the interview synopses which are being published separately.

In several Instances It was apparent from the Initial telephone contact that

there were several Independent software centers within a given organization, and

that it would be desirable to contact more than one of these. In two

organizations three Interviews were actually conducted, but It was found that

the third one contributed little additional material. It was then decided to

restrict Interviews to two within a given organization, and to select centers

that could be expected to reflect wide differences.

3.4 INTERVIEW PRACTICES

All interviews were conducted at the respondents' premises In accordance with a previously agreed to Interview schedule and, with a few exceptions cited In the

following, all appointments were adhered to. In one Government-support

organization where two Interviews had been set up, one of the prospective

respondents was not available. Information obtained at the site Indicated that

the tool usage was not materially different from that of the center at which the

Interview was conducted; no attempt was made to reschedule that Interview

because considerable travel was Involved. In one Federal agency the Initial

contact for the Interview was Involved In a high level meeting and the Interview

was conducted with a substitute who was very knowledgeable In most areas. Answers to the remaining questions were later obtained by phone from the

original contact.

To conserve time during the interviews, and also to permit the survey to

generate consistent Information, a standard set of questions was used

(reproduced In Appendix B) . The interview consisted of two parts. The first

part covered the user environment, significant software development practices.
PAGE 10

and general experience with software tools. During the second part of the

Interview the experience with one specific tool was explored In depth. Including

the motivation for tools acquisition, actual versus expected use of the tool,

recommendations for modifications or additional uses of the tool, and comments

on actual or possible Integration of the tool with other software development

aids. Where tools of significantly different scope were In use at a given

center (e.g., one tool for software development and another one for project

management), part II of the Interview covered each of these separately. Notes

were made at the time of the Interview but the Interviews were not recorded.

About midway through the Interview schedule. It became apparent that current

tool usage and future Interest In tools are Influenced by what the respondent

perceives as the major difficulty In generating high quality software. The

following question was therefore added to part I: "What are the major software

development problems In this environment?". Answers to this question are

discussed In Section 6,

Because details on tool options and performance were not always available at the

time of the Interview, descriptive literature on a tool or a user manual were

obtained In some Instances, In other cases, the Interviewees volunteered to

furnish tools catalogs or technical papers describing development practices In their environment. Material obtained In this manner Is, where applicable.

Integrated Into the Interview synopsis.

Two organizations requested that the record of the Interview be submitted to

them for review and additions or corrections were made In this process. In all

other cases, the Information obtained during the Interview formed the only basis

for the analysis In this report.
PAGE 11

SECTION 4

CLASSIFICATION OF ENVIRON»CNTS

This section considers a fairly large number of factors In the environment that

might affect software tool usage. Some of these are eliminated either because

they are highly correlated with more readily recognized factors, or because the

discussion shows that their effect on tool usage is difficult to assess. The

elimination of factors that are Irrelevant or have only a minor effect on tool

usage helps to focus the remainder of this report on factors of major

sign If Icance,

The following factors will be discussed In the order listed:

Size of software organization

Type of organization (private. Government-support, Government)

Applications (scientific, MIS) and language

Development environment (batch, interactive)

Program running environment (batch. Interactive, real-time)

Computer type

Involvement In tools development

Survey Coverage

4.1 SIZE OF THE SOFTWARE ORGANIZATION

Large organizations are expected to be more inclined to use software tools than

small ones because (1) they have greater resources at their disposal, (2) they can amortize the tool and training costs over a larger base, (3) their software

development procedures are usually more formal, (4) there Is more emphasis on use of tools for organizational control of the software development process, and

(5) they are more likely to be required by the customer to employ software tools

or to furnish data which are most effectively obtained by tools.

Some problems are encountered In defining what Is meant by 'size' and how to

measure It, Software development and maintenance budgets, or the number of

computer hours used for these functions, are suitable indicators of size, but

they are not always known or made public. The size of the programming staff Is an equally valid indicator, and one that Is usually more readily available. It Is used In the following as the size measure. It may also be argued that the

overall size of the organization, rather than Its software component, should be

considered. Resources, and possibly also the degree of formality In

development, are Indeed determined by overall size. Nevertheless, the narrower definition Is preferred because tool usage, as described below. Is based on the

scope of control exercised by software management.

Where all software development Is carried out In a single location and under a single line management, the size that Is significant for tool usage is taken as the total programming staff engaged In development, test, and maintenance. Where

software is being developed or maintained In multiple locations, or where it Is

1
PAGE 12

distributed among several line managements (e, g., one software group supporting

a revenue function and another one a disbursement function). It must be

determined on an Individual basis whether tool usage will be governed by

Identical criteria for all of the locations or groups, and in that case the

'size' is that of the total programmer population. If local criteria govern

tool usage, then the 'size' Is that of the programming group for which the

criteria are val i d.

Most software tools, and particularly development tools that provide the format

and audit functions, impart uniformity to the software which has been processed

by them. This uniformity is regarded as a distinct benefit In environments

where there Is a deliberate attempt to control the software development process

and software qual Ity, The size of a software organization, as far as Its effect on tool usage Is concerned, is In a significant way determined by the scope of

control of the management level that enforces uniformity of the development

process and of quality assurance. Thus, a large software organization. In which

Individual small groups set their own software quality standards represents a tool usage environment that is equivalent to that of several small

organizations.

This definition of size is particularly important in classifying the environment

represented by Federal agencies. It will be recognized as extremely naive to

represent all Federal software development as a single environment. But even major departments or individual agencies or bureaus usually do not enforce a sufficiently uniform approach to software quality that would make them the

proper unit for the assessment of size in the present context. The size of a software organization is in the following expressed as the number of programmers

who are subject to a single set of guidelines for software quality and the

software development process.

Size, as defined above, can of course range from a handful of programmers to a staff of several hundreds or even thousands. For the study of tool usage It has

been found sufficient to divide this continuum into four categories:

Small - up to 14 programmers

Medium - 15 to 39 programmers

Large - 40 to 99 programmers

Very large - 100 or more programmers

These categories were primarily established on the basis of the supervisory

structure by which they are controlled. The small environment is typically

managed by a single decisionmaking Individual. The medium environment is one where two layers of decisionmakers may be Involved, but where there still Is (or

can be) direct communication among all individuals engaged In software

development. The large environment usually Involves three layers of management

concerned with software quality or development practices, and formal

communication among programmers may be the rule. Very large environments

typically have staff departments that establish or coordinate software

development practices and that evaluate. Introduce, and frequently develop

software tool s.
PAGE 13

4.2 TYPE OF ORGANIZATION

This study distinguishes between three organization types: private. Government,

and Government-support. The latter Includes Federal Contract Research Centers,

National Laboratories, and similar quasi -Governmental bodies. These have fewer

procedural constraints than Government agencies and may be more motivated to

achieve a minimum cost solution to software development.

The conventional expectation Is that all private software development is profit

oriented and that tool usage will be governed primarily by economic

considerations, while Governmental agencies are procedure oriented and will use tools accordingly. As will be seen In the following section, these stereotypes

are seldom found In current software practice. To a large extent, tool usage Is

In all three types of organizations governed by preferences of one or a few

individuals who consider themselves 'experts' In this field. These preferences

may have been acquired by study, by generic or specific tools experience, or through meetings of user organizations of the major computer manufacturers.

A further classification of the private sector Into Government end use and

non-Government end use of the software or the service provided by the software

may also be considered. To some degree this distinction parallels that provided

by size, since the very large private sector organizations mostly produce

software that In some way services Government functions or Is subject to

Government regulation. This subdivision Is not further pursued here.

Similarly, a division of Government organizations Into civil and defense sectors

may be considered significant for tool usage. This division Is largely

paralleled by the distinction between scientific and MIS software which will be

discussed In the following subsection. Most of the scientific program

development Is carried out by agencies associated with the Department of Defense

or NASA, or by portions of other agencies which pattern their software

development practices deliberately after those In the defense sector.

Furthermore, in MIS software development there seems to be little difference

between the practices of DoD and non-DoD agencies.

Although the findings of this survey do not indicate a large difference In tool

usage between the three organization types discussed here. It Is believed that

continued use of this distinction Is warranted. Status as a Government

organization provides for the developer access to a large number of tools

currently In Government Inventory; easy communication with users (and frequently

also with the developers) of these tools; and. In some cases, the possibility of

using the tools on their present host via remote Job entry or Interactive

terminals. Also, it Is expected that there will be greater emphasis on uniform

or, at least, similar software development practices among most Federal

agencies. All of these will provide a motivation to use tools which Is not

present In the private sector.

Government-support organizations are expected to continue to occupy a middle

ground, being partly Influenced by the Increasing uniformity of development

practices among agencies which they serve and being more concerned with the

economic factors of tool usage than Governmental bodies.
PAGE 14

4.3 APPLICATIONS AND LANGUAGE

The two major program applications which affect tool usage are scientific and

administrative. Because the latter have recently expanded considerably into the area of information retrieval, the term management information systems (MIS) is

more descriptive of the types of programs handled and is generally used in this

report.

The predominant language for scientific programming is FORTRAN. PL/1, BASIC,

PASCAL and APL are used to a much lesser extent. In the aerospace and defense

sector, JOVIAL (Air Force), CMS-2 (Navy), and HAL-S (NASA) are required to be

used for embedded computer applications (and some others) but FORTRAN is a recognized and popular language there, also. The wide use of FORTRAN Is

significant for tool usage because many tools are specifically designed for this

language. Thus, the FORTRAN user has a much wider choice of tools than

programmers working in other languages.

The logic structures IF THEN ELSE, DO WHILE, and DO UNTIL which are preferred

for structured programming are not directly implemented in the conventional

FORTRAN dialects, and this has given rise to a large number of pre-processors

which translate these constructs into FORTRAN and usually also provide some

editing and format functions. These tools fill a unique, language dependent

need of the scientific programming environment. Another language specific tool

need arises from the large number of FORTRAN dialects in use which inhibits

transfer of programs from one environment to another. This need has been met,

on one hand, by analyzers that determine whether a program adheres to a standard

subset of FORTRAN and, on the other hand, by tools that translate from one

dialect to another.

In the 1^1 S environment COBOL Is the most widely used language, with PL/1 and

ALGOL having a very minor share. A number of tools are specific to COBOL, and a number of others (file and data base managers, computer utilization reporters)

address specialized needs of the MIS environment. Many current applications in

this field, and practically all new ones, run Interactively, usually from a large number of terminals. This places a high premium on computer resource

utilization and Implies a high benefit If the run time of programs can be

reduced. Dynamic analysis tools with resource utilization, timing, and tuning

features are frequently employed. Some of the more popular of these are

specifically designed to eliminate Inefficiencies due to current COBOL

compi I ers.

MIS programmers frequently have less formal training than those working In the

scientific data processing field. This Is particularly true of some Federal ADP

agencies where on-the-job training programs are used to upgrade personnel from

the computer operator and clerical categories to the programmer classification.

Unless tool usage Is made a specific part of the training, such personnel will

have no understanding of software tools and will therefore not be Inclined to

use them.
PAGE 15

A positive factor for use of tools In the MIS environment Is that commercial

computer applications (which are primarily MIS oriented) are still expanding at

a rapid rate which motivates tools development for this market. Many of the new computer installations are at the low end of the performance range (mini- and

micro-computers), and tools for this specific area are attaining an Importance

which they have never had before. Because these tools are Intended for less

sophisticated users than tools running on mainframes and because many address

basic program development and maintenance needs, they can be expected to be of

considerable benefit to small Federal agencies.

It is sometimes assumed that scientific programmers tend to be more

sophisticated than those In the MIS environment and are thus more Inclined to

use tools, to adapt tools (make minor revisions required for an application),

and develop tools. The survey showed much greater Involvement with tools

development among scientific programmers but no significant difference between

the two environments In the extent of tool usage. (The MIS population was heavily biased towards very large organizations and this may have affected this

latter observation.) A negative factor specific to the scientific environment

Is that much programming Is being done by personnel who are primarily scientists

or engineers and who have little training or Interest In the formal aspects of

computer programming. These groups are highly motivated to get a program

running In as short a time as possible and have little concern for

maintainability, portability, or documentation. They will use only those tools

that are obviously beneficial for the Initial creation of a program, e. g.,

editors, compilers, and debug aids.

The area of office automation (word processing, electronic mall, and

communications management) Is becoming distinct from that of management

Information systems. It Is being served by personnel who are usually without

any computer training and work with programs (which are really tools) that

generate prompts, menu selection, etc. This field Is certain to Increase In Importance and will require much standardization effort (e. g., to permit

Interchange of media and to facilitate personnel transfer). However, at present

It Is outside the scope of conventional software tools. One of the

organizations contacted In the survey Is a developer of word processing

software. Although the findings from that Interview are Incorporated In later

analysis, the sample Is much too small to permit any generalizations.

4.4 PROGRAM DEVELOPMENT ENVIRONMENT

Two distinctions In the program development environment may affect tool usage:

submittal mode (batch vs. Interactive), and assignment practice (Individual vs,

team)

Most software development environments today offer a choice of Interactive and

batch programming with the decision left either to the Individual programmer or to a team leader. Exceptions are found at both ends of the size scale. Some

small software organizations that depend exclusively on off-site equipment offer

only Interactive development. In some very large organizations, applications

development Is conducted only on Interactive terminals, partly as a means of

promoting tool usage (e, g., structure graphics, data reference generators).
PAGE 16

There Is usually a capability for developing systems programs and others that

are not tied Into the principal application In batch mode.

Very few, if any, tools are Inherently constrained to either batch or

Interactive use, although the specific Implementation may be directed to one of

these modes due to the prompts, graphics representations, etc., that are

employed. Some dynamic analysis tools are used predominantly in the batch mode

because they entail a large run-time expansion factor which makes It inefficient

to use them from a terminal and because the output Is usually desired In

hard-copy form. In general, however, the usefulness or efficiency of software

tools Is not greatly affected by whether the programming environment is batch

oriented or uses interactive terminals.

Only in small software organizations Is individual programming employed today.

Medium sized and larger organizations favor a team approach, even when the scope

of the effort would permit assigning It to a single Individual. Part of this

tendency Is due to the need for training Junior programmers which can be very

effectively handled by a team assignment. Another advantage Is that the project

schedule Is made less dependent on continued availability of a single Individual

when the effort Is assigned to a team. Because they tend to Impart a uniform

format to a software product and because they automate project recordl^eeping (e.

g,, the schedule feature of static analysis tools), tools tend to reduce some of

the differences between Individual and team programming. On small software

development efforts there does not seem to be a major distinction In the

motivation for tool usage between individuals and programming teams.

Larger projects are handled by larger organizations which, as a rule, employ

team programming. For some projects, large teams (more than 10 programmers) or multiple teams are required. In this environment the use of tools for

scheduling, cross reference, and program library management becomes desirable or mandatory. This aspect of tool usage Is, however, already accounted for by the

size of the software organization. Since this Is more permanent and more easily

determined than the attributes of the programming environment, a separate factor

for the latter does not appear to be required. The assignment practice

(individual or team) Is therefore not regarded as a significant factor in

determining software tool usage.

In summary, the program development environment as such has little effect on

tool usage In general, although a specific tool may be only (or more) useful in

a batch or Interactive setting. The size of the programming team (which In turn

Is governed by the magnitude of the program) has some effect on tool usage, but

this Is highly correlated with the effect of organization size and is better

modeled by the latter. Therefore program development environment is not further

considered as a classification factor.

4.5 PROGRAI^ RUNNING ENVIRONMENT

Three major categories of running environment may affect tool usage: batch.

Interactive, and real-time. Typical programs that run primarily In batch mode

are large scientific models (e.g., for determining neutron flux in a nuclear

reactor) or periodic report generators (e. g., for monthly transaction
PAGE 17

summaries). Typical Interactive programs In the scientific area are those that

perform spectral analysts or that generate graphics. In the MIS area,

practically all programs for aperiodic querying of data bases are Interactive,

e.g., those that are used to handle Inquiries about benefit payments or personal

status of an Individual. Real-time programs Interact with a physical

environment which Is subject to effects not controlled by the primary

application program. Aircraft or satellite attitude control programs and

real-time simulation programs are representative examples.

To the extent that batch programs are run Infrequently (not a necessary

Inference), there may be little Incentive to optimize run-time or memory

requirements. However, In all other respects, they represent the same tool

needs (for quality, documentation, etc.) as other programs of an equivalent

application.

Interactive programs, particularly those for MIS applications that are being run throughout the business day, place heavy demands on computer resources, and they are therefore prime candidates for the use of optimization tools. Because a large amount of current COBOL programming Is directed at the Interactive

programs, this aspect of tool usage Is more effectively considered as a language

and application (MIS) specific factor. Interactive scientific programs are usually not run on a continuous basis and therefore the benefits of optimization

are not as significant.

Real-time programs usually must execute very efficiently In order to meet the

time constraints Imposed by their environment; e. g., aircraft flight control

routines must furnish a complete three-axis output 25 to 40 times per second. In order to assure the required control over timing, these programs, or parts of

them, may be coded In assembly language. This may, at times, generate

requirements for special tools, such as decompilers (tools that combine the

static analysis feature of structure checking with the transformation feature of

restructuring) or flowcharters that accept assembly language input. In general,

however, the tool usage Is not significantly different from that encountered In

other running environments.

The critical Ity of the application (which tends to be particularly high In

real-time environments) may also Impose special tool requirements. However, It

will be found that a failure In practically any major application program,

partlcuarly In a Government environment, can have very serious consequences that

warrant application of test tools (dynamic analyzers) and static analyzers.

That these are not currently In wide use Is partly due to lack of Information

and partly due to the difficulties In applying some of these tools. Differences

In the critical Ity of programs among the classes discussed here are relatively

Insignificant In determining the need for these tools.

The security aspects of the running environment may also affect tool usage.

Typical of special security needs are programs that deal with the control of

nuclear weapons or with the guidance of vehicles used for the delivery of

nuclear weapons. This environment has not been Included In this survey, but It

Is Inferred from the general literature that It may Involve quite extensive tool

usage. The Defense Advanced Research Projects Agency has sponsored the

development of tools for formal verification of programs, and It surmised that
I PAGE 18

these are Intended for security sensitive applications C5,6ll. A specific

Instance of a program verification tool Is AFFIRM C6l].

Aside from the rather special case cited In the preceding paragraph, the various

program running environments do not represent significantly different tools

needs. This factor Is therefore not further considered In this classification.

4.6 COMPUTER TYPE

Both the development (host) and the target computer type can have a major effect

on tool usage. This effect Is due to the size of some of the software tools,

their execution time requirements, and the operating system and language

requirements posed by the tools. It Is quite natural for commercial tool

development to address the most widely used computers and operating systems.

Federal agencies have a large variety of computer types, and comparatively few

of these are those In common use In other environments. This poses a serious

obstacle to employment of commercial software tools, and the variety of

computers also restricts the benefits of common development or tools sharing

among agencies.

A number of approaches can overcome this obstacl,e but none of them are cheap or Immediately available (see Section 8), At present the dependence of tools on specific computers and operating systems must be considered a serious obstacle

to wider employment of tools. The great variety of computers used by Federal

agencies makes tools standardization and tools sharing very difficult, A

classification of tool usage by specific computer types may be beneficial but It was not within the scope of this survey.

4.7 INVOLVEMENT WITH TOOL DEVELOPMENT

As might be expected. Involvement In tool development provides a strong

motivation for tool usage, not only of the local product but also of that from

other sources. This was well borne out In the tool user survey, where every

tool developer used at least one general purpose tool. Since some tools

development takes place In small or medium organizations, this makes their tool

usage distinct from that of non-developers.

Several factors seem to be Involved: Interest In software technology (as

distinct from programming for a specific application), recognition of the

similarity of programming tasks that stimulates an attempt to automate them, and

the greater facility of tools use that comes from being active In their

development (recognition of options and switches, being familiar with the format

of an Instruction book, etc.).

Classification as to Involvement In tools development therefore seems to be

desirable In order to Identify the likelihood of tool usage In a given

environment. The classification as a tool developer Is based on primary or major product of the organization. Occasional tools development, particularly

development of simulations or report generators, was not so classified. Tools

development may not be Immediately useful for bringing about more widespread use
PAGE 19

of software tools. It Is neither possible nor particularly desirable to make

every lead programmer into a tool developer. However, the fact that familiarity

with one tool seems to stimulate interest in others should be noted. This may

motivate a greater effort for introduction of the first tool Into a given

environment, because it may be assumed that further tools will then be adopted

more read! ly.

4.8 SURVEY COVERAGE

The distribution of participants In the survey with regard to type of

organization and size Is indicated in Table 4-1. Where two centers in a given

organization were contacted each Is treated as an Individual respondent In Table

4 - 1 and in subsequent analyses in this report.

It Is seen from Table 4 - 1 that approximately equal participation by al I three

organization types was obtained. In terms of size, the sample is somewhat more

heavily weighted toward large and very large development groups than had been

anticipated. This Is partly due to the fact that Government organizations are more likely to fall Into these size groups. However, a fair representation of

all size groups Is present, and for each size group at least one participant was obtained In each of the organization types.

TABLE 4 - 1 ORGANIZATION TYPE AND SIZE OF PARTICIPANTS

Size Number of Participants Total

Private Gov't Government

Support

Smal 1 3 2 1 6

Med I urn 1 2 3 6

Large 2 1 4 7

Very 1

3 3 4 10

Total 9 8 12 29

The volunteer participants which returned written responses are not Included In

the above. These were all private organizations, one each in the small, medium

and very large categories.

Twenty-one of the participants listed In Table 4 - 1 did predominantly

scientific programming, practically all In FORTRAN. Six of the participants

worked predominantly In the MIS area, five using COBOL and one ALGOL. One

organization had an approximately balanced MlS/sclentlf Ic workload, and one was

engaged In word processing development, using a microprocessor assembly

language. All MIS organizations except one were Government agencies.

Seven of the organizations In the survey were tool developers. Two of these were Government-support organizations. All others were In the private sector.

All of the tool developers were from the scientific programming community.
PAGE 20

SECTION 5

PRESENT TOOL USAGE

This section summarizes the findings of the survey in several dimensions.

First, an overall classification by the extent of tool usage is presented. The

tools identified by users in part II of the interviews are then classified by

the previously generated taxonomy so as to permit an analysis of individual tool

features that were encountered. Finally, comments are presented on certain

groups of tools for which specific advantages or disadvantages could be

Identified from the survey findings. This material is organized under the

fol lowing topics:

Overal I extent of tool usage

Tool features classification

User reaction to selected tools

5.1 OVERALL EXTENT OF TOOL USAGE

It is nearly Impossible to Interact with a current computer without the use of some tools. Assemblers, Interpreters, debug aids, and compilers can all be

rightfully called software, tools but they are not of the type that one

associates directly with software quality tools. Admittedly, however, any

distinction based on the contribution to software quality will be quite

subjective. In the above grouping, a good case can be made that compilers

support software quality because a source program in a high order language Is

more readily reviewed and maintained than one coded in assembly language, A

better delimiter is that the tools listed above are normally part of the

operating system and are acquired without a deliberate decision on the part of

the user. Participants In the survey who used only tools that are customarily

furnished with an operating system were classified as 'minimal tool users'.

Another level of tool usage is represented by those who acquire, or In some cases develop, special purpose tools suited for their mission and with no direct

Impact on software quality. Simulators and file managers are typical tools in

this category, as are editors and precompilers that do not Include static

analysis features (see tool features classification below). Survey participants

who only used tools in this category are called 'intermediate tool users'.

The final group of participants Is made up of those who deliberately acquired or developed general purpose tools that include static or dynamic analysis features

or that otherwise directly support software quality. Test and audit tools,

program design languages, optimizers, and performance measurement tools are

representative of these tool classifications. Survey participants In this

category are called 'general purpose tool users', and these represent the most

extensive level of tool usage.
PAGE 21

The overall classification of participants by the extent of tool usage,

expressed In the three categories just described. Is presented In Table 5-1,

The participants are defined by the Interview number (the alphanumeric preceding

the colon), and the extent of tool usage Is Indicated by the letter following

the colon. An asterisk preceding the Interview number Indicates a tool

developer as defined In the preceding section.

TABLE 5 - 1 EXTENT OF TOOL USAGE

Staff Organization Type

Size Private Gov't Government

Support

Sma I 5:M

*9:G

14B:G

1 1A: I 22: I 10A:M

Med I um 2: I *1 1B:G

188: I 3

8

21

Large *14A:G

^17:G

18A: 4A

4B

15

20

Very I. *1 :G

*13:G

19:G

6:G

12A:G

*12B:G

7:G

10B:G

16:G

23 :G

Tool Usage Designation:

M - minimal I - intermediate

G - general purpose

One of the immediate observations suggested by this table Is that all tool

developers are general purpose tool users. Thus among tool developers In the

survey, the extent of tool usage was not affected by size or organization type

(note that none of the Government organizations was classified as a tool

developer by the criteria stated In Section 4, although some Incidental tools

development was carried out at two of the sites).

In the following analysis, tool developers are omitted from the data. To obtain

a better insight Into the overall status of tool usage. It Is convenient to

assign numerical (ordinal) Indices to tool usage. Arbitrarily, we assign 0 for

minimal tool usage, 1 for Intermediate tool usage, and 2 for general purpose

tool usage. The content of a eel I In Table 5 - 1 can then be characterized by a number that represents the average tool usage of the members of that cell.

Thus, among small private participants there are one minimal user (Index 0) and
PAGE 22

one general purpose user (index 2). The average Index for that cell Is

therefore 1

In order to permit meaningful aggregation of cells In a row or column, a weighting factor corresponding to the number of entries In a cell must also be

available. This weighting factor is shown In the left part of Table 5-2. The

first row average Is computed as {2x1+2xl+1x0)/5= 0.8.

It is emphasized again that tool developers have been omitted In this

quantitative evaluation of tool usage.

TABLE 5-2 NUMERICAL INDICES OF TOOLS USAGE

NUMBER OF USERS, N[i,j]

Size Organization Type

Private Gov't Gov't

Support

Smal 1 2 2 1

Med i urn 1 1 3

Large 0 1 4

Very 1

1 2 4

Total (N) 4 6 12 or Column Average (T)

TOOL USAGE INDEX, T[i,j]

Organization Type

Private Gov't Gov't Row

Support Average

1 1 1 0.8

1 1 0.7 0.8

1 1.5 1 .4

2 2 2 2

1 .3 1 .3 1 .3 1 .3

Before discussing these findings in detail. It should be recognized that the

total population involved here Is not large enough for a meaningful statistical

evaluation. Nevertheless, the difference in tool usage by small and medium

software organizations, as opposed to that by large and very large ones, is made

very clear in this presentation. Also, for the population analyzed here, the

organization type does not seem to affect tool usage. Small and medium

Government organizations had a lower score than equivalent private and

Government support organizations, but the population of the Individual cells

involved Is so small that this finding should not be generalized.

For the six MIS organizations in the survey, the average Index of tools

utilization is 1.2, slightly less than for the population as a whole. Again,

the small difference and the overall limitations of the survey suggest caution

In Interpreting this finding.
PAGE 23

5.2 TOOL FEATURES CLASSIFICATION

In a Work Assignment that preceded the Tool User Survey, a Taxonomy of Software

Tool Features was generated, and this formed the basis of an NBS Special

Pub I Icatlon [2],

This taxonomy deviates from previous efforts at tools classification In that It

evaluates a tool as a processor rather than by the purpose for which the tool

might be employed. The features of a processor are defined by

Input/function/output capabilities. The key tools Identified In part II of each

of the Interviews were classified by this taxonomy. A summary of the principal

functions found In these tools is presented In Table 5-3. The functions are

classified Into three major groups: transformation, static analysis, and dynamic

analysis. Each of these Is In turn divided Into detailed functions. Thus,

under transformation the detailed functions that were most frequently

encountered were editing, formatting. Instrumentation, optimization,

restructuring, and translation.

TABLE 5-3 UTILIZATION OF FUNCTION FEATURES OF KEY TOOLS

Feature Number of

Uses in

Kev Tools

Transformation 22

Translation 10

Editing 9

Formatting 8

Restructuring 6

Static Analysis 21

Management 1

1

Scanning 10

Auditing 3

Dynam ic Ana lysis 13

Coverage Analysis 5 Tracing 3

Simulation 3

The transformation feature was utilized by pre-processor s, editors, general

development tools, and test analyzers; all of these were represented by multiple

entries In the key tools list. The static analysis feature was used by some

pre-processor s, all test tools, general development tools, and libraries.

Dynamic analysis was Involved In simulations, performance optimization, and test

tools. The frequency with which certain detail features were utilized suggests
PAGE 24

that considerable simplification of tool usage may be possible In an Integrated

tools environment.

A very Interesting finding, somewhat related to tools Integration, Is that six

of the tools used all three major function features. With one exception, these

tools were In use at large or very large organizations. The exception Involved

an Integrated collection of small tools that was used and regarded as very

helpful In a small software organization. It must be stated, though, that this

small software organization was part of a very large overall organization, so that one assumes that the sophistication In tool usage was not completely

home-grown.

5.3 USER REACTION TO SELECTED TOOLS

The reaction of the Immediate user to the tools Identified above was not

uniformly favorable. Most tools, however, were regarded as useful. Comments

on some selected tool groups are presented below under headings of:

Immediate user oriented tools

Management oriented tools

Tools primarily used by large organizations

Special purpose tools

5.3.1 Immediate User Oriented Tools

The tools that elicited the most favorable comment from the Immediate user were

editors, pre-processors, and (general purpose) development tools. The detailed

features that most of these have In common are editing, formatting, scanning,

and, to a lesser extent, tracing. From the user's point of view, these tools

obviously simplify the generation and correction of programs, and the computer

Input operations. Because most of them address a restricted area of software

development and have few options, they are generally regarded as easy to use. User documentation seldom exceeds 20 or 25 pages. In most cases the

documentation was adequate for the original version of the tool. Some

complaints were stated regarding updating of documentation for revisions In the

too I s

5.3.2 Management Oriented Tools

From management's point of view, libraries, data base managers, and performance

analyzers received the most favorable comments. In addition to the management

function of static analysis, the detailed features most frequently found In these tools were editing, restructuring and optimization. Many of these tools

furnished reports In a format tailored for second and higher level management,

and they helped to Improve software or computer productivity. The reaction of

programmers and first level supervision was not always positive. The use of

these tools Involved some effort on their part and the payoff was not apparent

to them.
PAGE 25

The most adverse comments on the use of some of these tools arose from

misunderstanding of the purpose for which they were employed. A resource

utilization analysis tool was declared useless for achieving performance

Improvements on specific programs, the reports generated by the tool were

criticized as being too voluminous, and the admittedly small demands that the

tool made on computer resources were felt to Interfere with normal operation.

The Initial contact for this Interview had been made at a high management level,

and It was known from this conversation that there was above average

appreciation of software tools In this organization. There was an opportunity

to question the same manager at a later time about his evaluation of this tool.

He explained that the reports were very valuable to him because they clearly

Identified abnormal events In the computer center, and conversely, when these

reports showed no abnormal events, he was assured that the computer operations

were proceeding normally. He was well aware that this tool was not suitable for

fine-tuning an Individual program. In this case the difficulties perceived at

the working level were not due to shortcomings In the tool, but rather to lack

of information about its intended purpose,

5.3.3 Tools Primarily Used by Large Organizations.

For several tools used primarily by large and very large software organizations

difficulties were cited repeatedly by both the Immediate user and management.

Most test analyzers are in this category. The characteristic features of these

tools are instrumentation and coverage analysis. The most frequent complaint

was associated with run-time expansion, and problems were also voiced regarding

difficulties in use and restrictions on program size or structure. In spite of

these criticisms, test analyzers were regarded as useful tools because the

information which they furnished was Impossible to obtain by other means. The

problems cited should be evaluated In connection with any efforts aimed at

expanded use of software tools.

Requirements analyzers represent another generic type that was regarded as valuable but difficult to use by survey participants. Requirements analyzers

process a very high level language (VHLL) and their static analysis features

include completeness checking and consistency checking. These tools make

extensive demands on computer resources (the program for one of them was

reported as one million FORTRAN statements) and are usually operated by

specialist teams of four or five members. A tool of this type Is suitable only

for use by a very large software organization. Yet, automated requirements

analysis can make a very valuable contribution to software quality because

requirements errors are an ever present cause of software failures and one that

Is very costly to correct if it is not found in the early life cycle stages.

Requirements analyzers provide a particularly efficient means for handling

changes In requirements, with regard to checking for consistency with the

unchanged parts of the original requirements and assuring propagation Into

documents that Implement the requirements. An extensive review of requirements

by a specially convened team of experts can usually be arranged at the beginning

of a project, and this practice will not be completely eliminated, even If a requirements analysis tool is employed. However, such review teams cannot

usually be assembled when changes have to be phased In, frequently on short

notice, nor can they be depended on for perfect recall of the considerations
PAGE 26

that went into the original requirements formulation. Thus, even If there are

no economic benefits from a requirements analysis tool at the outset of a project. Its use will pay off over the years as the Inevitable changes In requirements are processed, and the established requirements data base can then

be utilized. Yet it Is difficult to obtain a commitment to use a requirements

analyzer if the benefits are, as they should be, described as accruing primarily

In the future. Hence the use of these tools Is sometimes being Induced by

unrealistic promises of Immediate payoffs. When these are not being achieved, a negative attitude Is being generated with regard to requirements analyzers and,

frequently, with regard to other software tools as well.

Since requirements analyzers are usually operated by spec i a 1 1 st teams, and

without interaction with the Immediate user, it should be possible to host them

on centralized facilities that can be accessed by multiple users on a fee for

service basis. Such an arrangement may remove some of the heavy start-up costs,

and it will make the services of these tools available to smaller users.

5.3.4 Special Purpose Tools

By their nature, special purpose tools can be expected to be well suited for the

environment in which they are employed, and user reaction is therefore

predictably favorable. Two types of tools encountered In the survey fall Into

the special purpose category: simulators and data base managers (for a unique

data base). The former are identified by the use of the simulation feature of

dynamic analysis (and usually very few additional features), the latter by

formatting, management and scanning features (this Identification is not

unique). Difficulties, insofar as they were mentioned, related to the

maintenance of these tools, particularly In the area of program documentation

and In the updating of user manuals.

Special purpose tools are typically developed as a temporary measure, or to

serve what is perceived to be a very short term need. If the tool Is

successful. It Is used on a continuing basis and requests will be received to

extend or adapt It for additional applications. The developer is enthusiastic

about the acceptance of the tool and seldom owns up to the fact that

considerable effort Is required to transform the hurriedly developed program

into a software product that can be maintained over a period of time or can serve multiple users. If the Inevitable difficulties are experienced, they are

frequently attributed to inherent problems in the use of software tools. A

definition of a basic programming format for a 'tool', and minimum standards for

both user and maintenance documentation may avoid some of these problems.
PAGE 27

SECTION 6

ADDITIONAL ANALYSIS OF THE SURVEY

This section provides a further analysis of selected questions from the tool user survey and an evaluation of the Impact of these findings on tool usage.

The subsections analyze, respectively, the tools acquisition process, the

experience with current tools, user statements about projected tool usage, and

user characteristics that may affect tool usage.

6.1 ACQUISITION OF TOOLS

A number of survey questions that deal with the tools acquisition process are analyzed for the purpose of capturing Information that may be useful for future

efforts In the Introduction of additional software tools. Specific headings

deal with the sources of current tools, the authority for tools procurement, and

the motivation for obtaining the tools.

6.1.1 Sources of Tools

Of 31 tools for which the source could be determined, 12 were developed

in-house. Three of these were special purpose simulations, and three others

also served rather specialized applications. The remaining six were general

purpose tools, and most of these were In use also In other organizations. The

high proportion of in house development is probably not typical for the overall

tool user population and may reflect that tool developers are lil<ely to be

overrepresented In a survey of this type. A summary of sources of tools that were procured from the outside Is shown in Table 6-1.

TABLE 6 - 1 SOURCES OF PROCURED TOOLS

Source

Open market

Pub I Ic domal

Computer vendor

Commissioned

Total

No. of Tools

7

5

5

2

19

The public domain category Includes tools acquired by a Government agency In the

survey from another Government agency. Both of the commissioned tools are dynamic analyzers that had been developed by private companies for specific

Government applications. The use of commissioned tools was encountered only in

Government agencies. There were no other significant distinctions in the source

of tools among the types or sizes of organizations.
PAGE 28

6.1.2 AcQuigitiPn Prpgedurgs

The authority for tools procurement rested primarily with local supervision

(defined as any level of supervision collocated with the tool user) as shown by

the summary of the authority for tools procurement presented In Table 6-2.

TABLE 6-2 AUTHORITY FOR TOOL PROCUREMENT

Authority No. of

Organizations

Local supervision 18

Headquarters 6

Comm I ttee 2

Diffuse 2

Tota I 28

The need to refer to headquarters was encountered In five Government

organizations and one Government-support organization. Committee approval was required In one private and one Government organization; In both cases the

committees were local. Diffuse authority was reported by two very large private

organizations. In addition to the normal authority shown In the table above,

two very large private organizations (both defense contractors) Indicated that

tools could be procured out of project funds If the contract so stated or

permitted. The project manager rather than the In-line supervisor then became

the procur^ent authority.

It was also quite apparent that Informal routes of tools procurement exist In large and very large organizations of all types. This Involves 'borrowing'

tools from a friend In another organization or through computer user groups or In-house development of tools without specific authorization. The latter

process does not always produce beneficial results. The tools thus developed

frequently lack adequate documentation and are not designed for portability. If

It Is attempted to apply them away from the native environment considerable

difficulties are experienced, and this may discourage further tool usage.

6.1.3 Motivation for Tools Acquisition

Motivation for tools acquisition was not expressed by the survey participants In ways that could be easily classified. In many cases several reasons were

undoubtedly at work at the same time. In only two organizations, both large and

private, was a deliberate cost-benefit study made prior to tools procurement or In-house development. In a few other Instances economic considerations were a factor In tools procurement, but no specific analysis was undertaken. At least

two of the survey participants mentioned that the tool was 'free', and It was

Implied that economic benefits were therefore obvious. This reasoning does not

account for the labor required to evaluate, adapt and Install the tool, or for

the training In Its use.
PAGE 29

In many cases the procurement or development of a tool was based on a clearly

Identified need, and economic factors were not explicitly considered. Examples

are pre-processors for structured programming, simulations, and management tools

that generate reports required or desired by a customer. In others, the tool

was perceived to present a better way of accomplishing a necessary task, and

technical management authorized Its procurement out of discretionary funds.

Program design languages and performance monitoring or optimization tools may be

acquired out of this motivation. In still other Instances the technical

challenge, the desire to Innovate, or the possibility of overcoming present

difficulties may lead to tool development or procurement. This was encountered

In the survey particularly with respect to overall software development

facilities and requirements analyzers.

6,2 TOOLS EXPERIENCE

The users were In general quite satisfied with the tools In current use. In only one case was a recently acquired tool characterized as unsatisfactory, and

that for some of Its Intended applications. This case Involved a dynamic

analysis tool procured by one Government agency for use by another In which user constraints had not been correctly Interpreted to the developer. Comparatively

minor modifications could probably have corrected most of the deficiency but a proper avenue for funding these was not currently available, A local 'tool smith'

would have been very effective In this case. Comments on non-use or abandonment

of tools, reliability/maintainability, adequacy of training and documentation,

and options used are analyzed Individually below.

6.2.1 Reasons for Non-Use or Abandonment

This question was asked with particular reference to tools used In the past.

Meaningful answers fall mostly Into three categories: the tool was too

difficult to use. It was functionally not suited, or a better tool was found.

The relative frequency of these reasons Is shown In Table 6-3. The last

category Includes two cases where documentation was unavailable or very

I nadequate.

TABLE 6-3 REASON FOR NON-USE OR ABANDONMENT OF TOOLS

Organ I zat Ions

Better tool available

Difficult to use Functional ly not suited

Other

9

5

4

3

Total 21
PAGE 30

6.2.2 Reliability/Maintainability of Tools

In only a few cases did the organization contacted during the survey maintain

the tool with regard to which this question was asked. In all of those cases

the maintenance was rated easy or without problems. The reliability of tools

was also generally rated high. Pertinent user comments have been classified

below as unqualified good, qualified good, and other. The latter category

includes the dynamic analysis tool mentioned in the introductory paragraph of

this subsection, and a computer simulator for which It was stated "There have

been some unexplained crashes. There Is no protection against Illegal

instructions". Typical comments that were translated Into qualified good are "Reliability Is now acceptable" and "No known bugs. Input syntax errors are not

always well diagnosed". There were no apparent differences In the reliability

ratings between various types or sizes of organizations.

TABLE 6-4 USER'S RELIABILITY RATING OF TOOLS

Rating No. of

Qrganl2;atlon?

Unqual I f led good 16

Qua I If led good 9

Other 2

Tot a I 27

6.2.3 Training and Documentation

The training provided for tool use was characterized as predominantly formal

(In-house or outside), sometimes formal, and predominantly Informal. The latter

frequently was restricted to making documentation available to potential users.

In the survey population as a whole the three types of training were pretty

evenly represented but small organizations relied primarily on Informal

training, while large ones preferred formal training. This Is shown In Table 6 - 5.

TABLE 6-5 TRAINING FOR INTRODUCTION OF TOOLS

Type of

Train Ina Total Smal

Size of Organization

1 Medium Large V. large

Forma 1 10 0 2 3 5

Var lab 1

11 1 4 3 3

1 nf orma

8 4 0 2 2

Total 29 5 6 8 10
PAGE 31

The type of organization did not have a significant effect on training method.

Large organizations are more likely to be general purpose tool users (see Table

5-1) and also depend more heavily on formal training. This produces an apparent correlation between formal training and general purpose tool usage

which is not considered to be a cause and effect relationship.

Specific comments on documentation were received for 30 tools. Of these, 13

were rated as good or equivalent (useful, helpful). Twelve were described as adequate or In other terms that indicated less than complete satisfaction. In five cases the documents were considered unsatisfactory by the user. Typical

comments In this category were "A User's Manual is available but it Is

considered marginal. It should be redone" or "Documentation for this tool is

not particularly good. It should be Improved". A manual that did not pertain

to an identified tool In the survey (and is not counted among the five responses

reported above) was characterized as unsatisfactory because "It did not warn the

user of all the special conditions". Documentation was the Item most criticized

by the survey population.

6.2.4 Tool Options Used

The use of options was described for 28 tools as shown In Table 6-6

TABLE 6-6 EXTENT OF TOOL OPTIONS USAGE

Extent No. of

Tool s

A! I options used 13

Partial use of options 8

Extent not known 6

No options available 1

Tota I 28

The first category contains some cases in which general purpose use was

apparently a rare occurrence. One user stated that 90 percent of the runs Involve only 10 percent of the available options. However, as long as a feature

Is used at least occasionally It was classified as 'used'. Partial use of

options was due to two causes: features that were not needed at al I In the

survey application, and features which might be desired but were awkward to use

or usurped excessive computer resources (this latter factor was mentioned with

regard to two dynamic analyzers). The large number of 'unknown' responses was

due to the fact that the Individual Interviewed did not use all of the options

and felt he could not, within reasonable effort, determine the extent of usage

In his entire organization.

There was slight evidence that partial use was more prevalent In Government

Installations than in the other types of organizations. This may be due to the
PAGE 32

presence of several dynamic analyzers with very comprehensive features among the

Government tools. Size of the organization did not seem to be a factor in

determining the extent of options use.

A number of the respondents indicated that options were removed at instal I at ion or after some experience had been gained with the tool. Note, however, that

several users would have liked more flexibility in the use of their tools (see

discussion of desired Improvements, 6.5.1). It may be concluded that when a new

tool Is Introduced the number of options should be kept to a minimum. This

simplifies training and will also reduce the computer resource requirements.

After experience has been gained with the basic tool function, the user may be

motivated to consider further options.

6.3 FUTURE TOOLS USAGE

Both the extension In the use of present tools and the acquisition of additional

tools are covered under this heading. About 80 percent of current users

Indicated a desire to Improve some features of their current tools, and all but one of the organizations contacted had plans for acquiring additional tools.

6.3.1 Improvements and Integration of Present Tools

Improvements in specific features of their current tools were mentioned in 15

organizations. The need for improved documentation has been discussed above and

is not repeated here. A classification of the desired Improvements Is shown In Table 6 - 7.

TABLE 6-7 DESIRED IMPROVEMENTS IN PRESENT TOOLS

Improvement No. of

Organizations

User Interface 5

Performance 4

Flexibility 4

Portability 2

Tota I 1

Examples of desired improvements in the user interface are (1) replacement of a very formal command language for a management tool with a less formal one, and

(2) provisions for forward and backward traceabllity between structured code and

FORTRAN in a preprocessor. The recommended performance Improvement Involved

processing speed and, in at least one case, the total size of the code. In the

flexibility area the suggestions Involved handling more language forms, removing

size restrictions on the code to be analyzed, and adaptation to new versions of

operating systems. The desire for portability was raised with regard to one

general software development system that is at present completely tied to a military host computer, and an improvement In portability was desired for a restructuring and analysis tool.
PAGE 33

The Issue of tools Integration was commented on with respect to 21 tools. For

five of these It was stated that they existed already In an Integrated

environment. Three of the tools had been developed from the outset as an Integrated system. In one case Integration of project management and test

records had been achieved around a dynamic analysis tool, and the remaining tool was a simulator of broad scope. The remaining comments on Integration were

classified as shown In Table 6-8,

TABLE 6-8 PROPOSED INTEGRATION

Type Qf I ntegration No. of

TqqU

With library 4

Development/test system 3

Non-specific 5

No Integration desired 4

Total 16

The suggested Integration with a library usually Involved the ability to Invoke

commands or data sets automatically, to Identify and store the results of the

tool's operation, and to compare various sets of tool outputs. These comments

were made with regard to editors and report generators. Integration Into a development/test systems was recommended for preprocessors and a test management

tool. A negative response on this question was furnished for several large

tools. Including requirements analysis and management tools.

6.3.2 Contemplated Acquisitions

Twenty of the responding organizations Indicated a desire to acquire tools In the near future. No distinction was made between desire at the technical level

and firm committment to procure the tool. The 28 tools Identified In these

responses are classified by purpose In Table 6-9. Tool features that might

serve these purposes are discussed later.

Source editing and control tools were primarily desired by small and medium

organizations, and test support tools primarily by large and very large

organizations. For all other classes In Table 6-2, size did not appear to be

a major determinant. The type of organization did not seem to affect the

purpose for which the tools were to be employed.
PAGE 34

TABLE 6-9 PURPOSE OF CONTEMPLATED TOOL ACQUISITIONS

PurpQ$e No. of

Tools

Source editing & control 5

Test support 5

Requirements analysis 4

Code analysis 4

Project management 3

Performance Improvement 3

Software development systems 2

Other 2

Tota I 28

Source editing and control tools Include sophisticated editors, library

managers, and file managers. These tools typically include some transformation

functions and the static analysis functions of management and scanning. Test

support tools all include the dynamic analysis functions of coverage analysis

and tracing, the static analysis function of structure checking, and usually a number of other static and dynamic analysis functions as well. Requirements

analysis tools are characterized by accepting very high level language as an Input on which a number of transformation and static analysis functions are performed. Tools for project management typically accept a data Input and

perform the static analysis functions of cost estimation, scheduling, and

management. Code analysis tools rely primarily on static analysis functions of

auditing, scanning, and structure checking. The static analysis tools were

desired to Improve both the uniformity and the portability of the code. The

performance analysis tools utilize dynamic analysis functions of timing and/or

tuning, and usually a number of static analysis functions as well. The two

tools in the 'other' category were both Intended for specialized purposes, one

for language research and the other one as a macro facility.

6.4 USER CHARACTERISTICS OF INTEREST FOR TOOL USAGE

Under this heading two user characteristics that were captured In the survey and

which may be pertinent to actual or potential tool usage are discussed: the

major software problems seen by the user, and the Involvement In standards and

professional activities.

6.4.1 Major Software Problems

Only a part of the survey population was asked to Identify what they regarded as their major software problem. Where there was contact with several levels of

management, the question was addressed to the highest management level.

Responses were obtained from 14 organizations (including the three volunteers)
PAGE 35

who identified 19 problem areas. These responses are classified In Table 6 - 10.

TABLE 6-10 MAJOR SOFTWARE PROBLEMS

Problem Organization Type

Total Private Govt. Govern

Support ment

Requirements 7 3 0 4

Mai ntenance 4 3 1 0

Personnel 3 2 0 1 Software engnrg. 3 2 0 1 Managmt, reports 2 1 0 1

Total 17 9 1 7

Inadequate requirements definition and traceablllty was Identified as a problem

In each of the four Government agencies which responded to this question. In the private sector It was less of a problem. The three organizations which

Identified It there were small software activities, although two were In large

corporations. In the other private organizations there Is either better

communication between user and developer, or the problem is not being

recognized. Where the private sector develops software for the Government,

inadequate requirements may be perceived as a Government problem rather than as

a software problem.

Requirements definition Is a very serious problem throughout the software

community. The above fragmentary data indicate that it Is being felt with

special severity In Government organizations. Current requirements analyzers

are technically capable of helping in this area but are not extensively employed

because of the personnel and computer resource requirements which they Impose

(see 5.3.3). This area deserves further Investigation because of the

potentially great benefits that can be derived from a systematic and, at least

partially, automated approach to requirements definition.

That maintenance was not seen as a problem area in the Government organizations

Is somewhat surprising. Two of the respondents contracted maintenance to

private companies and may thus have been shielded from this problem. Shortage

and skill levels of personnel were commented on by three organizations.

Software engineering practices were described as a problem In terms of the

difficulty of Introducing them and auditing for compliance. These may also be

regarded as personnel problems but they have been separated here because they

could be more directly resolved by the employment of tools than other personnel

problems. Inadequate management reports represent a similar category In that

the problem can be directly attacked by Introduction of proper tools.

None of the participants specifically mentioned reliability or downtime In

responding to this question. This may be due to the fact that comments on major

software problems In thel

environment were requested. The organizations were

software developers rather than users, and In their own operation was probably
PAGE 36

not severely impacted by software failures. At least two of the respondents

were working on software for reliability-critical applications. This points to

a possible Information gap in current software practice: the developer may be

aware that the software is for a critical application but receives no formal

feedback on the number of failures or the consequences of failures. Thus the

actual benefit of a test tool cannot be evaluated in that environment.

6.4.2 Standards and Professional Activities

Ten of the organizations contacted as part of the survey participated in

standards activities, either Governmental or voluntary (professional or trade

associations, ANSI, etc.). Nine of these were classified as general purpose

tool users In Section 5, and one as an intermediate tool user. This Is a significantly higher degree of tool use than is found In the survey population

(and presumably In the general software population) as a whole. It Is not

concluded that standards activities per se lead to tool usage. All but one of

the participants In standards activities were large or very large organizations,

and the exception was a tool developer. Thus their tool usage can be accounted

for by other factors. However, that standards and tools are closely related

must also be kept In mind. One user made a strong case for considering

standards as tools.

All of the participants in standards efforts were also active In the

professional area. In addition, seven organization that did not have a significant Involvement with standards indicated participation In professional

activities. Of these, four were classified as general purpose tool users and

three as Intermediate tool users in Section 5. All but two of the seven

additional participants In professional activities were also In categories that

had very high tool usage. The two exceptions were non-tool developers In small

organizations who were both classified as Intermediate tool users. These

findings do not Indicate a strong relationship between participation in either

standards or professional activities and tool usage.
PAGE 37

SECTION 7

INTERPRETATION OF FINDINGS

In this section the findings of the Survey of Software Tools Usage are

Interpreted In a broad context to define factors favorable to tool usage,

obstacles to tool usage, and requirements for effective tool usage derived from

these. The comments made here are reflections on the survey rather than

reporting of results, and they are presented for the purpose of stimulating

further evaluation of the survey findings on the part of the software community

as well as on the part of tool developers. Wherever possible, specific data

presented In earlier sections are referenced to support the Interpretations

discussed here.

7.1 FACTORS FAVORABLE TO TOOL USAGE

Factors that appear to promote the usage of software tools are considered here

under two headings: factors arising from current practice and additional factors

that can be expected to become significant In the near future.

7.1.1 Current Factors

Explicit economic analysis Is rarely used as a criterion for tools acquisition,

and It was not Identified as a determining factor by any of the 12 Government

organizations Included In the survey (see 6.1.3). The decision to acquire a tool Is most likely to be made by a local manager or committee who may take Into

account only that subset of cost and benefits which Is an Immediate constraint,

e. g., purchase or lease cost, or computer throughput (see 6.1.2). The tools

which received the most favorable comments, particularly In small organizations,

were those that met clearly perceived technical or management needs, e. g.,

precompilers for structured code and report generators. Where portability was emphasized, code auditors that checked for adherence to standard language syntax

were In the same category.

Another group of tools which received very favorable comments were those which

produced benefits In the Immediate area of concern to the tool user as pointed

out In 5.3.1. Use of these tools Is self-motivated, and there are few

obstacles to their Introduction as long as they are available In a suitable

format (for the computer, operating system, and programming language In use at a given facility), are reasonably well documented, and have a 'friendly' user

I nterf ace.

With regard to all of the tools mentioned above, there Is somewhat of an embarassment of riches: there are many good ones available, and several that

were functionally very similar were encountered In the comparatively small

population of this survey. Greater uniformity In tools selection among

Government and Government-support organizations would be beneficial In providing

uniformity of software practice and In facilitating transfer of code.
PAGE 38

documentation, personnel, and tools operating experience. It will also reduce

the cost of maintaining and Improving several sets of programs and

documentation.

7.1,2 Future Trends

Just as structured programming with 'non-structured' languages has made the use

of precompilers essential. It can be expected that further development of

software engineering practices and their adoption In organizational or broader

guidelines will lead to the acceptance of additional tools. This trend Is

already evident In the translation of design Into code. A design language and

two development systems that Include the equivalent of a design language were

encountered In the survey, and all of them received favorable comments from the

user.

Factors associated with software quality have been Identified In a number of

past studies, and metrics for these have been proposed [7,8]. In at least one

case, metrics of this type have been Included as a contractual requirement. The

evaluation of the metrics has been carried out by mostly manual means [9], but

If this practice spreads, software tools will undoubtedly be used extensively

for automated reporting of the metrics.

Cost modeling and reliability modeling are other areas In which large amounts of

data on software characteristics are required In order to validate or apply

current research. Software tools will be required to support the extensive data

collection. Further Impetus for tool usage may arise from the adoption of

naming conventions within large programs and data bases. Adherence to

conventions and verification of uniqueness of names can best be carried out by

software tool s.

Last but not least, the Increased Interest at high levels of project and

organizational management In gaining visibility and control of software

development Is expected to speed the Introduction of tools. Initially these may

be management oriented tools that generate timely reports on resource

utilization and software status. As a result of these efforts, certain software

practices will be recognized as beneficial, others as detrimental to efficient

development and design. Tools may be directly Introduced In order to support

the beneficial practices, or they may be Included In a broad effort to promote

uniformity of software development over larger aggregates of both Government and

private organizations. The survey participants frequently commented that

present software practices were fragmented but that there was perceptible

pressure or already existing directives toward greater uniformity.

7.2 OBSTACLES TO TOOLS USAGE

Obstacles to tool usage that were encountered during the survey are discussed

here under three headings: factors arising out of the computer environment,

organizational factors, and tool related factors.
PAGE 39

7.2.1 Obstacles In the Computer Environment

Some very elementary obstacles to the use of tools arose from the

Incompatibility of existing tools for a desired function with the computer

environment that wanted to use them. The tool may not be available for the

computer in use at that facility or for the operating system employed there, or

it may not have been able to handle the high order language and dialect In which

the programs were coded. In a few cases, the tool was compatible with respect

to these factors, but It required too much memory or Imposed too great a throughput penalty to be beneficial.

As far as new tools are concerned, these obstacles can be partly avoided by

greater emphasis on portability (see 7.2.3). For both existing and new tools,

the use of standard conversion packages or access to tools on a remote host are alternatives that need to be explored. One of the Government organizations

contacted In the survey mandates that all integration and testing be conducted

by remote access to Its own mainframe computers, using target computer

simulators that It supplies. This is a division of a defense agency that has

played a leading role in the standardization of both high order languages and

computers. The portability of both source and object code Is therefore no problem In this specific environment, and the tools compatibility problem has

been solved by having a permanent host for the tools.

7.2.2 Organizational Obstacles

The primary organizational obstacle to use of tools Is that many of the most

effective tools produce benefits that lie partly or entirely outside the scope

of the organization that Is responsible for their acquisition and use. The

following discussion focuses on requirements analyzers, test tools, and software

management systems. Several other tool types may be affected by the same considerations.

The potential benefits of requirements analyzers have already been mentioned

(see 6.4.1). The perceived obstacles to their use are (1) the personnel

required for Initial Installation and training, (2) the personnel required to

establish the requirements data base for each project, and (3) the computer

resources for performing requirements analysis. The first of these can be

overcome by funding from computer technology budgets which exist at most

agencies. In all cases where requirements analyzers are currently In use, both

In the private sector and in Government, the acquisition funding has come from

such a source, and a means for dealing with this difficulty is thus Identified.

The computer resources have been a serious obstacle in the past because both the

program and the data base for requirements analysis tend to be very large. With

higher performance computers, and with memory limitations being largely removed

by virtual memory techniques, this obstacle, too, may be overcome.

This leaves the second factor as the crucial one. The cost of establishing a data base has to be paid for out of funding In the early stages of a program

when budgets tend to be very slim. There Is some benefit from having an automated requirements analysis during the Initiation phase, but It Is not a compelling factor, and It is questionable whether It warrants the cost. The
PAGE 40

chief benefit of having a data base occurs when changes are Introduced later In the life cycle, and when the validation and documentation of these changes can

be accomplished automatically at a fraction of the cost and schedule that Is

required by present methods. Because of the high cost, validation and

documentation of software changes are sometimes omitted with very severe

consequences for the overal I project. The present organizational structure does

not usually provide Incentives for management to allocate resources during the

project Initiation phase to achieve benefits In later life cycle phases. Aside

from very broad directives (specific emphasis on this Item In life cycle cost

studies), no clear resolution Is currently envisioned. To provide for a more

comprehensive solution of this problem In the future, a creditable case study of

the use of a requirements analyzer through Initiation, development and

operations phases should be undertaken.

Similarly, the primary benefits from the use of test tools occur In the

operations phase whereas their cost has to be borne during development. Here,

again, the cost of the tool Itself and some of the start-up expenses may be

funded as part of a technology effort. But the use of a dynamic analyzer,

particularly If It performs a sophisticated coverage analysis, will Increase

both personnel and computer resource requirements during test, and the only

benefit to the test function may be knowledge that they have done a 'good job'.

Where test tools have been used consistently and successfully. It has usually

been either In the tool developer's own organization (or one affiliated with

It), or In an environment where the use has been mandated. Recently an Increasing number of test analyzers have been put Into use, and a there may be a tendency to consider a computer operation backward If It does not have one. However, there Is a difference between running a demonstration project and

efficient dally use of such a tool. Because of the large number of different

test tools In current use, and the somewhat differing objectives for their use,

conclusions about the merit of these as a group are not possible at this time.

Further study of functional, performance, and resource usage differences, and an overall assessment of their benefit, will be necessary If broad guidance on the

employment of test tools Is to be generated.

Software management systems as a minimum attempt to chart the progress of

development against milestones established at the project's Inception, In many

cases they also generate expected rates of expenditure on the basis of

historical trends or on the basis of Information gathered from the specific

software under development. The effort required to Initiate these management

systems frequently has to be borne by elements of the project organization which

are In close contact with the details of the program and who don't need a sophisticated tool to tell them whether the next scheduled review dates will

have to be postponed. The benefit of the tool Is felt at a much higher level of

management where the uniform format and the consistent standards for data

entries which are enforced by a software management system provide significant

savings In the review of multiple projects, A further benefit of the use of

these tools Is that they automatically generate a we I I -formatted data base that

can Improve cost estimation and scheduling of future projects. If these

benefits can be brought to the attention of the management level at which they are really targeted, a much more widespread acceptance of software management

systems may result.
PAGE 41

7.2.3 Obstacles Arising from the Tools

Not all tools provide a net benefit, regardless of the scope over which the

assessment is made. Difficulties reported with specific tools have already been

discussed in 6.2.1. In this connection, attention Is also called to a critical

evaluation of the benefits of maintenance tools In a recent publication [10],

The following addresses obstacles arising from generic features of some tools

categories.

Some tools automate fairly routine activities that are now being performed by

personnel of low skill levels, and will therefore reduce labor requirements In

that category. However, it was found that the use and maintenance of the tools

require additional personnel at high skill levels. Although there may be ample

economic Justification for the use of a tool in terms of overall labor

reduction, some agencies feel that they cannot provide the required skill level.

A frequent complaint was that Federal ADP organizations cannot attract or retain

highly skilled personnel whereas they can get lower skilled personnel or

trainees. This objection can be partly overcome If tools are shared among a number of agencies, and the effort for tool maintenance on the part of any one

agency can be proportionately reduced.

Comments made by users and non-users suggest that increased acceptance of tools

can be achieved if ease of use is specified as a primary criterion for tools

procurement. Tools are typically developed In very sophisticated organizations

who see the tool user as one who has at least an undergraduate degree in

computer science. This does not correspond to the present state of affairs, and

should not be expected to prevail In the future. Tool commands, interpretation

of output, and the general documentation must be addressed to personnel who have

either one year of on-the-job training or a Junior college education.

Although the use of a tool may provide labor or resource savings over the long

run, the Introduction of a new tool was In many organizations found to require

considerable personnel and computer time, and It therefore Impeded routine

operations over a short period. In some agencies, particularly the smaller

ones, the temporary unavailability of people and computing resources was

perceived to be a serious obstacle to investigating and using tools. It Is

undoubtedly true that most tools do not become fully effective as soon as they

are Introduced, and a temporary reduction of resource availability for routine

operations may therefore be unavoidable. This factor can be partly overcome by

providing good documentation and training aids (which will be facilitated If the

expense can be allocated among a number of users). To some extent, this must

also be addressed as an organizational problem, and a sufficiently high level of

management must be involved In tools acquisition to permit a reasonable

trade-off of long term benefits versus short term costs.

The final factor to be discussed In this category Is the usage of computer

resources by tools. This was not a significant factor with regard for most

users, but where It was mentioned as objectionable It was felt to be extremely

so. One major tool type was found to account for all of the adverse comments:

dynamic analysis tools, particularly coverage analyzers. By their nature, tools

In this category expand the run time of the programs which they analyze, and
PAGE 42

this was accepted by the user. However, when more than a lO-fold expansion

takes place, a serious curtailment of computer availability was experienced, and

the value of the tool — and of the Information furnished by It — was called

Into question. Some tools were reported to Increase the run time by a factor In excess of one 100! A careful, specialized study of this tool type, which has

already been suggested In 7.2,2, will be useful In clarifying and possibly

resolving this issue.

7.3 REQUIREMENTS FOR FUTURE TOOLS USAGE

This subsection summarizes the principal requirements for effective tool usage

that are derived from the above comments. The two headings deal, respectively,

with requirements that must be levied against the tools and requirements

pertaining to the environment, the latter Including organizational factors.

7.3.1 Requirements on Tools

A primary requirement for tools Is that they address a clearly recognized need

in the environment to which they are targeted. This means that a tool that

Improves software quality (reliability, maintainability, portability, or a related factor) will be efficiently applied in an organization that recognizes a responsibility for providing the quality factor supported by that tool. In any

Government sponsored future tool development It might be wel I to identify rather

specific objectives.

Unless the tool is intended for a single host application, portability is a very

important requirement. Past attempts at achieving portability have not been

uniformly successful. During the conduct of the survey it was related that a well-known portability auditor for FORTRAN did not fully comply with the

criteria that It enforced. Usually portability Is only achieved at a considerable throughput penalty, A suitable compromise might be to allow

machine-dependent code in identified sections of a tools program provided that

Its purpose and format are well documented so that a local toolsmlth can

accomplish required modifications without excessive effort.

That tools have to be useable by personnel of low skill levels was repeatedly

mentioned during the survey. This requires that a 'friendly' Interface be

provided for the control input and for the user output: descriptive names for

commands, prompts, and in output listings; the capability for corrections In the

tool set-up; comprehensive checking of the control input for syntax,

consistency, and completeness; and suitable diagnostics when irregularities are

detected in either the tool or control input.

The most frequently mentioned deficiency of existing tools is poor

documentation. To some extent this Is due to tools developed for local use being accepted on a wider scale, a development which should not be discouraged

per se. But the additional requirements for making It suitable for general use must be recognized and funded. As a minimum, tools for general use should have

the following documentation:
PAGE 43

Installation manual that Identifies computer resource requirements,

machine-dependent code, user settable variables, limits, etc., and which

Includes test cases.

User • s manual , If possible, organized In several levels for beginning

users and more advanced ones, with a detailed description of all user operations and commands, the effect of options on run-time and memory

requirements, and an explanation of diagnostics.

Systems manual for use by the tool smith or an advanced programmer for

making adaptations, trouble shooting, and performing minimal

ma I ntenance.

Maintenance manual If a central maintenance organization has not been

I dent I f led.

7.3.2 Environmental Requirements for Tools Usage

The following addresses primarily requirements for effective use of tools that

enhance software quality. While the preceding heading pointed out that a tool

should address a specific need for software quality, a corollary is that the

environment should define the quality factors which are considered Important In

a given application. If software maintenance Is a more significant Item than

software reliability, the acquisition of documentation tools may be preferred to

that of test tools.

A closely related consideration Is that those who use the tool and the tool

output should be aware of the objective that Is to be accomplished and should

have some metric available to determine that this objective Is being achieved.

Development and test groups are currently being tasked with using documentation

tools, code auditors, and test analyzers which support primarily maintenance and

operational reliability. In a few cases, organizations Interviewed as part of

the survey did not have an accurate understanding of the purpose of one of their

tools. But much more prevalent was the condition where the purpose was

understood yet a measure of accomplishment was not available. This Is

particularly true of test tools where no quantitative data on software

reliability are being collected.

As has been repeatedly stated, the benefits of tool usage must be explained at

the management level which stands most to gain. Because a significant benefit

of many tools Is that they provide uniformity and consistency, and facilitate

future software tasks, a management level concerned with those broader aspects

must be addressed.

Tools use within the Federal Government could be made more effective If a central clearinghouse for tools Information were available. The tools catalog

recently published by NBS/ICST and the software tools exchange being established

by GSA are a start In this direction. The collection and dissemination of tool

users' comments, the development of guidelines for tools Introduction and

Integration, the conduct of tools demonstration projects, and the comparative

evaluation of tools to eliminate duplicate development and maintenance
PAGE 44

activities (which were found In the survey particularly In precompilers and test

tools) are also required to facilitate efficient tools use. While none of the

Federal agencies contacted In the survey had conducted explicit cost/benefit

analyses regarding tools acquisition and use, most would have liked to have data

and techniques for such analyses made available to them. A tools clearinghouse

would be the logical source for supplying this Information which Is considered

essential for rational software practices In Federal agencies.
PAGE 45

SECTION 8

CONaUSIONS

This section summarizes findings of the survey that are significant for the

application of software tools to a broad range of Federal agencies. Conclusions

that affect Individual tools are discussed In Section 5, and those that affect

Individual user segments are discussed In Section 6.

8.1 OVERALL TOOL USAGE

All organizations contacted In the course of the survey used some software

tools. Where the tools set was restricted to those normally furnished as part

of the operating system (compilers, assemblers, debug aids, etc.) tool usage was classified as minimal. Three of 29 organizations were at this level. In other

organizations tool usage Included special purpose programs that were essential

for their mission (simulators and special purpose data base managers). These were classified as Intermediate tool users and comprised 8 of the 29 survey

participants. The remaining 18 organizations used at least one general purpose

tool In their regular operations. Although a few of the larger organizations

had tools that automated significant portions of software management, software

development, or test, no 'complete' tool usage was encountered. All

organizations contacted could have benefited from additional tool usage, and

many were taking steps In that direction.

The small sample size precludes a formal statistical evaluation of the

Information from the survey but two groups were clearly Identified as consistent

tool users: organizations that have more than 100 programmers and analysts

subject to a uniform software discipline, and tool developers. All survey

participants In either of these groups used general purpose tools. Among the

four size categories considered In this report (see 4.1), there was a consistent

Increase In general purpose tool usage with size. With regard to tool

development one Is faced with a chicken and egg problem: does tool usage lead

to tool development, or vice versa? The survey did not provide a firm answer to

this question. A high degree of sophistication and experience In software

development Is required to create a tool that can be successfully employed

outside Its native environment, and one can speculate that this experience must

In many cases have Included tool usage.

The survey Included three types of organizations: private sector. Government

support, and Government. When tool developers (which were found only among

private and Government-support organizations) were eliminated, there was no perceptible difference In tools utilization among these three types. In small

and medium organizations there was slightly higher tools utilization In the

private sector which may have been due to other characteristics of the specific

participants. Similarly, the finding that tools utilization was slightly higher

among scientific programming organizations than among MIS organizations need not

be general ly val Id.
PAGE 46

8.2 MOTIVATION FOR ACQUISITION OF TOOLS

Where motivation for tools acquisition could be clearly established. It usually

arose from a need which could not be readily met In any other way.

Precompilers, simulations, and some management tools (report generators) are

examples of this type of usage. Most tools acquisitions seemed to be motivated

by multiple factors, e. g., a tool Is described in the literature or seen In

operation, and it is recognized as offering a better way of accomplishing some present tasks. Where performance monitoring, software development systems or design languages were In use, their acquisition seemed to have followed this

pattern. Requirements analyzers and test tools may be Installed out of the same reasoning, and sometimes also because of the professional challenge which they

represent. The need In these areas can of course be demonstrated to management,

but the tool capabilities, the cost of Installing the tool, and the cost of the

operation and support activities are usually evaluated In a subjective manner,

if at all. In only two cases, both In commercial organizations, was a formal

cost benefit analysis made prior to tools acquisition.

The responsibility for tools acquisition was In the hands of local management in

approximately two- thirds of the organizations contacted. Approximately one-half

of the Government agencies and one Government-support organization needed to

refer the decision to higher (remote) headquarters. The localized control over

tools acquisition encourages experimentation, and this Is regarded as beneficial

at the present stage of tool development and application. It also promotes a proliferation of tool designs addressing a single need, and this Is not

beneficial, particularly among Government agencies, because It Inhibits the

development of a common software environment and the duplication causes

unnecessary costs.

8.3 BENEFITS OF TOOL USAGE

Some of the benefits of tool usage are evident to the user and need little

comment: the precompiler, the library system, and the optimization tool can be

evaluated on the basis of the service they render In the Immediate environment.

Many tools, and particularly those that can have the greatest Impact on software

quality and cost, achieve diffuse benefits not all of which are evident to the

Immediate user. Software development systems provide appreciable benefits In automating routine tasks of software development and permitting teams to

Interact In a systematic manner, and these are recognized by the development

group. They can also furnish management reports, help In documentation and

configuration management, and provide standardization of software development

within a large organization. The latter attribute. In turn, facilitates

transfer of personnel and activities, provides flexibility In project planning

and assignment of computer resources, and permits sharing of experience in the

use of the tool and in other aspects of software development which Is not

possible In present autonomous environments. These broader benefits need to be

brought to the attention of a management level where they will be fully

appreciated. The survey Indicates that this Is seldom the case at present.
PAGE 47

Requirements analyzers, test tools, and software management systems also provide

wide-ranging benefits, some of which will accrue only In much later life cycle

stages. The obstacles that arise from this have been described In 7.2. Here,

too. It Is necessary to alert high level management of the availability and

capabilities of the tools so that proper resources for their employment can be

made available. Another significant information gap that may Impact the proper

employment of tools is the lack of quantitative software reliability feedback

from the user to the developer. This was signaled In the survey by the fact

that no software developer Identified reliability or a related quality as a primary problem area although several were working on reliability-critical

programs and a fair number were using, or were getting ready to use, test

coverage analyzers.

8.4 POLICIES THAT PROMOTE TOOL USAGE

Before organizations can decide on the use of tools they must know about them.

In some cases that knowledge can by Itself motivate the tools use as discussed

in 8.2. The establishment of a software tools data base at NBS/ICST and a systematic classification of the tools listed there by means of the tools

features taxonomy are significant and desirable steps In that direction.

Once it knows about the tool, an organization may want documentation at various

levels of detail (see 7.3.1) and may benefit from the experience of a current

user. In some cases this information can be obtained from computer users'

organizations, from the vendor, or through professional acquaintances. Some of

these sources have an obvious bias, and none of them are universally available

(even the tool vendor may no longer support a tool). The software tools tools

exchange being established by GSA Is desirable for these reasons.

The training, 'hand-holding', and other Information transfer that Is required to

bring a tool to productive status within a new user environment are frequently

underestimated. Guidelines for these steps will be generated under a task that

will follow the survey reported on here.

Difficulties for many potential tool applications arise from the lack of

portability — the desired tool cannot be interfaced with the computer

Installation, the operating system, or the language used by the target

organization. Several avenues are available for Improving portability, but none produce Immediate results or are particularly cheap. Policies in this regard

need to be developed carefully.

The first possibility Is to make tools highly portable by coding In a standard

high order language and to program Interactions with the operating system (1/0

operations) In a symbolic manner. This involves a considerable loss of

efficiency because the non-standard features of high order languages are usually

provided to Improve efficiency, and because the symbolic designation of Input or output operations will slow down the operating system. Because many tools place

a heavy demand on computer resources even if coded for a specific host, this

additional requirement may Impair the usefulness of the tool.
PAGE 48

A second approach Is to restrict the commonality to the design of tools and to

have several versions, each coded for efficient execution on a different

computer. This obviously Increases the development expense and will require

considerable research In order to arrive at a suitable combination of tools and

computer adaptations.

Finally, network access to tools that are hosted on a suitable computer may be

considered. The most serious limitations of this approach will be encountered

In communication channel capabilities and In the ability of the tool to handle

language and data peculiarities arising from the target computer. The latter

factor may not present much difficulty If the program Is In a standard language

and uses standard data formats, e. g., COBOL with 32-blt data.

As an overall conclusion of this survey. It may be stated that an Interest by

high level management In software and a commitment to control software

development will be the most significant motivation for tool usage. It Is

presumed that this Interest exists now, or will soon exist, because of the

crucial dependence of most Government services on software and also because of

the cost, personnel, and management problems which have been reported [3]] and

can be expected to Increase In severity. The role of tools In providing

visibility of software development and test. In promoting uniform and systematic

procedures, and In providing flexibility of personnel and resource assignments

must be brought to the attention of the highest management levels so that tools

can be properly and speedily applied.
PAGE 49

REFERENCES

ClU D. J. Relfer and H. A. Montgomery, "Final Report, Software Tool Taxonomy",

Prepared for NBS/ICST under subcontract to SoHaR Incorporated by Software

Management Consultants, Torrance, CA, 1 June 1980.

Il2j Raymond C. Houghton, Jr., "Features of Software Development Tools", NBS

Special Publication 500-74, February 1981.

{J>2 The Comptroller General, "Wider Use of Better Computer Software Technology

Can Improve Management Control and Reduce Costs", FGMSD-80-38, General

Accounting Office, April 29, 1980.

C4!] H. Hecht, "Synopsis of Interviews from a Survey of Software Tool Usage",

forthcoming NBS Interagency Report, 1981.

[5 J S. L. Gerhart "Program Verification in the 1980s: Problems, Perspectives,

and Opportunities", ARPA Order No. 2223, Information Sciences Institute, Marina

del Rey CA, August 1978.

[6] S. L. Gerhart et a!., "An Overview of AFFIRM: A Specification and

Verification System" in Information Processing 80 (F. H. Lavington, ed.). North

Holland Publishing Co., 1980

[7] B. W. Boehm et al., "Characteristics of Software Quality", TRW Systems

Engineering and Integration Division, Redondo Beach CA, TRW-SS-73-09, Dec 73

[8] J. A. McCall, P. K. Richards, and G. F. Walters, "Factors in Software

Quality", RADC-TR-77-369, Nov 77

[9] G. F. Murine, "On the Selection of Requirements Analysis Metrics for a Distributed Network", Panel presentation at the Third Minnowbrook Workshop,

Syracuse University, Aug 1980

[10] B. P. Lientz and E. B. Swanson, "Impact of Development Productivity Aids

on Application System Maintenance", Data Base , Winter-Spring 1980, pp. 114-120
APPENDIX A

LETTER SOLICITING PARTICIPATION IN THE SURVEY

UNITED STATES DEPARTMEWT OF COMMERCE

fUational Bureau of Standards

Washington, D.C. 20234

The National Bureau of Standards' Institute of Computer

Science and Technology (ICST) has significantly increased its

effort in support of the Brooks Act (PL 89-306) which aims to aid

•ooyernment agencies to improve cost effectiveness in the

selection, acquisition and utilization of automatic data

processing resources. As part of these efforts, ICST has

contracted with SoHaR Incorporated to conduct a study of software

tool usage in both Government agencies and the private sector.

It is the purpose of this letter to solicit your participation

in this study.

The study is aimed at determining environmental factors

which inhibit or promote tool usage, and thus you may make a valuable contribution even if you currently do not use any tools . For publication of the results of this study outside ICST the

identification of the participants may be deleted, if desired.

SoHaR Incorporated is not engaged in tool development, marketing

or furnishing tool related services. They have also agreed

to safeguard any information which you consider proprietary in

the same manner in which it will be handled by a Government

agency

Unless you 'notify us to the contrary, you will be contacted

by SoHaR personnel during the April to June time period to set up

a discussion session with a person or persons knowledgeable in

your data processing activities. The discussions will be

conducted at your facility at a mutually convenient time and will

ordinarily be completed in less than one hour. Your cooperation

in this study will be helpful to the ADP community inside and

outside of Federal agencies and will be much appreciated.

Si ncerel y

,

Raymond C. Houghton, Jr.

Programming Science Division
PAGE 51

APPENDIX B

TOOL USER INTERVIEW

Part I - Tool User Information

1. Organizational Identification of user.

2. Approximate annual budget of programming and computer operation departments

(alternatively, the number of ana I yst/prgrammers and the number of computer

operators)

.

3. Relationship to ultimate software user, and the nature of user imposed

requirements (govt, standards, etc.).

4. Range of programmer skills.

5. On-site computer equipment and operating systems.

6. Off-site equipment, operating and communications systems pertinent to tools

uasage.

7. Programming languages now used and considered for the future.

8. Software requirements and design (formality, reviews, updates).

9. Programming environment (size of groups, on-line vs. batch).

10. Software quality control (organization, formality, standards).

11. Documentation (responslbl ity, formality, standards).

12. Changes and software maintenance (responsibility, procedures).

12A. What are the major software problems In this environment.

13. Participation in computer and software standards efforts.

14. Awareness of software engineering literature and meetings.

15. Circumstances of the first tool acquisition.

16. Tools currently In use.

17. Tools no longer In use.

18. Tools considered for future use.

19. Who decides on tool procurement and by what criteria,

20. How are new tools Introduced (formal and Informal training, documentation

only).
PAGE 52

Part II - Tool Information

21. Tool Identification, originator, and date of first use.

22. Tool owneership data (licensed, on time-share, etc.).

23. Original purpose and current appi ication(s) of tool.

24. Tool and target software data.

25. Tool usage data (set-up time, run time, analysis required)

26. Typical usage (case history),

27. Frequency and conditions of usage (customer specs),

28. Benefits of usage (vs. prior practice).

29. Standards serviced or enforced by the tool.

30. Principal original reason for tool acquisition; were other tools

considered, and what was the basis for selecting this tool.

31. Current reason for tool usage,; target software characteristics Important

for use (e, g., safety critical applications, very tight timing requirements),

32. Formal cost/benefit anlaysls If available.

33. Are all features of this tool used? Most and least used features,

34. Suggested tool Improvements.

35. Tool rel labl I Ity/aval I abl I Ity/mal ntal nabl I Ity.

36. General comments on tool use.

37. Recommendations for other app I I Icatl ons of this tool.

38. Compatibility of this tool with other tools and with the procedures of the

computing environment.

39. Are some functions of this tool also provided by other tools or general

support software? Where should these functions be located?

40. Suggestions for Integration of this tool Into a highly automated

environment or Into a library.

41. How will anticipated future software engineering developments affect the

usefulness of this tool.

42. Effect of tool usage on software quality.
PAGE 53

43. Effect of tool usage on personnel requirements.

44. What documentation was furnished with this tool? How useful is it?

45. Was special training provided with this tool?
NBS-lUA (REV. 2-BC) U.S. DEPT. OF COMM.

RiRi inriRAPHir data

SHEET (See instructions)

1. PUBLICATION OR

REPORT NO. MRc CD cnn-R?

2. Performing Organ. Report No. 3, Publication Date

Mri\/f^mh^r IQnl

4. TITLE AND SUBTITLE

Computer Science and Technology:

Final Report: A Survey of Software Tools Usage

5. AUTHOR(S)

Herbert Hecht

6. PERFORMING ORGANIZATION (If joint or other thon NBS, see /n s tru ction sj

NATIONAL BUREAU OF STANDARDS

DEPARTMENT OF COMMERCE

WASHINGTON, D C 20234

SoHaR Incorporated

1040 So. La Jol la Ave.

Los Angeles, CA 90035

7. Contract/Grant No.

8. Type of Report & Period Covered

F i na 1

9. SPONSORING ORGANIZATION NAME AND COMPLETE ADDRESS (Street. City. State, ZIP)

National Bureau of Standards

Washington, DC 2023^

10. SUPPLEMENTARY NOTES

Library of Congress Catalog Card Number: 81-600112

Document describes a computer program; SF-185, FlPS Software Summary, is attached.

11. ABSTRACT (A 200-word or less factual summary of most sigr)ificant information. I f document includes a si gnificant

bi bl iography or literature survey, mention it here)

The state of the art regarding the development and use of software

tools is presented. The information was gathered from an in-depth survey

of 23 sites with Government and industry. A comparison is made among the

levels of tool usage based on the size and type of programming group. The

survey examines aspects that both promote or inhibit tool usage and provide

requirements for the future.

12. KEY WORDS (Six to twelve entries; alphabetical order; capitalize only proper names; and separate key words by semicolons)

Programming aids; software automation; software development; software

engineering; software tools; software tools usage.

13. AVAILABILITY

\XJ Unlimited

(331 Fo'' Official Distribution. Do Not Release to NTIS

1X3] Order From Superintendent of Documents, U.S. Government Printing Office, Washington, D C

20402.

331 Order From National Technical Information Service (NTIS), Springfield, VA 22161

14. NO. OF

PRINTED PAGES

58

15. Price

$4.25

USCOMM-DC 6043-P80

t^'U.S. GOVERNMENT PRINTING OFFICE: 1 9 8 1-360- 997/ 1 858
ANNOUNCEMENT OF NEW PUBLICATIONS ON

COMPUTER SCIENCE & TECHNOLOGY

Superintendent of Documents,

Government Printing Office,

Washington, D. C. 20402

Please add my name to the announcement list of new publications to be issued in the series. National Bureau of Standards Special Publication 500-.

Name

Company

Address

City State Zip Code

(Notincation key N-S03)

I



NBS TECHNICAL PUBLICATIONS

PERIODICALS

JOURNAL OF RESEARCH—The Journal of Research of the National Bureau of Standards reports NBS research and develop- ment in those disciplines of the physical and engineering sciences in which the Bureau is active. These include physics, chemistry,

engineering, mathematics, and computer sciences. Papers cover a broad range of subjects, with major emphasis on measurement

methodology and the basic technology underlying standardization. Also included from time to time are survey articles on topics

closely related to the Bureau's technical and scientific programs. As a special service to subscribers each issue contains complete

citations to all recent Bureau publications in both NBS and non- NBS media. Issued six times a year. Annual subscription; domestic

$16; foreign S20. Single cop\. $3.75 domestic; $4.70 foreign. NOTE; The Journal was formerly published in two sections; Section A "Physics and Chemistry" and Section B "Mathematical

Sciences." DIMENSIONS/NBS—This monthly magazine is published to in- form scientists, engineers, business and industry leaders, teachers, students, and consumers of the latest advances in science and

technology, with primary emphasis on work at NBS. The magazine

highlights and reviews such issues as energy research, fire protection, building technology, metric conversion, pollution abatement,

health and safety, and consumer product performance. In addition, it reports the results of Bureau programs in measurement

standards and techniques, properties of matter and materials, engineering standards and services, instrumentation, and

automatic data processing. Annual subscription; domestic $11;

foreign SI 3.75. NONPERIODICALS

Monographs—Major contributions to the technical literature on

various subjects related to the Bureau's scientific and technical activities. Handbooks—Recommended codes of engineering and industrial practice (including safety codes) developed in cooperation with in- terested industries, professional organizations, and regulatory

bodies. Special Publications—Include proceedings of conferences sponsored by NBS, NBS annual reports, and other special publications

appropriate to this grouping such as wall charts, pocket cards, and

bibliographies. Applied Mathematics Series—Mathematical tables, manuals, and

studies of special interest to physicists, engineers, chemists,

biologists, mathematicians, computer programmers, and others engaged in scientific and technical work.

National Standard Reference Data Series—Provides quantitative data on the physical and chemical properties of materials, compiled from the world's literature and critically evaluated. Developed under a worldwide program coordinated by NBS under

the authority of the National Standard Data Act (Public Law

90-396).

NOTE; The principal publication outlet for the foregoing data is the Journal of Physical and Chemical Reference Data (JPCRD)

published quarterly for NBS by the American Chemical Society

(ACS) and the American Institute of Physics (AlP). Subscriptions,

reprints, and supplements available from ACS, 1 155 Sixteenth St., NW, Washington, DC 20056.

Building Science Series—Disseminates technical information developed at the Bureau on building materials, components,

systems, and whole structures. The series presents research results,

test methods, and performance criteria related to the structural and

environmental functions and the durability and safety characteristics of building elements and systems.

Technical Notes—Studies or reports which are complete in themselves but restrictive in their treatment of a subject. Analogous to monographs but not so comprehensive in scope or definitive in treatment of the subject area. Often serve as a vehicle for final reports of work performed at NBS under the sponsorship of other government agencies.

Voluntary Product Standards—Developed under procedures

published by the Department of Commerce in Part 10, Title 15, of

the Code of Federal Regulations. The standards establish nationally recognized requirements for products, and provide all concerned interests with a basis for common understanding of the

characteristics of the products. NBS administers this program as a supplement to the activities of the private sector standardizing

organizations. Consumer Information Series—Practical information, based on NBS research and experience, covering areas of interest to the con- sumer. Easily understandable language and illustrations provide

useful background knowledge for shopping in today's technological marketplace. Order the above NBS publications from: Superintendent of Documents. Government Printing Office, Washington, DC 20402. Order the following NBS publications—FIPS and NBSIR's—from

the National Technical Information Services, Springfield, VA 22161.

Federal Information Processing Standards Publications (FIPS

PUB)—Publications in this series collectively constitute the Federal Information Processing Standards Register. The Register

serves as the official source of information in the Federal Govern- ment regarding standards issued by NBS pursuant to the Federal Property and Administrative Services Act of 1949 as amended.

Public Law 89-306 (79 Stat. 1127), and as implemented by Executive Order 11717(38 FR 12315, dated May II, 1973) and Part 6 of Title 15 CFR (Code of Federal Regulations).

NBS Interagency Reports (NBSIR)—A special series of interim or

final reports on work performed by NBS for outside sponsors

(both government and non-government). In general, initial dis- tribution is handled by the sponsor; public distribution is by the National Technical Information Services, Springfield, VA 22161,

in paper copy or microfiche form.
U.S. DEPARTMENT OF COMMERCE

National Bureau of Standards

Washington. DC 20234 POSTAGE AND FEES PAID

U.S. DEPARTMENT OF COMMERCE

COM-215

OFFICIAL BUSINESS

Penalty for Private Use. $300

THIRD CLASS